Vulnerability Class

Trojan Source: Catching Hidden Bidirectional Unicode in Code Review and CI

Bidirectional control characters can make source code read differently to a reviewer than to the compiler. What the Trojan Source research and UTS #55 say, plus scanners for JavaScript, Python and Go and safe identity normalisation.

AppSec Brief · ·
JavaScriptPythonGo
Trojan SourceUnicodebidihomoglyphcode reviewCVE-2021-42574JavaScriptPythonGo

Code review assumes that what you see is what the compiler gets. Unicode bidirectional (bidi) control characters break that assumption: the compiler reads source in logical order, while an editor or diff viewer may display it in a different visual order. This is the class of attack called Trojan Source. We chose it as an evergreen topic because no existing article here covers it, and no fresh news item in the last 48 hours met our bar for developer relevance.

What the research says

Per trojansource.codes, the work is by Nicholas Boucher and Ross Anderson of the University of Cambridge, published at USENIX Security 2023 (August 2023). The site tracks two CVEs: CVE-2021-42574 for the bidirectional reordering attack and CVE-2021-42694 for the homoglyph variant, where visually confusable characters make distinct identifiers look identical. It states that compilers and interpreters process the logical order of source code, not the visual order.

The recommended mitigations on that page are:

  • Compilers, interpreters and build pipelines should warn or error on unterminated bidi control characters in comments or string literals, and on identifiers mixing confusable scripts.
  • Language specifications should formally disallow unterminated bidi controls in comments and string literals.
  • Editors and repository front ends should make these characters visible.

GitHub acted on the last point. Its 31 October 2021 changelog describes a warning shown on files containing bidirectional Unicode text and tells reviewers to open flagged files in an editor that reveals hidden characters.

The bidi controls to look for

The explicit directional formatting characters are the embeddings and overrides at U+202A to U+202E and the isolates at U+2066 to U+2069. (We listed these code point ranges from our own knowledge of the Unicode standard; the pages we opened name them by abbreviation: LRE, RLE, PDF, LRI, RLI, PDI and so on.) The attack pattern is to hide such a character inside a string literal or comment so that code after it displays in a misleading order, for example making a check appear to be inside a comment or a string appear to end earlier than it does.

Our assessment: the risk is highest where a reviewer sees code only in a web diff and where contributions come from outside the team, such as open-source pull requests, vendored code and AI-generated patches.

What the Unicode standard recommends

UTS #55, Unicode Source Code Handling is more nuanced than “ban everything”. It states in section 1.2.3 that the solution is not to forbid the directional formatting characters, since legitimate right-to-left text needs them. Instead it recommends:

  • Displaying explicit controls visibly by their abbreviations (section 4.2.2).
  • Warning when an explicit control’s effect extends across syntactic atoms (section 5.1.6).
  • Detecting confusables with the UTS #39 skeleton and checking mixed scripts per identifier chunk rather than whole identifiers (sections 5.1.1 and 5.1.2).
  • Not silently normalising string literal contents, and using normalization form C for case-sensitive languages’ identifiers (section 3.1.1).

Practical reading: a blanket CI ban is reasonable for a codebase with no RTL-language strings, but if you localise into Arabic or Hebrew, scan and allow-list instead.

A CI scanner

This Node.js script fails the build when a bidi control appears in a file. We ran it against a file containing U+202E (it reported the line and exited 1) and a clean file (exit 0).

// scan-bidi.mjs - usage: node scan-bidi.mjs $(git ls-files)
import { readFileSync } from "node:fs";
const BIDI = /[‪-‮⁦-⁩]/u;
let bad = false;
for (const file of process.argv.slice(2)) {
  readFileSync(file, "utf8").split(/\r?\n/).forEach((line, i) => {
    const m = line.match(BIDI);
    if (m) {
      console.log(`${file}:${i + 1}: U+${m[0].codePointAt(0).toString(16).toUpperCase()}`);
      bad = true;
    }
  });
}
process.exit(bad ? 1 : 0);

Python and Go equivalents follow. These two are untested: no Python or Go toolchain was available when we wrote this, so review and test them before relying on them.

# scan_bidi.py (untested)
import pathlib, sys

BIDI = {chr(c) for c in (*range(0x202A, 0x202F), *range(0x2066, 0x206A))}

def scan(path):
    text = path.read_text(encoding="utf-8", errors="replace")
    for n, line in enumerate(text.splitlines(), 1):
        for ch in line:
            if ch in BIDI:
                yield f"{path}:{n}: U+{ord(ch):04X}"

hits = [h for p in sys.argv[1:] for h in scan(pathlib.Path(p))]
print("\n".join(hits))
sys.exit(1 if hits else 0)
// scanbidi.go (untested)
package main

import (
	"bufio"
	"fmt"
	"os"
)

func isBidi(r rune) bool {
	return (r >= 0x202A && r <= 0x202E) || (r >= 0x2066 && r <= 0x2069)
}

func main() {
	bad := false
	for _, path := range os.Args[1:] {
		f, err := os.Open(path)
		if err != nil {
			fmt.Fprintln(os.Stderr, err)
			os.Exit(2)
		}
		sc := bufio.NewScanner(f)
		for n := 1; sc.Scan(); n++ {
			for _, r := range sc.Text() {
				if isBidi(r) {
					fmt.Printf("%s:%d: U+%04X\n", path, n, r)
					bad = true
				}
			}
		}
		f.Close()
	}
	if bad {
		os.Exit(1)
	}
}

Run the scanner as a required pull request check, and add a pre-commit hook so contributors see failures before review. Remember it covers only bidi controls; confusable identifiers need a UTS #39-based tool.

The application-side cousin: identity confusion

The same visual-versus-logical gap shows up in your running application. If usernames or emails are compared as raw strings, two visually identical values can be different accounts, or different-looking values can collapse into one after a later normalisation step. A short Node.js check (we ran it) shows how compatibility normalisation folds a fullwidth letter:

const canon = (s) => s.normalize("NFKC").toLowerCase();
console.log("admin" === "adｍin");              // false
console.log(canon("admin") === canon("adｍin")); // true

The pitfall is inconsistency: if your uniqueness check runs before normalisation and your login lookup runs after, an attacker can register a lookalike that later resolves to a victim. Normalise once, at the boundary, store the canonical form, and enforce uniqueness on it. For handles shown to other users, also reject mixed-script values using the UTS #39 mechanisms. Note that UTS #55 advises against silently normalising string literals in source code; that advice concerns language design, not your user table.

Checklist

  1. Add a CI scan for bidi controls and fail on any hit outside an explicit allow-list.
  2. Treat the GitHub bidi warning on a pull request as a blocker until someone has inspected the raw characters.
  3. Review dependencies and AI-generated patches the same way.
  4. Normalise identities once, at input, and enforce uniqueness on the stored canonical form.
  5. Check recent compiler and linter releases for built-in bidi diagnostics; we did not verify per-tool support here.

Sources