Appendix B — Character tables

Normative. Generated by gentables.py from the rules in §5; regenerate rather than hand-edit. Ranges are inclusive, hexadecimal, LO-HI or a single codepoint.

These tables exist so that normalization does not depend on a regular-expression engine’s Unicode property vocabulary, which is not portable (the Rust regex crate rejects \p{Cs}; neither Go nor Java can express \p{Dash}), and does not change when a runtime’s Unicode version changes.

B.1 SPLIT — 104 codepoints, 55 ranges

Each becomes U+0020 SPACE in step 3, and therefore a hyphen in step 4.

Derived from \p{Dash}, \p{Quotation_Mark} and \p{White_Space} as of Unicode 16.0, plus ampersand, period, comma and apostrophe, plus the script-specific terminators and separators listed below that are otherwise Po and would be discarded: U+2044 FRACTION SLASH, U+3001 IDEOGRAPHIC COMMA, U+3002 IDEOGRAPHIC FULL STOP, U+060C ARABIC COMMA, U+061B ARABIC SEMICOLON, U+06D4 ARABIC FULL STOP, U+0964/U+0965 DEVANAGARI DANDA and DOUBLE DANDA, U+104A/U+104B MYANMAR SIGN LITTLE SECTION and SECTION, U+17D4/U+17D5 KHMER SIGN KHAN and BARIYOOSAN, U+1362 ETHIOPIC FULL STOP, U+0E5A/U+0E5B THAI ANGKHANKHU and KHOMUT.

Without those additions, punctuation disposition varies arbitrarily by script: 東京、大阪 would join into one word while 東京。大阪 split into two, and U+2044 would fuse ½ into the digits 12.

0009-000D    0020         0022         0026-0027    002C-002E    0085         00A0         00AB
00BB         058A         05BE         060C         061B         06D4         0964-0965    0E5A-0E5B
104A-104B    1362         1400         1680         17D4-17D5    1806         2000-200A    2010-2015
2018-201F    2028-2029    202F         2039-203A    2044         2053         205F         207B
208B         2212         2E17         2E1A         2E3A-2E3B    2E40         2E42         2E5D
3000-3002    300C-300F    301C-301F    3030         30A0         FE31-FE32    FE41-FE44    FE58
FE63         FF02         FF07         FF0D         FF62-FF63    10D6E        10EAD

B.2 VIRAMA — 69 codepoints, 58 ranges

Canonical_Combining_Class 9. Licenses a following ZWJ or ZWNJ (§5.1).

094D         09CD         0A4D         0ACD         0B4D         0BCD         0C4D         0CCD
0D3B-0D3C    0D4D         0DCA         0E3A         0EBA         0F84         1039-103A    1714-1715
1734         17D2         1A60         1B44         1BAA-1BAB    1BF2-1BF3    2D7F         A806
A82C         A8C4         A953         A9C0         AAF6         ABED         10A3F        11046
11070        1107F        110B9        11133-11134  111C0        11235        112EA        1134D
113CE-113D0  11442        114C2        115BF        1163F        116B6        1172B        11839
1193D-1193E  119E0        11A34        11A47        11A99        11C3F        11D44-11D45  11D97
11F41-11F42  1612F

B.3 ARABIC — 448 codepoints, 4 ranges

Arabic-script blocks. A ZWNJ between two characters in these ranges is kept (§5.1). The Arabic Presentation Forms blocks are absent because NFKC folds them before §5.1 runs.

0600-06FF    0750-077F    0870-089F    08A0-08FF

B.6 CASEFOLD — 1557 entries

Complete Unicode Default Case Folding, as a per-character table.

An implementation MUST fold from this table and MUST NOT use its runtime’s lowercase operation. Case mapping and case folding are different operations, and every runtime tested diverges from folding somewhere:

runtime	operation	divergence
Go	`strings.ToLower`	simple mapping: `İ` U+0130 → `i`, where folding gives `i` + U+0307
Go	`strings.ToLower`	`ΟΔΟΣ` → `οδοσ` without the Final_Sigma context rule
Java	`toLowerCase()`	locale-sensitive: `ISTANBUL` → `ıstanbul` under `tr_TR`
all	lowercase	`ß` is preserved, where folding expands it to `ss`

104 of the 1557 entries expand to more than one character.

The table is emitted per language by gencode.py. The Python reference implementation uses str.casefold() instead, which is definitionally Default Case Folding; the differential test in difftest.py therefore checks the table against a standard implementation rather than against itself.

B.4 Folds

Applied to letters and numbers in step 3.

from	to	reason
U+02BC MODIFIER LETTER APOSTROPHE	U+02BB MODIFIER LETTER TURNED COMMA	Visually identical; no orthography distinguishes them. Uzbek standardised on U+02BB, Cherokee and several transliteration schemes use U+02BC. Without the fold they are aliases that never match.

Any Unicode decimal digit is folded to its ASCII equivalent inside a flag group (§6.2). Digits elsewhere in an alias are kept as written.

B.5 Elided by name

codepoint	reason
U+0640 ARABIC TATWEEL	Decorative elongation for justification, not a letter. `مـــرحبا` and `مرحبا` are the same word.

B.7 DIGITS — 760 entries

Every Unicode decimal digit, mapped to its ASCII equivalent. Used only inside flag groups (§6.2): a generator folds, a reader accepts. Digits elsewhere in an alias are kept as written.