go-ruby-unicode-normalize

Ruby's String#unicode_normalize in pure Go โ€” the four UAX #15 forms, MRI-compatible, no cgo.

pure Go ยท zero cgo unicode_normalize stdlib Normalize / IsNormalized NFC ยท NFD ยท NFKC ยท NFKD Unicode 17.0.0 x/text core + patch MRI byte-exact 100% coverage 6 arches
Documentation GitHub
Documentation (MkDocs Material + mike) License: BSD-3-Clause Go 1.26.4+ Coverage 100%

go-ruby-unicode-normalize is a pure-Go (no cgo) reimplementation of Ruby's String#unicode_normalize / String#unicode_normalized? โ€” the unicode_normalize standard library โ€” matching MRI 4.0.5 (Unicode 17.0.0) byte-for-byte. It normalizes UTF-8 text to any of the four Unicode Standard Annex #15 forms (NFC, NFD, NFKC, NFKD) and reports whether a string is already normalized โ€” without any Ruby runtime. The heavy lifting is delegated to the pure-Go golang.org/x/text/unicode/norm, with a small override table patching exactly the Unicode 16.0/17.0 characters x/text leaves un-normalized on the go 1.26.4 floor, so the result matches MRI on every toolchain. It is the Unicode-normalization backend for go-embedded-ruby, a standalone sibling of go-ruby-regexp, go-ruby-marshal and go-ruby-yaml โ€” differential-tested against MRI, 100% coverage, CI green across 6 arches and 3 OSes.

Normalize ready

Normalize(s, form) returns UTF-8 text in any of the four UAX #15 forms โ€” NFC, NFD, NFKC, NFKD โ€” matching Ruby’s String#unicode_normalize byte-for-byte (NFC is the default form, mirroring Ruby’s :nfc).

IsNormalized ready

IsNormalized(s, form) reports whether a string is already in the given form โ€” Ruby’s String#unicode_normalized? โ€” without rewriting the bytes.

The four UAX #15 forms ready

NFC / NFD / NFKC / NFKD as Form values mapping to Ruby’s :nfc / :nfd / :nfkc / :nfkd; the zero value is NFC, and Form.String returns the Ruby symbol name for diagnostics and the rbgo binding.

x/text core ready

Heavy lifting delegated to the pure-Go (cgo-free) golang.org/x/text/unicode/norm; both MRI 4.0.5 and x/text target the same Unicode standard, so they agree on Hangul composition, ligatures, full/half-width folding and combining-mark ordering.

Unicode 17.0.0 patch ready

x/text gates its Unicode 17.0.0 tables behind the go1.27 build tag and otherwise falls back to 15.0.0; an override table patches exactly the Unicode 16.0/17.0 characters that fallback would leave un-normalized, so the result matches MRI on every toolchain (and is a no-op on go1.27+).

MRI oracle & coverage ready

Deterministic ruby-free tests hold statement coverage at 100%; differential oracle tests sweep every assigned scalar value ร— all four forms against the system ruby. Green across all six 64-bit Go arches and three OSes, cgo disabled.

A faithful port of Ruby's unicode_normalize stdlib in pure Go, cgo disabled, so it cross-compiles and embeds anywhere. It normalizes UTF-8 to the four UAX #15 forms and reports normalization, delegating to the cgo-free x/text core and patching the Unicode 16.0/17.0 characters needed for MRI parity. Validated differentially against the system ruby binary across every assigned scalar value and all four forms. It is a standalone, reusable module and the normalization backend for the sibling org github.com/go-embedded-ruby.