README.md (4214B)
1 An implementation of grapheme cluster boundaries from [Unicode text segmentation](https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries) (UAX 29), for Unicode 17. 2 3 [](https://pkg.go.dev/github.com/clipperhouse/uax29/v2/graphemes) 4  5  6 7 ## Quick start 8 9 ``` 10 go get github.com/clipperhouse/uax29/v2/graphemes 11 ``` 12 13 ```go 14 import "github.com/clipperhouse/uax29/v2/graphemes" 15 16 text := "Hello, 世界. Nice dog! 👍🐶" 17 g := graphemes.FromString(text) 18 19 for g.Next() { // Next() returns true until end of data 20 fmt.Println(g.Value()) // Do something with the current grapheme 21 } 22 ``` 23 24 _A grapheme is a “single visible character”, which might be a simple as a single letter, or a complex emoji that consists of several Unicode code points._ 25 26 ## Conformance 27 28 We use the Unicode [test suite](https://unicode.org/reports/tr41/tr41-36.html#Tests29). 29 30  31  32 33 ## APIs 34 35 ### If you have a `string` 36 37 ```go 38 text := "Hello, 世界. Nice dog! 👍🐶" 39 g := graphemes.FromString(text) 40 41 for g.Next() { // Next() returns true until end of data 42 fmt.Println(g.Value()) // Do something with the current grapheme 43 } 44 ``` 45 46 ### If you have an `io.Reader` 47 48 `FromReader` embeds a [`bufio.Scanner`](https://pkg.go.dev/bufio#Scanner), so just use those methods. 49 50 ```go 51 r := getYourReader() // from a file or network maybe 52 g := graphemes.FromReader(r) 53 54 for g.Scan() { // Scan() returns true until error or EOF 55 fmt.Println(g.Text()) // Do something with the current grapheme 56 } 57 58 if g.Err() != nil { // Check the error 59 log.Fatal(g.Err()) 60 } 61 ``` 62 63 ### If you have a `[]byte` 64 65 ```go 66 b := []byte("Hello, 世界. Nice dog! 👍🐶") 67 68 g := graphemes.FromBytes(b) 69 70 for g.Next() { // Next() returns true until end of data 71 fmt.Println(g.Value()) // Do something with the current grapheme 72 } 73 ``` 74 75 ### ANSI escape sequences 76 77 By the UAX 29 specification, ANSI escape sequences are not grapheme clusters. To treat 7-bit ANSI escape sequences as a single cluster, set `AnsiEscapeSequences` to true. 78 79 ```go 80 text := "Hello, \x1b[31mworld\x1b[0m!" 81 g := graphemes.FromString(text) 82 g.AnsiEscapeSequences = true 83 84 for g.Next() { 85 fmt.Println(g.Value()) 86 } 87 ``` 88 89 To also parse 8-bit C1 controls (non-UTF-8 bytes), set `AnsiEscapeSequences8Bit` to true. 90 91 ```go 92 g.AnsiEscapeSequences = true // 7-bit forms (ESC ...) 93 g.AnsiEscapeSequences8Bit = true // 8-bit C1 forms (0x80-0x9F), not valid UTF-8 94 ``` 95 96 For ESC-initiated (7-bit) control strings, only 7-bit terminators are recognized. 97 For C1-initiated (8-bit) control strings, only C1 ST (`0x9C`) is recognized as ST. 98 99 We implement [ECMA-48](https://ecma-international.org/publications-and-standards/standards/ecma-48/) control codes in both 7-bit and 8-bit representations. 8-bit control codes are not UTF-8 encoded and are not valid UTF-8, caveat emptor. 100 101 ### Benchmarks 102 103 ``` 104 goos: darwin 105 goarch: arm64 106 pkg: github.com/clipperhouse/uax29/graphemes/comparative 107 cpu: Apple M2 108 109 BenchmarkGraphemesMixed/clipperhouse/uax29-8 142635 ns/op 245.12 MB/s 0 B/op 0 allocs/op 110 BenchmarkGraphemesMixed/rivo/uniseg-8 2018284 ns/op 17.32 MB/s 0 B/op 0 allocs/op 111 112 BenchmarkGraphemesASCII/clipperhouse/uax29-8 8846 ns/op 508.73 MB/s 0 B/op 0 allocs/op 113 BenchmarkGraphemesASCII/rivo/uniseg-8 366760 ns/op 12.27 MB/s 0 B/op 0 allocs/op 114 ``` 115 116 ### Invalid inputs 117 118 Invalid UTF-8 input is considered undefined behavior. We test to ensure that bad inputs will not cause pathological outcomes, such as a panic or infinite loop. Callers should expect “garbage-in, garbage-out”. 119 120 Your pipeline should probably include a call to [`utf8.Valid()`](https://pkg.go.dev/unicode/utf8#Valid).