Applies NFKC normalisation, maps Indic and Arabic-Indic digits to ASCII,
maps dash variants to - and removes invisible characters (zero-width
space/joiners, BOM, soft hyphen). Returns the normalised text together
with an offset map back to the original.
Arguments
- x
A character vector.
Value
A list with one element per element of x, each a list with
text (the normalised string) and map (an integer vector giving, for
each code point of text, the index of the original code point it came
from, or NULL when the text is unchanged).
Examples
cg_normalise("Aadhaar: \u0968\u0969\u096a\u096b")
#> [[1]]
#> [[1]]$text
#> [1] "Aadhaar: 2345"
#>
#> [[1]]$map
#> [1] 1 2 3 4 5 6 7 8 9 10 11 12 13
#>
#>