Skip to contents

Replace identifiers with tokens that can be restored

Source: R/tokenise.R
tokenise.Rd

Replaces every detected identifier with a numbered token such as [PAN_1] or [PERSON_NAME_2], and records in vault which value each token stands for, so restore() can put the values back into a model's answer. The same identifier gets the same token however it is written (ABCPE1234F and abcpe1234f; +91 98765 43210 and 9876543210), across texts and across calls with the same vault.

Usage

tokenise(text, vault, detectors = NULL, context = NULL, min_confidence = 0)

Arguments

text

A character vector.

vault

A vault from cg_vault().

detectors

Detector ids to run; NULL runs all registered detectors (see cg_detectors()).

context

Optional extra context per element of text (recycled), treated as if it appeared next to every match. scan_pii() (planned) passes column names here, so a column called voter_id supports Voter ID matches.

min_confidence

Drop matches scoring below this value.

Value

A character vector like text, with identifiers replaced by tokens. NA stays NA.

See also

restore(), redact() for irreversible placeholders.

Examples

vault <- cg_vault()
safe <- tokenise(c("PAN ABCPE1234F, mobile 9876543210", "pan abcpe1234f again"), vault)
safe
#> [1] "PAN [PAN_1], mobile [MOBILE_IN_1]" "pan [PAN_1] again"                
restore("The customer with [PAN_1] can be reached on [MOBILE_IN_1].", vault)
#> [1] "The customer with ABCPE1234F can be reached on 9876543210."