Detects personal data before it leaves the machine, with Indian identifiers built in: Aadhaar (with Verhoeff checksum), Aadhaar Virtual ID, PAN, GSTIN, voter ID, passport, driving licence, CKYC, ration card and NREGA job card numbers, UPI IDs, IFSC, bank accounts, payment cards with expiry dates and verification codes, +91 mobile numbers, ABHA, and names and addresses in Latin and Indic scripts.
The detection rules live in a shared, language-neutral spec, so this package and the Python package credgate find exactly the same matches. Part of the CredGate project; early development.
Use it with local LLMs (Ollama, vLLM)
Redact before the prompt reaches the model; the model only ever sees placeholders such as [PAN] and [PERSON_NAME]:
note <- "Customer name: Priya Nair. PAN ABCPN1234K, mobile +91 98765 43210, wants the EMI date moved."
preview(note, min_confidence = 0.5) # what will be masked, and why
library(credgate); library(ellmer)
safe <- redact(note, min_confidence = 0.5)
chat_ollama(model = "gemma3:1b")$chat(paste("Summarise:", safe))
# vLLM: chat_vllm(base_url = "http://localhost:8000/v1", model = "<served model>")The vignette vignette("local-llms", package = "credgate") walks through Ollama, vLLM, data frames and evaluation; runnable scripts are in the examples folder.
Check the preview. Names are found next to an honorific, relation or field label (
Shri,S/o,Name:,नाम:); a name in running text without such a cue (“Customer Priya Nair called”) is not detected yet. Redaction is irreversible; to get the real values back in the model’s answer, tokenise instead (see below). An automatic proxy is on the roadmap.
How it compares
| On 700 held-out synthetic documents | Precision | Recall | F1 | Time |
|---|---|---|---|---|
| credgate | 1.000 | 0.961 | 0.980 | 0.3 s |
| Presidio, India-configured | 0.609 | 0.754 | 0.674 | 11.6 s |
| Presidio, default | 0.565 | 0.498 | 0.529 | 12.1 s |
About the data: the 700 documents are synthetic - KYC forms, bank messages, call-centre notes, Hinglish chats, Hindi text and hospital summaries generated by benchmark/generate_corpus.py and included in the repository under benchmark/corpus/. No real personal data was used. Types both tools cover, exact span matching, thresholds tuned on a separate dev split for every system; time is for 1,400 documents. Synthetic data generated by us, so read with that in mind.
Where Presidio does better. credgate is built for Indian identifiers and Indian-language context, and that is what this benchmark measures. Presidio is broader and more mature:
- Voter IDs without context: Presidio scores 1.000 here, credgate 0.679 (credgate reports a bare voter ID format at low confidence, 0.3, because it could equally be a product code).
- Many more entity types: international IDs (US, UK, EU, …), IBAN, IP and MAC addresses, URLs, crypto wallets, dates, nationality and religion - credgate detects none of these.
- Names in running text: Presidio’s NER finds names without an honorific, field label or “ji”; credgate does not yet.
- Anonymisation: Presidio can mask, hash or encrypt with a key, and add custom operators; credgate redacts, or tokenises with an in-memory vault that can be saved encrypted (no hashing or encryption operators).
- Beyond text: image (OCR) and structured-data redaction, a REST service and Docker images.
- Maturity and extensibility: years in production, many languages and NLP back-ends, custom recognisers; credgate is pre-alpha.
- The corpus favours credgate: we generated it; on general English text Presidio may well do better.
Full method, per-type results and how to reproduce: benchmark/README.md. The comparison is independent and not affiliated with Microsoft.
Installation
CRAN submission of 0.1.0 is under review. Until it is accepted, install the development version from R-universe or GitHub:
# Development version, prebuilt for Windows, macOS and Linux (R-universe):
install.packages("credgate",
repos = c("https://dwarpal.r-universe.dev", "https://cloud.r-project.org"))
# or from GitHub (needs build tools on Windows and macOS):
# install.packages("remotes")
remotes::install_github("dwarpal/credgate", subdir = "pkg/credgate")With Nix: nix build github:dwarpal/credgate#credgate-r.
Usage
library(credgate)
x <- detect_pii("Name: Priya Nair; PAN: ABCPN1234K; UPI: priya.n@oksbi")
x
#> 3 PII matches (values masked; use `x$value` to see them):
#> doc start end type value confidence
#> 1 7 16 person_name XXXXX XXXX 0.80
#> 1 24 33 pan XXXXXX234K 0.97
#> 1 41 53 upi_id XXXXX.X@Xksbi 0.99
x$reasons[2]
#> [1] "matched PAN pattern; context 'PAN'; holder type: individual"detect_pii() takes a character vector and returns one row per match with the document index, character positions, type, data class, value, canonical form (for consistent tokenisation), confidence and reasons. Printing masks the values.
Context makes generic patterns trustworthy. Pass extra context, such as a column name, with context:
detect_pii("ABC1234567") # nothing: could be anything
detect_pii("ABC1234567", context = "voter_id") # Voter ID, confidence 0.97Indic digits, fullwidth characters and zero-width characters are normalised before matching, and positions refer to your original text:
detect_pii("आधार: ५४२२ २९३७ ११८२")$type
#> [1] "aadhaar"Redacting and previewing
redact() replaces each identifier with a placeholder naming its type; preview() shows what would be redacted and why, before anything is sent:
redact("Name: Priya Nair; PAN: ABCPN1234K")
#> [1] "Name: [PERSON_NAME]; PAN: [PAN]"
preview("Name: Priya Nair; PAN: ABCPN1234K; order no 878245671885")
#> 3 items would be redacted:
#> Name: [PERSON_NAME]; PAN: [PAN]; order no [AADHAAR]followed by a table of each placeholder, the masked value, its confidence and the reasons (here the order number scores only 0.42 because of the word “order”; use min_confidence to leave such matches alone). Both are new after 0.1.0: install the GitHub version to use them.
Tokenising and restoring
Redaction is one-way. To get the real values back in a model’s answer, tokenise() replaces identifiers with numbered tokens and keeps the mapping in a vault; restore() puts the values back:
vault <- cg_vault()
tokenise(c(
"Shri Ramesh Kumar, PAN ABCPE1234F, mobile +91 98765 43210",
"pan abcpe1234f called again from 9876543210"
), vault)
#> [1] "Shri [PERSON_NAME_1], PAN [PAN_1], mobile [MOBILE_IN_1]"
#> [2] "pan [PAN_1] called again from [MOBILE_IN_1]"
restore("[PERSON_NAME_1] (PAN [pan_1]) will be called on MOBILE_IN_1.", vault)
#> [1] "Ramesh Kumar (PAN ABCPE1234F) will be called on +91 98765 43210."The same identifier gets the same token however it is written, so the model can still tell customers apart. restore() accepts tokens as models tend to copy them (other case, spaces, missing brackets); strict = TRUE stops on a token the vault does not know. The vault prints with values masked.
To restore tokens in a later session - or in Python - save the vault to a file encrypted with a passphrase (at least 12 characters; from the argument, the CREDGATE_VAULT_PASSPHRASE environment variable, or a prompt):
vault_save(vault, "loans.cgvault") # needs the sodium package
vault <- vault_open("loans.cgvault") # later: same tokens, numbering continuesThe passphrase is never stored: without it the file cannot be opened. Keep *.cgvault files out of version control.
Keyed tokens make the same identifier get the same token everywhere - across sessions, datasets, machines and R or Python - so tokenised tables can still be joined and counted, and parallel workers agree (needs the openssl package):
key <- cg_new_token_key() # store it like a password, e.g. in CREDGATE_TOKEN_KEY
a <- cg_vault(key)
b <- cg_vault(key)
tokenise("PAN ABCPE1234F", a) # "PAN [PAN_bpg46o6x465d4]" (depends on the key)
tokenise("pan abcpe1234f", b) # the same token
vault_merge(a, b) # combine vaults from parallel workersWhoever holds the key can confirm a guessed value by computing its token, so keep it out of code and project files. The key is never saved in a vault file; vault_open(path, key = ...) needs it again.
Scanning data frames
scan_pii() reports which columns hold PII and how much, without sending anything anywhere. (New after 0.1.0: until the next CRAN release, install the GitHub version to use it.) Column names count as context, so a column called voter_id or cust_pan supports those types:
customers <- data.frame(
cust_pan = c("ABCPE1234F", "AAACR5055K", "XYZPQ9876R"),
notes = c("Called, Aadhaar 542229371182", "no issues", "mobile 9876543210")
)
scan_pii(customers)
#> Scanned 2 columns; PII found in 2: cust_pan and notes
#> column type rate matched whole confidence example
#> cust_pan pan 100% 3 of 3 3 0.97 XXXXXX234F
#> notes aadhaar 33% 1 of 3 0 0.97 XXXXXXXX1182
#> notes mobile_in 33% 1 of 3 0 0.93 XXXXXX3210whole counts values that are entirely the identifier (an ID column) rather than containing it (free text). Large columns are sampled (sample = 10000 values by default).
Checksums and test data
verhoeff_valid("542229371182") # Aadhaar / VID checksum
luhn_valid("4111 1111 1111 1111") # payment cards
gstin_valid("27AAPFU0939F1ZV") # GSTIN check character
synth_indian_pii(5, seed = 1) # synthetic records for tests and demosSynthetic values pass the same validation as real ones but are random; use them for tests, never as anonymised data.
Your own detectors
register_detector(new_detector(
id = "employee_id", class = "internal", label = "Employee ID",
pattern = bounded("EMP[0-9]{6}")
))
cg_detectors() # all registered detectorsLarge data: Arrow, Parquet, DuckDB
Tables too big for memory are read in chunks. Install the packages you use (arrow, duckdb, DBI, dbplyr):
scan_pii(arrow::open_dataset("calls.parquet")) # one column at a time
redact_parquet("calls.parquet", "calls_safe.parquet", columns = "notes")
con <- DBI::dbConnect(duckdb::duckdb())
scan_pii(dplyr::tbl(con, "calls")) # any dbplyr table
redact_query(con, "SELECT call_id, notes FROM calls", "calls_safe")Any DBI database works the same way, for example Postgres:
con <- DBI::dbConnect(RPostgres::Postgres(), dbname = "lending", host = "db.internal")
scan_pii(dplyr::tbl(con, "loan_notes")) # needs dbplyr
redact_query(con, "SELECT id, notes FROM loan_notes", "loan_notes_safe")redact(), tokenise() and detect_pii() also accept Arrow arrays. See vignette("large-data") and the large-data guide.
Policies
A policy decides, per destination, what happens to each identifier: allow, redact, tokenise, or block the whole request. The same YAML file works in Python:
policy <- cg_policy("bank-default.yaml")
d <- policy_apply(policy, c("Card 4111 1111 1111 1111 exp 12/27", "CVV 123"), "claude")
d$allowed # FALSE
d$blocked_by # "card-with-cvv-and-expiry"policy_decide() reports the decision without rewriting; policy_apply() also rewrites allowed requests (pass vault = when the policy tokenises). Loading checks everything: an unknown field, tier, action or type is an error. An example policy is in the repository’s examples/policies/bank-default.yaml.
Audit log
Policy decisions can be recorded in an append-only, hash-chained log: who sent which kinds of data to which endpoint, under which policy revision and rule - never the text or the values. Editing, deleting or reordering records is detected; with an audit key, identifiers are kept as keyed hashes; card verification codes are never recorded. R and Python share the format.
log <- cg_audit_log("audit.jsonl") # key from CREDGATE_AUDIT_KEY, if set
policy_apply(policy, messages, "claude", audit = log) # recorded, allowed or blocked
audit_verify("audit.jsonl") # intact? and the head hash
audit_read("audit.jsonl") # the records, as a data frameMasked output
Wherever credgate shows a detected value (printed matches, preview(), scan_pii() examples), it masks it:
| Value | Shown as |
|---|---|
| Identifiers (Aadhaar, PAN, mobile, account, card, …) | last 4 characters kept: XXXX XXXX 1182, XXXXXX234F
|
| Names and addresses | masked in full: XXXXX XXX
|
Keeping the last 4 characters lets you check which value matched, and stays within the display limits of UIDAI’s masked Aadhaar and PCI DSS (cards). If your policy allows no part of a value to be shown, mask everything:
options(credgate.mask = "full") # or CREDGATE_MASK=full in ~/.RenvironMasking changes only what is displayed; detection and redact() ([AADHAAR], [PERSON_NAME]) are unaffected.
Limits
Names and addresses are found only next to a cue: an honorific (Shri, Smt, श्रीमती, or “ji” after the name), a relation (S/o, D/o), a field label (Name:, पता:) or a house/village keyword followed by a PIN code. A deity addressed with “ji” (“Ram ji”) is taken as a name. A bare 12-digit number may pass the Aadhaar checksum by chance and is reported with lower confidence.