Detection

detect_pii

detect_pii(text: str | Sequence[str | None] | Any, detectors: Sequence[str] | None = None, context: str | Sequence[str | None] | None = None, min_confidence: float = 0.0, window: int | None = None) -> list[Span]

Detect PII in one or more documents.

Runs every registered detector over normalised text, validates candidates (checksums, code tables), scores them from nearby context and resolves overlapping matches (the more confident, then the longer, span wins - so a GSTIN is reported once, not also as the PAN inside it).

Parameters:

Name	Type	Description	Default
text	str | Sequence[str | None] | Any	A string, a sequence of strings, a pandas or Polars Series, or an Arrow array; missing values count as empty. Span doc is the position in this sequence.	required
detectors	Sequence[str] | None	Detector ids to run; None runs all registered ones.	None
context	str | Sequence[str | None] | None	Extra context per document (one string for all), treated as if it appeared next to every match - e.g. a column name, so a column called voter_id supports Voter ID matches.	None
min_confidence	float	Drop matches scoring below this value.	0.0
window	int | None	Characters either side of a match searched for context keywords; None uses the spec's context_window.	None

Returns:

Type	Description
list[Span]	Spans ordered by document and position.

preview

preview(text: str | Sequence[str | None], detectors: Sequence[str] | None = None, context: str | Sequence[str | None] | None = None, min_confidence: float = 0.0) -> Preview

Show what would be redacted and why, without sending anything.

Printing the result shows the redacted text and each detected identifier (value masked) with its confidence and reasons.

Span dataclass

Span(doc: int, start: int, end: int, type: str, data_class: str, value: str, canonical: str, confidence: float, reasons: str)

One detected PII match.

start and end are 0-based, end-exclusive code-point offsets into the original text of document doc. The representation masks value; read the attribute to see it.

to_dict

to_dict(reveal: bool = False) -> dict[str, Any]

A JSON-ready dict; value is masked and canonical omitted unless reveal.

scan_pii

scan_pii(data: Any, sample: int = 10000, min_rate: float = 0.0, detectors: Sequence[str] | None = None, seed: int = 42) -> ScanReport

Scan the columns of a table for PII.

Reports, per column, which PII types occur and in what share of values - for example that cust_id looks like a PAN in 98% of rows. Each column is sampled (evenly spaced, so results are reproducible) and every sampled value is checked with :func:detect_pii, using the column name as context. Whole numbers are checked too; other numbers, booleans and dates are skipped.

Parameters:

Name	Type	Description	Default
data	Any	A pandas or Polars DataFrame, a Polars LazyFrame, an Arrow table or dataset, a DuckDB relation, or a mapping of column name to values. Lazy frames, datasets and relations are read one column at a time, so the table may be larger than memory. A PySpark DataFrame is sampled inside the cluster (up to sample rows, randomly with seed) and only the sample is collected.	required
sample	int	Maximum number of values checked per column.	10000
min_rate	float	Report a type only if it occurs in at least this share of sampled values.	0.0
detectors	Sequence[str] | None	Detector ids to run; None runs all registered ones.	None
seed	int	Random seed for sampling Spark DataFrames; ignored otherwise.	42

ScanReport

ScanReport(findings: Iterable[ColumnFinding], n_columns: int)

Bases: list[ColumnFinding]

Findings of :func:scan_pii; prints as a readable table.

to_records

to_records() -> list[dict[str, Any]]

One dict per finding, e.g. for pandas.DataFrame(report.to_records()).

ColumnFinding dataclass

ColumnFinding(column: str, type: str, data_class: str, n_sampled: int, n_matched: int, n_full: int, rate: float, mean_confidence: float, example: str)

One PII type found in one column.

mask_value

mask_value(x: str, data_class: str | None = None, mode: str | None = None) -> str

Replace letters and digits with X, keeping the last 4 characters (2 for values shorter than 8) - the same convention as UIDAI's masked Aadhaar. Values of the personal class (names, addresses) keep nothing, and neither does any value when the mask mode is "full" (see :func:set_mask).

set_mask

set_mask(mode: str | None) -> None

Choose how values are masked wherever credgate shows them (printed spans, previews, column-scan examples, the command line).

Parameters:

Name	Type	Description	Default
mode	str | None	"partial" (default) keeps the last 4 characters of identifiers, as in UIDAI's masked Aadhaar; "full" keeps nothing, for policies that allow no part of a value to be shown. None falls back to the CREDGATE_MASK environment variable, then to "partial".	required

normalise

Text normalisation before detection.

Detectors run on a normalised copy of the text so that trivial variations (Devanagari digits, fullwidth characters, zero-width joiners inside an identifier) cannot hide PII. Every normalised code point remembers which original code point it came from, so spans are always reported against the text the caller passed in. The handled code points are listed in the spec (engine.yaml, normalisation).

Normalised dataclass

Normalised(text: str, map: tuple[int, ...] | None)

Normalised text and, per code point, the index of the original one.

map is None when every code point maps to exactly one, so positions are unchanged.

normalise

normalise(text: str | None) -> Normalised

Normalise text for PII detection.

Applies NFKC per code point, maps Indic and Arabic-Indic digits to ASCII, maps dash variants to - and removes invisible characters.

Detectors

detectors

Detector registry.

A detector finds candidate spans with a regex, validates them (checksums, code tables), and scores them using surrounding context. Built-in detectors are defined in the shared spec (detectors.yaml) and compiled on first use; users can add their own with :func:register_detector.

Detector dataclass

Detector(id: str, data_class: str, label: str, patterns: Sequence[str], capture: str | None = None, validate: Callable[[str], bool] | None = None, checks: str | None = None, base: float | Callable[[str], float] = 0.8, base_without_context: float | None = None, context: str | None = None, negative: str | None = None, context_required: bool = False, canonical: Callable[[str], str] = canonical_default, describe: Callable[[str], str] | None = None, ignore_case: bool = False, column_label: str | None = None, column_patterns: Sequence[str] = ())

A PII detector.

Parameters:

Name	Type	Description	Default
id	str	Unique identifier, reported as the span type.	required
data_class	str	Data class used by policies, e.g. "identity_strong".	required
label	str	Human-readable name.	required
patterns	Sequence[str]	Regular expressions (Python regex syntax) for candidates; matches of all patterns are combined. Wrap them with :func:bounded so they do not match inside a longer token.	required
capture	str | None	Name of a capture group whose span is reported instead of the whole match.	None
validate	Callable[[str], bool] | None	Called with each matched string (normalised text); candidates returning False are dropped.	None
checks	str | None	Short description of what validate checks, added to the reasons.	None
base	float | Callable[[str], float]	Confidence (0-1) without supporting context, or a function of the matched string returning it.	0.8
base_without_context	float | None	Confidence (0-1) used instead of base when no context keyword is found; for patterns that are strong evidence with context but weak without it.	None
context	str | None	Regex (case-insensitive) of keywords that raise confidence when they appear near the match.	None
negative	str | None	Regex (case-insensitive) of keywords that lower confidence.	None
context_required	bool	Drop matches without supporting context.	False
canonical	Callable[[str], str]	Maps a match to its canonical form (used for consistent tokenisation).	canonical_default
describe	Callable[[str], str] | None	Returns an extra reason for a match.	None
ignore_case	bool	Match the patterns case-insensitively.	False
column_label	str | None	Regex (case-insensitive) that, when it matches the caller's extra context (a column name), makes that context the label for column_patterns: for detectors whose patterns need a label in the text, so a column named card_cvv labels its values.	None
column_patterns	Sequence[str]	Patterns used, reporting the whole match, only when column_label matches the extra context.	()

canonical_default

canonical_default(x: str) -> str

Remove spaces and hyphens and upper-case.

bounded

bounded(core: str) -> str

Wrap a pattern so it only matches whole tokens.

The match may not be preceded or followed by a letter or digit, nor by a digit joined with a space or hyphen (so a 12-digit window inside a 16-digit card number is not reported as an Aadhaar number).

register_detector

register_detector(detector: Detector, overwrite: bool = False) -> str

Register a detector; returns its id.

reset_detectors

reset_detectors() -> None

Restore the built-in detectors, removing user-registered ones.

detectors

detectors() -> list[dict[str, Any]]

Registered detectors, one dict per detector.

expand_fragments

expand_fragments(x: str | None, depth: int = 0) -> str | None

Replace {{name}} with the fragment, or the list joined by |.

compile_detector

compile_detector(spec: dict[str, Any]) -> Detector

Build a :class:Detector from one entry of detectors.yaml.

builtin_detectors

builtin_detectors() -> list[Detector]

The built-in detectors, compiled from the spec.

Detector dataclass

Detector(id: str, data_class: str, label: str, patterns: Sequence[str], capture: str | None = None, validate: Callable[[str], bool] | None = None, checks: str | None = None, base: float | Callable[[str], float] = 0.8, base_without_context: float | None = None, context: str | None = None, negative: str | None = None, context_required: bool = False, canonical: Callable[[str], str] = canonical_default, describe: Callable[[str], str] | None = None, ignore_case: bool = False, column_label: str | None = None, column_patterns: Sequence[str] = ())

A PII detector.

Parameters:

Name	Type	Description	Default
id	str	Unique identifier, reported as the span type.	required
data_class	str	Data class used by policies, e.g. "identity_strong".	required
label	str	Human-readable name.	required
patterns	Sequence[str]	Regular expressions (Python regex syntax) for candidates; matches of all patterns are combined. Wrap them with :func:bounded so they do not match inside a longer token.	required
capture	str | None	Name of a capture group whose span is reported instead of the whole match.	None
validate	Callable[[str], bool] | None	Called with each matched string (normalised text); candidates returning False are dropped.	None
checks	str | None	Short description of what validate checks, added to the reasons.	None
base	float | Callable[[str], float]	Confidence (0-1) without supporting context, or a function of the matched string returning it.	0.8
base_without_context	float | None	Confidence (0-1) used instead of base when no context keyword is found; for patterns that are strong evidence with context but weak without it.	None
context	str | None	Regex (case-insensitive) of keywords that raise confidence when they appear near the match.	None
negative	str | None	Regex (case-insensitive) of keywords that lower confidence.	None
context_required	bool	Drop matches without supporting context.	False
canonical	Callable[[str], str]	Maps a match to its canonical form (used for consistent tokenisation).	canonical_default
describe	Callable[[str], str] | None	Returns an extra reason for a match.	None
ignore_case	bool	Match the patterns case-insensitively.	False
column_label	str | None	Regex (case-insensitive) that, when it matches the caller's extra context (a column name), makes that context the label for column_patterns: for detectors whose patterns need a label in the text, so a column named card_cvv labels its values.	None
column_patterns	Sequence[str]	Patterns used, reporting the whole match, only when column_label matches the extra context.	()

register_detector

register_detector(detector: Detector, overwrite: bool = False) -> str

Register a detector; returns its id.

reset_detectors

reset_detectors() -> None

Restore the built-in detectors, removing user-registered ones.

bounded

bounded(core: str) -> str

Wrap a pattern so it only matches whole tokens.

The match may not be preceded or followed by a letter or digit, nor by a digit joined with a space or hyphen (so a 12-digit window inside a 16-digit card number is not reported as an Aadhaar number).