Redaction and tokens¶
redact ¶
Irreversible redaction and previews (the spec's "Redaction" rules).
Preview
dataclass
¶
Preview(text: list[str], redacted: list[str], spans: list[Span])
What :func:redact would do: the redacted texts and the spans behind them.
placeholder ¶
placeholder(type_: str) -> str
The placeholder that replaces a span of this type, e.g. [PAN].
redact ¶
redact(text: str | Sequence[str | None] | Any, detectors: Sequence[str] | None = None, context: str | Sequence[str | None] | None = None, min_confidence: float = 0.0) -> Any
Replace every detected identifier with a placeholder naming its type.
For example "PAN ABCPE1234F" becomes "PAN [PAN]"; everything else
is kept exactly. Redaction is irreversible.
text can be a string, a sequence of strings, a pandas or Polars
Series, or an Arrow array; the result has the same type (a Series keeps
its index and name). Missing values stay as they are.
preview ¶
preview(text: str | Sequence[str | None], detectors: Sequence[str] | None = None, context: str | Sequence[str | None] | None = None, min_confidence: float = 0.0) -> Preview
Show what would be redacted and why, without sending anything.
Printing the result shows the redacted text and each detected identifier (value masked) with its confidence and reasons.
Preview
dataclass
¶
Preview(text: list[str], redacted: list[str], spans: list[Span])
What :func:redact would do: the redacted texts and the spans behind them.
tokenise ¶
Reversible tokenisation (the spec's "Tokenisation" rules).
UnknownTokenError ¶
Bases: ValueError
Raised by a strict :func:restore when text holds tokens the vault does not know.
TokenCollisionError ¶
Bases: ValueError
Two different values got the same keyed token (or the same token stands for different values in two vaults being merged).
VaultFileError ¶
Bases: ValueError
Raised when a vault file cannot be opened or saved: wrong passphrase, damaged or altered file, or not a vault file.
Vault ¶
Vault(key: str | bytes | None = None, *, keyed: bool = False)
The mapping between tokens and the values they stand for.
A vault starts empty and grows as :func:tokenise meets new
identifiers; the same identifier always gets the same token. It lives in
memory only. Printing it masks the values.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
str | bytes | None
|
A token key (hex text or bytes, at least 32 bytes) for keyed
tokens such as |
None
|
keyed
|
bool
|
Use keyed tokens with the key from |
False
|
key_fingerprint
property
¶
key_fingerprint: str | None
The fingerprint of this vault's token key, or None if numbered.
entries ¶
entries(reveal: bool = False) -> list[dict[str, Any]]
The vault's entries in order of creation; values are masked unless reveal.
merge ¶
merge(other: Vault) -> Vault
Add the entries of another keyed vault made with the same key (for example by a parallel worker). Returns this vault.
save ¶
save(path: str | PathLike[str], passphrase: str | None = None) -> None
Save the vault to an encrypted file, so its tokens can be restored later.
The passphrase comes from passphrase, else the environment variable
CREDGATE_VAULT_PASSPHRASE, else a prompt; it needs at least 12
characters and is never stored. Without it the file cannot be opened.
The file is written readable only by you; an existing file is
replaced only if it is itself a vault file.
open
classmethod
¶
open(path: str | PathLike[str], passphrase: str | None = None, key: str | bytes | None = None) -> Vault
Open a vault saved with :meth:save. Tokenising with it continues
its numbering, and :func:restore can put its values back.
A keyed vault needs its token key again (key, else
CREDGATE_TOKEN_KEY); the key itself is never saved.
tokenise ¶
tokenise(text: str | Sequence[str | None], detectors: Sequence[str] | None = None, context: str | Sequence[str | None] | None = None, min_confidence: float = 0.0) -> str | list[str | None]
Same as :func:tokenise with this vault.
new_token_key ¶
new_token_key() -> str
A new random token key, as 64 hex digits. Keep it like a password: whoever holds it can check a guessed value against a token.
key_fingerprint ¶
key_fingerprint(key: str | bytes | None = None) -> str
16 hex digits that identify a token key without revealing it
(None uses CREDGATE_TOKEN_KEY).
tokenise ¶
tokenise(text: str | Sequence[str | None] | Any, vault: Vault, detectors: Sequence[str] | None = None, context: str | Sequence[str | None] | None = None, min_confidence: float = 0.0) -> Any
Replace every detected identifier with a numbered token such as [PAN_1].
The vault records which value each token stands for, so
:func:restore can put the values back into a model's answer. The same
identifier gets the same token however it is written, across documents
and across calls with the same vault.
text can be a string, a sequence of strings, a pandas or Polars
Series, or an Arrow array; the result has the same type. Missing values
stay as they are.
restore ¶
restore(text: str, vault: Vault, strict: bool = False) -> str
Put the values behind tokens back into text, such as a model's answer.
Tokens are recognised with any letter case, with spaces inside the
brackets ([ pan_1 ]) and without brackets (PAN_1). Tokens the
vault does not know are left unchanged; with strict, a bracketed
unknown token (one the model invented or altered) raises
:class:UnknownTokenError.
restore ¶
restore(text: str, vault: Vault, strict: bool = False) -> str
Put the values behind tokens back into text, such as a model's answer.
Tokens are recognised with any letter case, with spaces inside the
brackets ([ pan_1 ]) and without brackets (PAN_1). Tokens the
vault does not know are left unchanged; with strict, a bracketed
unknown token (one the model invented or altered) raises
:class:UnknownTokenError.
Vault ¶
Vault(key: str | bytes | None = None, *, keyed: bool = False)
The mapping between tokens and the values they stand for.
A vault starts empty and grows as :func:tokenise meets new
identifiers; the same identifier always gets the same token. It lives in
memory only. Printing it masks the values.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
str | bytes | None
|
A token key (hex text or bytes, at least 32 bytes) for keyed
tokens such as |
None
|
keyed
|
bool
|
Use keyed tokens with the key from |
False
|
key_fingerprint
property
¶
key_fingerprint: str | None
The fingerprint of this vault's token key, or None if numbered.
entries ¶
entries(reveal: bool = False) -> list[dict[str, Any]]
The vault's entries in order of creation; values are masked unless reveal.
merge ¶
merge(other: Vault) -> Vault
Add the entries of another keyed vault made with the same key (for example by a parallel worker). Returns this vault.
save ¶
save(path: str | PathLike[str], passphrase: str | None = None) -> None
Save the vault to an encrypted file, so its tokens can be restored later.
The passphrase comes from passphrase, else the environment variable
CREDGATE_VAULT_PASSPHRASE, else a prompt; it needs at least 12
characters and is never stored. Without it the file cannot be opened.
The file is written readable only by you; an existing file is
replaced only if it is itself a vault file.
open
classmethod
¶
open(path: str | PathLike[str], passphrase: str | None = None, key: str | bytes | None = None) -> Vault
Open a vault saved with :meth:save. Tokenising with it continues
its numbering, and :func:restore can put its values back.
A keyed vault needs its token key again (key, else
CREDGATE_TOKEN_KEY); the key itself is never saved.
tokenise ¶
tokenise(text: str | Sequence[str | None], detectors: Sequence[str] | None = None, context: str | Sequence[str | None] | None = None, min_confidence: float = 0.0) -> str | list[str | None]
Same as :func:tokenise with this vault.
new_token_key ¶
new_token_key() -> str
A new random token key, as 64 hex digits. Keep it like a password: whoever holds it can check a guessed value against a token.
key_fingerprint ¶
key_fingerprint(key: str | bytes | None = None) -> str
16 hex digits that identify a token key without revealing it
(None uses CREDGATE_TOKEN_KEY).
TokenCollisionError ¶
Bases: ValueError
Two different values got the same keyed token (or the same token stands for different values in two vaults being merged).
UnknownTokenError ¶
Bases: ValueError
Raised by a strict :func:restore when text holds tokens the vault does not know.
VaultFileError ¶
Bases: ValueError
Raised when a vault file cannot be opened or saved: wrong passphrase, damaged or altered file, or not a vault file.