Integrations¶

Spark DataFrames work directly with scan_pii and pii_inventory; BigQuery has its own functions.

Arrow and Parquet¶

Redact Arrow tables and streams, and Parquet files larger than memory.

Requires pyarrow (pip install 'credgate[arrow]'). Text columns are processed batch by batch, so memory use depends on the batch size, not the size of the data. :func:credgate.redact and :func:credgate.tokenise also accept Arrow arrays directly.

text_columns ¶

text_columns(schema: Schema) -> list[str]

Names of the string columns in a schema.

redact_batch ¶

redact_batch(batch: RecordBatch, columns: Sequence[str] | None = None, **options: Any) -> pa.RecordBatch

Redact the given columns (default: all string columns) of one record batch.

options are passed to :func:credgate.redact (detectors, min_confidence).

redact_table ¶

redact_table(table: Table, columns: Sequence[str] | None = None, **options: Any) -> pa.Table

Redact the given columns (default: all string columns) of a table.

redact_batches ¶

redact_batches(batches: Any, columns: Sequence[str] | None = None, **options: Any) -> Iterator[pa.RecordBatch]

Redact a stream of record batches (for example a RecordBatchReader), yielding redacted batches one at a time.

redact_parquet ¶

redact_parquet(source: str | PathLike[str], destination: str | PathLike[str], columns: Sequence[str] | None = None, batch_size: int = 10000, **options: Any) -> int

Redact a Parquet file into a new one, batch_size rows at a time.

The source may be larger than memory. Columns not redacted are copied unchanged; the schema is kept. Returns the number of rows written.

Polars¶

Polars integration: pl.col("notes").credgate.redact().

Importing this module registers a credgate namespace on Polars expressions and Series (pip install 'credgate[polars]')::

import polars as pl
import credgate.polars  # noqa: F401

df.with_columns(pl.col("notes").credgate.redact(min_confidence=0.5))
df.filter(pl.col("notes").credgate.has_pii())

Numbered tokens (:func:credgate.tokenise) are not offered here: Polars may process batches in parallel and in any order, so token numbers would depend on scheduling. Tokenise a Series directly instead.

CredgateExpr ¶

CredgateExpr(expr: Expr)

redact ¶

redact(detectors: Sequence[str] | None = None, min_confidence: float = 0.0) -> pl.Expr

Replace identifiers with placeholders such as [PAN].

has_pii ¶

has_pii(detectors: Sequence[str] | None = None, min_confidence: float = 0.0) -> pl.Expr

Whether each value contains a detected identifier.

CredgateSeries ¶

CredgateSeries(s: Series)

redact ¶

redact(detectors: Sequence[str] | None = None, min_confidence: float = 0.0) -> pl.Series

Replace identifiers with placeholders such as [PAN].

has_pii ¶

has_pii(detectors: Sequence[str] | None = None, min_confidence: float = 0.0) -> pl.Series

Whether each value contains a detected identifier.

DuckDB¶

DuckDB integration: SELECT cg_redact(notes) FROM 'big.parquet'.

:func:register adds SQL functions to a DuckDB connection (pip install 'credgate[duckdb]')::

import duckdb
import credgate.duckdb

con = duckdb.connect()
credgate.duckdb.register(con, min_confidence=0.5)
con.sql("COPY (SELECT id, cg_redact(notes) AS notes FROM 'calls.parquet') "
        "TO 'calls_redacted.parquet'")

The functions run on Arrow batches, so DuckDB streams data through them and the input may be larger than memory. Numbered tokens are not offered: DuckDB processes batches in parallel, so token numbers would depend on scheduling.

register ¶

register(con: Any, detectors: Sequence[str] | None = None, min_confidence: float = 0.0, prefix: str = 'cg_') -> None

Add <prefix>redact(text) and <prefix>has_pii(text) to con.

detectors and min_confidence apply to every call of the functions.