Inventory and discovery

Where PII is, across tables: in-memory data frames, Arrow, DuckDB, Spark and BigQuery all produce the same inventory.

pii_inventory

pii_inventory(data: Any, source: str, table: str, sample: int = 10000, min_rate: float = 0.0, detectors: Sequence[str] | None = None, seed: int = 42) -> Inventory

Scan a table and record where PII is, for an inventory.

Uses the same rules as :func:scan_pii, but records every column - including those where nothing was found - with the source and table names, the sample size and method, and when and by what it was scanned. No values are recorded, not even masked ones. Combine the inventories of several tables with + and save them with :meth:Inventory.write.

Parameters:

Name	Type	Description	Default
data	Any	Any table :func:scan_pii accepts.	required
source	str	Name of the data source, chosen by you (e.g. "bigquery:prod").	required
table	str	Name of the table within the source (e.g. "crm.customers").	required
sample	int	As for :func:scan_pii.	10000
min_rate	float	As for :func:scan_pii.	0.0
detectors	Sequence[str] | None	As for :func:scan_pii.	None
seed	int	As for :func:scan_pii. A PySpark DataFrame is sampled inside the cluster; its row count is recorded.	42

Inventory

Inventory(rows: Iterable[InventoryRow] = ())

Bases: list[InventoryRow]

Rows of an inventory; combine inventories with + and save them with :meth:write. Prints as a summary per table.

to_records

to_records() -> list[dict[str, Any]]

One dict per row, e.g. for pandas.DataFrame(inventory.to_records()).

write

write(path: str | PathLike[str]) -> Path

Write the inventory as a CSV file, exactly as the spec describes.

InventoryRow dataclass

InventoryRow(source: str, table: str, column: str, type: str | None, data_class: str | None, n_rows: int | None, n_sampled: int, n_matched: int, n_full: int, rate: float, mean_confidence: float | None, sample_method: str, scanned_at: str, credgate: str, v: int = FORMAT_VERSION)

One column of one table, and one PII type found in it (type is None for a column scanned without finding any).

read_inventory

read_inventory(path: str | PathLike[str]) -> Inventory

Read an inventory CSV file written by :meth:Inventory.write (in R or Python).

Databases

PII inventories of whole databases (the spec's "Databases" rules).

Every table in the chosen schemas is sampled by the database where it can (Postgres and Db2 TABLESAMPLE SYSTEM, DuckDB a reservoir sample), and only the sample is read. With an inventory file, the scan can resume after an interruption, and later runs rescan only the tables that changed.

import sqlalchemy
from credgate.database import inventory_database

engine = sqlalchemy.create_engine("postgresql+psycopg://reader@db/crm")
inv = inventory_database(engine, source="postgres:crm", path="inventory.csv")

credgate makes no network calls of its own: it uses the engine you pass.

DatabaseInventory

DatabaseInventory(rows: Any = (), failed: dict[str, str] | None = None, skipped: Sequence[str] = ())

Bases: Inventory

An :class:~credgate.Inventory of a database, with the tables that could not be read (failed: schema.table -> error message) and those kept from an earlier run (skipped).

inventory_database

inventory_database(connection: Any, source: str, path: str | PathLike[str] | None = None, schemas: Sequence[str] | None = None, include: Sequence[str] | None = None, exclude: Sequence[str] | None = None, views: bool = False, sample: int = 10000, min_rate: float = 0.0, detectors: Sequence[str] | None = None, max_age: float = 30, on_error: str = 'warn') -> DatabaseInventory

Record where PII is in every table of a database.

Parameters:

Name	Type	Description	Default
connection	Any	A SQLAlchemy Engine or Connection.	required
source	str	Name of the data source for the inventory, e.g. "postgres:crm".	required
path	str | PathLike[str] | None	Optional inventory file (CSV). Existing rows are read first; tables scanned within max_age days whose row count has not changed are kept rather than scanned again; the file is rewritten after each table, so an interrupted run can continue.	None
schemas	Sequence[str] | None	Schemas to scan; default all except the database's own.	None
include	Sequence[str] | None	Shell-style patterns for schema.table to scan (e.g. ["crm.*"]); default all.	None
exclude	Sequence[str] | None	Patterns for schema.table to leave out.	None
views	bool	Also scan views (reading a view runs its query).	False
sample	int	Maximum number of rows read per table.	10000
min_rate	float	As for :func:credgate.scan_pii.	0.0
detectors	Sequence[str] | None	As for :func:credgate.scan_pii.	None
max_age	float	Days after which a table is scanned again even if unchanged.	30
on_error	str	"warn" (skip a table that cannot be read, and list it in failed) or "stop" (raise).	'warn'

sample_query

sample_query(family: str, schema: str, table: str, n: int, columns: Sequence[tuple[str, str]], sample: int = 10000) -> tuple[str | None, list[str], str, int | None]

The query that samples a table, the columns it reads, the sample method, and how many rows to read (None: all the query returns), as the spec's "Databases" rules say. columns are (name, type).

BigQuery

PII scans and inventories of BigQuery tables (the spec's "BigQuery tables" rules).

BigQuery samples the table and only the sample is downloaded; a dry run checks the cost of every query first. credgate makes no network calls of its own: it uses the google.cloud.bigquery.Client you pass, with its credentials and project.

from google.cloud import bigquery
from credgate import bigquery as cgbq

client = bigquery.Client(project="my-project")
cgbq.scan_pii(client, "my-project.crm.customers")
inv = cgbq.inventory_dataset(client, "my-project.crm")

CostLimitError

Bases: RuntimeError

A query would process more bytes than max_bytes allows; nothing ran.

scan_pii

scan_pii(client: Any, table: Any, sample: int = 10000, min_rate: float = 0.0, detectors: Sequence[str] | None = None, sampling: str = 'tablesample', max_bytes: int = MAX_BYTES) -> ScanReport

Scan the columns of a BigQuery table for PII (see :func:credgate.scan_pii).

Parameters:

Name	Type	Description	Default
client	Any	A google.cloud.bigquery.Client.	required
table	Any	"project.dataset.table", or a table reference.	required
sample	int	Maximum number of rows sampled.	10000
min_rate	float	As for :func:credgate.scan_pii.	0.0
detectors	Sequence[str] | None	As for :func:credgate.scan_pii.	None
sampling	str	"tablesample" (block sampling; only sampled blocks are billed) or "rand" (reads and bills whole columns). Views and external tables always use "rand".	'tablesample'
max_bytes	int	Refuse any query estimated to process more bytes than this (default 1 GB). Also sent to BigQuery as the billing limit, except with TABLESAMPLE, where BigQuery would check it against the whole table.	MAX_BYTES

pii_inventory

pii_inventory(client: Any, table: Any, source: str | None = None, sample: int = 10000, min_rate: float = 0.0, detectors: Sequence[str] | None = None, sampling: str = 'tablesample', max_bytes: int = MAX_BYTES) -> Inventory

Record where PII is in a BigQuery table (see :func:credgate.pii_inventory).

The source defaults to "bigquery:<project>" and the table name to "dataset.table". Other arguments as for :func:scan_pii.

inventory_dataset

inventory_dataset(client: Any, dataset: Any, source: str | None = None, sample: int = 10000, min_rate: float = 0.0, detectors: Sequence[str] | None = None, sampling: str = 'tablesample', max_bytes: int = MAX_BYTES) -> Inventory

The inventory of every table and view in a dataset ("project.dataset").

Stops at the first table whose query would exceed max_bytes. Other arguments as for :func:pii_inventory.

sample_query

sample_query(table: str, kind: str, n: int, fields: Sequence[tuple[str, str, str]], sample: int = 10000, sampling: str = 'tablesample') -> tuple[str | None, list[str], str]

The query that samples a table, the columns it reads and the sample method (spec: "BigQuery tables"). fields are (name, type, mode).