Record where PII is, for an inventory

Scans a table with the same rules as scan_pii(), but records every column - including those where nothing was found - with the source and table names, the sample size and method, and when and by what it was scanned. No values are recorded, not even masked ones. Combine the inventories of several tables with rbind() and save them with write_inventory(); the file format is shared with the Python package (see the spec's "Inventory" section).

Usage

pii_inventory(
  data,
  source,
  table,
  sample = 10000L,
  min_rate = 0,
  detectors = NULL,
  seed = 42L
)

Arguments

data

A data frame, an Arrow Table or Dataset (for example from arrow::open_dataset()), or a database table from dbplyr (dplyr::tbl(con, "calls"), for DuckDB, Postgres and other DBI databases). Arrow and database tables are read one column at a time, in chunks, so they may be larger than memory; the sample is the same as for the table in memory. For a database, rows must come back in the same order on each read (DuckDB and SQLite do for a plain table). A Spark table from sparklyr is sampled inside the cluster (up to sample rows, randomly with seed) and only the sample is collected.

source

Name of the data source, chosen by you (e.g. "bigquery:prod", "hive").

table

Name of the table within the source (e.g. "crm.customers", or a file path).

sample

Maximum number of values checked per column.

min_rate

Report a type only if it occurs in at least this share of sampled values (0 reports every type found).

detectors

Detector ids to run; NULL runs all registered ones.

seed

Random seed for sampling Spark tables; ignored otherwise.

Value

A data frame of class cg_inventory with one row per column and PII type found, and one row with type = NA for each column where none was found: v, source, table, column, type, class, n_rows (when known without reading the table), n_sampled, n_matched, n_full, rate, mean_confidence, sample_method ("all" or "even", or "spark-sample" for Spark tables), scanned_at (UTC) and credgate (the version that scanned).

Examples

customers <- data.frame(
  cust_pan = c("ABCPE1234F", "AAACR5055K", "XYZPQ9876R"),
  notes = c("Called, Aadhaar 542229371182", "no issues", "mobile 9876543210"),
  balance = c(1520.5, 99.75, 0.25)
)
inv <- pii_inventory(customers, source = "crm", table = "customers")
inv
#> Inventory of 1 table:
#>   crm: customers - 3 column(s) scanned, PII in 2 (pan, aadhaar, mobile_in)
path <- tempfile(fileext = ".csv")
write_inventory(inv, path)
read_inventory(path)
#> Inventory of 1 table:
#>   crm: customers - 3 column(s) scanned, PII in 2 (pan, aadhaar, mobile_in)