Reports, per column, which PII types occur and in what share of values -
for example that cust_id looks like a PAN in 98% of rows. Useful on its
own, before any data is shared, to find which columns need protection.
Arguments
- data
A data frame, an Arrow
TableorDataset(for example fromarrow::open_dataset()), or a database table from dbplyr (dplyr::tbl(con, "calls"), for DuckDB, Postgres and other DBI databases). Arrow and database tables are read one column at a time, in chunks, so they may be larger than memory; the sample is the same as for the table in memory. For a database, rows must come back in the same order on each read (DuckDB and SQLite do for a plain table). A Spark table from sparklyr is sampled inside the cluster (up tosamplerows, randomly withseed) and only the sample is collected.- sample
Maximum number of values checked per column.
- min_rate
Report a type only if it occurs in at least this share of sampled values (0 reports every type found).
- detectors
Detector ids to run;
NULLruns all registered ones.- seed
Random seed for sampling Spark tables; ignored otherwise.
Value
A data frame of class cg_scan with one row per column and type:
column, type, class, n_sampled, n_matched (values with a
match), n_full (values that are entirely the identifier), rate
(n_matched / n_sampled), mean_confidence and a masked example.
Details
Each column is sampled (evenly spaced, so results are reproducible), and
every sampled value is checked with detect_pii() using the column name as
context: a column named voter_id supports Voter ID matches that would
otherwise need a keyword in the text. Whole numbers are checked too
(mobile and account numbers are often stored as numbers); other numbers,
logicals and dates are skipped. The rules are shared with the Python
package (see the spec's "Column scan" section).
Examples
customers <- data.frame(
cust_pan = c("ABCPE1234F", "AAACR5055K", "XYZPQ9876R"),
notes = c("Called, Aadhaar 542229371182", "no issues", "mobile 9876543210")
)
scan_pii(customers)
#> Scanned 2 columns; PII found in 2: cust_pan and notes
#> column type rate matched whole confidence example
#> cust_pan pan 100% 3 of 3 3 0.97 XXXXXX234F
#> notes aadhaar 33% 1 of 3 0 0.97 XXXXXXXX1182
#> notes mobile_in 33% 1 of 3 0 0.93 XXXXXX3210