Redact a Parquet file, batch by batch

Writes a copy of a Parquet file with identifiers in the chosen text columns replaced by placeholders such as [PAN]. The file is read in record batches, so it may be larger than memory; other columns and the schema are kept.

Usage

redact_parquet(source, destination, columns = NULL, batch_size = 10000L, ...)

Arguments

source, destination

Parquet files to read and write; they must differ.

columns

Columns to redact; NULL redacts every string column.

batch_size

Rows per batch.

...

Passed to redact(): detectors, min_confidence.

Value

The number of rows written, invisibly.

See also

redact_query() for database tables, including DuckDB.

Examples

src <- tempfile(fileext = ".parquet")
dst <- tempfile(fileext = ".parquet")
arrow::write_parquet(data.frame(id = 1:2, notes = c("PAN ABCPE1234F", "no PII")), src)
redact_parquet(src, dst)
arrow::read_parquet(dst)
#> # A tibble: 2 × 2
#>      id notes    
#>   <int> <chr>    
#> 1     1 PAN [PAN]
#> 2     2 no PII