Scan and inventory BigQuery tables — bq_scan

BigQuery samples the table and only the sample is downloaded; only the columns credgate can scan are read, and a dry run checks the cost of every query first. credgate makes no network calls of its own: it uses bigrquery, with your credentials (bigrquery::bq_auth()). The rules are shared with the Python package (credgate.bigquery; see the spec's "BigQuery tables").

Usage

bq_scan_pii(
  table,
  billing = NULL,
  sample = 10000L,
  min_rate = 0,
  detectors = NULL,
  sampling = "tablesample",
  max_bytes = 1e+09
)

bq_pii_inventory(
  table,
  source = NULL,
  billing = NULL,
  sample = 10000L,
  min_rate = 0,
  detectors = NULL,
  sampling = "tablesample",
  max_bytes = 1e+09
)

bq_inventory_dataset(
  dataset,
  source = NULL,
  billing = NULL,
  sample = 10000L,
  min_rate = 0,
  detectors = NULL,
  sampling = "tablesample",
  max_bytes = 1e+09
)

Arguments

table: A table: "project.dataset.table" or a bigrquery::bq_table().
billing: Project billed for the queries; defaults to the table's project.
sample: Maximum number of rows sampled.
min_rate, detectors: As for scan_pii().
sampling: "tablesample" (BigQuery's block sampling; only the sampled blocks are billed) or "rand" (reads and bills whole columns). Views and external tables always use "rand".
max_bytes: Refuse any query estimated to process more bytes than this (default 1 GB). Also sent to BigQuery as the billing limit, except with TABLESAMPLE, where BigQuery would check it against the whole table.
source: Source name for the inventory; defaults to "bigquery:<project>". The table is recorded as "dataset.table".
dataset: A dataset: "project.dataset" or a bigrquery::bq_dataset(). Every table and view in it is scanned.

Value

bq_scan_pii() a scan report as from scan_pii(); bq_pii_inventory() and bq_inventory_dataset() an inventory as from pii_inventory(), with the table's row count and sample_method "all", "bigquery-tablesample" or "bigquery-rand".

Examples

if (FALSE) { # \dontrun{
bigrquery::bq_auth()
bq_scan_pii("my-project.crm.customers")
inv <- bq_inventory_dataset("my-project.crm")
write_inventory(inv, "inventory.csv")
} # }