Evaluate detection against labelled text — cg

Measures how well detect_pii() finds the PII in documents where every PII span has been labelled: true positives, false positives, false negatives, precision, recall and F1, per type and overall. Use it to check credgate on your own (labelled) data. The rules are shared with the Python package (see the spec's "Evaluation" section).

Usage

cg_evaluate(
  documents,
  min_confidence = 0,
  match = c("exact", "overlap"),
  detectors = NULL
)

Arguments

documents: Labelled documents: a list of lists with text and spans (each span a list with start, end - 0-based, end-exclusive character offsets - and type), or the path of a JSON Lines file in that format.
min_confidence: Ignore predictions scoring below this value.
match: "exact" (same type and offsets) or "overlap" (same type, overlapping spans).
detectors: Detector ids to run; NULL runs all registered ones.

Value

A data frame of class cg_evaluation with columns type, tp, fp, fn, precision, recall and f1: one row per type, then overall (micro-averaged). Rates are NA when undefined.

Examples

docs <- list(
  list(text = "PAN: ABCPE1234F", spans = list(list(start = 5, end = 15, type = "pan"))),
  list(text = "order no 878245671885", spans = list())
)
cg_evaluate(docs)
#> Evaluated 2 documents, exact matching:
#>     type tp fp fn precision recall     f1
#>  aadhaar  0  1  0       0.0     NA     NA
#>      pan  1  0  0       1.0      1 1.0000
#>  overall  1  1  0       0.5      1 0.6667
cg_evaluate(docs, min_confidence = 0.5)
#> Evaluated 2 documents, exact matching:
#>     type tp fp fn precision recall f1
#>      pan  1  0  0         1      1  1
#>  overall  1  0  0         1      1  1