Benchmark Methodology
A truthful, reproducible description of how Veridexa evaluates its Fraud Detection pipeline. No performance claims are published until the legitimate labelled dataset requirement is met.
Purpose
This benchmark exists so external readers can understand exactly how Veridexa evaluates itself internally and where the boundary between private evaluation data and public artifacts sits. It is not a certification and not a substitute for independent third-party auditing.
Evaluation scope
The runner executes each labelled document through the production Fraud Detection pipeline and records the final decision only. Intermediate evidence, provider outputs, and Core reasoning are never included in public artifacts.
Supported document categories
- Passport
- Government-issued ID card
- Academic certificate
- Academic transcript
- Bank statement
- Invoice
- Employment document
- Financial document
Ground-truth requirements
Every sample is labelled AUTHENTIC or FRAUDULENT. Labels must be independently verified. Unverified, ambiguous, or partially-manipulated documents are not admitted.
Dataset inclusion and exclusion criteria
- Real documents with verified provenance and redistribution rights.
- Excluded: demo assets, synthetic copies, duplicates, relabelled copies, and any document whose ground truth cannot be independently verified.
- Duplicate prevention: validation computes the SHA-256 of every source file. Any two entries with identical content are rejected before the benchmark runs.
Metrics
- Total documents, completed documents, error count.
- True/false positives and negatives.
- Accuracy, precision, recall, F1 — omitted when denominators are zero.
- Average processing time.
Headline metrics are withheld unless the number of resolved binary decisions is ≥ 30. Below that threshold the report status is insufficient_data and no ratio is claimed.
Treatment of MANUAL_REVIEW
The pipeline may return MANUAL_REVIEW. Those entries are counted as unresolvedCount and excluded from accuracy, precision, recall, and F1 — never scored as correct or incorrect.
Sanitization and public export
The public export pipeline emits only: documentType, groundTruthLabel, predictedDecision, processingTimeMs, a coarse errorCategory, and a deterministic pub_<sha256[0..16]> identifier. Source files, filenames, storage paths, hashes, OCR text, QR contents, PII, and provider evidence are never published. Any forbidden field causes the export to fail closed.
Limitations
- Small benchmark datasets are indicative of pipeline behavior on that set only.
- Labels are binary; partial manipulation and ambiguous provenance are not represented.
- Provider outputs can drift; a metric produced on one day may not reproduce on another.
- No adversarial robustness guarantee is claimed.
Minimum readiness sample requirement
The OpenAI readiness gate requires at least 30 legitimate labelled samples. Public results are not published until this requirement is met and the sanitized export completes successfully.
Current status
Current public results are not yet published because the legitimate dataset requirement has not been met. See /benchmark/results for live status.