Skip to content
CreatorPrivacyKit
Learn Blog Privacy @mollycoddle330

Research Protocol · Version 1.0

OCR Benchmark Methodology

Full specification of test fixture generation, normalization pipelines, metric formulas, and execution environments.

OCR Benchmark Methodology?

Full specification of test fixture generation, normalization pipelines, metric formulas, and execution environments. The benchmark uses 10 controlled document scenes located under tests/fixtures/ocr-multilingual/ . Each fixture is a photographic scene depicting a synthetic but realistic document in everyday conditions: Desk invoice (EN): Multi-line table, currency symbols, sales tax rate, address, phone, and contact email.

Last updated

1. Test Fixture Selection & Composition

The benchmark uses 10 controlled document scenes located under tests/fixtures/ocr-multilingual/. Each fixture is a photographic scene depicting a synthetic but realistic document in everyday conditions:

  • Desk invoice (EN): Multi-line table, currency symbols, sales tax rate, address, phone, and contact email.
  • Wall community notice (ZH-CN): Paper notice pinned to wood, containing office hours, room number, telephone, and URL.
  • Restaurant menu board (JA): Multi-column food items, prices in Japanese Yen, photo illustrations, and website.
  • Cafe poster on glass (KO): Poster taped behind reflective glass with coffee descriptions and opening hours.
  • Supermarket receipt (ES): Thermal paper receipt with physical folds, line-item Euro prices, and tax breakdown.
  • Event flyer (FR): Cultural program schedule pinned to a cork board with accented French prose and venue details.
  • Technical product sheet (DE): High-contrast spec sheet on an aluminum clipboard with German umlauts, mass units, and serial codes.
  • Shipping label (PT-BR): Cardboard package label with Brazilian postal code (CEP), tracking number, and barcode area.
  • Civic notice (HI): Devanagari civic service announcement with mixed Hindi text and Latin telephone digits.
  • Clinic form & card (AR): Right-to-left Arabic clinic sheet accompanied by an appointment card with mixed LTR email and URL.

Source Type: synthetic_controlled_fixture. Ground truth represents the visible text on the rendered pixels, not the generation prompt. Decorative incidental background items (such as coffee mugs, wall trim, or distant posters) are excluded from evaluation.

2. Normalization Rules

To maintain strict empirical validity, our scoring pipeline uses generic text normalization only:

  • Unicode NFC Normalization: All inputs and ground truth strings are normalized to Unicode Canonical Composition (NFC).
  • Line Break Harmonization: Windows CRLF ( ) is collapsed to standard Unix LF ( ).
  • Whitespace Folding: Consecutive horizontal spaces and tabs are collapsed to a single space. Leading and trailing whitespace is trimmed.
  • Zero Post-OCR Character Repair: The test harness deliberately avoids regex heuristics, word dictionaries, or character substitution rules (such as repairing letter 'O' to number '0' or lowercase 'l' to digit '1'). A miss remains a miss.

3. Scoring Metrics & Formulas

We report multiple complementary metrics because a single average error rate can conceal catastrophic single-digit financial errors:

Character Error Rate (CER)

Computed as the Levenshtein distance (substitutions (S), deletions (D), and insertions (I)) divided by the total reference character count (N):

CER = (Substitutions + Deletions + Insertions) / Total Reference Characters

CER is evaluated over the optimal matching window of the OCR dump against the concatenated expected ground truth. Lower values indicate fewer edits.

Word Error Rate (WER)

Evaluated on whitespace-delimited word tokens for languages that use word spacing (English, Spanish, French, German, Portuguese, Korean). Non-spaced scripts (Chinese, Japanese, Hindi, Arabic) report CER and Key Field Accuracy only, as arbitrary space tokenization produces misleading WER figures.

Key Field Accuracy (Primary Verdict Metric)

Each fixture defines structured ground truth fields (e.g., invoice IDs, dates, addresses, totals, email, telephone). A field is counted as a hit only if its exact normalized string appears verbatim as a substring in the OCR output. Meaning-similar or partial matches count as a miss.

PASS ≥ 95% Key Fields
WARN 85% – 94.9% Key Fields
FAIL < 85% Key Fields

Strict Dimension Sub-checks

Numeric values, telephone numbers, emails, URLs, and document identifiers are tracked separately. If a single digit in a financial price is corrupted (e.g., $145.06 vs $145.08), the numeric metric records a failure even if overall CER is under 2%.

4. Engine Run vs. Product E2E Run

The test suite executes two separate benchmark passes to distinguish core OCR engine accuracy from browser user-interface integration:

1. Engine Run

Runs Tesseract.js 5.1.1 directly in Node.js against the exact same vendored 4.0.0_best_int tessdata packs shipped to production. Measures raw LSTM character recognition speed and accuracy without DOM manipulation overhead.

2. Product E2E Run

Automates Google Chrome via Puppeteer against the live production route (/tools/ocr/). Simulates real user interaction: uploading the file, selecting language, clicking Run OCR, waiting for completion, asserting editor text, testing clipboard copy, and verifying downloaded TXT files.

5. Execution Environment & Hardware

Repository Commit 5fd71dc / ce817fa
Test Execution Date September 20, 2026
Operating System Linux 7.0.0-31-generic (x86_64)
Browser Runtime Google Chrome 153.0.0.0 (Headless) / Node.js v22
OCR Library Tesseract.js 5.1.1 (tesseract.js-core 5.1.1, WebAssembly SIMD enabled)
Model Traineddata tessdata_best (version 4.0.0_best_int)

Reproduce this benchmark locally

Every number published on this page can be re-executed directly from the source repository. No private keys or cloud accounts are required.

  1. Clone the repository: git clone https://github.com/ggggghhhhyyyy/creator-privacy-kit
  2. Install dependencies: npm install
  3. Run the official engine benchmark: 
    npm run benchmark:ocr -- --engine
  4. Run the browser E2E verification test: 
    npm run benchmark:ocr -- --e2e

Runtime environment: Node.js v22+ on Linux / macOS / Windows with Chrome installed for optional E2E. Ground truth strings and fixture hashes are defined in tests/fixtures/ocr-multilingual/manifest.json.

Cite this benchmark

Empirical research published by CreatorPrivacyKit Labs is freely citeable with attribution under CC BY 4.0.

APA / Text Format 
CreatorPrivacyKit Labs. (2026). OCR Benchmark Methodology (Version 1.0) [Data set]. CreatorPrivacyKit. https://creatorprivacykit.com/labs/ocr-benchmark/methodology/
View BibTeX entry 
@misc{creatorprivacykit_2026_ocr_benchmark,
  title = {OCR Benchmark Methodology},
  author = {{CreatorPrivacyKit Labs}},
  year = {2026},
  version = {1.0},
  howpublished = {\url{https://creatorprivacykit.com/labs/ocr-benchmark/methodology/}},
  note = {Tested 2026-09-20. In-browser Tesseract.js 5.1.1 multilingual evaluation}
}
Data sources: Dataset landing & download · results.json · results.csv