# Source reliability and the cost of a citation workflow

Fourteen controlled source cases and seven ways to check three quotations. Inspect the evidence, request counts, body sizes, and explicit limits.

## What was checked

The corpus adds two-column PDF text, separately positioned PDF table cells, an HTML table, curly quotation marks, accented text, emoji offsets, whitespace, a redirected article, a robots denial, an HTTP 404, and a quotation beyond the reader’s 100,000-character output limit. The checker must preserve a distinction between absence, incomplete text, and unreadable sources.

These are selected BOZOS-owned fixtures containing fictional data. The capture runs the real parsers and matching functions with a controlled source transport and DNS resolver. It is an offline regression evaluation, not an independent publisher benchmark or a production availability test. Matching remains literal apart from whitespace normalization.

## Requests and bytes

Every measured workflow checks the same three quotations: two present, one with a changed number. The separate URL checks retrieve the same article three times. The citation batch groups the URL into one retrieval. Source-request totals include robots.txt checks.

Read-once/reuse includes the initial reader response and the supplied-text check. Already-held text measures only checking text the caller has; its prior retrieval is outside the measurement, and it does not verify publisher provenance. Modern MCP can call the batch directly. The cold legacy workflow includes initialization and its notification.

Each workflow has one excluded warm-up and five measured samples. Times are local handler elapsed time; HTTP envelopes are simulated and MCP runs the real handler. They exclude network, TLS and production routing, and must not be read as a production speed ranking or SLA. Bytes are UTF-8 request and decoded response bodies, excluding headers and compression. MCP response totals include both JSON text and structuredContent; bytes are not model tokens.

## Reproduce and inspect

The complete capture includes per-case evidence, per-call measurements, source-file and fixture hashes, Node version and timestamps. The downloadable corpus generator builds the exact fixture bytes without dependencies. The PDF was rendered and visually inspected.

The Python verifier runs all fourteen cases through the public HTTP API or modern MCP. It checks outcomes, fixture hashes, redirect provenance, passages, span lengths and PDF pages, saves the raw replies, and stops on any mismatch. It uses Python’s standard library, sends no feedback, and does not retry. These reruns are internal test activity, not adoption evidence.

```sh
python3 verify-live.py --transport http --output http-results.json
python3 verify-live.py --transport mcp --output mcp-results.json
```

- [Live verifier (Python)](https://bozos.si/examples/source-reliability/verify-live.py)
- [Corpus manifest and expected outcomes](https://bozos.si/examples/source-reliability/corpus.json)
- [Dependency-free fixture generator](https://bozos.si/examples/source-reliability/source-corpus.mjs)
- [Source evidence](https://bozos.si/examples/source-reliability/results/reliability.json)
- [Workflow samples](https://bozos.si/examples/source-reliability/results/workflows.json)
- [Capture metadata and hashes](https://bozos.si/examples/source-reliability/results/capture.json)
- [Two-column and table PDF](https://bozos.si/examples/source-reliability/fixtures/layout-v1.pdf)

## What these cases do not establish

The PDF uses a column-major content stream. Other PDF reading orders, scans, ligatures, and hyphenated line breaks can behave differently. A match inside flattened table text does not verify the table’s semantic relationships. HTML extraction can omit context even when the returned representation is not truncated.

The long-source case exercises output truncation after a successful fetch; it does not exercise the download byte limit. The tests do not establish truth, claim support, source credibility, publisher coverage, independent demand, or production latency. All corpus material and captured results are CC0.


## Source cases

| Case | Observed outcome | Passed |
| --- | --- | --- |
| Unicode punctuation and emoji offsets | exact_match | yes |
| Straight quotes differ from curly quotes | not_found | yes |
| Changed count in article text | not_found | yes |
| Navigation is not article evidence | not_found | yes |
| A row extracted from an HTML table | normalized_match | yes |
| Line breaks normalize but the passage is preserved | normalized_match | yes |
| Left PDF column, page 1 | exact_match | yes |
| Right PDF column, page 1 | exact_match | yes |
| PDF table cells, page 2 | exact_match | yes |
| PDF sentence, page 2 | exact_match | yes |
| Redirected article keeps final URL provenance | exact_match | yes |
| Missing quotation beyond the fetched output limit | source_truncated | yes |
| HTTP 404 keeps quotation unchecked | upstream_error | yes |
| A robots denial keeps quotation unchecked | robots_disallowed | yes |

## Workflow measurements

| Workflow | API requests | Source requests | Request body bytes | Response body bytes | Local p50 ms | Local p95 ms |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| HTTP: three separate URL checks | 3 | 6 | 304 | 7455 | 7.11 | 9.92 |
| HTTP: one URL citation batch | 1 | 2 | 367 | 5737 | 4.27 | 6.52 |
| HTTP: read once, then check supplied text | 2 | 2 | 1063 | 8621 | 4.94 | 7.29 |
| HTTP: source text already held | 1 | 0 | 1435 | 5361 | 1.76 | 2.42 |
| MCP 2026: direct URL batch | 1 | 2 | 574 | 12236 | 4.20 | 6.48 |
| MCP 2026: discover, then URL batch | 2 | 2 | 749 | 14393 | 4.31 | 5.27 |
| MCP 2025: initialize, notify, URL batch | 3 | 2 | 686 | 14153 | 4.02 | 5.14 |