Table of Contents

How detection is tested

PDiffy's job is to fail your test when a change to your code changes a document. No tool can promise to catch every possible change, and PDiffy doesn't. This page describes how its own tests check that it catches the changes it should, and lists the changes it's known to miss.

What fails a test

A snapshot test fails whenever the new snapshot differs from the approved one. Whether the change matters for accessibility decides how the report flags it, not whether the test fails. So the question is always the same: does the change alter the snapshot text? Each kind of test below answers part of that question.

Mutation tests

The PDF mutation tests build one tagged PDF in code, with headings, paragraphs, a table, a link, an image, a form field, bookmarks, attachments, metadata and document settings. Each of more than 100 cases changes one thing about it. A few of them:

  • Structure: a heading level changed, alt text removed or emptied, reading order swapped, a table header's scope removed, tagged text made untagged or an artifact, text moved to another structure element.
  • Text and fonts: text moved by 1 pt, the font size changed by 0.1 pt, a font replaced by one with a different name, a font's ToUnicode map removed, text made invisible.
  • Visible content: an image's pixels changed, a line's dash pattern changed or its ends rounded or corners beveled, a shape's outline changed inside the same box, text hidden by a clipping path or by a box drawn over it.
  • Links, bookmarks and forms: a link's target or destination page changed, a bookmark's view changed, a form field's accessible name removed, a field's script edited.
  • Document: the language changed, the title no longer shown, the document no longer marked as tagged, the PDF/UA identification changed, the document encrypted, the PDF version raised past 1.7.

Each case must change the snapshot, show up in the expected section as the expected kind of change, and be flagged as affecting accessibility exactly when it should be. Most cases change the PDF itself rather than snapshot text, so they test the reading as well as the comparison. The Word mutation tests do the same for a DOCX, with more than 75 cases covering paragraph and character formatting, page and section breaks, section layout, style and list definitions, the document defaults and settings, tables, images, fields, headers, footnotes, tracked changes and comments.

A weekly job makes similar changes to real PDFs, one at a time. For each PDF/UA file in the veraPDF corpus, it retags a paragraph as a span, edits alt text, changes a language, removes a font's ToUnicode map, renames a font and removes a page's parent tree key, wherever the file has one to change. It fails when a change leaves the snapshot the same, or when a change applies to no file at all. The job keeps a list of changes known to leave a file's snapshot the same, such as renaming a form font no field uses, with the reason for each. The check also fails when a listed change starts changing the snapshot or isn't compared at all, so a fixed gap or a file the corpus moved comes off the list.

Printed keys are covered by mutations

A coverage test lists every key the sample PDF's snapshot prints (alt=, lang=, color= and so on) and fails if no mutation changes it. Another does the same for the sample Word document. A new key can't be added to either format without a mutation proving that a change to it is caught. The tests exempt the few keys still waiting for a mutation.

Every line is edited

The snapshot edit tests take real snapshots and edit each line in turn, then check that the diff reports that edit and nothing else, in both directions. A comparison rule that misreads some kind of line fails here, even when no test was written for that line.

Format changes

When a PDiffy release changes what a snapshot prints, a test compares a snapshot in the old format with the new one and expects only the documented additions. Upgrading shows you what's new, not a wall of moved or rewritten lines. Each change is listed in Format changes.

Changes that must not fail a test

Catching changes is only useful if unchanged documents pass. The equivalence tests write each document two ways that differ only in encoding (object numbers, compression, object streams, incremental updates, zip order) and expect the same snapshot. A weekly job does the same for the real PDF/UA files in the veraPDF corpus, rewriting each one with qpdf.

Damaged and hostile files

A change that breaks a document must not make content quietly disappear from its snapshot. Tests load malformed files (references that loop, missing objects, objects whose place in the file is damaged), randomly damaged files, and files built to exhaust the stack or memory. Each must load with the damaged part marked unreadable, or fail with a clear error.

Other writers

Most tests build their PDFs and Word files in code, so every input is small and its purpose is clear. Further tests read PDFs written by another PDF library (PDFsharp) and check what shouldn't depend on the writer, such as each element's text, page, alt text and language, rather than comparing exact snapshots.

On every change

The test suite, more than 2,000 tests, runs on Windows and Linux for every push and pull request. The veraPDF corpus checks run weekly.

Known gaps

These changes can alter how a document looks without changing its snapshot.

  • Curved clipping paths are traced in steps. A clipping path, or text drawn in a clipping mode (4 to 7), counts in its real shape with curves traced in small straight steps, so content within a hair of a curve can be judged either way.
  • Some clipping text counts as its letters' boxes. Text drawn in a clipping mode in a font whose outlines PDiffy can't read, such as a standard font that isn't embedded, clips to its letters' boxes, so content inside a letter's box but outside its shape isn't marked clipped=yes.
  • Covering is judged one shape or image at a time. Content is marked covered=yes only when one thing drawn after it hides all of it: an opaque shape filled with a plain color, or an image without a mask, outside any layer. A pattern or a thick outline drawn over text, two shapes that hide it only together, or a shape that hides only part of it isn't recorded, and content within a hair of a curved edge can be judged either way.
  • An appearance-only font's outlines aren't recorded. A font that only a text, combo box, list box or signature field's appearance, or a FreeText annotation's, draws with is listed by name, embedding and Unicode map, but its glyph outlines aren't hashed, since that would bring back the text the appearance draws. Swapping its embedded program for one with the same name doesn't show.
  • Small path changes can round away. A path's outline is hashed with its points rounded to a tenth of a point, so a point moved by less than that may not change the snapshot. A dash offset changed by less than a whole unit doesn't show either, since PdfPig reads it as a whole number.

Limitations and guarantees covers what PDiffy promises and what it doesn't do.