Skip to content

Add Hypothesis fuzzing + property tests for geotiff #1661

@brendancol

Description

@brendancol

Reason or Problem

The geotiff module has ~25k LOC of tests against ~17k LOC of parser/writer code, but every test in xrspatial/geotiff/tests/ is example-based: handwritten inputs built with make_minimal_tiff from conftest.py. Zero property-based or fuzz tests.

Recent commits show a pattern of bugs that property-based testing catches cheaply:

  • babb72e _coords_to_transform broke for 3D (y, x, band) DataArrays
  • 03d7380 georef not inherited when reading overview IFDs
  • c294578 / dd907b8 nodata sentinel poisoned cubic / COG overviews
  • 9a5f55e read_vrt leaked integer source nodata into a float-dtype VRT
  • f0a09c0 write_geotiff_gpu(file_like, cog=True) silently accepted an unsupported combination
  • 595ece8 signature/docstring drift between to_geotiff, write_vrt, write_geotiff_gpu
  • dbb5ceb user-defined CRS WKT not promoted to attrs on read
  • cdf7d43 LERC / JPEG 2000 decompression output not capped (block bomb)

These cluster into a few classes:

  1. Kwarg silently dropped or routed to the wrong code path
  2. Per-band / per-chunk metadata leaking (nodata, dtype)
  3. Georef / attrs not inherited when traversing IFD chains
  4. Round-trip equality broken for an unusual but valid combination of (dtype, compression, tiled, predictor, nodata)

Hand-written tests catch one variant at a time. Property tests cover the cross-product cheaply.

Proposal

Add Hypothesis-based property tests for the geotiff module.

Design:

A new test file xrspatial/geotiff/tests/test_fuzz_hypothesis_<issue>.py with three groups:

  1. Round-trip property. Strategies generate (width, height, dtype, compression, tiled, predictor, nodata), build a DataArray, write with to_geotiff, read with open_geotiff, assert array equality and attrs preservation. Bounded ranges (dims 1 to 32, a short dtype list, common codecs) so CI stays fast.

  2. IFD layout permutations. Use make_minimal_tiff with strategies over tag order, tile vs strip, big/little endian, with/without geo tags. Assert open_geotiff either returns a valid array or raises a typed exception from the geotiff module. Never a bare IndexError / struct.error / UnicodeDecodeError.

  3. Byte-level mutation fuzz. Take a valid TIFF from make_minimal_tiff, flip one byte at a Hypothesis-chosen offset, assert the reader either parses consistently or raises a typed exception. Never silently returns wrong data, never segfaults.

Each test uses @settings(max_examples=50, deadline=None) to bound CI time.

Hypothesis is not currently a test dependency. The implementation will guard the import with pytest.importorskip('hypothesis') so the suite still runs when hypothesis is missing. A follow-up PR can add it to setup.cfg [tests] once the harness has earned its keep.

Usage:

pip install hypothesis
pytest xrspatial/geotiff/tests/test_fuzz_hypothesis_*.py -v

Locally, --hypothesis-show-statistics shows shrinking behaviour on failure.

Value:

  • Catches the next instance of every bug class above before a user files it
  • Documents the parser's exception contract (what is "invalid input" vs "our bug")
  • Cheap maintenance: strategies are short, seeds make failures reproducible

Stakeholders and Impacts

Geotiff module maintainers. No runtime behaviour changes, no new public API. Impact limited to the test suite.

Drawbacks

  • Hypothesis adds ~1s to test collection time when installed
  • Bugs found by the fuzzer need triage (real bug vs strategy too broad)
  • Property failures are harder to read than example failures until the counterexample shrinks

Alternatives

  • Keep adding example-based tests one bug at a time (status quo)
  • AFL or afl-tiff: heavier infrastructure, harder to run in CI
  • Run hypothesis in a nightly CI job only: a possible follow-up if collection time hurts

Unresolved Questions

  • Add hypothesis to setup.cfg [tests] now or keep it opt-in? Defaulting to opt-in for this PR.
  • Seed a real-file corpus (GDAL COG, Sentinel-2-style)? Out of scope here; open a follow-up if synthetic strategies leave gaps.

Additional Notes or Context

Audit numbers:

  • Parser/writer LOC: 17,314 (xrspatial/geotiff/_*.py + __init__.py)
  • Test LOC: 24,904
  • Hypothesis tests: 0

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions