Skip to main content
FORGEJO ERA / ISSUE #616

Preference catalog tests do not detect many validation and presentation regressions

closedOpened by MelanX on

Original description​

By MelanX · October 23, 2026

Crate: tools/minecraft-options

Commit: 2b2af1e4f8166d02e4732d19e040a53745d97a51

Behavior

A mutation pass over preferences.py left all 45 tests passing for most mutations, including: the double-claim check, the inverted-domain check, step <= 0, "default below minimum", the unknown-tier and "excluded twice" checks, the historical/non-current field and unknown-registry-id checks, the "not manageable" check, the agrees min/max comparison, numeric kind chosen by field type instead of codec, the default_confidence == "high" gates for enum, numeric, and text/toggle defaults, the language-kind special case, modelPart section assignment, the enum bare-value fallback, enum sort by ordinal, the dumps flags, and help_translation_key.

Other defects in the tests:

  • test_a_numeric_step_is_kept_only_when_the_proven_domain_agrees never reaches the agrees mismatch branch because the fixture's fov codec min/max are None.
  • test_a_default_without_high_confidence_is_not_carried uses gui_scale, which has no enrichment entry, so it does not test the confidence gate.
  • STABLE_ID_OVERRIDES is only tested with assertIn, never the canonical-id-to-id mapping, and test_compatibility_only_changes_do_not_grow_the_catalog cannot catch build iterating the registry instead of the current ids.
  • main, --check, _load_normalized_record, and file writing have no tests.
  • RealCanonicalShape pins the committed dataset's counts (189/182/61/16/15), so an ordinary update.py run that adds a canonical setting breaks the unit tests even though the README calls growing the catalog a separate step.

Expected behavior

Each validation rule has a test that fails when the rule is removed, step and confidence tests exercise the branch they name, and counts are derived from the data instead of pinned.

Impact

Regressions in the generator or its validator ship unnoticed.

Locations

  • tools/minecraft-options/test_preferences.py
  • tools/minecraft-options/preferences.py

Validation and limitations

Mutation run on a scratch copy of the module (53 mutants, about 50 survived). The other points were read from the test source.