Preference catalog tests do not detect many validation and presentation regressions
Original description
By MelanX · October 23, 2026Crate: tools/minecraft-options
Commit: 2b2af1e4f8166d02e4732d19e040a53745d97a51
Behavior
A mutation pass over preferences.py left all 45 tests passing for most mutations, including: the double-claim check, the inverted-domain check, step <= 0, "default below minimum", the unknown-tier and "excluded twice" checks, the historical/non-current field and unknown-registry-id checks, the "not manageable" check, the agrees min/max comparison, numeric kind chosen by field type instead of codec, the default_confidence == "high" gates for enum, numeric, and text/toggle defaults, the language-kind special case, modelPart section assignment, the enum bare-value fallback, enum sort by ordinal, the dumps flags, and help_translation_key.
Other defects in the tests:
test_a_numeric_step_is_kept_only_when_the_proven_domain_agreesnever reaches theagreesmismatch branch because the fixture's fov codec min/max areNone.test_a_default_without_high_confidence_is_not_carriedusesgui_scale, which has no enrichment entry, so it does not test the confidence gate.STABLE_ID_OVERRIDESis only tested withassertIn, never the canonical-id-to-id mapping, andtest_compatibility_only_changes_do_not_grow_the_catalogcannot catchbuilditerating the registry instead of the current ids.main,--check,_load_normalized_record, and file writing have no tests.RealCanonicalShapepins the committed dataset's counts (189/182/61/16/15), so an ordinaryupdate.pyrun that adds a canonical setting breaks the unit tests even though the README calls growing the catalog a separate step.
Expected behavior
Each validation rule has a test that fails when the rule is removed, step and confidence tests exercise the branch they name, and counts are derived from the data instead of pinned.
Impact
Regressions in the generator or its validator ship unnoticed.
Locations
tools/minecraft-options/test_preferences.pytools/minecraft-options/preferences.py
Validation and limitations
Mutation run on a scratch copy of the module (53 mutants, about 50 survived). The other points were read from the test source.