Repeated exposure changes the experiment
If every configuration is adjusted against the same small query list, the system can become very good at that list without improving general usefulness. Retrieval benchmark research depends on defined datasets and judgments, while ir_datasets emphasizes consistent handling of evaluation material. For an organizational pilot, the same principle means separating examples used during development from questions reserved for later assessment, even when the overall collection is still relatively small.
Keep a simple evaluation ledger
ABOLTUS experiments could divide reviewed questions into a development set and a held-out set, recording the split and source snapshot. Developers can inspect failures in the first group while the second group remains untouched until a comparison is ready. If a held-out failure becomes a development example, the record should say so and a fresh assessment set should eventually replace it. This avoids presenting a familiar regression suite as independent evidence that a new configuration generalizes to unseen tasks.
Report the limits of the sample
A small held-out set still has uncertainty, and its composition matters as much as its size. It should include exact references, paraphrases, missing-answer cases and relevant document formats. The goal is not elaborate statistical ceremony before useful work begins. It is a transparent boundary between learning from examples and claiming improvement on new ones. For ABOLTUS, such a ledger would make research results easier to interpret and help prevent a successful demonstration from being mistaken for broad evidence of retrieval quality across the organization's future documents and questions.