Pick real pieces of work your team handles: a weekly report summary you found accurate, an email draft the tool got badly wrong, a project brief the summarizer flattened into mush. Save them in a private folder on your desktop or shared drive. Call it “test cases.” This collection becomes your private test set, a repeatable group of inputs you use to compare tool versions.
A repeatable test used to compare performance is what researchers call a benchmark. [1] The word sounds formal, but the idea is simple: you run the same inputs, compare the outputs, and decide whether things got better or worse. Your folder is a desk-scale version of that logic.
Write the Pass Rule Before You Look
Before you run a test, write down what “good” looks like for each case. Do this before you see the outputs.
Without a written standard, the first output can become the reference point. You stop asking whether it is good enough and start asking whether it is better than the last result. Writing the pass rule first prevents the output from quietly changing the standard.
For a report summary, the rule might be: the output names the main decisions the original report recommended, in order, without adding context that was not in the source. For an email draft: the tone matches the sender’s usual register, with no invented commitments. Keep each rule simple, specific, and checkable.
NIST’s preliminary, voluntary draft guidance on automated benchmark evaluations organizes the work around defining objectives and selecting benchmarks, then running evaluations, then analyzing and reporting results. [2] Your pass rule is the objective, written in language your team can apply.
Test the Version You Will Use
When a vendor announces a new tool version, run your saved cases against it before rolling it out or renewing the subscription. A published score does not answer this local question. Scores depend on the model, the evaluation setup, the attempts and tools used, the examples sampled, and whether the model had prior access to them. [1] Those conditions may differ from the work and setup your team will use.
Your folder gives you examples tied to your work, but it does not prove the model has never seen similar material. The non-peer-reviewed arXiv preprint says a private test set closes only the most direct route of prior exposure, not every way information can leak into an evaluation. [1] Treat the folder as practical local evidence, not proof of a clean laboratory test.
Run each case with the settings your team will use. Save the output and mark it pass or fail against the rule you wrote earlier. Also note any review or correction the output needed. That record becomes your comparison point when the tool changes again.
Write Down What the Test Cannot Tell You
After you record the results, add a short note about what the folder does not prove.
It does not prove the tool is safe for every use. It does not prove the absence of bias in outputs you have not tested. It does not prove the tool will perform well on work unlike your saved examples. NIST says automated evaluations cannot meet every evaluation objective. [2] Your desk test answers a narrower question: did this tool version meet your written standard on these cases under these conditions?
Write that narrow claim beside the results. It gives the next decision-maker evidence tied to the team’s work instead of a score remembered without its conditions.
Before the Next Renewal, Open the Folder
The folder matters because it changes a real decision. Build it before the renewal conversation starts, not after. Write the pass rules before you run the test. Test the version you are actually evaluating, not a demo environment. Record what passed, what failed, what needed correction, and what the test could not show.
That sequence gives you evidence tied to your work, your team, and your standards. Create the folder and write the pass rules before the next renewal or rollout decision.