Test an AI answer against a question you can actually check

An answer can sound finished too soon

NIST’s Generative AI Profile identifies confabulation as a risk, and its Generative AI evaluation program provides an official example of structured measurement work. For a community reader, the practical issue is an answer that presents an unsupported assertion with confidence.

A polished response about a local institution may still contain a wrong date, invented source or missing qualification.

This article proposes an original public-information exercise for a Southern Virginia AI observatory. It reports no benchmark already run and makes no ranking of current products.

The exercise should use a question whose answer can be checked in a primary document. The narrower the question, the easier it is to distinguish a factual error from a stylistic disagreement or a difference in scope.

Design one fair trial

Select a public document, record its title and date and write one question about a clearly stated fact. Prepare the verified answer before testing an AI tool.

Keep the question free of personal, school, medical or confidential business information. A harmless public test does not need a private document just to make the task look realistic.

Record the exact question, date, tool label visible to the user and answer. Preserve whether the system was allowed to browse or was given the document.

Those conditions matter. An answer produced with the source in front of it should not be compared casually with an answer produced without access to that source.

If a product changes, the earlier test remains a dated observation.

Score the claim, then inspect the source

Use a simple original rubric: supported, contradicted, not established or omitted. Apply it to individual factual claims, not the whole paragraph’s tone.

If an answer names a source, open the source and find the passage that supports the claim. A working link does not prove that the linked page says what the AI answer attributes to it.

Add a second review question: did the response admit a gap when the document did not answer the question? That behavior deserves attention alongside accuracy.

A system can be helpful by preserving uncertainty rather than filling it with a plausible detail. Keep a human reviewer’s notes separate from the generated text so a reader can tell which conclusions came from the test and which are editorial judgments.

Publish a bounded observation

A community note should identify the small sample, test conditions and errors actually observed. It should not say that one trial proves a system always succeeds or fails, or that all local jobs will change in a particular way.

If sharing the result, include the verified source and the rubric so others can repeat the exercise. A modest observatory becomes useful through a recoverable method: a clear question, a primary record, a dated answer and an honest account of the limits.

Keep the original response available alongside any correction. Rewriting the AI output before scoring it can accidentally remove the very error the trial was meant to examine.

If a reviewer corrects the answer for publication, label the correction as human work. That makes the observatory’s evidence clear and prevents a polished final paragraph from disguising how the tool actually performed.