Aglet

Investigate Why an Agent Sounds More Certain Than Its Evidence

Investigation should explain how confidence strength diverged from evidentiary support. Trace a result from sources and comparisons through wording or labels, then test matched cases with different evidence quality. Keep missing evidence, instruction pressure, and a flawed confidence rule as separate possibilities.

Build a useful investigation brief

  1. Build a claim ledger

    List claims, evidence, alternatives considered, source timing, calculation, uncertainty, and confidence wording for the affected result. Mark direct, derived, interpretive, and unsupported statements. Include a comparable cautious result so the investigation has a reference with a known evidence boundary.

  2. Trace confidence formation

    Follow task instructions, evidence selection, intermediate criteria, conclusion, and final label or phrase. Check whether certainty was copied from an input, inferred from completeness, forced by output format, or disconnected from alternatives. Record each revision and decision point.

  3. Test matched evidence cases

    Hold the question constant while varying evidence completeness, source freshness, contradiction, and discriminator strength. Compare claim, alternatives, uncertainty, and confidence wording. If the same evidence supports several valid interpretations, record the policy choice needed rather than inventing a calibration formula.

What to carry forward

Produce a confidence investigation with claim ledger, evidence lineage, formation points, matched cases, and unresolved policy. Recommend a wording rule, evidence requirement, or bounded review change. Do not call the result calibrated from one plausible example or from stylistic consistency alone.

Keep the decision with the work.

Use a Work Item in Aglet to record the problem, the evidence you have, and the next decision. Add an owner and priority, then keep updates in the discussion so the next person can follow the reasoning.

Create an account See the product workflow