Every model can answer. Almost none can tell you when to trust the answer. idk scores every AI output with a calibrated confidence percentage — and returns an honest "idk" instead of a confident hallucination.
Language models are trained to always produce an answer — never to weigh whether they should. The wrong answer and the right one arrive in the same confident voice. That's fine for trivia. It's catastrophic for medicine, law, finance, and code.
Times a raw LLM says "I'm not sure" unprompted, per thousand answers it should. Confidence is uniform; correctness isn't.
Your current visibility into which AI outputs in production are wrong. If you can't measure it, you're shipping it.
The three-letter answer that would have prevented the fabricated citation, the phantom API, and the invented refund policy.
idk sits between any model and your users. Every output comes back with a calibrated confidence score, supporting evidence, and — below your threshold — a clean abstention.
A drop-in calibration API. Send any prompt + any model's answer; get back {answer, confidence, evidence, abstain}. Model-agnostic. Two lines of code.
Middleware for production AI. Audits every output in-flight, flags overconfident claims, blocks or reroutes answers below threshold, and keeps a confidence ledger for compliance.
A consumer answer app where every response shows its confidence — and sometimes the answer is just "idk." Radical honesty as a habit-forming product.
The engine estimates uncertainty from independent signals, then calibrates the fused score against outcomes — so "85%" means right about 85% of the time.
Resample the model and measure semantic agreement. Answers that wobble under resampling are answers you shouldn't trust.
Claims are decomposed and matched against retrieved sources. Unsupported claims drag the score down — hard.
Raw signals are fused and mapped to real-world hit rates using domain-tuned calibration curves.
Above threshold τ: answer + score + evidence. Below: a structured "idk" with a fallback route — human, retrieval, or retry.
This is not a mockup. Connect an Anthropic API key and the engine runs a live two-pass check — draft, then calibrate — and returns a confidence percentage. Below threshold τ, it abstains. Ask it about the future and watch it shrug.
Calibration isn't a nice-to-have on top of AI — it's the control layer that makes AI deployable where money and liability live.
Every output ships with a score, evidence, and a ledger entry. When a regulator, court, or customer asks "why did your AI say that?" — you answer with an audit trail, not a shrug. (We handle the shrugs.)
Most teams cap AI autonomy because they can't tell good answers from bad. With per-route thresholds, high-confidence answers auto-send while risky ones route to humans — so automation rates climb without incident rates following.
Emerging AI rules (EU AI Act and successors) demand risk management, transparency, and human oversight for high-risk systems. An abstention log — what your AI declined to answer, and why — is the cleanest oversight artifact yet invented.
Users who see honest confidence scores — including the occasional "idk" — trust the confident answers more, and come back. Admitted unknowns are marketing for every answer you do stand behind.
If a wrong answer costs more than an API call, you're the customer.
"Our bot promised a refund policy that doesn't exist."
with idk → only ≥90% answers auto-send; the rest route to agents with the draft attached. Deflection up, incidents down.
"The brief cited a case that was never decided."
with idk → every citation is decomposed and verified against sources; unverifiable claims abstain before they reach a filing.
"The summary invented a dosage."
with idk → τ=0.99 on anything clinical; below it, structured escalation to a human — with the uncertainty reasons attached.
"We can't deploy AI we can't audit."
with idk → the confidence ledger turns every output into an auditable event: score, evidence, decision, route. Exam-ready.
"One viral hallucination undid a quarter of trust-building."
with idk → two lines of code, per-route thresholds, and a UI primitive users learn to love: the confidence chip.
"It didn't ask. It just booked the wrong thing."
with idk → pre-action confidence gates: agents check "how sure am I?" before they click buy. Below τ, they stop and say idk.
We only charge when the engine is confident enough to let an answer through. When it says idk, so does your invoice.
*Nothing is certain. We're the company that tells you that.
Overconfidence is a bug.File it. Fix it. Ship the patch.
"I don't know" is a feature.The most underrated string in computing.
Trust ships with a number.No score, no deploy.
Calibration beats charisma.85% should mean 85%.
Abstain, then escalate.A shrug with a plan attached.
Honesty compounds.Every admitted unknown buys the next answer credibility.
Our white paper lays out the full argument: why models bluff, how calibration works, and what an honest AI stack looks like.
Download the white paper (PDF) hello@idk.io