Technical articles on AI evaluation, failure modes, and what we're learning from production deployments.
26 May 2026
Domain-calibrated guardrails for AI scribes. Catches hallucinated medications, fabricated values, and unsupported clinical inference at the inference boundary, with sub-second latency.
Ryan Lail
15 Apr 2026
Most failure-mode analysis treats failures as a static taxonomy. In production they're a living distribution - some dominant, some emerging, some being squashed by recent fixes. Here's a clustering pipeline that tracks all three.
Luke Markham
15 Apr 2026
Build or buy eval infrastructure? The six layers after a baseline judge (ensemble through meta-eval), why a first slice of scoring is not the same milestone as release-grade depth, and how we benchmark quality (71% baseline judge to 83.6% on our internal set).
Michael Karotsieris
Domain-calibrated guardrails for AI scribes. Catches hallucinated medications, fabricated values, and unsupported clinical inference at the inference boundary, with sub-second latency.
Most failure-mode analysis treats failures as a static taxonomy. In production they're a living distribution - some dominant, some emerging, some being squashed by recent fixes. Here's a clustering pipeline that tracks all three.
Build or buy eval infrastructure? The six layers after a baseline judge (ensemble through meta-eval), why a first slice of scoring is not the same milestone as release-grade depth, and how we benchmark quality (71% baseline judge to 83.6% on our internal set).
Findings from clinical AI engagements - actual failure patterns from production clinical AI, categorised by type, with real examples. Discussions becoming decisions, dangerous omissions, dosage errors, and diagnostic leaps.
Our open-source research on RewardBench 2 shows that two simple techniques — task-specific criteria injection and ensembling — improve LLM judge accuracy by up to 13.5pp (71.7% baseline → 85.8%).
A structured taxonomy of 60+ failure modes across eight categories, synthesizing recent research into a practical framework for understanding how and why large language models fail.
Introducing Composo: AI evaluation that learns your standards. Not manual review, not LLM-as-judge -- a third option that gets better the more you use it.
A deep dive into Composo's generative reward model architecture that achieves 95% agreement with expert evaluators, compared to ~70% for LLM-as-judge approaches.
A practical guide to evaluating LangGraph multi-agent workflows using Composo's agent evaluation framework with quantitative scoring across 5 key dimensions.
LLM-as-a-judge consistency is largely illusory. The same hallucination produces wildly different scores depending on scale configuration, undermining evaluation trust.
Published AI scribe failure rates are higher than vendor marketing suggests. Real lawsuits have started. Here is what the specific failure patterns look like and what clinical evaluation needs to catch.
A component-based evaluation framework for agentic LLM systems covering tool call formulation, tool choice, response integration, reasoning evaluation, and system-level analysis.
A practical guide to evaluating clinical AI in production: specific failure modes to catch, why generic evaluation misses them, regulatory context, and what good quality infrastructure looks like.
A comprehensive guide to evaluating LLM classification quality, covering supervised metrics, generative reward models, and LLM-as-judge approaches.
A comprehensive guide to evaluating RAG applications, covering generation metrics, retrieval assessment, and advanced CAG-based oracle evaluation techniques.
Composo Align achieves 95% agreement with expert preferences vs 72% for LLM-as-judge, with 100% score consistency through its deterministic generative reward model.
Composo Align uses a generative reward model architecture to provide deterministic, consistent scoring for LLM evaluation, achieving 95% agreement with expert preferences.
A plain-language explanation of LLM evaluation: what it is, why LLM-as-judge plateaus, what production AI quality actually requires, and how to think about build vs buy.
LLM evaluations that worked last quarter can silently stop working this quarter. Evaluation drift is a first-class failure mode of production AI quality systems. Here is how to detect it and what to do about it.
An honest analysis of the real cost of building an LLM evaluation pipeline internally vs buying a deployed quality layer. Real numbers, real timelines, real trade-offs.
A structured guide to evaluating LLM applications, covering common challenges with human vibe checks and LLM-as-judge, and key steps to building a reliable evaluation framework.
A practical guide to AI guardrails for production LLM systems. What they catch, what they miss, and how to deploy domain-specific guardrails that actually work at inference time.