Evaluation

Featured · recent

A close read of arXiv:2610.19095. Chatbot Arena (LMArena) gives each model one Bradley–Terry score, which assumes Luce's choice axiom. On 1.8 million votes, versions of one model are judged 1.37x more decisively than their score gap implies, sizes of one generation 1.26x, models of one lab 1.12x. The overall order barely moves, but head-to-head odds between siblings are pulled toward even. The mechanism, how strong each piece of evidence is, how it relates to style control and the Leaderboard Illusion, and what to do when you read or build a pairwise leaderboard.

Featured · recent

An operator's synthesis of OpenRouter's cost–quality frontier, CI eval gates, and confidence escalation—plus provider routing, FrugalGPT/RouteLLM, and on-site eval-reliability posts—for coding-agent engineers.

Featured · recent

Jev made decision models a category. This survey groups a dozen-plus hosted and open models by how they work, checks vendor claims against Decision Index, JevBench and Red Hat's tests, explains why public scores, latency and "calibration" don't compare, and ends with a comparison table and an agent-loop checklist.

Featured · recent

OpenAI released 700+ AI math manuscripts, then withdrew three. We split "AI proved X" into five layers, show what a machine checks, and give an audit checklist.