Department series
Model Watch
What we know and do not know about models — opinions, briefs, and eval teasers.
Dispatches
Filed from the floor — the patterns unfreeze in Miami, and the Linux moment arrives on the laptop.
-
Which Open Model Should You Actually Run Locally?
We tested 7 model families across 3 runtimes on a 64 GB Mac. gemma-4-26b-a4b hits 95% in 4 minutes with thinking off. The runtime barely matters — but thinking mode changes everything.
-
Gemma 4 on Your Machine: How Google’s New Open Weights Stack Up (Model Showdown)
We benchmarked Gemma 4 (e2b, default, 26B MoE, 31B dense) through Ollama against 50+ hosted and local models on reasoning, knowledge, instruction, coding, and TezLab MCP tool use—same Umwelten harness as our other showdowns. Here’s where the new line shines, where frontier models still pull ahead, and how the biggest Gemma handles real EV data tools.
Proofs
Show your work — 131 models, honest margins, and the thesis that it's agents all the way down.
- Simple gotchas beat leaderboards — 76% of 131 models failed a one-question common-sense test. — The Car Wash Test · Mar 1
- The weights aren't the product: identical Nemotron weights scored 1/6 to 6/6 on tool use across four providers. — Same Weights, Different Results · Mar 24
-
Same Weights, Different Results
We ran the same Nemotron model on four providers and got wildly different results. MCP tool use ranged from 1/6 to 6/6. Speed varied 16x. The weights are identical. The results are not.
-
Can LLMs Use Real-World Tools? Mercury-2, ELO, and the Umwelten Setup
We ran 39 models against real Rivian driving data via MCP tools. Inception's Mercury-2 delivered a perfect 15/15 in 8.6 seconds. Here's the standout model, the ELO narrative rankings, and how the same Umwelten setup powers both chat and evals.
-
The Car Wash Test: Learning from Model Evals
We asked 131 AI models a simple question — should I walk or drive to the car wash? 76% got it wrong. Simple gotcha questions reveal more about model reasoning than any benchmark leaderboard.
Instruments
Reading the gauges — before you trust an agent, you need instruments that show you when it's lying.
The Console
Moving into the terminal — the agents live where the work lives, and code becomes disposable material.
- Local caught up for real work — gpt-oss and llama3.2 generate usable code on a laptop, no cloud required. — Code Generation with Local Models · Aug 20
-
gpt5 is smarter than you are
gpt5 can choose to be so smart it's almost impossible to judge. Lets see how it does on some unanswerable questions and if it can totally replace google.
-
Code Generation with Local Models
Small, local AI models deliver surprisingly effective results for everyday tasks. Also llama3.2 is surprisingly fast and gpt-oss is surprisingly good.
-
gpt-5 and gpt-oss
OpenAI’s GPT-5 launch stole headlines, but GPT-OSS quietly made local AI a lot more practical. This post covers what’s new, how to run it with Ollama or LM Studio, and why context size can change your results.
Field Trials
Take the tools outside and see what breaks — agents become products, and products can be raced.
- Fifteen agents, one job: raw capability is table stakes — workflow fit decides the winner. — June 2025 Coding Agent Report · Jun 15
-
June 2025 Coding Agent Report
A comprehensive analysis of 15 leading AI coding agents in 2025. We break down the strengths, weaknesses, and surprises from top tools, with clear winners for pros, tinkerers, and casual users alike.
-
Thoughts on gemini
Despite popular narratives about Google lagging in AI, their Gemini models reveal engineering excellence that's hard to ignore when you strip away the conservative product decisions and UI polish. From the lightweight yet powerful Gemma 3 to the multimodal capabilities of Gemini 2.5, Google's models demonstrate a level of speed, precision, and fundamental understanding that suggests they're not playing catch-up—they're just being cautious.
Blueprints
Drawing up the system — memory, MCP, structured output: the quarter the chat window stopped being the product.
- Models sort into castes: small and fast, foundation, and 'educated' models that carry a learned process. — How I Classify Models · Jan 21
-
Moral Vibe Check
Technical correctness and meaningful insight: well-formatted, detailed AI responses can mask a fundamental lack of understanding—a "raving lunatic" hidden behind impressive form. Maybe P-doom is less about malice and more of making us intellectually poorer by substituting form for substance, facts for understanding, and technical accuracy for wisdom.
-
How I classify models
Small models are smart yet limited in knowledge; foundation models possess both deep understanding and extensive knowledge but lack structured problem-solving approaches. Educated models like DeepResearch excel by combining learned reasoning processes with large memory capacities, enabling them to adapt effectively to complex tasks while handling vast information instantaneously.
-
AI for research: DeepResearch a clear winner
Asking the tough questions: DeepResearch excels in depth and comprehensiveness, while o1, Sonnet 3.5, and DeepSeek with DeepThought provide comparable results for complex inquiries. Smaller models like phi4 and llama3.2 are deemed inadequate for intricate topics.