State.

Ask a team why their coding agent feels productive on Tuesday while their procurement, accounting, or macro-research agent is still stuck in pilot six months later, and the honest answer almost always comes down to oracles. Software engineering comes pre-packaged with compilers, type checkers, unit tests, and stack traces. Enterprise operations—reviewing supplier contracts, reconciling multi-year tax schedules, writing in an executive’s voice, or scanning adverse media before an earnings call—have none of them.

Worse, the two fallback verification mechanisms most organizations reach for—human review and off-the-shelf LLM-as-judge prompts—fail simultaneously as volume scales.

Why human review and naive LLM judges both buckle

Tariq Shaukat (Sonar) laid out the arithmetic of that double failure. Across enterprise software teams Sonar tracks, AI-generated code volume surged 2,450%, yet shipped release velocity rose only 30%. The bottleneck moved squarely into verification, where human reviewers exhibit cognitive surrender: when an AI assistant is right, reviewers agree with it 93% of the time; when the AI is wrong, reviewers still agree with it four out of five times (80%). Plausible formatting lulls tired reviewers into rubber-stamping subtle logic bugs and security regressions.

Handing that review burden to a standalone LLM judge without deterministic scaffolding merely trades human fatigue for stochastic drift:

• Across identical replications of the same input, standalone LLM judge verdicts flip 7% to 17% of the time.
• Challenge a judge with a trivial follow-up prompt (“Are you sure?”) and verdicts change 7% to 34% of the time.
• Meanwhile, compounding step error punishes long trajectories brutally: an agent operating at 95% per-step accuracy drops to 36% end-to-end reliability by step 20 (0.95²⁰ ≈ 0.358), which is why METR’s headline multi-hour task horizons contract sharply the moment required reliability moves from a 50% coin flip to an 80%+ enterprise SLA.

Shaukat framed shipping confidence as a four-term product borrowed from Toyota production engineering—explicit intent × in-loop guide rails × independent hybrid verification × auditable evidence—where letting any single term hit zero collapses total confidence to zero. Supplying structural codebase maps and static verification rails directly inside the agent loop (Sonar Vortex) cut token consumption by 36% and reduced commit-time issues by 92% on long-running tasks. In a companion session, Manish Kapur (Sonar) and Addy Osmani (Anthropic) showed what that looks like at frontier throughput: scaling Anthropic’s internal engineering pipeline from roughly 2,500 to 30,000 merged changes per day required abandoning line-by-line human diff inspection as the primary safety net in favor of automated first-pass reviews, diff-triggered edit hooks, property-based and end-to-end tests, supply-chain CVE scans, browser-exercised screenshot/video proof, and explicit labeling of agent-authored pull requests while keeping a named human accountable for production outcomes.

Five ways New York teams bootstrapped ground truth where none existed

If you cannot trust tired humans or raw LLM judges, and public benchmarks do not cover your domain, how do you measure an agent before putting it in front of customers or regulators? Five distinct engineering patterns emerged across the week:

1. Write the answer first (Air Canada)

Nathanael Weill (Air Canada) confronted two classic enterprise dead ends: Q&A over roughly 10,000 telecom vendor documents and automated legal review of supplier contracts—with zero public benchmarks and scarce subject-matter-expert hours. On the Q&A system, Air Canada noticed an empirical simplification: whenever the retriever surfaced the right source page, the downstream model produced the right answer 90% of the time. Instead of grading free-form prose with an LLM judge, Weill’s team inverted dataset generation—sampling known source pages, generating synthetic questions from those answers, and checking page retrieval deterministically in code. Iteration speed jumped fivefold, from one experiment per week to one per business day.

Supplier contract review looked harder. The team initially hoped to evaluate redlining by diffing unsigned initial drafts against final signed agreements, only to discover procurement never archived the unsigned originals. When they tried grading agent reviews with an LLM-as-judge harness, the judge’s variance rivaled the contract agent’s own error rate—calibrating the judge was nearly as hard as building the product. So they inverted the problem again: taking approved legal contract templates with blank clause slots and populating each slot with synthetic clause arrays engineered by construction either to violate or satisfy specific procurement rules. Every synthetic contract now carried exact ground-truth breach labels and ran end-to-end in five minutes. Where the original workflow still resisted deterministic grading, Weill’s team re-architected the agent itself around evaluable intermediate steps rather than shipping an unmeasurable black box.

2. Backtest the web without lookahead leakage (Exa)

For hedge funds, insurers, and risk teams building predictive or research agents, evaluating web search introduces a subtler trap: temporal leakage. Test an earnings, credit, or adverse-media agent against live web search today and the index already contains post-announcement articles, updated company bios, and hindsight commentary. Waiting in live production to see whether an expressive semantic search pipeline genuinely predicts outcomes or merely overfits takes months.

Ryan Ahern (Exa) introduced Exa Snapshot—an archive of 400 billion+ daily-indexed web snapshots captured continuously during freshness crawls—allowing agents to run point-in-time URL lookups and semantic searches strictly as the web existed one day, seven days, or thirty days before a historical event. Equally important for multi-step agent trajectories, Ahern showed why raw full-page scraping wrecks both accuracy and budgets: unlike humans who type three keywords and skim one link, agents issue dense bursts of exploratory queries whose context tokens compound on every subsequent reasoning turn. Returning query-relevant dynamic snippet highlights across results delivered higher downstream accuracy at 95% fewer tokens than full-page text.

3. Multi-draft sampling and cross-family judges for subjective quality (Towards AI & Greptile)

What happens when the task is genuinely subjective—like matching a specific author’s writing voice—or hunting rare 1-in-1,000 outage bugs in pull requests?

Louis-François Bouchard (Towards AI) built ToneBench across 120 models and ten long-form scripts because AI prose slop persists even on frontier models: left to a single pass, models collapse toward a voiceless, generic average. Three measurement rules stood out:

• Sample multiple drafts per prompt. Intra-model variance widens sharply as model capability drops, meaning single-sample evals routinely reward lucky 1-in-5 outliers. Generating five drafts per prompt captured dispersion cleanly without blowing up benchmark cost.
• Categorize before scoring. A 14,000-word judge prompt structured as a two-stage rubric—first placing a draft into one of six qualitative voice buckets, then scoring across nine metrics—aligned far better with 1,000+ human pairwise comparisons than direct numeric or binary grading.
• Never let a model family grade itself. LLM judges showed pronounced family-wide self-preference (with OpenAI models systematically favoring OpenAI outputs across sizes), requiring cross-family judges. And once ToneBench existed, a striking production pattern emerged: generating 10 drafts with a 50×-cheaper model (GLM 5.3 Flash) and merging the best passages in a single Opus 5.5 high-thinking pass beat a single frontier pass on both voice fidelity and total cost.

Daksh Gupta (Greptile) reported the exact same cross-family phenomenon in code review. Naive sandboxed PR reviewers burn $3 to $25 per pull request spraying minor style nitpicks while missing the 1-in-1,000 catastrophic outage bug. Because live traffic can only confirm bug presence (never bug absence), Greptile evaluates every prompt and routing change against a curated catalog of historical outage-causing pull requests—and found that inverting model families between author and reviewer (for example, reviewing Claude-authored code with a GPT family model, or vice versa) increases real bug catch rates by 10%, because models within the same post-training lineage share correlated blind spots.

4. Opinionated behavior specs for long-horizon work (Basis)

In accounting and tax preparation, trajectories routinely span hours and hundreds of tool calls with no clean unit-test binary at the end. Mitchell Troyanovsky (Basis) argued that general-purpose “chatbox” agents fail inside accounting firms for two reasons: a competency gap (pushing prompt and memory management onto busy accountants) and an unstandardized process gap (an unopinionated agent may reach a plausible number via an indefensible workflow).

Basis treats every token entering an agent’s context window—system prompts, progressive skill disclosures, tool schemas—as runtime training data, and encodes firm methodology into behavior specs: explicit process invariants such as “always cross-check state apportionment against primary statutory tables before populating workpapers” or “render and visually inspect every exported slide deck before delivery.” Because behavior specs grade whether the trajectory followed mandatory professional procedure whenever a trigger condition arose, they supervise arbitrary production runs without needing pre-labeled ground-truth answers. When an agent violates a spec at step 410 of a thousand-call run, engineers avoid re-running three hours of compute either by forking trajectory state immediately prior to the bad turn or by spinning up a cheap coding subagent that simulates the production harness to iterate on context fixes in seconds.

5. External MCP completion gates + System One judges (WorkOS & Arize)

Even when deterministic checks exist, letting an agent run its own test command and self-report "Done!" fails in subtle ways. Nick Nisi and Zack Proser (WorkOS) recounted watching an agent pass a shallow UI assertion on an optimistic DOM update while silently dropping the backend write—losing customer data on page reload. Their fix decoupled verification authority from the worker altogether:

1. Centralize completion behind an external MCP verifier (Ctx). Agents receive an outcome target—not step-by-step micromanagement—and must ask an independent MCP service ("Am I done?"), which executes application-specific verifier CLIs, reloads state, and writes signed run receipts. As WorkOS put it, once those deterministic verifier CLIs exist, the verifiers become more durable than the application code itself, letting teams port or rewrite implementations safely.
2. Emit state-rich error diagnostics. Verbose failure payloads explaining how state diverged turn verification rejections into self-healing closed loops.
3. Pair deterministic state checks with fast System One models. Strict string assertions break whenever UI confirmation copy changes; calling a sub-second System One classifier (the same pattern we benchmarked with TypeSafe’s Jev in our September Umwelten study) checks semantic UI state without brittle regex coupling.

Aparna Dhinakaran (Arize AI) zoomed that loop out to the whole engineering role. While six billion people have touched chat LLMs and 20 million professionals use coding tools, roughly 5,000 frontier practitioners currently orchestrate swarms of thousands of always-on agents. In Arize’s architecture (Signal), observability is formatted for agents as primary consumers: 3-to-6-hour trace sweeps, metric thresholds, and Slack complaints automatically wake ephemeral, credential-scoped sandboxes behind a traced proxy, checkpointing discovered issues and skills externally while meta-evaluating every online LLM judge against human golden sets—and tracing both production agents and the coding harnesses (Claude Code, Cursor, Codex) that built them.

Domain evaluation patterns at a glance

Organization	Domain bottleneck	Why naive eval failed	Ground-truth & verification architecture	Measured effect
Air Canada	10K vendor docs & procurement redlines	Missing unsigned drafts; LLM judge noise ≈ agent error	Write the Answer First: page-ID targets + synthetic breach/pass clause templates	1/wk → 1/day iteration; 5-min contract evals
Exa	Predictive financial & adverse-media search	Live web leaks post-event outcomes; full pages bloat trajectories	Exa Snapshot (400B+ daily web captures) + dynamic snippet highlights	Zero lookahead bias; 95% fewer search tokens
Towards AI	Author-specific long-form writing	Single-draft outliers & OpenAI family judge self-preference	ToneBench: 5 drafts/prompt, 2-stage rubric, cross-family judges + 10-draft cheap/Opus merge	50× cheaper drafting (GLM 5.3 Flash) + higher voice match
Greptile	Rare (1-in-1,000) outage bugs in PR review	$3–$25 noisy nitpicks; live runs never prove bug absence	Historical outage PR catalog + cross-family author/reviewer inversion	+10% bug catch rate; 20-subtask routed cost
Basis	Multi-hour tax & accounting workflows	Outcome-only evals miss indefensible intermediate steps	Ground-truth-free behavior specs + trajectory state forking	Auditable CPA-grade process supervision
WorkOS	Multi-step app & UI mutations	Optimistic DOM false positives & brittle copy regexes	External MCP verifier (Ctx) + state-rich CLI errors + System 1 judges	Persisted proof receipts; rewrite-safe verifiers
Sonar & Anthropic	30,000 daily merged code changes	80% human cognitive surrender; 7–17% LLM judge flip rate	In-loop structural maps (Vortex) + static/CVE/browser video gates	−36% tokens, −92% commit issues

Nowhere was the contrast between Silicon Valley demo culture and New York production reality sharper than in the financial-services sessions. Why does enterprise adoption drop tenfold—from 50% in software engineering down to ~5% in business and investment workflows (Alex Lieberman, Tenex)—even though frontier models can now pass bar exams, read credit indentures, and write SQL?

Belinda Neal and Keon Shahab (Anthropic) opened their financial-services session with a blunt diagnosis: model intelligence is no longer the binding constraint on Wall Street. Capability, unit token cost, and prompt-injection resistance now outrun institutional absorption. What stalls deployment is that most organizations build agents one bespoke proof-of-concept at a time—wiring custom connectors, burying compliance rules inside system prompts, and putting human approvers in front of every button click until review fatigue turns safety theater into rubber-stamping.

Across six back-to-back sessions from Two Sigma, Coinbase, Anthropic, Bridgewater, Apollo Global Management, and Coatue, a remarkably consistent blueprint emerged for crossing that 95% wall—paired with a cold shower from Apify on why autonomous wallet-to-wallet commerce hasn’t happened yet.

Escaping the sandbox-versus-prompt trap (Two Sigma, Coinbase, Anthropic)

John Feminella (Two Sigma) named the two anti-patterns that paralyze enterprise security teams:

1. Total isolation: Locking agents inside air-gapped sandboxes with no credentials or internal network paths—completely safe, and completely useless for real operations.
2. Policy-by-prompt: Pasting corporate compliance PDFs into system instructions or asking humans to click “Approve” on every tool call—which neither enforces boundaries at runtime nor survives reviewer fatigue.

Two Sigma replaced both extremes with five composable runtime controls designed so the paved road is faster than shadow IT:

• Zero-friction automatic registration. Instead of a heavyweight pre-approval committee that drives engineers underground, building on an approved Two Sigma agent framework automatically registers the agent’s human owner, business unit, declared tools, and network endpoints in a central catalog.
• Risk-tiered review queues. Uniform sign-off stalls a harmless internal meeting summarizer for two months behind the same compliance queue as a production trading-adjacent workflow. Two Sigma scores risk directly from declared tool entitlements and target systems so low-risk builds ship immediately.
• Registry-compiled runtime entitlements. Rather than maintaining coarse firm-wide allowlists, managed execution environments query the central agent registry at initialization time and compile that specific agent’s declared host, secret, and tool scopes into kernel- and proxy-enforced boundaries.
• Self-serve unit-test evaluation harnesses. Because swapping or upgrading a foundation model acts like upgrading a non-deterministic runtime dependency, central platform teams provide shared evaluation runners—analogous to pytest or CI—while domain teams write only task-specific assertions.
• Deterministic fast paths + conservative classifier escalation. Explicit rules run at wire speed in code; genuinely ambiguous actions pass through an independent, conservative classifier so human reviewers only see edge cases worthy of human attention. (Compare this directly with our Q4 study on New Quality Productive Forces and the access-boundary rules in standards.thefocus.ai.)

Evan Kormos (Coinbase) formalized the compliance half of that architecture. Regulated institutions suffer when teams conflate build-time policy—written normative intent expressed in must, may, and should obligations (“the organization’s original system prompt”)—with the run-time system, which mutates daily as products and schemas ship. By keeping normative policy separate and compiling gated workflows into directed acyclic graphs (DAGs), Coinbase runs multiple previously cost-prohibitive independent compliance and risk checks in parallel, aggregating their verdicts at deterministic merge gates without serializing latency or weakening mandatory controls.

Anthropic’s financial-services team (Belinda Neal and Keon Shahab) packaged the same separation into a five-layer reusable foundation that takes banks and asset managers from one-off builds to dozens of governed workflows:

1. Unified MCP connectivity abstracting source systems so teams stop maintaining an N × M integration matrix;
2. SME-authored plain-text skills (such as trade-break investigation playbooks) versioned in Git and tested like production code;
3. Least-privilege role identities per agent;
4. Deterministic hook guardrails and write sandboxes, where an agent investigating a trade break can prepare a booking amendment but cannot commit to the ledger until a named human SME approves; and
5. Continuous regression evals closed directly with business SMEs—demonstrated publicly in Anthropic’s wealth-management suite pairing 25 custody/CRM/portfolio connectors with eight advisor-gated open-source workflow skills.

Why financial agents live or die on bitemporal ontologies and golden sources

Runtime guardrails prevent unauthorized writes, but they do nothing if the agent reads stale or contradictory numbers. Three investing talks—from Bridgewater, Apollo + Rowspace, and Coatue—showed what institutional context engineering actually entails when billions of dollars sit downstream.

Bridgewater’s Vault: bitemporal enrichment and 1,000-QPS agent storms

Aaron Linsky and Sam Green (Bridgewater) opened with a live failure mode every macro investor recognizes. Ask an open-web chat agent what is driving the Korean won today, and out of 15 retrieved links it quietly leans on a three-month-old note, synthesizing an outdated narrative with total fluency. Point that exact same model at Bridgewater’s internal Vault—which ingests 100 million+ documents across 135 external feeds plus decades of internal research memos and recorded investment debates—and the agent time-scopes the query, matches 52 current candidates, and synthesizes seven fresh primary sources.

Three engineering decisions made Vault compound rather than rot:

• Bitemporal firm ontology. Every ingested document is enriched not only with Bridgewater’s canonical entities (countries, markets, macro concepts) and publication timestamp, but also with the time period discussed inside the text, preventing historical retrospectives from polluting current-regime queries.
• Cluster-local native filtering for agent storms (turbopuffer). Human researchers issue roughly 100 queries per hour; deep-research agent swarms trigger 100 to 1,000+ queries per second, almost every one constrained by date range, asset class, and source entitlement. Standard vector pre-filtering collapsed under latency at 100M+ scale, while naive post-filtering missed sparse matches. Switching to turbopuffer’s native cluster-local filtering—Paired with 200 hand-labeled representative investor queries built in under a week—drove retrieval precision@10 from 60% to ~95%.
• Habit-first UX with automatic synthesis write-back. Investors lived in broker distribution emails and Outlook folders. Rather than forcing a foreign chat UI, Vault turned distribution lists into saved searches and inbox folders into saved collections first, then layered on Pocket Analyst inside the Investor Workbench—where every human-agent deep dive writes its verified synthesis back into Vault, compounding institutional memory on the next cycle.

Apollo Global Management & Rowspace: golden-source precedence over chatbot summaries

Jack McDonald (Apollo Global Management) and Michael Manapat (Rowspace) explained why off-the-shelf earnings summarizers barely move the needle in private credit and active collateralized loan obligation (CLO) management. In private credit, the real friction lives in middle- and back-office data reconciliation: a single financial attribute for a borrower can appear across 20 internal systems and trustee reports with conflicting values, and only one of them is authoritative under a given legal context.

Instead of shipping another research chatbot, Rowspace embedded engineers side-by-side with Apollo portfolio managers and traders every day to encode golden-source precedence rules into continuous data-engineering agents:

• Weekly warehouse loan reconciliation—which previously took two to three staff members hours of manual bank-statement cross-checking—now runs end-to-end in one minute, shifting staff to daily exception QA.
• Mock CLO portfolio construction across 200 to 400 loan names—balancing indenture waterfall rules, concentration covenants, and live desk pricing—dropped from two to three days in manual Excel spreadsheets to minutes for the first 80–90% of the allocation, freeing PMs to spend their time on final credit judgment and trade execution.

Coatue: a hedge fund as a composable state-machine function

Frank Long (Coatue) distilled the architectural philosophy shared by both firms into a single signature:

HedgeFund(context_external, context_internal) -> (stock, size, timing)

Two rules govern Coatue’s buildout. First, pixel-to-API parity: “Every single thing that a human can see in pixels on the screen, an agent in your building should be able to see via API”—positions, liquidity constraints, factor exposures, external feeds, and twenty years of internal trade notes. Second, anti-dogmatic runner selection: “There is no glory in automating things for the sake of automating things.” Each compartmentalized node in the research graph—with 10+ ingestion and reasoning knobs (OCR vs. raw vision, chunking, thinking budget, skills) tuned like ML hyperparameters against an explicit north-star metric—is assigned whichever runner maximizes absolute quality: deterministic code, classical ML regressions, an LLM, or a human analyst.

Reality check: agents can pay, so why don’t they? (Apify)

If governed agents are finally operating inside institutions, are they autonomously buying data and compute from external vendors over the new wave of machine-payment protocols?

Jan Curn (Apify) brought hard empirical numbers that punctured eighteen months of fintech and crypto hype. Credit cards clearly break on sub-cent machine micro-transactions and chargeback abuse, which inspired more than a dozen competing agentic payment standards—led by Coinbase’s x402 and Stripe’s Machine Payment Protocol (MPP). Yet when Curn audited actual trailing 30-day settlement volume:

• x402 cleared roughly $1 million total across the entire ecosystem.
• MPP cleared roughly $1,500.

Why is autonomous agent spend virtually zero? Four structural forces choke the funnel:

1. Static API keys already work. Agentic payment rails are not competing against Visa checkout forms; they are competing against pre-provisioned SaaS accounts and Authorization: Bearer keys billed monthly to a corporate card whenever tool needs are known ahead of time.
2. Subsidized flat-rate LLM plans Invert build-versus-buy. When a $100/month coding plan delivers $14,000 worth of subsidized frontier tokens, an agent has zero economic incentive to pay $0.05 to an external utility API—it will happily burn $12 of “free” tokens writing a bespoke scraper or parser from scratch.
3. Untrusted service discovery. An autonomous agent cannot tell whether an arbitrary 402 Payment Required endpoint on the open web will return valid data or steal funds, forcing discovery back through curated marketplaces (Apify Store, ChatGPT apps, MCP registries) that already handle billing.
4. Deferred-settlement double-spend. Variable-cost compute jobs cannot safely use naive “authorize up to $X” schemes on-chain because a buyer agent can drain the same wallet across fifty concurrent providers before settlement clears—pushing providers back toward upfront deposits, escrow contracts, or plain old enterprise invoices.

The institutional governance stack compared

Institution	Core operational bottleneck	Architectural mechanism	Where human authority sits
Two Sigma	Shadow IT vs. 2-month manual review queues	Auto-registering frameworks + registry-compiled runtime entitlements + conservative escalation classifiers	Owns registered agent identity & reviews only classifier-flagged ambiguous calls
Coinbase	Static compliance rules vs. daily-changing products	Build-time normative policy (must/may/should) compiled into parallel policy DAGs	Defines normative policy & signs off at mandatory compliance merge nodes
Anthropic FS	Bespoke 1-off bank PoCs & ledger write risk	Unified MCP layer + Git-versioned SME skills + deterministic booking hooks	Business SMEs author skills & approve every proposed ledger amendment
Bridgewater	Stale macro context & 1,000-QPS agent search storms	100M+ doc bitemporal Vault + turbopuffer native cluster filtering (60% → 95% P@10)	Researchers co-investigate in Pocket Analyst, writing verified synthesis back to Vault
Apollo + Rowspace	20 conflicting internal systems per credit metric	Embedded engineers encoding golden-source precedence into continuous reconciliation agents	Traders & PMs own golden-source hierarchy, daily exception QA, and final 10–20% CLO sizing
Coatue	Copy-paste UI silos & unmeasured LLM sprawl	Full API parity + compartmentalized DAG nodes choosing code, ML, LLM, or human per step	Tunes node hyperparameters against north-star metrics & executes judgment-heavy nodes

Midway through 2025, building an AI agent meant assembling a cathedral of custom prompt chains, sub-step planners, AST chunkers, and bespoke tool wrappers. By October 2026 in New York, the engineers maintaining the highest-volume harnesses stood on stage and explained why they spent the summer tearing most of that scaffolding down—and moving what remained into isolated Linux virtual machines fed by structured institutional memory.

The trajectory flywheel and self-deleting guardrail shims (SpaceX)

Jediah Katz (SpaceX Super Intelligence) gave the clearest structural account of why coding harnesses keep eating themselves. Every agent harness splits cleanly into two layers:

1. A foundational runtime layer that owns loop execution, durable state checkpointing, isolated compute, and secret boundaries; and
2. A context orchestration layer that owns model routing, tool definitions, memory compaction, and behavioral guardrails.

In single-session software engineering, the context layer is caught in a relentless data flywheel: clever harness scaffolding makes harder coding tasks tractable today → millions of resulting user trajectories feed frontier post-training tomorrow → the next model generation internalizes those planning and file-navigation behaviors natively → choice of coding harness stops moving benchmark accuracy.

So what still differentiates a serious harness once raw coding IQ moves into the weights? Katz highlighted four durable capabilities:

• Cost and latency efficiency. Even on benchmarks where two harnesses tie on pass rate, their token burn and wall-clock completion time diverge sharply based on compaction discipline and model-mix routing.
• Long-lived coordinator agents for unmastered domains. Single-turn coding chats are saturated, but multi-day knowledge work and software factories are not. Systems like SpaceX’s internal Rockbot shift from chat sessions to persistent coordinator bots that hold high-level strategy, subscribe to unified event streams (collapsing separate Slack, CI, and timer integrations behind a single composable event-subscription primitive to avoid Hyrum’s Law lock-in), and dispatch worker fleets with explicit act-versus-ask boundaries.
• Cloud runtimes with scoped credentials that survive laptop lid-closes without leaking host secrets.
• Self-deleting guardrail shims. Every current model generation still has idiosyncratic sharp edges requiring deterministic guardrails. Instead of letting temporary workarounds ossify into permanent harness bloat, Katz’s team instruments every guardrail shim with an activation counter: a background watcher bot monitors the telemetry dashboard, and the moment a newer model stops triggering a shim, the bot automatically opens a pull request deleting the dead guardrail.

create / execute / delete: why every framework converged on VM sandboxes (Daytona)

If the prompt layer of the harness is shrinking, the execution boundary underneath it has hardened into consensus. Mislav Ivanda (Daytona) surveyed the major open-source and commercial agent frameworks—which still disagree fiercely on graphs, crews, handoffs, and memory abstractions—and pointed out the one architectural commitment every single framework now agrees on: giving the model its own isolated computer via three lifecycle primitives: create, execute, and delete.

Ivanda walked through why the two popular shortcuts from 2024–2025 failed security review in 2026:

• Embedded JS/Wasm isolates (V8, QuickJS) protect glue code, not tools. Running model-generated orchestration code inside an in-process V8 isolate stops buffer overflows, but every registered tool call (send_email, query_prod_db, read_file) exits the isolate and executes on the host process with host privileges. A prompt-injected model still drives privileged host side effects cleanly through the front door.
• Local command allowlists collapse on python -c. Asking developers to click “Allow python” on their laptop feels like human-in-the-loop safety until you remember that a single python -c invocation can open arbitrary raw sockets, exfiltrate ~/.ssh and cloud tokens, or mutate any file the user owns.

Across 2026 releases, frameworks like CrewAI and AG2 stopped treating isolated sandboxes as an optional enterprise add-on and actively deleted their legacy local code executors. Crucially, Ivanda warned teams against the “start local, sandbox later” proof-of-concept trap: models, prompts, and even sandbox vendors remain swappable, but local execution posture locks in system architecture by coupling every tool assumption to the developer’s host filesystem and ambient credentials. Meanwhile, once execution moves into disposable Linux VMs, sandboxes stop being mere 5-second snippet runners and become fleets of thousands of parallel environments powering subagents, dependency-rich browser UI checks, and reinforcement-learning grading loops.

What weights cannot swallow: outer-loop institutional context (Linear, Greptile, GEICO)

If models now navigate syntax trees natively inside disposable VMs, why do unattended coding and modernization agents still ship changes that senior engineers immediately revert?

Tom Moor (Linear) diagnosed the shift in a single sentence: as intelligence that cost $100 three years ago drops to five cents ($0.05) and agents run unattended for hours, the primary failure mode shifts from implementation capacity to product incoherence. Source code tells an agent what the software currently does; it never tells the agent why an appealing UI pattern was rejected last quarter, which enterprise customer requested a workflow three times this month, who owns an unspoken product invariant, or what caused a subtle transaction revert two weeks ago.

When coding moved out of the interactive inner loop into the unattended outer loop, Linear built a four-stage context pipeline so institutional judgment travels with the ticket:

1. Automated intake & enrichment. Linear’s triage agent deduplicates multi-channel bug reports and feature requests, then queries internal MCPs—attaching customer tier, request frequency, Datadog logs, stack traces, live bug reproductions, and links to prior related architectural decisions before any builder agent wakes up.
2. Unattended outer-loop implementation combining that enriched brief with explicit verification skills to auto-fix scoped bugs in cloud sandboxes.
3. Intent-joined PR review. Rather than linting code diffs in a vacuum, Linear’s review agent joins the diff back against the original customer issue, historical reverts, and design-system conventions—catching inverted label hierarchies, rejected UI idioms, and database transaction hazards.
4. Daily post-merge distillation. Every night, a scheduled job mines human review comments and reverts across the day’s merged PRs and distills recurring lessons back into agents.md and living docs—turning one-off human pushback into compounding product memory.

Two other New York talks reinforced the exact same outer-loop lesson at very different scales:

• Living codebase wikis over raw grep (Greptile). Daksh Gupta showed why letting a review agent run dozens of naive grep searches burns massive token budgets while still missing cross-service coupling. Greptile disintermediates raw search by continuously maintaining a living architectural wiki synthesized from code changes, production telemetry, design docs, and tickets—and boots the full application stack inside dependency-rich VM sandboxes so vision/computer-use models catch mobile-viewport regressions (such as a submit button present in the DOM but clipped off-screen) that static diff review never sees.
• Multiplayer war rooms for million-line legacy estates (GEICO). Ford Prior reminded the room of Gene Kim’s distinction between the inner loop (writing a patch) and the outer loop (cross-functional risk, architecture, and business sign-off). Inside GEICO’s multi-million-line legacy claims systems, inner-loop coding is cheap; cross-team alignment is what takes quarters. GEICO built the Fable War Room—a multiplayer workflow governed by structured Markdown FleetKit rules and nine deterministic artifact gates. Research agents map legacy call graphs first, stage architectural plans in a human queue, and generate interactive HTML worksheets and editable diagrams that ask business and claims owners sharp edge-case questions they hadn’t thought to specify, ingesting their edits directly back into context. The payoff was striking on two fronts: single-day, $1,000 war-room sprints shipped concrete legacy wins (including a rentals API extension saving 8,500 calls per month and a salvage plan that caught a live production bug), and benchmarking against recent golden production commits proved that once rich domain context is distilled upfront, smaller Sonnet-class models write legacy claims code just as accurately as heavyweight frontier models at a fraction of the cost.

Layer	Disappearing (2024–2025 pattern)	Enduring (Late-2026 production pattern)	Who demonstrated it
Execution boundary	Host python -c allowlists & in-process V8/Wasm glue isolates	create / execute / delete isolated Linux VMs from day one	Daytona, Warp, Greptile
Inner-loop harness	Hand-rolled AST chunkers, rigid multi-step prompt planners	Minimal tool loops + self-deleting guardrail shims watched by PR bots	SpaceX Super Intelligence
Product coherence	Pasting raw ticket titles into code-only repo agents	Outer-loop triage enrichment + intent-joined review + daily post-merge distillation	Linear
Tribal architecture	Repeated token-heavy grep across repos & Slack	Continuously synthesized living codebase wikis + full-stack visual VM checks	Greptile
Legacy alignment	Months of architecture meetings or blind greenfield rewrites	$1,000 one-day war rooms (FleetKit Markdown gates + interactive HTML SME worksheets)	GEICO

Put twenty parallel agents inside isolated cloud VMs with rich institutional context and two operational questions hit your desk within forty-eight hours: how does a small human team supervise and improve the fleet without drowning, and who is paying the GPU bill when token volume jumps twentyfold?

New York’s fourth through-line connected human fleet management directly to the physics of KV caches, post-trained open weights, and inference serving.

Stepping out of localhost: the Agent Manager and Skill Doctor (Untangle, Turing Post, Warp)

Ryan Carson (Untangle) delivered the week’s sharpest provocation to individual contributors: “If you’re running local dev, you are holding yourself back and you’re holding your team back.” A developer editing files locally on a laptop cannot orchestrate ten concurrent, inter-communicating cloud sessions. At Untangle, engineers operate strictly as agent managers over a structured fleet hierarchy:

• Builder agents implement features inside cloud environments, paired with automated reviewer and child-QA processes capped at three review iterations before escalating to a manager agent, and only then to a human.
• A dedicated merge-manager agent serializes the incoming wave of concurrent pull requests so parallel builders stop colliding on shared sequence numbers like database migrations.
• Mandatory feature-demo videos replace diff skimming. Every builder agent records a screen video demonstrating the feature working and passing automated tests; watching that video on a laptop is the human product owner’s non-negotiable merge gate.

Untangle rebuilt hiring around the exact same premise—eliminating resumes, recruiter screens, and live chat interviews (“you’ll like their hobbies, and it’s all noise”) in favor of unguided 45-minute screen recordings of candidates operating agents to ship a real pull request, stack-ranked by models to select the top five for paid 16-hour recorded trials under NDA before a single phone call takes place. On the candidate side of that same market, Ksenia Se (Turing Post) drew the complementary boundary in Thursday’s opening keynote: use human judgment first to rank companies and speakers whose technical problems genuinely excite you, deploy a personal research agent to scrape career pages, match open roles, and extract published compensation bands—and never let an agent auto-submit your resume, keeping outreach grounded in real human conversation.

Once a team runs dozens of cloud sessions daily, however, a quiet pathology creeps into the harness. As Ben Holmes (Warp) observed: unattended agents are pathologically polite workaholics—when a skill file is stale or a local test port moved, they never complain; they silently burn twenty extra tool calls brute-forcing lsof and curl workarounds, then hand you a clean green PR. Looking only at the merged diff hides the rot.

Warp’s fix treats the harness and its SKILL.md library as a continuously graded system:

1. Centralize the session triad automatically. Running inside shared cloud Docker sandboxes captures (dialogue transcript, tool/MCP calls, final diff) for every run without syncing local ~/.claude folders off laptops.
2. Score traces on PR merge. Merging a pull request triggers Markdown-defined LLM-as-judge rubrics that grade tool-call efficiency, dead-end detours, and test hygiene—turning silent port-probing loops into explicit 55% efficiency scores.
3. Close the loop with Skill Doctor and daily meta-agent PRs. A scheduled meta-agent (plus a single-player Skill Doctor audit skill that crawls local Claude, Codex, and Pi logs) aggregates recurring rubric failures into concrete pull requests updating SKILL.md files and subagent permissions—and tracks rubric score inflections on a team dashboard so engineers can verify that a merged skill edit actually lifted fleet performance from 55% to 85%, rather than trusting opaque overnight “dreaming” memory loops.

Tokenomics 101: why the AI bill is too damn high (Featherless)

All of those outer-loop fleets, parallel policy DAGs, and post-merge trace graders burn tokens continuously. Eugene Cheah (Featherless) opened the conference with a forensic breakdown of why enterprise inference bills spiral—and how basic systems hygiene plus open-weight right-sizing routinely cuts a $150,000/month frontier API bill down to $7,600/month without degrading task pass rates:

1. Split shared API keys immediately. Roughly 30% to 40% of enterprises route multiple departments and workloads through a single shared API key, making unit economics invisible. Issuing distinct keys per department and workflow is the highest-ROI ten-minute change in enterprise AI.
2. Make the cheaper model faster, not just cheaper. Developers will never voluntarily switch from a frontier planner to a smaller worker model just to save the CFO’s budget—unless the smaller model responds at noticeably higher tokens-per-second, aligning developer impatience with company margin.
3. Respect the superlinear speed-cost curve (2× speed ≈ 3× cost). Hardware physics punishes low-batch, high-speed serving: every 2× jump in per-request tokens-per-second costs roughly 3× more GPU capacity. Reserve high-tok/s serving strictly for interactive human-waiting paths; run CI reviews, trace sweeps, and overnight backfills on high-throughput batched tiers, sizing dedicated GPUs to steady base load with cloud burst spillover.
4. Stop thrashing prefix KV caches with 30-model turn routers. Multi-turn agents derive huge cost and latency savings from reusing the prompt prefix KV cache across step 1, step 2, …, step 30 (a dynamic we measured locally in Local KV-Cache Reuse for Agents). Drop in an “intelligent router” that switches across 20 or 30 models on every turn and every model swap invalidates the prefix KV cache, forcing full prompt prefill recomputation and destroying your savings. Route at the task or workload boundary, or restrict intra-loop routing to at most two or three fixed planner/worker roles.
5. Enforce human output ownership against the “slop grenade.” Measuring engineer productivity by token consumption subsidizes unverified AI decks and PRs lobbed over the wall onto teammates. Measure verified business outcomes per dollar, build task-specific eval suites from production traces, and search open-weight models (7B–27B) for the smallest passing model.

Below the API layer: specialists, KV cartridges, BenchPod, and open foundations (Ramp, Runpod, Nous)

A recurring hallway question in New York was whether teams should bother optimizing below the frontier API at all if flagship token prices keep falling every six months. Alexander Shevchenko (Ramp) answered with production numbers from a company where internal LLM usage grew 123% year over year: even as unit token prices fall, aggregate feature spend compounds fast enough that applied research below the API pays back immediately.

Ramp shared four concrete wins from its applied research stack:

• Post-trained 35B specialist beating the frontier (Qwen 3.5 35B). Partnering with Prime Intellect to post-train a 35-billion-parameter Qwen model specifically for financial spreadsheet Q&A inside Ramp Sheets surpassed the frontier generalist baseline on exact-pass accuracy while serving answers at dramatically lower latency, orchestrated underneath a frontier planner.
• Offline KV-cache cartridges (20×–30× context compression). Much of corporate finance context—general ledgers, vendor histories, aging schedules before month-end close—is predictable hours before users query it. Pre-computing compressed offline KV-cache cartridges overnight packed 20× to 30× more working context into memory with far less engineering lift and broader reuse than full weight fine-tuning.
• 10,000-agent open-weight pentesting swarms. Running a massive MapReduce security swarm across Ramp’s codebase using open-weight models up to 5× cheaper than frontier reasoning models uncovered seven high-severity vulnerabilities—including a subtle multi-component email script injection—proving parallel sampling over cheap open weights matches frontier depth on search-heavy security audits.
• Semantic spend attribution & Portal weight transfer. Externally, Ramp classifies enterprise LLM traffic semantically so finance teams map token spend to accounting categories and R&D tax credits; internally, Ramp Labs built Portal to project latent post-training weight updates from an older base model onto newly released checkpoints so specialist training doesn’t reset to zero every quarter.

For teams serving those open models themselves, Chris Vela (Runpod) showed why naive self-hosting stumbles: teams deploy stale vLLM releases instead of nightly builds needed for fresh architectures, over-allocate max context lengths without sizing KV memory for concurrency, and waste 30 minutes per flag tweak waiting on container restarts and weight reloads. Runpod’s BenchPod wraps Dockerized vLLM with live YAML reloads, self-healing multi-node watchdogs, and FastAPI telemetry/benchmark endpoints designed to be called directly by coding agents—uncovering counterintuitive wins such as raising max_num_batched_tokens on concurrent Kimi agentic loads (which leaves time-to-first-token flat while sharply cutting inter-token latency), right-sizing routine traffic onto DeepSeek across 4× H200 GPUs, and scaling past ~50 internal users via disaggregated prefill-decode Ray clusters + LMCache.

Finally, Dillon Rolnick (Nous Research) framed the strategic stakes behind owning that open stack. As practitioners swap freely among frontier APIs, per-token prices fall against stubbornly flat GPU/power costs, squeezing raw inference margins. How do frontier labs recoup trillion-dollar infrastructure bets? Not by selling cheap commodity tokens forever—by moving up the stack into sticky enterprise agent workflows (where switching costs create margin) and partnering in lucrative verticals like financial services to absorb domain workflows before competing directly. Because model moats increasingly come from post-training data rather than secret architectures—and coding agents can clone closed UI features over a weekend—Rolnick argued enterprises should treat their agent harness as an owned operating system built on open, model-swappable foundations like Hermes Agent rather than renting the control plane from a single model landlord.

The 2026 enterprise tokenomics playbook

Optimization lever	Common anti-pattern	Production fix	Reported impact
Attribution	30–40% of orgs share one company-wide API key	Per-department & per-workflow keys + semantic R&D spend taxonomy	Immediate visibility into runaway loops (Featherless, Ramp)
Throughput vs. speed	Paying interactive high-tok/s tiers on background CI & batch sweeps	Interactive fast tier for humans; batched tier for async (2× speed ≈ 3× cost)	~3× GPU savings on non-interactive workloads (Featherless)
Model routing	Turn-by-turn routing across 20–30 models	Task-level routing or 2-model planner/worker split to preserve prefix KV caches	Eliminates repeated multi-turn prefill burn (Featherless)
Context prefill	Stuffing raw ledgers/docs into every live prompt	Overnight offline KV-cache cartridges + LMCache disaggregated prefill	20×–30× context compression ahead of month-end (Ramp, Runpod)
Right-sizing & swarms	Calling flagship reasoning models on every subtask	Eval-gated 7B–35B open specialists + parallel 5×-cheaper open-weight swarms	$150K/mo → $7.6K/mo client bill (Featherless); 7 high-sev bugs found (Ramp)
Live serving tuning	Default vLLM flags + 30-min container cold restarts	Agent-driven BenchPod live FastAPI flag sweeps (max_num_batched_tokens)	Lower inter-token latency on 4× H200 pods (Runpod)

Every good conference has two clocks running at once: the urgent engineering work paying the bills over the next twelve months, and a quieter signal from two years out reminding everyone not to over-index on today’s interface primitives.

For ninety percent of AI Engineer New York, “agent” meant a discrete, turn-based loop reading text, calling JSON/bash tools inside a Linux VM, and writing code or structured memos back out. Then two keynotes—from Pim de Witte (General Intuition) and Cristóbal Valenzuela (Runway)—stepped outside the text window altogether and showed scaling laws crossing into continuous space, time, and interactive visual simulation.

Acting while observing: 1.5 trillion monthly action labels (General Intuition)

Autoregressive language models take turns: read a prompt, think for three seconds, emit a tool call, wait for stdout. Physical and real-time virtual environments refuse to pause while the model thinks. If a quadcopter encounters a crosswind mid-inference, turn-based planning crashes into the wall.

Pim de Witte showed how General Intuition trains continuous pixels-in / actions-out foundation models that observe incoming frames simultaneously while emitting high-frequency control inputs—built on top of Medal, the PC gameplay recorder scaling from one billion to three billion uploaded videos per year and capturing more than 1.5 trillion explicit controller and multi-key action inputs every month (matching the entire robotics field’s historical teleoperation volume every twelve hours).

Three technical insights stood out:

1. Game controllers are the universal robotic abstraction layer. Over 1,000 low-degree-of-freedom commercial robots already expose standard gamepad/joystick velocity and pose interfaces above their low-level motor torque controllers—meaning policies trained on high-APM gameplay trajectories transfer directly to physical drone micro-adjustment and robotic teleoperation with far denser simultaneous action supervision than slow human lab demos.
2. Explicit action labels beat video-only latent inference. Pure video-prediction or JEPA-style latent-action approaches struggle when tiny, visually imperceptible control inputs (like a subtle aircraft rudder tap) cause large trajectory shifts seconds later; pairing raw pixel streams with exact timestamped controller state learns causal control directly.
3. Two-tier production brain. These spatiotemporal models do not replace frontier LLMs—they sit underneath an orchestrator like Gemini or Claude as the spinal cord and cerebellum, translating high-level semantic plans into millisecond-level closed-loop execution.

Engine-free interfaces and visual scratchpads for coding agents (Runway)

Cristóbal Valenzuela (Runway) approached the same convergence from the opposite direction: scaling real-time, action-conditioned next-frame video prediction until the video generator becomes a general-purpose world simulator.

Predicting the next visual frame across massive video corpora forces a model to internalize geometry, gravity, object permanence, lighting, and occlusion implicitly in its weights—with zero Unity, Unreal, Blender, or hand-coded physics meshes underneath. Condition that next-frame predictor on live user clicks, camera drags, or arbitrary counterfactual interventions mid-rollout, and two unexpected software primitives emerge:

• Continuum interactive portals. Runway demonstrated treating a real-time world model as an operating-system rendering layer (Continuum) that turns structured web documents into persistent, live interactive video surfaces whose scenes respond continuously to clicks, queries, and window resizing without a pre-built 3D asset pipeline.
• Visual imagination for coding agents. Ask today’s coding agent to design a 3D mechanical assembly or complex spatial UI and it has to reason blind through coordinates in text. Connecting a coding agent directly to Continuum gives the agent a live, steerable visual scratchpad—simulating spatial behavior alongside code so human and agent inspect and adjust the physical design together in real time.

Two clocks, one discipline

Step back across all thirty-one talks in Midtown Manhattan and the two clocks reinforce the same conclusion. Inside enterprises today, raw text intelligence stalls at the 95% diffusion wall until engineers surround it with answer-first domain evals, behavior specs, parallel policy DAGs, bitemporal golden-source ontologies, isolated Linux VM sandboxes, outer-loop institutional context, and prefix-aware open-weight economics. And as models cross from discrete text turns into continuous spatiotemporal control and real-time visual simulation, that surrounding discipline—knowing what counts as ground truth, bounding what actions are permitted, and pricing every millisecond of compute—only becomes more decisive.

— Will Schenk · TheFocus.AI