Jev Gate

A self-correcting agent loop that knows why it failed before it tries again.

Jev Gate is a critique-refine agent loop that fixes real bugs in Elixir code — chosen over a generic Python or LeetCode demo, since Elixir is rare enough in most models' training data that the loop has genuine work to do.

The loop

A cheap draft model attempts a fix. We actually execute it against real tests — pass/fail is execution-graded, never self-reported. On failure, TypeSafe's Jev model (System One's Choice primitive) diagnoses the specific failure class — logic_error, off_by_one, syntax_or_runtime_error, and so on — before a stronger model refines the fix using that diagnosis. Each retry is informed by a calibrated classification of what went wrong, not a blind re-ask.

Real results, not simulated — 3 repeated runs, not one

9 hand-authored Elixir tasks, each independently verified — buggy version fails, a correct fix passes — via real elixir execution before any LLM spend. Run 3 times: the cheap 2.6B draft model's first-try success genuinely varies (3/9, 1/9, 1/9) — don't trust a single number. What's stable across all 3 runs: diagnosis-gated escalation closed the gap to 9 / 9 · 100% every time, verified via the W&B API, not the console log.

fix_is_prime never passed on the first try in any of the 3 runs — one example: the draft model wrote math:sqrt(n), valid Erlang, invalid Elixir. TypeSafe correctly diagnosed it as a syntax_or_runtime_error each time; refine fixed it.

Tracks — why we entered these

• Best Loop Design — the loop is a deliberate escalation gated by a real diagnosis, not blind retry.
• Most Production-Ready — execution-graded pass/fail, full observability, honest documented limits.
• Best Use of Weave — explicit tracing on every call, plus auto-instrumented provider spans nested for free.
• Best Use of TypeSafe AI — Jev's diagnosis is the one decision the entire loop design pivots on.
• Best Use of marimo — a working notebook pulling live results, not a placeholder.

Sponsor tools — what we actually built, not claimed

Tool	Used for
TypeSafe AI	The escalation decision itself — classify_failure() calls Jev's Choice primitive on every test failure.
W&B Weave	Explicit @weave.op() on every LLM/TypeSafe call, so the trace stays complete regardless of provider — Weave's own auto-instrumented OpenAI spans nest inside for free.
W&B (classic)	A logged Run with per-task, per-iteration metrics and model config.
marimo	A notebook pulling this data live via wandb.Api() for the two plots above.
OpenRouter	Provider-agnostic LLM layer — draft and refine models are a config swap, not a code change.

Honesty

Critique-refine is Self-Refine/Reflexion lineage, not a novel loop shape — we're not claiming otherwise. Not tested against a public benchmark. W&B Serverless Sandboxes was investigated for real (working SDK, working auth) but blocked on org entitlement, so execution runs in a local subprocess with a 10-second timeout today, not isolated cloud compute.