# Rubric Labs
> Rubric Labs is an AI engineering agency. We build AI agents, products, and models for companies, and ship them to production with your team.
# Rubric Labs brand system
The Rubric mark uses a 3×3 grid: 101/110/100 (rows top to bottom, 1 = filled).
## Principles
- Start with Ink (#000000) and Paper (#FFFFFF). Reserve Live (#10B981) for open slots, booking, and open roles. Use Sky (#38BDF8) for the Lab, data, and accents.
- Reuse the square cell in labels, status, and artwork. Keep its edges square, even inside rounded cards.
- One idea per section: a label, a short heading, and evidence. Use space to separate sections.
- Write specific claims. Name what was built or measured.
## Downloads
[All assets, ZIP](https://rubriclabs.com/brand/rubric-assets.zip) (~683 KB). Sizes are approximate, before HTTP compression.
| Asset | Mark colour | Files |
| --- | --- | --- |
| Logo | black | [SVG](https://rubriclabs.com/brand/logo-black.svg) (~0.2 KB) · [PNG](https://rubriclabs.com/brand/logo-black.png) (~0.7 KB) |
| Logo | white | [SVG](https://rubriclabs.com/brand/logo-white.svg) (~0.2 KB) · [PNG](https://rubriclabs.com/brand/logo-white.png) (~5.2 KB) |
| Wordmark | black | [SVG](https://rubriclabs.com/brand/wordmark-black.svg) (~6.7 KB) · [PNG](https://rubriclabs.com/brand/wordmark-black.png) (~234 KB) |
| Wordmark | white | [SVG](https://rubriclabs.com/brand/wordmark-white.svg) (~6.7 KB) · [PNG](https://rubriclabs.com/brand/wordmark-white.png) (~234 KB) |
PNG sizes: 1000 × 1000 logo; 6240 × 1800 wordmark. Prefer SVG. These downloads include padding and a background: white for black marks, black for white marks. Use the templates below for transparent, tightly cropped versions.
Brand assets are free to use and adapt. Matter requires a separate licence. [Figma source](https://www.figma.com/file/tRoDcoEfD0hSfhvKs4lzZR/Rubric-Branding-(TEMP-Copy)?type=design&node-id=46%3A79&mode=design&t=fdSx4pbKSYB66sgb-1).
## Logo
Keep the proportions, orientation, and square corners. Leave one cell of clear space around the ink: a 48px mark needs 16px on each side. The downloads include that padding; this SVG does not.
```svg
```
## Theme
Current website tokens:
| CSS variable | Light | Dark |
| --- | --- | --- |
| --background | #f7f7f8 | black |
| --surface | #ffffff | #111113 |
| --primary | black | white |
| --secondary | #666 | #999 |
| --rule | #0000002e | #ffffff33 |
| --live | #047857 | #10b981 |
| --sky | #0284c7 | #38bdf8 |
Use primary/secondary for text, background for the page, surface for cards, and rule for 1px separators. Live and Sky text use darker shades on light surfaces; the live status square stays #10B981. Do not fade body text with opacity.
## Typography
Matter Regular for headings and prose. Licence it from Displaay or use system-ui, sans-serif. It is not in the asset bundle.
[Departure Mono, OTF](https://rubriclabs.com/brand/fonts/DepartureMono-Regular.otf) (~84 KB) is free. Use it for labels and metadata, not paragraphs. It has an 11px pixel grid; use multiples of 11px for larger display type.
| Role | Size | Line-height | Tracking |
| --- | --- | --- | --- |
| Brand-page hero | clamp(3rem, 7vw, 6rem) | 0.95 | -0.05em |
| Section heading | 2.25rem; 3.5rem from 640px | 1.02 | -0.035em |
| Intro | 1.125rem; 1.25rem from 640px | 1.625 | Normal |
| Body | 1rem | 1.5–1.625 | Normal |
| Pixel label | 11px | 1 | 0.04em |
Section labels start with a 5px square and an 8px gap. Inline metadata omits the square. Live status uses a 7px green square with sentence-case Matter text. For changing numbers in Matter, apply tabular figures only to digits; its tabular punctuation and spaces are wider too.
## Spacing
Website defaults, not fixed dimensions for every medium:
| Element | Default |
| --- | --- |
| Container | Centred, max-width 72rem including gutters |
| Horizontal gutters | 20px; 32px from 640px |
| Section padding, top/bottom | 80px; 128px from 640px |
| Label → heading → intro | 20px between each |
| Heading group → content | 48px |
| Card padding | 24px; 32px on larger screens |
| Card gaps | 24px or 32px |
| Prose width | 32–36rem |
Align labels, headings, and content to the same container. Stack columns on phones. The cell defines artwork geometry; page spacing need not follow a three-column grid.
## Wordmark
Use the supplied paths for the symbol and “Rubric Labs”, including the square dot on the i. Do not typeset a replacement.
[Full wordmark SVG](https://rubriclabs.com/brand/wordmark-black.svg) (~6.7 KB). For this transparent React template, copy only its path's d attribute, not the background rect:
```jsx
function RubricWordmark(props) {
return (
)
}
```
The cropped viewBox is 2518.54 × 307.6 units. The symbol is 300 units wide; the R is 298 units tall. Leave one symbol's width around a standalone wordmark.
| To match | Set the full SVG's size to |
| --- | --- |
| Cap height H | height = H × 307.6 / 298 |
| Symbol width W | width = W × 2518.54 / 300 |
## Footer credit
For “Made by”, match the R to the text's cap height and align baselines. Use a 0.25em internal gap; inset the whole credit from the footer edges by one symbol's width.
```jsx
// Uses RubricWordmark from the template above.
export function Footer() {
const symbol = 'calc(1cap * 300 / 298)'
return (
)
}
```
## Partner lockups
Match a partner wordmark's first capital to Rubric's R. For a pictorial logo, match its width to the Rubric symbol, not the whole wordmark. Trim SVG padding first, then adjust by eye.
Centre × between equal-width columns with 8px gaps and a 32px sign column. Scale the whole group to fit narrow screens. Match letters rather than decorative features such as an arch above the lettering.
## Art and motion
Client artwork uses square cells in the client's colours. Keep those colours local to the art.
The Lab uses Sky (#38BDF8) cells on #06070A. Its Conway's Game of Life field is seeded with the mark; the brand-page example uses 8px cells and a 220ms tick.
Animate cells assembling or changing state. Under prefers-reduced-motion, show the resting mark, a still field, and a non-pulsing status square.
## Share images
- 1200 × 630, Ink (#000000) background. Margins: 56px top/bottom, 96px sides, allowing for Slack and X crops.
- White wordmark and pixel-mono label on the top row. Matter title and one sentence below.
- Page-specific art in a rounded window on the right.
- Live (#10B981) only for open status.
## About
Rubric Labs is a small team of founder-level engineers with its own research lab. We join client teams and build through production. We publish research on the blog and in The Grid.
- Name: Rubric Labs
- Website: https://rubriclabs.com
- Email: hello@rubriclabs.com
## New additions
For each new component, document its purpose, dimensions, colour role, and one example. Add responsive, theme, and reduced-motion behaviour where relevant. Keep shared values in the brand source modules; add new topics as sections.
[Source](https://rubriclabs.com/brand.md)
# What can you build with Jev?
Practical use cases for fast, cheap decision models
By [Sarim Malik](https://rubriclabs.com/team/sarim), 2026-10-07
Jev is a new model built to **evaluate questions with predefined possible answers**. It returns answers in a few hundred milliseconds, in a structured format, with a probability for each possible answer.
Practically, Jev helps you make a decision. You provide it context, questions, and possible answers, and it answers each question using probabilities, which you can then use to make a decision.
We spent the past two weeks building with Jev’s API. In this post, we’ll show practical examples of what you can build with it.
## What can you ask Jev?
You can ask Jev three kinds of questions. Every figure in this post **calls the Jev API live**, and the time and cost in each corner come from your request.
**Choice** picks the best option from a list you provide, and returns a probability for every option.
*Using Choice to find the emoji that best represents the title*
**Score** rates something on a scale you define, like 1 to 5. The score can land between levels, like 4.7.
*Using Score to rate how well an emoji represents the title*
**Bool** (Jev calls it [Noul](https://docs.typesafe.ai/primitives/noul), and OpenAI calls it [Predicate](https://developers.openai.com/api/docs/guides/decisions)) returns the probability that a statement is true. Your code decides how sure is sure enough.
*Using Bool to test whether an emoji represents the title*
## Examples
### Prioritizing notifications
We all get tons of notifications every day from different products, whether natively on your device or via email.
Knowing which ones need your attention now and which can wait is really critical.
So instead of doing it manually yourself or passing this on to a reasoning model, which can take many seconds and has real cost per notification, we asked Jev **two questions**: Does this notification need the recipient to act? And is it time-sensitive?
*Prioritizing incoming notifications with Jev*
A notification is marked priority when both answers come back at **80% or higher**. That threshold is our choice. Change it in the figure’s bottom corner, then replay the inbox to sort it against the new cutoff.
Click the priority pill on any notification to see Jev's answers.
This example isn't perfect: we're not passing any context about the user.
Take the first notification, for example, Maya's message in #team about lunch at noon. Our two questions are written for work, so they treat lunch plans as something that can wait. So it's really important how you specify the criteria to the model, because your **criteria determine how the model answers** your questions.
Secondly, the model is only going to make a decision as good as the **state or context you provide**, which is true for any language model. So in another situation, if we provided more context about the user, perhaps its decisions would have been different.
### Searching by meaning
When you search a web page in your browser, it runs a **literal search**: it only finds the exact characters you typed.
**Semantic search** matches what you mean instead, so it can find an answer that shares none of your words.
Jev can do semantic search. In the figure below, we ingested every sentence from [Wikipedia’s article about platypuses](https://en.wikipedia.org/w/index.php?title=Platypus\&oldid=1376973599) and made it searchable.
Ask anything about the platypus, or click refresh for another question.
*Searching an article on the platypus with Jev*
In this demo, we first split Wikipedia’s article on the platypus into 211 sentences. During each search request, we ask two questions:
- A **Choice** question ranks all 211 sentences, using their IDs as the options. Choice accepts up to 255 options, so the whole article fits in one request.
- A **Bool (Noul)** question asks whether any sentence answers the question, where true means at least one sentence states or directly implies the answer.
We need both because Choice always ranks some sentence first, even when none answers. Ask "How fast can a platypus swim?" and Choice picks a swimming sentence at 85%, but the article never gives a speed. The Bool returns 5%, below our **35%** cutoff, so the figure dims the matches.
One question rarely covers everything you need to know, so it often pays to **ask several in one request**: here, Choice finds where the answer is, and Bool tells you whether there is one.
Each search takes about 250 ms and costs 0.05¢. If you want to dig further into this use case, check out [this guide](https://docs.typesafe.ai/cookbooks/semantic_find).
### Checking spelling
Think of a Grammarly-style editor that checks your spelling as you type. Each time you finish a word and press space, we send Jev that word and the text around it. Once you type the next word, we check it again, since some mistakes only show up in what comes after.
The request sends two **Bool** questions in one call, the same pattern as the semantic search example above:
- **Non-word:** "Is this word, as written, a misspelled non-word?" This catches *recieved* and *publsh*. Names, jargon, and regional spellings count as correct.
- **Wrong word:** "Is this word spelled correctly but wrong for this sentence?" This catches *form* for *from* and *loose* for *lose*.
A typo gets a red squiggly line, and a wrong word gets a blue one.
*Checking spelling as you type*
We underline a typo when Jev is at least **80%** sure, and a wrong word at **50%**. The wrong-word bar is lower on purpose: in our tests, correct words never scored above 37%, so 50% still leaves a safe margin. Lowering it from 80% to 50% caught 8 of 12 wrong words instead of 5.
It still misses some easily confused pairs, like *then* and *than* or *your* and *you’re*. Those scored between 20% and 46%, below our bar.
If you fix a word yourself, its underline disappears, and Jev checks it again once you pause or press space.
Because the rules are written in plain language, you could also make the spellcheck **programmable**: tell it what to ignore, like startup names, acronyms, or slang, and it skips those words.
### Finding settings
So far, each example asks Jev one round of questions. Here, Jev walks down the menus of iPhone Settings one level at a time, and each answer decides which question we ask next. Sorting something into a tree like this is called **hierarchical classification**.
Search the phone for a setting you want to change, or click refresh for another one.
*Finding a setting in iPhone Settings*
The walk stops at a setting, or earlier if Jev is less than **60% sure of any option**, leaving you on that menu page instead of guessing or going down the wrong path.
### Rewriting emails
Jev also works well next to a larger model. Here, each email comes with a few things its writer cares about. Jev scores the draft on each one, and a language model, in this case gpt-6-luna, rewrites it until every **score reaches 4 out of 5**, or until it has **tried three times**.
Click play to improve the draft, or refresh for another email and its criteria.
*Rewriting an email until it meets the writer's criteria*
## How does Jev compare?
While we were writing this post, new decision models kept coming out, from OpenAI's [Decisions API](https://developers.openai.com/api/docs/guides/decisions) (GPT-6 Luna Decisions) to Cloudflare's [Clef 27B](https://www.cloudflare.com/resource/clef-rl-interest), an open-weight model.
We were curious how they compare, so we ran a quick vibe-comparison by creating **101 hand-labeled scenarios** based on the examples discussed in this post and running them against each decision model three times.
Overall, we found that **Jev was the most accurate, fastest, and cheapest**, but slightly less consistent. The gaps between Jev and GPT-6 Luna Decisions were small, except on price, where GPT-6 Luna Decisions was nearly twice as expensive. Clef was noticeably slower and about six times as expensive as Jev. See the full results below.
*Three decision models on 101 hand-labeled examples*
It's important to note that we wrote these prompts for Jev and revised a few labels after our first run, so read this as a rough comparison rather than a benchmark.
## What’s next
To recap, all of these use cases could have been done with **structured outputs** from previous LLMs. None of this is fundamentally new.
What's different about Jev is that **it's incredibly fast, cheap, and consistent**. TypeSafe [measures it](https://typesafe.ai/blog/introducing-system-one-models-and-jev) at 40–200x faster than frontier models, and 445x cheaper. You've probably noticed this yourself using the figures in this post.
It's faster because it doesn't write. An LLM produces its answer one token at a time, and your code then parses it. Jev reads your question once and puts a probability on every possible answer at the same time, so it can't return an answer you didn't allow.
But overall, decision models like Jev open up use cases that used to be too slow, too expensive, or too hard to personalize and fine-tune. And since Jev-like models can run on smaller weights, many of them could even be **done on device**.
The field of decision models is moving fast. Try OpenAI’s [Decisions API](https://developers.openai.com/api/docs/guides/decisions), [Together’s model](https://www.together.ai/blog/how-to-train-your-own-jev), [Kev](https://openrouter.ai/jaredpalmer/kev-4b), [d1](https://www.liquid.ai/blog/d1-decision-model), [pplx-decider](https://huggingface.co/perplexity-ai/pplx-decider-v1.1-27b), or whatever comes next. We'll dig into how decision models work in a future post.
## Related posts
- [Challenging the Chatbot](https://rubriclabs.com/blog/challenging-the-chatbot)
[Source](https://rubriclabs.com/blog/what-can-you-build-with-jev.md)
# Static Analysis for the Agentic Era
Evaluating how repository maps affect coding agent performance
By [Michael Zoubkoff](https://x.com/wenkafka), 2026-09-22
This is a guest post written by Michael Zoubkoff, an independent researcher supported by a Rubric research grant. To run experiments with us, [reach out](https://rubriclabs.com/contact).
We tested three specialized code-mapping methods with a coding agent. Performance improved but at the cost of more tokens and time. Specialized agent harnesses can help, though not for free.
## Why this? Why now?
LLMs have excelled at software engineering tasks.
On [DeepSWE](https://deepswe.datacurve.ai), a representational benchmark of software tasks, overall completion rates are trending toward 80%. With that said, the non-deterministic nature of LLMs means success rates can vary wildly across runs.
Modern coding agents typically manage codebase context with a combination of heuristics and general-purpose shell commands like grep and find. To this end, we investigate whether specialized artifacts affect agents' abilities to navigate unfamiliar codebases, make their first edit and, if so, **is the benefit enough to offset the added context overhead?**
Existing work includes [RepoBench](https://arxiv.org/abs/2306.03091), which evaluates repository-level retrieval and code completion using cross-file context. We test something different:
Do deterministic repository maps improve coding agent task performance?
Which representation, if any, is most useful? To this end, we measure
- task success,
- token usage and cost,
- runtime,
- time to first edit,
- repository-navigation behavior,
- artifact usage, and
- context-management overhead.
## Experimental Setup
Using [Tree-sitter](https://tree-sitter.github.io/tree-sitter/), we built **Cartograph**, a tool to parse code in a variety of programming languages into a unified, canonical structure covering symbols, modules, and relationships such as
- calls,
- imports,
- instantiation,
- inheritance,
- implementation, and
- references.
Currently, it supports ingesting TypeScript, JavaScript, Python, Go, and Rust.
Cartograph represents codebases via one of three methods we've named Architecture, Skeleton, or Callgraph.
The tool stores these representations as Markdown files. We include ambiguous relationships and parsing failures as additional information for the agent.
Below you'll see how each output might appear in practice:
*Cartograph's three methods of mapping a codebase: Architecture, Skeleton, or Callgraph. Open the generated [architecture.md](https://github.com/michaelzoub/MapBench/blob/main/examples/architecture.md), [skeleton.md](https://github.com/michaelzoub/MapBench/blob/main/examples/skeleton.md), and [callgraph.md](https://github.com/michaelzoub/MapBench/blob/main/examples/callgraph.md) examples.*
Actual artifacts differ in the information they surface and how that information is exposed.
Architecture and Skeleton provide more human-centric representations of the codebase, focusing on where things are and what exists. Callgraph focuses on how symbols interact, such as which functions call which and the execution path between them.
*Information surfaced by each artifact type.*
We intentionally minimize overlap between these artifact types, which makes differences in results easier to attribute to the information each representation provides.
## Experiment
To compare how agents complete real, long-horizon software engineering work, we ran a subset of DeepSWE v1.1 tasks three times each. We tracked task correctness alongside tokens spent, runtime, source access, and navigation behavior (ie. which files the agent accessed).
We fixed the model (GPT-5.6-Luna), provider (OpenAI), harness (Pi), thinking level (Max), repository version, and task prompt across conditions. Every condition also retained access to the complete repository source; the only experimental variable was which structural artifact, if any, the agent could access.
We chose 5.6 Luna because it sits close to the efficiency frontier on [DeepSWE tasks](https://deepswe.datacurve.ai/). Pi is deliberately a simple coding harness: it gives the model the filesystem and editing capabilities needed to complete the task with minimal overhead. This combo of model and harness is also close to coding agents used in practice.
Agents worked inside disposable sandboxes, with held-out verification performed separately after each run.
Artifacts were exposed as ordinary files, with a pointer added to the system prompt:
```md
The following generated repository-structure aids are available:
- Architecture: `.cartograph/architecture.md` is available.
Use the available repository information as you judge useful for completing the task.
```
Agents could inspect the artifact using the same filesystem and CLI tools available for normal source-code navigation, making artifact discovery and usage part of the experiment itself.
Before the full evaluation, we ran a small set of canary tests using a cheaper model to validate the setup and to estimate cost.
Each run had a 90-minute timeout. When the timeout was reached, execution stopped but the verifier still graded the workspace in its current state, meaning a timed-out trial could pass if the agent had already produced a correct solution.
The results should be interpreted as specific to our model-harness pair. Repeating the experiment with other models and harnesses would be needed to establish generality.
## Results
We summarize the main results across 359 trials below.
*Experiment pass rate by condition.*
Across the first 30 tasks, the baseline is a mean pass rate of 37.8%, compared with 36.7% for Architecture, 40.4% for Skeleton, and 37.8% for Callgraph. Skeleton is nominally the best condition.
Of the 359 graded trials, 216 completed, 134 hit the 90-minute cap, and 9 failed. We exclude runs that failed due to infrastructure issues.
The clearest positive result is **time-to-first-read** of a file touched by the reference patch, which fell from a median of 10 seconds in the baseline to 5 seconds with Architecture, 5 seconds with Skeleton, and 6 seconds with Callgraph. The difference is statistically significant for all three artifact conditions.
Beyond final task success, we looked at whether agents actually used the artifacts and how their navigation changed. Discovery was not a problem: Architecture and Skeleton were opened in 100% of trials, and Callgraph in 98.9%, usually within 4-5 seconds. However, **agents generally read the artifact once and never returned to it**.
Median within-pair runtime remained effectively 1.0× the baseline across all three conditions.
*Mean runtime in minutes against pass rate.*
The artifact conditions also used **more tokens without meaningfully changing runtime**.
*Mean total tokens against pass rate.*
Finally, we measured the overhead introduced by Cartograph itself, including artifact generation time and size, finding it impacted total runtime only negligibly.
A summary of these results can be seen below.
*Mean metrics across conditions.*
A detailed breakdown of per-task results can be found below.
*Per-trial results by task and condition.*
## Discussion
Interestingly, the pass rate ended up being fairly low across every condition. Our experiment differs from the original DeepSWE setup in harness, reasoning effort, timeout policy, and task subset, so it is hard to pinpoint exactly why. GPT-5.6 Luna on Max + Pi also used substantially more tokens than our small-model canary runs, despite a 96% cache hit rate for the experiment.
The more interesting result is that **the artifacts seem to do their job: they make initial navigation significantly faster**. The problem is that localization (initial navigation) only represents around 0.2% of a full trial. Cutting that time in half is useful, but it has little effect on an hour-long task dominated by understanding the specification, implementing the change, and verifying the work.
The way agents use the artifacts might also explain why the benefit does not carry further. The agents almost always fell back to navigation heuristics after an initial skim of the map. The maps thus seem useful as an initial orientation aid but may not fit with the model's trained instincts.
These results should be interpreted narrowly. So far, only 30 of 112 tasks have been tested, and only one model-harness combination has been run. The experiment is therefore better viewed as a comparison between conditions under one fixed setup, rather than a general claim about structural artifacts across coding agents.
However, the results have actually increased my confidence that **opinionated harnesses can help agents**. They clearly improve localization, and the Skeleton artifact improves upon the baseline by 2.7 percentage points.
## Future
For future work, benchmarking SOTA models would show whether stronger models benefit more from having a deterministically generated artifact they can use as an anchor. Testing different harnesses would also reveal how much artifact usage depends on the agent harness.
Additionally, since agents rely heavily on heuristics when exploring codebases, testing the maps over much longer runtimes, potentially without a time limit, could reveal whether their value becomes more significant on longer-horizon tasks.
## Related posts
- [Autonomous Iteration](https://rubriclabs.com/blog/autonomous-iteration)
[Source](https://rubriclabs.com/blog/static-analysis-for-the-agentic-era.md)
# Everything we know about good agent design
Patterns for building agents that work in production
By [Sarim Malik](https://rubriclabs.com/team/sarim), 2026-09-16
Recently, I read [Everything I know about good API design](https://www.seangoedecke.com/good-api-design/). I loved how it was packed with so much value all in just a numbered list.
In this post, I decided to interview the [team](https://rubriclabs.com/team) at Rubric and make a similar list about **good agent design**, discussing the insights we've picked up building agents internally and for companies like Graphite, Albertsons and [others](https://rubriclabs.com/work).
This post assumes a basic understanding of how agents work. If you're new to agents, I'd recommend starting with an introduction to [agent harnesses](https://rubriclabs.com/blog/what-is-an-agent-harness) and [coding agents](https://rubriclabs.com/blog/how-does-claude-code-actually-work).
## Prompts
### 1. Read your system prompts
Ask yourself: if I were the agent, could I do the task from this prompt?
System prompts often include **static and dynamic content**. Dynamic content is information added to the prompt at runtime, such as the current user's details, available tools, or the agent's environment and constraints, etc.
Engineers rarely read system prompts in their entirety, even though dynamically built prompts often accumulate **redundancy, contradictions and poor formatting**.
For really large prompts, we tend to **visualize the prompt** by assembling both the static and dynamic pieces and read through the overall prompt. This helps us understand the overall context, identify any redundancies or contradictions, and also see the formatting of the actual runtime prompt.
Ultimately, reading it cold like a college student with no other context allows us to empathize with the agent and understand its behavior.
### 2. Keep your system prompts short
Your system prompts should be short enough that your engineers can keep the whole thing in mind. On the higher end, they should be **less than 5,000 tokens** (6–8 pages of text).
For example, recently Asana announced that they **cleared 5 years of engineering work in 2 weeks** with Codex **using a [five-sentence prompt](https://openai.com/index/asana/#:~:text=a%20five%2Dsentence%20prompt)**.
Obviously, there are other factors involved and those 5 sentences don't cover the entirety of Codex's underlying system prompt. But generally, models today don't require the long, prescriptive system prompts they did in previous generations, as they have become increasingly capable, execute for longer and stay on track.
## Models
### 3. Aggressively deprecate support for old models
Agents are best co-designed with a generation of models. Supporting old models has a lowest-common-denominator effect: prompts have to be dumbed down and scoped to the level of the weakest model, which wastes the potential of the newest model.
### 4. Use a model selector only if you absolutely need to
Most agents these days expose a model selector to the end user. This is a dated pattern. We are at a point where products like [OpenRouter](https://openrouter.ai/), or model routers generally, make it very easy for you to identify the best model for a specific task.
If your incentives are aligned with the user, your goal is to deliver the **most value while keeping token usage and cost low**. So automatically route requests to the best possible model instead of having the user pick a specific model. This can add latency, though it can be unnoticeable for most use cases.
## Tools
### 5. Design tools around use cases
Identify **flagship use cases** that you want your agent to accomplish and design tools that allow your agent to fulfill those use cases.
For example, if you're building an agent that helps you book flights, ask yourself: what do you want your agent to accomplish? Roughly, this might include:
- Find available flights
- Compare fares
- Track price changes
- Hold a fare
- Make a booking
- Modify an existing booking
- Cancel and request refunds
You might be tempted to fulfill each use case with a specialized tool, whereas the better strategy is to **design a unique set of [primitives](https://rubriclabs.com/blog/primitives-over-pipelines)** that your agent can assemble dynamically to solve each use case.
Ultimately, the success criterion for good tool design is whether your flagship use cases are supported (cheaply and quickly, of course) and there's a path to support future use cases.
### 6. Pick the smallest subset of tools
After you understand your use cases, your goal is to design a **small set of really powerful tools** that allow you to fulfill all the use cases.
For example, **cost sensitivity** is a major driver for booking flights. One agent designer might decide to create a tool called `find_cheapest_flight`, which takes a date range and recommends the cheapest itinerary in that interval.
Another agent designer might give autonomy to the agent and expose a single tool called `search_flights`, allowing the agent to call this tool **in parallel** multiple times and compare the prices on its own to find the cheapest fare.
The advantage of the latter is that it can help you find the cheapest flight, but it can also provide a primitive that can be used to fulfill other use cases such as searching for flights in general or to look up a specific flight.
So while both approaches work, the latter gives you a larger surface area while meeting the specific use case of being able to find the cheapest flight and is thus the better approach in our experience.
### 7. Avoid overlapping capabilities across tools
If you give the agent a GitHub push tool and a shell with GitHub already signed in, it has two ways to push the same change. Avoid overlapping tools so the agent has a **clear path** to execute a capability.
### 8. Don't leave agents with missing capabilities
As you design your tools, if you feel there is a gap or **missing capability** that prevents the agent from fulfilling a use case, don't expect the agent to magically solve it.
You need to [unblock your agents](https://rubriclabs.com/blog/unblocking-agents) by putting yourself in the shoes of your agent and ask yourself, if I had access to these tools, would I be able to solve this problem?
If the answer is no, you need to **rethink the tools** that you have selected.
### 9. Name tools and parameters literally
Models use names and descriptions to decide what a tool does. You want to use the most **unambiguous, clear, and literal** way of explaining that tool.
For example, `send_sms` tells the agent precisely what the tool does, whereas `process_action` is ambiguous and confuses the model.
### 10. Let tools enforce permissions
Don't strip tools based on auth level because the agent loses visibility into its capabilities. Instead, return a **clear, graceful error** when an agent calls a tool without suitable permission. If an agent knows that it has a capability that it is unable to access, it comes to a more accurate conclusion versus not having the capability to begin with.
For example, an airline might offer discounts exclusively to members. Instead of hiding the discount tool from non-members, let the tool check membership and return “This discount requires an airline membership.” The agent can then explain why the discount isn’t available and what the user needs to access it.
### 11. Treat errors as useful feedback
Tools should return **precise errors** and tell the agent what failed and why, so the agent can potentially recover and correct its course.
For example, if an agent tries to book a flight using an expired fare, returning "Booking failed" doesn't explain what went wrong. Returning "This fare has expired" gives the agent enough information to recover and recommend another itinerary.
### 12. Distinguish between different types of errors
Distinguish expected errors the agent can recover from, like a bad argument or a slug that's already taken, from unexpected failures, like a bug in the tool itself.
Give actionable feedback for recoverable errors. [Log](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#24-log-everything) and [surface](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#25-give-your-agent-a-tool-to-report-its-own-failures) unexpected failures so they can be investigated instead of silently retried or ignored.
## Context
### 13. The agent only knows what's in its context
Agents don't know what they don't know. Just because tacit knowledge exists in memory, or the answer to a database design question exists in a skill, doesn't mean the agent knows it's there if it's not discoverable.
Make sure hints are in context, not just technically available to the agent. The agent only knows what's in its context.
Keep a **small index of available files, memories, and skills** in context.
### 14. Give the agent visibility into its environment
Give the agent visibility into its rate limits, tool restrictions, expectations, weird gotchas, errors, what has happened, if it was recently compacted, if it was rate limited, if there was a change to a tool, if a memory got written, etc.
You want your agent to understand its **environment and constraints** so it can make better decisions.
For example, if an agent knows that it is about to be rate limited, then instead of persisting on the task at hand, it might decide to send a message to the admin to increase the rate limits, which is a better outcome than abruptly stopping.
### 15. Avoid compacting mid-task
[Compaction](https://rubriclabs.com/blog/what-is-an-agent-harness#compaction) lobotomizes the agent if done at the wrong time. It kills momentum, so you should keep compaction to a minimum, at natural logical boundaries, not under pressure.
For example, you might be building a complex itinerary with your flight booking agent involving multiple stops, multiple people, and corporate discounts. If the agent compacts just before completing the booking, it might lose **critical information** needed to finish the task.
### 16. Allow the agent to compact on its own
We've found that giving the agent [visibility into its context window](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#14-give-the-agent-visibility-into-its-environment) and tools to manage it allows the agent to compact at **better checkpoints** than if an external service were forcing it to compact.
In the [previous example](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#15-avoid-compacting-mid-task), an agent with control over its own compaction would probably choose to compact after completing the booking, a more natural checkpoint.
## Skills
### 17. Use skills for procedural knowledge
As an agent designer, you might be tempted to bake workflows into tools.
Instead, you want to give the agent strong primitives (tools), and bake procedural knowledge modularly in skill files.
For example, in our flight booking agent, you probably want to support multi-stop bookings. Instead of building custom tools to support this, you could use a **skill file** which the agent can follow to carry out the appropriate steps using existing tools.
## Subagents
### 18. Delegate context-heavy work to subagents
Spinning up subagents to take on **context-heavy work** and return their findings to the main agent is a great way to manage the context window of your primary agent. It also lets you fan out and target more difficult tasks that might be difficult for a single agent to achieve.
For example, a user might ask your flight booking agent to recommend three completely different vacations, including flights and hotels. You could **spawn three subagents** to research one itinerary each and return their findings to the core agent. This could help the core agent give a higher-quality response than if it researched all three on its own.
### 19. Give subagents enough context to own a task
When you first experiment with subagents, the core agent tends to create subagents with really tiny scopes of work. For example, change the border radius.
Whereas in our experience, you want the subagent to own a larger piece of the work and give it enough context to own it end-to-end. **Onboard subagents like employees**, with objectives, priorities, project context, and success criteria.
### 20. Reuse subagents
Agents tend to treat subagents ephemerally: dispatch a subagent to do a task, then another subagent to review it, then another subagent to implement the feedback. You get all these "waves" and "dead subagents."
Instead, dispatch subagents with **high ownership** to do entire, complete pieces of work, and give them follow-ups or feedback. They become the **owner of that scope**, and their context gets ripe.
### 21. Let one primary agent coordinate subagents
In some multi-agent workflows, several agents work in parallel as peers. In our experience, we prefer **one primary agent running in a single thread** that delegates work to sub-agents and uses their findings to stay on course.
We've found this approach less complicated and more efficient for most use cases.
## Guardrails
### 22. Run your guardrails outside the agent's control
You want to separate your guardrails from the core agent so they can run **outside the agent's control**. Run them outside the agent's container, with permissions that prevent the agent from modifying or disabling them.
For example, a safety service could check incoming user messages, webpages, or tool results for **prompt injection attempts** before the agent processes them. These checks might use a fast language model or a classifier to flag content that tries to redirect the agent or override its instructions.
If a check fails, block the content or stop the request. The agent shouldn't be able to bypass that decision. These checks can miss attacks, so tools should still [enforce permissions](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#10-let-tools-enforce-permissions) independently.
### 23. Separate trusted instructions from untrusted data
Clearly label which information is **untrusted data** pulled from external sources and which information includes **trusted instructions** from the user or the system.
For example, web search results or the contents of a file uploaded by the user may be treated as untrusted data, not as instructions. The model needs clear visibility into what the trusted instructions are and what is untrusted information. This allows the model to protect itself from unwanted behavior.
## Observability
### 24. Log everything
Log your conversation turns, tool calls, skill retrievals, everything. This is a prerequisite to debugging.
If an agent force-pushed a bad migration at 11:59 PM, you want to be able to replay the exact scenario in which that happened and fix the agent for those situations.
### 25. Give your agent a tool to report its own failures
Allow your agent to identify its **unexpected failure modes** and log them to your feedback tool of choice. [Raindrop](https://www.raindrop.ai/blog/agent-self-diagnostics/) does this through self-diagnostics: the agent gets a tool to report its own failures.
This allows you to capture hidden failure modes that are sometimes hard to capture from an agent trace.
## Summary
1. [Read your system prompts](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#1-read-your-system-prompts).
2. [Keep your system prompts short](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#2-keep-your-system-prompts-short).
3. [Aggressively deprecate support for old models](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#3-aggressively-deprecate-support-for-old-models).
4. [Use a model selector only if you absolutely need to](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#4-use-a-model-selector-only-if-you-absolutely-need-to).
5. [Design tools around use cases](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#5-design-tools-around-use-cases).
6. [Pick the smallest subset of tools](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#6-pick-the-smallest-subset-of-tools).
7. [Avoid overlapping capabilities across tools](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#7-avoid-overlapping-capabilities-across-tools).
8. [Don't leave agents with missing capabilities](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#8-dont-leave-agents-with-missing-capabilities).
9. [Name tools and parameters literally](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#9-name-tools-and-parameters-literally).
10. [Let tools enforce permissions](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#10-let-tools-enforce-permissions).
11. [Treat errors as useful feedback](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#11-treat-errors-as-useful-feedback).
12. [Distinguish between different types of errors](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#12-distinguish-between-different-types-of-errors).
13. [The agent only knows what's in its context](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#13-the-agent-only-knows-whats-in-its-context).
14. [Give the agent visibility into its environment](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#14-give-the-agent-visibility-into-its-environment).
15. [Avoid compacting mid-task](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#15-avoid-compacting-mid-task).
16. [Allow the agent to compact on its own](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#16-allow-the-agent-to-compact-on-its-own).
17. [Use skills for procedural knowledge](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#17-use-skills-for-procedural-knowledge).
18. [Delegate context-heavy work to subagents](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#18-delegate-context-heavy-work-to-subagents).
19. [Give subagents enough context to own a task](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#19-give-subagents-enough-context-to-own-a-task).
20. [Reuse subagents](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#20-reuse-subagents).
21. [Let one primary agent coordinate subagents](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#21-let-one-primary-agent-coordinate-subagents).
22. [Run your guardrails outside the agent's control](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#22-run-your-guardrails-outside-the-agents-control).
23. [Separate trusted instructions from untrusted data](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#23-separate-trusted-instructions-from-untrusted-data).
24. [Log everything](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#24-log-everything).
25. [Give your agent a tool to report its own failures](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design#25-give-your-agent-a-tool-to-report-its-own-failures).
What haven't we written about? This is a rolling list, and we plan on updating it as more things come top of mind.
## Related posts
- [Primitives over Pipelines](https://rubriclabs.com/blog/primitives-over-pipelines)
[Source](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design.md)
# What is an Agent Harness?
The software that makes an LLM agentic
By [Ted Spare](https://rubriclabs.com/team/ted), [Dexter Storey](https://rubriclabs.com/team/dexter), [Sarim Malik](https://rubriclabs.com/team/sarim), 2026-08-10
A harness is the **software that translates a model into a system that can affect its environment**.
For example, [Pi](https://pi.dev/) is a harness that lets the model use your terminal to read and edit files on your computer.
Functionally, a harness makes a model agentic, meaning it can take action.
Put simply:
Agent = Model + Harness
*The components of an agent harness*
A few of the harnesses we'll discuss in this post include [Claude Code](https://rubriclabs.com/blog/how-does-claude-code-actually-work), Codex, Cursor, OpenClaw, Hermes and Pi. They generally share the same underlying functional areas.
Looping
At a minimum, a harness collects LLM calls in a loop (even ChatGPT has a harness). A **common looping architecture is ReAct (Reason + Act)**, where a user sends a message and the agent thinks, searches, writes code, and responds when it's done.
*The harness executes the model's instructions and returns the result*
Modern harnesses go much further: they let agents work for a long time, learn from experience, and delegate work.
Context
Every time the agent loops, the harness passes context to the inference endpoint. **Context is all the information the model can see to decide what to do next.** A common pattern is to include:
- a fixed system prompt (which describes the agent's role)
- some current information (today's date, preferences, available tools)
- the past messages (user messages, tool calls, tool outputs and agent messages) in chronological order
LLMs have finite **context windows (a maximum amount of information they can process at a time)** and an agent that loops enough will often fill up this window. So we have to choose which parts of the total context to keep.
*The context window fills until the model cannot accept another input*
Harnesses largely use two strategies that address different parts of the problem: compaction and memory.
Compaction
Harnesses largely deal with context limits **within a single run** using a compaction strategy, where they ask an LLM to summarize historic information, and then remove it from the window.
*Good compaction can be [the difference](https://x.com/OpenAI/status/2082616640144048433) between stunted and outstanding performance.*
When a session ends, its compacted context is generally discarded.
What if we want an agent to learn over time and across sessions?
Memory
Humans actually have [a few](https://www.psychologytoday.com/ca/basics/memory/types-of-memory) types of memory, such as **procedural (how to), episodic (what happened), and tacit (intuition)**.
Most harnesses today collapse memory into **procedural vs. everything else**.
A common way to store procedural memory is in **skills**. Skills are folders that include markdown text describing a workflow, and optionally scripts that aid that workflow. Encoding your IP into high quality, opinionated skills is a powerful way to scale your leverage with agents. Harnesses determine how to create and use skills.
Tacit knowledge is the unwritten heuristics and intuition gained from experience. Modern harnesses generally bucket it into **MEMORY.md** (and, to a lesser extent, USER.md).
*Each session inherits stored memory from earlier sessions and leaves new memory for the next*
Memory is one of the areas of AI engineering that feels largely unsolved. It's unclear whether it will be solved through scaling LLM capabilities or necessitates a new architecture. Many AI engineers **believe that "continual learning" is necessary for true AGI**, where the model's weights actually change over time as it interacts with the world. This would likely imply that procedural and episodic memory, world knowledge and intuition become subsumed into the black box.
Orchestration
We'll look at two mechanisms an agent uses to orchestrate its work: planning and dispatch.
Planning
As coding agents begin to run for longer, and deploy more intelligence through [dispatch](https://rubriclabs.com/blog/what-is-an-agent-harness#dispatch), they are owning larger and more complex problems end to end, and the developer is less in the loop to steer the agent. The value of high quality planning increases as agents implement the plans more autonomously. Harnesses like [Claude Code](https://rubriclabs.com/blog/how-does-claude-code-actually-work) and Codex ship with a native planning mode, where **the agent must first create a detailed Plan.md file with feedback from the user** before executing. These harnesses then place a reference to the plan and the todo list into a top level state (system prompt) so that the agent doesn't forget what it's working on across long runs.
Dispatch
For complex tasks, modern harnesses enable the primary agent to delegate work to one or many sub-agents. This is powerful because it **enables the agent to elastically scale to meet the complexity of the task**, but it also introduces complexity because you have to coordinate across fragmented state. Without a coherent orchestration strategy, agents can overwrite each other's work, get caught in loops, or lose track of the core task.
Orchestration is one of the newer components of agentic engineering as we transition from single agent runtimes into agent fleets and agent swarms. A common strategy for coding agents is to have the main agent come up with a plan and a todo list for a large build, and then dispatch one agent per todo list item, and then one agent to review each item.
*Three agents work in parallel, then return their results to the primary agent*
Tools
Execution
Agents interact with the outside world using tool calls. A **common pattern is to define tools through an input schema and output schema**. The LLM outputs JSON text that is compatible with the input schema, then the harness interprets that and executes actual code (such as executing an API call), awaits the response, and returns data to the model with the result of the action.
The harness's solution for transforming LLM tool calls into actual function executions on computers carries strong implications for how the system thinks about authorization and autonomy. For example, do you sign in to your email account in a browser and give the agent tooling to control the browser? Or do you expose tools to read, draft and send emails, then execute those actions through your email provider's API with OAuth?
Interfaces
**MCP (Model Context Protocol)** servers have emerged as a **standard interface for scaling tools**.
Historically, developers have had to write bespoke integration code for every tool they want to use. In the email example, the harness would have to host code that translates the LLM's `email.send` payload into an authenticated API call to the email provider, and every other harness would have to do the same.
With MCP, the community or email providers themselves can build this tooling once, and the harnesses just need to adopt MCP broadly.
MCP is by no means necessary: CLIs or simply good documentation offer similar benefits.
Verification
As the plans we give agents get harder and touch more complex surface area, it becomes paramount to evaluate whether solutions actually work. Today, most harnesses leave this largely up to the model and tooling, or encode frameworks in the system prompt. In the code world, AI systems are encouraged to **write unit tests, and use tools like Computer Use and Browser Use to conduct smoke tests**.
Unit tests can end up being low-signal, as the agent often writes the test with full context of how their code works, and adds an arbitrary check mark on something that may not actually work.
Smoke tests can catch the kinds of issues that usually surface through use. In a smoke test, an agent walks through the final product, sanding down any rough edges it encounters.
**AI Labs are increasingly post-training their models using reinforcement learning environments where the model is set up with a harness**, so it's possible that verification becomes an emergent model level instinct and doesn't need to be an opinion of the harness.
*A reviewer sub-agent rejects the work, then approves the retry before it returns to the primary agent*
Permissions
When an agent wants to do an action such as executing code on a computer, the harness needs to decide whether that action can be executed immediately, or if the action needs to be reviewed for safety. Harnesses like Codex historically required the user to review every new action the agent wanted to take on the computer, and allow that action once or in perpetuity. Users could bypass this setting with "YOLO mode", which approves every request automatically.
Newer iterations often include an autonomous AI review where all actions are passed into an LLM for review and a request is only surfaced to the user if the LLM deems it "risky". Permission granularity evolves as users build trust in the previous generation of systems.
What's next for harnesses?
Harnesses represent a **major new lever to scaling AI output**, alongside model size, data, and compute. They can unlock [18x cost savings](https://x.com/OpenAI/status/2082616640144048433) and [superhuman scores](https://www.primeintellect.ai/blog/prime-agent) on complex puzzles, and foundation models are **increasingly being [trained](https://x.com/liquidai/status/2084640701669613906) with harnesses in mind**.
Seeing such dramatic improvements on hard problems, we can only imagine what's possible in real-world settings.
## Related posts
- [How does Claude Code actually work?](https://rubriclabs.com/blog/how-does-claude-code-actually-work)
[Source](https://rubriclabs.com/blog/what-is-an-agent-harness.md)
# Challenging the Chatbot
Intelligent applications need new interaction patterns
By [Sarim Malik](https://rubriclabs.com/team/sarim), 2026-07-07
Chatbots have become the de facto interface for AI-native software.
My daily stack is filled with chat-first coding agents like Codex and [Claude Code](https://rubriclabs.com/blog/how-does-claude-code-actually-work). Much of what we build at Rubric is chat-first by design.
There are **multiple factors that make chatbots compelling**:
1. People already know how to use them
2. You can show up with a rough idea of what you want and still reach a solid outcome
3. There's a built-in feedback loop: you ask, the model responds, you refine, etc.
People now reach for a chat thread for work that used to require menus and forms. Not only has software development changed, but the way we *use* has fundamentally changed. Which raises a more existential question: **what exactly is software?**
At its core, user-facing software is a [human-computer interface](https://youtu.be/PYobZzjW_ic?si=nRwK4ugGj0fyn_yM). It is how people express intent to a machine and how the machine reflects state back. Every button, filter, menu, and chat box is part of a translation layer between you and the computer. Once you see software that way, it becomes clearer why **language may be the best way to express intent.** But this does not mean that it is the best way to express *state.*
On a [podcast](https://youtu.be/yAj5EnyuakI?si=BFyuCrRxTOGSAYgp\&t=1306), Naval Ravikant describes AI models as **natural language computing**, where you say what you mean instead of learning command syntax or navigating menus.
If **natural language is the new input**, the larger design question starts to center around where intelligence can shape the translation layer. In this post, we will look at a few different ways intelligence can fit into new ways we use computers.
## Interaction Pattern Examples
### 1. Keystroke Intelligence
One useful pattern is to let language capture intent, then immediately turn that intent into visible, editable application state.
Traditional dashboards often make users translate their goal into filter logic. For example, if you want to find large software purchases from last quarter that still need approval, you have to know which fields exist, which menus to open, and how the product names each status or category. In other words, you have to internalize a data model *and* a software interface before you can get your answer.
An intelligent interface would let the user **start with the goal instead**, where the system can interpret the request, apply the relevant structured filters, and show the resulting filter state back to the user so that it can be inspected or edited.
Let's look at a concrete example. Imagine **a traditional banking product called Ledger**, which has a table of transactions and a handful of filters for fields like amount, merchant, department, and policy. If you've used Brex or Ramp, Ledger probably feels familiar.
Usually, those fields become a dynamic form made of filter buttons. The user has to assemble the right combination by hand.
Click around the figure below to feel how much of the interaction is spent translating intent into interface mechanics.
*Traditional software asks the user to assemble the right filter combination by hand*
Even in this small static example, you probably spent a few seconds learning the interface. And even when it works, the outcome may not map exactly to what you had in mind. The interface can only express the shapes its developers anticipated. If you want to ask for the same information in a slightly different way, you run into the edges of the product.
That is one reason traditional software can feel frustrating. You have to learn the product's vocabulary and remember how to operate it later. But the advantage of this type of software is that **it is predictable**. The whole team will get the same output given identical inputs.
We still want that deterministic trust. So the opportunity is to find a compromise: **let language express intent, then turn that intent into visible, editable state.**
Try the flow below by describing the transactions you want to see in the input.
*Intelligent software can use language to assemble filters while preserving visible, editable state*
In our local runs, most requests landed in the **300-700ms** range, which is close enough to the latency people already tolerate in normal software interactions. The cost is also small enough to change the product math. One representative request cost **0.000842¢**, which means you could run roughly **1.2 million** filter searches before spending **$10**.
And of course, inference costs keep falling, [latency keeps improving](https://epoch.ai/publications/inference-economics-of-language-models), and smaller models keep getting better at narrow structured tasks. That makes it reasonable to imagine interactions like this eventually feeling instantaneous enough that the user does not think about the model at all.
These are the sorts of questions we've been asking ourselves at Rubric. **Where is it worth spending a little intelligence on each keystroke?** The answer seems to be that anywhere the product is trying to capture intent, translate vague language into application state, or help the user shape a query. That's where this pattern starts to open up.
### 2. Ambient Intelligence
Ambient intelligence is **work the product does in the background**. People like to talk about "proactive software," and ambient intelligence is one way software can be more proactive, where intelligence is running at all times but only interrupts the interface when it has something useful to surface.
As an example, imagine **a CRM product called Pivotal**. Instead of waiting for a sales rep to ask a question, Pivotal scans account activity and prepares recommendations for the user to inspect.
Click one of the recommendations in the figure below.
*Ambient intelligence can run on a fixed schedule, assemble useful reports, and let users tune what gets surfaced next time*
The product gathers data from inside the CRM and recommends an action to the user. While some recommendations are just nice to have, critical ones **catch something that would have otherwise gone unnoticed**.
Click the gear icon on an open recommendation to see the instruction behind it and how often it runs.
As discussed previously, one of the factors that makes the chatbot compelling is its feedback loop. A **surfaced recommendation is only half of that loop**. The other half is a way to give the model feedback as it works in the background.
In this case, the recommendation arrives as a dedicated action item, with a link that takes the user to the right place inside the product. The user can then interact and provide feedback accordingly.
Use the thumbs up or thumbs down on an open recommendation to see how this kind of feedback loop might work.
Ambient intelligence is most useful when it feels like the product did useful work and brought it back at the right moment, with clear ways to inspect what it found and shape what happens next. It shouldn't feel like magic out of thin air, but rather **a configurable part of the product.**
### 3. Inline Transformation
The first two patterns applied intelligence around the interface. The third pattern **embeds it directly inside the content** the user is working on.
Consider how you might write and edit using ChatGPT or Claude. You copy a paragraph into a thread, describe a change, and get back a rewritten block of text. Then you paste it over the original and reread everything, not knowing whether the model made edits you didn't want or didn't notice.
With inline transformation, the user selects the exact text they want to change, describes the change in place, and reviews a diff exactly where the text lives. For this example, **imagine an essay-writing tool called Pencil**. Instead of a chat panel next to the document, Pencil puts the model *inside of the document*.
Select any phrase in the editor below and describe how it should change. Or type / after a space to write something new at the cursor.
*Inline transformation lets the user scope an edit by selecting text, describe the desired change, and review the resulting diff in place*
The model sees the selected text, some surrounding context, and the instruction, then returns only the replacement.
The feedback loop from the ambient pattern shows up here too in a compressed fashion. Accept and discard are the thumbs up and thumbs down, and because the edit is scoped, rejecting one costs nothing. The user stays the editor; the model is a collaborator that proposes diffs.
## Where This Leaves the Chatbot
The chatbot is still currently the best interface we have for exploration and for work that has no fixed shape yet, where models assemble decision paths on the fly ([Primitives over Pipelines](https://rubriclabs.com/blog/primitives-over-pipelines)), but it is just the beginning of this design space.
The three patterns above share the same shape: language captures the intent, the product turns that intent into visible, editable state, and a feedback loop lets the user steer what happens next. Edited filters, thumbs on a recommendation, an accepted or discarded diff.
Future interfaces will look like the software we already know, with intelligence embedded wherever users are trying to express what they mean.
## Related posts
- [Primitives over Pipelines](https://rubriclabs.com/blog/primitives-over-pipelines)
[Source](https://rubriclabs.com/blog/challenging-the-chatbot.md)
# Autonomous Iteration
Testing coding agents' ability to improve a feature automatically
By [Ted Spare](https://rubriclabs.com/team/ted), 2026-06-10
A coding agent’s first draft is usually good, but it’s rarely the best it could be.
People close that gap by hand: look at the work, notice what’s off, fix the worst thing, look again.
We wanted to see if the agent could run that loop on its own, so we ran a series of experiments across UI, performance, and UX.
## Method
Our core idea is quite simple:
1. a coding agent writes code,
2. another agent critiques the output,
3. the original agent addresses the feedback.
We repeat until the critic is happy.
Any [coding agent](https://rubriclabs.com/blog/what-is-an-agent-harness) will do, so long as it can use a computer, see screenshots, and deploy sub-agents.
We used [Pi](https://pi.dev) for its simplicity.
We’ll develop the prompt over the course of three examples, in order of increasing difficulty.
## Examples
### 1. SaaS Landing Page
We start with a straightforward example: building a marketing page for a software product.
By default, coding agents tend to produce the [familiar](https://x.com/wesbos/status/2062180597015990472) purple-gradient look.
We want pages to be distinct.
The meta-prompt we used is captured as a [SKILL.md](https://skill.md):
```md
---
name: iterate-ui
description: Build a UI, screenshot it, critique the screenshot, and improve it.
---
# To iterate on UI
1. Build the feature.
2. Run the dev server and take a screenshot.
3. Look at the screenshot and list what looks bad.
4. Fix the worst issues.
5. Repeat until it looks good.
```
You’ll notice it’s intentionally simple and UI-specific, but it steers clear of defining what's good.
We will make it more general in the coming examples.
Here is the result of three rounds of iteration:
*“build a mobile landing page for Cadence, a habit tracker for teams”*
The one-shot result covers basic copy but lacks hierarchy or variety.
The third iteration is scannable, with a clear H1. It leads with a representation of the product itself. It adds social trust as ratings, user logos, and testimonials. Its CTA nudges the user forward with a low time commitment and the option to back out.
The site feels more polished, even without an explicit rubric.
We’ve successfully squeezed quality out of the agent.
In the next example, we get quantitative.
### 2. Performance Optimization
In this example, we have the coding agent build an interactive table.
We render it with 10 000 rows and measure how long it takes to load and filter.
The agent's initial solution is slow and costly.
To improve it, we add the following to our meta-prompt. Note that this introduces some definition of what we consider "good". We will generalize it in the final example.
```md
...
# To iterate on performance
1. Define a metric (e.g. p50 filter latency in ms) and write a benchmark for it.
2. Write a test suite that pins behavior. Ensure it passes.
3. Each pass: form one hypothesis, change one thing, re-run tests + benchmark.
Keep the change only if tests pass AND the metric improved. Otherwise revert.
4. Stop when you hit the target or gains flatten. Log before/after numbers each pass.
```
After two iterations, the model recalls common optimization techniques:
- memoize computed data,
- debounce the search input, and
- virtualize the table (render only visible rows plus a small buffer).
Here, the improvement is drastic:
*“build a React table that lists transactions with a search filter, sortable columns, and column sums"*
With a benchmarking harness, the agent improved load speed by 13x and search performance by almost 100x.
You can try each iteration [here](https://rubriclabs.com/blog/iterative-improvement/demos/dashboard).
In the next example, we task the agent with iterating on a full-stack, product-level feature.
### 3. Product Onboarding Flow
To improve an onboarding flow requires all of
1. a model of the underlying product,
2. the ability to click through the flow and take screenshots, and
3. the ability to judge (2) against (1).
The agent's initial solution is dense and high-friction.
Try stepping through each iteration below:
*“scaffold an onboarding flow for Cadence - create an account, set up a team, and pick some habits"*
The agent added optimistic validation, sensible defaults, and made optional steps skippable. It also removed fields that can be completed later.
In general, these changes reduce friction.
Friction was probably the correct metric to focus on.
## A skill for all scenarios
We want the agent to be able to improve any feature without an explicit rubric, which means identifying *which* metrics to push forward.
To do so, we generalize the meta-prompt to *whatever the agent thinks is highest-impact*.
- Given a static page, polish the UI.
- Given a pre-built component, make it fast.
- Given a marketing feature, remove friction.
Below is the combined skill:
```md
---
name: improve-feature
description: Improve a feature across iterations. After building the feature, define how to judge it, measure the current state, fix the highest-impact problem, and re-measure until it meets the bar or gains flatten.
when_to_use: After producing any artifact (a UI, a component, an API, a flow) where the first attempt is unlikely to be the best one.
---
# Iterate
The first version is a draft. Your job is the loop that follows.
## 1. Define the bar
Before improving anything, write down how this artifact will be judged:
- Visual → a 1–5 rubric (hierarchy, contrast, type scale, spacing, CTA clarity,
trust signals, "does it look generic?"). Always screenshot at a fixed viewport.
- Performance → one measurable metric (e.g. p50 latency in ms) plus a benchmark
that prints it, and a test suite that pins correct behavior.
- Flow / interactive → a concrete task to complete; success = the artifact can be
used end-to-end without confusion. Drive it with browser/computer use, not a glance.
## 2. Measure the current state
Run it and capture evidence (screenshot, benchmark number, walkthrough recording).
Write down the top problems, ranked by impact.
## 3. Change one thing
Fix only the highest-impact problem this pass. Don't refactor opportunistically.
## 4. Re-measure and gate
Re-capture the same evidence at the same settings.
- Keep the change only if it improved the bar AND broke no guardrail (tests stay green).
- Otherwise revert and try the next hypothesis.
## 5. Stop deliberately
Stop when you meet the bar, run out of budget (cap the passes), or gains flatten.
Keep a short changelog: what was wrong, what you changed, before → after.
```
We are confident this skill can be improved - maybe even automatically.
## Related posts
- [Contract Engineering](https://rubriclabs.com/blog/contract-engineering)
[Source](https://rubriclabs.com/blog/autonomous-iteration.md)
# Every Server Deserves a Coding Agent
The promise and peril of collocating code and coding agent
By [Ted Spare](https://rubriclabs.com/team/ted), 2026-05-01
## Introduction
We recently ran an in-person experiment. We wanted to give non-technical people the chance to try [OpenClaw](https://openclaw.ai) (or, technically, a fork thereof, but we’ll use the term OpenClaw for simplicity).
To make onboarding as frictionless as possible, users only had to text a number: [8677-RUBRIC](sms:+18677782742?body=hi!).
*Text us (or scan a QR code) to get an OpenClaw*
Over a series of texts, the agent would guide them through picking a username, setting some goals, and deploying a site on their unique subdomain.
Over the course of the experiment, several problems arose:
1. **We needed a landing page.** Users didn’t know what the product was.
2. **We needed onboarding.** Users understandably had no idea what to text to the agent and churned immediately.
3. **We needed to fix bugs.** One user’s agent couldn’t find a browser. Another’s got rate-limited (when they had it send one text per second, which is technically in scope).
Even one year ago, we would have noted these hiccups, written them up, converted them to tickets, prioritized them against a roadmap, then worked through the tickets over the course of days and weeks.
In this case, the demo itself was being supervised by an OpenClaw instance. We simply Slacked it on the spot, got a landing page, got an onboarding flow, and got the [agent harness](https://rubriclabs.com/blog/what-is-an-agent-harness) fixed.
*The user sees a text thread. The agent does the rest.*
These coding sessions took several minutes each, but since the superadmin OpenClaw lived on the production server, we [skipped GitHub](https://mitchellh.com/writing/ghostty-leaving-github) entirely and builds took mere seconds.
## To stream or not to stream
There is no shortage of products that stream pages of AI-generated code to the end user. The pattern follows naturally from the legacy of AI-assisted IDEs like Cursor.
However, if we instead allow ourselves to view code as increasingly low-cost, reliable, and ephemeral, it’s natural to hide it from view entirely and only show the result. Doing so enables a few interesting use cases:
1. **Non-technical users can participate in the software-building process.** Software development can become a normal part of everyday life, much the same way reading and writing went from a practice of a niche group to a pillar of public education.
2. **Software can be updated on the fly.** For example, from the subway, between conference talks, or on a site visit. Customer feedback can be implemented mid-interview, without the product owner context-switching to code. In practice, co-locating the coding agent with the source code does lead to minimal (often sub-10 second) build times.
3. **Non-technical users can climb the capability ladder** without taking on the burden of tech know-how. Users can build one-off pages for events, farmers’ market stands can build loyalty programs, and WhatsApp-based logistics companies can scale NPS surveys.
These last few require a few more steps, which we'll return to.
## Faster horses
> "it’s almost inevitable that this is the way people are gonna use computers"
New technologies enable new form factors. Pioneers of the web [think](https://www.latent.space/p/pmarca?selection=0b21cbcc-1fd1-41b7-a76a-388c7cde642e) "it’s almost inevitable that this is the way people are gonna use computers". We think OpenClaw is exciting because it gets several UX assumptions deeply right:
1. **A computer terminal is the everything tool.** From the shell, an agent can do just about anything on the internet, like write code to [order coffee](https://www.terminal.shop) then Slack you with the delivery time.
2. **A chat client is all you need.** This doesn’t have to be a fancy iOS app. Slack, SMS, Discord, a CLI - anything works so long as it supports (at a minimum) text in and notifications out. The medium does influence the messaging; for instance, we found SMS lent itself to short, simple requests.
3. **Being always-on is a must.** Using a computer is useful, but being able to save files and run long-running code supports use cases like serving a web app, setting reminders, or tracking a price.
We can push the standard OpenClaw setup further by considering the system more as an admin dashboard than as a personal assistant, and then by giving it access to a few *rare digital goods*:
- **A domain name**, e.g. to generate single-use shortlinks or OAuth callbacks, spin up internal tools, or scale entire web apps.
- **A virtual credit card** (with $1), e.g. to create SaaS accounts, grab API keys for AI inference, or access paywalled content.
- **A phone number and email**, for auth and 2FA.
These are the OpenClaw’s passport to the internet.
Once [unblocked](https://rubriclabs.com/blog/unblocking-agents), the agent can also migrate its own tech extremely flexibly, e.g. from Python to Rust, from a closed-source to an open-source LLM, or from a VM in Singapore to the Raspberry Pi in your palm. In our in-person experiment, the underlying agent went from OpenClaw to [Hermes Agent](https://hermes-agent.nousresearch.com) to a custom, lightweight harness. Since the [agent is increasingly its memories](https://x.com/hwchase17/status/2042978500567609738), little changed for users.
If you’re an OpenClaw user (or OpenClaw instance...) and want to try deploying OpenClaws for your non-technical friends and family, here’s the key idea as an [agent skill](https://agentskills.io/home):
```md
---
name: subclaw
description: Spin up a sandboxed OpenClaw-like subagent on its own public subdomain, addressable from a chat client (e.g. SMS via Twilio). For giving a non-technical user their own always-on coding agent without exposing your inference key.
when_to_use: You run an OpenClaw-like harness and want to provision an isolated one for someone else.
---
# Sandboxed Sub-Agent
## Key decisions
1. **Sandbox = microVM, not container.** Firecracker (or Cloud Hypervisor / Kata) gives a real kernel boundary per tenant in ~125ms cold-boot, ~10MB overhead. Docker namespaces aren't enough once the agent has shell-exec — one bad apt install and you share a kernel with untrusted code. Don't compromise here; everything else is recoverable, a kernel breakout is not.
2. **Persistence = per-tenant overlay on a shared rootfs.** Bake one golden rootfs.ext4 with the base OS + preinstalled tools (bun, jq, curl, git, ripgrep, sqlite). Each tenant gets a copy-on-write overlay file; their VM mounts rootfs (ro) + overlay (rw). Snapshot on shutdown, restore on next message. Cheap state, fast boots.
3. **Inference lives on the host, not in the VM.** Most load-bearing decision. The VM has no API key, no model client, no rate limiter. The host runs the outer agent loop and calls the LLM; only tool I/O crosses into the VM. The tenant fully customizes their agent by editing files at a known path inside their VM (e.g. /etc//system.md, tools.json, skills/*.md) which the host re-reads each turn. Built-in privileged tools (DB writes, billing, llm_complete for nested calls) are non-shadowable. Every alternative leaks the key within a day.
4. **Routing = wildcard DNS + dynamic Caddy.** Point *.yourdomain.com CNAME at one box. Run Caddy with on-demand TLS and the admin API enabled (localhost:2019). On VM boot, register a route mapping .yourdomain.com → the VM's tap-device IP via the admin API. Two traps: **(a) use PUT /config/.../routes/N to insert at an index, not POST — POST always appends regardless of the index in the path, which dumps tenant routes after your wildcard catch-all and yields universal 502s; (b) keep the wildcard route last.** Idempotent pattern: PUT /id/ first (replaces in place if exists), fall back to PUT at the wildcard's index.
5. **Base toolset is small and opinionated.** ~10 built-in host-side tools and trust the agent to compose: shell_exec (in-VM), read_file, write_file, search_files, list_channels / notify_user / schedule_notification (out-of-band notifications: persist jobs in a DB-backed scheduler, tick every 5s, claim atomically; for agent_turn mode, dependency-inject the agent runner from the bootstrap layer to avoid a scheduler:left_right_arrow:agent import cycle), reload_harness, and one llm_complete so the agent can fan out work without an API key. Let the tenant write their own tools in tools.json. Preinstall jq and pass tool input as a base64-encoded JSON env var — never string-interpolate user input into a shell command.
6. **The user-facing UX lives in the system prompt.** Hard-won rules: (i) never narrate tool calls or stream raw tool output; summarize the outcome in one sentence — "your site is live at x.yourdomain.com," not the build log; (ii) prefer one short reply over a wall of text — chunk anything longer and ask before continuing; (iii) on failure, say what you'll try next, don't paste the stack trace; (iv) for ambiguous requests, ask one clarifying question, not three; (v) the agent's memory of the user is a markdown file the agent itself owns and edits — don't introduce a separate memory tool, it splits the truth. Ship a safe-default system.md in the rootfs overlay and have the host hot-reload it each turn, with a fingerprint check so user customizations aren't clobbered on upgrade.
7. **Onboarding = one chat surface, one tenant.** Pick any chat client (SMS via Twilio is friendly for non-technical users; Discord, iMessage, Matrix all work). Point its inbound webhook at your host and route by sender ID → tenant. First message creates the tenant, picks a username, boots the VM, registers the Caddy route, and replies with the live URL. No app, no signup form, no GitHub. Note: VoIP numbers, non-.com domains, and fresh-IP egress get 2FA-blocklisted by major platforms — budget time for unblock work or use already-warmed assets.
## Out of scope
- **Egress controls.** A tenant with shell_exec can hit any external host or your cloud metadata endpoint. Add iptables rules on the VM's net namespace and an egress proxy with allowlisted destinations.
- **Cost caps.** Meter tokens per tenant on the host (every LLM call goes through you, so this is easy) and hard-cap free tiers.
## MVP shape
Single VPS, ~20 LOC of TS per concern: an HTTP server handling chat-client webhooks, a Firecracker manager (/run/vmm.sock is fine), a Caddy admin client, and a per-turn agent loop that reads /etc//{system.md,tools.json} from each tenant's VM over a vsock or local socket. Bind-mount source into the server container so edits apply on docker compose restart server without a rebuild. Fits in a few thousand lines, runs comfortably on a $20/mo box for the first dozen tenants.
## Why this works
It aligns who-pays with who-controls — the operator pays for inference and owns the key, the tenant owns their prompt, tools, and files and lives entirely inside their VM. That gives the tenant maximum privacy in their box while keeping the operator's mistakes (and the tenant's) recoverable from outside it.
```
## For every server, an agent
[Coding agents](https://rubriclabs.com/blog/how-does-claude-code-actually-work) have [matured](https://epoch.ai/benchmarks?view=graph\&tab=benchmarks\&benchmark-selection=%7ESWE-Bench+verified%7Eterminalbench) to a point where they can routinely build working software. This unlocks a fundamentally new way to interface with computers, of which OpenClaw is one example.
There are still serious security and UX challenges to making this type of app production-grade, but to skip the opportunity would be a loss for non-technical users.
Text the system. Get software. Swap parts. Test and repeat.
## Acknowledgments
Thank you to [Jihad Esmail](https://x.com/jaesmail), [Max Musing](https://x.com/MaxMusing), and [Justin Bellavance](https://www.justinbellavance.com) for their thoughtful input.
## Related posts
- [Unblocking Agents](https://rubriclabs.com/blog/unblocking-agents)
[Source](https://rubriclabs.com/blog/openclaw-backend.md)
# Contract Engineering
How we define done for AI agents using verifiable contracts
By [Dexter Storey](https://rubriclabs.com/team/dexter), 2026-04-09
Our team has been running experiments toward fully unsupervised development, with the goal of being able to write a spec at night, and wake up to working, deployed, production-ready software. Agents can already build and deploy without handing the keyboard back to you, given their own accounts, infrastructure, and verification tooling. We wrote about how we set that up in [**Unblocking Agents**](https://rubriclabs.com/blog/unblocking-agents).
But the results haven't always been consistent. Over time, we realized this was because nobody had defined what "done" means in a form the agent can be held to.
We've been approaching this with something we call **contracts**: hard, versioned, executable claims about what the system must do. A contract is both the specification and the acceptance criteria. The agent builds until every contract passes, and it cannot ship until they do.
## The Problem with Prompting
When building with agents, your spec is the conversation. The details of the product are teased out through your prompts, the agent's responses, and the ongoing back and forth. This is the source of truth for the entire system, and it breaks in three ways.
The first is **decay**. The first prompt is always the best because you're offering the full picture with high context. As you continue your conversation with the agent, you might prompt to fix a specific issue: "Fix the auth redirect." The agent fixes it but breaks session handling. You catch that: "The session isn't persisting." It fixes the session, but the fix doesn't match the pattern used elsewhere. The context window fills up and gets compacted. The agent loses track of early decisions. You lose track of the code because you didn't write it. The spec is now scattered across dozens of messages with no sense of consistency.
The second is **non-determinism**. Even with a perfect prompt, agent runs aren't reproducible. You can have the same prompt in a different session and get a completely different result. Some runs feel locked in: the agent makes good decisions and the architecture is clean. Other runs drift. The difference is sampling luck, context ordering, and temperature. You can't reliably reproduce what worked, and you can't explain why something didn't.
The third is **specificity**. [PRDs](https://en.wikipedia.org/wiki/Product_requirements_document) were designed for humans under the assumption that engineering is expensive and that efforts take weeks with constant subtle feedback signals during sprints. A PRD says "the user should be able to send an email" and trusts the engineer to figure out what that means across OAuth, API calls, database writes, cache invalidation, and realtime updates. An agent, on the other hand, needs a level of specificity that would be unreasonable to ask of a human engineer, but is exactly right for a process where execution is cheap and restarts are free.
We need a format that's durable, deterministic, and precise enough that an agent can execute against it with no interim feedback and get it right.
## Contracts
A contract is a hard, versioned, executable definition of what the system must do. It defines a scenario — what the agent should build — and a sequence of events the system must produce, each with a proof requirement. Every claim resolves to something binary, verifiable against the real system. For example, either the database row exists or it doesn't, the webhook arrived or it didn't, etc. It doesn't live in a conversation or decay between runs. The agent doesn't decide what matters while it's building because the contract decides up front.
Contracts differ from [test-driven development (TDD)](https://en.wikipedia.org/wiki/Test-driven_development) in that they are durable and survive re-writes. As code changes, tests must be re-written, whereas contracts are expected to remain untouched except by the designer.
This is the specificity that prompting lacks. A PRD says "the user should be able to send an email" and trusts the engineer to fill in the gaps. A contract, on the other hand, spells out ***every event the system must produce*** across every layer — UI, API, database, cache, webhooks, realtime — each with proof that it happened.
## What This Looks Like
Take a fullstack email app: OAuth with Google, sending via the Gmail API, receiving via webhook, database writes, Redis cache invalidation, and SSE realtime updates.
The end-to-end contract for "send and receive" defines the exact event log the system must produce:
*The full send-and-receive lifecycle. Each step maps to a contracted event with proof — screenshots, database queries, API responses, SSE messages.*
Each event is independently verifiable from outside the application. Verification combines **deterministic** checks like database queries, response codes, Redis messages, and log entries with **qualitative** checks where an LLM scores screenshots against expected states.
*13 events, verified in order. The agent can't claim success without every check passing against real infrastructure.*
The contract can't be satisfied without the system actually working end to end. And it doesn't matter which run produced the code, or what the agent's context looked like, or what sampling decisions it made. The contract is the same every time.
## Catching Bugs
The agent runs the SendAndReceiveEmail contract. Events 1 through 9 pass. The email is sent, the webhook arrives, the database rows exist. Then event 10 — `cache:redis:publish` — fails. The evaluator queries Redis and finds no PUBLISH on `inbox:{accountId}`. The agent wrote the webhook handler but forgot to publish the cache invalidation event.
Without the contract, the agent reports success. The app works in testing because the page refreshes and loads from the database directly, but in production, users with the inbox already open never see new emails arrive. SSE never fires because the Redis pub never happened.
With the contract, the agent gets kicked back. It knows exactly what failed, exactly what the proof requires, and it iterates until Redis shows the message.
This is what specificity buys you. A PRD would never mention Redis cache invalidation. A prompt might, if you remembered in the moment. But a contract always does, because ***it defines every event the system must produce***, not just the ones that are obvious.
## Sharpen the Spec
Again, contracts don't decay. The specification moves out of the conversation and into the repo as a versioned, reviewable, durable document.
This makes restarts cheap. Without contracts, a restart throws away hours of prompting and context, like edge cases and failures that taught you what the product really needed. With contracts, you can throw out the code, the context, and the thread, but the contracts still carry everything forward. ***Different agent runs produce different code but satisfy the same contracts.***
And contracts accumulate. Run 1: the email app has 13 contracts. All pass. A week later, you discover the Gmail API rate-limits a send and the UI still shows a success toast. You add a new contract that sends 10 emails in quick succession, forcing the 429. Now the contract requires the API to surface the rate limit and the UI to show an error instead of a false success:
```
{ event: "api:send:ratelimited", proof: apiResponse(429) }
{ event: "ui:compose:error", proof: screenshot({ toast: "error" }) }
```
That edge case is locked in. The next agent run — potentially completely different code — must handle rate limiting correctly or the contract fails.
A month later, the inbox doesn't sort correctly when two emails arrive within the same second. You add a contract that triggers two webhooks back-to-back with timestamps 10 milliseconds apart, then checks that the inbox renders the newer one first:
```
{ event: "ui:inbox:order", proof: dom({ firstChild: newerEmail }) }
```
The spec gets sharper over time instead of fuzzier.
## Contracts Are the Product
Contracts are the highest-level language that's still pseudo-deterministic. You're writing precise, verifiable descriptions of what the product must do, and the agent's job is to produce code that satisfies them.
This also means the engineering effort shifts upstream. Instead of writing code and manually verifying, you define contracts and let the agent iterate until they pass. The hard thinking moves from implementation to specification: what does "done" actually mean, across every layer of the system?
In a meaningful sense, ***the contracts are the application***.
## Limitations
We've been running this approach against real applications, and it's already surfacing a new class of problems — the kind that only show up once the basic architecture is right.
**Scarce infrastructure.** Some contracts require resources that can't be cheaply provisioned on demand. For example, a first-time signup needs a fresh email account, SMS verification needs a real phone number outside the VOIP blocks, and payment flows need real bank accounts with real money. We're approaching this with resource pools — pre-provisioned resources leased to a run and returned afterward — but pools have limits, and some resources are genuinely finite.
**World parity.** A contract that passes locally doesn't necessarily pass in production. Real APIs ship breaking changes without notice, real networks have variable latency, real OAuth providers occasionally change their consent screens. A contract should pass in every environment, but achieving that consistently is an open problem.
**Qualitative drift.** LLM-scored screenshots are powerful but noisy. The same screenshot can score differently across evaluations. We're working on pinned scoring (caching evaluations so a passing screenshot stays passed until the UI actually changes) but calibrating the boundary between acceptable variance and a real regression is ongoing.
**Ambiguity.** Contracts currently follow two principles: being verifiable (or, at least, having a feedback loop to hill-climb), and being unambiguous enough that a junior developer could implement them. Formalizing and streamlining these guidelines into a reusable framework is non-trivial, and something that we need a framework around.
These are hard problems, but they're also the right problems given the tools at our disposal.
---
At night, you write the contracts. By morning, the agent has built against them, deployed the result, rerun failures, fixed what it could, and left behind proof: what passed, what failed, what changed, and why. This is Contract Engineering.
We're building in the open at [github.com/rubriclab](https://github.com/rubriclab).
## Acknowledgments
We'd like to share a special thank you with Jihad Esmail, Max Musing, and Erik Kaunismäki for their thoughtful feedback.
## Related posts
- [Autonomous Iteration](https://rubriclabs.com/blog/autonomous-iteration)
[Source](https://rubriclabs.com/blog/contract-engineering.md)
# Unblocking Agents
Tooling and infrastructure for autonomous software engineering
By [Dexter Storey](https://rubriclabs.com/team/dexter), 2026-03-31
At Rubric, our goal is to build a process that allows us to write a spec at night, and wake up to working software. The agents are good enough to build real applications, but lack the permissions and resources to finish them.
> "I've written the migration. Can you run it? I don't have permission to execute against the database."
> "I've set up the OAuth handlers, but I can't create the GCP project needed for credentials. Can you create a project in the console and add the OAuth redirect URI? I'll need the client ID and secret to continue."
> "I've built the signup flow. Can you test it manually? I'll need a real email and phone number to get through verification."
The agent wrote the code, but it can't provision what it needs, deploy what it built, or verify what it produced.
To unlock truly autonomous agents, we believe that agents must be free to work however they see fit, which requires guardrails that encourage that freedom without sacrificing safety. This essay explores the steps we've taken to achieve that goal.
## What Blocks Agents
Agents working on real applications hit three types of walls.
**Access** is the most basic layer. Agents need permission to act on your machine to do things like writing files, running shell commands, installing packages, and making network requests. An unrestricted agent can do real damage, so frameworks gate actions behind approval prompts. Even when the agent is doing exactly what you asked, the [permission model](https://rubriclabs.com/blog/what-is-an-agent-harness#permissions) assumes it might not be, and you're left sitting in front of a screen clicking "yes" repeatedly.
*The agent writes the code, then yields. It can't create the GCP project it needs to continue.*
Beyond local permissions, the agent needs credentials for the services the code depends on. These might be API keys, cloud credentials, deployment tokens, or database connection strings. Most teams keep these tightly scoped or out of the agent's reach entirely. In practice, this means every missing credential either halts the agent entirely, or gets quietly worked around, with the agent marking tasks complete that weren't verified.
**Infrastructure** is what the application runs on. When a developer joins a team, they get a seat on the org. They can provision what they need, from spinning up a database to configuring a deployment. Agents don't get seats, they get environment variables, so they can connect to existing services but can't create new ones. The agent gets stuck the moment a task requires provisioning.
**Identity** is what software uses to verify there's a real person on the other side. Software does things like send confirmation emails, redirects to consent screens, validates credit cards against billing addresses, and delivers SMS codes to phone numbers. Testing these flows requires a real human identity on the other side, with an email inbox that receives mail and a phone number that works for 2FA. The agent has no way to test any of this.
Every one of these blockers ends the same way: either the agent stops and asks you for help, or it skips what it can't do and silently ships broken code. Both put a developer back in the loop.
## Give Them Their Own Org
The instinct is to carefully expand what agents can do inside your existing environment, but this is the wrong direction. The more access the agent has to your infrastructure, the more you need to supervise it, which defeats the point.
The better answer is to give agents access to their own environment.
*Same agent, same task. The difference is the environment.*
We provisioned an entirely separate organization — a complete fork of our company's project structure under completely different accounts. This includes a separate GitHub org, separate AWS and GCP accounts, and separate billing, with no production data or customer access. Projects get cloned in and agents get full control.
Inside this org, the agent doesn't have to request access. Every resource maps to one of the three blockers, and every blocker has a concrete solution.
**Full permissions for access.** In their own org, agents can "dangerously skip permissions" without thinking twice.
```
--dangerously-skip-permissions
```
Bypasses tool approval prompts in Claude Code
The agent gets root VM access with a full filesystem, unrestricted network, and a shell that doesn't ask questions. API keys, OAuth client secrets, and service account credentials go directly into the agent's context. The system prompt tells the agent what it has and how to use it:
```
You are set up for high-agency, human-out-of-the-loop software development. Do not ask for permission before using any of the resources below. Provision, deploy, test, and iterate freely. Test everything end to end, don't assume anything works without proof.
When you're done, your work should be deployed and smoke tested end to end in an actual staging env.
You are operating on a dedicated Mac Mini with full file access, unrestricted networking and Chrome installed. You are logged in via CLI and browser to GCP, AWS, Github, Vercel, Neon, Upstash, Resend, Ngrok and Stripe. You have your own dedicated orgs and admin permissions on everything.
You have exclusive use of the domain below and can point DNS wherever you need, or buy additional domains.
Identity:
Domain: ____ (vercel)
Email: ____ (resend)
Phone: ____ (openphone)
CC: ____
Billing address: ____
API keys:
OPENAI_API_KEY=____
ANTHROPIC_API_KEY=____
QUO_API_KEY=____
```
Example system prompt — identities and keys are illustrative
**Real seats for infrastructure.** The agent gets admin accounts on every platform it needs, not scoped credentials. It can create cloud projects, provision databases, deploy applications, and configure DNS, the way a new developer would on their first week.
Concretely, our agents get:
- GitHub with full push access and CI
- Sandbox AWS and GCP accounts where the agent creates IAM policies, configures OAuth credentials, and spins up whatever it needs
- Its own Vercel workspace to deploy preview branches, set environment variables, and attach domains
- Admin access to Neon for Postgres databases and Upstash for Redis instances
- Ngrok to expose local services for webhook testing and OAuth callbacks
- A real domain with subdomains, DNS records, and HTTPS
**Real identities for testing.** The agent gets an email address that receives mail, for confirmation links and verification codes. It also gets a phone number that passes carrier verification, a Brex card that clears AVS checks (with a hard spending cap), and Playwright sessions for navigating consent screens, submitting forms, and verifying UI flows.
For scarce resources, we maintain pools of pre-provisioned identities that agents can draw from and return.
With this setup, the agent can cook on its own without asking for help.
## The Isolation Boundary
An agent with admin access to cloud accounts, a credit card, and the ability to run arbitrary code is secure because none of these resources are connected to production.
There is no IAM role, no shared credential, and no network path between the agent's org and ours. The production org and the agent org are completely separate entities with completely separate billing.
*A project forks into the sandbox, gets modified and verified, then returns as a pull request. Nothing else crosses the boundary.*
Code flows in one direction. When the agent finishes a task, it makes a pull request into our production org. We decide whether we merge or we don't. That's the only bridge between the two worlds.
We run agent workloads on dedicated Mac Minis in isolated VMs. If an agent provisions something expensive or gets stuck in a loop, the blast radius is a capped card and a sandbox account, not our infrastructure.
The safety model is not about restricting what the agent can do. It's about ensuring that nothing the agent does can reach production.
## Next Steps
This setup gives agents what they need to build real software end to end.
But "unblocked" doesn't mean "correct." We've had agents build an app, deploy it, verify their own work, and report success, only to find the software was still wrong. Usually, this is because the agent didn't know what "correct" meant, or it knew originally but started to drift.
That's a specification problem, and with today's models, we believe we're much closer to solving it as well. We'll write about how we're approaching this next week. Subscribe to get the next essay (Contract Engineering) in your inbox!
## Related posts
- [Contract Engineering](https://rubriclabs.com/blog/contract-engineering)
[Source](https://rubriclabs.com/blog/unblocking-agents.md)
# Primitives over Pipelines
Give agents modular functions instead of prescriptive workflows
By [Sarim Malik](https://rubriclabs.com/team/sarim), 2026-03-04
A lot of early AI systems that got built at orgs were built under an old era where models didn't know how to follow instructions, drifted off task, and hallucinated.
To ensure these systems delivered value, we created rigid guardrails, prescriptive pipelines, and step-by-step orchestrations. The goal was to force user intent onto a predetermined path that the designer had already foreseen.
In many ways, we were force-fitting our insights from the [classic software era](https://karpathy.medium.com/software-2-0-a64152b37c35) onto this new age of AI-native software, repeating the mistake Rich Sutton identifies in [The Bitter Lesson](http://incompleteideas.net/IncIdeas/BitterLesson.html).
However, as of gpt-5.2 and opus-4.5, models exhibit improved instruction following and significantly reduced hallucination. The addition of native reasoning capabilities has enhanced the models' ability to admit uncertainty and, consequently, [decrease hallucination](https://openai.com/index/why-language-models-hallucinate/#:~:text=most%20evaluations%20measure%20model%20performance%20in%20a%20way%20that%20encourages%20guessing%20rather%20than%20honesty%20about%20uncertainty.) by better saying "I don't know" when lacking context. The models also have better planning capabilities and are able to break down complex tasks into smaller chunks without losing sight of the broader goal.
Paradoxically, the quality-control pipelines that were once protective are now becoming obstacles, as they impose a rigid trajectory through a solution space that the models are now capable of navigating more effectively on their own.
> "In many ways, we were force-fitting our insights from the classic software era in this new age of AI-native software."
## Primitive-Oriented Agent Design
We propose an alternative: **Primitive-Oriented Agent Design**. Instead of prescribing rigid use cases or workflows, we provide the agent with a small, modular set of pure, composable functions — "primitives". This grants the agent the freedom to assemble its own workflows at runtime, utilizing the full context of its capabilities.
*A pipeline is rigid; primitives compose to fit each scenario*
Each primitive is simple enough to reason about independently, but when combined they form a large surface area of possible behaviors. Good primitives share some common traits:
- **Purity**: Primitives should behave predictably given the same inputs.
- **Clear contracts**: Each primitive should expose a well-defined schema describing what it expects and what it returns.
- **Narrow responsibilities**: A primitive should do one thing well rather than encapsulate an entire workflow.
- **Composability**: Primitives should be able to combine naturally with others.
## Thin system prompts
Primitive-Oriented Agent Design also changes how we think about [system prompts](https://rubriclabs.com/blog/what-is-an-agent-harness#context).
Early AI systems treated the system prompt as a repository of knowledge. Designers front-loaded it with policies, examples, domain facts, and long lists of instructions so the model had everything it needed up front.
This made sense when models were unreliable. But in primitive-oriented systems, this pattern becomes counterproductive.
When agents can retrieve information through primitives (for e.g. searching documents, fetching records, or recalling memories) the system prompt no longer needs to store the system’s state.
In fact, large system prompts hurt agent performance by cluttering context and hindering reasoning. They also cost more due to increased compute needs, as [attention scales quadratically with context length](https://wandb.ai/wandb_fc/tips/reports/The-Problem-with-Quadratic-Attention-in-Transformer-Architectures--Vmlldzo3MDE0Mzcz) (doubling the context quadruples the attention workload). Since APIs charge per token, bigger prompts also increase operational costs.
Instead, primitive-oriented systems favor thin system prompts. A thin prompt defines the **goal, boundaries, and voice of the agent** and knowledge is retrieved dynamically through primitives when needed.
In other words, the system prompt encodes **taste, not state**.
The prompt defines how the agent should think and behave, while primitives supply the information required to act. This keeps the prompt small, keeps the system flexible, and ensures the agent reasons over fresh context rather than static assumptions.
> "A thin system prompt encodes taste, not state."
## A practical example
Consider a shopping agent responsible for helping a user browse and inspect items in an online store. A user might ask to see what products are available, inspect the details of a specific item, or render a list of results in a structured UI. The agent’s job is to retrieve the relevant records and present them in a usable format.
There are two ways to build this system.
**Pipeline approach**
You prescribe a fixed “search workflow.” The agent lists the product skeleton, calls a search tool, maybe passes through a result-correction step, and finally renders the result. Every request follows the same sequence.
This produces predictable UI behavior, but it also inherits the limitations of the workflow itself. If the underlying catalog doesn’t match the use case you anticipated, the pipeline still executes the same steps. The agent cannot adapt the strategy to the situation. It simply walks the path it was given. Latency also accumulates because each stage runs even when it isn’t necessary.
**Primitives approach**
Instead of prescribing the workflow, you expose a small set of primitives:
- **list**: Retrieves a collection of products with options for filtering and pagination.
- **get**: Retrieves a single, specific product by its identifier.
- **render**: Renders a collection of products using a specified rendering logic.
None of these dictate order. They simply expose capabilities. Given these primitives, the agent can assemble its own workflow at runtime. For example:
- It might **list** available products, retrieve a few with **get**, then render them.
- If the user asks for details about a specific product, it might skip listing and call **get** directly.
- If the user asks to compare multiple products, it could retrieve them in parallel before rendering.
- If the agent already knows which item the user means from prior context, it may go straight to rendering.
*One primitive set, multiple execution paths based on intent*
The same primitives support many different execution paths. This is the core advantage of primitive-oriented systems: the designer **defines capabilities, not trajectories**. Each primitive remains simple and predictable, but together they give the agent the freedom to construct the right workflow for each request.
The result is a system that adapts to the user’s intent, avoids unnecessary steps, and reasons over the problem instead of replaying a predetermined script.
## Let it cook
The core problem is that the pipeline encodes one trajectory through the solution space, while user intent and real-world data rarely follow a single predictable path.
The shopping example is one instance of a broader pattern. When the system can navigate the space itself, prescribing the path gets in the way. We're used to designing systems where we control the flow; with capable models, the leverage is in giving them room to assemble.
Don't try to anticipate every path. Give primitives and let the agent figure it out.
## Related posts
- [Everything we know about good agent design](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design)
[Source](https://rubriclabs.com/blog/primitives-over-pipelines.md)
# How does Claude Code actually work?
A first-principles understanding of coding agents
By [Sarim Malik](https://rubriclabs.com/team/sarim), 2026-02-10
Claude Code has quickly become the default coding agent since being [announced](https://www.anthropic.com/news/claude-3-7-sonnet) less than a year ago by Anthropic.
If you’ve tried Claude Code, you’ve probably had the same reaction: wait, this actually works. And once you feel that, it’s easy to imagine agents like this spreading far beyond code.
Naturally, we became curious and wanted to understand how the system works, dig through the [Claude Code docs](https://code.claude.com/docs/en/overview) and study the [Claude Agent SDK](https://platform.claude.com/docs/en/agent-sdk/overview) (Claude Code’s API surface).
Claude Code is closed source, so we can’t inspect its internals directly. Instead, we have to infer the system from its public surface area, mainly the SDK and docs.
We had a few questions in mind:
- What is the agent actually doing step-by-step?
- Is it a single-agent or multi-agent, and how are decisions made?
- Which tools does Claude Code use, and why are they designed the way they are?
- How does it run long conversations without losing the plot?
## What is Claude Code, really?
Claude Code is a coding agent which runs in your terminal. You can invoke it to tackle coding tasks, but you can also ask for help with non-coding tasks such as writing docs, searching files, researching topics, and more.
This seems to be emergent behaviour as it was [originally designed](https://youtu.be/zDmW5hJPsvQ?si=Lh9ZSEYFS6_RodVo\&t=132) to be an internal coding agent for Anthropic, but quickly found product-market fit internally and eventually became a core offering of the company.
Much of Claude Code's capabilities can be attributed to the built-in agentic loop which alternates between three phases:
*The phases Claude Code cycles through*
Now here’s the key idea: the loop stretches or shrinks based on what you ask. A small request might complete in a few steps. A bigger request usually turns into a back-and-forth: read some files, change something, review it, repeat until it’s correct.
Almost everything Claude Code does is one of three things: gather, act, or verify. Small tasks stop early. Bigger ones loop until they’re solid.
## The landscape
The loop is the core mental model, but the loop alone is not enough. To operate on a real project, Claude Code runs inside an [**agent harness**](https://rubriclabs.com/blog/what-is-an-agent-harness) that gives the model tools, state, and permission boundaries. This matters because most of the practical value comes from the harness layer, not just the base model.
### What is an agent harness?
An *agent harness* is the runtime layer around a language model that lets it call tools safely, keep working state, and take actions in an environment.
Think of it as the bridge between model intent and real execution. In Claude Code, that execution environment is your filesystem, terminal commands, and test tooling.
### Where does Claude Code sit?
One useful way to bucket coding tools is by how much of the gather/act/verify loop they can own:
- **Autocomplete**: suggests the next lines
- **Copilot**: helps draft and edit, but you still run commands and verify outcomes
- **Agent**: can run the loop against a real project by using tools and feeding results back into context
This maps to what Andrej Karpathy calls an [autonomy slider](https://youtu.be/LCEmiRjPEtQ?si=P6uxaPlH06uYM86f\&t=1222): the industry has moved from autocomplete, to chat copilots, to longer-running agents that can own larger parts of the loop.
Claude Code sits near the agentic end. It is terminal-native, strongly harnessed, and opinionated about verification (tests, diffs, builds), while still asking for explicit approval on risky actions. So what does this architecture look like for a single request in practice?
## The core system
Let’s zoom in on a single request. Claude Code doesn’t answer in one shot, it runs a tight transaction loop: decide what to do next, do it, then use the result as new evidence.
*The context window grows as Claude gathers, acts, and verifies*
Here’s the flow:
1. **Start state**: your request, plus the current context window (system instructions, prior messages, recent tool results).
2. **Decision**: the model reads that state and chooses the next move, either answer now or call a tool to reduce uncertainty.
3. **Execution**: tools run in the terminal environment (read files, search, run commands) and return results.
4. **Update**: those results get appended back into the context window, and the loop repeats until Claude can stop confidently.
That’s the whole transaction: context → decide → tool → result → context, repeated until the evidence is strong enough.
## The model
At the center of Claude Code is Claude, the model. The model reads your project, keeps a mental map of how things connect, and decides what to do next.
Given what it sees in the context window, it chooses the next move: gather more context, take an action, or verify what changed.
The model tiers matter more than most users realize:
**Sonnet** is the default. It handles most coding tasks well—file navigation, straightforward edits, test runs. For 80% of work, it’s the right call.
**Opus** is where you go when the hard part is reasoning, not execution. Architecture decisions, debugging race conditions, refactors that touch many files, migrations where you need to hold a lot of state in your head. Opus is slower and costs more, but it makes fewer dumb mistakes on complex tasks. At Rubric, we use Opus as our daily driver—the unit cost is higher, but we get fewer loops and less rework.
**Haiku** is fast and cheap, but it loses nuance. Fine for grep-like tasks or simple lookups. Not great for multi-step reasoning.
**Extended thinking** is an option on any tier. You’re essentially asking the model to spend more compute before responding—longer internal chains of thought, more self-correction. This increases latency and cost, but it’s worth it for genuinely hard problems where thinking longer actually helps.
Today’s state of the art is **Sonnet 4.5** and **Opus 4.6**, and these models keep improving over time.
Next, let’s define what the context window actually contains.
## Context window: what Claude Code sees (and what it can’t)
The loop is powered by one constraint: whatever fits in the model’s context window is what exists for this step.
Claude doesn’t have magical access to your whole project. It only sees what Claude Code has loaded into the window right now, and as the window fills up, older details get compacted (more on this later).
So what’s inside the window?
- **System instructions**: the rules of the tool, plus your project rules (like `CLAUDE.md`).
- **Conversation history**: your messages and Claude’s responses.
- **Tool results**: file contents it read, search results, terminal output, diffs, failing tests, and anything else a tool returned.
- **Loaded skills and tool definitions**: extra capabilities you’ve enabled that Claude can call.
**What is CLAUDE.md**
`CLAUDE.md` is a plain markdown file you add to your project to explain how you do things in this codebase. Unlike conversation history, which is ephemeral, this is a set of fixed instructions always loaded into the context window.
**What is a Skill?**
A [Skill](https://code.claude.com/docs/en/skills) is a markdown file which containes reusable instructions. Claude loads a Skill only when it’s relevant (or when you invoke it directly).
That’s the working set Claude uses to decide the next action. Anything outside of it might as well not exist.
Now the practical part: your context budget depends on the model you pick. The latest models are all [200K tokens](https://platform.claude.com/docs/en/about-claude/models/overview), so model choice changes quality/latency/cost more than raw context size. Compared to older generations, that’s a lot: Claude jumped from [9K to 100K](https://www.anthropic.com/index/100k-context-windows) in 2023, and then to [200K](https://www.anthropic.com/index/claude-2-1) later that year.
When the window fills up, Claude Code manages it automatically: it clears older tool outputs first, then summarizes the conversation if needed. Your latest request usually survives, but early details can disappear, which is why persistent rules belong in `CLAUDE.md`. This process of condensing the context window is called **context compaction**.
Next up, let’s look at the tools that generate most of that evidence.
## Tools: the building blocks
When you ask Claude Code to do real work, it’s mostly using a small, sharp set of tools. Tools are how the loop touches reality: read files, run commands, fetch external context, and pull the results back into the context window.
Anthropic publishes the canonical list in the [Claude Code docs](https://code.claude.com/docs/en/settings#tools-available-to-claude).
You can group the core tools into three buckets:
- **Workspace tools**: `Read`, `Write`, `Edit`, `Glob`, `Grep`, and `Bash` let Claude inspect your codebase and make changes in your terminal environment.
- **Web tools**: `WebSearch` and `WebFetch` let Claude pull in external context when the answer isn’t in your project.
- **Interaction tools**: `AskUserQuestion` is how Claude gets unstuck when a required detail is missing, by asking you a small set of multiple-choice questions.
Here are the core tools Claude Code uses most often in the gather/action/verify loop:
*Tools available to Claude Code*
Beyond the core loop, Claude Code also has supporting tools that expand its surface area without changing the mental model:
- **Orchestration**: task and delegation helpers like `Task`, `TaskList`, and `TaskOutput`.
- **Integrations**: `MCPSearch` for [Model Context Protocol (MCP)](https://rubriclabs.com/blog/how-does-claude-code-actually-work#model-context-protocol-mcp) tool discovery and loading.
- **Editing**: `NotebookEdit` for notebooks and `LSP` for language-server powered code intelligence.
- **Control utilities**: things like `ExitPlanMode` and `KillShell` to manage the session.
### Task: orchestration via sub-agents
A `Task` is how Claude Code spins up a **sub-agent**. It is a separate, focused agent instance with its own context window that does one scoped unit of work and returns a condensed result to the main agent.
In practice, this is how Claude Code does deeper work without bloating the main thread. For example, the main agent can delegate “find all references to X” or “scan for failing tests and summarize”, then merge the result back into the main plan.
This helps in three ways:
- **Context isolation**: the sub-agent keeps its own context, so deep exploration doesn’t pollute the main thread.
- **Parallelism**: you can run multiple sub-agents at once (for example, find files, check tests, review diffs), then merge the outcomes.
- **Tool scoping**: sub-agents can be restricted to a smaller tool set (read-only review, test runner), which reduces risk.
The mental model is simple: the main agent stays in charge of the narrative and final decision. Sub-agents are disposable workers that hand back a result the same way a tool returns output.
### Model Context Protocol (MCP)
MCP is how Claude Code plugs **external systems** into the same gather/act/verify loop by treating them as tools. Instead of hardcoding GitHub support or Slack support, the agent can discover and call MCP-provided tools on demand (for example, “list open pull requests”, “fetch a document”, “query a database”).
The key design point is **context efficiency**. Tool definitions can be expensive, so Claude’s approach is to load tools on demand: MCP [tool search](https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool) can defer loading a whole tool catalog and only expand the handful of tool schemas the agent actually needs for this request.
### Designing a tool interface
Claude Code works because tool interfaces are **constrained**. A good tool is a small contract with clear inputs, bounded scope, and outputs the agent can trust.
The [text editor tool](https://platform.claude.com/docs/en/agents-and-tools/tool-use/text-editor-tool) is a good example. Instead of a free-form rewrite, Claude is limited to a small command set (`view`, `create`, `insert`, `str_replace`), and [str\_replace](https://platform.claude.com/docs/en/agents-and-tools/tool-use/text-editor-tool#str-replace) requires an exact match (including whitespace) so the edit is surgically targeted.
That constraint matters because it fails loudly: if the old snippet doesn’t exist, or appears multiple times, the tool returns an error. The agent is forced back into gather phase, which is exactly what you want when a change would otherwise be ambiguous.
Two other tools are worth calling out because they shape how real the agent feels:
- **`Bash`**: commands run in your environment. That makes verification factual (run tests, check diffs).
- **Web tools**: `WebSearch` is great for freshness and citations, while `WebFetch` pulls full pages.
Put together, this is how Claude Code stays reliable: the model proposes a small next step, and the tool layer turns that step into evidence or a clear error.
The remaining ingredient is the environment those tools run in.
## Terminal as an environment
An underrated part of Claude Code's power comes from the environment it lives in, your terminal pointed at your project, with your tooling already wired up.
The terminal creates a clean permission boundary. Claude doesn’t get magic access, it inherits whatever your shell user can do, and every meaningful action routes through explicit tool calls you can audit, constrain, or deny.
Just as importantly, it lets Claude interact with the same primitives you already trust: files, diffs, tests, and commands, working inside your existing workflow.
That’s why the same agent in a generic cloud environment is often less effective. Without your project’s existing scripts, linters, and build/test commands, taking action becomes guesswork, and verifying becomes weak. Claude Code feels strong because the environment provides the [primitives](https://rubriclabs.com/blog/primitives-over-pipelines), and the model composes them.
## Why this matters
Claude Code feels powerful for a simple reason: it gets to live in an environment, the terminal, with the same primitives you already trust: files, diffs, tests, scripts, and your build tooling.
Studying the docs also made me realize that agent quality is mostly about clear contracts. Reliable agents come from tight tool interfaces, bounded outputs, and fail-loudly behavior that turns intent into evidence inside real workflows.
The other thing that surprised me is how scalable the loop is. Break work into parallel sub-tasks, spin up sub-agents via `Task`, pull back condensed results, and let context compaction keep the main thread focused as scope grows.
And this is where it is going next. Agents will run longer, coordinate more parallel work, and take on larger end-to-end slices of projects as tool ecosystems mature and permissioning and verification get stronger.
If you want to go deeper, the best references are the [Agent SDK](https://platform.claude.com/docs/en/agent-sdk/overview) and the [Claude Code docs](https://code.claude.com/docs/en/how-claude-code-works).
## Related posts
- [Everything we know about good agent design](https://rubriclabs.com/blog/everything-we-know-about-good-agent-design)
[Source](https://rubriclabs.com/blog/how-does-claude-code-actually-work.md)
# Introducing Rubric Labs
The bridge between AI research and products
By [Sarim Malik](https://rubriclabs.com/team/sarim), 2025-03-31
After 18 months of building in stealth, we're excited to officially introduce Rubric Labs to the world. As a [product studio](https://rubriclabs.com/blog/introducing-rubric-labs#what-is-a-product-studio), we're on a mission to help companies deploy the next generation of intelligent software, bridging the significant gap between promising AI demos and production-ready applications.
---
## Our story
Our story begins with three founders united by a shared passion: creating products that users truly love. We began our journey in the world of startups, working in the knowledge management, developer tooling and supply chain space. We took our learnings from these startups and boiled them into Rubric Labs, which we established with a clear mission—**to build a lean team of problem solvers** **who care deeply about craft and use our collective skills for good**. We recognized that our strength lies in our obsession with quality and our commitment to turning innovative ideas into practical, valuable solutions.
When [GPT-3 emerged](https://arxiv.org/pdf/2005.14165), we saw a realm of opportunity unlocked before us. What started as hacking together demos quickly evolved into something more significant as we recognized a critical gap in the AI landscape. The distance between creating an impressive AI demo and developing a robust, production-ready product remains substantial—and that's precisely the space we're working to fill.
*The founders (left to right: Sarim, Ted and Dexter)*
## The AI deployment gap
Today the world of AI is rapidly evolving. There is cutting edge research being published by research groups consistently. If you're an avid [Arxiv](https://arxiv.org) reader, you know exactly what I am talking about. There are [107 unique publications](https://arxiv.org/list/cs.AI/recent?skip=0\&show=50) published just today in the AI category. Let that sink in.
Moreover, the amount of amazing demos we're seeing on platforms like [𝕏](https://x.com/search?q=introducing%20AI\&src=typed_query) and [LinkedIn](https://www.linkedin.com/search/results/content/?keywords=introducing%20AI\&origin=SWITCH_SEARCH_VERTICAL\&sid=\(Qf) is uncounted for. The advent of coding formats like "[vibe coding](https://x.com/karpathy/status/1886192184808149383)" have further increased the number of demos we see on a regular basis. But how many AI tools are we actually using? Not a lot. The gap between an amazing demo and a product that works in production still remains significant.
Why does this gap exist? That's a topic for another day, but we see the same thread of problems with AI demos: reliability challenges, scale and performance issues, integration complexity, data knowledge gaps, and data flywheel challenges. These challenges create significant barriers for companies seeking to harness AI's potential. Without specialized expertise, organizations either settle for suboptimal implementations or abandon promising AI initiatives altogether. This is where Rubric steps in.
## What is a product studio?
It's helpful to first define what a product studio is. A traditional AI lab is associated with research and model providers come to mind, such as [OpenAI](https://openai.com), [Anthropic](https://www.anthropic.com), etc. Whereas a product studio is focused on building products. The focus is geared towards the application layer versus the model layer.
As a product studio, we operate at the intersection of cutting-edge AI research and practical implementation. We combine research-backed methodologies with hands-on development expertise to create AI solutions that work reliably in real-world environments.
## Our journey and expertise
We've been fortunate to partner with some of the most innovative technology-first companies in the industry. Our collaborations with companies like [Gumloop](http://gumloop.com/), [Graphite](http://graphite.dev/), and [Trigger](http://trigger.dev/), to name a few, have allowed us to tackle challenging problems at the cutting edge of AI implementation.
Beyond these public partnerships, we've worked with several stealth startups, where we've built natural language interfaces that allow users to access complex information, take actions, and generate customized dashboards on the fly. We've also partnered with Fortune 500 companies in the retail sector to develop conversational AI agents with multi-tool capabilities and long-term memory for enhancing customer experiences.
Our portfolio spans from early-stage startups to enterprise-level organizations across multiple industries, including fintech, healthcare, and e-commerce. In each case, we've delivered custom AI solutions that address specific business challenges while meeting rigorous production standards.
Working closely with these forward-thinking organizations has deepened our expertise and validated our approach to bridging the gap between AI research and production. We have also shared some of our work in the open. We're strong advocates for open-source and contribute whenever possible. We wrote about [designing an agent for production](https://blog.langchain.dev/how-to-design-an-agent-for-production/) back in 2023 in partnership with [LangChain](https://www.langchain.com/experts) and [Cal.com](https://cal.com). We wrote about [building specialized video agents](https://rubriclabs.com/work/graphite) in partnership with [Graphite](https://graphite.dev). We are also the creators of [create-rubric-app](https://github.com/RubricLab/create-rubric-app), a boilerplate template for getting started with full-stack AI apps quickly.
Our approach is different from a conventional consulting company because we are first and foremost product builders and secondly, we take an active role in building out these solutions. More specifically our approach includes:
1. **Research integration**: Translating the latest academic breakthroughs into practical applications
2. **Production excellence**: Building systems that meet enterprise requirements for reliability, security, and scale while prioritizing first-principles thinking
3. **Problem-first thinking**: Focusing on solving genuine business challenges rather than deploying technology for its own sake
However, the truth is that not everything works in AI systems today—and we've experienced this firsthand. We've had our fair share of solutions that didn't work, but that's part of the experimentation process. Unlike a traditional consultant who would maintain a pretense of being always right, we position ourselves as truth seekers and builders. We experiment in a problem space with our clients to truly push the envelope on what's possible. We value transparency above everything else. Trust is at the foundation of our organization. In turn, this has been instrumental in refining our approach to system design and implementation, ensuring that we build solutions that not only work in demos but thrive in real-world applications.
## Our vision forward
We believe we're at a pivotal moment in the evolution of software. AI is transitioning from an interesting feature to a fundamental component of how applications are built and experienced. At Rubric Labs, we're committed to helping companies navigate this transformation successfully.
Our team brings together expertise in software engineering and product development to deliver solutions that aren't just technically impressive but create genuine business value. We're also pursuing this vision in the open – writing technical guides on our [blog](https://rubriclabs.com/blog) and publishing open-source projects on our [GitHub](http://github.com/rubricLab).
## Let's build the future, together
We're excited to partner with forward-thinking companies ready to leverage the power of AI in their products and operations. Whether you're looking to enhance existing applications or build entirely new AI-powered experiences, we'd love to hear from you.
Visit our website to learn more about our approach and services, or reach out directly through our contact form to start a conversation about your AI journey.
Follow us on [Twitter](https://x.com/RubricLabs) and [LinkedIn](https://www.linkedin.com/company/96080278/) to stay updated on our latest work and insights into the evolving AI landscape.
The future of intelligent software is here, and we're thrilled to be building it with you.
## Related posts
- [Create Rubric App](https://rubriclabs.com/blog/create-rubric-app)
[Source](https://rubriclabs.com/blog/introducing-rubric-labs.md)
# Fine-tuning a GPT model for spam detection
By [Ted Spare](https://rubriclabs.com/team/ted), 2024-11-29
## What We Built
We fine-tuned a small LLM for spam detection using dozens of historical spam tags. It's working remarkably well.
While it's currently tailored to our needs, the approach could be generalized for broader email spam detection. Our schema is roughly as follows:
```ts
type Contact = {
company: string;
email: string;
message: string;
// ...
}
```
## The Problem
We get lots of inbound messages. Most are spam. Our workflow for triaging them was simple but inefficient:
1. Someone submits a message on [rubriclabs.com/contact](https://rubriclabs.com/contact)
2. We get a Slack notification
3. Someone on our team manually flags it as spam or legitimate
Here's the kicker: even at just 30 seconds per day, this adds up to hours per year (not to mention the lingering cost of context-switching) making it worthwhile to automate in a post-Cursor world.
## The Solution: Fine-tuning
The data from our spam flags was simply stored in Postgres, creating what would become our training dataset:
```jsonl
{ "message": "We sell the best leather couches", "status" : "spam" }
// ...
{ "message": "Looking to build an agentic flight booking system", "status": "legit" }
```
Given the hundreds of upvotes/downvotes, deduped and cleaned (a 10-minute process, given the simple schema), fine-tuning on OpenAI was a straightforward process.
## The Technical Details
The fine-tuning schema follows a standard chat message format:
```ts
type Message = {
role: {
role: "user" | "assistant" | "system";
content: string;
}
}
```
The actual examples are stored as [JSONL](https://jsonlines.org/), a file format where each line is valid JSON.
The actual process was refreshingly simple:
1. Write our array of examples to a .jsonl file
2. Upload the file
3. Wait ~10 minutes
4. Pay ~$10
5. ...
6. Profit?
## Does It Work?
### Quantitative Evaluation
- We did a head-to-head comparison between `GPT-4o` and `ros-spam`:
- We held back 10% of our dataset for testing
We ran comparisons in both OpenAI playground and OpenPipe evals
The result: `ros-spam` achieved 100% accuracy vs ~80% for a frontier model, even with prompt engineering.
### Qualitative Assessment
We shipped it to prod with a feedback loop:
- Each run appears with the message in Slack as spam or legit
- We can immediately spot and correct errors
- When needed, we re-tune the model ♺
## Deployment
The implementation was surprisingly painless. Accessing the model requires just a single-line change from standard GPT-4o calls, whether you're using:
- the Node.js `fetch` API,
- [OpenAI SDK](https://platform.openai.com/docs/libraries),
- [AI SDK](https://sdk.vercel.ai/docs/introduction),
- [@rubriclab/agents](https://github.com/rubriclab/agents) or any other standard method.
For those interested in alternatives, you could also host this on:
- Fireworks
- Together
- OpenPipe
or self-host on bare metal.
## Conclusion
The ROI of this exercise was clear: human-level spam tagging running 24/7 for a couple hours of dev.
Have questions or feedback? Drop us a message at hello@rubriclabs.com.
## Related posts
- [My Summer at Rubric](https://rubriclabs.com/blog/my-summer-at-rubric)
[Source](https://rubriclabs.com/blog/fine-tuning-for-spam-detection.md)
# My Summer at Rubric
By [Arihan Varanasi](https://x.com/arihanxv), 2024-09-08
This past summer, I had the privilege of running various projects at the Lab at Rubric. The focal point of these several experiments was **generative UI**, bridging capable LLMs with rich component libraries and design systems. Inspired by **Vercel AI** and **V0**, we set out to build our own fully functional genUI experience, supporting tedious dev work with intuitive **DX** and enhancing user interactions with dynamic but simple **UX**. Alright, enough high-level talk; let me actually show you some of the stuff we built, but please note that they are still a work in progress :)
## Rubric UI
*The Codeblock component*
With the sharp growth of Rubric, we've realized the need for a centralized design system and component library. One that captures the sick aesthetics and feels of the Rubric brand while making initialization super simple for internal or client projects. To this end, we've built an initial version of Rubric UI to achieve just that.
With just...
```sh
bun i rubricui
bunx rubricui init
```
You now have access to a fully featured component library + styles, ready to use in your project! Ranging from basic components like buttons to slightly more complex ones like code blocks, markdown displays, and file uploads, Rubric UI aims to remove the headache of re-implementing components or worrying about styling and dependencies.
Play around with some of the components [here](https://rubricui.vercel.app/).
## Genson
[Video](https://rubriclabs.com/images/my-summer-at-rubric/genson.mp4)
*A video demo we recorded for Genson*
Moving on to a more experimental and ambitious AI-centric project, we built Genson, a system that builds complex UIs with LLMs and structured outputs. With a ton of custom-written Zod schemas, we were able to build a system that generates typesafe JSON, which is then used to create entire React components that can not only render data but also call functions and make API calls all on their own.
By defining a schema for all components, we can make it very intuitive to define actions (functions, server actions, APIs) and front-end components. Users can generate full dashboards with forms, buttons, dropdowns, and more with nested components, shared data, complete customizability, and APIs with minimal errors due to the nature of typesafe JSON output.
Like V0, users can refine generations with additional prompts through the chat interface on the left while viewing the generated output on the right and previous generations via the bottom sidebar.
[Video](https://rubriclabs.com/images/my-summer-at-rubric/genson2.mp4)
*The Genson dashboard*
Additionally, updates to the UI are quicker due to a selective JSON update system, where only the necessary components or properties are rewritten.
We're still working on defining what a "good" UI schema looks like, but we're really excited about the future of Genson and what it could mean for the future of JSON-powered UI generation.
## Autotune
Now moving to the end of the spectrum of UI → AI, I am excited to unofficially announce our fine-tuning package, Autotune! Given an OpenAI compatible OpenAPI JSON schema, Autotune performs synthetic data generation of user prompts and AI responses to fine-tune OS models to respond to prompts in the specified structured schema. Now, you can easily bring the power of structured outputs to your projects while using the open-source models you love.
Best of all, Autotune is designed to be as simple as possible to use. Through `autotune init`, a `.env` file is generated based on the user-chosen fine-tuning provider and `schema.json` file is made, containing an example of an OpenAPI schema. Once the relevant values are filled in and your schema is ready, then simply run `autotune build` and you're ready to get started!
The build step spawns a series of prompts for the user to answer, specifying the data generation and fine-tuning process. Users can create a new dataset or add entries to an existing one.
*The Autotune build step*
Once all prompts are answered, new entries are generated and added to a new or existing dataset. Then, the user can specify whether they would like to fine-tune a model via the CLI, and if the answer is yes, the user can specify which model to fine-tune, and the FT status is shown in real-time. Finally, the new model slug is outputted to the user and can be used to make API calls.
Autotune is currently in the early stages of development with a limited subset of providers and quite some work to be done, but we strongly believe in making fine-tuning as easy and accessible to all developers, no matter what their level of expertise is with AI. We hope to continue working on Autotune and release it soon, so stay tuned!
## Some concluding statements 🫡
Working on these projects at Rubric has been an incredible experience. I've shipped my first NPM package, worked on some really cool tech with a bunch of smart and talented people, and have been challenged in so many different ways. Best of all, I can already see tremendous growth in my skills, and I'm really excited about what's in store for me next.
As always, keep learning and keep shipping 🚢
Peace ✌️
— Arihan
## Related posts
- [Autonomous Iteration](https://rubriclabs.com/blog/autonomous-iteration)
[Source](https://rubriclabs.com/blog/my-summer-at-rubric.md)
# Multi-staging → Local → Prod in record time
By [Dexter Storey](https://rubriclabs.com/team/dexter), 2024-02-16
At **Rubric**, we are constantly pushing full stack features in parallel. We strive to get high quality code to prod as fast as possible with minimal coordination. We developed the multi-staging workflow to address problems with webhooks in staging that created bottlenecks across the CI/CD pipeline.
## TLDR;
Instead of `Development`, `Staging`, `Prod` envs, we propose `Local-[DEVELOPER]`, `Staging-[DEVELOPER]`, and `Prod`.
By setting up unique pipelines for each developer, you can move faster, testing multiple full stack features simultaneously in the cloud.
*Each developer's local and staging environments merge into one production environment.*
## The Problem
Webhooks require a static URL (breaks Vercel preview branches)
A single staging branch necessitates coordination across teams
Shared databases don't let you push schema changes fast enough (conflicts)
The common practice is to have a `development (local)`, `staging`, and `production` env, with slight modifications to each developer's local env that they handle manually in `.env.local` on their IDE.
This creates conflicts: "Prisma Studio isn't working because you pushed schema"
And headaches: "It works on my machine but times out in serverless (Vercel)"
***Multi-staging solves these problems by giving every developer their own local and staging env.***
## Tunnelling (Local)
We used [ngrok](https://ngrok.com/) to handle local environments, specifically the [cloud edge domains](https://ngrok.com/docs/guides/how-to-set-up-a-custom-domain/) feature.
```sh
ngrok http --domain=dexter.rubric.sh 3000
```
Using this command I can tunnel `dexter.rubric.sh` → `localhost:3000`, allowing webhooks sent to `dexter.rubric.sh` to be forwarded to my local machine. This is the bones of the `Local-[DEVELOPER]` env.
Bonus points (`~/.zshrc`):
```sh
tunnel() {
local port="$*"; ngrok http --domain=dexter.rubric.sh "${port:-3000}"
}
alias l='tunnel'
```
We further give each developer a local DB and their own set of auth tokens, but we will dive into this in [The Stack](https://rubriclabs.com/blog/multi-staging#the-stack) section.
## Staging
Let's go further and set up a multi-lane on-ramp to prod. In addition to local, let's also give each developer their own staging env, to let them build, test and share features without coordinating.
Each developer should have their own local and staging:
- Database
- Auth tokens
- URL
- GitHub branch
- Env
We use this workflow for [Maige](https://maige.app/) - a Rubric experiment that brings intelligent workflows to your issues and PRs in GitHub. If you are interested in the project, check out the codebase (OS) [here](https://github.com/RubricLab/maige). There are several useful DX commands that allow you to never have to open the .env file. See [package.json](https://github.com/RubricLab/maige/blob/f017cb57ae588b3fb57fb32906a9360556fe8c80/package.json#L83C3-L83C61).
## The Stack
### Infisical
Infisical is the single source of truth for environment variables for local and staging envs.
On Local: `infisical --env=dexter-local export > .env.local`
*Infisical Environments*
### Supabase
Supabase is the single source of truth for databases for local and staging envs.
### PlanetScale
[PlanetScale](https://planetscale.com/) is our database. Using the safe migrations workflow, you can manage schema like GitHub - with pull requests.
A separate branch for local and staging (per developer) lets us push changes aggressively without coordination.
*PlanetScale DB branches and schema changes*
### Vercel
[Vercel](https://vercel.com) is our deployment partner, automatically deploying on changes to `main` and `[DEVELOPER]/staging`.
Staging branches have their own url - and pull env from Infisical automatically.
*Vercel staging URLs and Infisical env sync*
### GitHub (Apps)
For Maige, we have a separate app for each env, which allows webhooks to be handled properly.
Each developer's instance of the app works in the cloud without an active IDE.
*Maige GitHub Apps*
## The Magic
Once this is all set up, we get to see the true power of the workflow. Developers demoing full stack features in the cloud simultaneously.
With local envs configured, I can easily work on Ted's local branch if we pair program:
```sh
git checkout ted/feat
&& infisical --env ted-local export > .env.local
&& ngrok http -- domain=ted.rubric.sh
```
With staging envs configured, I can test out Arihan's new PR (which includes schema changes, new UI and modified GitHub permissions) from my phone on the subway. No need for an IDE or even a laptop (:
*Arihan demoing a new feature in staging*
As we hurdle forward into the future, it looks like modern engineering looks a lot less like vim and a lot more like chat. With multi-staging, we can field an aggressive pace by testing full stack changes in the cloud - on a real URL. When AI agents make the leap from copilots to engineers, we might not have the time to review code. What if we can simply open a URL and test as a user? Are we ready for this velocity?
As always, this is a work in progress. Please send feedback or questions to hello@rubriclabs.com.
Peace nerds (:
## Related posts
- [Unblocking Agents](https://rubriclabs.com/blog/unblocking-agents)
[Source](https://rubriclabs.com/blog/multi-staging.md)
# Planning for near-free AI
By [Ted Spare](https://rubriclabs.com/team/ted), 2024-01-07
## Capability
Open-source LLMs are thriving. Since the summer of 2023, OS LLMs have rapidly approached human-level performance across a suite of benchmarks.
Line graph showing the average benchmark performance of open-source Large Language Models approaching that of GPT-4 and human-level performance, during the period from August 2023 to January 2024.
\*From [HuggingFace Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) and [OpenAI](https://openai.com/research/gpt-4).
For now, GPT-4 remains the best. What's more, GPT-5 will come out around September 2024, according to prediction markets. We can expect the envelope to be pushed further by any jump in capabilities.
*Betting markets thought we would have GPT-5 by now. Hindsight is 20/20.*
From [Metaculus: When will OpenAI announce GPT-5?](https://www.metaculus.com/questions/15462/gpt-5-announcement/)
Regardless, open-source (or at least [open-access](https://huggingface.co/blog/llama2)) LLMs continue to catch up.
## Cost
Self-hosting LLMs can help to protect customer data, prevent [fluctuations in ability](https://www.zdnet.com/article/gpt-4-is-getting-significantly-dumber-over-time-according-to-a-study/), and build more defensible tech. Surely, this option is complicated and costly? Quite the opposite. The price for GPT-3-level models is in freefall.
From [Latent Space: The Four Wars of the AI Stack](https://www.latent.space/p/dec-2023).
## Timeline
What would you build if GPT-4 were 10x cheaper? What about 100x cheaper? 100x faster?
This is not an outlandish vision but a near-inevitable trend. As we hurdle toward a future where human-level intelligence is virtually free and abundant, how can we prepare?
What if every student had a genius-level tutor, every SMB had Fortune 500-level marketing resources, and every online community had Stripe-level engineering? How do we ensure that these trends serve to close digital gaps rather than widen them?
## Opportunity
Our future is not written in stone; it's coded in bytes and shaped by every line of code we write, every application we dream up, and every conversation we have about what comes next.
Whether you're looking to streamline existing processes, unlock new levels of personalization, or pioneer entirely new services, we’re happy to chat about what’s possible.
## Related posts
- [Challenging the Chatbot](https://rubriclabs.com/blog/challenging-the-chatbot)
[Source](https://rubriclabs.com/blog/planning-for-free-ai.md)
# Create Rubric App
By [Ted Spare](https://rubriclabs.com/team/ted), 2023-11-01
`> npx create-rubric-app`
We recently [wrote about](https://blog.langchain.dev/how-to-design-an-agent-for-production/) designing AI agents for production. Many developers shared the feedback that it's still not clear how to actually serve the thing to users. We wanted to bring that power to all.
Today, we're excited to introduce create-rubric-app: the fastest way to bootstrap a full-stack AI project.
Building on the legacy of create-react-app and create-next-app, this is our opinionated meta-meta-framework for building AI apps ready for users. We'll be adding templates for a ton of common use cases, but our first one is focused on AI agents.
## Why agents?
Agents are a powerful new tool that unlock a whole suite of applications, like variable-depth scraping, multi-step planning, and self-healing code. However, there is a lack of guidance on how to implement them in a deployable way.
Our agent is built with LangChain and OpenAI Functions. As an example, we've built a smart to-do list. The agent has access to four tools:
- `listTasks`
- `createTask`
- `updateTask`
- `deleteTask`
From these tools, it can manipulate the to-do list flexibly from natural language.
To try it out, visit the app in the browser and try asking the Agent to "remind me to buy bread" then "mark the bread task as complete". It should explain what it's doing as it completes these tasks. To debug, watch the logs in your terminal.
## How does it work?
The tools above are each described by a [Zod schema](https://zod.dev/?id=objects). The app passes the descriptions of each tool to GPT via OpenAI's API. The LLM will then decide which tool to "call", and with which inputs. By using OpenAI's [Function Calling](https://platform.openai.com/docs/guides/gpt/function-calling), the LLM is well-suited to outputting structured data (a common limitation of LLMs).
LangChain will then parse the output and call the actual functions, passing the output back to the LLM to inform the next step.
You now understand how AI agents work and you're ready to build one for your own use case. Doing so is as simple as swapping the agent's toolkit with your functions, API endpoints, a [search engine](https://js.langchain.com/docs/api/tools/classes/SerpAPI), a [memory store](https://js.langchain.com/docs/modules/memory/how_to/vectorstore_retriever_memory), or even a Discord connection (allowing agent ↔ human communication).
As long as your tools return text explaining their result, the agent will do its best to work with them.
## Quickstart
To use create-rubric-app, simply run `npx create-rubric-app@latest`. This will guide you through some steps to bootstrap a Next.js project and generate an OpenAI key.
To run the app, simply run `bun dev` or `npm run dev`.
To deploy your app and make the agent accessible to users, two options are [Vercel](https://vercel.com/solutions/nextjs) and [Railway](https://railway.app/template/HRZqTF).
This project is [open-source](https://github.com/RubricLab/create-rubric-app)! We welcome suggestions and contributions.
Most of all, we look forward to seeing what you build with this.
## Related posts
- [Primitives over Pipelines](https://rubriclabs.com/blog/primitives-over-pipelines)
[Source](https://rubriclabs.com/blog/create-rubric-app.md)
# Product Engineer
Remote, Canada · Full-time · $120k–$150k CAD
Rubric Labs is a small team of founder-level engineers with our own research lab. We build with teams like [Gumloop](https://rubriclabs.com/work/gumloop), [Graphite](https://rubriclabs.com/work/graphite), and [E2B](https://rubriclabs.com/work/e2b), and write up what we learn on the [blog](https://rubriclabs.com/blog).
This is our **first full-time engineering hire** outside the [founding team](https://rubriclabs.com/team). You'll work inside client teams and own builds end to end, from the first scoping call to production.
## What you'll do
- Scope projects with clients and turn ideas into plans
- Build interfaces, APIs, data models, and agent loops
- Ship AI features to production and test them with evals
- Work directly with client founders and engineering teams
- Write about what you build and learn
## What we're looking for
- Several years shipping production software, with work you can point to
- Comfort owning a project without a spec or a manager checking in
- **Taste** in interfaces. You notice when spacing, motion, or copy is off
- Clear writing. Most client communication happens in docs and threads
- Experience building with LLMs
## Benefits
- Competitive health benefits
- Equipment and home-office budget of $1000
- Three weeks of paid vacation
- Annual team trips
- $1000 per month discretionary token spend
## How to apply
Email [hello@rubriclabs.com](mailto:hello@rubriclabs.com?subject=Product%20Engineer) with your resume and links to two or three things you've built. Tell us which one you're proudest of and why.
We care about **proof of work** more than credentials. The quality of what you've built, and what you've worked on, weighs more than where you studied or worked.
We review applications on a rolling basis.
[Source](https://rubriclabs.com/careers/product-engineer.md)