Giving coding agents whole-repository maps

We tested three specialized code-mapping methods with a coding agent. Performance improved but at the cost of more tokens and time. Specialized agent harnesses can help, though not for free.

Why this? Why now?

LLMs have excelled at software engineering tasks.

On DeepSWE, a representational benchmark of software tasks, overall completion rates are trending toward 80%. With that said, the non-deterministic nature of LLMs means success rates can vary wildly across runs.

Modern coding agents typically manage codebase context with a combination of heuristics and general-purpose shell commands like grep and find. To this end, we investigate whether specialized artifacts affect agents' abilities to navigate unfamiliar codebases, make their first edit and, if so, is the benefit enough to offset the added context overhead?

Existing work includes RepoBench, which evaluates repository-level retrieval and code completion using cross-file context. We test something different:

Which representation, if any, is most useful? To this end, we measure

• task success,
• token usage and cost,
• runtime,
• time to first edit,
• repository-navigation behavior,
• artifact usage, and
• context-management overhead.

Experimental Setup

Using Tree-sitter, we built Cartograph, a tool to parse code in a variety of programming languages into a unified, canonical structure covering symbols, modules, and relationships such as

• calls,
• imports,
• instantiation,
• inheritance,
• implementation, and
• references.

Currently, it supports ingesting TypeScript, JavaScript, Python, Go, and Rust.

Cartograph represents codebases via one of three methods we've named Architecture, Skeleton, or Callgraph.

The tool stores these representations as Markdown files. We include ambiguous relationships and parsing failures as additional information for the agent.

Below you'll see how each output might appear in practice:

Actual artifacts differ in the information they surface and how that information is exposed.

Architecture and Skeleton provide more human-centric representations of the codebase, focusing on where things are and what exists. Callgraph focuses on how symbols interact, such as which functions call which and the execution path between them.

We intentionally minimize overlap between these artifact types, which makes differences in results easier to attribute to the information each representation provides.

Experiment

To compare how agents complete real, long-horizon software engineering work, we ran a subset of DeepSWE v1.1 tasks three times each. We tracked task correctness alongside tokens spent, runtime, source access, and navigation behavior (ie. which files the agent accessed).

We fixed the model (GPT-5.6-Luna), provider (OpenAI), harness (Pi), thinking level (Max), repository version, and task prompt across conditions. Every condition also retained access to the complete repository source; the only experimental variable was which structural artifact, if any, the agent could access.

We chose 5.6 Luna because it sits close to the efficiency frontier on DeepSWE tasks. Pi is deliberately a simple coding harness: it gives the model the filesystem and editing capabilities needed to complete the task with minimal overhead. This combo of model and harness is also close to coding agents used in practice.

Agents worked inside disposable sandboxes, with held-out verification performed separately after each run.

Artifacts were exposed as ordinary files, with a pointer added to the system prompt:

The following generated repository-structure aids are available:
 
- Architecture: `.cartograph/architecture.md` is available.
 
Use the available repository information as you judge useful for completing the task.

Agents could inspect the artifact using the same filesystem and CLI tools available for normal source-code navigation, making artifact discovery and usage part of the experiment itself.

Before the full evaluation, we ran a small set of canary tests using a cheaper model to validate the setup and to estimate cost.

Each run had a 90-minute timeout. When the timeout was reached, execution stopped but the verifier still graded the workspace in its current state, meaning a timed-out trial could pass if the agent had already produced a correct solution.

The results should be interpreted as specific to our model-harness pair. Repeating the experiment with other models and harnesses would be needed to establish generality.

Stay in touch

Join our mailing list to receive future posts straight in your inbox.

Results

We summarize the main results across 359 trials below.

Resource usage for each of the conditions can be found below.

Across the first 30 tasks, the baseline is a mean pass rate of 37.8%, compared with 36.7% for Architecture, 40.0% for Skeleton, and 37.8% for Callgraph. Skeleton is nominally the best condition.

Of the 359 graded trials, 216 completed, 134 hit the 90-minute cap, and 9 failed. We exclude runs that failed due to infrastructure issues.

The clearest positive result is time-to-first-read of a file touched by the reference patch, which fell from a median of 10 seconds in the baseline to 5 seconds with Architecture, 5 seconds with Skeleton, and 6 seconds with Callgraph. The difference is statistically significant for all three artifact conditions.

Beyond final task success, we looked at whether agents actually used the artifacts and how their navigation changed. Discovery was not a problem: Architecture and Skeleton were opened in 100% of trials, and Callgraph in 98.9%, usually within 4-5 seconds. However, agents generally read the artifact once and never returned to it.

The artifact conditions also used more tokens without meaningfully changing runtime. Compared with the baseline at 23.6M tokens per trial, Architecture used 27.7M, Skeleton 26.9M, and Callgraph 28.8M. Median within-pair runtime remained effectively 1.0× the baseline across all three conditions.

Finally, we measured the overhead introduced by Cartograph itself, including artifact generation time and size, finding it impacted total runtime only negligibly.

A summary of these results can be seen below.

A detailed breakdown of per-task results can be found below.

Discussion

Interestingly, the pass rate ended up being fairly low across every condition. Our experiment differs from the original DeepSWE setup in harness, reasoning effort, timeout policy, and task subset, so it is hard to pinpoint exactly why. GPT-5.6 Luna on Max + Pi also used substantially more tokens than our small-model canary runs, despite a 96% cache hit rate for the experiment.

The more interesting result is that the artifacts seem to do their job: they make initial navigation significantly faster. The problem is that localization (initial navigation) only represents around 0.2% of a full trial. Cutting that time in half is useful, but it has little effect on an hour-long task dominated by understanding the specification, implementing the change, and verifying the work.

The way agents use the artifacts might also explain why the benefit does not carry further. The agents almost always fell back to navigation heuristics after an initial skim of the map. The maps thus seem useful as an initial orientation aid but may not fit with the model's trained instincts.

These results should be interpreted narrowly. So far, only 30 of 112 tasks have been tested, and only one model-harness combination has been run. The experiment is therefore better viewed as a comparison between conditions under one fixed setup, rather than a general claim about structural artifacts across coding agents.

However, the results have actually increased my confidence that opinionated harnesses can help agents. They clearly improve localization, and the Skeleton artifact improves upon the baseline by 2.7 percentage points.

Future

For future work, benchmarking SOTA models would show whether stronger models benefit more from having a deterministically generated artifact they can use as an anchor. Testing different harnesses would also reveal how much artifact usage depends on the agent harness.

Additionally, since agents rely heavily on heuristics when exploring codebases, testing the maps over much longer runtimes, potentially without a time limit, could reveal whether their value becomes more significant on longer-horizon tasks.