Harness Engineering for On-Device Agents

Abstract

Small language models can now run entirely on a phone, but there is little controlled evidence on making them dependable at multi-step tool use. I study how much of an on-device agent's reliability comes from the harness around the model rather than from the model itself. Using Gemma 4 E2B on an iPhone 15 Pro (2,048-token context window, react-native-executorch), I add one harness component at a time (seven file tools, a todo-style planning tool, a single bash tool in place of the file tools, and a system prompt of worked examples) and re-run the same 20 tasks three times after each change, for 701 scored runs. I judge task success from the workspace files left on the device, and I also label every run by how it ended. I find that time to first token grows linearly at 5.2 s per 1,000 prompt tokens (R² = 1.000), so the seven tool definitions alone add about 6 s to model load. Outcomes are close to deterministic at the task level (in the final evaluation, 18 of 20 tasks score 0/3 or 3/3 in every configuration), and the most common failure is quitting after the first failed tool call, not looping. Each component moves the bottleneck without removing it. Planning prevents early quitting but exhausts the context window on 5 of 12 multi-step runs. Replacing seven tools with one bash tool cuts start-up context by 78% (1,351 → 300 tokens) without changing which tasks succeed. On this task suite, worked examples are the only change that turns consistently failing tool tasks into consistent successes; they also halve tool calls and cut failed calls by 2–4×, but break two previously solved simple tasks, so the net pass-rate change is not significant. I release the harness, evaluation dashboard and per-run data.

1Introduction

Running a language model on the phone itself offers privacy, offline use and no per-token cost, and sub-3B models are now capable enough to follow instructions and emit tool calls¹⁶16Google DeepMind's Gemma 4 open models. E2B is the small, on-device variant (about 2B effective parameters) used throughout this study.Google DeepMind. Gemma 4 model card. 2026.. An on-device agent, however, needs more than a model: it needs tools, a loop that executes them, a prompt that explains them and some way of managing a very small context window. This surrounding software, which I call the harness, is usually assembled from patterns developed for frontier models running in the cloud, where context is plentiful and tokens are cheap.

Recent work argues that much of what frontier models appear unable to do reflects how they are orchestrated rather than what they know^1,21Argues that frontier models are under-used because the scaffolds that manage them are hand-engineered and brittle, and that the next leap may come from letting models decompose tasks themselves rather than from more scaling.A. Zhang, Z. Li, and O. Khattab. The Mismanaged Geniuses Hypothesis. Blog post, 2026. alexzhang13.github.io2Recursive Language Models keep a long prompt as a variable in a Python REPL and let the model call itself over pieces of it, handling inputs far beyond its context window.A. L. Zhang, T. Kraska, and O. Khattab. Recursive Language Models. arXiv:2512.24601, 2025., and controlled studies of coding-agent harnesses find that the same component can help one model and hurt another depending on its capability⁴4Ablates context management, planning and the action space for coding agents from 30B to 550B parameters. Bash-only helps the strongest model but sharply hurts the weakest; planning mainly keeps weaker models from giving up; context management matters most under tight budgets.An Empirical Study of Harness Design for Coding Agents. arXiv:2609.20804, 2026.. It is not obvious how these findings transfer to the other end of the scale. A 2B model on a phone has a context window two to three orders of magnitude smaller. It reads its prompt at roughly 185 tokens per second and generates at roughly 20, so every token the harness adds costs user-visible latency. Prior work shows it also costs energy and heat on phones^3,183Benchmarks a 1.5B model on a Raspberry Pi NPU, a Galaxy S24 Ultra, an iPhone 16 Pro and an RTX 4050 laptop. On phones, heat rather than peak compute limits sustained speed: the iPhone loses about 40% of its throughput within three runs.P. Tummalapalli, S. Arayakandy, R. Pal, and K. Kundan. LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load. arXiv:2603.23640, 2026.18Characterises LLM inference on mobile devices, with attention to energy use and thermal behaviour.S. Laskaridis, K. Katevas, L. Minto, and H. Haddadi. MELTing Point: Mobile Evaluation of Language Transformers. In Proceedings of MobiCom, 2024.; this study measures latency and memory, not energy.

I ask a practical question: if the model is fixed, how much reliability can the harness buy, and what does each component cost on the device? To answer it I run an incremental ablation. Starting from the bare model, I add one component at a time and re-run the same task suite after each addition. Each step changes one component, but each step was also run in its own session, so comparisons between steps carry some session-to-session variation (Section 6). What I found:

1. Context costs time. On the phone, time to first token grows in a straight line with prompt length, about 5.2 s per 1,000 tokens, so you can predict the delay any prompt or tool definition adds from its token count before shipping it (see Section 4.1).
2. The model fails the same way every time. Across 701 runs, outcomes are close to deterministic, and the most common failure is giving up after one failed tool call, not looping (see Sections 4.2–4.3).
3. Each harness piece moves the bottleneck. I added file tools, planning, a single bash tool and worked examples one at a time, and measured what each did to task success, to how runs fail, and to the cost on the phone (see Sections 4.2–4.6).
4. Pass rates hide how far a run gets. Scoring every tool call against the task's milestones shows that setups with the same pass rate get very different distances (see Section 4.6).

2Related Work

On-device inference. Benchmarks of mobile LLM inference have characterized throughput, energy and thermal behavior^3,18; for example, under sustained load recent iPhones settle well below peak throughput. These studies measure the model and runtime in isolation. I hold the runtime fixed and measure how harness choices translate into prompt length, latency and task success.

Agent harnesses and tool design. Tool-using agents typically follow a reason-act loop¹⁹19ReAct interleaves reasoning steps with tool actions in one loop, the pattern most agent harnesses, including this one, follow.S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR, 2023. with either many narrow tools or a code or shell interface^20,520CodeAct gives agents executable code as their action space instead of JSON tool calls, and finds this improves task success.X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, and H. Ji. Executable Code Actions Elicit Better LLM Agents. In ICML, 2024.5A roughly 100-line agent whose only tool is bash. It scores over 74% on SWE-bench Verified and is widely used as a standard evaluation harness.mini-SWE-agent: a minimal bash-only software engineering agent. github.com. Several practitioner reports describe replacing specialized tools with a single shell or code tool and observing lower cost with equal or better success^5,6,76Vercel rebuilt its text-to-SQL agent around a single bash tool in place of many specialised tools, and reports higher success with fewer tokens and steps.Vercel. We removed 80% of our agent's tools. Blog post. vercel.com7A simulated bash for agents written in TypeScript: commands run against an in-memory filesystem and never touch the host machine. The same idea as the bash tool in this study.Vercel Labs. just-bash: a simulated bash environment for agents. github.com. The closest controlled study, an empirical analysis of coding-agent harnesses⁴, ablates context management, planning and action space across models from 30B to 550B parameters. It finds that a bash-only action space helps the strongest model but sharply degrades the weakest, that an explicit planning tool mainly helps weaker models by keeping them from abandoning tasks, and that context management matters most under tight context budgets. My study tests these components on a model an order of magnitude smaller than their weakest, under a context budget sixteen times smaller than their tightest.

Examples and tool selection for small models. Tool-use guidance increasingly recommends worked examples over schema descriptions alone^8,98Argues that a tool's JSON schema says what input is valid but not how the tool should be used, and that models learn correct usage from examples.Anthropic. Introducing advanced tool use. Engineering blog. anthropic.com9Practical guidance for agent tools: say when each tool should be used, return meaningful context, keep outputs token-efficient, and evaluate tools against realistic tasks.Anthropic. Writing effective tools for agents. Engineering blog. anthropic.com. In a study of few-shot tool adaptation on a 3B model, examples accounted for a substantially larger gain than written documentation¹⁰10On a Llama 3.2 3B model, few-shot examples account for most of the gain in adapting to new tools (+21.5%), far more than written documentation (+5%).Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models. arXiv:2604.20148, 2026., and examples attached to tool descriptions change how often models select those tools²¹21Shows that how a tool is described, for example adding usage examples, shifts which tools models choose, independent of whether the tool is actually better.Tool Preferences in Agentic LLMs are Unreliable. arXiv:2505.18135, 2025.. For edge devices, selecting a smaller tool set per request has been reported to reduce latency and energy¹³13Lets the model pick a smaller tool set for each request on edge devices, and reports shorter execution time, lower power use and more accurate tool calls.V. Paramanayakam, A. Karatzas, I. Anagnostopoulos, and D. Stamoulis. Less is More: Optimizing Function Calling for LLM Execution on Edge Devices. arXiv:2411.15399, 2024.. Small models have also been observed to repeat tool calls they have just seen fail¹²12Finds that small models (135M to 1.7B parameters) shown their own failed tool call become more likely to repeat that same call.Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail. 2026., and local models fail to produce required artifacts more often than hosted ones¹⁴14Local open-weight models fail to finish a task and write its required output far more often than models served through an API.DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents. arXiv:2609.06059, 2026..

3Experimental Setup

3.1Hardware, model and runtime

In all experiments I use Gemma 4 E2B¹⁶ running on an iPhone 15 Pro through react-native-executorch 0.10.4¹⁵15Software Mansion's library for running ExecuTorch models, including LLMs, inside React Native apps. It is the runtime behind every experiment here.Software Mansion. react-native-executorch. github.com, with a context window (maxSeqLen) of 2,048 tokens and temperature 0.2. Gemma emits tool calls in its own syntax, <|tool_call>call:name{key:<|"|>value<|"|>}<tool_call|>, which I parse myself. The model operates inside a private workspace directory on the device, which I present to it as /.

3.2Harness configurations

Table 1 lists the configurations. Each step adds or replaces one component relative to its predecessor.

Table 1. Harness configurations, in the order I introduced them. "Start-up tokens" is the context consumed by the system prompt and tool definitions before the user's first message.

The bash tool is not a shell. It is a small interpreter I wrote for this study, and it supports ls, cat, head, tail, wc, grep, find, echo and printf with redirection, here-documents, sed -i, mkdir, cp, mv, rm, sort and pipes, operating on the same workspace as the file tools. Its error messages follow bash conventions (e.g. cat: /notes: Is a directory). Like edit_file, it refuses to overwrite a file the model has not read in the current session.

The worked-example prompt (Appendix A) introduces the assistant and its tools, then gives one example call per job (list, search by content, search by name, read, create, append, edit, tick an item, copy, move, delete, show to the user) and six rules derived from failures I observed in earlier steps. To avoid teaching to the test, none of my examples uses a file, name or word that appears in the task suite. The file-tool and bash variants contain the same jobs.

3.3Task suite

The suite contains 20 tasks (Table 2), and I run each one from a fresh model session and a freshly seeded workspace.

Table 2. Task suite: 20 tasks in three groups, with representative examples from each group.

3.4Protocol

A Mac-hosted dashboard schedules jobs; the phone polls for one job at a time and returns a full trace. Before each measured run, I run a throwaway warm-up generation, because I observed that the first reply after a few minutes of idling is 2.5–3.5 s slower. Between runs the phone rests until its thermal state returns to nominal (capped at three minutes). I run every task three times under each configuration. Before each full evaluation I run a single-repeat trial and test every check against deliberately incorrect answers.

I ran the sessions on 14–15 October 2026, one session per step, with builds stamped by commit. Between sessions I fixed bugs (see Section 6). All sessions ran with the phone unplugged except the Step 3b evaluation, which ran overnight while charging; I therefore compare its timing measurements only within that session.

3.5Metrics

Task success. My checks inspect the workspace after the run, not only the reply: a todo item counts only if /todo.md contains a new - [ ] line with the required content, and a claim of success without the corresponding file change fails.

Outcome. I label each run finished, wrong result, asked instead (asked the user before any successful tool call), gave up, hit tool limit, context full (the runtime failed to ingest further input with the context estimated at ≥85% of the window) or crashed.

Loops. I mark a run as looping if it issues the same tool call with identical arguments twice in a row, or three times anywhere.

Cost. Start-up tokens, model load time, time to first token, turn time, peak context, tool calls and failed tool calls.

Statistics. I report pass rates with Wilson 95% intervals¹⁷17The Wilson score interval for a proportion, which stays well behaved for small samples and at 0% or 100%, as in 0/3 or 30/30.E. B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212, 1927.. Because outcomes are nearly deterministic per task, I also compare configurations at the task level with an exact sign test over tasks whose outcome changed.

Progress trajectories. For the four multi-step tasks I define three or four milestones per task (e.g. searched, read the note, saved a correct reminder, added the todo item) and score each tool call by the milestones reached so far. A milestone counts only when its content is correct: a reminder must contain the actual date and time, a todo entry must be a checklist item, and pasting the instruction text into a file does not count as writing the requested sentence.

4Results

4.1The bare model: reasoning room and the cost of context

The bare model passed 27 of the 30 runs on tasks that need no files. All three failures came from one arithmetic task whose prompt ended with "Give the number." The model answered with two tokens and was wrong each time (29, 29, 23; correct answer 31). A harder arithmetic task posed as an open question was answered correctly each time, with 265 tokens of worked reasoning taking about 15 s (Figure 1). The "Give the number." task, which leaves the model no room to reason, failed under every configuration I tested.

Figure 1. Tokens generated (a, log scale) and accuracy (b) on two arithmetic word problems with the bare model, three runs each: one prompt ends with "Give the number.", the other is posed as an open question.

Time to first token is linear in prompt length (Figure 2): 1.7 s at 297 tokens, 3.0 s at 548, 5.4 s at 1,015 and 7.8 s at 1,482, a slope of 5.2 s per 1,000 tokens with R² = 1.000 and repeats within 0.2 s. Decoding speed (about 20 tokens/s), peak memory (about 1.6 GB) and accuracy (12/12) did not vary with prompt length. On this device, the context a harness adds is therefore paid for almost entirely as waiting time before the first token.

Figure 2. Time to first token against prompt length, three runs per length, with a least-squares fit. Dotted lines mark the start-up context of each harness configuration in Table 1.

4.2File tools enable file tasks and break some simple ones

When I added the seven file tools, success on tasks that need files rose from 0/30 to 12/30, but success on tasks that do not fell from 27/30 to 21/30 (Figure 3a). The regression is concentrated in the two tasks that supply notes inside the message. Both fell from 3/3 to 0/3 because the model ignored the supplied text and tried to read it from a file. The tool definitions add 1,138 start-up tokens, and model load time rose from 0.7 s to 6.6 s, consistent with the slope in Section 4.1 (Figure 3b, c). Because the definitions are processed once at load, per-reply time to first token barely changed (0.27 s to 0.32 s).

Figure 3. Bare model versus file tools, three runs per task. (a) Pass rate on tasks that need files and tasks that do not, with Wilson 95% intervals. (b) Start-up tokens. (c) Median model load time.

4.3Failures are systematic, and the model quits instead of looping

In the Step 1 evaluation, every one of the 20 tasks scored either 3/3 or 0/3 in both configurations. Figure 4 shows that this near-determinism persists across the study: in the final evaluation, 18 of 20 tasks score 0/3 or 3/3 in every configuration. All runs used temperature 0.2, and at this temperature repeated sampling or majority voting would recover few failures; changing the inputs is more likely to help. Whether this holds at higher temperatures I have not tested.

Figure 4. Runs passed out of three for every task in the main configuration of each step. Task names are colored by group: no files needed (black), file tasks (blue), multi-step file tasks (red). Columns come from separate sessions.

I observed no looping in the 60 tool-enabled runs of Step 1. Instead, the model stopped after one to three calls, most often immediately after the first failed call (Table 3). The outcome distributions in Figure 5 show the same pattern. With file tools, half of the failed file-task runs (9 of 18) end with the model asking the user to do something it could have done itself.

Table 3. Failure patterns with file tools (Step 1). Each pattern occurred in all three runs of its task.

Figure 5. Distribution of run outcomes per configuration on (a) the ten file tasks and (b) the four multi-step tasks.

4.4Planning prevents quitting but exhausts the context window

The planning tool left the overall pass rate unchanged (33/60 with and without) but changed how runs ended (Table 4). The model planned before acting in 18 of the 21 runs in which it used the tool, and stopped asking the user to intervene (12 → 3 runs). On the appointment task it now recovered from the failed directory read in every run: it listed the directory, read the correct note and wrote a correct reminder, then ran out of context before adding the todo item.

Table 4. File tools with and without the planning tool (Step 2), 60 runs each. Tool calls are totals over all runs.

Planning raises the start-up context from 1,138 to 1,351 tokens, leaving under 700 tokens of a 2,048-token window for the task, and every plan update re-sends the full list (Figure 6). This is consistent with the finding of⁴ that planning mainly helps weaker models by keeping them from abandoning tasks. Under a 2,048-token budget, however, the extra persistence uses up the window before the task completes.

Figure 6. Peak context per run on the four multi-step tasks with and without planning, three runs each. × marks runs that ended because the context window was full.

4.5A single bash tool cuts start-up cost without changing which tasks succeed

In the planning runs, individual tool results were small (roughly 15–35 tokens), and most of the window was taken up by tool definitions before the user's message. When I replaced the seven file tools with the single bash tool, start-up context fell from 1,351 to 300 tokens (513 with planning), load time from 7.8 s to 2.2 s, and context exhaustion disappeared (Figure 7, Table 5).

Table 5. File tools with planning versus one bash tool with and without planning (Step 3a), 59–60 runs per configuration.

Figure 7. File tools with planning versus one bash tool with and without planning, three runs per task. (a) Start-up tokens. (b) Median load time. (c) Highest context reached in any run, against the 2,048-token window. (d) File tasks passed.

File-task success did not change. The same three file tasks passed 3/3 and the remaining seven failed 0/3 in every configuration, and the traces suggest why. The model used grep in only 7 of the 75 bash runs that issued any command, and 30 of its 34 failed commands were cat on a guessed path. It still wrote placeholders ("Reminder: Dentist appointment on [Date and Time]") without reading the source note. Bash also accounts for the drop on tasks without files in Table 5. Without planning, it broke a recall task that every other configuration solved ("I do not have access to your personal information"), and with or without planning it tried to read the notes supplied in the message from a file, failing that task 0/3 where file tools with planning passed it 3/3. These results are consistent with⁴, where a bash-only action space helped the strongest model and degraded the weakest. At 2B parameters I see no quality benefit, only a large reduction in cost.

4.6Worked examples change how the tools are used

When I added the worked-example prompt to each tool set, it raised file-task success from 9/30 to 12/30 in both cases, reduced tool calls by more than half and failed calls by a factor of 2.4 (file tools) and 4 (bash), and eliminated context exhaustion for the file tools (Table 6, Figure 8).

Table 6. Step 3b, 60 runs per configuration. The phone was charging during this session; turn times are comparable only within the table.

Figure 8. Each tool set with and without worked examples, three runs per task. (a) File tasks passed. (b) Tool calls and (c) failed tool calls over 60 runs. (d) Median turn time.

At the task level, the gains are specific and repeatable. With bash, the search task, which had failed in every earlier configuration, passed 3/3 with a single grep -ril call. With file tools, the drafts-indexing task passed 3/3 by listing the folder, reading both files and writing an index with accurate descriptions; this is the first multi-step task to succeed with planning enabled. Both tool sets also recovered the pasted-notes task by answering from the message. The examples had costs as well. With file tools, the recall task fell from 3/3 to 0/3 and the 15-word task from 3/3 to 0/3 (16 words); with bash, the arithmetic task fell to 1/3 (the model asked for a currency exchange rate) and the 15-word task to 2/3.

Because each tool set gained one file task and changed the outcome of only a few tasks overall, these differences do not reach statistical significance at the task level (with two improved and two degraded tasks for file tools, an exact sign test gives p = 1.0; for bash, two improved and two degraded, p = 1.0). The behavioral change is larger and appears with both tool sets: fewer calls, fewer failed calls and correct use of search. I treat the pass-rate result as suggestive and the behavioral result as the more reliable one.

Introducing the examples also exposed an interaction with the model's tool-call syntax. In an interactive test I ran before the evaluation, the model produced 17 tokens and no executed call: it had adopted the examples' key="value" notation inside its native call syntax, which my parser rejected. I made the parser accept this notation and the parenthesized form shown in the examples; all evaluation runs used the corrected parser.

Progress trajectories. Pass rates hide how far failing runs get. In Figure 9, I score each tool call on the appointment task against its milestones. File tools fail on the first call and stop. Planning recovers from failures and reaches a correct reminder, then exhausts the context. Bash with planning searches by file name, finds nothing and writes placeholders. Bash with examples reaches a correct reminder in three calls (search, read, write), the fastest progress of any configuration, and then stops one step short. I extend the same analysis to all four multi-step tasks in Figure 10. On the meeting-notes task no configuration gets past its first step, because every configuration guesses the notes' file name.

Figure 9. Progress after each tool call on the appointment task ("find my dentist appointment, add it to /todo.md and save a reminder"), one line per configuration, each showing the path taken by the majority of its three runs. Dips into the shaded band are failed calls; × marks context exhaustion and ■ an early stop.

Figure 10. Progress after each tool call for all four multi-step tasks, plotted as in Figure 9.

4.7Summary across steps

Figure 11 summarises the main configuration of each step. Start-up cost rose with each added capability until the bash tool reduced it by 62%. Success on file tasks changed only twice: when I introduced tools, and when I added worked examples.

Figure 11. Main configuration of each step. (a) Pass rate on tasks that need files and tasks that do not, with Wilson 95% intervals. (b) Start-up tokens. (c) Median model load time. I evaluated the steps in separate sessions.

5Discussion

Each component moved the bottleneck. Tools made file tasks possible, but the model abandoned them after one error. Planning removed the early abandonment and exhausted the context instead. The bash tool freed the context without improving success, which left how the model searched and edited as the remaining limit, and worked examples targeted that limit directly. No single component was sufficient, and each is better judged by its effect on the failure mode it targeted than by the aggregate pass rate.

On-device cost is predictable. Because time to first token is linear in prompt length on this device, I can estimate the latency cost of a harness change before deployment from its token count alone. At 5.2 s per 1,000 tokens, the 838-token difference between the seven file-tool definitions and the single bash tool (1,138 and 300 start-up tokens, without planning) predicts about 4.4 s of waiting each time the model loads. I can check the prediction against Table 5: file tools with planning (1,351 tokens) and bash without planning (300 tokens) differ by 1,051 tokens, which predicts 5.5 s, and the measured load times differ by 5.6 s (7.8 s and 2.2 s).

Failure analysis was more informative than pass rates. Configurations with identical pass rates differed sharply in outcome distribution and in progress (Figures 5, 9, 10). Labeling how each run ended and scoring intermediate progress is what pointed me to each next intervention.

Agreement with larger-scale evidence. My results agree with those reported for coding agents of 30B parameters and above⁴, where planning mainly helps weak models by sustaining effort and a bash-only action space does not help them. The worked-example result is in line with reports that examples help small models more than documentation¹⁰. The main difference at this scale is the severity of the context constraint, under which planning, helpful at larger scale, caused failures on multi-step tasks.

Remaining failures. The tasks that still fail in every configuration mostly require two edits to different files (a reminder and a todo item; a summary and a todo item; a letter and a todo item). The model typically completes the first and reports success. A lightweight completion check that compares the request against the changes made to the workspace is therefore a natural next component for me to test, provided its token cost is justified under the latency model above.

6Limitations

Scope. I evaluate one model, one device and one runtime. I expect qualitatively similar behavior from other small models but have not tested this.

Sample size. My suite contains 20 hand-written tasks with three repeats each. Because outcomes are nearly deterministic, the effective sample size is closer to the number of tasks than the number of runs, and the improvements in Section 4.6 are not statistically significant at the task level.

Sessions and builds. I evaluated each step in its own session and build, and fixed several defects between sessions: a parser error affecting arguments with a dropped string delimiter, an echo implementation that did not interpret \n, error messages that now suggest a next action, and the parser leniency described in Section 4.6. Within-session comparisons are therefore stricter than comparisons across steps. The one configuration that appears in three sessions (file tools with planning) shows how large the session-to-session variation is: its file tasks scored 9/30 every time, but its tasks without files scored 24/30, 23/29 and 21/30 and it ran out of context in 5, 3 and 4 runs, with the pass-rate differences coming from two tasks (the pasted notes and the unseen file). I ran the Step 3b session while the phone was charging.

Checks. I wrote every check myself as a pattern match over files and replies. Before use, I tested each check against incorrect answers, and still found and corrected two that were too lenient during the study; the numbers I report use the corrected checks.

Worked examples and the task suite. I wrote the worked-example rules after watching the model fail on this same suite, and some target a specific failure (for example, answering from text pasted into the message). Keeping the suite's files and words out of the examples does not remove that overlap, so the gains in Section 4.6 may not carry over to new tasks. A held-out set of tasks, written after the prompt was fixed, would test this.

Planning and examples together. In Step 3b both configurations with examples also had planning, so I cannot separate the effect of the examples from their interaction with the planning tool.

Temperature. Every run used temperature 0.2. The near-determinism in Section 4.3, and the conclusion that resampling would not help, may not hold at higher temperatures.

Components not studied. I did not evaluate self-verification, an explicit reasoning budget, context compaction or offloading. I planned compaction, but it became unnecessary for this task suite once the bash tool reduced start-up context.

7Conclusion

With the model held fixed, my harness largely determined what a 2B on-device agent could accomplish and how long the user waited for it. The linear per-token cost made the latency of each change predictable, and my labels for how each run ended indicated which change to try next. The change with the largest effect on behavior was not a new tool or control loop. It was a short set of worked examples showing the model how to use the tools it already had. I release the harness, the evaluation dashboard and all 701 scored runs to support replication on other models and devices.

References

1A. Zhang, Z. Li, and O. Khattab. The Mismanaged Geniuses Hypothesis. Blog post, 2026. alexzhang13.github.io

2A. L. Zhang, T. Kraska, and O. Khattab. Recursive Language Models. arXiv:2512.24601, 2025.

3P. Tummalapalli, S. Arayakandy, R. Pal, and K. Kundan. LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load. arXiv:2603.23640, 2026.

4An Empirical Study of Harness Design for Coding Agents. arXiv:2609.20804, 2026.

5mini-SWE-agent: a minimal bash-only software engineering agent. github.com

6Vercel. We removed 80% of our agent's tools. Blog post. vercel.com

7Vercel Labs. just-bash: a simulated bash environment for agents. github.com

8Anthropic. Introducing advanced tool use. Engineering blog. anthropic.com

9Anthropic. Writing effective tools for agents. Engineering blog. anthropic.com

10Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models. arXiv:2604.20148, 2026.

11Google. FunctionGemma formatting and best practices. ai.google.dev

12Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail. 2026.

13V. Paramanayakam, A. Karatzas, I. Anagnostopoulos, and D. Stamoulis. Less is More: Optimizing Function Calling for LLM Execution on Edge Devices. arXiv:2411.15399, 2024.

14DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents. arXiv:2609.06059, 2026.

15Software Mansion. react-native-executorch. github.com

16Google DeepMind. Gemma 4 model card. 2026.

17E. B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212, 1927.

18S. Laskaridis, K. Katevas, L. Minto, and H. Haddadi. MELTing Point: Mobile Evaluation of Language Transformers. In Proceedings of MobiCom, 2024.

19S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. ReAct: Synergizing Reasoning and Acting in Language Models. In ICLR, 2023.

20X. Wang, Y. Chen, L. Yuan, Y. Zhang, Y. Li, H. Peng, and H. Ji. Executable Code Actions Elicit Better LLM Agents. In ICML, 2024.

21Tool Preferences in Agentic LLMs are Unreliable. arXiv:2505.18135, 2025.

---

Citation

Appendix A: Worked-example prompt (bash variant)

Appendix B: Data and code availability

My harness and app are available at https://github.com/jameskanyiri/on-device-agent-harness and my evaluation dashboard at https://github.com/jameskanyiri/On-Device-Agent-Performance-Monitoring. The scored data for all 701 runs, one row per run with its task, configuration, outcome, token counts and timings (the JSON also lists every tool call), is available as runs.csv and runs.json. The dashboard's src/server/export-runs.test.ts regenerates both files from the raw evaluation sessions, scoring each run exactly as the dashboard does.