The Orb Knows

Abstract

Agents without memory are amnesiac tool-callers; agents with memory are colleagues. We describe the memory subsystem behind Jarvis, Tensor, and Tensor Code: a persistent, self-consolidating store that writes continuously while the agent acts and compresses on idle — analogous to the role sleep plays in biological memory. The system is built on SQLite with FTS5, runs entirely on-device, and exposes its state through an orb interface so the user can see what the agent knows about them. We report p99 recall latency under 40 ms, a >6× improvement in multi-session task success over memoryless baselines on our internal evaluation, and a simple, auditable data model that users can inspect, edit, or delete.

1. Motivation

The current generation of "chat" assistants is stateless by default. Context windows are stuffed on each turn; anything beyond the window is forgotten. This is fine for one-shot answers and catastrophic for agency. A useful agent has to know what you asked yesterday, who "my sister" refers to, which project directory you meant, and whether it already tried a failed approach last Tuesday.

We set three design goals:

1. Local-first. Memory never leaves the device without explicit opt-in.
2. Continuous, not batch. The agent writes as it acts, not in a separate ingest job.
3. Legible. The user can see the memory — the orb visualizes density, growth, and topic clusters.

2. Data model

Three tables, no vectors at the storage layer:

FTS5 provides BM25 ranking over all three tables without a separate index service. Embedding retrieval is a secondary path computed on-demand, not at write time — a deliberate choice to keep the write path cheap and the system easy to inspect.

3. Idle consolidation

When the agent is idle for more than a 90-second window, a consolidation pass runs:

• Summarize the last N events into a single episode row; drop events older than a configurable TTL.
• Link the episode back to any related prior episodes via shared entities (people, files, URLs).
• Promote any repeated-across-episodes assertion into the facts table.
• Forget facts that contradict newer observations; keep the old one as an episode reference, not a live fact.

The consolidator is itself the agent's own model — no separate dedicated model — run at low priority. Users can trigger it manually (/sleep in Jarvis) or inspect its output.

4. Results

5. Visualizing knowing

The orb interface isn't decorative. Its density encodes fact count, its temperature encodes recency, and its "breathing" corresponds to active consolidation. When the agent sleeps, the orb dims; when it wakes with new facts, it flickers. This gives the user a first-person view into whether the agent has seen them enough to be useful yet — a question that has historically been answered with "trust us."

6. Limitations

Multi-user shared memory (e.g., a household) is out of scope for this version. Fact contradictions currently require manual review if they span more than two episodes. Embedding-based recall is on-demand only; latency drops to 40 ms hold for BM25 queries, not ANN searches.

One Model, Many Surfaces

Abstract

We argue that the right factorization for a personal agent is one base model, per-surface adapters — not one agent per product. A single 4B base, saturday-4b, drives all three MingLLM surfaces: voice (Jarvis), web (Tensor), and code (Tensor Code). Surface specialization comes from (a) small LoRA adapters trained per surface, (b) a routing head that dispatches to the right adapter based on the user's current surface and the nature of the request, and (c) a shared memory backbone (see The Orb Knows). This keeps capability accretive: a skill learned in one surface lifts the others, memory is common, and inference cost stays fixed at 4B.

1. Why one base

The naïve approach is a product per model: one LLM for voice, another for browsing, a third for coding. We tried this. The result is three separate context silos, three separate training pipelines, and three models that have to re-learn the same user every Monday. The unified-base approach solves all three:

• Shared context. A fact learned by Jarvis is immediately available to Tensor.
• Single training loop. Improvements to the base lift all surfaces; improvements to an adapter are cheap and targeted.
• Predictable inference cost. The user pays for 4B params, not 12B spread across products.

2. Architecture

Each adapter is trained via expert iteration on surface-specific trajectories (see Against Scale for the method). Training is fully independent per surface, so work on the Web adapter doesn't block Voice.

3. Per-surface results

4. Cross-surface transfer

Because the base is shared, we see positive transfer: a capability learned in one surface shows up partially in the others without explicit training. For example, training the Web adapter on multi-step planning raised the Voice adapter's task-completion rate by 7.3 points despite no Voice retraining. This is the point — the surfaces are three hands on the same mind.

5. Why not a larger single model?

A 27B dense model would solve the three-surfaces problem by brute force but wouldn't fit in a laptop RAM budget at acceptable inference latency. MoE (our target Gemma 4 27B MoE with 3B active) sidesteps this — it's the natural next base. The router-and-adapter pattern composes with MoE rather than competing with it.

6. Limitations

The router head is currently trained supervised on labeled surfaces and can misroute on ambiguous inputs ("can you show me this in my calendar and book it?"). We're evaluating a latent, multi-adapter-mix approach where both Voice and Calendar adapters can be active simultaneously.

Against Scale

Abstract

The dominant thesis of the last five years is that capability scales with compute, parameters, and data — a thesis that justifies an industry of billion-dollar training runs. On bounded task distributions — the kind a personal agent actually faces — this thesis is wrong. Expert iteration on a 4B base, run on a single MacBook over a weekend, matches frontier cloud models that are 50–400× its size. We ran the experiment. Three cycles, ~3 hours each, $0 in cloud compute, 15.3% → 94.9% on a real-world browser agent benchmark. This paper makes the position explicit: for bounded agency, scale is not the moat. Trajectories are.

1. The scaling orthodoxy

The argument for scale has three legs: (i) emergence — capabilities appear only above a parameter threshold; (ii) sample efficiency — bigger models learn from less; (iii) generality — bigger models transfer to more tasks. Each is true in the limit and misleading in practice.

For a specific agent facing a specific task distribution — book my flights, drive my browser, ship my code — the relevant question is not "how many capabilities can this model have in principle?" but "how many of the capabilities this user needs does it have reliably?" We show that this narrower question is answerable with loops, not scale.

2. The experiment

Setup. Base: Gemma-4 4B. Hardware: single Apple M5 Max MacBook, 64 GB unified memory. Cloud compute: $0. Training framework: MLX. Task distribution: 59 real end-to-end browser tasks drawn from production traffic.

Method. Expert iteration (ReST-style). Each cycle:

1. Roll out. The current model attempts each of the 59 tasks; traces are recorded.
2. Filter wins. Only trajectories that successfully complete the task are kept. No human labels.
3. Fine-tune. SFT on wins mixed with baseline data (3:1 for tool use, 2:1 for knowledge) — a LoRA of rank 16 and α=32, 800 iterations, LR 1e-5 (AdamW). ~2–3 hours per cycle.
4. Fuse + evaluate. LoRA fused into base weights. Fused model evaluated on held-out and WebArena-lite; if metrics improve, the fused model becomes next cycle's base.

3. Results

Cycle 3 matches or beats the reported scores of every public frontier model we can compare against on the same benchmark class, at 50–400× fewer parameters and ~nine hours of laptop time.

4. Against scale

Three claims, in increasing order of contentiousness:

1. Narrow. For bounded task distributions, a small model plus expert iteration dominates a frontier model plus prompting on both capability and cost.
2. Actionable. The ingredients required — a 4B base, a scoring function, and a laptop — are within reach of anyone who wants an agent.
3. Strong. The marginal return on scale for agentic tasks, conditional on access to the user's task distribution, is approximately zero. The moat is trajectories, not parameters.

We expect disagreement on the strong claim. We note only that the moat argument is empirically testable, and that our test fails to find it.

5. What this doesn't claim

We do not claim scale is never useful. For genuinely open-ended generation, open-world reasoning, or extremely long horizons, scale helps. We claim that personal agency — the kind MingLLM builds — is not one of those regimes.

6. What this implies

If the moat for personal agency is the task distribution and not the model, the economics of the industry inverts. The interesting asset is no longer a trillion-parameter checkpoint; it is a tight loop between a real user, a scoring function, and a 4B model on that user's laptop. This is the thesis MingLLM is built on.

Step 1: Rollout

The model generates trajectories by attempting real production tasks — navigating websites, filling forms, answering queries. Each rollout is a complete execution trace from initial state to final action, recorded with full observation-action pairs.

We use Gemma-4 4B with 30+ browser tools (tcb_smart_click, tcb_deep_inspect, tcb_type, tcb_navigate, etc.) to interact with real web environments. Each rollout produces a trajectory of 5–20 steps depending on task complexity.

Batch size per cycle: 59 unique E2E tasks, with multiple rollout attempts per task to ensure diversity.

Step 2: Filter Wins

Each trajectory is scored against the ground-truth task completion. Winning trajectories — those that successfully complete the full task — are selected for training. Failed trajectories are discarded or kept at low ratio for negative sampling.

The filtering step is critical: it creates a high-quality training signal without any human annotation. The model learns exclusively from its own successful demonstrations.

Anti-forgetting mix: winning trajectories are mixed with baseline (general capability) data at a 3:1 ratio for tool-use and 2:1 for knowledge preservation.

Step 3: Fine-tune

Supervised fine-tuning on the filtered winning trajectories using LoRA (rank 16, alpha 32). Training runs for 800 iterations with learning rate 1e-5 (AdamW) on the MLX framework, entirely on Apple M5 Max hardware.

After fine-tuning, the LoRA adapter is fused into the base model weights. This fused model becomes the starting point for the next iteration cycle, progressively accumulating agentic capabilities.

The entire fine-tuning cycle takes approximately 2-3 hours on consumer Apple Silicon hardware, making it practical for iterative development.

Step 4: Evaluate

The updated model is evaluated on held-out test tasks (never seen during training), WebArena-lite benchmark (35 tasks), and an 80-test multi-turn assessment suite.

Evaluation gates: a model must pass a minimum threshold on held-out tasks before being promoted as the new production model. This prevents regression from any single training cycle.

If evaluation metrics improve, the cycle repeats from Step 1 with the new model. If metrics plateau or regress, the cycle terminates and the best checkpoint is selected.