Updated
| Stage 1 | Stage 2 | Stage 3 | |
|---|---|---|---|
| Focus | Alignment | Sync reason | Streaming reason / ICL |
| Train | WM: ot, at → ℓ(ot+1) IDM: ot, ot+1 → at ot, ot+1 → ℓ(at), at |
ot−k, at−k, …, ot → reason, at | — |
| Eval | WM, IDM Multiple choice, etc. |
Task success In-distribution, zero-shot |
— |
| Comparison | Action-as-text | VLA w/o reason | — |
The handwritten function symbol is transcribed as ℓ(·); its exact notation is slightly unclear.
Original version:
Obs 1 -> Reason 1 -> (Interrupt) Action 1 -> Reason 1 cont'd -> (Interrupt) Obs 2 -> Reason 2 ...Simplified version (assuming the action and new observation arrive simultaneously):
Obs 1 -> Reason 1 -> (Interrupt) Action 1 -> Obs 2 -> Reason 2 ...Here, action tokens may serve the following purposes:
The desired approach to data annotation is the “free lunch” in existing large-scale robotic datasets—we use future observations as oracle information to generate supervision for action-language alignment.
Here are some objectives:
Notes:
The following are post-discussion thoughts.
Solutions:
– Combining two topics: harmonic reasoning + “free lunch” in robotic datasets
Frontier LLMs (GPT Astra, Claude Fable) can control robots by directly controlling end effectors or calling skills/primitives.
But the limitations are:
Solutions:
Assumption: We need to output an action every dt seconds. The initial thinking should not take more than T seconds—for example, dt = 1.0, T = 10.0. We receive a new observation (short video or image) every dt seconds.
Key question: How do we keep a sliding-window context with recent media while also having a long-term text-based summary?
Solution: Cache observations and evict them in FIFO order when the context length is exceeded (see StreamingLLM).
[Recent turns with observations: cached] [Textual Memory M_t: fresh] [Action query] → response
Update rules:
Persistent base: [... Oₜ₋₁ Oₜ]
Temporary use: [... Oₜ₋₁ Oₜ] [Mₜ] [query] [response]
Next base: [... Oₜ₋₁ Oₜ Oₜ₊₁]
Next use: [... Oₜ₋₁ Oₜ Oₜ₊₁] [Mₜ₊₁] [query] ...If we want, we can even remove the explicit textual memory because historical information is implicitly contained in cached turns.
Requirement: Can be served on 8 × H100 GPUs
Modern LLM agents solve tasks by searching repositories and reading files through tool calls. This raises a question: can a model internalize such external knowledge—not merely memorize it, but understand it well enough to act more effectively at test time?
Aviral explored this in two stages:
Internalize repository knowledge. The model explored a codebase, generated questions about it, found the answers, and was fine-tuned on those question–answer pairs. Although this should have taught the model about the repository, agentic performance did not improve. Knowledge acquisition alone was therefore insufficient.
Learn to use the knowledge. The agent first collected multi-turn interaction rollouts in the environment. Another model then compressed parts of those interactions into reasoning traces framed as recalling relevant internal knowledge before acting. Fine-tuning on both the repository knowledge and these action-oriented reasoning traces improved performance.
The reasoning data required neither human demonstrations nor a stronger expert model: it was generated automatically by summarizing the agent’s own interactions.
Possessing useful knowledge and knowing how to use it are distinct capabilities. Auxiliary training may inject knowledge without improving the target task unless the model is also trained to retrieve and apply that knowledge when choosing actions.
The central challenge is therefore not only what knowledge to internalize, but also how to create training data that teaches the model to use it.
The experiment suggests a possible direction for robotics: use robot trajectories themselves to generate reasoning-oriented training data, without relying on additional human or expert annotations.
However, robotics introduces several harder problems:
Can we automatically summarize (part of) a robot’s trajectories into supervision that somehow helps it learn important reasoning-related abilities, such as having memory and using reasoning to derive actions?
I call it the “free lunch” in current robotic datasets, similar to “self-supervised learning,” but for reasoning purposes.
I want the main gain to come from understanding the data themselves, not from distilling large expert models—for example, asking GPT how to understand something. We can use VLMs, but mainly for data generation, not for gaining knowledge.
From now on, these are LH’s own thoughts.
Note: Here, “reasoning” refers to the general human thinking process. It may take the form of chain-of-thought, as in LLMs, but may not be limited to that.
One-sentence reason: A robotic policy involves multiple modalities: vision, language, action, and reward/value. In contrast, an LLM works only with language, which makes history, context, and reasoning feasible. As a result, there are no robotic foundation models trained on all of them.
Existing pre-training paradigms (disentangled):
Given so many modalities and so many abilities to learn, the above are not enough. Even worse, in methods involving language, language may serve only as a label without involving in-depth reasoning. It is therefore hard to say that those models “understand language,” and “reasoning for action” remains missing in robotics.