A selective map of reinforcement-learning methods that treat interaction as a temporal decision problem—not a long transcript masquerading as a one-step completion—and a concrete offline-to-online design for a VLM choosing language subtasks for a low-level VLA.
is the right primary object.
It can rank several feasible instructions at the same visual/history state and admits one-step Bellman updates from a replay buffer. A pure V(s) only scores states; action selection still needs candidate sampling, a model, or Q-like lookahead.
is the closest existing algorithmic template.
ArCHer explicitly learns utterance-level Q/V off-policy, then sends the utterance advantage into token-level policy learning. Its reported ~100× sample-efficiency advantage over on-policy baselines is unusually aligned with costly robot rollouts, though it was tested on text agents, not robots.1↗
usefulness × feasibility × long-term return.
SayCan already demonstrates language candidate scoring with learned robotic affordances, but its high-level planner is not learned by RL and its original feedback is myopic.2↗ The project should retain the safety/feasibility gate and add a high-level return critic.
Key negative conclusion. GiGPO, RAGEN/StarPO, turn-level GRPO and similar methods are useful controls and simulation baselines, but remain fundamentally on-policy/group-rollout methods. They improve multi-turn credit assignment, not real-world data reuse. For the intended project they should not be the main algorithm.
Project assumption. At high-level time t, the VLM observes recent images, task goal, execution history, low-level success/failure signals, and optionally a compact memory; it emits a short language instruction executed for a variable-duration option by a mostly fixed VLA. A real rollout therefore yields a small number of semantically meaningful transitions, not thousands of motor actions.
| Family | Representative work | Where temporal structure enters | Data regime | Fit to expensive robotics |
|---|---|---|---|---|
| Hierarchical actor–critic | ArCHer | Utterance MDP with Bellman Q/V; token MDP receives utterance advantage | Off-policy replay; online and offline variants | Best fit |
| Language-valued critic | NLAC | Language Bellman backup predicts and critiques future consequences | Off-policy prioritized replay; no policy gradients | High-upside |
| Conservative offline language RL | CHAI; human-centric dialogue RL | Sequence/utterance return under dataset support constraints | Fully offline; KL/pessimism or CQL-style control | Strong prior |
| Grouped turn credit | GiGPO; turn-level GRPO; G2PO | Compare actions from repeated/identical states; graph TD edges | Mostly on-policy grouped trajectories | Sim ablation |
| Search / model-based improvement | Agent Q; LATS; RWML; WorldCoder | Branch, predict next state, critique or back up terminal outcomes | Often environment- or inference-heavy; can distill offline | Use in sim / model |
| Embodied language agents | CALM, WebShop, Embodied Planner-R1; SayCan (adjacent) | Language action candidates, closed-loop feedback, skill affordances | Mixed; many are rollout-heavy | Evidence bridge |
Mechanism. The high-level critic learns utterance Q and V with target-network Bellman bootstrapping. The actor’s token log-probabilities are weighted by the utterance advantage. An offline variant uses IQL-style expectile V estimation plus advantage-weighted regression (AWR), explicitly preventing unconstrained out-of-distribution language actions. The authors report about 100× sample efficiency relative to existing methods across Twenty Questions, Guess My City, a detective game, and WebShop; their PPO implementation required at least 1,024 new trajectories per update.1↗
Burden/evidence. Research code supports replay buffers and GPT-2/RoBERTa-scale actor/critic models, but the repository warns that even Twenty Questions can take a week on one GPU.3↗ Evidence is multi-turn but text-only; 7B was the largest reported model and no physical deployment was attempted.
Exact transfer. Make each completed VLA subtask one utterance-level transition. Train double Q(s,a) and expectile V(s) from all robot logs; update a VLM subtask generator/ranker by AWR. Preserve a frozen or slowly updated VLA. This is the report’s primary blueprint.
Mechanism. A language successor model predicts future consequences; a language Bellman target merges the observed one-step successor with predicted downstream futures. A language evaluator turns this into an action critique, and a refinement policy proposes a better action. The policy distills refinements rather than using a policy-gradient ratio. Priority is tied to critic loss so replay emphasizes likely-suboptimal steps. On 20 Questions and τ-retail, the paper reports roughly 30% improvement over standard RL fine-tuning on multi-step tasks.4↗
Limitations. Its convergence claims rely on strong ordering and representation assumptions; one model plays several prompted roles; evaluations remain language/tool tasks. A fluent critique can be causally wrong. No code link was identifiable on the canonical paper page at search time.
Exact transfer. Pair scalar Q with a generative “future consequence” head: “if the VLA attempts a, likely object-state change / failure / recovery is …”. Use it to relabel or refine bad logged subtasks while scalar Q remains the decision/safety anchor. Especially attractive when random free-form instruction exploration is unsafe.
Mechanism. Complete trajectories from the same task give a macro relative advantage. Repeated observations across those trajectories form anchor groups; actions from the same state are compared to produce micro advantages without an auxiliary critic or extra rollouts. Reported gains exceed 12 points on ALFWorld and 9 points on WebShop over GRPO at the same rollout and GPU-memory budget.5↗
Limitations. “No extra rollouts” means no extra relative to grouped GRPO—not few rollouts. Exact repeated states are common in deterministic text simulators and scarce in camera-based robotics. It cannot naturally exploit an arbitrary historical behavior mixture.
Exact transfer. In simulation, reset to saved states and branch 4–8 language subtasks to create a clean action-ranking set. In reality, use naturally recurring canonical states or learned state clusters cautiously; do not make grouped branching the core real-data pipeline. Code.
Mechanism/evidence. StarPO structures state–thinking–action–reward trajectories. StarPO-S adds trajectory filtering, a critic, and decoupled clipping after observing an “Echo Trap”: loss of reward variance accompanied by gradient spikes. Three stylized environments support useful findings—diverse initial states, medium action granularity, frequent resampling, and fine-grained reasoning-aware reward matter.6↗
Limitations. Primarily a stability/diagnostic contribution on small synthetic environments; still rollout-centric, and its reward-shaping conclusions may be environment-specific.
Exact transfer. Instrument reward variance, gradient norms, action/reason repetition and subtask diversity. Reject degenerate trajectories and vary initial object configurations. Treat it as an experimental hygiene checklist, not the core algorithm. Code/environments.
Mechanism/evidence. Recasts a bandit-like trajectory objective as an MDP and estimates finer turn advantages. On search/tool-use tasks, the paper reports 100% tool execution and 50% exact match versus baselines with no tool invocation and 20–30% exact match.7↗
Limitations. Strong local results but limited environments; does not solve replay or distribution shift. Credit quality depends on trustworthy intermediate reward or a learned value estimator.
Exact transfer. Use as a controlled on-policy baseline showing whether temporal advantages themselves help. If it beats terminal-only GRPO but loses badly in real interactions per episode, that supports the off-policy thesis.
Mechanism/evidence. Merges identical states across trajectories into a global transition graph, estimates state values from grouped visits, and uses edge-centric TD errors. The preprint reports gains up to 22.2% over GRPO on WebShop, ALFWorld, and AppWorld.8↗
Limitations. Very recent preprint; identical observations need not be Markov-equivalent, especially under hidden robot state. Graph construction presumes enough revisitation.
Exact transfer. Build an offline semantic transition graph over object-centric scene states, but merge only when task, scene state, held objects, subtask outcome, and relevant history agree. Use graph TD error as a replay priority, not as the sole truth.
Mechanism. Treats arbitrary agent execution as an MDP, records calls as structured spans, decomposes trajectories into per-call training transitions, and decouples execution from training. Microsoft reports experiments on text-to-SQL, RAG and tool-use; the open-source system supports external harnesses.9↗
Limitations. Much of the contribution is infrastructure. Decomposition alone is not a solution to long-horizon credit; the actual reward propagation/optimizer choice remains decisive. Therefore it is included as a useful implementation layer, not ranked as a scientific blueprint.
Exact transfer. Adopt its trace schema idea: log VLM prompt, image/memory digest, proposed subtask, safety gate, VLA result, duration, intervention, and terminal outcome as separable spans. Feed those records to ArCHer/IQL rather than assuming the middleware’s default learner solves the problem. Code.
Mechanism/evidence. CHAI combines a pretrained language model with offline RL to pursue task goals from static human dialogue. Earlier human-centric offline dialogue RL adds KL-control to a language prior and pessimism under uncertainty, explicitly targeting overestimated out-of-distribution actions.10↗, 11↗
Limitations. Dialogue dynamics, reward quality, and action consequences differ substantially from physical manipulation. These methods predate modern VLMs and evaluate relatively constrained language domains.
Exact transfer. Pretrain only within the support of demonstrated/replayed language subtasks; add behavior-model KL or conservative Q penalties; require human/VLM paraphrase augmentation so “support” reflects semantic equivalence rather than exact wording. CHAI code.
Mechanism/evidence. Guided MCTS branches from agent states, a self-critic supplies process guidance, outcomes are backed up, and an off-policy DPO variant learns preferences from both successes and failures. The paper reports large gains in WebShop and booking tasks after iterative collection.12↗
Limitations. Search multiplies interactions; reported booking numbers involve web environments and online search, not irreversible physical actions. DPO preferences do not necessarily produce a calibrated Q.
Exact transfer. Run MCTS only in simulation or a learned world model; distill pairwise “subtask a beats b at state s” comparisons into the VLM, then validate the best branch once in reality. Do not branch physical rollouts merely to produce preferences.
Mechanism/evidence. Trains action-conditioned textual world models to align imagined and realized successor states in representation space. Combined with success reward, it reports +6.9 points on ALFWorld and +5.7 on τ²-Bench over direct success-reward RL, while matching expert-data training.13↗
Limitations. Text-state semantic distance is much easier than predicting contact-rich visual dynamics. A plausible verbal successor can conceal an impossible grasp or irreversible physical failure.
Exact transfer. Predict symbolic/object-centric deltas and option success—not pixels. Train from every transition. Use uncertainty to veto model-planned candidates; periodically recalibrate on real transitions.
Mechanism/evidence. CALM learns an action proposal distribution from human play and lets an RL agent re-rank the compact candidate set; it reported a 69% relative score improvement on unseen Jericho games.14↗ WebShop supplies a stateful web MDP with 1.18M products and 12,087 crowd instructions, plus IL, RL, and IL+RL baselines and public code/data.15↗
Limitations. Candidate bottlenecks cap performance when the right action is never proposed. Text games/websites have cheap resets and exact rewards.
Exact transfer. Separate proposal from evaluation: a pretrained VLM produces 4–16 short, executable subtask candidates; Q ranks them after feasibility filtering. Always include recovery, observe, and terminate candidates.
Mechanism/evidence. Candidate skill score is the product of LLM usefulness and a learned affordance/value estimate; selection repeats after execution. It completed abstract long-horizon instructions on a real mobile manipulator.2↗
Why only adjacent. It does not learn a high-level policy from long-term task return, and the original system receives environment grounding through current-step skill values; its project page explicitly notes that failures or environmental changes may not be available to planning.
Exact transfer. Preserve SayCan’s feasibility factor p(success of VLA | s,a), but multiply/add it with a learned high-level Q for downstream task completion. Do not confuse option feasibility with task value: “can pick cup” is not “should pick cup now.”
Mechanism/evidence. Interactive Policy Optimization trains from grouped in-environment exploration and sparse completion. Reported completion is 97.78% on ALFWorld and 79.92% on ScienceWorld, with a 3.66-point drop on unseen environments.16↗
Limitations. It explicitly relies on parallel exploration—the opposite of this project’s real-world constraint—and uses text simulations with resettable state. Despite “embodied” branding, it is not physical VLM control.
Exact transfer. Include only as an upper-bound simulation comparator and a reminder that pure sparse success can work when rollouts are abundant. It should not determine the real-robot algorithm.
Mechanism/evidence. MCTS-like selection/expansion/simulation uses actual environmental feedback, self-reflection and outcome backup. It reported strong results including 75.9 WebShop average score; code and trajectories are public.17↗
Limitations. It is not RL training and consumes multiple environment branches. LM self-values can be miscalibrated.
Exact transfer. Its node/value interface is useful for simulator evaluation and for planning inside a learned symbolic world model. Distill resulting branches to Q/policy; real execution remains single-path.
Recommended core
One replay transition per executed VLA option; double Q over (visual/history state, language subtask); expectile V; AWR or weighted SFT actor. Highest direct evidence for off-policy multi-turn language learning and most compatible with heterogeneous logged robot behavior.
Safety layer
Factor downstream task return from immediate executability. CQL/KL support penalty prevents attractive but unsupported instructions. The low-level VLA success model can be pretrained independently and updated from every attempted option.
Research upside
Scalar critic makes ranking and calibration testable; language successor/critique generates repairs and supplies interpretable failure hypotheses. More speculative than ArCHer, but unusually well matched to open-ended instruction actions.
Auxiliary
Agent Q or GiGPO-style branches from checkpointed simulator states generate action comparisons; distill them by DPO/AWR, then run only the chosen path on hardware. This converts cheap branching into offline data without demanding physical resets.
Experimental
G2PO-inspired merging may pool repeated scene configurations and identify pivotal subtask edges. Requires robust object-centric state identity and tests against perceptual aliasing.
Baseline only
Valid multi-turn method and useful scientific control, but group rollouts are costly and old data expires. Use in simulation to measure the value of turn credit; do not center real-world learning on it.
Important distinction. The low-level option-success model answers “will this VLA execute the instruction from here?” Q answers “does choosing it now increase eventual task return?” They must be separately supervised and jointly used.
Segment demonstrations and autonomous runs at semantic option boundaries. Log failures, interventions, duration, and terminal outcomes—not only successes.
Behavior-clone the proposal model; train feasibility, double Q, expectile V, and AWR actor. Use simulator branches and paraphrase augmentation.
Deploy conservatively, prioritize uncertain/high-value states, add transitions to replay, update critic frequently and actor slowly behind safety gates.
Paraphrases fragment Q data; superficially similar instructions differ in grounding. Canonicalize slots and regularize paraphrase Q consistency while retaining raw text.
Same image can hide grasp force, prior failed attempts, or object damage. Include belief, option result, recent actions, and uncertainty; test recurrent memory against snapshot state.
Q overvalues unseen, eloquent instructions. Use conservative Q, behavior likelihood, ensemble disagreement, and low-level feasibility; force an ask/observe fallback.
Easy skills dominate even when irrelevant. Train feasibility only on local completion and Q only on downstream task return; report both calibrations.
If the low-level policy changes, replay transition dynamics become stale. Version every VLA checkpoint; condition critics on version or freeze VLA during high-level RL phases.
Progress detectors reward moving objects without satisfying relations. Use terminal predicates, negative tests, intervention cost, and video audits of top-Q failures.
Fixed-step discount treats 2-second and 30-second options equally. Use semi-MDP γk and report wall-clock/robot-time efficiency.
Branch-generated policy learns visual or dynamics artifacts. Domain randomize, keep real-only validation, and down-weight synthetic transitions using uncertainty.
A VLM actor and VLM critic share blind spots. Use separate seeds/architectures, grounded predicates, double critics, and human review of disputed high-impact actions.
| Experiment | Comparison | Primary metric | Decision it resolves |
|---|---|---|---|
| E1 · Q vs V vs no critic | Q candidate ranker; V + sampled lookahead; SayCan-style feasibility×LM; VLM policy alone | Success per 100 real episodes; calibration/Brier score | Whether action-conditioned long-term value is necessary. |
| E2 · data reuse | Offline IQL/AWR; online ArCHer; turn-level GRPO; terminal GRPO | Learning curve by environment transitions, not optimizer steps | Whether off-policy reuse creates the claimed real-rollout advantage. |
| E3 · critic factorization | Task Q only; feasibility only; product/additive combination; joint monolithic score | Invalid-action rate, task success, recovery success | Whether separating “can” and “should” matters. |
| E4 · offline support | BC; IQL; IQL+conservative penalty; unconstrained Q actor | OOD action rate and real success | Whether conservative control prevents language extrapolation. |
| E5 · state/history | single frame; frame stack; object belief; recurrent episodic memory | Success on tasks with hidden state/repeated mistakes | What must be in the high-level state. |
| E6 · sim branching | no synthetic; random alternatives; Q-uncertain alternatives; Agent-Q/MCTS alternatives | real transfer per synthetic transition | Whether branch data helps or induces sim bias. |
| E7 · language action form | fixed skill IDs; structured language slots; free-form instruction | held-out compositional tasks + VLA execution validity | Whether language adds generalization beyond a discrete option library. |
| E8 · consequence critique | scalar Q; NLAC-like critique; scalar+critique | recovery quality and human-rated causal accuracy | Whether language-valued criticism adds actionable signal. |
| E9 · frozen vs changing VLA | frozen; periodically fine-tuned; joint alternating updates | critic calibration under policy versions | How damaging non-stationary low-level dynamics are. |
| E10 · generalization | train task compositions A/B/C; test unseen D/E/F sharing atomic skills | zero/few-shot success and subtask-sequence edit distance | Whether RL learns strategy rather than memorized scripts. |
Go/no-go criterion. If offline Q does not outperform behavior cloning under a fixed real-transition budget and its calibration does not predict failures, do not scale real-world RL. Improve state representation/reward/support before collecting more episodes.
No included agentic-LLM RL paper demonstrates off-policy training of a visual high-level language planner over a real manipulation VLA under a deliberately small rollout budget. Most “embodied” evidence is textual ALFWorld/ScienceWorld; the closest physical system, SayCan, does not RL-train its high-level LM.
Scalar critics are easy to rank and calibrate but hard in open language spaces; language critics are expressive but hard to verify. A hybrid is promising, not settled.
GiGPO/G2PO assume repeated or mergeable states. Visual manipulation needs a defensible equivalence relation over observations, latent object state, and history.
Leaderboards usually report success or score, not success per real transition, robot-minutes, irreversible failures, intervention load, or calibration—the metrics that should govern this project.
Cross-cutting disagreement. Recent group-policy work argues that a learned critic is avoidable and potentially unstable; ArCHer/NLAC argue that explicit temporal value is precisely what yields sample efficiency. Both can be true in cheap simulators. Under expensive physical data, the decisive axis is not GPU memory but how many independent environment transitions an update can reuse; this strongly favors off-policy critics unless their extrapolation error dominates.
Search date: 3 September 2026. Default related-work depth: 1. Queries combined “multi-turn/agentic/interactive/embodied” with “reinforcement learning, actor critic, Q learning, offline, off-policy, credit assignment, replay, tree search, world model, language action.” Anchors were resolved to arXiv/OpenReview/PMLR/ACL, official project pages, and official GitHub repositories. Backward/forward neighbors were semantically filtered rather than included by citation proximity.
venue peer-reviewed venue or proceedings. preprint not treated as established. system implementation evidence. Numeric claims in this report are author-reported unless explicitly stated; no cross-paper leaderboard comparison is implied because models and environments differ.
Ordinary RLHF/RLVR, single-turn reasoning, prompt-only agents, pure harness papers, and low-level robot control were not enumerated. WebGPT is a historical interactive-RL anchor but uses largely trajectory/answer-level human preference optimization and lacks the multi-turn temporal design required here, so it is bibliographic context rather than a recommended method.19↗
Several 2026 works are extremely recent and only available as preprints. Full reproducibility was not independently rerun. Citation counts were deliberately not used because age dominates. Some commercial agent-training details are not public. Robotics transfer statements are labeled as this survey’s inference, not authors’ claims.
Search stopped after all requested families had at least one primary representative; repeated queries yielded variants of grouped policy optimization or generic RLVR rather than new off-policy high-level mechanisms; and the main design alternatives—Bellman critic, conservative offline policy, group-relative turn credit, search/model, and affordance factorization—were saturated. The remaining gap is empirical, not another citation branch.