Beyond Imitation:
Self-Improving Robot Policies via Off-Policy Q-Planning

Varun Giridhar, Anant Khandelwal, Jeremy A. Collins, Ignat Georgiev, Animesh Garg

arXiv Code BibTeX
Q-Planning overview diagram
Q-Planning overview. Left: at inference, the frozen BC policy samples N candidate action chunks; the Q-function scores them, and the executed chunk is a single-step Q-weighted average. Right: rollouts (successful and failed) are appended to a replay buffer and used to fine-tune only the Q-function. The updated $Q_\phi$ returns to the next inference iteration; the BC policy is never updated.

Abstract

Behaviour Cloning (BC) has driven remarkable progress in robot manipulation, yet it is fundamentally limited by its inability to self-improve: a policy that fails cannot learn from that failure without additional human demonstrations. Reinforcement Learning fine-tuning offers a path to self-improvement but has proven difficult to scale to the multi-billion-parameter models underpinning modern robot policies. We propose Q-Planning, which equips a large visuomotor BC policy with a small off-policy Q-function. Because a Q-function estimates value rather than imitates actions, it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts, an asymmetry BC does not have. We exploit this asymmetry to enable value-guided action selection at inference (a single-step $Q$-weighted average over BC draws) and online self-improvement that fine-tunes only the Q-function, leaving the BC weights untouched. On LIBERO and bimanual RoboTwin, ten iterations of self-improvement lift every benchmark score we tested (LIBERO-10 93 → 99%, RoboTwin 83.8 → 91.4%) and shorten successful episodes on the near-ceiling suites (LIBERO-Object, LIBERO-Goal). On two contact-rich bimanual real-robot tasks, the same loop (BC frozen, no human intervention) improves purely from its own deployment rollouts: stack-cups 40 → 90% and insert-wallet 25 → 80% in five iterations, whereas SFT on successful rollouts alone stalls at 55% and 30%. Under an identical online budget Q-Planning is the only method, among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR, that improves stably from failures without training an auxiliary actor.

TL;DR

  1. An off-policy Q-function for large BC policies. A Q-chunking architecture with HL-Gauss categorical outputs and its own DinoV2 and T5 encoders, trained on the same demonstrations used to train the BC policy.
  2. A real-time value-guided action selector. A single-step Q-weighted average over N BC flow-matching draws, amortising the encoders across all candidates so a planning step costs 400ms on RoboTwin (1.6× faster than a single 10-step BC inference and comfortably inside the 960 ms replan budget).
  3. Self-improvement from failures without touching the BC. A loop that folds successful and failed deployment rollouts into Q-only updates: LIBERO-10 93 → 99%, RoboTwin 83.8 → 91.4%, and on a real bimanual robot stack-cups 40 → 90% and insert-wallet 25 → 80%.

Self-improvement on a real robot

Two contact-rich bimanual tasks: stack-cups, stacking plastic cups, and the harder insert-wallet, slotting a credit card into a wallet. From 100 base demonstrations and 20 online episodes per iteration, with the BC policy frozen and no human intervention, Q-Planning improves purely from its own deployment rollouts.

Where Q-Planning recovers

A BC policy can only be trained on successful demonstrations. An off-policy Q-function has no such restriction: it can be trained on any trajectory, successful or failed, because it estimates value rather than imitates actions.

✗ frozen BC

✓ Q-Planning

Q-Planning, iteration 5

Same task, same frozen BC policy on both sides. Only the Q-function changed.

Method

Two components: an off-policy Q-function over action chunks, and a self-improvement loop that fine-tunes only that Q-function. At every stage the base BC policy is frozen.

1. Off-policy Q-function over action chunks

Standard scalar Q-regression under sparse terminal rewards is unstable; most transitions carry no reward signal, and the long horizons of manipulation tasks amplify bootstrapping error. We therefore (i) treat the length-H action chunk as a single super-action so the effective bootstrapping horizon shrinks by a factor of H, and (ii) replace scalar regression with HL-Gauss categorical regression, which projects each scalar target onto a fixed grid of bins via a Gaussian kernel and trains the Q-network with a cross-entropy objective.

Q-function architecture: DinoV2 and T5 encoders feed a transformer decoder that cross-attends to visual and language tokens and takes the action chunk as query tokens; the decoder output goes through an HL-Gauss head to produce Q.
The Q-function has its own DinoV2 visual encoder and T5 language encoder (parameter-disjoint from the BC policy). A transformer decoder cross-attends to visual and language tokens and takes the candidate action chunk as query tokens. The HL-Gauss head outputs B bin logits over discounted returns.
$$ Q_\phi(o_t, \ell, a_{t:t+H}) \;=\; \sum_{b=1}^{B} v_b \cdot \operatorname{softmax}\bigl(\ell_\phi(o_t,\ell,a_{t:t+H})\bigr)_b $$

At inference we draw N action chunks from a truncated 3-step flow-matching pass of the frozen BC, score each with $Q_\phi$, and execute a softmax $Q$-weighted average with temperature $\lambda$ rather than any single sample: averaging beats selecting.

$$ w^{(n)} \propto \exp\bigl(Q_\phi(o_t, \ell, a^{(n)}_{t:t+H})/\lambda\bigr), \qquad \bar{a}_{t:t+H} = \sum_{n=1}^{N} w^{(n)}\, a^{(n)}_{t:t+H} $$

2. Self-improvement loop

A BC policy trained only on successful demonstrations has never seen the failure modes that emerge under autonomous execution. Q-Planning closes this loop by deploying the planner, collecting both successful and failed rollouts, and using them to refine only the Q-function. Because only $Q_\phi$ (∼1B parameters) is updated, one self-improvement iteration is dramatically cheaper than a full-policy gradient step.

Algorithm 1 · Q-Planning self-improvement
  1. Input: BC dataset $\mathcal{D}_{\text{BC}}$, frozen policy $\pi^{\text{BC}}$, Q-function $Q_\phi$, target $Q_{\bar\phi}$
  2. $\mathcal{D} \leftarrow \mathcal{D}_{\text{BC}}$
  3. for iteration $i = 1, 2, \ldots$ do
  4. // Phase 1: collect rollouts under Q-Planning (BC frozen)
  5. for task $k \in \mathcal{T}$, episode $j = 1, \ldots, M$ do
  6. Reset environment; receive $\mathbf{o}_0$
  7. while episode not done do
  8. Draw $N$ chunks from $\pi^{\text{BC}}$, score with $Q_\phi$, and take the $Q$-weighted average $\bar{\mathbf{a}}_{t:t+H}$
  9. Execute $\bar{\mathbf{a}}_{t:t+H}$; observe $r_{t:t+H},\; \mathbf{o}_{t+H}$
  10. end while
  11. Append full episode to replay buffer $\mathcal{D}$
  12. end for
  13. // Phase 2: refine only $Q_\phi$ (BC frozen)
  14. for $s = 1, \ldots, S$ do
  15. Sample minibatch $\mathcal{B} \sim \mathcal{D}$ and update: $\phi \leftarrow \phi - \alpha\, \nabla_\phi\, \mathcal{L}(\phi;\mathcal{B})$
  16. EMA target update: $\bar{\phi} \leftarrow \eta\, \phi + (1 - \eta)\, \bar{\phi}$
  17. end for
  18. end for

Results

Benchmark results

Before any environment interaction, Q-Planning already improves over the frozen FastWAM BC policy on 4 of 5 benchmarks, by +1.3pp on average. Ten iterations of self-improvement then lift success wherever there is headroom (LIBERO-Spatial to 98.5%, LIBERO-10 to 99%, bimanual RoboTwin 83.8 → 91.4%) while shortening successful episodes on the near-ceiling suites, where success has no room to grow (LIBERO-Object 139 → 120, LIBERO-Goal 110 → 99).

Benchmark FastWAM Q-Planning (offline) Q-Planning (online)
Success ↑ Ep. len. ↓ Success ↑ Ep. len. ↓ Success ↑ Ep. len. ↓
LIBERO-Spatial 90.5106 91.5115 98.5107
LIBERO-Object 100.0138 99.5139 100.0120
LIBERO-Goal 97.0107 99.0110 99.099
LIBERO-10 90.0274 93.0261 99.0224
RoboTwin (47 tasks) 83.2220 83.8231 91.4232
Mean 92.1— 93.4— 97.6—

Both Q-Planning columns use the same frozen BC policy; only the online one collects rollouts and fine-tunes $Q$. The offline column is $Q$-weighted selection over BC draws with no self-improvement; the online column is the same policy after ten iterations of the loop.

Each cell in the online column is the endpoint of a ten-iteration run rather than a single measurement. RoboTwin climbs from 83.8% to 91.4% and LIBERO-Spatial reaches 98.5%; the two suites already at or above 99% have no room left in success, so the loop instead shortens successful episodes: the Ep. len. column above.

Against other self-improvement methods. Every method here starts from the same frozen BC policy and gets the same online budget: ten iterations of 100 rollouts per task, run on LIBERO-10. Best-of-N (the same loop with argmax selection in place of the weighted average) plateaus at 95%: value-guided selection captures part of the gain, but only the Q-weighted average plus self-improvement carries success to 99%. SFT on successes, the direct test of “learns from failures”, plateaus at 93.5%: re-imitating only successes cannot absorb the failure signal $Q$ can. IBRL collapses, DSRL swings between 69% and 91%, and DAWR hovers below the frozen BC. The clips show one LIBERO-10 task over the first six iterations.

Positioning against the closest methods

V-GPS steers a frozen policy with an offline-trained value and never iterates online. DSRL adapts online but through a latent-noise auxiliary actor. IBRL keeps its IL policy frozen but trains a fresh RL actor from scratch, which at VLA scale means re-learning behaviour the BC already encodes; DAWR, the advantage-weighted regression baseline from the DPPO paper, fine-tunes the policy weights directly, which is expensive at scale and risks degrading the BC prior. Q-Planning alone is BC-frozen, plug-and-play with a multi-billion-parameter policy, and self-improving from failures with no auxiliary actor.

Method BC frozen Plug-and-play large VLA Learn from failures Offline-to-online $Q$ No aux. actor
V-GPS✓✓✗✗✓
DSRL✓✓✓✓✗
IBRL✓✗✓✗✗
DAWR✗✗✓✗✓
Q-Planning (ours) ✓✓✓✓✓

Planning latency

Drawing candidates with 3-step flow matching makes planning real-time: 400ms/step on RoboTwin at N = 32, 42% of the 960 ms replan budget and 1.6× faster than a single 10-step BC inference. On LIBERO it is latency-neutral: the deployed N = 64 configuration costs 640 ms, essentially matching the 646 ms the frozen-BC baseline pays for one 10-step draw.

Full latency profile (1× NVIDIA L40S, median ms over 60 timed steps)

We profile a full Q-Planning planning step (one action chunk) on a single NVIDIA L40S GPU, batch 1, median + p95 over 60 timed steps after 10 warm-up steps, with torch.cuda.synchronize around every timed region. The strict setting uses lerobot-eval parity (deterministic algorithms, TF32 off, cudnn.benchmark off, BC in bf16, $Q$ in fp32), which reproduces the reported success rates. The replanning cadence is the deadline a planning step must beat: the planner emits an H = 32-step chunk, of which the first 10 (LIBERO, 30 Hz) or 24 (RoboTwin, 25 Hz) steps execute before replanning, giving budgets of 333 ms and 960 ms.

Configuration$Q$-evals LIBERO strict% RoboTwin strict%
BC only, 3 denoise steps0273 (275)82273 (275)28
BC only, 10 steps (eval baseline)0646 (647)194640 (643)67
Q-Planning, N=16, 3 steps16337 (338)101352 (354)37
Q-Planning, N=32, 3 steps ◂ RoboTwin32371 (372)111400 (402)42
Q-Planning, N=64, 3 steps ◂ LIBERO64640 (641)192716 (725)75
Earlier iterative sample-based variant (3 iters, N=64)1921114 (1117)3341276 (1282)133

Planning-step latency (ms) on an NVIDIA L40S. Median over 60 timed steps; parenthesised numbers are p95. Percentages express the fraction of the replan budget (LIBERO 333 ms, RoboTwin 960 ms). Strict = eval parity settings. Rows marked ◂ are the deployed configurations. The final row is an earlier iterative sample-based variant we prototyped, retained here for comparison.

Findings. (i) Under strict eval-parity settings, RoboTwin is comfortably real-time: the deployed N = 32 planner uses only 42% of the 960 ms budget, and is 3.2× faster than the earlier iterative variant, which would have missed the budget at 133%. (ii) LIBERO is latency-neutral relative to its own frozen-BC baseline: the deployed N = 64 configuration takes 640 ms per step, essentially matching the 646 ms cost of a single 10-step BC draw the baseline itself pays. Lowering to N = 16 fits the 333 ms budget with comparable success. (iii) Encoders (DinoV2 + T5) run once per planning step at 23–27 ms; only the ∼500M $Q$ decoder scales with N (roughly 2–3 ms per candidate), which is why doubling N does not double the total step cost.

Q-value trajectories

Q-value over time on two LIBERO-10 episodes. On success, Q climbs from about 0.26 to 0.63; on failure it oscillates in a lower range and never converges.
Q-value over time. On a successful rollout $Q_\phi$ climbs from ∼0.26 to ∼0.63; on a failure it oscillates in a lower range and never converges, a useful sanity signal for the value estimate.

Limitations

BC policy dependence and exploration boundary. Q-Planning cannot bootstrap from scratch: any behaviour the BC head cannot produce with non-negligible probability is a behaviour the planner cannot select and the loop cannot learn to exploit. Multi-modal flow-matching draws widen this envelope beyond a single Gaussian around the BC mode, but tasks where the base BC generates no successful chunks at all remain out of reach. Gains therefore scale with BC quality and diversity.

$Q$-decoder scaling with N. The BC and $Q$ encoders are amortised once per planning step, but the $Q$ decoder scales linearly with the candidate count N (∼2–3 ms per candidate on an L40S). Our deployed configurations (N = 64 on LIBERO, N = 32 on RoboTwin) fit inside the frozen BC baseline's latency budget, but pushing N toward the hundreds to attack harder exploration would eventually make the decoder pass, not the BC draw, the bottleneck.

Terminal-reward supervision. Our loop assumes a per-episode success detector: the environment success bit in simulation and a human-provided per-episode label on the real robot. Extending to open-ended tasks would require language-conditioned or learned success models.

BibTeX

@misc{qplanning2026,
  title         = {Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning},
  author        = {Varun Giridhar and Anant Khandelwal and Jeremy A. Collins and
                   Ignat Georgiev and Animesh Garg},
  year          = {2026},
  eprint        = {2608.21204},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO}
}