Nav4Manip: From Navigation to Dexterous Mobile Manipulation through Co-Evolution

Anonymous Submission
REBUTTAL

Thank you for your careful reading of Appendix D.2 and for framing the evaluation-cell filtering boundary as a clear, constructive question. We address each part of your inquiry below.

Q1:

What exactly causes rejection before an affordance evaluation cell is created? ... Please distinguish invalid initializations, rejected action candidates within a task, and entire valid initializations for which the dock permits no feasible action.

A1:To directly answer your question: only physically invalid object initializations (Category 1) can be rejected before cell creation; a valid object initialization is never omitted even if the current dock admits no feasible manipulation target (Category 3).Show moreShow less

To directly answer your question: only physically invalid object initializations (Category 1) can be rejected before cell creation; a valid object initialization is never omitted even if the current dock admits no feasible manipulation target (Category 3).

Specifically, the three categories you highlighted are distinguished and handled as follows:

  • Case 1 (Invalid initializations): When constructing the simulation scenes, we enforce physical validity constraints and collision checks on initial object placements and randomization. Configurations with physical penetration or out-of-bound placement are prevented during scene generation and never enter the evaluation pipeline.
  • Case 2 (Rejected action candidates within a task — feasible plan, failed physics verification): For valid object configurations, if the planner generates a trajectory that subsequently fails physics verification (e.g., due to grasp instability), the attempt creates a record in the Affordance Evaluation Cell, is scored as 0, and is counted in the denominator of Eq. (34).
  • Case 3 (Valid initializations where the dock permits no feasible action): If the object configuration is valid but the candidate dock is too far or poorly positioned for the planner to find any feasible trajectory, the attempt also creates a record in the Affordance Evaluation Cell, is scored as 0, and is fully included in the denominator of Eq. (34).
ζi,j,ceval=1[∃φ∈Φ^i,j,c:Yi,j,c,φeval=1]q(bi,j)={1Ni,jeval∑c=1Ni,jevalζi,j,ceval,Ni,jeval>0,0,Ni,jeval=0.(34)
Q2:

If the last category can be excluded, how is it accounted for when comparing candidate docks? The frequency of such exclusions and their effect on the selected endpoints would help determine whether the scores remain comparable across docks.

A2:Because Category 3 cases are not excluded prior to cell creation, all candidate docks for a given task are evaluated over the exact same set of valid initial object configurations. To provide quantitative transparency on how frequently each outcome occurs, we audited all candidate dock evaluations across our training set:Show moreShow less

Because Category 3 cases are not excluded prior to cell creation, all candidate docks for a given task are evaluated over the exact same set of valid initial object configurations. To provide quantitative transparency on how frequently each outcome occurs, we audited all candidate dock evaluations across our training set:

  • Case 1 (Invalid initializations): 0%
  • Case 2 (Planning succeeded, physics verification failed): 19.72%
  • Case 3 (No feasible plan due to poor or distant dock pose): 67.69%
  • Successful execution: 12.59%

As these statistics show, Category 3 accounts for the majority of failed trials (67.69%), and including them in the denominator is essential for properly penalizing unsuitable docks.

Grasp Explorer · Simulation

Open in a new tab ↗

Results from inference in simulation.

A bidirectionally coupled co-evolution system for long-horizon dexterous mobile manipulation.


Abstract

Dexterous mobile manipulation is essential for robots to perform useful tasks in complex, unstructured environments. However, existing systems face two challenges. First, navigation remains too coarse for downstream manipulation, which often requires a precise base position and approach orientation rather than merely reaching a meter-scale goal region. Second, navigation and manipulation are commonly trained in isolation, lacking a feedback loop that enables the two capabilities to co-evolve. These limitations prevent current systems from reliably executing long-horizon dexterous mobile-manipulation tasks in real-world environments. To address them, we introduce Nav4Manip, a bidirectionally coupled co-evolution system for long-horizon dexterous mobile manipulation. It comprises two complementary architectures: Nav4Manip-N learns long-horizon navigation and refines its policy through manipulation feedback, while Nav4Manip-M trains a world-action model (WAM) for downstream dexterous manipulation. We train Nav4Manip-N in three stages: multi-task co-tuning learns navigation behavior and manipulation-aware affordance prediction; rejection sampling fine-tuning (RFT) rejects low-quality policy-generated paths and fine-tunes the model on retained trajectories; and reinforcement learning from AI feedback (RLAIF) uses counterfactual preferences to further improve navigation decisions. To support this framework, we establish the Nav-N-Manip benchmark and construct Nav4Manip-Pairs from benchmark rollouts, providing large-scale paired navigation–manipulation trajectories and affordance supervision. Extensive experiments on public vision-language navigation (VLN) benchmarks and Nav-N-Manip demonstrate state-of-the-art performance, while real-world evaluations show that Nav4Manip completes diverse dexterous mobile-manipulation tasks.



Method

Nav4Manip architecture with navigation, affordance estimation, manipulation, and selective RLAIF feedback

Our framework trains its navigation model in three stages: multi-task co-tuning learns navigation behavior and manipulation-aware affordance prediction; rejection sampling fine-tuning (RFT) rejects low-quality policy-generated paths and fine-tunes the model on retained manipulation-successful trajectories; and reinforcement learning from AI feedback (RLAIF) uses execution-grounded counterfactual preferences to further improve navigation decisions.



Nav-N-Manip and Nav4Manip-Pairs

Data generation pipeline from scene assets and manipulation affordances to navigation rendering, language annotation, and paired trajectory data

We establish the Nav-N-Manip benchmark and construct Nav4Manip-Pairs from benchmark rollouts, providing large-scale paired navigation–manipulation trajectories and rollout-derived affordance supervision.



Real-World Evaluation

Real-world evaluations show that Nav4Manip completes diverse dexterous mobile-manipulation tasks.



Mobile Manipulation Rollouts

Navigation and dexterous manipulation in simulation (4× speed).



Navigation Rollouts

First-person navigation rollouts, including instruction following and object-goal navigation.



Manipulation Rollouts

First-person manipulation rollouts in simulation.



R2R Navigation Comparisons

Navigation rollouts on R2R.



RxR Navigation Comparisons

Navigation rollouts on RxR.



VLNVerse Navigation Comparisons

Navigation rollouts on VLNVerse.