Using VLM Reasoning to Constrain Task and Motion Planning

1Robotics Institute, Carnegie Mellon University, 2Department of Computer Science, Purdue University, 3Department of Computer Science, Rice University
*Equal contribution
VIZ-COAST overview: a VLM reads an example instance and the PDDL domain, writes constraints.py, and the Z3 planner applies those constraints to an unseen instance.

VIZ-COAST uses VLMs and SMT-based task planning to constrain TAMP problems. Our constraints generalize to unseen problem instances within a broader domain.

Abstract

In task and motion planning, high-level task planning is done over an abstraction of the world to enable efficient search in long-horizon robotics problems. However, the feasibility of these task-level plans relies on the downward refinability of the abstraction into continuous motion. When a domain's refinability is poor, task-level plans that appear valid may ultimately fail during motion planning, requiring replanning and resulting in slower overall performance.

Prior works mitigate this by encoding refinement issues as constraints to prune infeasible task plans. However, these approaches only add constraints upon refinement failure, expending significant search effort on infeasible branches. We propose VIZ-COAST, a method of leveraging the common-sense spatial reasoning of large pretrained Vision-Language Models to identify issues with downward refinement a priori, bypassing the need to fix these failures during planning.

Experiments on three challenging TAMP domains show that our approach is able to extract plausible constraints from images and domain descriptions, drastically reducing planning times and, in some cases, eliminating downward refinement failures altogether, generalizing to a diverse range of instances from the broader domain.

Method

VIZ-COAST architecture: the Visual Reasoning Module produces constraints for the SMT-based task planner, whose plans are grounded by a streams-based planner.

VIZ-COAST consists of three components:

  • SMT-based task planner. A task planner built on the Z3 SMT solver with a Madagascar or SATPlan encoding, which allows flexible constraint specification while introducing minimal search overhead.
  • Visual Reasoning Module (VRM). Before planning begins, the VRM uses a VLM to analyze an example scene (image and geometric state), the PDDL domain and problem, and the grasp specification. It foresees downward-refinement failures and infers general constraints to prevent them, written as a Python file of constraints on the SMT solver.
  • Stream-based grounding. Task plans are grounded to motion with the streams framework used in PDDLStream and COAST.

On a new problem, the task planner applies the constraints to block infeasible task plans. After K refinement failures, the VRM revises its constraints using the cached failure feedback.

Paraphrased example of a VRM-generated constraints file that blocks picking from and placing into closed containers.
Paraphrased example of a constraints file produced by the VRM to block picking and placing from closed containers. The VRM produces a function that calls a high-level API to block action assignments under certain conditions, constraining the search space of the task planner.

Domains

The Shelf, Packing and Cooking domains in simulation.
  • Shelf. Rearrange blocks in a restrictive shelf to move the red block to the center tile. The shelf only allows grasps from one direction, so any block in the approach corridor causes a collision, which the symbolic domain does not encode.
  • Packing. Pack slabs, big blocks and small blocks into a container and close the lid. The symbolic domain ignores the container's limited volume, so items must be stacked in a configuration that fits under the lid.
  • Cooking. Two arms use a frying pot and a steaming pot to cook and serve ingredients. The symbolic domain neglects lids blocking pots, each arm's limited reach, unstable lid stacking, and lids occupying plates.
Executing a VIZ-COAST plan in the Packing domain.

Results

Shelf

Success rates, refinement failures and planning times in the Shelf domain.
VIZ-COAST was prompted once on the first 3-block problem, producing constraints that generalized to all 30 unique problems containing up to 8 blocks. It plans faster than COAST-based baselines while avoiding refinement failures.

Packing

Success rates, refinement failures and planning times in the Packing domain.
VIZ-COAST achieves 100% success and the fastest planning times on larger problem instances, whereas COAST-based methods become increasingly slower or fail completely. The failures of the no-feedback ablation on larger problems show the importance of closing the constraint generation–planning loop. As VIZ-COAST refines its constraints over problems, it encounters fewer refinement failures and plans faster.

Cooking

Success rates, refinement failures and planning times in the Cooking domain.
VIZ-COAST achieves 100% success and the fastest planning times, whereas COAST-based methods and the no-feedback ablation fail completely beyond 2 ingredients.

Choice of VLM

Success rate, average time per constraint update (Prompting), and average total downward-refinement failures over 20 independent trials of 4 sequential Shelf problems with 4–7 blocks. Metrics are averaged over runs in which all problems succeeded.

VLMSuccessPrompting (s)Ref. failures
Gemini 3.5 Flash (high)100%49.41 ± 8.360.70 ± 0.57
Gemini 3.5 Flash (medium)100%39.91 ± 9.610.55 ± 0.94
Gemini 3.5 Flash (low)100%22.87 ± 4.370.10 ± 0.31
Gemini 3.5 Flash-Lite (low)100%6.28 ± 2.207.05 ± 8.87
GPT-5.6 Terra (high)100%81.17 ± 26.700.45 ± 0.51
GPT-5.6 Luna (medium)100%55.31 ± 36.850.85 ± 1.23
GPT-5.6 Luna (low)100%25.24 ± 9.772.22 ± 6.03
Qwen 3.7 Plus (medium)100%236.15 ± 84.420.35 ± 0.49
Qwen 3.5 Flash (medium)73%241.83 ± 108.521.38 ± 0.89

BibTeX

@misc{yan2025vizcoast,
  title         = {Using {VLM} Reasoning to Constrain Task and Motion Planning},
  author        = {Yan, Muyang and Mengdibayev, Miras and Floros, Ardon and Guo, Weihang and Kavraki, Lydia E. and Kingston, Zachary},
  year          = {2025},
  eprint        = {2510.25548},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
}