Differentiable Particle Optimization for Fast Sequential Manipulation

Lucas Chen1, Shrutheesh R. Iyer1, Zachary Kingston1
1Purdue University
Paper arXiv Video Code

SPaSM solves sequential manipulation problems in milliseconds

Abstract

Sequential robot manipulation tasks require finding collision-free trajectories that satisfy geometric constraints across multiple object interactions in potentially high-dimensional configuration spaces. Solving these problems in real-time and at large scales has remained out of reach due to computational requirements. Recently, GPU-based acceleration has shown promising results, but prior methods achieve limited performance due to CPU-GPU data transfer overhead and complex logic that prevents full hardware utilization.

To this end, we present SPaSM (Sampling Particle optimization for Sequential Manipulation), a fully GPU-parallelized framework that compiles constraint evaluation, sampling, and gradient-based optimization into optimized CUDA kernels for end-to-end trajectory optimization without CPU coordination. The method consists of a two-stage particle optimization strategy: first solving placement constraints through massively parallel sampling, then lifting solutions to full trajectory optimization in joint space.

Unlike hierarchical approaches, SPaSM jointly optimizes object placements and robot trajectories to handle scenarios where motion feasibility constrains placement options. Experimental evaluation on challenging benchmarks demonstrates solution times in the realm of milliseconds with a 100% success rate; a 4000x speedup compared to existing approaches.

Video

Visual Effects

Using nerfies you can create fun visual effects. This Dolly zoom effect would be impossible without nerfies since it would require going through a wall.

Matting

As a byproduct of our method, we can also solve the matting problem by ignoring samples that fall outside of a bounding box during rendering.

Animation

Trajectory Optimization

We perform joint object placement and trajectory optimization by using an augmented langrangian method to iteratively improve the plan. Use the slider here to move through the steps of the tower environment optimization between the initial and optimized states.

Interpolate start reference image.

Initial State

Loading...
Interpolation end reference image.

Final State


More Trajectory Optimization

Here is another example of the optimization process on the tetris environment.

Interpolate start reference image.

Initial State

Loading...
Interpolation end reference image.

Final State


Re-rendering the input video

Using Nerfies, you can re-render a video from a novel viewpoint such as a stabilized camera by playing back the training deformations.

Related Links

There's a lot of excellent work that was introduced around the same time as ours.

Progressive Encoding for Neural Optimization introduces an idea similar to our windowed position encoding for coarse-to-fine optimization.

D-NeRF and NR-NeRF both use deformation fields to model non-rigid scenes.

Some works model videos with a NeRF by directly modulating the density, such as Video-NeRF, NSFF, and DyNeRF

There are probably many more by the time you are reading this. Check out Frank Dellart's survey on recent NeRF papers, and Yen-Chen Lin's curated list of NeRF papers.

BibTeX (preprint)

@article{chen2025spasm,
    author={Lucas Chen and Shrutheesh Raman Iyer and Zachary Kingston},
    title={Differentiable Particle Optimization for Fast Sequential Manipulation}, 
    year={2025},
    eprint={2510.07674},
    archivePrefix={arXiv},
    primaryClass={cs.RO},
    url={https://arxiv.org/abs/2510.07674}, 
}