NL2TL-Tree: Natural Language to Nested Temporal-Logic Trees for Robustness-Guided Flow-Matching Planning
Companion of the 2026 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2026 ACM International Symposium on Wearable Computers (UbiComp/ISWC '26), Poster, Shanghai, China, Oct 11, 2026
Summary in three lines
- NL2TL-Tree translates natural-language robot instructions into nested signal temporal logic trees without additional training.
- The tree exposes scope errors and lets a differentiable robustness score guide a frozen flow-matching planner.
- The full pipeline completed six tested TurtleBot3 tasks in simulation, compared with four when A* repair was disabled.
Abstract
Letting non-expert users command robots in plain language is a long-standing goal of ubiquitous computing, but free-form instructions are ambiguous and give no guarantee that the resulting behavior is correct. We present NL2TL-Tree, an early-stage prototype exploring whether a nested Signal Temporal Logic (STL) tree can serve as an interpretable, checkable interface between natural-language robot instructions and motion planning. The pipeline is training-free (no model is trained or fine-tuned): a frozen LLM is few-shot-prompted to parse a smart-device task into the tree (human-readable and error-localizable to a single sub-node), and a frozen graph-encoded flow-matching planner (TeLoGraF), steered by a differentiable STL-robustness term, turns the tree into motion. Test-time mechanisms (operator normalization, nested-tree decomposition, a feasibility self-check that defers to a classical backend) extend the frozen planner, and an online robustness monitor can trigger a stop/re-plan; a planning-latency-aware deadline further debits the planner's own wall-clock from timed goals. We report preliminary evidence (a small parsing pilot and a TurtleBot3 Gazebo simulation) and outline an evaluation plan for parsing accuracy, failure localization, and user-facing interpretability. Code, prompts, and configs are released.
How to cite
J. Gong, W. Mao, E. Javanmardi, Y. Li, Q. Zhou, and M. Tsukada, “NL2TL-Tree: Natural Language to Nested Temporal-Logic Trees for Robustness-Guided Flow-Matching Planning,” in Companion of the 2026 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2026 ACM International Symposium on Wearable Computers (UbiComp/ISWC '26), Poster, Shanghai, China, 2026.
BibTeX
@misc{Gong2026,
title = {NL2TL-Tree: Natural Language to Nested Temporal-Logic Trees for Robustness-Guided Flow-Matching Planning},
author = {Jiachen Gong and Wencan Mao and Ehsan Javanmardi and Yun Li and Quanxi Zhou and Manabu Tsukada},
year = {2026},
date = {2026-10-11},
urldate = {2026-10-11},
booktitle = {Companion of the 2026 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2026 ACM International Symposium on Wearable Computers (UbiComp/ISWC '26), Poster},
address = {Shanghai, China},
abstract = {Letting non-expert users command robots in plain language is a long-standing goal of ubiquitous computing, but free-form instructions are ambiguous and give no guarantee that the resulting behavior is correct. We present NL2TL-Tree, an early-stage prototype exploring whether a nested Signal Temporal Logic (STL) tree can serve as an interpretable, checkable interface between natural-language robot instructions and motion planning. The pipeline is training-free (no model is trained or fine-tuned): a frozen LLM is few-shot-prompted to parse a smart-device task into the tree (human-readable and error-localizable to a single sub-node), and a frozen graph-encoded flow-matching planner (TeLoGraF), steered by a differentiable STL-robustness term, turns the tree into motion. Test-time mechanisms (operator normalization, nested-tree decomposition, a feasibility self-check that defers to a classical backend) extend the frozen planner, and an online robustness monitor can trigger a stop/re-plan; a planning-latency-aware deadline further debits the planner's own wall-clock from timed goals. We report preliminary evidence (a small parsing pilot and a TurtleBot3 Gazebo simulation) and outline an evaluation plan for parsing accuracy, failure localization, and user-facing interpretability. Code, prompts, and configs are released.},
howpublished = {Companion of the 2026 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2026 ACM International Symposium on Wearable Computers (UbiComp/ISWC '26), Poster},
keywords = {Other},
pubstate = {published},
tppubtype = {misc}
}