Tsukada Lab. Computer Network Group

NL2TL-Tree: Natural Language to Nested Temporal-Logic Trees for Robustness-Guided Flow-Matching Planning

Jiachen Gong, Wencan Mao, Ehsan Javanmardi, Yun Li, Quanxi Zhou, Manabu Tsukada

Companion of the 2026 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2026 ACM International Symposium on Wearable Computers (UbiComp/ISWC '26), Poster, Shanghai, China, Oct 11, 2026

Summary in three lines

  • NL2TL-Tree translates natural-language robot instructions into nested signal temporal logic trees without additional training.
  • The tree exposes scope errors and lets a differentiable robustness score guide a frozen flow-matching planner.
  • The full pipeline completed six tested TurtleBot3 tasks in simulation, compared with four when A* repair was disabled.

Abstract

Letting non-expert users command robots in plain language is a long-standing goal of ubiquitous computing, but free-form instructions are ambiguous and give no guarantee that the resulting behavior is correct. We present NL2TL-Tree, an early-stage prototype exploring whether a nested Signal Temporal Logic (STL) tree can serve as an interpretable, checkable interface between natural-language robot instructions and motion planning. The pipeline is training-free (no model is trained or fine-tuned): a frozen LLM is few-shot-prompted to parse a smart-device task into the tree (human-readable and error-localizable to a single sub-node), and a frozen graph-encoded flow-matching planner (TeLoGraF), steered by a differentiable STL-robustness term, turns the tree into motion. Test-time mechanisms (operator normalization, nested-tree decomposition, a feasibility self-check that defers to a classical backend) extend the frozen planner, and an online robustness monitor can trigger a stop/re-plan; a planning-latency-aware deadline further debits the planner's own wall-clock from timed goals. We report preliminary evidence (a small parsing pilot and a TurtleBot3 Gazebo simulation) and outline an evaluation plan for parsing accuracy, failure localization, and user-facing interpretability. Code, prompts, and configs are released.

How to cite

J. Gong, W. Mao, E. Javanmardi, Y. Li, Q. Zhou, and M. Tsukada, “NL2TL-Tree: Natural Language to Nested Temporal-Logic Trees for Robustness-Guided Flow-Matching Planning,” in Companion of the 2026 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2026 ACM International Symposium on Wearable Computers (UbiComp/ISWC '26), Poster, Shanghai, China, 2026.

BibTeX

@misc{Gong2026,
  title         = {NL2TL-Tree: Natural Language to Nested Temporal-Logic Trees for Robustness-Guided Flow-Matching Planning},
  author        = {Jiachen Gong and Wencan Mao and Ehsan Javanmardi and Yun Li and Quanxi Zhou and Manabu Tsukada},
  year          = {2026},
  date          = {2026-10-11},
  urldate       = {2026-10-11},
  booktitle     = {Companion of the 2026 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2026 ACM International Symposium on Wearable Computers (UbiComp/ISWC '26), Poster},
  address       = {Shanghai, China},
  abstract      = {Letting non-expert users command robots in plain language is a long-standing goal of ubiquitous computing, but free-form instructions are ambiguous and give no guarantee that the resulting behavior is correct. We present NL2TL-Tree, an early-stage prototype exploring whether a nested Signal Temporal Logic (STL) tree can serve as an interpretable, checkable interface between natural-language robot instructions and motion planning. The pipeline is training-free (no model is trained or fine-tuned): a frozen LLM is few-shot-prompted to parse a smart-device task into the tree (human-readable and error-localizable to a single sub-node), and a frozen graph-encoded flow-matching planner (TeLoGraF), steered by a differentiable STL-robustness term, turns the tree into motion. Test-time mechanisms (operator normalization, nested-tree decomposition, a feasibility self-check that defers to a classical backend) extend the frozen planner, and an online robustness monitor can trigger a stop/re-plan; a planning-latency-aware deadline further debits the planner's own wall-clock from timed goals. We report preliminary evidence (a small parsing pilot and a TurtleBot3 Gazebo simulation) and outline an evaluation plan for parsing accuracy, failure localization, and user-facing interpretability. Code, prompts, and configs are released.},
  howpublished  = {Companion of the 2026 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2026 ACM International Symposium on Wearable Computers (UbiComp/ISWC '26), Poster},
  keywords      = {Other},
  pubstate      = {published},
  tppubtype     = {misc}
}