Tsukada Lab. Computer Network Group

An Emergency Accident Reporter System for Autonomous Vehicles using Retrieval-Augmented Vision-Language Models

Muhammad Sulthan Adhipradhana

Master Thesis, Graduate School of Information Science and Technology, The University of Tokyo, Sep 30, 2026

Summary in three lines

  • This thesis develops a vision-language accident reporter that grounds emergency dispatch in Japanese response rules.
  • It annotates 500 accident clips and separates observed facts, rule retrieval and dispatch reasoning.
  • Experiments found fewer unnecessary dispatches, and ablations identified rule grounding as a major contributor.

Figures from the paper

Figure 4.1: Overview of the DispatchRAG framework
Figure 4.1. Overview of the DispatchRAG frameworkSource: Figure 4.1 from Master's thesis, The University of Tokyo, 2026 (author's version)A dashcam clip is turned into an observation-only scene description, rewritten as a retrieval query, matched against Japanese statutes and EMS protocols, and finally reasoned into a three-perspective dispatch plan in which every claim cites a rule.
Figure 5.1: Qualitative comparison on a minor vehicle-to-vehicle contact
Figure 5.1. Qualitative comparison on a minor vehicle-to-vehicle contactSource: Figure 5.1 from Master's thesis, The University of Tokyo, 2026 (author's version)For a minor contact that needs only the police, end-to-end VLMs given the same rules over-triage and add ambulance or fire units, some by hallucinating a severe detail. The proposed framework predicts the correct urgency tier and dispatch.

Abstract

Traffic accidents cause 1.19 million deaths worldwide in 2023. For each accidents, the casualty's chance of survival depends on how quickly the emergency units are dispatched. However, the average time from an accident to a report call is about 4.6 minutes in urban areas and can go up to 7.2 minutes in rural areas. This is correlated to the fact that few bystanders exist in rural areas, where they can make an emergency call. To resolve this, an automatic emergency responder can detect an accident, assess the severity, and dispatch the appropriate emergency service needed to shorten the emergency response time and reduce the mortality rate of these victims. Vision-Language Models (VLMs) become a good tool candidate as an emergency responder since they can describe the accident scene well, but they have a limitation of answering only from parametric memory. This nature is not suitable for a safety-critical role since it doesn't have any knowledge of a real-life safety protocol; thus, it is prone to hallucination. A safety-critical dispatch decision system should ground its knowledge in authoritative protocols and produce a actionable dispatch decision that can be justified to the rules that govern it. This thesis presents DispatchRAG, a retrieval-augmented framework that turns a dash-cam accident clip into a protocol-grounded emergency-dispatch decision. The framework decomposes the task into three stages: a Perception stage that converts the clip into an observation-only scene description; a Rule-Retrieval Agent that rewrites this description into a retrieval query written in the register of the protocol sing Hypothetical Document Embeddings (HyDE), and fetches the applicable rules from a corpus of authoritative Japanese traffic-law and emergency-medical protocols with a multilingual dense retriever; and a Reasoning Agent that composes a three-perspective (3-POV) dispatch plan for the dispatching agency, the on-scene civilian, and the arriving EMS crew. Every directive cites both the rule that licenses it and the observed evidence that motivates it, so each dispatch decision can be verified against its sources. To evaluate the framework, I constructed the Accident Dispatch Dataset: 500 dash-cam accident clips adapted from MM-AU, annotated manually and blind to any model output with an urgency tier, the emergency units to dispatch, a 3-POV response plan, and the set of applicable protocol rules. To my knowledge, it is the first benchmark to annotate accident video with an actionable, protocol-grounded dispatch decision rather than an unstructured description. Against frontier end-to-end VLM baselines given the same retrieved rules, DispatchRAG wins the structured decision metrics (Emergency Unit F1 0.85 against 0.75-0.82; Urgency-tier exact match 0.47 against 0.23-0.35) and is the best calibrated on the binary ambulance decision, producing 58 false positives against 175 for the highest-recall baseline, which dispatches an ambulance in nearly every clip. An ablation study shows that removing the urgency guidebook and the rule grounding degrades the results the most, and that the reliability comes from decomposing the task rather than from the scale of the model. Given faithful observations and the right rules, a modest open-weight reasoner produces a better dispatch decision than a frontier model.

How to cite

M. S. Adhipradhana, “An Emergency Accident Reporter System for Autonomous Vehicles using Retrieval-Augmented Vision-Language Models,” in Master Thesis, Graduate School of Information Science and Technology, The University of Tokyo, 2026.

BibTeX

@mastersthesis{Adhipradhana2026b,
  title         = {An Emergency Accident Reporter System for Autonomous Vehicles using Retrieval-Augmented Vision-Language Models},
  author        = {Muhammad Sulthan Adhipradhana},
  year          = {2026},
  date          = {2026-09-30},
  urldate       = {2026-09-30},
  school        = {Master Thesis, Graduate School of Information Science and Technology, The University of Tokyo},
  abstract      = {Traffic accidents cause 1.19 million deaths worldwide in 2023. For each accidents, the casualty's chance of survival depends on how quickly the emergency units are dispatched. However, the average time from an accident to a report call is about 4.6 minutes in urban areas and can go up to 7.2 minutes in rural areas. This is correlated to the fact that few bystanders exist in rural areas, where they can make an emergency call. To resolve this, an automatic emergency responder can detect an accident, assess the severity, and dispatch the appropriate emergency service needed to shorten the emergency response time and reduce the mortality rate of these victims. Vision-Language Models (VLMs) become a good tool candidate as an emergency responder since they can describe the accident scene well, but they have a limitation of answering only from parametric memory. This nature is not suitable for a safety-critical role since it doesn't have any knowledge of a real-life safety protocol; thus, it is prone to hallucination. A safety-critical dispatch decision system should ground its knowledge in authoritative protocols and produce a actionable dispatch decision that can be justified to the rules that govern it. This thesis presents DispatchRAG, a retrieval-augmented framework that turns a dash-cam accident clip into a protocol-grounded emergency-dispatch decision. The framework decomposes the task into three stages: a Perception stage that converts the clip into an observation-only scene description; a Rule-Retrieval Agent that rewrites this description into a retrieval query written in the register of the protocol sing Hypothetical Document Embeddings (HyDE), and fetches the applicable rules from a corpus of authoritative Japanese traffic-law and emergency-medical protocols with a multilingual dense retriever; and a Reasoning Agent that composes a three-perspective (3-POV) dispatch plan for the dispatching agency, the on-scene civilian, and the arriving EMS crew. Every directive cites both the rule that licenses it and the observed evidence that motivates it, so each dispatch decision can be verified against its sources. To evaluate the framework, I constructed the Accident Dispatch Dataset: 500 dash-cam accident clips adapted from MM-AU, annotated manually and blind to any model output with an urgency tier, the emergency units to dispatch, a 3-POV response plan, and the set of applicable protocol rules. To my knowledge, it is the first benchmark to annotate accident video with an actionable, protocol-grounded dispatch decision rather than an unstructured description. Against frontier end-to-end VLM baselines given the same retrieved rules, DispatchRAG wins the structured decision metrics (Emergency Unit F1 0.85 against 0.75-0.82; Urgency-tier exact match 0.47 against 0.23-0.35) and is the best calibrated on the binary ambulance decision, producing 58 false positives against 175 for the highest-recall baseline, which dispatches an ambulance in nearly every clip. An ablation study shows that removing the urgency guidebook and the rule grounding degrades the results the most, and that the reliability comes from decomposing the task rather than from the scale of the model. Given faithful observations and the right rules, a modest open-weight reasoner produces a better dispatch decision than a frontier model.},
  keywords      = {Thesis},
  pubstate      = {published},
  tppubtype     = {mastersthesis}
}