This is that robot's own recording. Once it settled into being wrong, its own numbers still said it was fine.
0 s400 s
healthy, on floor 2on the floor with no mapback, but 19.4 m off
PR2 · 400 s of a recorded run · 80 detector events · MIT Stata Center dataset (CC BY 3.0)
01
Grey is the walls of a real building, from its map. Green is not where the robot went. It is where the robot's software believed it was, drawn from its own recorded log as the run plays back.
02
Partway through, the robot rode an elevator up to a floor its map did not have, drove around, then came back down. From that point its belief and the truth split apart: its belief settled into a corridor 19.4 metres from where it actually stood, while its own report said it was confident to within 8 centimetres.
03
Amber marks the moments my detectors caught something wrong: 30 of them after it came back and settled into the wrong corridor, more while it was on the floor with no map. They caught it while the wrong belief was still forming. Once it stopped changing, both the checks and the robot's own confidence number went quiet, because neither had anything outside the robot's own estimate of itself to compare against.
04
I read recordings like this one for a living. I build small checks that watch for exactly this kind of trouble, run them the same way every time, and say plainly what they catch and what they miss. If a robot in your fleet has ever come back looking fine when something was wrong, and you still have the recording, that is what the twenty-minute call below is for.
Théo Alessandro Hermann, independent practitioner
I read robot navigation logs. I froze four fault detectors before I looked at any results, ran them unchanged on 108 minutes of recordings from five platforms, and published what they caught, what they missed, and the raw counts behind both.
The problem
A fleet's worst localisation incidents are the quiet ones: the robot that was somewhere else while reporting centimetre confidence, the alarm that fired on a healthy machine, the failure nobody can reproduce from the bag, which is the robot's own recording. Nav2, the ROS 2 navigation stack, ships its localiser AMCL with kidnap recovery switched off, so a displaced robot has no mechanism to conclude it is lost. Someone has to notice, and the bag is usually all they have.
Three results
01
The same frozen thresholds did not transfer. The covariance threshold, set on a simulated TurtleBot3, sits inside a real Tiago's healthy yaw noise and fired five times on a recording with nothing wrong, graded wrong in the case log. A threshold is a property of the machine it was measured on.
Against labels somebody else wrote, years before this tool existed: 16 of 16 gaps found, nothing else flagged. The labels give how many gaps each recording has, not when, so the match is a match of counts, one for one.
Two Cartographer backpack recordings on a shared timeline: the dataset's Known Issues column counts 2 and 14 laser gaps, and the frozen detector found 2 and 14 events, nothing elsewhere.
Source results/labelled/ · docs/figures/labelled-recall.png
03
A localiser 19.4 metres wrong reported 8 centimetres of uncertainty. The detectors caught the transition, then went silent once the wrong pose settled. No monitor built on the robot's own estimates can see steady-state confident wrongness.
Amber flagged 80 moments across the whole 400-second run: 2 while the robot was healthy, 47 while it was on the floor with no map, and 30 once it came back and settled into the wrong corridor. One more came after the ground truth ran out, too late for anything to grade.
Monitoring rules tuned on one robot mean something different on the next one, and a localiser that has settled into a wrong pose looks healthy to any check that trusts its own estimate. Triage on a fleet is therefore a calibration-and-ground-truth problem before it is a tooling problem.
What this does not claim
These are localisation-layer detectors, not an intrusion detection system and not a product. I recorded, ran and self-graded everything, under a rubric committed before any real result existed; the case log carries two verdicts of wrong and three of partial across fifteen graded rows. Three predictions I committed before one benchmark run: two held, one did not. Nothing in the stock config is calibrated for your robot.
The ask
If a robot in your fleet was somewhere it should not have been, and the bag exists, twenty minutes over that bag is the conversation I am proposing.
Analysis runs on your infrastructure or under NDA on mine; nothing is retained after handover.
Twenty minutes, free: you bring the incident, I say whether I can help.
Written proposal: fixed scope, fixed price, before you commit.
Weekly written updates: what shipped, what is next, what is at risk.
Handover: running code, documentation and a written report.
