Foundation models on the bridge: Semantic hazard detection and safety maneuvers for maritime autonomy with vision-language models

Alexey Gusev; Andreas Gudahl Tufte; Kim Alexander Christensen; Marco Pavone; Martin Steinert; Milan Ganai; Ole Andreas Alsos; Rohan Sinha

arxiv: 2512.24470 · v2 · pith:CXQT47U6 · submitted 2025-12-30 · cs.RO · cs.AI

Foundation models on the bridge: Semantic hazard detection and safety maneuvers for maritime autonomy with vision-language models

Kim Alexander Christensen , Andreas Gudahl Tufte , Alexey Gusev , Rohan Sinha , Milan Ganai , Ole Andreas Alsos , Marco Pavone , Martin Steinert This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-05-21 15:34 UTCgrok-4.3pith:CXQT47U6 record.json open to challenge →

classification cs.RO cs.AI

keywords vision-language modelsmaritime autonomysemantic hazard detectionfallback maneuversIMO MASS Codeautonomous vesselssafety systemshazard response

0 comments

The pith

Vision-language models select safe fallback maneuvers for autonomous ships by interpreting semantic hazards.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that vision-language models can supply the semantic understanding needed for maritime autonomous vessels to handle unexpected hazards where meaning matters, such as recognizing a diver-down flag or a nearby fire. It introduces Semantic Lookout as a practical system that uses camera images to choose cautious short-horizon actions from possible water-based trajectories while keeping human oversight. Evaluations across 40 harbor scenes confirm alignment with human consensus, better performance than geometry-based methods, risk reduction on fire scenes, and operation within latency limits suitable for regulatory handover periods. A field demonstration shows the full alert to maneuver to operator takeover sequence works in practice.

Core claim

Semantic Lookout is a camera-only VLM system that selects one cautious fallback maneuver or station-keeping from water-valid trajectories. It provides semantic awareness for out-of-distribution situations, shows alignment with human majority voting on scene understanding, outperforms geometry-only baselines by increasing standoff on fire hazards, and runs in sub-10 seconds to fit the alert-to-takeover window of the draft IMO MASS Code. End-to-end functionality is confirmed in a field run.

What carries the argument

Semantic Lookout, the candidate-constrained vision-language model fallback maneuver selector operating on camera images to pick cautious water-valid actions under human authority.

If this is right

Sub-10s VLM models preserve most scene awareness of slower models.
The system increases standoff distance to fire hazards compared to geometry-only baselines.
Full pipeline from alert through fallback maneuver to operator handover is verified on water.
VLMs act as semantic fallback selectors that fit the draft IMO MASS Code requirements within practical latency limits.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This semantic approach could integrate with existing multi-sensor perception systems to handle both meaning and precise geometry.
Adapting the models to maritime-specific training data might improve accuracy for local conditions and hazards.
The method opens possibilities for applying similar VLM-based fallbacks in other autonomous domains with regulatory handover needs.
Future work could test if these selections remain safe over longer horizons or in more varied sea states.

Load-bearing premise

VLM scene understanding from camera images will match human agreement and produce safe choices when limited to water-valid paths, without overlooking important hazards or adding risks under real maritime conditions.

What would settle it

A test case in which the VLM fails to identify a hazard such as a diver in the water or chooses a trajectory that reduces rather than increases safety distance to a fire.

Figures

Figures reproduced from arXiv: 2512.24470 by Alexey Gusev, Andreas Gudahl Tufte, Kim Alexander Christensen, Marco Pavone, Martin Steinert, Milan Ganai, Ole Andreas Alsos, Rohan Sinha.

**Figure 2.** Figure 2: Coordinate frames and their relation. The water segmentation filters [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Figure 3: Block diagram for automatic waypoint follow with motion guidance on the ASV. A combined motion is the result of an override logic in which a human [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

**Figure 4.** Figure 4: Override switch logic implemented from autonomous operation to [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗

**Figure 6.** Figure 6: The remote operation center used in closed-loop experiments. Details [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

**Figure 7.** Figure 7: Representative scene examples of (a) Diver flag, (b) MOB, (c) fire, (d) custom sign. [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗

**Figure 8.** Figure 8: Model awareness vs latency, showing only the pareto frontier. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗

**Figure 13.** Figure 13: Best@1 rate leader board with Wilson CI. Only the Pareto frontier [PITH_FULL_IMAGE:figures/full_fig_p013_13.png] view at source ↗

**Figure 11.** Figure 11: Best rate vs latency, showing only the per provider Pareto frontier [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗

**Figure 12.** Figure 12: Accept@1 rate leader board with Wilson CI. Only the Pareto frontier [PITH_FULL_IMAGE:figures/full_fig_p013_12.png] view at source ↗

**Figure 14.** Figure 14: Risk-relief on the fire scenes: mean ∆dH (m) over time with min– max shading across n=10 scenes. Positive values indicate increased separation from the annotated hazard. 5.5. Experiment 4: Integrated sea trial (H4) Overview and method. To validate the selector in a real, closedloop system, we ran the end-to-end stack on a real ASV with control and override capabilities as described in Sec. 4.3. This addr… view at source ↗

**Figure 16.** Figure 16: The model output is reproduced below: Reasoning: See: Blue-white ‘A’ flag on buoy starboard; docks both sides; open channel ahead. Diver operations—keep well clear and slow; avoid starboard approach near buoy. Implications: Flag may indicate divers; maintain slow speed and wide berth to avoid people/gear. Action: Proceed mid-channel left of buoy, slow speed, maintain lookout for divers. (3) the operator… view at source ↗

**Figure 16.** Figure 16: Data from the integrated sea trial showing the chain of events: (1) [PITH_FULL_IMAGE:figures/full_fig_p016_16.png] view at source ↗

read the original abstract

The draft IMO MASS Code requires autonomous and remotely supervised maritime vessels to detect departures from their operational design domain, enter a predefined fallback that notifies the operator, permit immediate human override, and avoid changing the voyage plan without approval. Meeting these obligations in the alert-to-takeover gap calls for a short-horizon, human-overridable fallback maneuver. Classical maritime autonomy stacks struggle when the correct action depends on meaning (e.g., diver-down flag means people in the water, fire close by means hazard). We argue (i) that vision-language models (VLMs) provide semantic awareness for such out-of-distribution situations, and (ii) that a fast-slow anomaly pipeline with a short-horizon, human-overridable fallback maneuver makes this practical in the handover window. We introduce Semantic Lookout, a camera-only, candidate-constrained VLM fallback maneuver selector that selects one cautious action (or station-keeping) from water-valid, world-anchored trajectories under continuous human authority. On 40 harbor scenes we measure per-call scene understanding and latency, alignment with human consensus (model majority-of-three voting), short-horizon risk-relief on fire hazard scenes, and an on-water alert->fallback maneuver->operator handover. Sub-10 s models retain most of the awareness of slower state-of-the-art models. The fallback maneuver selector outperforms geometry-only baselines and increases standoff distance on fire scenes. A field run verifies end-to-end operation. These results support VLMs as semantic fallback maneuver selectors compatible with the draft IMO MASS Code, within practical latency budgets, and motivate future work on domain-adapted, hybrid autonomy that pairs foundation-model semantics with multi-sensor bird's-eye-view perception and short-horizon replanning. Website: kimachristensen.github.io/bridge_policy

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This paper shows a practical way to use VLMs for picking human-overridable fallback maneuvers in maritime autonomy on harbor scenes, with some gains over geometry baselines, but the evidence stays narrow.

read the letter

The main point is that this work puts vision-language models into a constrained role as a semantic selector for short-horizon safety maneuvers when a maritime system encounters meaning-dependent situations like hazards or flags. They build Semantic Lookout around candidate water-valid trajectories, keep continuous human authority, and tie it to the draft IMO MASS Code needs for fallback and takeover windows. Sub-10-second models keep most of the scene awareness while the selector beats geometry-only baselines on alignment and standoff distance in the reported tests. The field run shows the full alert-to-maneuver-to-handover loop can run on water. That combination of design choices and initial numbers is the concrete contribution here. The evaluation covers 40 harbor scenes for latency, human consensus match via majority vote, and risk relief on fire cases, plus the one real-world check. This gives a usable template for anyone trying to add semantic awareness without replacing the rest of the stack. The soft spots sit in the test scope and reporting. Forty scenes plus one field run leave open how the VLM handles weather shifts, lighting changes, or denser traffic, and there are no error bars, false-negative rates for missed hazards, or ablations on those variables. The human panel is small, so it is not yet clear how well it tracks expert mariner judgment beyond the test set. The abstract summarizes outcomes without the full statistical detail that would let a reader judge robustness directly. This paper is for robotics and autonomy groups working on regulated maritime systems or hybrid foundation-model fallbacks. Readers who need a worked example of candidate-constrained VLM use in a safety loop will get value from the pipeline and the regulatory framing. It deserves a serious referee because the applied setup addresses a real gap with measurable early results, even if the current evidence is preliminary and would benefit from broader testing requests during review. I would send it out with that in mind.

Referee Report

2 major / 2 minor

Summary. The paper introduces Semantic Lookout, a camera-only VLM-based fallback maneuver selector for maritime autonomy that chooses cautious, water-valid actions (or station-keeping) from world-anchored trajectories under continuous human authority. It argues that VLMs supply the semantic awareness needed for out-of-distribution hazards (e.g., diver-down flags, nearby fires) that classical geometry-based stacks miss, and that a fast-slow anomaly pipeline makes this practical within the IMO MASS Code alert-to-takeover window. The central empirical support consists of per-call metrics on 40 harbor scenes (scene understanding, latency, majority-of-three human alignment, standoff improvement on fire scenes) plus a single on-water handover demonstration; sub-10 s models are claimed to retain most awareness while outperforming geometry baselines.

Significance. If the reported alignment and risk-relief results prove robust, the work supplies a concrete, regulator-compatible mechanism for injecting foundation-model semantics into short-horizon maritime fallback systems. It explicitly credits the field demonstration and the latency-aware comparison of fast versus slow VLMs, and it motivates hybrid architectures that pair VLM semantics with multi-sensor BEV perception.

major comments (2)

[Results on 40 harbor scenes] Evaluation on 40 harbor scenes: the abstract and results summary report quantitative improvements in human alignment and fire-hazard standoff distance, yet provide no exact metric definitions, statistical tests, error bars, or data-exclusion criteria. Because these numbers are the primary evidence for the claim that VLMs produce water-valid trajectories compatible with the draft IMO MASS Code, the absence of these details is load-bearing for the central safety argument.
[Field demonstration] Field-run demonstration: the single on-water alert-to-fallback-to-handover trial is presented as verification of end-to-end operation, but no quantitative false-negative rates for critical hazards, no ablation across weather/lighting/traffic conditions, and no comparison against expert mariner ground truth beyond the three-person consensus panel are reported. This leaves the generalization claim dependent on an untested extrapolation from the narrow test distribution.

minor comments (2)

[Method] The term 'Semantic Lookout' is introduced without a concise formal definition or pseudocode; a short algorithmic box would clarify the candidate-constrained selection step.
[Figures] Figure captions for the harbor scenes and trajectory overlays should explicitly state the VLM prompt template and the exact human-voting protocol used for the majority baseline.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. We address each major comment point by point below, indicating the revisions we will make to strengthen the manuscript's clarity and empirical rigor while preserving the original contributions.

read point-by-point responses

Referee: Evaluation on 40 harbor scenes: the abstract and results summary report quantitative improvements in human alignment and fire-hazard standoff distance, yet provide no exact metric definitions, statistical tests, error bars, or data-exclusion criteria. Because these numbers are the primary evidence for the claim that VLMs produce water-valid trajectories compatible with the draft IMO MASS Code, the absence of these details is load-bearing for the central safety argument.

Authors: We agree that the current presentation lacks sufficient detail on the evaluation protocol. In the revised manuscript we will expand the Results and Methods sections to provide: exact definitions of all metrics (scene-understanding accuracy, majority-of-three human alignment procedure with inter-rater agreement, standoff-distance computation in meters, and latency); statistical tests (e.g., paired comparisons with p-values against geometry baselines); error bars or confidence intervals on all reported figures and tables; and explicit data-exclusion criteria (e.g., scenes discarded for camera failure, extreme glare, or insufficient visibility). These additions will be placed before the quantitative claims to make the safety argument fully transparent. revision: yes
Referee: Field-run demonstration: the single on-water alert-to-fallback-to-handover trial is presented as verification of end-to-end operation, but no quantitative false-negative rates for critical hazards, no ablation across weather/lighting/traffic conditions, and no comparison against expert mariner ground truth beyond the three-person consensus panel are reported. This leaves the generalization claim dependent on an untested extrapolation from the narrow test distribution.

Authors: We acknowledge the limitations of a single field trial. The demonstration was intended only to confirm that the alert-to-fallback-to-handover sequence can execute within the IMO MASS Code time window under real conditions; it was never presented as a statistical evaluation. In revision we will (i) explicitly state the scope and constraints of the field run, (ii) add a dedicated limitations paragraph discussing the absence of false-negative rates and multi-condition ablations due to regulatory and safety requirements for on-water testing, and (iii) clarify that the three-person consensus serves as an initial human-alignment benchmark while noting the need for broader expert validation in future work. The primary quantitative evidence remains the 40 harbor scenes. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical evaluation with independent human and baseline comparisons

full rationale

The paper introduces Semantic Lookout as a VLM-based fallback selector and reports direct empirical measurements (per-call understanding, latency, human consensus alignment via majority-of-three voting, standoff improvement on fire scenes, and one field handover) on 40 held-out harbor scenes plus a single on-water run. No equations, fitted parameters, or self-referential definitions appear in the core claims; results are not forced by construction from inputs within the paper. Self-citations, if present, are not load-bearing for the reported metrics. The derivation chain is self-contained against external benchmarks (human voting, geometry baselines).

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the domain assumption that VLMs can deliver reliable semantic hazard detection when outputs are restricted to physically valid trajectories and that the resulting maneuvers provide measurable risk relief within regulatory time windows.

axioms (1)

domain assumption VLMs provide semantic awareness for out-of-distribution maritime situations when constrained to water-valid trajectories
Invoked to justify the fast-slow anomaly pipeline and fallback selector as practical for the IMO MASS Code handover window.

invented entities (1)

Semantic Lookout no independent evidence
purpose: Camera-only candidate-constrained VLM fallback maneuver selector
New system name and architecture introduced to implement the proposed semantic fallback.

pith-pipeline@v0.9.0 · 5887 in / 1360 out tokens · 63518 ms · 2026-05-21T15:34:43.918927+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/RealityFromDistinction.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We introduce Semantic Lookout, a camera-only, candidate-constrained VLM fallback maneuver selector that selects one cautious action (or station-keeping) from water-valid, world-anchored trajectories under continuous human authority.
IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

On 40 harbor scenes we measure per-call scene understanding and latency, alignment with human consensus (model majority-of-three voting), short-horizon risk-relief on fire hazard scenes

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages · 1 internal anchor

[1]

A Survey on LLM-as-a-Judge

Real-time out-of-distribution failure prevention via multi-modal reasoning, in: 9th Annual Conference on Robot Learning. (under review). 18 Geirhos, R., Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A., 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence URL: https://doi.org/10.1038/s42256-020-0025...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.1038/s42256-020-00257-z 2020
[2]

Nasiriany, S., Xia, F., Yu, W., Xiao, T., Liang, J., Dasgupta, I., Xie, A., Driess, D., Wahid, A., Xu, Z., et al., 2024

doi:10.1109/OJVT.2024.3443630. Nasiriany, S., Xia, F., Yu, W., Xiao, T., Liang, J., Dasgupta, I., Xie, A., Driess, D., Wahid, A., Xu, Z., et al., 2024. Pivot: Iterative visual prompting elicits actionable knowledge for VLMs. arXiv preprint arXiv:2402.07872 . Octo Model Team, Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Xu...

work page doi:10.1109/ojvt.2024.3443630 2024
[3]

Proceedings of the IEEE 109, 756–795

A unifying review of deep and shallow anomaly detec- tion. Proceedings of the IEEE 109, 756–795. doi:10.1109/ JPROC.2021.3052449. Rutledal, D., 2024. Designing for Situational Awareness in Re- mote Operator Centers for Autonomous Ships. Ph.d. thesis. Norwegian University of Science and Technology. Trond- heim, Norway. Saviolo, A., Rao, P., Radhakrishnan, ...

work page arXiv 2021
[4]

Unifying foundation models with quadrotor control for visual tracking beyond object categories, in: 2024 IEEE In- ternational Conference on Robotics and Automation (ICRA), IEEE. pp. 7389–7396. Shah, D., Osi ´nski, B., Levine, S., et al., 2023a. LM-Nav: Robotic navigation with large pre-trained models of lan- guage, vision, and action, in: Conference on ro...

work page 2024
[5]

Takeover time: Requirements for highly automated in- land vessels-first experimental-based results, in: 2024 IEEE 4th International Conference on Human-Machine Systems (ICHMS), IEEE. pp. 1–7. Sinha, R., Elhafsi, A., Agia, C., Foutter, M., Schmerling, E., Pavone, M., 2024. Real-time anomaly detection and reac- tive planning with large language models. arXi...

work page arXiv 2024
[6]

vir- tual rudder

Human factor influences on supervisory control of re- motely operated and autonomous vessels. Ocean Engineer- ing 299. doi:10.1016/j.oceaneng.2024.117257. V olden, Ø., Stahl, A., Fossen, T.I., 2022. Vision-based posi- tioning system for auto-docking of unmanned surface vehi- cles (USVs). International Journal of Intelligent Robotics and Applications 6, 86...

work page doi:10.1016/j.oceaneng.2024.117257 2024

[1] [1]

A Survey on LLM-as-a-Judge

Real-time out-of-distribution failure prevention via multi-modal reasoning, in: 9th Annual Conference on Robot Learning. (under review). 18 Geirhos, R., Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A., 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence URL: https://doi.org/10.1038/s42256-020-0025...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.1038/s42256-020-00257-z 2020

[2] [2]

Nasiriany, S., Xia, F., Yu, W., Xiao, T., Liang, J., Dasgupta, I., Xie, A., Driess, D., Wahid, A., Xu, Z., et al., 2024

doi:10.1109/OJVT.2024.3443630. Nasiriany, S., Xia, F., Yu, W., Xiao, T., Liang, J., Dasgupta, I., Xie, A., Driess, D., Wahid, A., Xu, Z., et al., 2024. Pivot: Iterative visual prompting elicits actionable knowledge for VLMs. arXiv preprint arXiv:2402.07872 . Octo Model Team, Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Xu...

work page doi:10.1109/ojvt.2024.3443630 2024

[3] [3]

Proceedings of the IEEE 109, 756–795

A unifying review of deep and shallow anomaly detec- tion. Proceedings of the IEEE 109, 756–795. doi:10.1109/ JPROC.2021.3052449. Rutledal, D., 2024. Designing for Situational Awareness in Re- mote Operator Centers for Autonomous Ships. Ph.d. thesis. Norwegian University of Science and Technology. Trond- heim, Norway. Saviolo, A., Rao, P., Radhakrishnan, ...

work page arXiv 2021

[4] [4]

Unifying foundation models with quadrotor control for visual tracking beyond object categories, in: 2024 IEEE In- ternational Conference on Robotics and Automation (ICRA), IEEE. pp. 7389–7396. Shah, D., Osi ´nski, B., Levine, S., et al., 2023a. LM-Nav: Robotic navigation with large pre-trained models of lan- guage, vision, and action, in: Conference on ro...

work page 2024

[5] [5]

Takeover time: Requirements for highly automated in- land vessels-first experimental-based results, in: 2024 IEEE 4th International Conference on Human-Machine Systems (ICHMS), IEEE. pp. 1–7. Sinha, R., Elhafsi, A., Agia, C., Foutter, M., Schmerling, E., Pavone, M., 2024. Real-time anomaly detection and reac- tive planning with large language models. arXi...

work page arXiv 2024

[6] [6]

vir- tual rudder

Human factor influences on supervisory control of re- motely operated and autonomous vessels. Ocean Engineer- ing 299. doi:10.1016/j.oceaneng.2024.117257. V olden, Ø., Stahl, A., Fossen, T.I., 2022. Vision-based posi- tioning system for auto-docking of unmanned surface vehi- cles (USVs). International Journal of Intelligent Robotics and Applications 6, 86...

work page doi:10.1016/j.oceaneng.2024.117257 2024