REVIEW 5 major objections 5 minor 42 references
MELLON - Multimodal Enhanced LLM for Online Navigation
T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read MELLON, a multimodal web navigation agent, raises step-wise action accuracy on WebShop from 5.11% to 14.37% after one epoch of training.
desk verdict A course project with a real architecture and honest negative results, but the headline claim about task completion accuracy is not supported by the reported step-wise metric on a curated test set. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the projection layer: a single trainable linear mapping placed between the Q-Former's visual output and the frozen LLM's embedding space. It converts visual information into prompt tokens inserted at an <ImageHere> slot in a shortened ReAct prompt, allowing the LLM to reason over text and image together. Because only the projection layer and Q-Former are trained, the whole adaptation fits in one epoch on a 24 GB GPU.
What would settle it
Train MELLON for one epoch as described and evaluate it on the standard 500-instance WebShop test set, reporting success rate and reward score. If the score and success rate do not exceed the ReAct baselines of 66.6 and 40.0 percent, the claimed 9.26-point improvement does not carry over to the benchmark's standard metric.
Extended reading notes
Core claim
The paper's central claim is that a frozen large language model can become a multimodal web navigation agent through a small trainable bridge. MELLON uses an EVA ViT to encode product images, a Q-Former to compress them, and a projection layer to turn the result into soft prompt tokens that CodeLlama can read alongside the WebShop accessibility tree and a ReAct-style reasoning prompt. After one epoch of training the projection layer and Q-Former with a cross-entropy generative loss, step-wise action accuracy on the paper's 338-instance test set rises from 5.11% to 14.37%, which the abstract reports as a 9.26% increase in task completion accuracy. The authors also find that a ViT-based encoder with a trainable Q-Former matches the ResNet baseline on score and success rate, while a frozen Q-Former does not improve over ResNet. On their own terms, the result is that multimodal alignment improves an LLM's action selection on WebShop without unfreezing the LLM.
Load-bearing premise
The result depends on treating step-wise action accuracy on a 338-instance test set, which the authors curated because it was hard for baselines, as task completion accuracy; if that metric and test set do not represent the standard WebShop benchmark, the headline improvement may not hold.
Editorial extensions
If this is right
- MELLON can be adapted to new frozen LLMs without retraining the model, so stronger base models should translate directly into better web agents.
- The one-epoch training budget makes the approach practical for environments where collecting large trajectory datasets is expensive.
- The authors' error analysis suggests frozen CodeLlama's empty or invalid outputs are the main remaining bottleneck, so better prompting or light LLM tuning is the next lever.
- The architecture is defined on WebShop but is not benchmark-specific; the same projection-layer recipe could be applied to other multimodal web environments.
Reading between the lines
- Editorial inference: the reported gain is measured on step-wise action accuracy, not success rate or reward score; on the standard 500-instance WebShop test the improvement could be smaller or absent, and the paper does not provide that number.
- Editorial inference: because the test set was curated to be challenging for baselines, the 9.26-point gain may overstate average-case benefit if hard cases are over-represented.
- Editorial inference: the failure of VQAgent and Multimodal Ranker suggests that in WebShop, text usually carries more information than images; a testable implication is that MELLON's gains concentrate on attributes that appear only in images, such as color or material.
- Editorial inference: a direct comparison on standard WebShop success and reward metrics against ReAct and the WebShop IL baseline would settle whether the step-wise gain translates into better task completion.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes three multimodal enhancements for the WebShop web-navigation benchmark: MELLON, which projects aligned ViT/Q-Former visual embeddings into a frozen CodeLlama model with redesigned ReAct prompts; VQAgent, which frames WebShop as a multiple-choice VQA task using BEiT3; and Multimodal Ranker, which augments action selection with BERT and CLIP similarity scores. The authors report that one epoch of MELLON training improves action accuracy from 5.11% to 14.37% on a 338-instance test set, while VQAgent and Multimodal Ranker both underperform the IL baseline. The abstract interprets the MELLON gain as a 9.26% improvement in 'task completion accuracy,' but Section 4.5 defines the reported metric as step-wise action accuracy, and no task-level success rate or reward score is reported for MELLON on the standard 500-instance WebShop test set.
Significance. If the step-wise accuracy gain were accompanied by task-level improvements on the standard WebShop test set, the parameter-efficient recipe of freezing the LLM and ViT while training only the projection layer and Q-Former would be a useful contribution, and the paper is candid in reporting that VQAgent and Multimodal Ranker fail to beat the baseline. However, the central positive claim is currently supported only by a non-public, nonstandard test set and a metric that does not measure task completion. The paper's only task-level evidence involving the visual architecture shows no improvement over the ResNet baseline, so the significance as stated is not established.
major comments (5)
- [Abstract; §4.5; §5.1.1] The abstract and Section 5.1.1 claim a 9.26% improvement in 'task completion accuracy,' but Section 4.5 explicitly defines the MELLON metric as step-wise accuracy, i.e., the fraction of steps where the predicted action matches the human action. No success rate or WebShop reward score for MELLON is reported on the standard 500-instance test set. The 9.26% gain is therefore a step-wise accuracy increase on the paper's own test set, not evidence of improved task completion; the conclusion in Section 6 itself reverts to 'step-wise accuracy,' confirming the discrepancy.
- [§4.3] The MELLON test set consists of 338 instances 'curated from the midterm report' and described as 'particularly challenging for baseline models,' but the curation procedure and the set itself are not described, justified, or released. Reporting results only on this nonstandard set makes the 9.26% gain non-comparable to existing WebShop results such as ReAct, IL, and IL+RL, and selecting instances specifically hard for baselines can inflate the apparent improvement. The paper should evaluate MELLON on the standard 500-instance test set or provide a clear argument that the curated set is representative.
- [§5.1.1, Table 3] The only task-level results connected to MELLON's visual architecture show that ViT+Trainable QFormer achieves a score of 39.92, identical to the ResNet baseline, and a success rate of 9.6% versus 9.8%. This is direct evidence that the improved step-wise action accuracy does not translate into improved task completion, and it contradicts the abstract's claim. The authors should report full MELLON success rate and score on the standard test set and reconcile these numbers with the claimed task-completion improvement.
- [§3.1.2] This section states that 'we integrated the WebShop completion score into the loss function as a contributing factor,' but the loss function defined immediately afterward is a pure cross-entropy loss over action tokens with no completion-score term. Either the score integration should be described precisely, for example as a weighted auxiliary term, or the claim should be removed; as written, the training objective is not actually specified.
- [§2.2 versus §5.1.2] Section 2.2 reports that ReAct achieves a score of 66.6 and a success rate of 40.0 on WebShop, but Table 4 lists ReAct (CodeLlama-7b) with a score of 24.98. These numbers are inconsistent, and the discrepancy is not explained. Because ReAct is a key comparison baseline, the paper must clarify the configuration and evaluation conditions used in each table.
minor comments (5)
- [Throughout] There are several typos and formatting errors, including 'backbond' in §3.1.1, 'batsh size' in §4.4, 'we doesn't train' in §5.1.1, and 'MELON' instead of 'MELLON' in Table 7.
- [§4.3] The MELLON training set of 8,237 instances is not related to the standard WebShop training split of 10,587 instances; please state explicitly how the training instances were filtered or subsampled.
- [§5.1.1, Table 2] The phrase '9.26% increase' is ambiguous between an absolute percentage-point gain and a relative gain; from Table 2 it is an absolute increase from 5.11% to 14.37%, so the wording should be clarified.
- [§3.1.2] The transition from the initially trained projection layer to the 'dual-focused' training strategy is described only qualitatively; a precise description of the data collection, loss weighting, and training schedule would be needed for reproducibility.
- [§5.1.3 and Appendix] Several textual references such as 'example 7', '11', '12', and '13' should be replaced with consistent figure or appendix numbers.
Circularity Check
No significant circularity: MELLON's reported 9.26% gain is an empirical held-out measurement, not a derivation from its own inputs.
full rationale
The paper's central MELLON result is obtained by training a projection layer and Q-Former on human WebShop trajectories using a cross-entropy generation loss (Sec. 3.1.2) and comparing step-wise action accuracy before and after one epoch of training on a separate 338-instance test set (Secs. 4.3, 4.5, 5.1.1). This is an empirical measurement, not a derivation from the conclusion. There is no fitted parameter that is later renamed as a prediction: the 9.26% figure is simply 14.37% minus 5.11% in Table 2. The paper does not rely on load-bearing self-citations; all cited architectural components (ViT, Q-Former, BLIP-2, ReAct, WebShop) are external prior work, and no uniqueness theorem from the authors is invoked. VQAgent and Multimodal Ranker are reported as negative results and are evaluated with standard WebShop reward/success metrics, so their conclusions are not circular either. The genuine concerns in this paper are metric-validity and comparability issues: calling step-wise action accuracy 'task completion accuracy' (Sec. 5.1.1 vs Sec. 4.5) and using a test set 'curated from the midterm report' that is 'particularly challenging for baseline models' (Sec. 4.3). These are validity and comparability concerns, not circularity, because they do not make the measured outcome equivalent to the training signal or to a self-cited premise. Under the stated circularity definitions, the appropriate finding is no significant circularity (score 0).
Assumptions & free parameters
free parameters (3)
- Projection layer weights =
Learned on WebShop trajectories (not reported)
- Q-Former fine-tuned weights =
Fine-tuned on WebShop data (details not specified)
- Multimodal Ranker learnable balance parameter =
Trained with MSE loss (not reported)
assumptions (4)
- standard math Cross-entropy loss is an appropriate objective for action generation
- domain assumption WebShop is a valid and representative benchmark for multimodal web navigation
- ad hoc to paper Step-wise action accuracy is a meaningful proxy for task completion
- ad hoc to paper The curated 338-instance test set is representative enough for evaluation
Cite this review
Pith. "Pith review of MELLON - Multimodal Enhanced LLM for Online Navigation." pith.science (2026). https://pith.science/paper/QZFVBYPX
@misc{pith2026260809121,
author = {Pith},
title = {Pith review of: MELLON - Multimodal Enhanced LLM for Online Navigation},
year = {2026},
howpublished = {\url{https://pith.science/paper/QZFVBYPX}},
note = {Machine review of arXiv:2608.09121}
}
read the original abstract
Web navigation agents are capable of addressing various types of tasks on different websites. Current baselines on web navigation are either unimodal or lack strong reasoning abilities given multimodal inputs. Focusing on the WebShop benchmark, a real-world website simulation, we explore the alignment of text and images, as well as multimodal reasoning and planning abilities, to enhance the performance of web navigation agents. We propose three innovative multimodal enhancements: Multimodal Enhanced LLM for Online Navigation (MELLON), VQAgent, and Multimodal Ranker. MELLON demonstrates a significant improvement in task completion accuracy, with a 9.26% increase after just one epoch of training. Our findings suggest the necessity of further exploration into multimodal approaches, with a focus on more extensive training and alignment strategies to enhance the effectiveness of web navigation agents.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
Langley , title =
P. Langley , title =. Proceedings of the 17th International Conference on Machine Learning (ICML 2000) , address =. 2000 , pages =
2000
-
[2]
T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980
1980
-
[3]
M. J. Kearns , title =
-
[4]
Machine Learning: An Artificial Intelligence Approach, Vol. I. 1983
1983
-
[5]
R. O. Duda and P. E. Hart and D. G. Stork. Pattern Classification. 2000
2000
-
[6]
2022 , eprint=
PaLM: Scaling Language Modeling with Pathways , author=. 2022 , eprint=
2022
-
[7]
Suppressed for Anonymity , author=
-
[8]
Newell and P
A. Newell and P. S. Rosenbloom. Mechanisms of Skill Acquisition and the Law of Practice. Cognitive Skills and Their Acquisition. 1981
1981
Show all 42 references
-
[9]
A. L. Samuel. Some Studies in Machine Learning Using the Game of Checkers. IBM Journal of Research and Development. 1959
1959
-
[10]
2023 , journal=
Language Is Not All You Need: Aligning Perception with Language Models , author=. 2023 , journal=
2023
-
[11]
Junnan Li and Dongxu Li and Silvio Savarese and Steven Hoi , year=
-
[12]
arXiv preprint arXiv:2305.16355 , year=
PandaGPT: One Model To Instruction-Follow Them All , author=. arXiv preprint arXiv:2305.16355 , year=
-
[13]
arXiv preprint arXiv:2305.11854 , year=
Multimodal Web Navigation with Instruction-Finetuned Foundation Models , author=. arXiv preprint arXiv:2305.11854 , year=
-
[14]
arXiv:2306.06070 , year=
Mind2Web: Towards a Generalist Agent for the Web , author=. arXiv:2306.06070 , year=
-
[15]
arXiv:2307.12856 , year=
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis , author=. arXiv:2307.12856 , year=
-
[16]
arXiv:2207.01206 , year=
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents , author=. arXiv:2207.01206 , year=
-
[17]
arXiv:2307.13854 , year=
WebArena: A Realistic Web Environment for Building Autonomous Agents , author=. arXiv:2307.13854 , year=
-
[18]
arXiv:2301.13280 , year=
WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semantics , author=. arXiv:2301.13280 , year=
-
[19]
arXiv:1808.09132 , year=
Mapping natural language commands to web elements , author=. arXiv:1808.09132 , year=
-
[20]
Proceedings of the AAAI Conference on Artificial Intelligence , author=
WIERT: Web Information Extraction via Render Tree , volume=. Proceedings of the AAAI Conference on Artificial Intelligence , author=. 2023 , month=. doi:10.1609/aaai.v37i11.26546 , abstractNote=
2023 doi
-
[21]
CoVA: Context-aware Visual Attention for Webpage Information Extraction , journal =
Anurendra Kumar and Keval Morabia and Jingjin Wang and Kevin Chen. CoVA: Context-aware Visual Attention for Webpage Information Extraction , journal =. 2021 , url =. 2110.12320 , timestamp =
2021 arXiv
-
[22]
Muslea, Ion and Minton, Steve and Knoblock, Craig , year =
-
[23]
Advances in neural information processing systems , volume=
Neural discrete representation learning , author=. Advances in neural information processing systems , volume=
-
[24]
arXiv preprint arXiv:2206.08916 , year=
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks , author=. arXiv preprint arXiv:2206.08916 , year=
-
[25]
arXiv preprint arXiv:2304.10592 , year=
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models , author=. arXiv preprint arXiv:2304.10592 , year=
-
[26]
2022 , eprint=
WebGPT: Browser-assisted question-answering with human feedback , author=. 2022 , eprint=
2022
-
[27]
2023 , eprint=
ReAct: Synergizing Reasoning and Acting in Language Models , author=. 2023 , eprint=
2023
-
[28]
2022 , eprint=
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale , author=. 2022 , eprint=
2022
-
[29]
arXiv preprint arXiv:2304.11477 , year=
Llm+ p: Empowering large language models with optimal planning proficiency , author=. arXiv preprint arXiv:2304.11477 , year=
-
[30]
arXiv preprint arXiv:2307.02485 , year=
Building Cooperative Embodied Agents Modularly with Large Language Models , author=. arXiv preprint arXiv:2307.02485 , year=
-
[31]
doi:10.5281/zenodo.5143773 , url =
Ilharco, Gabriel and Wortsman, Mitchell and Wightman, Ross and Gordon, Cade and Carlini, Nicholas and Taori, Rohan and Dave, Achal and Shankar, Vaishaal and Namkoong, Hongseok and Miller, John and Hajishirzi, Hannaneh and Farhadi, Ali and Schmidt, Ludwig , title =. doi:10.5281...
-
[32]
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
Reproducible scaling laws for contrastive language-image learning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
-
[33]
ICML , year=
Learning Transferable Visual Models From Natural Language Supervision , author=. ICML , year=
-
[34]
2022 , url=
Christoph Schuhmann and Romain Beaumont and Richard Vencu and Cade W Gordon and Ross Wightman and Mehdi Cherti and Theo Coombes and Aarush Katta and Clayton Mullis and Mitchell Wortsman and Patrick Schramowski and Srivatsa R Kundurthy and Katherine Crowson and Ludwig Schmidt a...
2022
-
[35]
2023 , eprint=
Code Llama: Open Foundation Models for Code , author=. 2023 , eprint=
2023
-
[36]
2023 , eprint=
AgentBench: Evaluating LLMs as Agents , author=. 2023 , eprint=
2023
-
[37]
2017 , eprint =
Reinforced Mnemonic Reader for Machine Reading Comprehension , author =. 2017 , eprint =
2017
-
[38]
2023 , eprint=
Vision Transformer with Quadrangle Attention , author=. 2023 , eprint=
2023
-
[39]
2016 , eprint=
VQA: Visual Question Answering , author=. 2016 , eprint=
2016
-
[40]
2022 , eprint=
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks , author=. 2022 , eprint=
2022
-
[41]
2016 , eprint=
Generative Adversarial Imitation Learning , author=. 2016 , eprint=
2016
-
[42]
arXiv preprint arXiv:2304.15010 , year=
Llama-adapter v2: Parameter-efficient visual instruction model , author=. arXiv preprint arXiv:2304.15010 , year=
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.