REVIEW 4 major objections 5 minor 11 references
Predicting Actions to Help Predict Translations
T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Action-recognition video features improve translation quality by up to 0.4 BLEU, with the effect concentrated when action verbs are masked in the source text.
desk verdict A well-designed probing study of action-aware visual features for MMT on How2, whose headline BLEU gains are too small and under-tested to carry the central claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the pairing of action-specific visual features with a transformer decoder that can attend to them. The features come from a 3D CNN fine-tuned to classify the 339 action categories in the Moments in Time dataset: conv4 is the final convolutional feature map, treated as 49 video regions; emb is a 339 by 300 matrix of word2vec embeddings of the action labels, each scaled by the CNN's softmax posterior for that label. The model conditions on these features either by adding a projected video summary to encoder outputs (AIC) or by adding a cross-attention layer over the visual features in the decoder (AIF). Two probes complete the argument: masking action verbs or all verbs in the source text, and incongruent decoding, which reverses the visual feature order to test whether the model is genuinely using them.
What would settle it
Re-run each configuration in Table 1 with several random seeds and compute a paired significance test on the test set; if the action-tuned visual models do not consistently beat text-only by more than the seed-induced noise floor, the central claim of visual benefit is not supported.
Extended reading notes
Core claim
The paper's central claim is that action-tuned visual features can improve English-to-Portuguese translation on How2 beyond a text-only transformer baseline, and that the improvement is strongest when the source text is missing action-related words. The best unmasked result is 56.2 BLEU with the AIF-emb model versus 55.9 for text-only; with action verbs masked, AIF-conv4 reaches 53.8 versus 53.6; with all verbs masked, AIF-emb reaches 44.5 versus 44.1. The authors also report that feeding incongruent visual features lowers BLEU in almost every setting, up to a 1.0-point drop, which they take as evidence that the multimodal models rely on the visual information rather than ignoring it. A complementary human evaluation, on the action-masked setting, ranks AIF-emb above both text-only and AIF-conv4.
Load-bearing premise
The premise that the small BLEU improvements, such as 56.2 versus 55.9, are stable effects rather than run-to-run noise is load-bearing, because the paper reports single runs without significance tests or variance.
Editorial extensions
If this is right
- Visual features trained for action classification should be preferred over generic video summaries for translating video-derived text; in every masking setting the action-tuned models produce the best BLEU scores.
- Action-aware multimodal models are more robust when the source text is noisy or incomplete, because visual attention can compensate for masked action verbs: incongruent visual features cause drops up to 1.0 BLEU, with larger drops in the masked settings.
- Representing action posteriors in word-embedding space (emb) is a viable way to condition translation on visual actions, and human evaluators preferred it for preserving meaning in action-masked sentences.
- Visual features help but do not close the gap between unmasked and masked input; even the best multimodal models remain below the unmasked text-only baseline when verbs are removed.
Reading between the lines
- A direct test of the paper's small BLEU deltas would be to run each configuration with several random seeds and report confidence intervals; until then, the 0.2-0.4 point differences could be within run-to-run noise.
- Because the reported benefit concentrates in verb-masked settings, the same action features could be tested on speech-to-text translation with automatic-speech-recognition noise, where verbs are commonly misrecognized or dropped.
- The emb representation creates a shared space for visual actions and source verbs, so it could be used to add a visual grounding objective or to steer the decoder toward action-verb translations without a separate visual attention mechanism.
- The experiments use videos that often focus on the speaker; on datasets with more scene diversity, the action features might produce larger improvements than 0.4 BLEU.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper asks whether action-related visual features extracted from How2 videos can improve English-to-Portuguese translation with a transformer-based model, under three input conditions: original text (ORG), source text with action verbs masked (ACT), and source text with all verbs masked (ALL). It compares three visual feature types (a pre-extracted videosum, a conv4 feature from a 3D CNN fine-tuned on Moments in Time, and a softmax-weighted word-embedding matrix over that CNN's 339 action categories), using additive and attention-based conditioning. It reports modest BLEU gains in Table 1, incongruent-decoding degradation in Table 2, and a human ranking study in Table 3. The central claim is that action-classification features are beneficial, especially when action-related words are masked in the source.
Significance. If the claimed effects are real, the paper would make a modest but useful contribution to multimodal machine translation: it provides evidence that features tuned to an action taxonomy, rather than generic video representations, can help recover masked action verbs, and it offers a probing methodology (masking, incongruent decoding, human ranking) that others can reuse. The design has real strengths: multiple feature types are compared, the masking protocol is clearly defined, incongruent decoding is used as a sanity check that the model relies on the visual stream, and the human evaluation targets the exact masked-verb phenomenon. The paper also connects the How2 and Moments in Time resources, which makes the feature-extraction pipeline reproducible. The main caveat is that the quantitative evidence is currently too weak to establish the improvement claims: the reported deltas are small, single-run, and not accompanied by significance tests or confidence intervals.
major comments (4)
- [Section 5, Table 1] The entire quantitative case for the central claim rests on single-run BLEU deltas of +0.3 (AIF-emb on ORG), +0.4 (AIF-emb on ALL), and +0.2 (AIF-conv4 on ACT) against the text-only baseline. The test set contains only 2,305 sentences, transformer training is known to be seed-sensitive, and the paper explicitly says it uses Multeval (Clark et al., 2011), a tool designed for paired significance testing and for controlling optimizer instability. Reporting no confidence intervals, p-values, or multiple-seed runs leaves open the possibility that the reported differences are run-to-run noise. This is load-bearing because the Introduction and Conclusion claim that visual features 'can be beneficial'; please add paired bootstrap tests or multiple-seed means with variance, and report which of the Table 1 differences are statistically reliable.
- [Section 3 with Section 2] There is a partial circularity that needs to be discussed explicitly. The ACT mask is constructed from the 339 action verbs in Moments in Time (Section 2), and the two best-performing feature types, conv4 and emb, are both produced by a CNN fine-tuned on exactly those 339 categories (Section 3). Consequently, when an ACT verb is replaced by V, the visual classifier output is a direct cue for the masked category: the model can, in principle, read off the verb identity from the softmax or conv4 representation and map it to a translation. This makes the ACT gains partly a consequence of the closed-set label overlap rather than evidence that visual features provide general semantic support for translation. The paper acknowledges the relation only as 'closely related' in Section 5; it should be discussed as an unavoidable confound. A concrete control would be to evaluate on verbs that are not in the 339 action set, or to use action features from a different taxonomy (e.g., Kinetics-400) and check whether the masking gains persist.
- [Section 5.1, Table 3; Abstract and Section 6] The claimed confirmation by human evaluation is not supported by the reported numbers. On ACT, the automatic evaluation in Table 1 favors AIF-conv4 (53.8) over text-only (53.6) and AIF-emb (53.5), but the human evaluation in Table 3 favors AIF-emb (0.81) over text-only (0.75) and AIF-conv4 (0.73). The abstract states that human evaluation confirms the automatic results, but the rankings disagree on the best system. In addition, the human study uses only 50 samples per annotator, four annotators, and reports no inter-annotator agreement or significance on the 0.06-point difference. Please reconcile the two evaluations and either add reliability statistics or soften the confirmation claim to what the data actually show.
- [Section 5, Table 2] The interpretation of the incongruent-decoding results is logically too strong. The paper says that the performance drops under reversed visual features 'prove that multimodality indeed exerts positive influence on the translation.' A model that is sensitive to the visual stream will degrade when that stream is corrupted, even if the visual features are neutral or slightly harmful in the congruent setting. The incongruent test is therefore evidence that the model uses the visual modality, not proof that using it helps translation. Please rephrase this conclusion as evidence of reliance on the visual stream, and keep the claim of benefit tied to the (ideally significance-tested) congruent-condition results.
minor comments (5)
- [Section 1] There is a duplicated 'and' in 'and and different ways' and a typo in 'representating'; please proofread the introduction.
- [Section 2] The masking description is ambiguous: 'The masking is performed in all sentences containing (action) verbs in the source language' reads as if only sentences containing action verbs are affected. Please clarify whether ALL masking replaces every verb in every sentence or only verbs in sentences that contain at least one action verb.
- [Section 3] For conv4 and emb, the paper says 'We sample 16 equi-distant frames for each video, feed them to the network,' but it is not clear whether these frames are used as a single 16-frame clip or as multiple clips; please specify the input construction.
- [Table 1 caption] The caption says 'Bold highlights our best results,' but the rendered table does not show any bold formatting; please ensure that the best score in each column is actually marked.
- [Figure 2 caption] The caption says 'Underlined text denotes masked words and their translations,' but no underlining is visible in the figure; please mark the relevant spans clearly.
Circularity Check
The ACT masking condition and the conv4/emb visual features are defined over the same 339-verb Moments-in-Time taxonomy, so the masking-setting gains are partly self-referential; ORG and incongruent-decoding results provide partial independent grounding.
-
self definitional
[Section 2 'Dataset and Masking Strategies' and Section 3 'Visual features' (with results in Table 1 and Table 2).]
"All verbs which correspond to an action as defined in the action categorisations of the Moments in Time dataset (Monfort et al., 2019) are replaced by a placeholder. ... conv4: the final convolutional layer of a 3D ResNet-50 CNN trained to classify the 339 action verbs from Monfort et al. (2019); ... emb: a word embedding matrix for the 339 action verbs, with the embedding of each verb weighted by the final softmax layer of the same CNN for conv4."
The ACT masking condition removes exactly the source tokens that match the 339 lemmatised action verbs from Monfort et al. (2019), while the two visual features that produce the reported gains (conv4 and emb) come from a CNN trained to classify that same set of 339 verbs. For emb, the feature is literally a softmax-weighted embedding matrix indexed by those 339 action labels, so attending to it gives the decoder a posterior distribution over the very categories from which the masked tokens were drawn. Therefore the claim that action-classifier visual features help 'in particular for masking settings' is, for ACT, supported by a condition where the masked vocabulary and the visual-feature label vocabulary are the same set by construction.
full rationale
The central quantitative claim is that visual features from an action-classification CNN improve translation, especially when the source masks action-related verbs. The strongest masking-specific evidence is not fully independent of its own construction: ACT masks exactly the tokens matching the 339 action verbs from Monfort et al. (2019), and the conv4/emb features are produced by a CNN trained to recognise those same 339 verbs, with emb even being a softmax-weighted embedding matrix over them. This makes the ACT improvement partly tautological and matches the self-definitional circularity pattern. However, the paper also reports an improvement on unmasked ORG text (AIF-emb 56.2 vs text-only 55.9) and a substantial incongruent-decoding penalty (up to 1.0 BLEU drop), both of which are not explained by the ACT/feature vocabulary overlap and provide independent evidence that the visual features are being used. The absence of significance tests and multiple seeds is a robustness concern, not a circularity concern under the rules. Self-citations to Ive et al. (2019) and Caglayan et al. (2019) are methodological rather than load-bearing. Overall, partial circularity due to the ACT/feature overlap warrants a score of 4.
Assumptions & free parameters
assumptions (3)
- domain assumption Visual features from pre-trained action-recognition CNNs (ResNeXt-101, ResNet-50 on Moments in Time) are informative for translating How2 instructional videos.
- domain assumption Masking action verbs or all verbs with a placeholder simulates realistic noisy or incomplete input text.
- domain assumption The Monfort et al. (2019) list of 339 action verbs is a suitable lexicon for identifying action words in How2 transcripts.
Cite this review
Pith. "Pith review of Predicting Actions to Help Predict Translations." pith.science (2026). https://pith.science/paper/B6VR662N
@misc{pith2026190801665,
author = {Pith},
title = {Pith review of: Predicting Actions to Help Predict Translations},
year = {2026},
howpublished = {\url{https://pith.science/paper/B6VR662N}},
note = {Machine review of arXiv:1908.01665}
}
read the original abstract
We address the task of text translation on the How2 dataset using a state of the art transformer-based multimodal approach. The question we ask ourselves is whether visual features can support the translation process, in particular, given that this is a dataset extracted from videos, we focus on the translation of actions, which we believe are poorly captured in current static image-text datasets currently used for multimodal translation. For that purpose, we extract different types of action features from the videos and carefully investigate how helpful this visual information is by testing whether it can increase translation quality when used in conjunction with (i) the original text and (ii) the original text where action-related words (or all verbs) are masked out. The latter is a simulation that helps us assess the utility of the image in cases where the text does not provide enough context about the action, or in the presence of noise in the input text.
Figures
Reference graph
Works this paper leans on
-
[5]
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,
-
[8]
doi: 10.1109/TPAMI.2019.2901464
ISSN 0162-8828. doi: 10.1109/TPAMI.2019.2901464. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics , pp. 311–318,
arXiv 2019
-
[10]
Sennrich, R., Haddow, B., and Birch, A
URL http://arxiv.org/ abs/1811.00347. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909,
-
[12]
N., Gouws, S., Jones, L., Kaiser, Ł., Kalchbrenner, N., Parmar, N., et al
Vaswani, A., Bengio, S., Brevdo, E., Chollet, F., Gomez, A. N., Gouws, S., Jones, L., Kaiser, Ł., Kalchbrenner, N., Parmar, N., et al. Tensor2tensor for neural machine translation. arXiv preprint arXiv:1803.07416,
-
[13]
Aggre- gated residual transformations for deep neural networks
Predicting Actions to Help Predict Translations Xie, S., Girshick, R., Doll´ar, P., Tu, Z., and He, K. Aggre- gated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vi- sion and pattern recognition, pp. 1492–1500, 2017
work page 2017
-
[2011]
URL https: //www.aclweb.org/anthology/P11-2031
As- sociation for Computational Linguistics. URL https: //www.aclweb.org/anthology/P11-2031. Elliott, D., Frank, S., Sima’an, K., and Specia, L. Multi30k: Multilingual english-german image descriptions. In 5th Workshop on Vision and Language , pp. 70–74, Berlin, Germany,
-
[2013]
Monfort, M., Andonian, A., Zhou, B., Ramakrishnan, K., Bargal, S
URL http://arxiv.org/abs/ 1301.3781. Monfort, M., Andonian, A., Zhou, B., Ramakrishnan, K., Bargal, S. A., Yan, T., Brown, L., Fan, Q., Gutfruend, D., V ondrick, C., et al. Moments in time dataset: one million videos for event understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–8,
-
[2014]
Efficient estimation of word representations in vector space
Mikolov, T., Chen, K., Corrado, G., and Dean, J. Efficient estimation of word representations in vector space. In Bengio, Y . and LeCun, Y . (eds.),Proceedings of the 1st International Conference on Learning Representations, ICLR 2013, Workshop Track Proceedings, Scottsdale, AZ, USA, May
work page 2013
Show all 11 references
-
[2016]
URL http://www.aclweb
doi: 10.18653/v1/W16-2346. URL http://www.aclweb. org/anthology/W16-2346. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems, pp. 5998–6008,
-
[2018]
URL http://aclweb.org/anthology/ W18-6402. Bojar, O., Chatterjee, R., Federmann, C., Graham, Y ., Had- dow, B., Huang, S., Huck, M., Koehn, P., Liu, Q., Lo- gacheva, V ., Monz, C., Negri, M., Post, M., Rubino, R., Specia, L., and Turchi, M. Findings of the 2017 conference on m...
2017
-
[2019]
Clark, J
URL http://arxiv.org/abs/1903.08678. Clark, J. H., Dyer, C., Lavie, A., and Smith, N. A. Bet- ter hypothesis testing for statistical machine translation: Controlling for optimizer instability. In Proceedings of the 49th Annual Meeting of the Association for Com- putational Lin...
1903 arXiv
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.