REVIEW 5 major objections 5 minor 23 references
Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation
T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read An entropy term in the loss is claimed to sharpen video attention and improve scene-aware dialogue answers.
desk verdict A real but small official benchmark win undercuts by a garbled training objective and a missing hyperparameter; the sign objection in the reader's take is mathematically backward. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Entropy-Enhanced Dynamic Memory Network (DMN). A DMN reads encoded input facts with an attention-based GRU inside episodic memory episodes; the paper runs one module on I3D visual features and one on Vggish audio features, each LSTM-encoded, and appends the entropy penalty $-\gamma \sum_m \sum_r p(r)\log p(r)$ to the cross-entropy sequence-generation loss (Eq. 11), so that diffuse attention over the $N$ video segments becomes expensive. The resulting audio and visual context vectors enter a question-guided fusion gate together with soft-attention summary and caption contexts, and the decoder receives the previous answer's last hidden state as dialogue context.
What would settle it
Train the Entropy-Enhanced DMN exactly as described but with $\gamma = 0$, keeping all other settings and hyperparameters fixed, and compare scores on the DSTC7 test set; if the model without the entropy term matches the entropy-enhanced scores, or if increasing $\gamma$ does not reduce the entropy of the attention gates over video segments, the paper's stated concentration mechanism is not what produces the improvement.
Extended reading notes
Core claim
The central claim is that attaching an entropy regularizer to the attention gates inside a Dynamic Memory Network makes the model allocate its video attention to fewer, more relevant segments, and that this sharper focus is what lifts answer quality on the Audio Visual Scene-aware Dialog task. The paper argues that ordinary soft attention over long videos produces diffuse distributions, so it adds $-\gamma \sum_m \sum_r p(r)\log p(r)$ to the loss, with $r$ indexing the attention distributions in the audio and visual memory modules. In the authors' telling, the resulting Entropy-Enhanced DMN selects video content that answers the question better, and when combined with attention over summary and caption plus the dialogue history, the complete system beats the official baseline on the challenge's objective and subjective evaluations.
Load-bearing premise
The result stands on the assumption that the entropy term, at the amount and sign used in training (the paper does not report $\gamma$), actually makes the video attention distribution more concentrated, and that this concentration, not some side effect of changing the loss, is what improves the generated answers.
Editorial extensions
If this is right
- With all five modalities (audio, visual, dialogue history, summary, and caption), the entropy-enhanced system is reported to beat the released baseline on the official objective and subjective evaluation metrics.
- Removing video entirely and relying only on the text modalities produces worse results, so the video memory network carries information that caption and summary alone do not.
- Feeding the previous answer's final hidden state into the decoder improves most objective metrics, supporting the dialogue-modeling hypothesis.
- Increasing the number of episodic memory updates from two to three yields only marginal gains on some metrics and costs substantially more computation.
- Audio plus text is reported to do better than visual plus text, suggesting that audio features are less noisy in this dataset.
Reading between the lines
- The paper does not report the value of $\gamma$, so the experiment that would pin the mechanism is a sweep: varying $\gamma$ from zero upward while measuring attention-gate entropy would show whether score gains track concentration directly.
- The same entropy-regularized attention idea should transfer to other long-sequence multimodal generation tasks, such as video captioning or embodied question answering, where attention over long inputs tends to flatten; the paper does not test this transfer.
- Because no ablation against generic regularizers is reported, an equal-cost baseline such as dropout on attention scores or label smoothing could in principle produce the same gains; that comparison remains open.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an entropy-enhanced Dynamic Memory Network (DMN) for the DSTC7 Audio Visual Scene-aware Dialog task. The model encodes visual and audio features with separate DMNs, applies soft attention to captions and summaries, and fuses all modalities with a gating mechanism. The claimed novelty is an entropy term added to the training loss (Eq. (11)) that is intended to make the video attention distribution more concentrated, so each question focuses on a small set of video segments. The authors report improvements over the released baseline on the official DSTC7 objective and subjective metrics, and present ablations on a prototype test set that examine the contribution of each modality, dialogue modeling, and the number of episodic memory updates.
Significance. If the central contribution is valid, an entropy regularizer on attention distributions is a simple and potentially useful idea for multimodal dialogue systems, and the reported official-evaluation gains over the baseline are encouraging. However, the paper's only formal statement of the proposed objective, Eq. (11), is not a coherent training loss as written: the first term is not standard negative log-likelihood, and the second term's target distributions are unspecified. The hyperparameter γ is never reported, no attention-entropy measurements are given, and the comparison lacks statistical significance tests. The paper also contains inconsistent baseline numbers between Table 1 and Table 2. These issues make the central claim not reproducible from the manuscript and prevent a clear assessment of whether the reported gains are caused by the entropy mechanism. The paper does not provide code, but the official evaluation is against an external benchmark, which is a strength.
major comments (5)
- [Entropy, Eq. (11)] The first term of Eq. (11), -1/K * sum_{i=1}^K p(y_i) log p(y_i), is not a standard negative log-likelihood for sequence generation. If p(y_i) is the model's probability assigned to the ground-truth token, the conventional term is -1/K * sum_i log p(y_i). As written, the term is maximized when p(y_i) = 1/e and would not drive the model toward the correct tokens. This is load-bearing because Eq. (11) is the only formal specification of the proposed entropy-enhanced training objective. Please correct the equation, define p(y_i) explicitly, or state that it is a typo.
- [Entropy, Eq. (11)] The second term of Eq. (11) is also underspecified. The text says 'r represents each distribution in m,' but it is not stated whether r ranges over the attention gates g_i^t in Eq. (5), the episode-level attention weights, the fusion weights β in Eq. (8), or some other set of distributions; nor are the summation ranges over time steps and episodes given. Additionally, the value of γ is never reported in the experiments. Without these details, the exact training objective cannot be implemented or reproduced. Note that, with the formula as written, -γ * sum_m sum_r p(r) log p(r) = +γ * sum_m H[p(r)] for γ > 0, so minimizing the loss would indeed penalize high entropy and thus concentrate the distributions; the sign is consistent with the stated goal, contrary to a possible naive reading. The problem is the missing specification of r and γ.
- [Experiments, Entropy Analysis] The paper repeatedly claims that the entropy mechanism makes the attention distribution 'more concentrated' and 'control[s] the attention distribution higher,' but no quantitative evidence is provided. The Entropy Analysis section only cites the BLEU/METEOR differences between DMN and Entropy-Enhanced DMN in Table 1. Please report direct measurements of the attention entropy or sparsity (e.g., mean entropy or effective number of attended segments) for DMN vs. Entropy-Enhanced DMN, ideally as a function of γ. Without this, the paper's central mechanistic claim is unsupported.
- [Experiments, Tables 1 and 2] The baseline results are inconsistent across tables: Table 1 reports Baseline (Hori et al. 2018) with BLEU-1 = 0.273, BLEU-4 = 0.084, and METEOR = 0.117, while Table 2 reports the same baseline with BLEU-1 = 0.626, BLEU-4 = 0.309, and METEOR = 0.215. The text states that 'We test all experiments on the prototype testing set' and that 'Table 2 is the final result released by the official,' but Table 1's caption says 'The first part is the official objective evaluation values.' Please clarify which rows correspond to the prototype test set and which to the official test set, and ensure that comparisons in the text and tables use the same evaluation configuration.
- [Experiments, Table 1] The claimed improvements over the DMN baseline (e.g., DMN (A, V, D, S, C) vs. Entropy-Enhanced DMN (A, V, D, S, C) in Table 1) are small, and no statistical significance tests, confidence intervals, or error bars are reported. Since the evaluation is on a single test split, it is unclear whether the differences are meaningful. Please add significance testing or variance estimates over multiple runs, or at least state the number of runs and their standard deviations.
minor comments (5)
- [Abstract] The abstract contains a spacing error: 'TheDialog System Technology Challenge' should be 'The Dialog System Technology Challenge'.
- [Data] The sentence 'we will first elaborate the date format' should read 'data format'.
- [Discussion] The qualitative analysis section refers to 'the dialogue in Table 2' and 'the last question in Table 3', but the sample dialogue appears to be in Table 3 and the question in Table 4. Please fix the cross-references.
- [Entropy] The phrase 'control the attention distribution higher' is unclear and appears to conflict with 'more concentrated'. Please rephrase to state the intended effect precisely (e.g., lower entropy, lower effective support).
- [Experiments] There is a typo in the sentence 'DMN (A, V, D, S, C) is not camparable with our submitted system' — 'camparable' should be 'comparable'.
Circularity Check
No circularity: the entropy-enhanced DMN is evaluated against an external DSTC7 benchmark and a released baseline, with no fitted quantity renamed as a prediction.
full rationale
The paper's central claim is that adding an entropy regularizer to the Dynamic Memory Network loss improves scene-aware dialogue generation. The evaluation is performed against the official DSTC7 test set and the released baseline of Hori et al. (2018), which are external to the paper and not constructed from the paper's own outputs. The entropy term in Eq. (11) is a regularizer added to the training loss, not a parameter fitted to a subset of the test data and then reported as a prediction. No step in the derivation chain reduces to its own inputs: the DMN architecture is cited from prior work, the video features are extracted with I3D and Vggish, the attention mechanisms are standard, and the reported gains are measured by external BLEU, METEOR, ROUGE-L, CIDEr, and human evaluation scores. The lack of a reported value for gamma, the ambiguity in the definition of the distributions r, and the questionable first term of Eq. (11) are reproducibility and correctness concerns, not circularity. The paper does not rely on a self-citation chain to justify its central premise, nor does it rename a known pattern or import a uniqueness theorem from its own authors. Therefore, the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (4)
- gamma (entropy weight) =
not reported
- M (number of episodic memory updates) =
2 or 3
- hidden size =
128
- beam size =
not reported
assumptions (5)
- domain assumption I3D and Vggish extract features that contain information useful for answering dialogue questions.
- domain assumption The AVSD dataset annotations are of sufficient quality to learn from.
- standard math The softmax attention and GRU equations in Eqs (1)-(5) behave as standard.
- ad hoc to paper Minimizing the entropy of the attention distribution concentrates it on a few segments.
- ad hoc to paper Passing the previous answer's last hidden state into the decoder helps answer the current question.
Cite this review
Pith. "Pith review of Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation." pith.science (2026). https://pith.science/paper/WIQRS446
@misc{pith2026190808191,
author = {Pith},
title = {Pith review of: Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/WIQRS446}},
note = {Machine review of arXiv:1908.08191}
}
read the original abstract
With increasing information from social media, there are more and more videos available. Therefore, the ability to reason on a video is important and deserves to be discussed. TheDialog System Technology Challenge (DSTC7) (Yoshino et al. 2018) proposed an Audio Visual Scene-aware Dialog (AVSD) task, which contains five modalities including video, dialogue history, summary, and caption, as a scene-aware environment. In this paper, we propose the entropy-enhanced dynamic memory network (DMN) to effectively model video modality. The attention-based GRU in the proposed model can improve the model's ability to comprehend and memorize sequential information. The entropy mechanism can control the attention distribution higher, so each to-be-answered question can focus more specifically on a small set of video segments. After the entropy-enhanced DMN secures the video context, we apply an attention model that in-corporates summary and caption to generate an accurate answer given the question about the video. In the official evaluation, our system can achieve improved performance against the released baseline model for both subjective and objective evaluation metrics.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
, " * write output.state after.block = add.period write newline
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 '...
-
[3]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[4]
Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2017. Bottom-up and top-down attention for image captioning and vqa. arXiv preprint arXiv:1707.07998
arXiv 2017
-
[5]
Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Lawrence Zitnick, C.; and Parikh, D. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , 2425--2433
work page 2015
-
[6]
Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473
arXiv 2014
-
[7]
Carreira, J., and Zisserman, A. 2017. Quo vadis, action recognition? a new model and the kinetics dataset. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on , 4724--4733. IEEE
work page 2017
-
[8]
Gao, J.; Ge, R.; Chen, K.; and Nevatia, R. 2018. Motion-appearance co-memory networks for video question answering. arXiv preprint arXiv:1803.10906
work page Pith review arXiv 2018
Show all 23 references
-
[9]
P.; Gemmeke, J
Hershey, S.; Chaudhuri, S.; Ellis, D. P.; Gemmeke, J. F.; Jansen, A.; Moore, R. C.; Plakal, M.; Platt, D.; Saurous, R. A.; Seybold, B.; et al. 2017. Cnn architectures for large-scale audio classification. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE Internati...
2017
-
[10]
Hochreiter, S., and Schmidhuber, J. 1997. Long short-term memory. Neural computation 9(8):1735--1780
1997
-
[11]
K.; Cartillier, V.; Lopes, R
Hori, C.; Alamri, H.; Wang, J.; Winchern, G.; Hori, T.; Cherian, A.; Marks, T. K.; Cartillier, V.; Lopes, R. G.; Das, A.; et al. 2018. End-to-end audio visual scene-aware dialog using multimodal attention-based video features. arXiv preprint arXiv:1806.08409
2018 arXiv
-
[12]
Kumar, A.; Irsoy, O.; Ondruska, P.; Iyyer, M.; Bradbury, J.; Gulrajani, I.; Zhong, V.; Paulus, R.; and Socher, R. 2016. Ask me anything: Dynamic memory networks for natural language processing. In International Conference on Machine Learning , 1378--1387
2016
-
[13]
LeCun, Y.; Bengio, Y.; and Hinton, G. 2015. Deep learning. nature 521(7553):436
2015
-
[14]
Luong, M.-T.; Pham, H.; and Manning, C. D. 2015. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025
2015 arXiv
-
[15]
Pennington, J.; Socher, R.; and Manning, C. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , 1532--1543
2014
-
[16]
Ren, S.; He, K.; Girshick, R.; and Sun, J. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems , 91--99
2015
-
[17]
Tran, D.; Bourdev, L.; Fergus, R.; Torresani, L.; and Paluri, M. 2015. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision , 4489--4497
2015
-
[18]
Wang, J.; Jiang, W.; Ma, L.; Liu, W.; and Xu, Y. 2018. Bidirectional attentive fusion with context gating for dense video captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 7190--7198
2018
-
[19]
Williams, R. J. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3-4):229--256
1992
-
[20]
Xiong, C.; Merity, S.; and Socher, R. 2016. Dynamic memory networks for visual and textual question answering. In International conference on machine learning , 2397--2406
2016
-
[21]
Xu, H., and Saenko, K. 2016. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In European Conference on Computer Vision , 451--466. Springer
2016
-
[22]
Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhudinov, R.; Zemel, R.; and Bengio, Y. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning , 2048--2057
2015
-
[23]
F.; Polymenakos, L.; Gunasekara, C.; Lasecki, W
Yoshino, K.; Hori, C.; Perez, J.; D'Haro, L. F.; Polymenakos, L.; Gunasekara, C.; Lasecki, W. S.; Kummerfeld, J.; Galley, M.; Brockett, C.; Gao, J.; Dolan, B.; Gao, S.; Marks, T. K.; Parikh, D.; and Batra, D. 2018. The 7th dialog system technology challenge. arXiv preprint
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.