REVIEW 4 major objections 6 minor 30 references
Semi-Supervised Image-Based Narrative Extraction: A Case Study with Historical Photographic Records
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A semi-supervised algorithm extracts coherent visual storylines from historical photo collections.
desk verdict A legitimate, clearly written proof-of-concept for adapting narrative maps to visual data, but the effectiveness claim is undercut by an evaluation that feeds the expert's own labels and dates into the algorithm while withholding them from the random baseline. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the adapted narrative maps pipeline: a directed acyclic graph whose nodes are photographs and whose edge weights encode coherence, computed as a combination of DETR feature similarity and label-spreading outputs for thematic category and date. A linear program maximizes the weakest-link coherence of the path from a fixed source to a fixed target while enforcing coverage constraints over thematic clusters, and the main storyline is the maximum-likelihood path through that graph. The key adaptation is the replacement of the original unsupervised clustering with semi-supervised label spreading, which is what allows partial expert labels and date seeds to steer the narrative structure.
What would settle it
Conduct the comparison with a second expert who is blinded to the first expert's labels and independently builds the ground-truth timelines and seed labels; if the narrative-maps advantage over random sampling disappears or becomes non-significant for 10+ image timelines, the reported performance depended on shared expert knowledge rather than intrinsic narrative extraction.
Extended reading notes
Core claim
The paper's central claim is that the narrative maps algorithm, originally designed for news text, can be adapted to visual data by replacing its unsupervised clustering step with semi-supervised label spreading and by defining edge coherence from DETR image features combined with propagated thematic and temporal labels. The extracted narrative is the most coherent directed path from a fixed start image to a fixed end image, chosen by a linear program that maximizes the weakest-link coherence subject to coverage constraints over thematic clusters. Against expert-curated timelines of lengths 5 through 30, the method significantly outperforms random sampling on DTW distance and cosine similarity for timelines of 10 or more images in the UMAP-reduced space, and for most longer timelines in the original high-dimensional space. The authors interpret this as evidence that the coherence-based graph objective captures narrative structure rather than merely following the dataset's class distribution, while acknowledging that random sampling also performs well on the shortest timeline and on this small, visually homogeneous collection.
Load-bearing premise
The expert-curated timelines serve as ground truth, yet the same expert supplied the thematic labels and date seeds used by the algorithm, and the start and end images are fixed to that expert's choices; if this ground truth is not independent, the advantage over random sampling may largely reflect injected expert knowledge.
Editorial extensions
If this is right
- For collections with partial expert annotations, the method produces a narrative timeline without requiring every image to be labeled.
- The advantage over random sampling is clearest for timelines of 10 or more images, so the method is best suited to reconstructing extended event sequences rather than very short stories.
- Using UMAP-reduced embeddings preserves the advantage while lowering computational cost, though the paper reports mixed results in the high-dimensional space for some timeline lengths.
- The extracted narratives are not exact replicas of an expert's selection; they offer alternative, verifiable orderings that keep human oversight in the loop.
- The authors expect the benefit over random sampling to be larger in noisier, more diverse collections, where random selections are less likely to land on coherent sequences.
Reading between the lines
- If the single-expert ground truth is the main confound, a decisive follow-up is to have a second, independent expert construct timelines and seed labels from the primary texts alone; a large performance drop would indicate the algorithm is partly recovering that annotator's own labeling rather than intrinsic narrative structure.
- The same pipeline could transfer to other partially ordered visual corpora, such as press photo archives or personal albums, where the narrative is retrospective rather than authored.
- Because only partial labels and a start/end pair are needed, the method could become an interactive exploration tool that proposes the most coherent storyline between two chosen images and lets a researcher edit and rerun.
- Domain-specific fine-tuning of the visual encoder on historical photography, rather than generic object-detection features, is a testable improvement the paper leaves open.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper adapts the text-based Narrative Maps algorithm to visual data for semi-supervised narrative extraction from historical photographs. Using DETR features on a 500-image subset of the ROGER collection, the method applies label spreading to propagate expert-provided thematic and date labels, builds a directed coherence graph, and extracts a main storyline between fixed start and end images. The extracted narratives are compared with six expert-curated timelines (lengths 5–30) using DTW distance and average cosine similarity, against a random sampling baseline, and a qualitative assessment by a domain expert is reported. The central claim is that the Narrative Maps approach generally outperforms random sampling for longer timelines (10+ images, p < 0.05), with expert evaluation confirming historical accuracy and coherence.
Significance. If the effectiveness claim were established, the paper would offer a useful proof-of-concept for computational narrative analysis in visual cultural heritage, and the release of source code and experiments is a practical strength. However, the evaluation design does not currently support the central claim: the same expert knowledge is used to construct the ground truth, to provide the labels and date seeds consumed by the method, and to judge the qualitative output. The comparison against random sampling therefore conflates algorithmic narrative extraction with injected expert supervision. The authors are transparent about several limitations, including a small and highly homogeneous dataset and the reliance on expert labels, but no experiment isolates the effect of this supervision. The significance of the reported quantitative results is additionally weakened by the lack of multiple-comparison control and by small effect sizes.
major comments (4)
- [Sections 3.2, 3.5, 3.6] The ground truth and the supervision given to the proposed method come from the same expert. Section 3.5 states that a domain expert created the six evaluation timelines and that the expert's partial labels and dates are fed into the semi-supervised pipeline; Section 3.2 uses these expert-provided thematic labels and date approximations as seeds for label spreading; and Section 3.6 fixes the start and end images to the expert-curated timeline. The random sampling baseline receives none of this supervision. As a result, the NM-over-RS advantage reported in Table 1 may reflect injected expert knowledge rather than intrinsic narrative extraction ability. This is load-bearing for the paper's central claim. The manuscript's own limitation paragraph in Section 5.1 acknowledges that expert-provided labels can introduce bias, but no experiment isolates this effect. I would need to see either an independently constructed ground truth, an ablation that withholds the expert labels and dates from NM, or a baseline that is given the same cluster and date information before accepting the effectiveness claim.
- [Section 4.1, Table 1] The statistical comparison uses 20 trials and raw t-tests for each timeline length and each metric, with six lengths and four metrics resulting in 24 tests. No multiple-comparison correction is applied. With this many tests, several p < 0.05 results are expected by chance; for instance, after a Bonferroni correction the high-dimensional similarity differences at L=10 (p=0.0399), L=15 (p=0.0257), L=20 (p=0.0033), and L=30 (p=0.0087) would no longer meet the family-wise significance threshold. The effect sizes are also small (e.g., L=10 high-dimensional similarity 0.915 vs 0.895), and the L=25 high-dimensional similarity comparison is not significant (p=0.3514). The abstract's sweeping claim that NM generally outperforms random sampling for 10+ images is therefore stronger than the currently reported evidence supports.
- [Section 4.2] The qualitative evaluation is performed by the same domain expert who created the baseline timelines and supplied the labels and dates used by the method, and there is no indication of blinding. This cannot serve as an independent confirmation of historical accuracy and coherence. In particular, the statement that the algorithmic selections could sometimes be interpreted as possessing greater relevance from an objective, image-content-based perspective is itself a subjective judgment by the same expert. A blind evaluation by a second expert, or at minimum a clearly independent evaluator, is needed to support the qualitative claims.
- [Section 5] The authors note that the dataset is small and highly similar, that random sampling performs surprisingly well in the high-dimensional space, and that both NM and RS tend to emulate the underlying class distribution. These observations are consistent with the small absolute differences in Table 1 and suggest that the reported advantages may partly reflect distributional properties of the collection rather than narrative structure. To contextualize the NM advantage, the paper should report a stronger baseline that is matched in information access, such as stratified random sampling by the expert's own thematic clusters, or a distribution-matching baseline. Without such a baseline, the comparison against uniform random sampling is too weak to establish that the method extracts meaningful storylines rather than merely reproducing the cluster distribution.
minor comments (6)
- [Author affiliation] The word 'Departament' in the affiliation for the first author should be corrected to 'Department'.
- [Sections 3.2 and 3.5] It is ambiguous whether the 'domain experts' in Section 3.2 and the 'domain expert' in Section 3.5 are the same individual; this should be stated explicitly because the independence of the ground truth and the supervision depends on it.
- [Section 4.1] The text says 'we do not explicitly evaluate coherence' and then immediately reports that 'average coherence values of the extracted NM are higher compared to the random sampling,' which is confusing; please clarify whether coherence is a reported metric or not.
- [Figure 3 caption] The caption refers to the 'unsupervised narrative maps algorithm,' but the paper's method is semi-supervised; the wording should be aligned.
- [Section 3.6] The description of the t-test analysis does not state whether the tests are paired or unpaired, nor whether the 20 trials are independent; adding this information would improve reproducibility.
- [Reference [23]] Reference [23] contains a typo: 'Darthmouth College' should be 'Dartmouth College.'
Circularity Check
NM-over-RS advantage is partly built from the same expert's labels, dates, and reference timelines, so the ground truth is not independent of the algorithm's inputs.
-
self definitional
[Sections 3.2, 3.5, and 4.1 (Table 1)]
"These expert-curated timelines serve as ground truth sequences against which we compare our algorithmically extracted narratives... Furthermore, the expert labeled the images in the baselines according to the stages of the expedition. These partial labels and dates provided by the expert are fed to the semi-supervised narrative maps extraction pipeline."
The evaluation target is not independent of the algorithm's inputs: the same expert who built the ground-truth timelines supplied the thematic labels and date seeds consumed by the label-spreading pipeline. Those seeds are used to construct the temporal and thematic structure of the coherence graph, so NM is scored against a timeline derived from the very expert judgments fed into it. The RS baseline receives no such seeds, so the Table 1 advantage (e.g., L=10 low-dim similarity 0.299 vs 0.116, p=2.5e-4) conflates algorithmic narrative extraction with injected expert chronology and theme knowledge. Section 5.1 concedes that 'dependency on expert-provided labels to induce a temporal ordering could introduce biases,' but no control or ablation isolates this effect.
-
other
[Section 4.2 (Qualitative evaluation: Expert-based Evaluation)]
"Finally, we complement our quantitative evaluation with an expert-based qualitative evaluation. To do this, we engage with the domain expert who created the baseline timelines to evaluate our extracted narratives. The expert assesses the coherence, relevance, and historical accuracy of one of our algorithmically generated storylines compared to their manually curated timelines."
The qualitative confirmation cited in the abstract, 'expert evaluation confirming the historical accuracy and coherence,' comes from the same expert whose labels and dates seeded the algorithm and whose manually curated timelines are the reference. The evaluation is unblinded, so the expert's positive assessment is an internal consistency check rather than independent validation: the output was built from that expert's prior chronological and thematic decisions and is then praised for matching those same decisions.
full rationale
The central quantitative claim is not a pure equation-level circularity: a real algorithm is run against a real random baseline, and both arms share fixed start/end images and matched sequence lengths. However, the ground truth is contaminated by the model's inputs. Section 3.5 explicitly states that the expert-curated baselines and the partial labels and dates fed to the pipeline come from the same expert, and Section 3.2 uses those date labels to induce temporal ordering for the directed graph. Random sampling is not given this expert supervision, so NM's advantage for timelines of length 10+ may simply reflect the value of having the expert's chronology and thematic structure encoded in the graph. Section 4.2 adds an unblinded qualitative assessment by the same expert, which cannot serve as independent confirmation. The paper's Section 5.1 limitation acknowledges the bias risk but does not test it. The self-citations to [13] and [6] are not the main issue: [13] is the external algorithm being adapted, and [6] is used only as supporting interpretation. Because the central effectiveness claim is partially circular but still contains independent visual-feature and optimization content, the score is 6 rather than higher.
Assumptions & free parameters
free parameters (5)
- K (map size) =
5, 10, 15, 20, 25, 30
- mincover =
not specified
- Coherence combination weights =
not specified
- Label-spreading parameters =
not specified
- UMAP parameters =
not specified
assumptions (4)
- domain assumption The 500-image Sacambaya subset contains an underlying narrative structure.
- domain assumption Pre-trained DETR features transfer to historical 1928 photographs.
- domain assumption The expert-provided chronology is correct.
- standard math The narrative maps optimization and label-spreading algorithms work as described in the cited references.
Cite this review
Pith. "Pith review of Semi-Supervised Image-Based Narrative Extraction: A Case Study with Historical Photographic Records." pith.science (2026). https://pith.science/paper/6PJ2HCJU
@misc{pith2026250109884,
author = {Pith},
title = {Pith review of: Semi-Supervised Image-Based Narrative Extraction: A Case Study with Historical Photographic Records},
year = {2026},
howpublished = {\url{https://pith.science/paper/6PJ2HCJU}},
note = {Machine review of arXiv:2501.09884}
}
read the original abstract
This paper presents a semi-supervised approach to extracting narratives from historical photographic records using an adaptation of the narrative maps algorithm. We extend the original unsupervised text-based method to work with image data, leveraging deep learning techniques for visual feature extraction and similarity computation. Our method is applied to the ROGER dataset, a collection of photographs from the 1928 Sacambaya Expedition in Bolivia captured by Robert Gerstmann. We compare our algorithmically extracted visual narratives with expert-curated timelines of varying lengths (5 to 30 images) to evaluate the effectiveness of our approach. In particular, we use the Dynamic Time Warping (DTW) algorithm to match the extracted narratives with the expert-curated baseline. In addition, we asked an expert on the topic to qualitatively evaluate a representative example of the resulting narratives. Our findings show that the narrative maps approach generally outperforms random sampling for longer timelines (10+ images, p < 0.05), with expert evaluation confirming the historical accuracy and coherence of the extracted narratives. This research contributes to the field of computational analysis of visual cultural heritage, offering new tools for historians, archivists, and digital humanities scholars to explore and understand large-scale image collections. The method's ability to generate meaningful narratives from visual data opens up new possibilities for the study and interpretation of historical events through photographic evidence.
Figures
Reference graph
Works this paper leans on
-
[1]
ACM Computing Surveys (CSUR)54(5), 1–38 (2021)
Alhussain, A.I., Azmi, A.M.: Automatic story generation: A survey of approaches. ACM Computing Surveys (CSUR)54(5), 1–38 (2021)
work page 2021
-
[2]
Alvarado, M., Matthews, M., Möller, C., Gerstmann, R.: Roberto Gerstmann: Fo- tografías, paisajes y territorios latinoamericanos. Pehuén (2009)
work page 2009
-
[3]
Digital Scholarship in the Humanities34(Supplement_1), i3–i16 (2019)
Arnold, T., Tilton, L.: Distant viewing: analyzing large visual corpora. Digital Scholarship in the Humanities34(Supplement_1), i3–i16 (2019)
work page 2019
-
[4]
In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) Computer Vision – ECCV 2020. pp. 213–229. Springer International Publishing, Cham (2020)
2020
-
[5]
IEEE Access 8, 144184– 144196 (2020)
Chumachenko, K., Männistö, A., Iosifidis, A., Raitoharju, J.: Machine learning based analysis of finnish world war ii photographers. IEEE Access 8, 144184– 144196 (2020)
work page 2020
-
[6]
Concha Macías, S., Keith Norambuena, B.: Evaluating the ability of computation- ally extracted narrative maps to encode media framing. In: Text2Story@ ECIR. pp. 17–28 (2024)
work page 2024
-
[7]
In: Proceedings of the 22nd Conference on Computational Natural Language Learning
Derby, S., Miller, P., Murphy, B., Devereux, B.: Using sparse semantic embeddings learned from multimodal text and image data to model human conceptual knowl- edge. In: Proceedings of the 22nd Conference on Computational Natural Language Learning. pp. 260–270 (2018)
work page 2018
-
[8]
History and Theory 48(4), 130–150 (2009)
Edwards, E.: Photography and the material performance of the past. History and Theory 48(4), 130–150 (2009)
work page 2009
Show all 30 references
-
[9]
University of Texas Press (1990)
Fleischman, S.: Tense and narrativity: From medieval performance to modern fic- tion. University of Texas Press (1990)
1990
-
[10]
arXiv preprint arXiv:2005.10107 (2020)
Ghalandari, D.G., Ifrim, G.: Examining the state-of-the-art in news timeline sum- marization. arXiv preprint arXiv:2005.10107 (2020)
2020 arXiv
-
[11]
In: Proceed- ings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies
Huang, T.H., Ferraro, F., Mostafazadeh, N., Misra, I., Agrawal, A., Devlin, J., Girshick, R., He, X., Kohli, P., Batra, D., et al.: Visual storytelling. In: Proceed- ings of the 2016 conference of the North American chapter of the association for computational linguistics: Hum...
2016
-
[12]
John Long, London (1934)
Jolly, S.D.: The Treasure Trail. John Long, London (1934)
1934
-
[13]
Proceedings of the ACM on Human- Computer Interaction 4(CSCW3), 1–33 (2021)
Keith Norambuena, B.F., Mitra, T.: Narrative maps: An algorithmic approach to represent and extract information narratives. Proceedings of the ACM on Human- Computer Interaction 4(CSCW3), 1–33 (2021)
2021
-
[14]
ACM Computing Surveys55(14s), 1–39 (2023)
Keith Norambuena, B.F., Mitra, T., North, C.: A survey on event-based news narrative extraction. ACM Computing Surveys55(14s), 1–39 (2023)
2023
-
[15]
ACM Computing Surveys (CSUR)52(5), 1–40 (2019)
Labatut, V., Bost, X.: Extraction and analysis of fictional character networks: A survey. ACM Computing Surveys (CSUR)52(5), 1–40 (2019)
2019
-
[16]
Forecasting5(3), 536–549 (2023)
Makridakis, S., Petropoulos, F., Kang, Y.: Large language models: Their success and impact. Forecasting5(3), 536–549 (2023)
2023
-
[17]
arXiv preprint arXiv:2204.02149 (2022)
Männistö, A., Seker, M., Iosifidis, A., Raitoharju, J.: Automatic image content extraction: Operationalizing machine learning in humanistic photographic studies of large visual archives. arXiv preprint arXiv:2204.02149 (2022)
2022 arXiv
-
[18]
In: Text2Story@ ECIR
Matus, M., Urrutia, D., Meneses, C., Keith, B.: ROGER: Extracting narratives using large language models from Robert Gerstmann’s historical photo archive of the Sacambaya expedition in 1928. In: Text2Story@ ECIR. pp. 53–64 (2024)
2024
-
[19]
arXiv preprint arXiv:1802.03426 (2018) Semi-Supervised Image-Based Narrative Extraction 15
McInnes, L., Healy, J., Melville, J.: Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018) Semi-Supervised Image-Based Narrative Extraction 15
2018 arXiv
-
[20]
Information retrieval for music and motion pp
Müller, M.: Dynamic time warping. Information retrieval for music and motion pp. 69–84 (2007)
2007
-
[21]
Robert Gerstmann y el tesoro de Sacambaya
Quisbert Condori, P.: Entre ingenieros y aventureros. Robert Gerstmann y el tesoro de Sacambaya. In: Imágenes de la Revolución Industrial: Robert Gerstmann en las Minas de Bolivia (1925 - 1936), chap. 3, pp. 47–64. Plural Editores, La Paz, Bolivia (2015)
2015
-
[22]
Ricoeur, P., Ricoeur, P.: Time and narrative, vol. 3. University of Chicago press (1984)
1984
-
[23]
Rauner Special Collections Library, Darthmouth College (1928)
Sanders, E.: The Story of the Jesuit Gold Mines in Bolivia and of the Treasure Hidden by the Sacambaya River. Rauner Special Collections Library, Darthmouth College (1928)
1928
-
[24]
Artificial Intelligence Review 56(8), 8393–8435 (2023)
Santana, B., Campos, R., Amorim, E., Jorge, A., Silvano, P., Nunes, S.: A survey on narrative extraction from textual data. Artificial Intelligence Review 56(8), 8393–8435 (2023)
2023
-
[25]
Archival science2, 1–19 (2002)
Schwartz, J.M., Cook, T.: Archives, records, and power: The making of modern memory. Archival science2, 1–19 (2002)
2002
-
[26]
In: Proceed- ings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
Shahaf, D., Guestrin, C.: Connecting the dots between news articles. In: Proceed- ings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 623–632 (2010)
2010
-
[27]
In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Wang, X., Chen, W., Wang, Y.F., Wang, W.Y.: No metrics are perfect: Adversarial reward learning for visual storytelling. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 899–909 (2018)
2018
-
[28]
Digital Scholarship in the Humanities35(1), 194–207 (2020)
Wevers, M., Smits, T.: The visual digital turn: Using neural networks to study historical images. Digital Scholarship in the Humanities35(1), 194–207 (2020)
2020
-
[29]
000 historical press photos? the challenges and opportunities of applying a scene detection algorithm to a digitised press photo collection
Wevers, M., Vriend, N., De Bruin, A.: What to do with 2.000. 000 historical press photos? the challenges and opportunities of applying a scene detection algorithm to a digitised press photo collection. TMG Journal for Media History25(1), 1 (2022)
2022
-
[30]
Advances in neural information processing systems16 (2003)
Zhou, D., Bousquet, O., Lal, T., Weston, J., Schölkopf, B.: Learning with local and global consistency. Advances in neural information processing systems16 (2003)
2003
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.