REVIEW 4 major objections 5 minor 25 references
Towards Cross-modal Retrieval in Chinese Cultural Heritage Documents: Dataset and Solution
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A new 5,726-pair Chinese cultural heritage dataset, plus a training-free local alignment method, lifts cross-modal retrieval to a mean recall of 47.9 on CulTi.
desk verdict A genuine new dataset for a neglected niche, saddled with a method claim that the reported numbers do not support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the local alignment module defined by Eqs. (5)-(6): patch weights $$w_k^i = \frac{\exp(\$\alpha$ \, S(u_k^i, u_I^i))}{\sum_{t=1}^n \exp(\$\alpha$ \, S(u_t^i, u_I^i))}$$ and final similarity $$S_{\mathrm{final}} = \sum_{k=1}^n w_k^i \, S(u_k^i, u_T^i).$$ Here $u_k^i$ is the embedding of a random crop, $u_I^i$ the full-image embedding, and $u_T^i$ the text embedding. The weight makes a crop count more if its embedding resembles the whole image, on the assumption that such crops contain the main subject the text describes. The dataset construction pipeline (OCR extraction, layout-specific cropping, and LLM-based text augmentation) supplies the training pairs needed to fine-tune the base model so that the local weights operate on domain-aligned embeddings.
What would settle it
Run LACLIP on CulTi with the patch weights in Eq. (5) replaced by uniform weights, keeping the same random crops and all other settings; if mean recall does not drop below 47.9, the similarity-based weighting is not doing the work, and the result reduces to fine-tuning plus random-crop averaging.
Extended reading notes
Core claim
CulTi is the paper's first contribution: 5,726 image-text pairs in Simplified Chinese, assembled from scanned pages of two authoritative publication series, with four data categories (pattern, original textile, cropped pattern, and mural) and a per-category 7:1:2 train/validation/test split. The second contribution is LACLIP, which starts from a Chinese-CLIP model with ViT-H/14 and RoBERTa-wwm-Large backbones fine-tuned on CulTi and adds a local alignment module at inference. For each image, several random crops are encoded, and each crop embedding is given a weight proportional to its cosine similarity with the full-image embedding; the final image-text similarity is the weighted sum of the crop-text similarities. The paper reports that this raises mean recall from 47.3 for the fine-tuned CN-CLIP ViT-H baseline to 47.9, mainly through text-to-image retrieval, while image-to-text recall slightly decreases.
Load-bearing premise
The central premise is that a random crop whose embedding resembles the full image is the region the text actually describes; if that mapping fails, the weighting adds noise rather than alignment.
Editorial extensions
If this is right
- CulTi gives the cross-modal retrieval community a domain-specific benchmark in Simplified Chinese where generic models perform poorly in zero-shot mode, with the best baseline mean recall at 22.6 before fine-tuning.
- LACLIP's inference-stage weighting can be attached to any fine-tuned Chinese-CLIP without retraining, so the reported improvement is portable across backbone choices.
- The reported gain is concentrated in text-to-image retrieval; image-to-text recall is slightly lower than the baseline, so the method is a targeted fix rather than a symmetric win.
- The per-category 7:1:2 split supports standard supervised evaluation, and the four data categories allow retrieval difficulty to be studied separately for patterns, original textiles, cropped patterns, and murals.
Reading between the lines
- If the similarity-weighting premise holds, the same random-crop weighting could be applied to other repetitive decorative arts, such as ceramics, brocades, or architectural ornament, where a text describes one motif among many repeats.
- The method's reliance on random crops suggests a natural extension: replacing random crops with detected motif regions or saliency proposals should either sharpen the gain or reveal that random cropping alone is sufficient.
- The 0.6-point margin over the best baseline is small relative to the 24.7-point gain from fine-tuning itself, so the practical contribution may be the dataset and fine-tuning procedure rather than the local alignment module.
- A controlled test varying the number of crops and the scaling factor $\alpha$ would show whether the weighting is robust or whether performance depends on these free parameters.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents CulTi, a Chinese cultural heritage multimodal dataset of 5,726 image-text pairs built from two book series on ancient silk patterns and Dunhuang murals. The construction pipeline is described in four steps: source selection, document digitization, OCR-based information extraction, and data summarization with ChatGPT-4o augmentation and manual proofreading; category and volume distributions are reported. The paper also proposes LACLIP, which fine-tunes Chinese-CLIP (ViT-H/14, RoBERTa-wwm-Large) on CulTi and, at inference, computes a weighted average of similarities between random image crops and the text, with crop weights derived from cosine similarity to the full image embedding (Eqs. 5-6). Experiments compare zero-shot and fine-tuned models on R@K and mean recall; LACLIP is reported at MR 47.9 versus 47.3 for CN-CLIP ViT-H.
Significance. If the dataset is as described, CulTi addresses a real gap: a domain-specific Chinese image-text retrieval benchmark for cultural heritage, with a documented construction process and a public code/data link. This is a potentially useful community resource. The method contribution is currently much weaker: the reported improvement over the strongest baseline is 0.6 mean-recall points, the image-to-text direction declines, no statistical support is provided, and the key design assumption is untested. The dataset part is credible and worth publishing; the method claim needs substantial additional evidence before it can be accepted. I therefore see this as a major-revision rather than a rejection, because the dataset contribution is fixable and the method issues are addressable with ablations and more rigorous evaluation.
major comments (4)
- [§5.2, Table 3] The central claim that LACLIP 'significantly outperforms existing models' rests on a 0.6-point mean-recall gain (47.9 vs 47.3) over CN-CLIP ViT-H, while the image-to-text direction declines at every recall level (R@1: 23.6 vs 24.5; R@5: 49.9 vs 50.4; R@10: 62.9 vs 64.0). No variance estimates, significance tests, or repeated runs with different seeds are reported, so the reported gain cannot be distinguished from run-to-run noise. The abstract, §5.2, and §6 should either provide statistical support or use more measured language.
- [§4.2, Eqs. (5)-(6)] The method's key premise, that patches similar to the full image contain the subjects the text describes, is not tested. Eq. (5) weights each patch by exp(α·S(u_i^k, u_i^I)), i.e., similarity to the global image embedding, not to the text. For a repetitive pattern where the text describes a single motif, a crop containing that motif can receive low weight while a dominant-background crop receives high weight. The paper provides no ablation replacing Eq. (5) with uniform weights, no cropping, or text-selected patches, and no sensitivity analysis for α; without this, the reported gain could come from random-crop ensembling or fine-tuning noise rather than the proposed local-alignment mechanism.
- [§4.2 and §5.1] The inference procedure is not reproducible as reported. The paper states that the image is 'randomly cropped to multiple local patches' but never specifies the number of patches n, the crop size, the scale/aspect-ratio range, or the random seed, even though n and the crop distribution directly determine Eq. (6). The fine-tuning description is also incomplete: no optimizer, learning-rate schedule, image resolution, or per-split data sizes are given. These details are needed before the LACLIP numbers in Table 3 can be independently reproduced or compared.
- [§5.2] The evaluation reports only aggregate MR and R@K over the full test set. Since CulTi contains four visually distinct categories (pattern, original textile, cropped pattern, mural) from two book series, and the local-alignment assumption is argued from the pattern case in Fig. 2, a per-category or per-source breakdown is necessary to show where the method helps and where it hurts. Without it, the claim that LACLIP 'handles fine-grained semantic associations' is not localized.
minor comments (5)
- [Throughout] There are several typographical errors, including 'T able 1' in §2.1 and 'Mciji Mountain' in Fig. 5 and the text; these should be corrected to 'Table 1' and 'Maiji Mountain'.
- [Figures 4, 7, 8] The qualitative examples contain small, low-resolution Chinese text that is difficult to verify from the PDF; higher-resolution crops or zoomed insets would help readers assess the failure cases.
- [§6] The conclusion contains grammatical errors (e.g., 'CulTi fulfill the gap in multimodal cultural area provides valuable resources') that should be corrected.
- [§3 and GitHub link] The paper should specify the license and copyright status of the images extracted from the two published book series, and whether the dataset is intended for non-commercial research use only.
- [§2.2] The Related Work mentions local-alignment methods LexVLA [11] and SEA [22] but does not compare LACLIP to them; a comparison or a clear explanation of why they are inapplicable would help position the method.
Circularity Check
No significant circularity: the LACLIP gain and the CulTi benchmark rest on independent empirical evaluation, not on self-citations or definitional identities.
full rationale
The paper's derivation chain is empirical rather than definitional. LACLIP's final similarity, Eq. (6), is a weighted sum of patch-text cosine similarities, with weights in Eq. (5) computed from patch-full-image similarity; neither equation defines the target result in terms of itself, and no fitted parameter is renamed as a prediction. The reported 47.9 versus 47.3 mean recall is a benchmark comparison on the newly introduced CulTi test split, and the fine-tuning and inference protocols are standard. The dataset construction is independent of the retrieval claim. The only self-citations ([16] and [19]) appear in related-work lists as examples of prior multimodal alignment methods and are not load-bearing for LACLIP's design or for the CulTi evaluation. Concerns about stochasticity, missing error bars, and the assumption behind Eq. (5) that full-image-similar patches contain the text-described motif are experimental-validity issues rather than circularity. Therefore, no circular step is established.
Assumptions & free parameters
free parameters (4)
- alpha =
1.02
- tau =
learnable, initialized to 1.0
- number of local patches n =
not reported
- random crop size and stride =
not reported
assumptions (3)
- domain assumption The chosen book series are authoritative and their printed descriptions correspond to the displayed images.
- domain assumption OCR extraction and ChatGPT-4o summarization preserve the meaning and alignment of the original descriptions.
- domain assumption Local patches whose embeddings are most similar to the full image contain the main subject described by the text.
Cite this review
Pith. "Pith review of Towards Cross-modal Retrieval in Chinese Cultural Heritage Documents: Dataset and Solution." pith.science (2026). https://pith.science/paper/DBMQVUT7
@misc{pith2026250510921,
author = {Pith},
title = {Pith review of: Towards Cross-modal Retrieval in Chinese Cultural Heritage Documents: Dataset and Solution},
year = {2026},
howpublished = {\url{https://pith.science/paper/DBMQVUT7}},
note = {Machine review of arXiv:2505.10921}
}
read the original abstract
China has a long and rich history, encompassing a vast cultural heritage that includes diverse multimodal information, such as silk patterns, Dunhuang murals, and their associated historical narratives. Cross-modal retrieval plays a pivotal role in understanding and interpreting Chinese cultural heritage by bridging visual and textual modalities to enable accurate text-to-image and image-to-text retrieval. However, despite the growing interest in multimodal research, there is a lack of specialized datasets dedicated to Chinese cultural heritage, limiting the development and evaluation of cross-modal learning models in this domain. To address this gap, we propose a multimodal dataset named CulTi, which contains 5,726 image-text pairs extracted from two series of professional documents, respectively related to ancient Chinese silk and Dunhuang murals. Compared to existing general-domain multimodal datasets, CulTi presents a challenge for cross-modal retrieval: the difficulty of local alignment between intricate decorative motifs and specialized textual descriptions. To address this challenge, we propose LACLIP, a training-free local alignment strategy built upon a fine-tuned Chinese-CLIP. LACLIP enhances the alignment of global textual descriptions with local visual regions by computing weighted similarity scores during inference. Experimental results on CulTi demonstrate that LACLIP significantly outperforms existing models in cross-modal retrieval, particularly in handling fine-grained semantic associations within Chinese cultural heritage.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
In: Proceedings of the IEEE international confer- ence on computer vision
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: Vqa: Visual question answering. In: Proceedings of the IEEE international confer- ence on computer vision. pp. 2425–2433 (2015)
2015
-
[2]
ACM Journal on Computing and Cultural Heritage17(1), 1–20 (2024)
Bahrami, M., Albadvi, A.: Deep learning for identifying iran’s cultural heritage buildings in need of conservation using image classification and grad-cam. ACM Journal on Computing and Cultural Heritage17(1), 1–20 (2024)
work page 2024
-
[3]
Chen, W., Yao, L., Jin, Q.: Rethinking benchmarks for cross-modal image-text retrieval. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 1241–1251 (2023)
work page 2023
-
[4]
arXiv preprint arXiv:2211.06679 (2022)
Chen, Z., Liu, G., Zhang, B.W., Ye, F., Yang, Q., Wu, L.: Altclip: Altering the language encoder in clip for extended language capabilities. arXiv preprint arXiv:2211.06679 (2022)
arXiv 2022
-
[5]
arXiv preprint arXiv:2010.11929 (2020)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
arXiv 2020
-
[6]
Advances in Neural Information Processing Systems35, 26418–26431 (2022)
Gu, J., Meng, X., Lu, G., Hou, L., Minzhe, N., Liang, X., Yao, L., Huang, R., Zhang, W., Jiang, X., et al.: Wukong: A 100 million large-scale chinese cross-modal pre-training benchmark. Advances in Neural Information Processing Systems35, 26418–26431 (2022)
work page 2022
-
[7]
arXiv preprint arXiv:2410.21276 (2024)
Hurst, A., Lerer, A., Goucher, A.P., Perelman, A., Ramesh, A., Clark, A., Os- trow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)
arXiv 2024
-
[8]
In: International conference on document analysis and recognition
Jahagirdar, S., Gangisetty, S., Mishra, A.: Look, read and ask: learning to ask questions by reading text in images. In: International conference on document analysis and recognition. pp. 335–349. Springer (2021)
work page 2021
Show all 25 references
-
[9]
In: Pro- ceedings of the 25th ACM international conference on Multimedia
Lan, W., Li, X., Dong, J.: Fluency-guided cross-lingual image captioning. In: Pro- ceedings of the 25th ACM international conference on Multimedia. pp. 1549–1557 (2017)
2017
-
[10]
IEEE Transactions on Multimedia 21(9), 2347–2360 (2019)
Li, X., Xu, C., Wang, X., Lan, W., Jia, Z., Yang, G., Xu, J.: Coco-cn for cross- lingual image tagging, captioning, and retrieval. IEEE Transactions on Multimedia 21(9), 2347–2360 (2019)
2019
-
[11]
Advances in Neural Information Processing Systems 37, 1141–1161 (2024)
Li, Y., Wang, Y., Fu, Y., Ru, D., Zhang, Z., He, T.: Unified lexical representa- tion for interpretable visual-language alignment. Advances in Neural Information Processing Systems 37, 1141–1161 (2024)
2024
-
[12]
International Circular of Graphic Education and Research pp
Liu, J., Krotova, T., Yezhova, O., Pashkevich, K.: Traditional elements of chinese culture in logo design. International Circular of Graphic Education and Research pp. 66–75 (2018)
2018
-
[13]
arXiv preprint arXiv:1907.11692 (2019)
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
2019 arXiv
-
[14]
ACM Computing Surveys (CSUR)54(6), 1–37 (2021)
Nguyen, T.T.H., Jatowt, A., Coustaty, M., Doucet, A.: Survey of post-ocr process- ing approaches. ACM Computing Surveys (CSUR)54(6), 1–37 (2021)
2021
-
[15]
Halmstad University Press (2018)
Nilson, T., Thorell, K.: Cultural heritage preservation: The past, the present and the future. Halmstad University Press (2018)
2018
-
[16]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Pan, Z., Wu, F., Zhang, B.: Fine-grained image-text matching by cross-modal hard aligning network. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 19275–19284 (2023) TowardsCross-modalRetrievalinChineseCulturalHeritageDocuments:Datase...
2023
-
[17]
Gondwana Research26(3-4), 1216–1221 (2014)
Qu, J., Cao, S., Li, G., Niu, Q., Feng, Q.: Conservation of natural and cultural heritage in dunhuang, china. Gondwana Research26(3-4), 1216–1221 (2014)
2014
-
[18]
In: International conference on machine learning
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021)
2021
-
[19]
Expert Systems with Applications255, 124811 (2024)
Wu, F., Wang, Q., Liu, X., Chen, Q., Zhao, Y., Zhang, B., Lim, E.G.: Discrimina- tive feature enhancement network for few-shot classification and beyond. Expert Systems with Applications255, 124811 (2024)
2024
-
[20]
In: Proceedings of the 31st ACM International Conference on Multimedia
Xie, C., Cai, H., Li, J., Kong, F., Wu, X., Song, J., Morimitsu, H., Yao, L., Wang, D., Zhang, X., et al.: Ccmb: A large-scale chinese cross-modal benchmark. In: Proceedings of the 31st ACM International Conference on Multimedia. pp. 4219– 4227 (2023)
2023
-
[21]
arXiv preprint arXiv:2211.01335 (2022)
Yang,A.,Pan,J.,Lin,J.,Men,R.,Zhang,Y.,Zhou,J.,Zhou,C.:Chineseclip:Con- trastive vision-language pretraining in chinese. arXiv preprint arXiv:2211.01335 (2022)
2022 arXiv
-
[22]
arXiv preprint arXiv:2408.11813 (2024)
Yin, Y., Zhao, Y., Zhang, Y., Lin, K., Wang, J., Tao, X., Wan, P., Zhang, D., Yin, B., Zhang, W.: Sea: Supervised embedding alignment for token-level visual-textual integration in mllms. arXiv preprint arXiv:2408.11813 (2024)
2024 arXiv
-
[23]
arXiv preprint arXiv:1907.04589 (2019)
Yu, T., Zhang, S., Lin, C., You, S., Wu, J., Zhang, J., Ding, X., An, H.: Dunhuang grottoes painting dataset and benchmark. arXiv preprint arXiv:1907.04589 (2019)
2019 arXiv
-
[24]
IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)
Zhang, J., Huang, J., Jin, S., Lu, S.: Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)
2024
-
[25]
Światowit11(52), 42–57 (2013)
Żuchowska, M.: From china to palmyra: The value of silk. Światowit11(52), 42–57 (2013)
2013
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.