Pith. sign in

REVIEW 5 major objections 6 minor 46 references

FuDoBa: Fusing Document and Knowledge Graph-based Representations with Bayesian Optimisation

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FuDoBa fuses LLM text embeddings with a global knowledge graph and a locally extracted graph, using Bayesian optimisation to choose per-modality projection sizes and weights, and claims the resulting ~100-dimensional representations match…

desk verdict FuDoBa is a solid, clearly described method for compressing LLM embeddings with KG fusion, but the evaluation protocol needs clarification to rule out transductive leakage before the parity claim can be trusted. read the letter →

arxiv 2507.06622 v1 pith:NBC32NZG submitted 2025-07-09 cs.CL

Boshko Koloski , Senja Pollak , Roberto Navigli , Blaž Škrlj This is my paper
classification cs.CL
keywords documentclassificationBayesianoptimisationrepresentationlearningknowledgegraphsearlyfusionmultimodalLLMembeddingsAutoML
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FuDoBa sets out to show that domain-specific structured knowledge can replace the dimensionality of LLM embeddings. Starting from a 1,536-dimensional proprietary text embedding, it adds two knowledge channels—a global graph built by linking entities to an external knowledge base, and a local graph built by extracting relation triplets from the dataset itself—projects each channel to a small dimension, and scales each by a learned importance weight. A Bayesian optimiser chooses these dimensions and weights by maximising the cross-validated macro-F1 of an off-the-shelf AutoML classifier. Across six sentiment and news datasets, the paper claims the fused ~100-dimensional representations perform on par with, or better than, the full LLM embeddings alone. If correct, this is a low-resource route to domain adaptation that also exposes which knowledge source matters for a given task.

What carries the argument

The load-bearing object is the hyperparameter vector $\theta = (l_{\mathrm{llm}}, l_{\mathrm{kg}}, l_{\mathrm{loc}}, \alpha_{\mathrm{llm}}, \alpha_{\mathrm{kg}}, \alpha_{\mathrm{loc}})$ governing how each modality is compressed and weighted before concatenation. Each modality embedding is Elastic-Net normalised, projected with truncated SVD to $l_m \le 64$, scaled by $\alpha_m$, concatenated, and normalised again; a Gaussian-process Bayesian optimiser with expected-improvement acquisition searches $\theta$ to maximise the 5-fold cross-validated macro-F1 of an AutoML classifier. This is what makes the fusion interpretable: the optimised $\alpha$ values are read as the importance of the LLM, global-knowledge, and local-knowledge channels for a given dataset.

What would settle it

Run the same FuDoBa search against the LLM-only baseline on a held-out benchmark of, say, 15-20 datasets with the same 50-trial, 5-minute AutoML budget. If the corrected significance test puts FuDoBa in a strictly worse rank group, or its average macro-F1 falls more than one point below the baseline on more than a third of the datasets, the equivalence claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that early fusion with per-modality low-dimensional projection preserves the accuracy of full-size LLM embeddings. Each modality is normalised, projected by truncated SVD to 16, 32, or 64 dimensions, scaled by a weight from 0 to 1, concatenated, and renormalised; Bayesian optimisation with a Gaussian-process surrogate and expected-improvement search maximises the 5-fold cross-validated macro-F1 of the AutoML classifier. Compared against the 1,536-dimensional LLM-only representation on six datasets, FuDoBa's best configurations land at 80-144 final dimensions and the paper reports them as statistically indistinguishable or better (Friedman test with Nemenyi correction), with an average macro-F1 of 83.34 versus 82.58 for the LLM baseline. The paper also reports that the full high-dimensional LLM+KG+LocKG concatenation scores highest in absolute terms, so the method's point is not that fusion beats everything, but that the same signal can be compressed to roughly one-tenth the size without losing the comparison to LLM-only.

Load-bearing premise

The 'on par' claim rests on a significance test computed from only six datasets, which cannot reliably detect moderate performance gaps; the paper's own table shows FuDoBa numerically below the LLM baseline on three of the six.

Editorial extensions

If this is right

  • FuDoBa's fused representations end at 80-144 dimensions, roughly one-tenth to one-sixteenth of the 1,536-dimension LLM baseline, while remaining within about one F1 point of it on five of six datasets and beating it by 5.6 points on XGENRE.
  • The optimal per-modality weights vary by dataset; LLM embeddings receive the highest average importance ($\alpha \approx 0.78$), but no modality is consistently dominant at $N=6$, so task-specific optimisation is needed rather than a fixed weighting.
  • A locally extracted knowledge graph covers documents the global graph misses: in the short-text hate-speech corpus the fraction of documents with no mapped entities falls from 22% to 17%, and the average number of mapped entities per document rises from 2.83 to 4.99.
  • The relation-extraction model can be swapped from a large proprietary system to smaller open models without a statistically significant change in macro-F1 (within about 1.25 points, $p=0.9931$), so the local-knowledge benefit is not tied to one extractor.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the learned per-modality weights might transfer across related datasets or tasks; a cheap test would be taking $\alpha$ and $l$ from one dataset and applying them to a held-out sibling dataset, which the paper does not report.
  • Editorial inference: because each modality is projected separately, the same optimisation should work for vision-language fusion with frozen encoders; testing on an image-plus-text benchmark would extend the claim beyond the six textual datasets.
  • Editorial inference: the cost comparison in the appendix implies embeddings are much cheaper than LLM prompting; if the on-par result holds, a practical pattern is a stored low-dimensional domain-adapted index that avoids repeated API calls, with the caveat that overlapping training data may inflate LLM baselines.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes FuDoBa, a Bayesian-optimisation-based early fusion method that combines 1536-dimensional LLM embeddings with two knowledge-graph modalities: a global WikiData5m KG branch (via Babelfy entity linking and RotatE embeddings) and a novel local KG branch built by relation extraction from the target corpus. Each modality is normalized, projected with truncated SVD to a low dimension (16/32/64), and scaled by an importance weight alpha before concatenation and a final ElasticNet normalization. Bayesian optimization (50 trials, 5-fold CV macro-F1 with AutoGluon) selects the per-modality dimensions and weights. The method is evaluated on six English datasets (sentiment and news genre), with the central claim that the resulting low-dimensional representations perform on par with, or better than, the proprietary LLM-only baseline while drastically reducing dimensionality.

Significance. If the empirical claim holds, FuDoBa is a practically useful contribution: it compresses 1536-dim LLM embeddings to roughly 100 dimensions while preserving classification performance, it runs on commodity hardware, and its per-modality weights are arguably interpretable. The paper also includes useful ablations: comparison of three relation extractors (showing open-source alternatives are competitive), a concat-then-project baseline, a cost analysis of prompting versus embeddings, and per-dataset Bayesian-search diagnostics. The claim is falsifiable and clearly stated, and the Appendix contains a candid limitations section. However, the strength of the evidence is limited by the small number of datasets, single reported runs without confidence intervals, an evaluation protocol that may be transductive, and the absence of released code and data.

major comments (5)
  1. [Sec. 3.3, Algorithm 1] The evaluation protocol is potentially transductive. In Step 1 (lines 1-6), the ElasticNet normalization N(E_m) and the truncated SVD projection are computed from the full modality matrices E_m before the CV split in Step 2, and Step 3 refits the classifier on the full dataset. If the reported test scores are obtained by applying Algorithm 1 to the combined train+test embeddings, test documents contribute to the SVD directions and normalization statistics; within the CV loop, validation folds also contribute. The LLM-only baseline uses fixed raw embeddings and receives no such transductive information, so the reported average advantage (approximately 83.5 vs 82.6 macro-F1) could be an artifact of leakage rather than of the fusion itself. The paper must state and implement a leak-free protocol: fit normalization and SVD on the training split only, apply the fitted transforms to validation and test splits, and, for the LocKG branch (Sec. 3.1), confirm that relation extraction and RotatE training use training documents only.
  2. [Sec. 5.1, Table 3] The 'on par or better' claim rests on a Friedman test with Nemenyi post-hoc correction over only six datasets, and the paper itself reports FuDoBa numerically losing on three of them (HateSpeech 74.61 vs 75.68, MLDoc 96.19 vs 96.74, Music 92.70 vs 92.90). With N=6, the test has very low power, so a non-significant result cannot distinguish parity from a moderate disadvantage. Each entry in Table 3 appears to be a single AutoGluon run with no variance or confidence interval. The authors should either evaluate on more datasets, run repeated trials with different seeds and report confidence intervals, or pre-specify an equivalence margin (such as the 1% band in Figure 3) and test against it.
  3. [Sec. 7 and Table 3] The conclusion that FuDoBa is 'comparable to or better than high-dimensional baselines (including LLM-only and simple concatenation approaches)' is not supported by the reported numbers. The LLM+KG+LocKG concatenation has a higher average macro-F1 (approximately 83.96 vs 83.51) and beats FuDoBa on HateSpeech, MLDoc, and XGENRE. The abstract's parity claim is specifically against LLM-only embeddings, and that is the defensible comparison; the broader claim about simple concatenation should either be removed or subjected to the same statistical analysis.
  4. [Sec. 5.2, Figure 5] The alpha and l values are the result of maximizing the cross-validated F1 objective f(theta), so the observation that alpha_LLM = 0.78 on average is a property of the optimized configuration, not independent evidence that LLM embeddings are inherently the 'most important' modality. The paper's own caveat about pretraining overlap and the non-significant paired tests (N=6) reinforce this concern. If interpretable early-fusion weights are a contribution, the interpretation should be validated against held-out tasks, equal-weight baselines, or an ablation in which the optimized weights are replaced by uniform weights.
  5. [Availability and reproducibility] The availability statement says code, embeddings, and scripts 'will be made available upon acceptance,' and the pipeline depends on proprietary OpenAI embeddings and gpt-4o-mini for relation extraction. Although Appendix C shows that open-source relation extractors achieve similar results, the exact reported numbers cannot be reproduced from the manuscript alone. The revision should release the code, precomputed embeddings, and a clear description of train/test split handling (including how SVD and normalization are fitted), or provide a Docker/API-free version of the pipeline.
minor comments (6)
  1. [Sec. 5.1] There is a typo 'statistifically' in the paragraph discussing the Friedman/Nemenyi test; also, 'Mattern Kernel' in Sec. 3.4 should be 'Matérn kernel'.
  2. [Sec. 3.3, Algorithm 1] The notation P_m = SVD_pm(Normalize(E_m)) is ambiguous: the preliminaries define P = A V_p, but the algorithm does not specify how the projection matrix V_p is estimated or reused. Please define SVD_pm and clarify that the projection must be fitted on the training portion only.
  3. [Table 3 caption] The caption says 'Bolded entries represent best scores per metric,' but the bold formatting is not visible in the text; also, the use of 'x' for exclusion/not-applicable should be stated in the caption.
  4. [Appendix B] The full CPU and memory listing is not useful to readers; a one-line hardware description would suffice.
  5. [References [5] and [17]] Both references are titled LLM2Vec but with different titles; please verify that these are distinct works and cite them consistently.
  6. [Availability] Phrase the availability statement as a concrete commitment with a repository link or supplement, rather than a future promise made upon acceptance.

Circularity Check

1 steps flagged · score 3.0 of 10

The main classification comparison is self-contained; the only by-construction element is the interpretation of optimized alpha weights as evidence of modality importance.

  1. self definitional [Section 5.2, 'Analysis of the Impact of Modality Importance and Dimensionality' (Figure 5 discussion)]
    "In Figure 5, the left subplot shows that modality importance, α, varies notably across the six benchmark datasets. On average, LLM embeddings have the highest importance (αLLM = 0.78), compared to Local KGs (αLocKG = 0.50) and Global KGs (αKG = 0.43), as expected."

    The α values are the hyperparameters optimized by the paper's own procedure: Section 3.4 defines θ* = arg max f(θ), and Algorithm 1 defines f(θ) as the 5-fold cross-validation macro-F1 of AutoGluon on the fused representation. The reported 'importance' is therefore the optimized parameter value, so the observation that αLLM is largest states the outcome of the fit rather than an independent property of the data. Interpreting this as evidence that LLM embeddings are the most important modality for the task is a conclusion that holds by construction and cannot independently validate the fusion or explain performance beyond the optimization objective.

full rationale

The central claim—that FuDoBa's low-dimensional fused representations perform on par with or better than 1536-dimensional LLM-only embeddings—is supported by a direct comparison on six held-out test splits and a Friedman/Nemenyi analysis; no equation defines the test F1 in terms of the optimized θ, so the main evaluation is not circular. The method openly optimizes α and l on cross-validated F1, which is supervised model selection rather than a disguised prediction. The reliance on BabelFusion [9], authored by the same group, for the KG embeddings and the low-dimensional projection premise is self-citation, but it is not load-bearing in a circular sense because the current paper re-evaluates the resulting representations against the LLM baseline on six datasets. The only notable by-construction element is the Section 5.2 reading of the fitted α values as evidence of modality importance; that is a partial interpretive circularity, not a flaw in the main parity result. The transductive-leakage concern about SVD and normalization being computed before the CV split is a protocol validity risk rather than a circularity, so it is not scored here.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

FuDoBa introduces no new theoretical entities. Its reliance is on fitted per-modality weights and projection dimensions (searched per dataset), hand-chosen normalisation weights, and a set of domain assumptions about the usefulness of SVD, RotatE embeddings, relation extraction, AutoGluon, and the representativeness of six datasets.

free parameters (4)
  • alpha_llm, alpha_kg, alpha_loc = e.g., Books: 1.0, 0.8, 0.1; XGENRE: 0.8, 0.8, 0.8
    Per-modality importance weights, fitted per dataset by Bayesian optimisation to maximise 5-fold CV macro-F1 (Section 3.3-3.4). These are the core free parameters of the method.
  • l_llm, l_kg, l_loc = e.g., Books: 64, 32, 16; final dims 96-144
    Per-modality SVD projection dimensions, searched over {16,32,64} by Bayesian optimisation (Section 3.3).
  • Elastic Net weights w1, w2 = 0.5, 0.5
    Chosen by hand for pre/post fusion normalisation (Section 3.2); not tuned.
  • BO trial count and AutoGluon time limit = 50 trials; 5 minutes
    Search budget set by the authors (Section 4.2); affects the quality of the found optimum. The paper notes in the Discussion that a larger budget 'would generate more promising results'.
assumptions (5)
  • standard math Truncated SVD preserves the most useful information in the first p principal directions
    Invoked in Section 3.2 and Algorithm 1 to justify projecting each modality to 16-64 dimensions.
  • domain assumption RotatE embeddings of WikiData5m entities capture useful semantics for document classification
    Used as the global KG modality, inherited from BabelFusion [9] (Section 3.1).
  • domain assumption Relation extraction from gpt-4o-mini yields useful, denoised knowledge triplets for short and long texts
    Defines the LocKG modality (Section 3.1). Appendix C shows alternative extractors give similar downstream F1, so this axiom is partially tested.
  • domain assumption AutoGluon with the good quality preset and 5-minute limit is a strong enough learner to rank representations fairly across dimensions
    Used for every objective evaluation and final test score (Section 4.2). High-dimensional features might be handicapped by the fixed time budget.
  • domain assumption Six datasets in two domains are representative enough for a statistical claim about all such tasks
    The Friedman/Nemenyi comparison in Section 5.1 covers N=6, which is weak for detecting meaningful differences.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FuDoBa: Fusing Document and Knowledge Graph-based Representations with Bayesian Optimisation." pith.science (2026). https://pith.science/paper/NBC32NZG

@misc{pith2026250706622,
  author       = {Pith},
  title        = {Pith review of: FuDoBa: Fusing Document and Knowledge Graph-based Representations with Bayesian Optimisation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NBC32NZG}},
  note         = {Machine review of arXiv:2507.06622}
}
read the original abstract

Building on the success of Large Language Models (LLMs), LLM-based representations have dominated the document representation landscape, achieving great performance on the document embedding benchmarks. However, the high-dimensional, computationally expensive embeddings from LLMs tend to be either too generic or inefficient for domain-specific applications. To address these limitations, we introduce FuDoBa a Bayesian optimisation-based method that integrates LLM-based embeddings with domain-specific structured knowledge, sourced both locally and from external repositories like WikiData. This fusion produces low-dimensional, task-relevant representations while reducing training complexity and yielding interpretable early-fusion weights for enhanced classification performance. We demonstrate the effectiveness of our approach on six datasets in two domains, showing that when paired with robust AutoML-based classifiers, our proposed representation learning approach performs on par with, or surpasses, those produced solely by the proprietary LLM-based embedding baselines.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 31 canonical work pages

  1. [1]

    In: Vlachos, A., Augenstein, I

    Muennighoff, N., Tazi, N., Magne, L., Reimers, N.: MTEB: Massive text embed- ding benchmark. In: Vlachos, A., Augenstein, I. (eds.) Proceedings of the 17th Conference of the European Chapter of the Association for Computa- tional Linguistics, pp. 2014–2037. Association for Computational Linguistics, Dubrovnik, Croatia (2023). https://doi.org/10.18653/v1/2...

    doi:10.18653/v1/2023.eacl-main.148 2023

  2. [2]

    In: Burstein, J., Doran, C., Solorio, T

    Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers...

    work page 2019

  3. [3]

    Grattafiori, A., al.: The Llama 3 Herd of Models (2024)

    work page 2024

  4. [4]

    In: Inui, K., Jiang, J., Ng, V., Wan, X

    Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982–3992. Association for Computat...

    doi:10.18653/v1/d19-1410 2019

  5. [5]

    In: First Conference on Language Modeling (2024)

    BehnamGhader, P., Adlakha, V., Mosbach, M., Bahdanau, D., Chapados, N., Reddy, S.: LLM2Vec: Large language models are secretly powerful text encoders. In: First Conference on Language Modeling (2024)

    work page 2024

  6. [6]

    IEEE Internet of Things Journal9(12), 9205–9213 (2022) https://doi.org/10.1109/JIOT.2021.3093065

    Chen, Q., Wang, W., Huang, K., Coenen, F.: Zero-shot text classification via knowledge graph embedding for social media data. IEEE Internet of Things Journal9(12), 9205–9213 (2022) https://doi.org/10.1109/JIOT.2021.3093065

    work page arXiv 2022

  7. [7]

    Neurocomputing496, 208–226 (2022) https://doi.org/10.1016/ j.neucom.2022.01.096

    Koloski, B., Stepiˇ snik Perdih, T., Robnik- ˇSikonja, M., Pollak, S., ˇSkrlj, B.: Knowledge graph informed fake news classification via heterogeneous representa- tion ensembles. Neurocomputing496, 208–226 (2022) https://doi.org/10.1016/ j.neucom.2022.01.096

    work page 2022

  8. [8]

    Machine Learning (2021) https://doi.org/10.1007/s10994-021-05968-x

    ˇSkrlj, B., Martinc, M., Lavraˇ c, N., Pollak, S.: autobot: evolving neuro-symbolic representations for explainable low resource text classification. Machine Learning (2021) https://doi.org/10.1007/s10994-021-05968-x

    work page doi:10.1007/s10994-021-05968-x 2021

Show all 46 references
  1. [9]

    In: Discovery Science: 27th International Conference, DS 2024, Pisa, Italy, October 14–16, 2024, Proceedings, Part I, pp

    Koloski, B., Pollak, S., Navigli, R., ˇSkrlj, B.: Automl-guided fusion of entity and llm-based representations fordocument classification. In: Discovery Science: 27th International Conference, DS 2024, Pisa, Italy, October 14–16, 2024, Proceedings, Part I, pp. 101–115. Springe...

    2025

  2. [10]

    In: Hajiˇ c, J., Carberry, S., Clark, S., Nivre, J

    Navigli, R., Ponzetto, S.P.: BabelNet: Building a very large multilingual semantic network. In: Hajiˇ c, J., Carberry, S., Clark, S., Nivre, J. (eds.) Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 216–225. Association for Computat...

    2010

  3. [12]

    Transactions of the Association for Computational Linguistics2, 231–244 (2014) https://doi.org/10.1162/tacl a 00179

    Moro, A., Raganato, A., Navigli, R.: Entity linking meets word sense disam- biguation: a unified approach. Transactions of the Association for Computational Linguistics2, 231–244 (2014) https://doi.org/10.1162/tacl a 00179

    2014 doi

  4. [13]

    ArXiv preprintabs/2003.06505(2020)

    Erickson, N., Mueller, J., Shirkov, A., Zhang, H., Larroy, P., Li, M., Smola, A.: Autogluon-tabular: Robust and accurate automl for structured data. ArXiv preprintabs/2003.06505(2020)

    2020 arXiv

  5. [14]

    In: Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014

    Le, Q.V., Mikolov, T.: Distributed representations of sentences and doc- uments. In: Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014. JMLR Workshop and Conference Proceedings, vol. 32, pp. 1188–1196. JMLR.org, ??...

    2014

  6. [15]

    In: Moens, M.-F., Huang, X., Specia, L., Yih, S.W.-t

    Gao, T., Yao, X., Chen, D.: SimCSE: Simple contrastive learning of sentence embeddings. In: Moens, M.-F., Huang, X., Specia, L., Yih, S.W.-t. (eds.) Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894–6910. Association for Computa...

    2021 doi

  7. [16]

    In: Ku, L.-W., Martins, A., Srikumar, V

    Li, X., Li, J.: AoE: Angle-optimized embeddings for semantic textual sim- ilarity. In: Ku, L.-W., Martins, A., Srikumar, V. (eds.) Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pp. 1825–1839. Association for ...

    2024 doi

  8. [17]

    ArXiv preprintabs/2405.13413(2024)

    Aghajanyan, A., BehnamGhader, P., Lewis, M., Yih, W.-t., Zettlemoyer, L.: LLM2Vec: Simple unsupervised approach to transform decoder-only LLMs into strong text encoders. ArXiv preprintabs/2405.13413(2024)

    2024 arXiv

  9. [18]

    ArXiv preprintabs/2501.08648(2025)

    Khosla, S., Tiwari, A., Kafle, K., Jenni, S., Zhao, H., Collomosse, J., Shi, J.: MAGNET: Augmenting generative decoders with representation learning and infilling capabilities. ArXiv preprintabs/2501.08648(2025)

    2025 arXiv

  10. [19]

    In: Ku, L.-W., Martins, A., Sriku- mar, V

    Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., Wei, F.: Improving text embeddings with large language models. In: Ku, L.-W., Martins, A., Sriku- mar, V. (eds.) Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2024 doi

  11. [20]

    Cocchi, F., Moratelli, N., Cornia, M., Baraldi, L., Cucchiara, R.: Augment- ing Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering (2025)

    2025

  12. [21]

    Computer Speech & Language65, 101104 (2021) https://doi.org/10.1016/j.csl.2020.101104

    ˇSkrlj, B., Martinc, M., Kralj, J., Lavraˇ c, N., Pollak, S.: tax2vec: Constructing interpretable features from taxonomies for short text classification. Computer Speech & Language65, 101104 (2021) https://doi.org/10.1016/j.csl.2020.101104

    2021

  13. [22]

    ArXiv preprintabs/1909.08402(2019)

    Ostendorff, M., Bourgonje, P., Berger, M., Moreno-Schneider, J., Rehm, G., Gipp, B.: Enriching bert with knowledge graph embeddings for document classification. ArXiv preprintabs/1909.08402(2019)

    2019 arXiv

  14. [23]

    In: Proceedings of the 5th ACM International Conference on AI in Finance

    Sarmah, B., Mehta, D., Hall, B., Rao, R., Patel, S., Pasquali, S.: Hybridrag: Inte- grating knowledge graphs and vector retrieval augmented generation for efficient information extraction. In: Proceedings of the 5th ACM International Conference on AI in Finance. ICAIF ’24, pp....

    2024

  15. [24]

    In: Inui, K., Jiang, J., Ng, V., Wan, X

    Fan, A., Gardent, C., Braud, C., Bordes, A.: Using local knowledge graph con- struction to scale Seq2Seq models to multi-document inputs. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing an...

    2019 doi

  16. [25]

    In: Proceedings of the 8th Interna- tional Conference on Database Theory

    Aggarwal, C.C., Hinneburg, A., Keim, D.A.: On the surprising behavior of distance metrics in high dimensional spaces. In: Proceedings of the 8th Interna- tional Conference on Database Theory. ICDT ’01, pp. 420–434. Springer, Berlin, Heidelberg (2001)

    2001

  17. [26]

    In: 2021 IEEE International Conference on Data Mining (ICDM), pp

    ˇSkrlj, B., Petkovi´ c, M.: Compressibility of distributed document representations. In: 2021 IEEE International Conference on Data Mining (ICDM), pp. 1330–1335 (2021). https://doi.org/10.1109/ICDM51629.2021.00166

    2021

  18. [27]

    In: 2012 UKSim 14th International Conference on Computer Modelling and Simulation, pp

    Koduri, S.: Multisensor data fusion with singular value decomposition. In: 2012 UKSim 14th International Conference on Computer Modelling and Simulation, pp. 422–426 (2012). https://doi.org/10.1109/UKSim.2012.65

    2012 doi

  19. [28]

    Bioinformatics36(1), 250–256 (2020)

    Le, T.T., Fu, W., Moore, J.H.: Scaling tree-based automated machine learning to biomedical big data with a feature set selector. Bioinformatics36(1), 250–256 (2020)

    2020

  20. [29]

    Proceedings of the IEEE103(9), 1449–1477 (2015) 21

    Lahat, D., Adali, T., Jutten, C.: Multimodal data fusion: an overview of methods, challenges, and prospects. Proceedings of the IEEE103(9), 1449–1477 (2015) 21

    2015

  21. [30]

    Computers, Materials and Continua80(1), 1–35 (2024) https://doi.org/10.32604/cmc.2024

    Jiao, T., Guo, C., Feng, X., Chen, Y., Song, J.: A comprehensive survey on deep learning multi-modal fusion: Methods, technologies and applications. Computers, Materials and Continua80(1), 1–35 (2024) https://doi.org/10.32604/cmc.2024. 053204

    2024 doi

  22. [31]

    In: Singh, S.P., Markovitch, S

    Speer, R., Chin, J., Havasi, C.: Conceptnet 5.5: An open multilingual graph of general knowledge. In: Singh, S.P., Markovitch, S. (eds.) Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 4444–445...

    2017

  23. [32]

    In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019

    Sun, Z., Deng, Z., Nie, J., Tang, J.: Rotate: Knowledge graph embedding by relational rotation in complex space. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, ??? (2019).https://openreview.net/foru...

    2019

  24. [33]

    IEEE Transactions on Automatic Control25(2), 164–176 (1980) https://doi.org/10.1109/TAC.1980.1102314

    Klema, V., Laub, A.: The singular value decomposition: Its computation and some applications. IEEE Transactions on Automatic Control25(2), 164–176 (1980) https://doi.org/10.1109/TAC.1980.1102314

    1980

  25. [34]

    In: Bartlett, P.L., Pereira, F.C.N., Burges, C.J.C., Bottou, L., Weinberger, K.Q

    Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimiza- tion of machine learning algorithms. In: Bartlett, P.L., Pereira, F.C.N., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25: 26th Annual Conference on Ne...

    2012

  26. [35]

    The MIT Press, ??? (2006)

    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press, ??? (2006)

    2006

  27. [36]

    The MIT Press, ??? (2005)

    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press, ??? (2005). https://doi.org/10.7551/mitpress/3206.001.0001 . https://doi.org/10.7551/mitpress/3206.001.0001

    2005 doi

  28. [37]

    In: Webber, B., Cohn, T., He, Y., Liu, Y

    Ranasinghe, T., Zampieri, M.: Multilingual offensive language identification with cross-lingual embeddings. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds.) Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 5838–5844. Associa...

    2020 doi

  29. [38]

    Schwenk, H., Li, X.: A corpus for multilingual document classification in eight languages. In: Calzolari, N., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Hasida, K., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., 22 Odijk, J., Piperidis, S., Tokunaga, T. (e...

    2018

  30. [39]

    In: Calzolari, N., B´ echet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk, J., Piperidis, S

    Kuzman, T., Rupnik, P., Ljubeˇ si´ c, N.: The GINCO training dataset for web genre identification of documents out in the wild. In: Calzolari, N., B´ echet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk...

    2022

  31. [40]

    In: Advances in Neural Information Pro- cessing Systems 37 (NeurIPS 2024) (2024)

    Tennenholtz, G., Chow, Y., Hsu, C.-W., Shani, L., Liang, Y., Boutilier, C.: Embedding-aligned language models. In: Advances in Neural Information Pro- cessing Systems 37 (NeurIPS 2024) (2024)

    2024

  32. [41]

    Journal of Machine learning research7(Jan), 1–30 (2006)

    Demˇ sar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research7(Jan), 1–30 (2006)

    2006

  33. [42]

    In: Moens, M.-F., Huang, X., Specia, L., Yih, S.W.- t

    Huguet Cabot, P.-L., Navigli, R.: REBEL: Relation extraction by end-to-end language generation. In: Moens, M.-F., Huang, X., Specia, L., Yih, S.W.- t. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2370–2381. Association for Computational Lin...

    2021 doi

  34. [43]

    entity1

    Barba, E., Orlando, R., Cabot, P.-L.H., Navigli, R.: ReLiK: Retrieve, Read and LinK: Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget (2024). https://openreview.net/forum?id=b0IRscfEOb Appendix A Limitations Despite these promising results, FuDoBa...

    2024

  35. [44]

    - N o r m a l i z e en tit ie s by using lower - case and s ing ul ar forms when a p p l i c a b l e to avoid near d u p l i c a t e s

    S t a n d a r d i z e d and Unique Ent it ies : - Extract only clear , general c on cep ts exactly as they appear in the text . - N o r m a l i z e en tit ie s by using lower - case and s ing ul ar forms when a p p l i c a b l e to avoid near d u p l i c a t e s

  36. [45]

    Concise Ent it ies : - Ensure each entity r e p r e s e n t s a single , clear concept ; avoid c o m b i n i n g mu lt ipl e c onc ep ts into one entity

  37. [46]

    Robust Mapping : - Do not derive or r e i n t e r p r e t entities - use the exact wording from the text so that each entity can be d ire ct ly traced back

  38. [47]

    ""{ d oc ume nt }

    S i m p l i c i t y in R e l a t i o n s h i p s : - Use the allowed r e l a t i o n s to denote simple and clear i n t e r a c t i o n s between en ti tie s . Do cu men t : """{ d oc ume nt }""" Listing 2: Prompt text for knowledge extraction and graph construction Appendix C...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.