{"id":"3e25bf99-90f7-4a63-8738-4622545589c6","arxiv_id":"2412.03592","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A method that represents each word by the concatenated autoencoder latent codes of images of its dictionary definition terms, evaluated on word similarity, categorization, and outlier detection.","lead":"This paper builds word vectors by auto-encoding images of dictionary definition terms, without using text contexts. The approach trains faster than context-based embeddings, but its similarity scores are mostly lower than standard baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'comparable performance' claim is unestablished: Table 1 compares against borrowed baseline scores with no confidence intervals, and the image method is below baselines on most word-similarity benchmarks.","rationale":"The reader's stated weakest assumption is the image-latent mechanism: that a reconstruction-only autoencoder on noisy web images preserves semantic information. My concern is one step earlier in the argument. The paper's headline is an empirical comparative claim, and the comparison is not controlled. This is load-bearing because if the comparison were valid, the method is still clearly worse on several benchmarks, and if invalid, the claim has no support. The reader's rationale does also mention borrowed baselines and missing statistical tests, so there is partial overlap, but the primary emphasis differs. I do not see an internal logical contradiction; the issue is missing evidence and non-matched baselines. Therefore the REJECT verdict remains appropriate, but the strongest reason is the uncontrolled empirical support for the central claim.","tokens_in":7922,"tokens_out":5120,"duration_ms":50277,"concrete_test":"Re-run the comparison in a controlled setting: use the released evaluation code from Wang et al. (2019) and evaluate original SGNS, CBOW, GloVe, FastText, and Dict2vec embeddings on the same vocabulary-filtered benchmark instances and cosine-similarity/Spearman protocol used for the image method. Compute bootstrap 95% confidence intervals (at least 1000 resamples) for each dataset. If the image-method score falls outside the baseline confidence interval on most datasets, the 'comparable' claim is refuted; if the intervals overlap broadly, the claim can be provisionally accepted, and the next step would be to probe whether the latent codes actually separate semantic categories.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 5 claim that the proposed image-based vectors 'perform comparably' to context-based models and train faster. The only quantitative support for this claim is Table 1, where all baseline rows are copied from Wang et al. (2019), a study with its own preprocessing, vocabulary, and evaluation protocol. No error bars, significance tests, or matched re-runs are provided, and the paper does not define what 'comparable' means. Under the usual reading (overlapping confidence intervals or comparable point estimates), Table 1 fails: the image method scores below SGNS on 6 of 8 similarity datasets (e.g., SimLex-999 0.23 vs 0.39, Verb-143 0.30 vs 0.45, RW 0.36 vs 0.46), and is far below the best baseline on MC-30 (0.70 vs 0.85 for ngram2vec) and RG-65 (0.69 vs 0.85 for Dict2vec). If these differences are real, the central claim is false; if they are not, the paper has not shown it, because the comparison is not controlled. The training-time advantage is also unsupported: no wall-clock training time for the baselines is reported, so 'much less training time' cannot be evaluated. The load-bearing evidence for the headline is therefore absent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for learning word vectors without using text context. For each word in a BERT-derived vocabulary, it retrieves images of the word and of the words in its dictionary definition, forming an ordered image-set of up to 100 images per word (5 images per term, up to 20 terms). A convolutional autoencoder is trained on 577,290 images to reconstruct 32x32 inputs through a 32-dimensional bottleneck; the 32-d latents for the image-set are concatenated into a 3200-d word vector. The method is evaluated on word similarity (eight datasets), outlier detection (two datasets), and concept categorization (four datasets). The paper claims the resulting vectors perform comparably to context-based embeddings while training in about ten hours.","tokens_in":8163,"tokens_out":6991,"duration_ms":61961,"significance":"If the central claims were established, the work would be a distinctive contribution: it would show that visual depictions of definition terms can support context-free word vectors, and the released image dataset (577,290 images for 115,458 terms) could be reused by the community. The paper is also honest about the dependence of vector quality on image selection. However, the experimental support is not at the level required for the claims. Baselines are not run under matched conditions, no uncertainty is reported, and the paper's own Table 1 shows the proposed method below common baselines on most similarity benchmarks. The significance of the direction is real, but the current evidence does not establish the headline result.","major_comments":[{"comment":"The headline claim that the method 'performs comparably to context-based methods' is not supported by the evidence presented. All baseline rows are copied from Wang et al. (2019), whose models were trained on Wiki2010 text and evaluated with their own preprocessing; no confidence intervals, significance tests, or matched re-runs are reported. The point estimates in Table 1 show the proposed method below SGNS on six of eight datasets (e.g., SimLex-999 0.23 vs 0.39, Verb-143 0.30 vs 0.45, RW 0.36 vs 0.46) and below the best baseline on MC-30 (0.70 vs 0.85) and RG-65 (0.69 vs 0.85). The term 'comparable' is never defined; under the usual reading, the table contradicts it. A controlled comparison with identical evaluation code, matched vocabulary and dimensionality, and bootstrap or permutation intervals is required.","section":"Section 5, Table 1"},{"comment":"The claim of 'much less training time' is not quantified against baselines. The paper reports 10 hours for the autoencoder on one NVIDIA Ampere GPU, but no wall-clock times for SGNS, CBOW, GloVe, FastText, ngram2vec, or Dict2vec are given, and the hardware and implementation for those baselines are not stated. The time spent collecting and filtering 577,290 images is also not included. Without a matched runtime comparison, the training-efficiency claim cannot be evaluated.","section":"Section 4.1 and Section 5"},{"comment":"No evidence is provided that the 32-d autoencoder latents encode meaning rather than low-level visual statistics. The only training objective is reconstruction on 32x32 images, and the paper's own conclusion states that vector quality depends on the selected images. Because the method is motivated by semantic independence, the authors should include direct checks such as nearest-neighbor inspection, probing on semantic relations, or an ablation in which image sets are perturbed or replaced with random images; otherwise the core mechanism is untested.","section":"Section 3.2 and Section 6"},{"comment":"The outlier-detection comparison repeats the uncontrolled-baseline problem, and the concept-categorization table reports v-measure scores for the proposed method only, with no baseline values, so 'decent v-measure' is not interpretable. Additionally, the proposed vectors are 3200-d while typical baselines are 100-300-d, and the paper does not discuss whether dimensionality or vector-length normalization affects the comparisons.","section":"Section 5, Tables 2 and 3"},{"comment":"The image-retrieval pipeline has no explicit quality filter, which matters for abstract words and function words. The paper notes that question words, conjunctions, and punctuation are retained, and for these categories image search can return arbitrary or empty images. This is acknowledged as a limitation in Section 6, but it also means the premise that images can represent definition terms is not validated for a large fraction of the vocabulary. The authors should report coverage statistics (how many terms had fewer than five usable images) and show that the results are robust to excluding low-coverage terms.","section":"Section 3.1"}],"minor_comments":[{"comment":"The title contains 'V ector' with a stray space; it should read 'Vector'.","section":"Title"},{"comment":"The reference list contains duplicate entries for Luong et al. 2013a and 2013b, which appear to be the same paper.","section":"References"},{"comment":"In Table 3, 'ESSLI-2008' should be 'ESSLLI-2008'.","section":"Table 3"},{"comment":"The Almuhareb and Poesio dataset is listed without describing the clustering algorithm or hyperparameters used to compute v-measure.","section":"Section 4.2.3"},{"comment":"There are minor wording issues, including 'we finally evaluate' in the abstract and 'obtain the find word vector representation' in the conclusion.","section":"Abstract and Conclusion"},{"comment":"Figures 2 and 3 are described but never explicitly referenced in the running text.","section":"Figures 2 and 3"}],"recommendation":"reject","confidential_remarks":"The paper has a useful dataset and an interesting direction, but the evaluation does not support the abstract's claims; Table 1 actually points in the opposite direction. I would encourage the author to run a properly controlled comparison and, if the results are competitive on a defined subset of benchmarks, resubmit with a more cautious claim. As it stands, I cannot recommend publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The actual novelty here is real: using dictionary definitions, web images for each definition term, and an autoencoder to build context-free word vectors. I don't know of another paper that puts these pieces together. The dataset construction is also substantial — 577k images for 115k terms — and the author is upfront in the conclusion that vector quality will depend on image selection. That honesty counts for something.\n\nThe central problem is the evaluation, and the stress-test note is right about this. Table 1 borrows all baseline scores from Wang et al. (2019), which trained on text (Wiki2010). That is not a controlled comparison: different modality, different corpus, different preprocessing. There are no confidence intervals, no significance tests, no ablations, and no definition of what \"comparable\" means. Worse, the paper's own numbers undercut the claim. On 6 of 8 similarity benchmarks the image vectors are below SGNS (SimLex-999 0.23 vs 0.39, Verb-143 0.30 vs 0.45), and far below the best baselines on MC-30 (0.70 vs 0.85) and RG-65 (0.69 vs 0.85). The headline \"performs comparably\" is not established by this table. The training-time argument also has no baseline timings, so \"much less training time\" cannot be evaluated.\n\nThe weaker spots are in proportion: the core idea is fine and the writing is clear, but the autoencoder latent space is never directly probed for semantic content. For abstract words and function words (the paper removes some common words but keeps conjunctions and punctuation), image search will return noise or empties, and the fixed 32-dimensional latent + 3200-dimensional concatenation is never justified. These are fixable issues. Minor typos (\"V ector,\" \"T able\") are cosmetic.\n\nWho is this for? Anyone interested in visually grounded, low-resource or multimodal embeddings. The paper is not ready in its current form, but it is coherent and the dataset, if actually released, has independent value. I would not cite it today, but I would bring it to a reading group to discuss the evaluation design. This deserves a serious referee, not a desk reject — the novelty and dataset are enough to warrant a major-revision round with matched baselines, error bars, and a direct test of what the latent codes encode.","headline":"A genuinely new image-only word-vector pipeline that is undermined by an uncontrolled, borrowed-baseline evaluation; the idea deserves a revised look, not a desk reject.","tokens_in":8720,"tokens_out":1204,"would_cite":false,"duration_ms":21656,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that word vectors trained only on images of a word's dictionary definition terms match the quality of context-based embeddings while training in a fraction of the time.","keywords":["word embeddings","image-based representation","autoencoder","dictionary definitions","context-independent semantics","word similarity","concept categorization","outlier detection"],"falsifier":"Retrain the identical auto-encoder on the same images but with the mapping from words to images randomly shuffled, then compare Spearman correlations on WordSim-353; if the shuffled model scores about the same as the original, the image content itself is not carrying the semantic signal.","tokens_in":7665,"feed_emoji":"🖼️","tokens_out":10145,"duration_ms":82886,"temperature":0.7,"pith_summary":"The paper claims that a word can be represented in vector space without reading any text: you look up the word's dictionary definition, collect images for the word and for each term in the definition, train a small auto-encoder to compress those images into 32-number codes, and concatenate the codes in definition order to get the word's vector. This gives a context-free representation because meaning comes from definitions and pictures rather than from the word's usage in sentences. On word-similarity, outlier-detection, and concept-categorization benchmarks, the resulting vectors score comparably to established context-based embeddings such as skip-gram and GloVe, while the whole training run takes only ten hours on a single GPU. If the claim holds, it offers a cheap route to word embeddings for languages or domains that lack large text corpora but have observable images for their vocabulary.","feed_headline":"Image-only word vectors match text-trained quality","feed_subtitle":"Ten hours of auto-encoder training on definition images matches skip-gram and GloVe scores.","key_machinery":"The load-bearing mechanism is the image-set auto-encoder with a 32-number latent bottle-neck. Each word's vector is the flattened concatenation of the latent codes of all images in its definition chain, so the vector's structure mirrors the recursive structure of the dictionary definition: the word's own image, then the images of the words that define it, in the same order. The reconstruction objective forces the auto-encoder to compress each 32x32 image into 32 numbers, and the paper's premise is that this compression retains enough semantic content to make the concatenated codes usable as word vectors. The five images per term are meant to cover multiple senses of a word, and empty tokens pad shorter definitions to a fixed length of 100 images.","core_discovery":"The central discovery is that a purely visual pipeline can reproduce the semantic geometry of text-based word vectors. For each of 115,458 terms, the author builds an ordered image-set: five images of the term itself plus five images for each of up to 19 words in its dictionary definition, fetched from CommonCrawl and DuckDuckGo, yielding 100 images per word. A five-layer convolutional auto-encoder is trained on 577,290 resized 32x32 images to reconstruct them from a 32-dimensional latent code, and that code is treated as a meaningful representation of each image. Appending the latent codes of the image-set in the definition order produces a 3,200-dimensional word vector. The paper reports Spearman correlations on eight similarity benchmarks, accuracy on two outlier-detection sets, and v-measure on four categorization sets, and reads the results as 'comparable performance' to SGNS, CBOW, GloVe, FastText, ngram2vec, and Dict2vec, with much shorter training time.","pith_inferences":["The method's ceiling likely sits at the image-retrieval step: a cleaner or curated image collection, or a higher-resolution latent code, could improve similarity scores without changing the architecture.","Because the latent codes are never supervised by word identity, the auto-encoder could be swapped for a contrastive model that explicitly aligns same-word images; the paper does not test this, but it is a natural extension.","The 'comparable' claim is made against static embeddings trained on Wiki2010; against contextual models, the image vectors would likely lose, so the practical window for this method is in lightweight or low-resource settings."],"forward_implications":["Word vectors can be produced for any vocabulary whose terms have retrievable images, without needing a large text corpus.","The ten-hour training time on a single GPU makes the method accessible for low-resource settings where context-based embeddings are expensive.","Because the representation is built from definitions rather than usage, it may remain stable across genres, styles, or domains in ways that corpus-trained vectors do not.","The author suggests the approach may transfer across languages, since the same object images appear regardless of the language used to define them."],"supporting_citations":[{"why":"Supplies all baseline scores for SGNS, CBOW, GloVe, FastText, ngram2vec, and Dict2vec that the comparison tables rely on.","marker":"Wang et al., 2019"},{"why":"Provides the WordSim-353 human similarity judgments used as a primary similarity benchmark.","marker":"Finkelstein et al., 2002"},{"why":"Provides SimLex-999, a benchmark of genuine similarity rather than relatedness.","marker":"Hill et al., 2014"},{"why":"Provides SimVerb-3500, the verb-specific similarity evaluation.","marker":"Gerz et al., 2016"},{"why":"Provides the 8-8-8 outlier-detection benchmark.","marker":"Camacho-Collados & Navigli, 2016"},{"why":"Provides BLESS, a concept-categorization benchmark.","marker":"Baroni & Lenci, 2011"},{"why":"Provides the Battig & Montague category norms used for concept categorization.","marker":"Battig & Montague, 1969"}],"fun_headline_variants":["Image-only word vectors rival text-based embeddings","Visual pipeline produces text-quality word vectors","Word vectors from definition images, no text needed","Auto-encoder on word images matches text-trained accuracy","Images alone give word vectors on par with word2vec"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands or falls on whether a 32-number code produced by an auto-encoder whose only job is to recreate images actually captures a word's meaning, especially for abstract or function words whose web images may be arbitrary or missing.","fun_headline_variants_meta":{"raw":{"variants":["Image-only word vectors rival text-based embeddings","Visual pipeline produces text-quality word vectors","Word vectors from definition images, no text needed","Auto-encoder on word images matches text-trained accuracy","Images alone give word vectors on par with word2vec"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000521,"raw_usage":{"total_tokens":2458,"prompt_tokens":821,"completion_tokens":1637,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":1566}},"tokens_in":437,"tokens_out":1637,"duration_ms":10132,"temperature":1.0,"reasoning_tokens":1566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:38:51.149614+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the identical auto-encoder on the same images but with the mapping from words to images randomly shuffled, then compare Spearman correlations on WordSim-353; if the shuffled model scores about the same as the original, the image content itself is not carrying the semantic signal.","supporting_citations":[{"cited_title":"Placing search in context: T he concept revisited","cited_arxiv_id":null,"evidence_quote":"Provides the WordSim-353 human similarity judgments used as a primary similarity benchmark."},{"cited_title":"SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity","cited_arxiv_id":"1608.00869","evidence_quote":"Provides SimVerb-3500, the verb-specific similarity evaluation."},{"cited_title":"and Navigli, R","cited_arxiv_id":null,"evidence_quote":"Provides the 8-8-8 outlier-detection benchmark."},{"cited_title":"and Lenci, A","cited_arxiv_id":null,"evidence_quote":"Provides BLESS, a concept-categorization benchmark."}],"review_version":1}