{"id":"e3dab6d1-7dee-4c59-af5f-a9964f2a4b45","arxiv_id":"2412.05035","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A dictionary-based multi-item codec sparsely projects CLIP image embeddings onto learned semantic atoms and regenerates images with unCLIP, reaching about 1e-4 BPP per image on a 5000-image collection.","lead":"This paper builds a compression codec for image collections that encodes each image as a few coefficients from a learned 'semantic dictionary' in CLIP space, then regenerates images with Stable unCLIP at extremely low bitrates. It claims rates near 10^-5 BPP per image for large collections while keeping semantic fidelity roughly on par with single-image semantic codecs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The codec rests on Eq. (4)'s semantic linearity, which the paper illustrates rather than proves and concedes has failure cases; Eq. (8) cannot detect those failures, so the 'not sacrificing semantic fidelity' claim is unsupported for general collections.","rationale":"I agree with the reader's identification of the load-bearing assumption. The paper's own text is the strongest evidence: it calls the linearity a proof but presents visual examples, and it explicitly limits the property to concept combinations seen during CLIP training. Since CLIP training cannot cover every combination present in arbitrary collections, and Eq. (8) always returns coefficients without estimating reconstruction error in semantic space, the codec has no way to detect its own failure. This is not a disagreement with the rate calculations; it is a correctness risk in the fidelity claim. The paper could address this by adding the quantitative test above and reporting failure rates for out-of-distribution collections. Until then, CONDITIONAL is the right verdict, and my read does not change it. The experiments on dictionary interpretability (Fig. 7-9) are a useful contribution, but they do not establish the linearity premise. The absence of released code and error bars increases the burden on the proposed quantitative validation.","tokens_in":14570,"tokens_out":11385,"duration_ms":115542,"concrete_test":"Directly test Eq. (4) on 200 image pairs: 100 pairs from the Landscape dataset and 100 from a disjoint dataset (e.g., Kodak indoor/urban). For lambda in {0.25, 0.5, 1, 2}, generate y = UnCLIP(normalize(CLIP(x1)+lambda*CLIP(x2))) and compute the CLIP cosine similarity between CLIP(y) and normalize(CLIP(x1)+lambda*CLIP(x2)). Report the median similarity and the fraction of pairs for which the generated image is no closer to the weighted-sum target than to either input alone. If that fraction is not near zero, Eq. (5) fails for a material share of real images and the dictionary projection in Eq. (8) has no way to know which images are affected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on Eq. (4)-(5): CLIP latents compose additively, so every image latent can be written as a sparse sum of dictionary atoms and UnCLIP of that sum has the intended semantics. The paper states that it 'proves' Eq. (4), but the evidence shown is a handful of qualitative interpolation figures (Fig. 4-5), and the final paragraph of Sec. III concedes that the addition fails when the added concepts were not seen together during CLIP training. The claim that linearity holds for all 'natural combination[s] of semantic concepts' is asserted without measurement. This matters because Eq. (8) projects every latent onto the learned dictionary with no check for whether the image's semantics lie in the additive span of atoms. A failure produces a plausible but semantically wrong image; the residual is discarded at the encoder, and normalization before UnCLIP can further mask the error. Fig. 10 already demonstrates the failure mode for an out-of-collection input, where the projection generates database-like content. Consequently, the reported bitrate advantage can stand while the semantic-fidelity half of the headline claim is not established for collections containing concept combinations outside the dictionary's additive span.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SMIC, a multi-item semantic compression scheme for image collections. Images are encoded with CLIP, a semantic dictionary is learned by sparse coding (Eq. 7), and each image's CLIP latent is approximated by a sparse combination of atoms; the coefficients and dictionary are quantized, and decoding uses Stable unCLIP to generate images. The authors claim that CLIP's latent space is semantically linear (Eq. 4), that the learned dictionary has a semantic interpretation, and that the resulting codec achieves extremely low rates (around 10^-5 BPP per image) without sacrificing semantic fidelity, outperforming single-item generative codecs.","tokens_in":14820,"tokens_out":7237,"duration_ms":65073,"significance":"The proposal is conceptually interesting and addresses a real gap: most semantic compression schemes are single-image, while collections have semantic redundancies. The paper provides a clear framework, includes a rate-distortion analysis with dictionary overhead, and honestly states limitations of the linearity assumption. If the linearity property holds sufficiently well and the rate model is accurate, the method could be valuable for cold storage and browsing of large collections. However, the central evidence is qualitative, the bitrate is computed from a fitted curve rather than measured, and the evaluation uses the same dataset for parameter selection and testing. These issues currently prevent the quantitative claims from being fully supported.","major_comments":[{"comment":"The paper claims to 'prove' semantic linearity, but the evidence is a handful of qualitative interpolation images (Figs. 4-5) and the text explicitly concedes that the addition fails when the added concepts were not seen together during CLIP training. Since every latent is approximated by a sparse sum of dictionary atoms (Eq. 8) and the residual is discarded, there is no mechanism to detect or correct images whose semantics are not in the additive span of the atoms. Figure 10 demonstrates exactly this failure for an out-of-collection image. The 'not sacrificing semantic fidelity' claim is therefore not established for general image collections; a quantitative evaluation of Eq. (4) over diverse concept pairs is needed.","section":"Section III, Eq. (4)"},{"comment":"The reported rates, including the headline comparisons in Tables III and IV, are computed using Eq. (13), which the authors describe in Section V-C as a 'rough' fit to Fig. 12 with a 'debatable' exponent. This is not a measurement of an actual coded bitstream. The rate claim should be validated by encoding the collection with the proposed coding scheme and measuring the total length, or by providing the fitted model with error bars and validating it on held-out data.","section":"Section V-B, Eqs. (12)-(14)"},{"comment":"The R-SF-O procedure selects the best parameter sets (na, λ, bdict, bcoef) on the Landscape dataset, and the same dataset is then used for the comparison in Table IV (n=5000). This selection on the evaluation set can inflate the reported rate-semantic performance. Please use a separate validation set for parameter selection and a held-out test set for the final comparison, or justify why overfitting is not a concern.","section":"Section V-C"},{"comment":"The abstract and contributions state a rate of 'around 10^-5 BPP per image', but Table IV reports 1.4×10^-4 BPP for the selected medium model at n=5000, and the n=∞ limit from Eq. (14) does not appear to reach 10^-5 for the given parameter sets. Please clarify the conditions for the 10^-5 figure and ensure the units and database size are consistent throughout.","section":"Abstract vs Table IV"}],"minor_comments":[{"comment":"The semantic metrics CC, BSS, and CSS are cited but not defined; please provide their formulas or precise equation references so the fidelity claims can be checked.","section":"Section IV-C"},{"comment":"Eq. (5) has an index error: the sum runs over j but the atom is written as t_i; it should be t_j.","section":"Section IV-A, Eq. (5)"},{"comment":"The notation for the coefficient bit depth is inconsistent: Eq. (12) uses bc while the text and parameter lists use bcoef.","section":"Section V-B, Eq. (12)"},{"comment":"The Fig. 13 caption mentions n=1000 and n=∞, but the text says results are for n=100, 10000, 100000 and n=∞; please correct the caption.","section":"Section V-C, Fig. 13"},{"comment":"There are several grammatical errors in the abstract and introduction, e.g., 'shown to have attains better' and 'induce semantic additions and subtractions in the images are generated with UnCLIP'; a language edit would improve readability.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper's central claim rests on a property that is asserted rather than rigorously demonstrated; if the authors can provide a quantitative validation of Eq. (4) and measure actual bitrates, the contribution would be much stronger. The discrepancy between the abstract's 10^-5 BPP and Table IV's 1.4×10^-4 BPP should be resolved before publication. The heavy reliance on the authors' own prior work [21] for metrics and comparisons is acceptable but should be made self-contained for a journal readership."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is real: a dictionary learned directly in CLIP latent space, used to compress a whole image collection with sparse coefficients, plus a nice demonstration that the learned atoms behave like semantic frequencies. That is a credible extension of the authors' own single-item CLIP codec and I have not seen it elsewhere. The random-access property is preserved, the dictionary overhead is analyzed honestly, and the paper openly concedes that latent addition fails when concepts were not seen together during training. That admission is more than many papers offer.\n\nThe soft spots are load-bearing. The \"proof\" of Eq. (4) is a handful of interpolated images, not a measurement, and the failure cases the authors admit in Section III are exactly the cases that matter for a codec: Eq. (8) discards the residual and then normalizes, so a projection that falls outside the additive span of the dictionary will produce a plausible but semantically wrong image. Fig. 10 already shows this for an out-of-collection input, and the paper does not check how often it happens for in-collection images. So the \"not sacrificing semantic fidelity\" claim in the abstract is not established.\n\nThe headline bitrate also deserves caution. Eq. (13) is a fitted approximation to the measured sparsity of Fig. 12, not a coded bitstream measurement, and the authors themselves call the exponent \"debatable.\" The total rate in Eq. (14) inherits that fit. The parameter sets for the final comparisons are selected on the evaluation dataset itself, which softens the rate-distortion curves. Metrics CC, BSS, CSS are self-defined or drawn from prior work with no error bars. No code is released.\n\nHaving said that, the central idea is not invalidated by these issues. The dictionary projection is a reasonable coding scheme; the rate advantage is plausible even if the exact numbers are fuzzy; and the semantic separation experiment (projection vs. residual) is a genuinely interesting observation that could support future work. The comparison against single-item codecs is fine for the claim, since existing multi-item codecs work at pixel level and are not in the same regime.\n\nWho gets value from this? Researchers in ultra-low-rate and machine-centric compression, and anyone pushing CLIP-based generative codecs. It deserves a serious referee, but the referee should push for code release, a real coded bitstream for the rate numbers, a quantitative test of linearity in-distribution, and a discussion of failure rates. I would not desk-reject it.","headline":"A genuinely new multi-item semantic codec with a weak proof of its central linearity assumption; worth refereeing, but the semantic-fidelity headline is not supported.","tokens_in":15378,"tokens_out":1151,"would_cite":true,"duration_ms":223260,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLIP's latent-space arithmetic is semantically linear, letting one shared dictionary compress entire image collections.","keywords":["semantic compression","multi-item compression","CLIP","dictionary learning","sparse coding","generative compression","rate-distortion","image collections"],"falsifier":"Take an image whose content combines two concepts that rarely co-occur in natural images (e.g., a cathedral interior made of coral), compute its SMIC reconstruction with a dictionary trained on a disjoint set of scenes, and measure the semantic distance $d_\\Phi(x,\\hat{x})$ alongside the reconstruction error $\\|z-\\mathbf{T}c\\|_2$; if a human annotator judges the regenerated image semantically unrelated while in-distribution images stay faithful, the linearity assumption is falsified for that input, and with it the universal rate-fidelity claim.","tokens_in":14337,"feed_emoji":"🗜️","tokens_out":8911,"duration_ms":85749,"temperature":0.7,"pith_summary":"This paper tries to establish that CLIP's image-embedding space is semantically linear: adding and subtracting CLIP vectors adds and subtracts the high-level content of the corresponding images, and this property is enough to build a multi-image codec. The codec, called SMIC (Semantic Multi-Item Compression), learns a small semantic dictionary from a collection, encodes each image as a few sparse coefficients over that dictionary, and regenerates the image with a CLIP-conditioned diffusion generator. The authors report rates as low as around $10^{-5}$ bits per pixel per image (with a dictionary-inclusive rate of $1.4\\times 10^{-4}$ BPP for their medium model on a 5000-image database), beating single-image CLIP codecs and classical video codecs while keeping comparable semantic fidelity. They also show that the learned atoms are interpretable high-level concepts and that the dictionary separates in-collection semantics from out-of-collection semantics. If the claim holds, collections of a few hundred images or more can be stored far more compactly than any per-image codec allows.","feed_headline":"A shared CLIP dictionary compresses image sets to ~10^-5 BPP","feed_subtitle":"CLIP vectors add and subtract like concepts, so one learned dictionary can code a whole collection and beat per-image codecs.","key_machinery":"The load-bearing object is the semantic linearity of CLIP's latent space, together with the learned semantic dictionary $\\mathbf{T}$. Eq. (4) is the operation that lets latent additions and subtractions stand in for semantic additions and subtractions; Eq. (5) generalizes it to a weighted sum of dictionary atoms, so an image's semantics becomes a sparse coefficient vector over shared concepts. Eq. (8), an $\\ell^1$-regularized projection solved by coordinate descent, recovers those coefficients for each image, and the quantized dictionary plus coefficient list is the entire compressed representation. The dictionary thus plays the role of the statistical side information in multi-item compression: it is transmitted once and its cost is amortized over the collection, while individual images remain randomly accessible.","core_discovery":"The central claim is the semantic linearity identity $\\Phi(\\mathrm{CLIP}(x_1)+\\lambda\\,\\mathrm{CLIP}(x_2))=\\Phi(\\mathrm{CLIP}(x_1))+\\lambda\\,\\Phi(\\mathrm{CLIP}(x_2))$, stated as Eq. (4): applying the CLIP-to-image generator $\\Phi$ to a weighted sum of two CLIP embeddings is the same as generating each embedding separately and combining the resulting semantics. The authors extend this to multiple vectors, so any image whose concepts lie in the span of a learned dictionary $\\mathbf{T}$ can be written as $z=\\sum_j c_j t_j$, with the coefficients $c_j$ found by an $\\ell^1$-penalized projection. The paper argues that the dictionary learned this way is genuinely semantic: each atom generates an interpretable scene (e.g., mountain, sea, sunset), and the sparse-coefficient representation preserves semantic coherence across dictionary sizes from $n_a=2$ to $n_a=128$. On this basis, SMIC transmits a quantized dictionary once and then, for each image, a short list of quantized coefficients; the decoder reconstructs the latent vector and generates the image with unCLIP. Against the single-image CLIP-unCLIP codec, SMIC's medium configuration achieves a rate of $1.4\\times10^{-4}$ BPP, and the compression advantage begins at a collection size of 150 images.","pith_inferences":["The linearity is only as reliable as CLIP's training distribution: for combinations of concepts never seen together, Eq. (4) fails, so a production codec would need an out-of-dictionary detector or a fallback single-image mode; the paper identifies the failure but does not implement detection.","A controlled test could map the failure boundary: take a set of concept pairs, measure where the reconstructed latent $\\hat{z}$ stops generating the intended semantics, and check whether the semantic-fidelity gap correlates with the reconstruction error $\\|z-\\mathbf{T}c\\|_2$; this would tell practitioners how far they can safely push the dictionary.","The same dictionary-as-semantic-filter idea suggests a task-oriented quantization: instead of spending bits uniformly, an agent could spend more coefficients on atoms whose semantics matter for the downstream task, which is the semantic quantization the authors leave as future work.","If other large multimodal encoders share this near-linear structure, the SMIC scheme or its dictionary-learning core could transfer to video collections or multimodal documents, where inter-item semantic redundancy is even larger."],"forward_implications":["A collection of 150 images is already enough for SMIC to beat its single-image CLIP-unCLIP counterpart in total rate; for less competitive codecs the breakeven point is below 50 images.","The dictionary is learned once from the collection and can be reused for new images from the same distribution, so the collection can be added to without recompressing the existing items.","The sparse-coefficient representation is an interpretable semantic decomposition, so a user can edit or reconstruct an image by modifying individual coefficients or by adding or subtracting whole atoms.","Because the dictionary cost is fixed and the per-image rate is tiny, the rate advantage grows without bound as the collection size increases.","The projection/residual separation shows which semantics belong to the database and which do not, enabling a future semantic quantization scheme based on task-relevant concepts."],"supporting_citations":[{"why":"Supplies the single-image CLIP-unCLIP codec and the CC/BSS/CSS semantic metrics that SMIC is compared against.","marker":"[21]"},{"why":"Defines the CLIP encoder whose latent space is claimed to be semantically linear.","marker":"[22]"},{"why":"Supplies the unCLIP latent-diffusion generator used to decode latent vectors into images.","marker":"[23]"},{"why":"Provides the Stable Diffusion backbone that unCLIP is fine-tuned from.","marker":"[24]"},{"why":"Provides the online dictionary-learning algorithm used to solve Eq. (7).","marker":"[26]"},{"why":"Supplies the coordinate-descent solver used to compute sparse coefficients in Eq. (8).","marker":"[29]"},{"why":"Motivates semantic compression by establishing the perception-distortion trade-off at low bitrates.","marker":"[14]"},{"why":"Supplies the Landscape image database on which the dictionaries are learned and evaluated.","marker":"[27]"}],"fun_headline_variants":["CLIP dictionary compresses image sets to ultra-low BPP","One learned dictionary codes entire image collections semantically","Semantic compression: CLIP vectors add like concepts, dict beats codecs","SMIC: sharing a semantic dictionary slashes image set bitrate","Image collection down to 10^-5 BPP via CLIP semantic dictionary"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that every image's meaning can be assembled by adding and subtracting a fixed set of dictionary concepts inside CLIP's embedding space; when two concepts were never seen together during CLIP's training, this addition stops working, and the codec has no detector to catch the failure.","fun_headline_variants_meta":{"raw":{"variants":["CLIP dictionary compresses image sets to ultra-low BPP","One learned dictionary codes entire image collections semantically","Semantic compression: CLIP vectors add like concepts, dict beats codecs","SMIC: sharing a semantic dictionary slashes image set bitrate","Image collection down to 10^-5 BPP via CLIP semantic dictionary"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001031,"raw_usage":{"total_tokens":4363,"prompt_tokens":986,"completion_tokens":3377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":3288}},"tokens_in":602,"tokens_out":3377,"duration_ms":25893,"temperature":1.0,"reasoning_tokens":3288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:58:08.197195+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an image whose content combines two concepts that rarely co-occur in natural images (e.g., a cathedral interior made of coral), compute its SMIC reconstruction with a dictionary trained on a disjoint set of scenes, and measure the semantic distance $d_\\Phi(x,\\hat{x})$ alongside the reconstruction error $\\|z-\\mathbf{T}c\\|_2$; if a human annotator judges the regenerated image semantically unrelated while in-distribution images stay faithful, the linearity assumption is falsified for that input, and with it the universal rate-fidelity claim.","supporting_citations":[{"cited_title":"Can image compression rely on clip?","cited_arxiv_id":null,"evidence_quote":"Supplies the single-image CLIP-unCLIP codec and the CC/BSS/CSS semantic metrics that SMIC is compared against."},{"cited_title":"High- resolution image synthesis with latent diffusion models,","cited_arxiv_id":null,"evidence_quote":"Provides the Stable Diffusion backbone that unCLIP is fine-tuned from."},{"cited_title":"Online dictionary learning for sparse coding,","cited_arxiv_id":null,"evidence_quote":"Provides the online dictionary-learning algorithm used to solve Eq. (7)."},{"cited_title":"Coordinate descent algorithms for lasso penalized regression,","cited_arxiv_id":null,"evidence_quote":"Supplies the coordinate-descent solver used to compute sparse coefficients in Eq. (8)."},{"cited_title":"The Perception-Distortion Tradeoff","cited_arxiv_id":"1711.06077","evidence_quote":"Motivates semantic compression by establishing the perception-distortion trade-off at low bitrates."},{"cited_title":"Histogan: Controlling colors of gan-generated and real images via color histograms,","cited_arxiv_id":null,"evidence_quote":"Supplies the Landscape image database on which the dictionaries are learned and evaluated."}],"review_version":1}