{"id":"9c68315c-81ac-4591-80d9-ba692083db87","arxiv_id":"1908.02013","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Ensembling classifiers over joint, reconstructed visual, and reconstructed semantic spaces, with temperature scaling, improves GZSL harmonic-mean accuracy on CUB, AWA1 and AWA2.","lead":"This paper combines three classifiers built from a shared visual-semantic model and averages their outputs with learned temperature scaling, aiming to improve generalized zero-shot learning. It reports state-of-the-art results on three of four standard benchmarks, but with small gains and without released code or error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Decoded spaces are deterministic functions of the same latent z, so the ensemble cannot add information; Table 2 shows uncalibrated averaging hurts (CUB H 41.5 vs 52.4) and the reported gains may come from validation-fit temperatures rather than the reconstructed spaces.","rationale":"The reader's weakest assumption is that Dx and Da produce unseen-class reconstructions retaining enough discriminative information for the extra classifiers to help. My concern is related but sharper: because both decoded spaces are deterministic functions of the same z, they cannot add information in the mutual-information sense, and Table 2's ablation suggests the uncalibrated decoded classifiers actually hurt. The large jump from the uncalibrated ensemble (H=41.5 on CUB) to the calibrated ensemble (H=54.0) indicates that the three validation-fit temperatures, not the reconstructed features, may be doing the work. That does not refute the empirical claim that the calibrated procedure improves harmonic mean over the reproduced CADA-VAE baseline, but it makes the central explanation unsupported until the contribution of the decoded spaces is isolated. The paper deserves credit for reproducing CADA-VAE, reporting an ablation, and providing AUSUC results; however, the lack of error bars and withheld code leave the small gains (0.1 to 1.6 H points) within the range of run-to-run variation and alternative explanations. I would keep the reader's CONDITIONAL verdict: the claim is plausible but requires the concrete ablation above, a stated calibration objective, and code/error bars before it can be accepted as demonstrating that multi-modal reconstructed spaces are the source of improvement.","tokens_in":11644,"tokens_out":7932,"duration_ms":94173,"concrete_test":"Re-run the Table 2 CUB and AWA2 ablations with Dx(z) and Da(z) replaced by two fixed random linear projections of z (same output dimensionalities as x and a), using the same classifier architectures and the same temperature-calibration procedure on the validation split. If the calibrated ensemble with random projections matches MCADA-VAE (H roughly 54.0 on CUB and 65.0 on AWA2), the decoded spaces contribute no class-discriminative signal and the reported gain is attributable to calibration/weight selection rather than to the reconstructed embeddings. If H drops materially, the reconstructions do carry signal beyond z that the ensemble exploits.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing issue is not only whether Dx and Da reconstruct unseen classes well; it is whether the decoded spaces can contribute at all beyond the joint latent z. In Eq. (4), x~=Dx(z) and a~=Da(z) are deterministic functions of z=Ex(x), so any classifier on them is a classifier on z. The ensemble therefore cannot, as the paper claims in Section 3.3, let the classifier 'see more information' than z already contains; at best it supplies different function classes over the same information. Table 2 makes the empirical situation stark: on CUB, the individual decoded classifiers have unseen accuracy 28.0 and 25.0 versus 48.4 for z, and the uncalibrated average of the three spaces ('MCADA-VAE (tau=1)') drops H from 52.4 (z alone) to 41.5. After fitting three temperatures on the validation split, H jumps to 54.0. With only three scalar temperatures, the calibrated ensemble can effectively re-weight or down-weight the two weak classifiers, so the reported gains over z-CADA-VAE (H +1.6 on CUB, +0.1 on SUN, +0.9 on AWA1, +1.1 on AWA2) may be a validation-set model-selection effect rather than evidence that the reconstructed spaces add class-discriminative signal. The paper does not state the objective used to fit the temperatures (e.g., NLL versus harmonic mean), reports no error bars, and withholds code, so this alternative explanation is not excluded by the current evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MCADA-VAE, a generalised zero-shot learning method that extends CADA-VAE by classifying in three spaces: the joint latent space z, the reconstructed visual space x~ = Dx(z), and the reconstructed semantic space a~ = Da(z). Three linear softmax classifiers are trained on these spaces, their outputs are temperature-scaled with one scalar per classifier fit on the validation split, and the final prediction is the average of the three calibrated probability vectors. The paper claims that the three embedding spaces contain complementary information, that temperature calibration removes the need for ensemble-weight model selection, and reports state-of-the-art harmonic-mean accuracy on CUB, AWA1 and AWA2, with competitive performance on SUN.","tokens_in":12050,"tokens_out":5990,"duration_ms":63123,"significance":"If the empirical claims were robust, the paper would make a useful practical observation: post-hoc ensembling of decoded spaces from a multi-modal VAE can improve GZSL accuracy, and scalar temperature scaling is a simple combination rule. The paper follows the standard Xian et al. protocol, evaluates on four benchmarks, and includes an ablation (Table 2) that transparently shows the uncalibrated average underperforms the joint-space classifier. However, the central conceptual claim is questionable, the reproduced CADA-VAE baseline differs from the published numbers, and the reported improvements over that baseline are small and unreplicated. As it stands, the manuscript does not establish that the reconstructed spaces contribute class-discriminative signal beyond the joint latent space.","major_comments":[{"comment":"Because x~ = Dx(z) and a~ = Da(z) are deterministic functions of z = Ex(x), the classifiers p(y|x~) and p(y|a~) are functions of z only. By the data processing inequality, the ensemble cannot 'see more information' than z already contains; at best it supplies different function classes over the same information. Table 2 is consistent with this concern: on CUB, the decoded-space classifiers have unseen accuracy 28.0 and 25.0 versus 48.4 for z, and the uncalibrated average drops H from 52.4 to 41.5. To support the paper's central claim, the authors should provide evidence that the decoded spaces add class-discriminative signal beyond z, for example by comparing against a calibrated ensemble of multiple classifiers trained directly on z under the same temperature-fitting procedure.","section":"Section 3.3, Eq. (4)"},{"comment":"The reproduced CADA-VAE baseline differs from the published CADA-VAE numbers in the seen/unseen trade-off while preserving the same harmonic mean. For example, on CUB the published values are YS=53.5, YU=51.6 but the reproduced values are YS=57.2, YU=48.4; on SUN they are 35.7/47.2 versus 36.8/45.1; on AWA1 they are 72.8/57.3 versus 76.6/55.0. This unexplained discrepancy is load-bearing because the claimed improvements are computed relative to the reproduced baseline, and the paper does not state which implementation details (e.g., hyperparameters, training length, or evaluation code) caused the difference.","section":"Section 4.4 / Table 3"},{"comment":"Fitting three temperature parameters on the validation set is itself a model-selection step, so the claim in Sections 3.3 and 5 that calibration 'mitigates the problem of model selection' is overstated: the method replaces two ensemble weights with three validation-fit temperatures. The paper does not report the objective used to fit the temperatures (e.g., negative log-likelihood versus harmonic mean) or the selected temperature values. Moreover, the statement in Section 2.3 that calibration 'does not change the classification accuracy' is contradicted by Table 2, where moving from tau=1 to the calibrated ensemble changes CUB H from 41.5 to 54.0; this is because the temperatures change the relative weighting of the ensemble members.","section":"Section 4.4 / Table 2"},{"comment":"The reported harmonic-mean improvements over the reproduced z-CADA-VAE baseline are small: +1.6 on CUB, +0.1 on SUN, +0.9 on AWA1, and +1.1 on AWA2, with no error bars or multiple-run statistics. Given that Table 2 shows the uncalibrated ensemble substantially hurts performance, it is important to show that the final gains are not within run-to-run variance or an artifact of validation-set temperature fitting. The authors should report mean and standard deviation over multiple seeds, and ideally the temperature values, to allow the reader to assess the stability of the result. The code is also withheld ('Link to our Github repository here after review'), which further hinders verification.","section":"Section 5 / Table 3"}],"minor_comments":[{"comment":"Equation (4) has mismatched parentheses in the last term and does not explicitly define lambda_x and lambda_a; the text later states they are set to unity after calibration, but this should be stated where the equation appears.","section":"Section 3.3, Eq. (4)"},{"comment":"There are incorrect table references: 'Table 4 contains some basic information about the data sets' should refer to Table 1, and 'The ablation study in Table 4' in Section 5 should refer to Table 2.","section":"Section 4.1 / Section 5"},{"comment":"Figure 3 is qualitative; the claim of complementary class structure across the three spaces would be strengthened by a quantitative measure, such as nearest-class separability or retrieval accuracy in each space.","section":"Section 5 / Figure 3"},{"comment":"The footnote and repository URL for the CADA-VAE code contain formatting artifacts ('edgarschnfld' and a space in 'CADA-V AE-PyTorch'); the actual URL should be checked and typeset correctly.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The incremental contribution over CADA-VAE is modest, and the central claim that decoded spaces add information beyond the joint latent space is not currently supported by the evidence. The unexplained difference between the reproduced and published CADA-VAE numbers, the lack of error bars, and the withheld code are additional concerns. If the authors can supply a concrete ablation isolating the effect of the decoded spaces, report temperatures and a proper multi-seed evaluation, and either reconcile the baseline discrepancy or restrict comparisons to their reproduced baseline, the paper could become publishable; without those elements, the empirical case is too thin for a journal paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a respectable but incremental follow-up to CADA-VAE, and the paper's headline explanation is wrong. The reconstructed visual and semantic spaces are deterministic functions of the same latent z, so they cannot carry 'more information' than z; at best they give the classifier a different function class over the same information.\n\nWhat's genuinely useful: the ensemble idea is clean, the temperature-scaling calibration is a sensible way to avoid manually tuning the combination weights, and the paper follows the standard GZSL protocol on the four usual benchmarks. The ablation table is honest enough to show that the individual reconstructed-space classifiers are weak (CUB unseen accuracy 28.0 and 25.0 vs 48.4 for z), and the uncalibrated average actually hurts (H drops from 52.4 to 41.5). That's a useful negative result.\n\nWhere it gets soft. The reported gains over z-CADA-VAE are small (H +1.6 on CUB, +0.1 on SUN, +0.9 on AWA1, +1.1 on AWA2), with no error bars or multiple runs. Three temperature parameters are fit on the validation set, and with those you can effectively down-weight the two weak classifiers, so the improvement may be validation-set model selection rather than evidence that the reconstructed spaces help. The paper doesn't state the objective used to fit the temperatures. Also, the reproduced CADA-VAE baseline differs from the original numbers (CUB YS 57.2 vs 53.5, YU 48.4 vs 51.6) with no explanation. Code is promised only after review. These are all addressable, but as submitted the central claim is not well supported.\n\nMy read: the stress-test concern is on target, but I wouldn't call it load-bearing in the sense that the method is incoherent. It's an empirical claim with a wrong motivation. If the authors release code and show the gains survive multiple seeds and a proper calibration analysis (e.g., what happens if you fit a single temperature, or use NLL vs harmonic mean), the paper could be a solid workshop-level or modest conference contribution. As is, it's a borderline reject for a top venue.\n\nWho is it for: GZSL researchers who care about ensembling and calibration. It deserves a serious referee because it's complete and on a standard benchmark, but I'd expect the referee to ask for code, error bars, and a much more careful discussion of what the reconstructed spaces actually contribute.\n\nRecommendation: send to peer review, but the bar at a top venue should be conditional accept at best, likely reject until the empirical questions are answered.","headline":"Incremental GZSL extension with a wrong information-theoretic justification; the small gains likely come from validation-set temperature fitting rather than the reconstructed spaces.","tokens_in":12524,"tokens_out":2610,"would_cite":false,"duration_ms":24317,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that generalized zero-shot classification improves when three embedding spaces—joint, reconstructed visual, and reconstructed semantic—are combined as a temperature-calibrated ensemble.","keywords":["generalized zero-shot learning","ensemble classification","multi-modal embedding","temperature scaling","variational autoencoder","harmonic mean","visual-semantic alignment","calibrated classifiers"],"falsifier":"Run the same trained model twice on the test sets: once with the two reconstruction branches as designed, and once with their inputs replaced by random vectors of the same dimension while keeping the joint branch unchanged; if harmonic-mean accuracy does not fall meaningfully, the reconstructed spaces are not carrying the claimed ensemble gain.","tokens_in":1413,"feed_emoji":"🎯","tokens_out":2228,"duration_ms":85061,"temperature":0.7,"pith_summary":"Generalised zero-shot learning has to label images from classes never seen during training, using only text or attribute descriptions of those classes. The paper argues that every common solution funnels the test image through one embedding space—joint, visual, or semantic—and that each space discards information the others keep. It proposes to classify in all three at once: map the image into a joint latent space, decode back into visual and semantic spaces, train one softmax classifier in each, and average their outputs. Because naive averaging hurts, each classifier's scores are rescaled by temperature calibration before the average. On CUB, AWA1 and AWA2 this ensemble reports state-of-the-art harmonic-mean accuracy and improves over the single-space baseline on all four benchmarks.","feed_headline":"Ensembling three embedding spaces lifts zero-shot accuracy","feed_subtitle":"Temperature-scaled averaging over joint, reconstructed visual, and reconstructed semantic spaces improves generalized zero-shot…","key_machinery":"The load-bearing object is the aligned variational autoencoder: two encoders, $E_x:\\mathcal{X}\\to\\mathcal{Z}$ and $E_a:\\mathcal{A}\\to\\mathcal{Z}$, push visual and semantic inputs into one shared latent space, and two decoders, $D_x$ and $D_a$, pull latent vectors back into the visual and semantic domains. Training aligns the two modalities through reconstruction and distribution-matching losses. The paper adds one linear softmax classifier per space and calibrates each with temperature scaling, a single scalar per classifier that softens the softmax so the three probability outputs are comparable. The calibrated average in Eq. (4) is the final predictor.","core_discovery":"The paper's central claim is the ensemble rule $p(y|x)=p(y|E_x(x))+\\lambda_x p(y|D_x(E_x(x)))+\\lambda_a p(y|D_a(E_x(x)))$: a test image is encoded by $E_x$ into the joint latent space $z$, and three classifiers vote—one on $z$ itself, one on the visual reconstruction $\\tilde{x}=D_x(z)$, and one on the semantic reconstruction $\\tilde{a}=D_a(z)$. The weights $\\lambda_x$ and $\\lambda_a$ are not tuned; instead each classifier is temperature-calibrated so that a fixed equal-weight average works. The authors claim this exposes the classifier to complementary information in each space, and they support it with ablations showing the three spaces have different class geometries and with benchmark results where the ensemble outscores its best single member.","pith_inferences":["The same calibration-then-average recipe could be applied to any collection of classifiers whose scores are not directly comparable, not just to the three spaces considered here; this is a testable extension the authors do not pursue.","Because the reconstructed-space classifiers alone perform poorly on unseen classes in the paper's ablation, a dedicated experiment varying decoder quality—more capacity or stronger alignment loss—would show whether the ensemble gain scales with reconstruction fidelity or saturates.","On the SUN benchmark the gain is smaller; a natural follow-up is to check whether ensembling helps most when seen and unseen classes are visually similar rather than when the class set is large and diverse.","If the benefit comes from information preserved in reconstructions, then feeding semantic attributes with more class-discriminative detail should increase the ensemble's advantage over the joint-space classifier."],"forward_implications":["Classifying over reconstructed visual and semantic spaces converts information that a joint-space classifier would discard into usable votes.","Temperature calibration removes the cross-validation search for ensemble weights, making the multi-classifier combination a one-line average at test time.","The ensemble improves harmonic-mean accuracy over the strongest single space on CUB, AWA1 and AWA2, and improves AUSUC over the baseline on all four data sets.","Each embedding space has a different class geometry, so no single learned space is a sufficient statistic for GZSL classification.","The approach carries over to unseen classes whose images were never seen in training, because all three classifiers are trained on synthetic samples generated from semantic attributes."],"supporting_citations":[{"why":"Supplies the aligned variational autoencoder whose joint and reconstructed spaces the ensemble reuses.","marker":"[19]"},{"why":"Defines the GZSL evaluation protocol, seen/unseen splits, and benchmark numbers the method must beat.","marker":"[25]"},{"why":"Provides temperature scaling, the calibration used to make the three classifiers' probabilities comparable.","marker":"[11]"},{"why":"Supplies the AUSUC metric used to evaluate the seen/unseen trade-off across the ensemble.","marker":"[5]"},{"why":"Prior generative GZSL baseline whose reported results are used as comparison.","marker":"[9]"},{"why":"Prior feature-generating GZSL method providing baseline results and benchmark context.","marker":"[26]"},{"why":"Prior approach that combines seen and unseen classifiers, the closest comparison to the proposed ensemble.","marker":"[3]"}],"fun_headline_variants":["Three spaces beat one in zero-shot learning","Ensemble of visual and semantic spaces aids zero-shot","Calibrated classifier ensemble improves generalized zero-shot","Mixing embedding spaces boosts zero-shot classification","Ensembling embeddings: a new GZSL approach"],"cache_read_input_tokens":14592,"weakest_assumption_plain":"The whole result rests on the decoders, trained only on seen classes, reconstructing unseen-class visual and semantic vectors that still carry class-discriminative information; if those reconstructions are mostly noise, the two extra classifiers cannot add signal and the method collapses to the joint-space classifier.","fun_headline_variants_meta":{"raw":{"variants":["Three spaces beat one in zero-shot learning","Ensemble of visual and semantic spaces aids zero-shot","Calibrated classifier ensemble improves generalized zero-shot","Mixing embedding spaces boosts zero-shot classification","Ensembling embeddings: a new GZSL approach"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000159,"raw_usage":{"total_tokens":1228,"prompt_tokens":945,"completion_tokens":283,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":211}},"tokens_in":561,"tokens_out":283,"duration_ms":3707,"temperature":1.0,"reasoning_tokens":211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:55:57.201712+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same trained model twice on the test sets: once with the two reconstruction branches as designed, and once with their inputs replaced by random vectors of the same dimension while keeping the joint branch unchanged; if harmonic-mean accuracy does not fall meaningfully, the reconstructed spaces are not carrying the claimed ensemble gain.","supporting_citations":[{"cited_title":"Schonfeld, S","cited_arxiv_id":null,"evidence_quote":"Supplies the aligned variational autoencoder whose joint and reconstructed spaces the ensemble reuses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides temperature scaling, the calibration used to make the three classifiers' probabilities comparable."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the AUSUC metric used to evaluate the seen/unseen trade-off across the ensemble."},{"cited_title":"Felix, B","cited_arxiv_id":null,"evidence_quote":"Prior generative GZSL baseline whose reported results are used as comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior feature-generating GZSL method providing baseline results and benchmark context."},{"cited_title":"Atzmon and G","cited_arxiv_id":null,"evidence_quote":"Prior approach that combines seen and unseen classifiers, the closest comparison to the proposed ensemble."}],"review_version":1}