{"id":"1c203125-1e17-469e-9db5-068d8a264ad7","arxiv_id":"2411.19475","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A tri-modal CLIP fine-tuning approach that aligns galaxy images, schematic symbols, and text labels improves galaxy classification and similarity search without large-scale astronomical pretraining.","lead":"GalaxAlign is a method for teaching general-purpose AI vision models to recognize galaxy shapes by aligning galaxy images with schematic symbols and short text labels. It reports classification accuracy close to a specialized astronomy model while avoiding the high cost of training on huge galaxy datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Symbol-modality contribution is the least-secure pillar of the tri-modal claim: the w/o-Symbol gains in Table 5 are small and unreplicated, the ConvNeXT 'Loss w/o text-image' collapse is unexplained, and Table 3 omits the CLIP baseline for retrieval.","rationale":"I agree with the reader's weakest-assumption analysis: the premise that schematic symbols add information beyond text-image alignment is the load-bearing element of the tri-modal claim. The strongest version of the paper's contribution—that mimicking citizen scientists' multimodal guidance enables efficient fine-tuning without astronomical pretraining—requires the symbol modality to matter, and Table 5 is the only direct test. That table is undermined by an implausible ConvNeXT value, by small absolute gains that are reported without variance, and by the absence of a CLIP retrieval baseline. The missing CLIP entries in Table 3 matter because CLIP is the initialization for GalaxAlign; without them, the retrieval result could in principle be achieved by the base model alone. None of this disproves the central claim, and the full-model numbers in Table 2 are internally consistent, so rejection would be too strong. However, the conditional acceptance recommended by the reader is appropriate: the authors should reproduce the anomalous ablation, add a random-symbol control, and include CLIP in retrieval before the tri-modal advantage is treated as established. I therefore leave the verdict unchanged at CONDITIONAL.","tokens_in":15324,"tokens_out":8851,"duration_ms":84239,"concrete_test":"Using the released code, rerun the ConvNeXT 'Loss w/o text-image' ablation from Table 5 with five seeds and record per-seed Galaxy10 accuracy/F1; simultaneously run a negative-control ablation in which the schematic-symbol stream is replaced by random, class-agnostic images (e.g., shuffled or blank symbols) with all other settings identical. If the 0.5781 value does not reproduce, Table 5 is not a faithful ablation; if the random-symbol control matches the real-symbol GalaxAlign accuracy, the schematic content is not the source of the gain. In the same protocol, add CLIP ViT-16 and CLIP ConvNeXT (zero-shot and text-image fine-tuned) to the Table 3 similarity-search evaluation and report mAP@5 and mAP.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.3—that tri-modal alignment with schematic symbols is what lets GalaxAlign match Zoobot and beat all other methods—rests on the assumption that the symbol modality supplies real morphological signal beyond a text-image CLIP fine-tune. The evidence for that assumption is thin and partly anomalous. In Table 5, removing the symbol stream (w/o Symbol) costs only 0.0097 accuracy on Galaxy10 and 0.0147 on GalaxyMNIST for ViT-16, and 0.0085 and 0.0062 for ConvNeXT; these differences are comparable to the run-to-run scatter reported in Table 2 and Table 5 reports no standard deviations. More seriously, the ConvNeXT row 'Loss w/o text-image' drops to 0.5781 accuracy / 0.5503 F1 on Galaxy10, about 39 points below the full model, while the corresponding ViT-16 ablation is essentially unaffected (0.9617). Section 3.3 provides no mechanism for such a backbone-dependent collapse; removing the text-image pair still leaves symbol-image and symbol-text contrastive terms, so the image and text remain indirectly aligned through the symbol encoder. This suggests either a broken training configuration or an unstable ablation, making Table 5 unreliable as support for the symbol-premise. The similarity-search claim in Section 4.4 is especially exposed because Table 3 omits CLIP entirely, so there is no direct comparison against the very pretrained model that GalaxAlign fine-tunes. If the symbol pathway does not robustly add signal, GalaxAlign reduces to a two-tower CLIP fine-tune and the claimed tri-modal advantage—and the retrieval superiority—is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents GalaxAlign, a tri-modal contrastive learning framework that fine-tunes CLIP for galaxy morphology classification and similarity search. The method aligns galaxy images, schematic symbols from the Galaxy Zoo 2 decision tree, and textual descriptions in two stages: first a shared image encoder processes images and symbols jointly with text, then three separate encoders are fine-tuned with pairwise contrastive losses. Experiments on Galaxy10 and GalaxyMNIST report classification accuracy/F1 comparable to Zoobot and better than several general-domain baselines, and higher mAP for similarity search. The paper also includes few-shot, data-size, and ablation studies, and releases code.","tokens_in":15668,"tokens_out":5524,"duration_ms":45843,"significance":"If the claims hold, GalaxAlign would offer a low-cost alternative to training domain-specific astronomical foundation models from scratch, by showing that multimodal guidance (text plus schematic symbols) can adapt generic vision-language models to a specialized scientific domain. The idea of mimicking citizen scientists' use of schematic symbols and text is well motivated, and the paper ships code, evaluates on two public datasets, and provides several baselines and ablations. However, the evidence is currently weakened by the absence of CLIP in the retrieval comparison, by non-uniform baseline training protocols, and by an unexplained ablation collapse in Table 5. These issues must be resolved before the central contribution can be firmly credited.","major_comments":[{"comment":"The ConvNeXT row 'Loss w/o text-image' reports 0.5781 accuracy / 0.5503 F1 on Galaxy10 and 0.7390 / 0.7379 on GalaxyMNIST, while the corresponding ViT-16 row is nearly unchanged from the full model (0.9617 vs 0.9732 on Galaxy10). In the tri-modal loss defined in Eq. (2), removing the text-image contrastive term still leaves the symbol-image and symbol-text terms, so the image and text modalities remain aligned transitively through the symbol encoder; the dramatic, backbone-dependent degradation therefore has no stated mechanism in §3.3. The 'w/o Text' rows also drop sharply for both backbones (to 0.7965 for ViT and 0.7599 for ConvNeXT on Galaxy10) without discussion. Because Table 5 reports no standard deviations, these extreme drops read as training instabilities rather than meaningful ablation results. Please either provide a mechanistic explanation, re-run these ablations with multiple seeds and report variance, or remove the anomalous rows.","section":"Table 5, §3.3"},{"comment":"The similarity-search comparison in Table 3 omits the CLIP models entirely, even though GalaxAlign is initialized from CLIP and then fine-tuned. Without a CLIP retrieval baseline (either unfine-tuned or fine-tuned with only text-image alignment), the claim of 'superior performance over baseline methods' does not establish that the tri-modal alignment or the schematic-symbol stream improves retrieval over the starting backbone. Please add CLIP(ViT-16) and CLIP(ConvNeXT) rows to Table 3, and ideally report the zero-shot (unfine-tuned) CLIP embeddings as well.","section":"Table 3, §4.4"},{"comment":"The baseline evaluation protocols are not uniform: Appendix A.2 states that MAE, DINOv2, MSN, and ViT-16 are evaluated with frozen encoders and a fine-tuned classification head following Lastufka et al., while ResNet-18/50, Zoobot, and GalaxAlign are fully fine-tuned. This makes the head-to-head numbers in Table 2, such as ViT-16 (0.7304) versus Ours (ViT-16, 0.9732) on Galaxy10, confound the training protocol with the method itself. The central claim that GalaxAlign 'outperform[s] all other methods' would be much more convincing if the key general-domain baselines were also evaluated under full fine-tuning (or, alternatively, under the same frozen-head protocol).","section":"A.2, Table 2"}],"minor_comments":[{"comment":"The main text states that 'just over 10 epochs' is sufficient for Stage 1, while Appendix A.1 reports training 50 epochs for the best results. Please reconcile this discrepancy and state the exact epoch count and any early-stopping criterion used for the reported numbers.","section":"Section 3.2 / Appendix A.1"},{"comment":"The spelling of ConvNeXT is inconsistent: Table 1 and Table 2 use 'ConvNext' in some places and 'ConvNeXT' in others, and Table 1 labels the backbone 'ConvNext-Base'. Please unify the naming.","section":"Table 1"},{"comment":"These figures show performance across data sizes and shot counts for many methods but contain no error bars or markers of statistical significance, making it difficult to judge whether the reported gaps are robust; adding variance would strengthen the few-shot and data-efficiency claims.","section":"Figures 6 and 7"},{"comment":"The text describing the ablations refers to 'the results in Table 4' and then immediately discusses the CLIP comparison in Table 5; please clarify in the table captions the definitions of Ours_v1, Ours_v2, Ours_v3, and Ours_Scratch, and state clearly which table corresponds to which ablation.","section":"§4.6"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a multimedia venue and the core idea is attractive, but I would not accept it in its current form. The main blockers are the unexplained catastrophic rows in Table 5 and the missing CLIP retrieval baseline in Table 3; both are addressable in a revision. I would also ask the authors to make the training configuration fully transparent (exact epoch counts, seeds, and learning-rate schedules) because the two-stage fine-tuning recipe is a central component of the claimed contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what you should know: this paper introduces a genuinely new trick — fine-tuning CLIP with three modalities (image, text, and schematic symbol) for galaxy morphology, using a two-stage warm-up then split encoders. The headline result, that this cheap fine-tune matches Zoobot's classification accuracy on Galaxy10 and GalaxyMNIST, is backed by the tables. That is a useful practical result for astronomy groups who cannot pretrain a domain foundation model.\n\nThe novelty is real but modest: the text side is just 'a picture of {class name}', and the real question is whether the symbol stream adds signal on top of a standard CLIP fine-tune. The ablation says yes, but weakly: the w/o-Symbol numbers in Table 5 are only about 0.006–0.015 below full, and there is no variance reported. More troubling, the ConvNeXT 'Loss w/o text-image' row collapses to 0.5781 accuracy on Galaxy10 while the ViT version is unaffected (0.9617). Section 3.3 offers no mechanism for that backbone-dependent collapse; it looks like a training instability or a bug. That anomaly does not kill the central classification claim — the full models are consistent and the Zoobot comparison holds — but it makes Table 5 unreliable as evidence for the symbol's contribution.\n\nThe retrieval section has a bigger gap: Table 3 compares against Zoobot and vision models but omits CLIP, the very model GalaxAlign starts from. Given that fine-tuned CLIP is within a point or two of the method on classification, the large mAP lead over Zoobot (0.9645 vs 0.8492 on Galaxy10) is hard to interpret without the CLIP retrieval baseline. It could be that the two-stage scheme helps retrieval a lot; it could also be that a plain CLIP fine-tune already does. The paper does not let you tell.\n\nAlso minor: Section 3.2 says 'just over 10 epochs' for Stage 1, Appendix A.1 says 50; the abstract promises code at a URL while the appendix says code will be public if published. These are easy to reconcile but need to be.\n\nNet: this is a solid, refereeable paper with a real if incremental contribution. The symbol-modality evidence is the soft spot, but the main result — matching Zoobot without domain pretraining — stands. I would send it to peer review, asking for the ConvNeXT ablation to be explained or fixed, CLIP added to the retrieval table, and the inconsistencies cleaned up. The reader's conditional verdict is about right; I would not desk-reject, and I would not accept as-is either.","headline":"GalaxAlign is a credible two-stage tri-modal fine-tune that matches Zoobot on classification, but the symbol modality's contribution is under-supported and the retrieval comparison omits CLIP itself.","tokens_in":16221,"tokens_out":2748,"would_cite":true,"duration_ms":24226,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a general vision-language model on galaxy images, text labels, and the schematic symbols used to instruct citizen scientists claims to match a galaxy-specialist foundation model on classification and to beat it on similarity…","keywords":["galaxy morphology","multimodal learning","CLIP fine-tuning","schematic symbols","citizen science","contrastive learning","similarity search","few-shot classification"],"falsifier":"Retrain GalaxAlign on Galaxy10 with the class-to-symbol mapping randomly permuted while keeping text and images intact; if accuracy and similarity-search mAP stay at the intact-symbol levels, the schematic modality is not carrying the claimed signal.","tokens_in":15073,"feed_emoji":"🌌","tokens_out":15855,"duration_ms":122914,"temperature":0.7,"pith_summary":"GalaxAlign sets out to show that expensive, from-scratch pretraining on large astronomical datasets is not required for strong galaxy morphology analysis. The method fine-tunes a general vision-language model while aligning three modalities: galaxy images, their textual class labels, and the schematic symbols from the Galaxy Zoo 2 decision tree that human citizen scientists are shown. In the paper's experiments, these fine-tuned models perform on par with or better than the compared astronomical foundation model on classification and achieve the highest similarity-search scores on both benchmarks. If this holds, astronomers could reuse off-the-shelf models and modest labeled sets instead of waiting years for large volunteer-annotated pretraining corpora.","feed_headline":"Images, symbols, and text let general AI match galaxy specialists","feed_subtitle":"A tri-modal fine-tuning recipe matches galaxy-specialist AI without large-scale astronomical pretraining.","key_machinery":"The load-bearing mechanism is a two-stage tri-modal alignment of CLIP-style encoders. Stage 1, a warm-up phase, runs a shared image encoder on photographs and schematic symbols paired with text through a contrastive loss, teaching a joint embedding space for visual and symbolic galaxy structure. Stage 2 copies the trained parameters into a dedicated symbol encoder and then fine-tunes image, symbol, and text encoders together with three pairwise contrastive losses (image-text, image-symbol, symbol-text). The parameter copy is what lets the symbol encoder start already aligned with images, so the encoders can specialize while remaining in a common embedding space. The schematic symbols themselves are simple line drawings from the Galaxy Zoo 2 decision tree, the same visual vocabulary used to instruct human volunteers.","core_discovery":"The paper's central claim is that a tri-modal contrastive alignment of images, text, and schematic symbols lets a general pretrained vision-language model reach the level of a galaxy-specialist foundation model without large-scale astronomical pretraining. GalaxAlign fine-tunes CLIP in two stages: Stage 1 uses a shared image encoder on both galaxy photos and schematic symbols, paired with text descriptions through a contrastive loss, so the encoder learns a joint representation of photographic and symbolic galaxy structure; Stage 2 copies the shared encoder's parameters into a dedicated symbol encoder and fine-tunes image, symbol, and text encoders together using pairwise contrastive losses. On Galaxy10 the method reports 97.32% accuracy with a ViT backbone and 97.10% with ConvNeXT, compared with 96.00% for the stronger Zoobot baseline; on GalaxyMNIST it reports 92.72% and 93.72%, bracketing Zoobot's 93.60%. For similarity search, GalaxAlign reports mAP of 0.9645 and 0.9640 on Galaxy10, above Zoobot's 0.8492. The authors interpret these results as evidence that multimodal instructions can substitute for domain-specific pretraining in galaxy morphology tasks.","pith_inferences":["The authors do not report total training cost, but their central claim implies substantial savings: if the approach holds, the practical value is avoiding the multi-year, hundred-thousand-galaxy labeling campaigns that specialist pretraining requires, so a direct GPU-hour comparison would quantify that.","Because the schematic symbols are simple and synthetic, a testable extension is replacing the exact Galaxy Zoo artwork with symbols generated on the fly from the textual class descriptions; if performance is unchanged, the information content of the decision tree, not the particular drawings, carries the effect.","The same recipe should transfer to other science domains with decision-tree or atlas-style schematic guidance (radio galaxies, cell biology, mineralogy), where symbol-image-text triplets already exist; a positive result there would confirm that the mechanism is domain alignment rather than galaxy-specific tuning."],"forward_implications":["Galaxy morphology classification becomes reachable by fine-tuning existing vision-language models, so groups without access to large annotated astronomical corpora can start from a smaller labeled set.","On Galaxy10 and GalaxyMNIST the fine-tuned models match or slightly exceed the compared astronomical foundation model in accuracy, making the approach a plausible substitute for domain-specific pretraining in these benchmarks.","Similarity search improves to 0.964 mAP on Galaxy10 from 0.849 for the best compared specialist model, strengthening the practical case for finding rare or structurally analogous galaxies from learned embeddings.","Few-shot and reduced-data experiments show the method keeps the highest or nearly highest accuracy at every tested data size, indicating the multimodal recipe is data-efficient as well as annotation-light.","The paper argues the same image-text-symbol recipe transfers to other sciences where schematic diagrams accompany imagery and descriptions, such as biology and geology."],"supporting_citations":[{"why":"Establishes that general vision foundation models fine-tuned on astrophysical images perform poorly, motivating the domain-alignment strategy and supplying the fine-tuning protocol for vision baselines.","marker":"[31]"},{"why":"Provides the state-of-the-art astronomical foundation model Zoobot and its two best backbones, the main comparison targets for classification and retrieval.","marker":"[33]"},{"why":"Supplies the CLIP architecture and the pretrained image-text encoders that GalaxAlign initializes and fine-tunes.","marker":"[23]"},{"why":"Supplies the Galaxy10 DECaLS benchmark used for the main classification, similarity-search, few-shot, and ablation experiments.","marker":"[12]"},{"why":"Supplies the GalaxyMNIST benchmark used to confirm the results on a second, smaller morphology dataset.","marker":"[32]"},{"why":"Supplies the Galaxy Zoo 2 decision tree whose schematic symbols form the third modality in the tri-modal alignment.","marker":"[38]"},{"why":"Shown as the citizen-science annotation guidance page where volunteers see symbols alongside text, the human workflow GalaxAlign mimics.","marker":"[39]"}],"fun_headline_variants":["Tri-modal alignment lets generic AI equal galaxy specialists","GalaxAlign: images, symbols, text align to match specialist AI","Fine-tune CLIP with schematic symbols to beat galaxy models","No massive pretraining needed: multi-modal fine-tuning wins","Schematic symbols teach CLIP galaxy shapes without big data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's edge depends on schematic symbols adding morphological information beyond what the text labels already provide, so that tri-modal alignment is doing real work rather than acting as a regularizer on a two-tower image-text model.","fun_headline_variants_meta":{"raw":{"variants":["Tri-modal alignment lets generic AI equal galaxy specialists","GalaxAlign: images, symbols, text align to match specialist AI","Fine-tune CLIP with schematic symbols to beat galaxy models","No massive pretraining needed: multi-modal fine-tuning wins","Schematic symbols teach CLIP galaxy shapes without big data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2963,"prompt_tokens":1015,"completion_tokens":1948,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":1874}},"tokens_in":631,"tokens_out":1948,"duration_ms":11504,"temperature":1.0,"reasoning_tokens":1874,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:08:49.410287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain GalaxAlign on Galaxy10 with the class-to-symbol mapping randomly permuted while keeping text and images intact; if accuracy and similarity-search mAP stay at the intact-symbol levels, the schematic modality is not carrying the claimed signal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the state-of-the-art astronomical foundation model Zoobot and its two best backbones, the main comparison targets for classification and retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Galaxy10 DECaLS benchmark used for the main classification, similarity-search, few-shot, and ablation experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GalaxyMNIST benchmark used to confirm the results on a second, smaller morphology dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Galaxy Zoo 2 decision tree whose schematic symbols form the third modality in the tri-modal alignment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shown as the citizen-science annotation guidance page where volunteers see symbols alongside text, the human workflow GalaxAlign mimics."}],"review_version":1}