{"id":"3063b730-6bf8-417b-a575-b3db91e5a0bc","arxiv_id":"2505.11386","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A mutual information framing for NeRF that selects low-redundancy views and adds CLIP and color regularization, reporting small PSNR gains in few-shot settings.","lead":"MutualNeRF wraps two known NeRF strategies, CLIP-based semantic consistency and camera-distance-based view selection, in a mutual information framework for limited-sample 3D reconstruction. The paper reports consistent but modest gains over FreeNeRF and ActiveNeRF baselines, while the theoretical derivation has gaps.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 1 is ill-posed: one conditional entropy cannot be proportional to both CLIP and pixel distances unless those distances are proportional; without it, the near-optimality and MI framing are unsupported.","rationale":"The reader's weakest-assumption selection is correct: Assumption 1 is the load-bearing bridge between information theory and the implemented losses/selection rules. All downstream claims — the greedy 2-approximation, the interpretation of L_macro and L_micro as maximizing mutual information, and the abstract's 'theoretically robust' claim — depend on it. The stress-test adds a sharper point: the assumption is not merely unverified; as stated it is internally inconsistent unless s and d are proportional, and the paper itself treats them as separate, sequential criteria. The proof of Lemma 2 explicitly requires a triangle inequality for H, which is available for d (as a metric) but not for s = 1 − cosine; no combined metric is provided. There is also a separate directional error in the few-shot section: Lemma 3 gives an upper bound on color difference in terms of camera distance, but the text inverts it to justify minimizing color difference as a way to reduce pixel-space distance. These are internal correctness risks, not merely disagreements with the current consensus. At the same time, the empirical contribution should be credited: Tables 1, 3, and 4 show consistent small gains over strong baselines, and the ablations support the usefulness of both CLIP-based and color-based regularization. The correct disposition remains CONDITIONAL: the empirical method is worth building on, but the theoretical wrapping should be corrected or removed, and code release is needed to verify the experiments. The proposed correlation and triangle-inequality checks are cheap and would settle whether Assumption 1 can be salvaged. Other issues, such as Definition 4's incorrect claim that multi-image mutual information equals the maximum pairwise MI, reinforce the same conclusion but are not needed for this verdict.","tokens_in":18028,"tokens_out":8933,"duration_ms":95348,"concrete_test":"On the Blender candidate set used in Table 1 (96 held-out views per scene), compute s(R,R) = 1 − CLIP cosine similarity and d(R,R) (via Lemma 1, equivalently camera-position squared distance) for every pair. Then compute the Spearman rank correlation between s and d across all pairs. If |ρ| is not close to 1, the two proportionality statements in Assumption 1 cannot both describe the same scalar H, so the theoretical derivation is internally inconsistent. As a corroborating check, enumerate any triple of views where s(A,C) > s(A,B) + s(B,C); one such triple would directly disprove the metric property that Lemma 2's proof requires of H.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Assumption 1 (Sec. 4) is the hinge of the paper: H(R|R) ∝ s(R,R) and H(R|R) ∝ d(R,R). As written, this cannot hold for a single scalar H unless s and d are themselves proportional, since H = c1 s and H = c2 d imply s = (c2/c1) d. The paper never defines a combined H = αs + βd, and the actual sparse-view selection uses s and d as separate sequential criteria (S→P / P→S), so the claimed 'unified metric' is not well defined. Lemma 2's 2-approximation proof then transfers the triangle inequality from d to H 'by Assumption 1'; but 1 − cosine similarity is not a metric (the triangle inequality fails), so even if H ∝ s held, the proof would not go through. The few-shot branch also reverses Lemma 3: Lemma 3 upper-bounds color difference by camera distance, yet the text concludes that color difference is a lower bound and that minimizing it reduces pixel-space distance, which reverses the implication. If Assumption 1 is abandoned, L_macro, L_micro, and the greedy view-selection rule remain plausible heuristics, but the abstract's 'theoretically robust' MI claim and the near-optimality guarantee have no demonstrated basis. The paper's own limitations section does not flag any of this; it only notes missing diffusion baselines.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MutualNeRF proposes a general framework, motivated by mutual information, for improving NeRF under limited samples via two mechanisms: sparse view sampling and few-shot view synthesis. The paper models the conditional information H(R|R) between images as proportional to both CLIP semantic distance and a pixel-space distance based on camera position (Assumption 1), then derives a greedy view-selection algorithm with a claimed 2-approximation guarantee and two plug-and-play regularization terms, L_macro (CLIP semantic consistency) and L_micro (color-distance regularization), intended to maximize mutual information between rendered and training views. Experiments on Blender, DTU, and LLFF under active-learning and few-shot settings report consistent improvements in PSNR, SSIM, and LPIPS over several baselines, including ActiveNeRF, DietNeRF, RegNeRF, and FreeNeRF. The abstract and introduction describe the mutual-information metric as 'theoretically robust' and the greedy algorithm as 'near-optimal.' The main contribution of the paper is this unification of view-selection and regularization under an information-theoretic framing, along with the associated experiments.","tokens_in":18345,"tokens_out":2718,"duration_ms":28809,"significance":"If the theoretical framework were sound, the paper would provide a unified and interpretable principle for both sparse view selection and few-shot NeRF regularization, with an approximation guarantee for the greedy selection algorithm. The empirical results are a genuine strength: the tables are internally consistent, the gains over ActiveNeRF in sparse-view sampling and over RegNeRF/FreeNeRF in few-shot synthesis are consistent across datasets and metrics, and the ablations support the usefulness of each proposed term. The paper also explicitly documents comparison limitations with diffusion-based baselines in its limitations section. However, the central formal claims are not currently load-bearing in the way the paper presents them, because the key assumption and the proofs that rest on it have serious gaps. The practical methods may still be valuable as heuristics, but the claimed theoretical support, the 2-approximation guarantee, and the mutual-information interpretation need either rigorous derivation or a substantial repositioning of the paper's claims.","major_comments":[{"comment":"Assumption 1 is ill-posed as stated: it asserts both H(R|R) ∝ s(R,R) and H(R|R) ∝ d(R,R), which for a scalar H implies s(R,R) ∝ d(R,R) unless one of the proportionalities is vacuous. The paper never establishes such a relation between CLIP semantic distance and camera-position distance, and the actual algorithms use s and d as separate, sequential criteria rather than as components of a single quantity. Since every theoretical conclusion, including Lemma 2 and the mutual-information interpretation of L_macro and L_micro, depends on this assumption, the theoretical claims are unsupported without a repaired definition and empirical or analytical validation of the proportionality.","section":"Section 4, Assumption 1"},{"comment":"The proof of the claimed 2-approximation transfers a triangle inequality from d to H 'by Assumption 1,' but the proof's triangle inequality step is not valid. The pixel-space distance d is an expectation of squared L2 distances between points on rays, and squared Euclidean distance does not satisfy the triangle inequality. The semantic distance s, defined as 1 minus cosine similarity, is also not a metric and does not satisfy a triangle inequality. Therefore the pigeonhole argument in Cases 1 and 2 cannot establish H(R_i1|R_i2) ≤ δ, and the 2-approximation guarantee is not proven as written.","section":"Appendix A.2, proof of Lemma 2"},{"comment":"Lemma 3 provides an upper bound ||Ĉ(r)−Ĉ(r)|| ≤ 3L||o−o||+C, but the text immediately concludes that color difference is a lower bound for camera-position difference and therefore a lower bound for pixel-space distance, and that minimizing color difference reduces pixel-space distance. This reverses the logical direction of the bound: an upper bound does not imply that color difference is a lower bound, and minimizing the left-hand side need not decrease the right-hand side. The connection between L_micro and the pixel-space distance d is therefore not established by Lemma 3.","section":"Section 6.1, use of Lemma 3"},{"comment":"The claimed distinction from prior work is partly undermined by the fact that L_macro is the DietNeRF consistency loss, as the paper itself notes in the Table 4 caption, and L_micro is a color-variance regularizer. Without a valid derivation from mutual information, the framework reduces to a combination of an existing semantic-consistency loss and a color-distance regularizer. The experiments show that this combination is empirically useful, but the paper's central novelty claim of a unified, theoretically robust mutual-information metric is not supported by the evidence as presented.","section":"Section 6.2, Table 4"}],"minor_comments":[{"comment":"The notation R and R for known and unknown images is visually indistinguishable in the text and equations; please use distinct symbols such as R_i and R_j or different fonts.","section":"Throughout"},{"comment":"The phrase 'board challenges' appears to be a typo for 'broad challenges.'","section":"Introduction"},{"comment":"The claim that the mutual information of multiple images is the maximum pairwise mutual information, attributed to Williams and Beer, should be stated as a modeling choice or heuristic rather than as a direct consequence of multivariate information decomposition; the paper should clarify that this is not a standard identity.","section":"Section 4, Definition 4"},{"comment":"The limitations section mentions the absence of diffusion-based baselines, which is appropriate, but it does not acknowledge the limitations of Assumption 1 or the proof gaps in Lemma 2 and the usage of Lemma 3; these should be discussed openly in a revision.","section":"Section 7"},{"comment":"The tables report average metrics but do not report standard deviations or per-scene breakdowns for the few-shot experiments; since the gains are sometimes small (e.g., +0.50 dB PSNR over FreeNeRF in Table 3), reporting variance would help assess significance.","section":"Tables 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The empirical work is solid and the paper may be publishable as a heuristic framework after revision. The main concern is that the title and abstract promise a theoretically robust mutual-information framework and a near-optimal selection algorithm, but Assumption 1 and the subsequent proofs do not support those claims. The authors should either provide a rigorous derivation of the proportionality, prove Lemma 2 under a valid metric or explicitly weaken the claim, and correct the logical use of Lemma 3, or rewrite the paper to present the method as a practical heuristic with the experiments as the primary contribution. The lack of code release in the current version is also a factor for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical parts of this paper are more solid than the theory. If you strip away the mutual-information claims, you have a reasonable few-shot NeRF paper with small, consistent gains from adding a CLIP loss and a color-variance loss on top of FreeNeRF, and an active-view-selection heuristic that beats ActiveNeRF on Blender. The gains are modest—about +0.5 dB PSNR on DTU/LLFF and +1.4 dB on the Blender active setting—but they are consistent across settings and the ablations are honestly reported.\n\nWhat is actually new: the sequential S→P and P→S selection procedure, and the plug-in L_micro on top of existing frameworks. The paper correctly notes that L_macro alone is DietNeRF's loss, so the novelty is in the combination and the selection rule. The experiments follow FreeNeRF's protocol, include per-object tables in the appendix, and acknowledge missing diffusion baselines.\n\nThe soft spots are in Sections 4–6 and they are not minor. Assumption 1 says one conditional entropy H(R|R) is proportional to both semantic distance s and pixel distance d. That can't hold as written unless s and d are themselves proportional. The paper never defines a combined metric. The greedy 2-approximation proof (Lemma 2) relies on H being proportional to a metric d for the triangle inequality, but the actual algorithm runs a sequential heuristic that doesn't optimize a single objective, so the near-optimality guarantee doesn't apply to what is implemented. And Lemma 3 is used in the wrong direction: it upper-bounds color difference by camera distance, yet the text concludes color difference is a lower bound and minimizing it reduces camera distance. That's a logical reversal. The mutual-information framing in the abstract ('theoretically robust') is not supported.\n\nThat said, the heuristics themselves are plausible. Minimizing redundancy in selected views and enforcing semantic/color consistency in few-shot training are reasonable ideas even without the information-theoretic veneer. The paper would be improved by removing or heavily revising the MI claims and presenting the method as what it is: two empirical regularizers plus a two-stage selection heuristic.\n\nWho is this for: people working on few-shot NeRF or active view selection. It deserves a serious referee because the empirical method is genuinely testable and the gains, though small, are consistent. I'd suggest 'major revision' with a request for code release and a rewrite of the theory section. I'd cite the selection heuristic and the empirical results, but not the MI theory.","headline":"The experiments are worth a look; the mutual-information theory around them is not.","tokens_in":18877,"tokens_out":3946,"would_cite":true,"duration_ms":37903,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mutual information, measured through CLIP semantic distance and camera/color distance, is proposed as a unified metric that guides NeRF sparse view sampling and few-shot view synthesis, with consistent quality improvements over baselines.","keywords":["Neural Radiance Fields","mutual information","sparse view sampling","few-shot view synthesis","active learning","CLIP semantic distance","greedy approximation","regularization"],"falsifier":"One could test Assumption 1 directly: for a fixed scene, estimate $H(R|R')$ from histogram binning of rendered or training images across many viewpoint pairs and compare it with $s(R,R')$ and $d(R,R')$; if the rank correlation between the measured conditional entropy and the sum $s(R,R')+d(R,R')$ is near zero or negative across those pairs, the proportionality assumption collapses and the 2-approximation guarantee loses its basis.","tokens_in":17815,"feed_emoji":"📷","tokens_out":9112,"duration_ms":79201,"temperature":0.7,"pith_summary":"This paper aims to show that mutual information can serve as a single, interpretable metric for making NeRF work well when training images are scarce. The authors operationalize mutual information at two levels: a macro level using semantic distance from CLIP embeddings and a micro level using pixel and ray distance driven by camera positions and RGB differences. Under this metric, sparse view sampling becomes a problem of minimizing mutual information between selected views, solved with a greedy look-ahead algorithm they claim is a 2-approximation. Few-shot view synthesis becomes a problem of maximizing mutual information between rendered and training images, implemented as two plug-and-play regularizers. The paper reports that this unified framing improves PSNR, SSIM, and LPIPS over strong baselines on Blender, DTU, and LLFF in both settings.","feed_headline":"Mutual information picks better NeRF views with fewer images","feed_subtitle":"CLIP + camera geometry select informative views and regularize few-shot NeRF, improving PSNR, SSIM, and LPIPS.","key_machinery":"The load-bearing object is the mutual information quantity $I(R,R') = H(R) - H(R|R')$, made computable through Assumption 1: $H(R|R')\\propto s(R,R')$ and $H(R|R')\\propto d(R,R')$, where $s(R,R') = 1 - \\cos(f(R), f(R'))$ with $f$ the CLIP image encoder, and $d(R,R')$ is the expected ray-distance, which Lemma 1 rewrites as $T_1 T_2 \\|o - o'\\|_2^2 + C$ with $T_1,T_2$ fixed integration lengths and $C$ a constant independent of camera positions. Multi-image mutual information is reduced to the maximum pairwise mutual information following the cited multivariate information decomposition. Lemma 3 bounds the rendered-color distance by $3L\\|o-o'\\| + C$ under an $L$-Lipschitz assumption on the radiance field, which justifies using color differences as the micro regularizer. The greedy look-ahead sampling algorithm, with its claimed 2-approximation guarantee, is the algorithmic consequence of this setup.","core_discovery":"The central claim is that the conditional information $H(R|R')$ needed to infer an unknown image $R'$ from a known image $R$ can be expressed by two measurable proxies: $s(R,R')$, the 1-cosine similarity of CLIP features, and $d(R,R')$, a pixel-space distance that Lemma 1 reduces to the squared Euclidean distance between camera positions. On this basis the paper defines mutual information for multiple images as the maximum pairwise mutual information, and uses it as a unified objective: sparse view sampling minimizes redundancy among chosen views with a greedy algorithm and a 2-approximation guarantee, while few-shot training maximizes information flow from training images to rendered images through a macro semantic regularizer and a micro color-distance regularizer. The paper reports consistent gains over ActiveNeRF in the sampling setting and over DietNeRF, RegNeRF, and FreeNeRF in the few-shot setting, with ablations showing both terms contribute and the semantic term contributing more.","pith_inferences":["The paper does not explore this, but if Assumption 1 holds, the same mutual-information acquisition function should transfer to any capture-planning setting where candidate camera poses are known before images exist, such as robot or drone view planning, and the greedy algorithm would offer the same 2-approximation.","The ablations place most of the few-shot gain on the macro term, which suggests a direct testable consequence: substituting a stronger or more scene-appropriate semantic encoder than CLIP should improve results further even if the micro term is unchanged.","Because Lemma 3 makes the micro term depend on color differences, the framework likely underestimates useful information in scenes with strong appearance changes such as shadows, highlights, or exposure shifts, where close cameras can still produce very different images; testing on such scenes would clarify where the proxy breaks."],"forward_implications":["In sparse view sampling, both orderings of the sequential selection, semantics-then-pose and pose-then-semantics, beat random sampling, furthest-view sampling, and ActiveNeRF on Blender in the paper's two settings, with the pose-then-semantics ordering giving the highest PSNR and semantics-then-pose giving the best LPIPS.","In few-shot synthesis, adding the macro and micro regularizers on top of NeRF, DietNeRF, RegNeRF, and FreeNeRF improves PSNR, SSIM, and LPIPS on Blender with 8 views, DTU with 3 views, and LLFF with 3 views, with the semantic term contributing most of the gain.","The greedy algorithm reduces per-iteration constraints from $O(N^2)$ to $O(N)$ while keeping a claimed 2-approximation guarantee for the optimal subset of views.","Because the regularizers are plug-and-play, the same mutual-information losses can be attached to other radiance field variants without retraining them from scratch.","Ablations show that the semantic and pixel terms are complementary: either alone helps, but combining them is better under both task settings."],"supporting_citations":[{"why":"Defines NeRF, the volume rendering formulation, and the Blender dataset used in both task experiments.","marker":"[Mildenhall et al., 2020]"},{"why":"Provides the active-learning sparse view sampling protocol and the ActiveNeRF baseline that MutualNeRF modifies and compares against.","marker":"[Pan et al., 2022]"},{"why":"Supplies the CLIP semantic encoder used to define $s(R,R')$ for both macro regularization and view selection.","marker":"[Radford et al., 2021]"},{"why":"Provides the multi-image mutual information decomposition as maximum pairwise mutual information used in Definition 4.","marker":"[Williams and Beer, 2010]"},{"why":"TupleInfoNCE is cited as the conceptual inspiration for using mutual information across modalities to model image-image correlation.","marker":"[Liu et al., 2021]"},{"why":"DietNeRF contributes the semantic consistency loss that the paper frames as an instance of the macro term, serving as a baseline and a comparison.","marker":"[Jain et al., 2021]"},{"why":"RegNeRF is a few-shot baseline whose framework the regularizers are added to, and a source of the DTU evaluation protocol.","marker":"[Niemeyer et al., 2022]"},{"why":"FreeNeRF supplies the few-shot evaluation settings on Blender, DTU, and LLFF and is improved by adding the mutual-information losses.","marker":"[Yang et al., 2023]"}],"fun_headline_variants":["Mutual info drives sparse-view NeRF sampling","Information-theoretic view selection for few-shot NeRF","Greedy mutual information sampling improves NeRF","CLIP-guided mutual information for NeRF sampling","Mutual information picks better NeRF views with fewer images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is Assumption 1 in Section 4, which says the conditional entropy $H(R|R')$ is proportional to the CLIP semantic distance $s(R,R')$ and to the pixel-space distance $d(R,R')$; this proportionality is stated without derivation, and no experiment in the paper checks that either proxy actually tracks conditional entropy.","fun_headline_variants_meta":{"raw":{"variants":["Mutual info drives sparse-view NeRF sampling","Information-theoretic view selection for few-shot NeRF","Greedy mutual information sampling improves NeRF","CLIP-guided mutual information for NeRF sampling","Mutual information picks better NeRF views with fewer images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1470,"prompt_tokens":932,"completion_tokens":538,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":464}},"tokens_in":548,"tokens_out":538,"duration_ms":5287,"temperature":1.0,"reasoning_tokens":464,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:53:49.091030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One could test Assumption 1 directly: for a fixed scene, estimate $H(R|R')$ from histogram binning of rendered or training images across many viewpoint pairs and compare it with $s(R,R')$ and $d(R,R')$; if the rank correlation between the measured conditional entropy and the sum $s(R,R')+d(R,R')$ is near zero or negative across those pairs, the proportionality assumption collapses and the 2-approximation guarantee loses its basis.","supporting_citations":[{"cited_title":"Srinivasan, Matthew Tancik, Jonathan T","cited_arxiv_id":null,"evidence_quote":"Defines NeRF, the volume rendering formulation, and the Blender dataset used in both task experiments."},{"cited_title":"Activenerf: Learning where to see with uncertainty estimation","cited_arxiv_id":null,"evidence_quote":"Provides the active-learning sparse view sampling protocol and the ActiveNeRF baseline that MutualNeRF modifies and compares against."},{"cited_title":"Contrastive multimodal fusion with tupleinfonce","cited_arxiv_id":null,"evidence_quote":"TupleInfoNCE is cited as the conceptual inspiration for using mutual information across modalities to model image-image correlation."},{"cited_title":"Putting nerf on a diet: Semantically consistent few-shot view synthesis","cited_arxiv_id":null,"evidence_quote":"DietNeRF contributes the semantic consistency loss that the paper frames as an instance of the macro term, serving as a baseline and a comparison."},{"cited_title":"Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs","cited_arxiv_id":null,"evidence_quote":"RegNeRF is a few-shot baseline whose framework the regularizers are added to, and a source of the DTU evaluation protocol."},{"cited_title":"FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization","cited_arxiv_id":"2303.07418","evidence_quote":"FreeNeRF supplies the few-shot evaluation settings on Blender, DTU, and LLFF and is improved by adding the mutual-information losses."}],"review_version":1}