{"id":"0a4ad85f-12db-4957-9d16-e4370e986e19","arxiv_id":"2412.18276","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"UNet-- replaces U-Net's four skip connections with a single compact multi-scale feature map plus an expansion module, reducing skip-connection memory by 93.3% while slightly improving restoration accuracy.","lead":"This paper introduces a U-Net variant that stores one compact feature map instead of four full-resolution skip connections, cutting the memory held for those connections by about 93%. The trade-off is a small accuracy gain on denoising, deblurring, super-resolution, and matting tasks, at the cost of a few percent more compute and parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Total activation memory is never measured; transient IEM maps generated during decoding may offset or reverse the claimed skip-connection memory savings.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the paper counts only persistent skip-connection maps (Msc) and assumes the IEM-generated multi-scale maps are either not held or cost less. My analysis shows the generated maps are transient but can be large, especially at the full-resolution stage, and may push the total activation memory above the baseline. The paper provides no total-memory table, so the central memory-efficiency claim is unverified. This does not contradict the paper's literal statement about Msc, but it undermines the practical motivation for resource-limited devices. The performance improvements, while lacking error bars, are consistent across tasks and are secondary to the memory claim. Therefore the reader's CONDITIONAL verdict remains appropriate: the paper should supply a total activation memory comparison and clarify the scope of the memory savings.","tokens_in":13177,"tokens_out":9370,"duration_ms":80983,"concrete_test":"Compute peak activation memory (in bytes) for NAFNet and NAFNet+UNet-- at inference on a 256x256 input using a memory profiler (e.g., torch.profiler or manual per-tensor accounting), including all intermediate IEM activations. If UNet-- peak total memory is lower than NAFNet's, the concern is resolved; if it is higher or comparable, the memory-efficiency claim must be revised or scoped to skip-connection storage only.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline memory claim is scoped to Msc, the persistent feature maps held between encoder and decoder (Sec. 3.1). In UNet--, Msc is reduced to the single-scale MSIAM map (0.25 MB vs. 3.75 MB). However, the IEM reconstructs full-resolution multi-scale maps during decoding; these are transient but occupy activation memory comparable to the original skip maps. For example, at the D1 stage, generating the 32-channel 256x256 map from the 256-channel 32x32 single-scale map requires pixel-shuffle with stride 8, which needs an intermediate 2048-channel 32x32 map (2 MB) plus the output 2 MB, in addition to the 0.25 MB single-scale map. The original NAFNet at D1 holds only E1 (2 MB). Thus the per-stage activation peak in UNet-- can exceed NAFNet's, even though the persistent skip memory is far smaller. The paper never measures total or peak activation memory, so the central 'memory-efficient' claim is unverified for actual deployment. The 93.3% figure applies only to a narrow accounting that excludes the very maps that replace the skip connections.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes UNet--, an architecture that replaces the multi-scale skip-connection feature maps of a U-Net with a single compact feature map produced by a Multi-Scale Information Aggregation Module (MSIAM) in the encoder, and regenerates enhanced multi-scale feature maps in the decoder through an Information Enhancement Module (IEM). The method is applied to NAFNet for image denoising, deblurring, and super-resolution, and to MSCANtiny for image matting. The authors report a 93.3% reduction in the memory held for skip connections (Msc), with small PSNR gains on denoising, deblurring, and super-resolution, and claim improvements on matting.","tokens_in":13411,"tokens_out":8684,"duration_ms":72021,"significance":"The core idea—aggregating multi-scale encoder features into a compact representation and regenerating them in the decoder—is a plausible and potentially useful direction for memory-efficient U-Net design, and the module is simple enough to be adopted in other U-Net variants. The reported Msc reductions are consistent with the architecture's design. However, the central 'memory-efficient' claim is currently scoped only to persistent skip-connection buffers, not to total or peak activation memory; the transient generation cost of IEM is never measured, leaving the deployment-oriented claim unverified. In addition, the matting results as reported contradict the stated improvements. If these issues are resolved, the contribution would be of interest to the efficient image restoration community.","major_comments":[{"comment":"The 93.3% memory reduction is measured only for Msc, the persistent feature maps held for the four skip connections (peak 15/8 ME1 = 3.75 MB for NAFNet, versus 0.25 MB for UNet--). The paper does not measure total or peak activation memory for either network. In the decoder, IEM produces feature maps with the same resolutions and channel counts as E1-E4, and the pixel-shuffle operation for restoring the E1-resolution map (stride 8) requires an intermediate 2048-channel 32x32 tensor (2 MB at the reported 256x256 input) plus the 2 MB output, in addition to the persistent 0.25 MB single-scale map. At the D1 stage, the original NAFNet holds only the 2 MB E1 map, so UNet--'s per-stage activation peak can exceed NAFNet's. Please report full-network activation memory traces (or at least peak activation memory during inference) and clarify whether IEM generates all four maps up front or stage-by-stage; without this, the title/abstract claim of memory efficiency is not established for actual deployment.","section":"Sec. 3.1, Fig. 1(b), Tabs. 4-5"},{"comment":"The text states that MSCANtiny with UNet-- 'outperforms the baseline model MSCANtiny with 0.246 (whole image), 0.402 (unknown region) in MSE', but Table 7 shows the opposite: every whole-image and unknown-region MSE entry for UNet-- is worse than the corresponding baseline entry (e.g., whole-image MSE on RWP-636: 14.835 vs 14.107, on P3M-500-NP: 2.391 vs 2.369, on P3M-500-P: 3.042 vs 2.998). The SAD entries are also mostly worse. This contradiction undermines the claim of consistent improvement across tasks. Please correct the text or provide the missing experiments/definitions of the quoted deltas.","section":"Sec. 4.4, Tab. 7"},{"comment":"The reported accuracy gains are small (e.g., +0.04 PSNR and +0.000 SSIM in denoising, Table 4) and no variance or multiple-seed results are provided. Without error bars or repeated runs, these differences may be within run-to-run noise. Please report mean +/- std over at least three training runs for the main comparisons, or otherwise demonstrate that the improvements are reproducible and statistically meaningful.","section":"Tabs. 4-6"}],"minor_comments":[{"comment":"There are several typos: 'IEM moduel' should be 'IEM module', 'enhancement header' should be 'enhancement head', and 'unsampling' should be 'upsampling'.","section":"Sec. 1, Sec. 3.3"},{"comment":"The three architecture variants in Fig. 3 are described but not experimentally compared. An ablation of the target resolution (e.g., minimum vs maximum resolution aggregation) would help justify the choice used in the NAFNet experiments.","section":"Sec. 3.3, Fig. 3"},{"comment":"The 'representative ability' metric (variance of pairwise cosine distances between channels) is non-standard, and the paper does not demonstrate a direct relationship between this metric and end-task performance. Please provide a more standard diversity metric or an explicit correlation analysis.","section":"Sec. 3.3, Tabs. 2-3"},{"comment":"The super-resolution results omit Msc, MACs, and Params columns; including them would make the memory and complexity claims complete, especially since the abstract claims consistent memory improvements across tasks.","section":"Tab. 6"},{"comment":"The text reports SSIM improvements for matting, but Table 7 lists MSE and SAD; please align the reported metrics.","section":"Sec. 4.4"},{"comment":"No code or model release is mentioned, which would aid reproducibility of the reported numbers; consider adding a footnote with the intended release plan.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The matting claim in Sec. 4.4 is directly contradicted by the paper's own Table 7, which is a significant reporting error and must be fixed. The memory claim, while correctly scoped to Msc in the tables, is presented in the title and abstract as general memory efficiency; I would ask the authors to either add full-network activation memory measurements or soften the claims. The architecture idea itself is reasonable, but the small PSNR gains need variance reporting before the improvements can be considered reliable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real architecture idea, clearly described, with consistent wins over NAFNet on four tasks. The modules are simple and the paper gives enough detail to reimplement. The 93.3% figure is honest only if you read it as Msc—the persistent skip-connection feature maps—and not as total activation memory. I think that scope is defensible, but the abstract doesn't flag it, and no total/peak memory measurement appears anywhere.\n\nWhat's new: MSIAM compresses the four encoder maps into one compact multi-scale aggregate; IEM regenerates per-scale features at decode time with a ConvNeXtV2-based enhancement block. That's a different approach from UNet++/ACC-UNet style dense fusion and from Tailor's pruning. The paper shows IEM outputs are not just copies of encoder features (SSIM ~0.02 and diversity stats), and Table 1 is a nice ablation showing no single skip connection is indispensable. Gains over NAFNet are +0.04 PSNR denoising, +0.19 deblurring, +0.398 SR, with only ~8% MACs and ~3% params. Matting also improves. The consistency across tasks argues the effect is real, though the denoising gain is within the range where I'd want multiple seeds.\n\nSoft spots, in order. (1) No error bars or seed variation; the 0.04 denoising gain could be noise. (2) No code release; rebuilding is possible but not trivial. (3) The memory claim: Msc drops from 3.75 to 0.25 MB, 93.3%, but IEM's generated maps are transient and have the same resolutions as the encoder maps they replace, so the paper should report peak and total activation memory for the whole network. The concern that the savings disappear is not proven—the original holds all four maps simultaneously while UNet-- may only hold the compact map plus one generated map at a time—but the paper doesn't measure it, so we can't tell. (4) No direct comparison to Tailor on denoising or SR; it appears only in deblurring, where UNet-- wins.\n\nVerdict: this deserves a serious referee. It's not a field-changer, but it's a useful, honest engineering contribution if the memory scope is stated more carefully. The right revision would add a total activation memory table, error bars, and code. I'd accept it for review and recommend the authors fix those gaps.","headline":"UNet-- is a credible, well-scoped engineering contribution that cuts skip-connection storage by an order of magnitude, but the headline memory claim is only about one memory component and the paper never measures total or peak activation memory.","tokens_in":13946,"tokens_out":4715,"would_cite":false,"duration_ms":41183,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing U-Net's four stored skip-connection feature maps with a single compact map cut the memory they demand by 93.3% while improving restoration accuracy.","keywords":["U-Net","skip connections","memory efficiency","image restoration","multi-scale feature aggregation","feature enhancement","NAFNet","image denoising"],"falsifier":"Measure the peak activation memory of the full UNet-- during inference on a representative input (e.g., 256×256, 32 channels) including all intermediate decoder feature maps; if the peak is not at least 50% below the baseline NAFNet's peak, the claimed memory reduction is not realized.","tokens_in":12994,"feed_emoji":"💾","tokens_out":5755,"duration_ms":44325,"temperature":0.7,"pith_summary":"The paper claims that the memory cost of U-Net's skip-connections can be reduced by 93.3% by replacing the four multi-scale encoder feature maps held in memory with a single compact aggregated map, then regenerating the multi-scale features in the decoder. The regeneration is done by a proposed Information Enhancement Module (IEM) that produces feature maps with higher diversity than the originals, which is why accuracy does not drop—it improves. Applied to NAFNet, the architecture gains 0.04 PSNR on denoising, 0.19 on deblurring, and 0.398 on super-resolution, at 7.9% more MACs and 2.8% more parameters. The method is presented as universal, with experiments also on image matting using a different backbone.","feed_headline":"Skip-connection memory in U-Net drops 93.3% with one compact map","feed_subtitle":"Aggregating encoder features to one map and regenerating them in the decoder cuts memory and improves PSNR.","key_machinery":"The central mechanism is the pair MSIAM + IEM. MSIAM reduces each encoder feature map's channels with point-wise convolutions, resizes them to the coarsest resolution (H/8 × W/8) with pixel unshuffles, concatenates them, and fuses them with a point-wise convolution; this produces a single compact map that is the only thing held in memory. IEM then pixel-shuffles that map back to the four target resolutions and passes each through an enhancement head made of a ConvNeXt V2 block and a separable convolution, producing feature maps with the same shapes as the originals but with enhanced information. The stored state becomes the compact map rather than the sum of four full-resolution encoder maps.","core_discovery":"The central claim is that the multi-scale feature maps U-Net stores for its skip connections can be collapsed into a single-scale, low-memory representation during encoding and then expanded back into richer multi-scale features during decoding, so the network only needs to hold the compact map between the two phases. The paper proposes MSIAM to aggregate the encoder's four feature maps—after channel reduction and resolution alignment—into one map of the smallest resolution, and IEM to resize and enhance that map back into the four original resolutions. The authors report that the memory held for the skip connections drops from 3.75 MB to 0.25 MB for NAFNet-sized inputs, a 93.3% reduction, while PSNR improves on all three restoration tasks; they also report that the IEM-generated features are not imitations of the encoder outputs (low SSIM) and have higher channel diversity (higher variance of pairwise cosine distances).","pith_inferences":["If the 93.3% figure is confirmed against total activation memory, the same aggregation-then-regeneration idea could be applied to other multi-scale architectures, such as feature pyramid networks, to compress their stored feature hierarchies.","The low SSIM between IEM outputs and encoder outputs suggests the IEM learns a mapping to a different, possibly more decorrelated feature space; this could be tested by measuring whether the downstream decoder's effective receptive field changes.","The paper's memory analysis only counts skip-map storage; a fair comparison of peak on-chip memory would require measuring the full activation footprint, including the IEM's intermediate maps.","One could test the enhancement claim directly by ablating the ConvNeXt V2 block in the IEM: if PSNR does not drop, the improvement may come from the aggregation rather than the enhancement."],"forward_implications":["Deploying U-Net-like restoration models on memory-constrained hardware becomes more practical: the stored activation for skip connections drops from 3.75 MB to 0.25 MB at 256×256 inputs, with total MACs and parameters increasing only modestly.","The same MSIAM/IEM swap can be applied to any U-Net variant; the matting experiment on MSCANtiny shows 94.5% skip-memory reduction with improved or comparable quality.","Because the IEM feature maps are generated on the fly during decoding, the encoder does not need to hold multiple resolution maps, which changes how U-Net memory is budgeted on accelerators.","The gain is not from simply dropping skip connections: the no-skip baselines lose 0.4 PSNR on denoising, while UNet-- gains 0.04, so the enhancement matters rather than just the removal."],"supporting_citations":[{"why":"Introduces the U-Net encoder-decoder with skip connections whose memory consumption is the target of this work.","marker":"[23]"},{"why":"Defines NAFNet, the baseline restoration model that the proposed UNet-- modifies and compares against in all restoration experiments.","marker":"[5]"},{"why":"Provides the ConvNeXt V2 block used inside the IEM enhancement head for representative ability.","marker":"[33]"},{"why":"Describes Tailor, a competing method that removes or shortens skip connections for resource efficiency, and serves as a baseline that achieves zero skip memory but no PSNR gain.","marker":"[32]"},{"why":"Proposes structural re-parameterization for merging skip connections into plain convolutions, an approach the paper argues is inapplicable to U-Net's long-distance skips.","marker":"[7]"},{"why":"Supplies the MSCANtiny backbone used in the image matting experiment to demonstrate generality beyond NAFNet.","marker":"[10]"},{"why":"Provides KBNet, a second denoising baseline that shows the method transfers to a different restoration model.","marker":"[40]"},{"why":"Supplies the SIDD dataset used for denoising training and validation.","marker":"[1]"}],"fun_headline_variants":["One map instead of four: U-Net skip memory down 93.3%","U-Net--: 93.3% less skip memory, better PSNR","Single compact map cuts U-Net skip memory by 93.3%","Collapse skip features, regenerate decoder: 93.3% memory saved","U-Net--: collapse skip features, save 93.3% memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The memory comparison counts only the four feature maps held for the skip connections; it assumes the IEM-generated multi-scale maps, which are created during decoding and have the same resolutions as the original skip maps, either do not need to be held in memory simultaneously or cost less than the maps they replace.","fun_headline_variants_meta":{"raw":{"variants":["One map instead of four: U-Net skip memory down 93.3%","U-Net--: 93.3% less skip memory, better PSNR","Single compact map cuts U-Net skip memory by 93.3%","Collapse skip features, regenerate decoder: 93.3% memory saved","U-Net--: collapse skip features, save 93.3% memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001185,"raw_usage":{"total_tokens":4918,"prompt_tokens":992,"completion_tokens":3926,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":3819}},"tokens_in":608,"tokens_out":3926,"duration_ms":22321,"temperature":1.0,"reasoning_tokens":3819,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:50:00.860631+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the peak activation memory of the full UNet-- during inference on a representative input (e.g., 256×256, 32 channels) including all intermediate decoder feature maps; if the peak is not at least 50% below the baseline NAFNet's peak, the claimed memory reduction is not realized.","supporting_citations":[{"cited_title":"In: International Conference on Medical Image Computing and Computer-Assisted Intervention (2015)","cited_arxiv_id":null,"evidence_quote":"Introduces the U-Net encoder-decoder with skip connections whose memory consumption is the target of this work."},{"cited_title":"ACM Transactions on Reconfigurable Tech- nology and Systems17, 1–23 (2024) UNet−−:U-Net with Reduced Skip-Connections 17","cited_arxiv_id":null,"evidence_quote":"Describes Tailor, a competing method that removes or shortens skip connections for resource efficiency, and serves as a baseline that achieves zero skip memory but no PSNR gain."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Proposes structural re-parameterization for merging skip connections into plain convolutions, an approach the paper argues is inapplicable to U-Net's long-distance skips."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SIDD dataset used for denoising training and validation."}],"review_version":1}