{"id":"42cb7ec7-f6b6-48c4-9e2e-467af40fe109","arxiv_id":"2505.20892","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"One-time soft alignment between forward and fixed feedback weights at initialization substantially improves feedback alignment training and approaches backpropagation accuracy on small image tasks.","lead":"This paper proposes initializing feedback-alignment networks with forward and backward weights already aligned, then leaving them free, and shows this one-time step improves training speed and accuracy on several image benchmarks. The result is a simple, cheap way to get closer to backpropagation-level learning without the weight transport and memory overhead that backpropagation needs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's deep-network parity claim is undercut by the paper's own Table S1: on the 5-conv AlexNet variant IFA is 65.55% vs BP 73.10%, and the Limitations section concedes the gap.","rationale":"The reader's verdict is CONDITIONAL, and our stress-test supports that verdict without changing it. The shallow-network experiments are internally consistent and well controlled: the angle sweep in Fig 4 cleanly shows that the correlation between forward and feedback weights at initialization, not the initialization distribution itself, drives the improvement. However, the strongest claim in the abstract is broader than the evidence. The paper's own Table S1 and Figure S3 show a widening performance gap as the architecture deepens: IFA is more than 7 points behind BP on the 5-conv AlexNet variant, and Section 6 admits this gap. This directly contradicts the phrase deep networks achieve performance comparable to backpropagation if taken literally. The reader's weakest assumption, that the one-time alignment remains informative for the whole training run, is essentially the same concern: the persistence of the alignment is the mechanism that would make the abstract claim true, and the paper provides only empirical angle trajectories for a shallow MLP, not for the deeper CNNs where the claim is made. We therefore agree with the reader's identification. Our concrete test would quantify the decay of the IFA error signal relative to the BP error signal across depth, which would determine whether the gap is caused by alignment decay or by something else, such as optimization difficulty or architecture. The verdict remains CONDITIONAL: the core idea is promising and the shallow evidence is solid, but the headline claim needs to be narrowed or supported by additional deep-network evidence before acceptance as stated.","tokens_in":977,"tokens_out":1051,"duration_ms":100326,"concrete_test":"Compute, on the 5-conv AlexNet variant of Table S1, the per-layer cosine similarity between the IFA feedback signal B_l delta_{l+1} and the exact BP signal W_l^T delta_{l+1} at each epoch, alongside the same quantity for the 3-layer MLP of Fig 4. If the cosine decays to near zero within the first few epochs for the CNN while staying high for the MLP, the one-time alignment does not persist at depth, and the abstract's comparable to backpropagation claim must be restricted to shallow networks. As a complementary check, re-run the CNN while re-aligning B_l to the current W_l every epoch; if that closes the 7.55-point gap, alignment persistence, not architecture or optimization, is the bottleneck.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 implements IFA by setting W_l = B_l^T cos(theta_init) + R sin(theta_init), so at theta=0 the feedback matrices B_l are exactly the transpose of the initial forward weights and the first backward pass is identical to BP. The central question is whether this one-time identity remains informative as W_l moves away from B_l^T. The paper's only evidence on this is the angle trajectories in Fig 4b and Fig S8, measured on a 3-layer MLP; no analytical bound or layer-wise error-signal measure is provided. The paper's own Table S1 shows the premise degrades with depth: on the 2-conv+3-FC CNN, IFA is 56.37% vs BP 62.13%; on the 5-conv+3-FC AlexNet variant, 65.55% vs 73.10%, a 7.55-point gap. Section 6 (Limitations) explicitly concedes that a performance gap remains between the approach and full backpropagation in deeper or more complex models. Thus the strongest claim in the abstract is not supported for the deep networks it names; it is supported only for shallow MLPs. This is an overclaim rather than an internal inconsistency, but it is load-bearing because the paper's stated contribution is precisely that deep networks achieve performance comparable to backpropagation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes initial feedback alignment (IFA), which initializes the forward weights as a soft projection of fixed random feedback weights (Algorithm 1) and then trains with standard feedback alignment. The main empirical claim is that this one-time initialization, without any further weight transport, yields learning comparable to backpropagation on shallow MLPs, smoother Hessian spectra, better corruption robustness, and improved adversarial robustness. Experiments on CIFAR-10, SVHN, CIFAR-100, STL-10, and two CNN variants are reported.","tokens_in":19524,"tokens_out":4539,"duration_ms":45924,"significance":"The idea is of interest: if a single initialization event is sufficient, it offers a simple and arguably more biologically plausible alternative to exact weight transport. The MLP experiments are carefully controlled (ten runs, rank-sum tests), and the Hessian and robustness analyses are systematic. However, the abstract's deep-network claim is not supported by Table S1, and the central mechanism (why the initial alignment remains informative) is analyzed only empirically. With revision, the paper could still support a useful, narrower claim.","major_comments":[{"comment":"The abstract states that \"a one-time soft alignment ... enables deep networks to achieve performance comparable to backpropagation,\" but Table S1 shows a 7.55-point gap against BP on the 5-conv+3-FC network (65.55% vs 73.10%) and a 5.76-point gap on the 2-conv+3-FC network (56.37% vs 62.13%). Section 6 explicitly concedes that \"a performance gap remains between our approach and full backpropagation in deeper or more complex models.\" The headline claim should be restricted to the shallow MLP regime or accompanied by results that actually support deep-network parity.","section":"Abstract; Section 6; Table S1"},{"comment":"At theta_init = 0 degrees, Algorithm 1 line 4 sets W_l = B_l^T, so the first backward pass is exactly backpropagation; the initial improvement over FA at this point is therefore partly definitional. The paper's only evidence that the fixed B_l remains useful as W_l moves away from B_l^T is the angle trajectory in Fig. 4b and Fig. S8 for a three-layer MLP. No analytical bound, layer-wise gradient alignment measure, or error-signal quality metric is provided, and Table S1 shows the approach degrades with depth. Please add a quantitative analysis of how the instructive power of B_l decays, or substantially narrow the claim.","section":"Section 3, Algorithm 1; Section 4.3, Fig. 4"},{"comment":"The claim that IFA achieves performance \"not significantly different from BP\" rests on a two-sided rank-sum test with n = 10 and P = 0.082. Absence of significance at this sample size is not evidence of comparability; an equivalence test, a confidence interval for the accuracy difference, or an effect-size report is needed if the abstract's \"comparable to backpropagation\" wording is retained.","section":"Section 4.1, Fig. 1h"},{"comment":"The adversarial-robustness claim is confounded by clean accuracy. Fig. 7c reports that larger initial misalignment (larger theta_init) improves accuracy under FGSM, but Fig. 4d shows that the same angles reduce clean accuracy. Since FA (theta = 90 degrees) has the lowest clean accuracy, the robustness gain may simply reflect the known accuracy-robustness tradeoff rather than an advantage of IFA. Please report robust accuracy at matched clean accuracy, or otherwise separate the effect of initial alignment from the accuracy penalty.","section":"Section 4.4, Fig. 7"}],"minor_comments":[{"comment":"The word \"emplyoed\" should be \"employed\"; the manuscript also contains LaTeX artifacts such as \"f anin\" in Algorithm 1 and Section B.2 that should be cleaned up.","section":"Section 4.3"},{"comment":"The text refers to \"Figure S1b\" and \"Figure S1c\" when describing layer-wise alignment dynamics for the second and final layers, but the relevant panels appear to be in Figure S8; the cross-reference should be corrected.","section":"Supplementary A.3.1"},{"comment":"The code availability statement says the code \"will be made available after the paper is published\"; since the paper is under review, please provide the code or a supplement to make the reported ten-run experiments reproducible.","section":"Section 7"},{"comment":"The memory-access comparison in Figure 1d lacks axes and units, making the claimed computational advantage difficult to quantify.","section":"Figure 1d"}],"recommendation":"major_revision","confidential_remarks":"The shallow-network contribution appears sound and the paper is a good candidate after revision, but the abstract overstates the deep-network results and the mechanism section is underdeveloped. I would encourage the editor to require a revised abstract, a sharper statement of the regime in which IFA matches BP, and additional analyses of misalignment decay and adversarial robustness before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, well-controlled empirical study of a simple initialization trick for feedback alignment. The trick works: setting initial forward weights to a mixture of the fixed feedback transpose and a random matrix dramatically stabilizes FA on MLPs, and the systematic angle sweep plus Hessian analysis are genuinely useful. But the abstract says deep networks perform \"comparable to backpropagation,\" and the paper's own Table S1 contradicts that: on the 5-conv AlexNet variant, IFA gets 65.55% vs BP 73.10%. The Limitations section admits the gap. That mismatch is the main thing to fix.\n\nWhat's actually new: the reparameterization W_l = B_l^T cos(theta) + R sin(theta) is a clean way to interpolate between FA and BP at initialization, and the paper is the first to my knowledge to sweep the angle systematically and tie it to Hessian spectra, corruption robustness, and adversarial robustness. The adversarial result — moderate misalignment helps against FGSM, BIM, and PGD — is interesting and not something I'd have predicted. The experiments look honest: ten runs, rank-sum tests, consistent architecture controls. The Hessian-vector product methodology is standard and properly referenced.\n\nSoft spots, in proportion: the load-bearing claim about deep parity is simply not supported by the data they ship. On a 3-layer MLP, IFA is statistically indistinguishable from BP, which is a real result. But the CNN numbers show a 6-8 point gap, so \"comparable\" is doing too much work. Also, part of the trainability benefit at theta=0 is definitional — the first backward pass exactly equals BP. The paper could say that more plainly instead of presenting it as an emergent discovery. There's no code release, which is annoying but not disqualifying. The decay of alignment over training is only characterized empirically; no bound or layer-wise error-signal measure. That's a minor issue because the empirical trajectories are clear, but it limits the theory.\n\nThe stress-test note is fair. I read the paper the same way: it's an overclaim in the abstract, not an internal inconsistency. The framing should be narrowed to \"shallow and moderately deep networks,\" and the CNN gap should be in the abstract. The developmental story about prenatal noise pretraining is speculative but clearly labeled as such.\n\nBottom line: this is a useful contribution to the bio-plausible learning subfield. A careful referee could help the authors fix the framing, release code, and maybe add one analytic observation about why the first-pass alignment helps. I'd send it out. Who's it for: anyone working on feedback alignment, weight transport, or neuromorphic training. I'd bring it to reading group and likely cite it if I were writing on FA initialization.\n\nRecommendation for peer review: accept with revisions if the authors narrow the claim and release code; desk rejection would be too harsh given the solid shallow-network results and the angle sweep.","headline":"A useful, well-controlled study of a simple initialization trick for feedback alignment, but the abstract overstates deep-network parity; the core results hold for shallow MLPs.","tokens_in":20071,"tokens_out":2289,"would_cite":true,"duration_ms":23089,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single soft alignment of forward and feedback weights at initialization lets deep networks train with fixed random feedback to backpropagation-level accuracy, with no weight transport during learning.","keywords":["initial feedback alignment","weight transport problem","feedback alignment","biologically plausible learning","loss landscape","Hessian spectral analysis","adversarial robustness","neural network initialization"],"falsifier":"Track the per-layer alignment angle between forward and feedback weights during IFA training on networks of depth 2 through 10: if the angle returns toward 90 degrees within the first few hundred update steps on the deeper nets while their final accuracy still trails backpropagation, the one-time alignment has stopped being load-bearing and the method collapses to baseline feedback alignment. A sharper version is to measure the early-training decay rate of alignment, $\\mathrm{d}\\theta/\\mathrm{d}t$, and check whether it grows with depth in a way that predicts the performance gap already visible in Table S1.","tokens_in":19006,"feed_emoji":"🧠","tokens_out":8346,"duration_ms":76795,"temperature":0.7,"pith_summary":"Backpropagation's need to copy forward weights into the backward pass at every step is both an energy cost and a biological implausibility, known as the weight transport problem. This paper claims that a single, approximate alignment of forward and feedback weights at initialization removes that need: networks then train with fixed random feedback, never re-synchronizing, and still reach test accuracy statistically indistinguishable from backpropagation on CIFAR-10. The broader point is that what makes feedback alignment unstable is largely a bad starting configuration, and a one-time fix suffices. If the claim holds, biologically plausible learning and neuromorphic hardware get a minimal recipe: mirror the weights once at startup, then let learning run with local signals.","feed_headline":"One-time weight alignment matches backprop without weight transport","feed_subtitle":"Starting weights aligned once lets fixed random feedback match backprop's accuracy — and survive attacks better.","key_machinery":"The load-bearing object is the initialization rule $W_l = B_l^\\top \\cos(\\theta_{\\mathrm{init}}) + R \\sin(\\theta_{\\mathrm{init}})$, where $B_l$ is the fixed random feedback matrix, $R$ is an independent random matrix of matching variance, and $\\theta_{\\mathrm{init}}$ is the soft-alignment angle. At $\\theta_{\\mathrm{init}} = 0^\\circ$ the forward weights equal the feedback weights exactly; at $\\theta_{\\mathrm{init}} = 90^\\circ$ the rule reproduces standard feedback alignment; intermediate angles interpolate between the two. This single scalar controls everything the paper reports: whether the angle between pathways relaxes or tightens during training, how smooth the loss landscape remains, how far the trainability envelope extends in depth, variance, and data scarcity, and the trade-off between clean accuracy and adversarial robustness.","core_discovery":"On the paper's own terms, the central discovery is that the learning trajectory of a feedback-alignment network is governed by its initial alignment state rather than by ongoing coupling between pathways. When forward weights are initialized to coincide with the fixed random feedback weights, or at a controlled angle to them, training proceeds by gradually relaxing that alignment while preserving backpropagation-like error signals; standard feedback alignment, which starts with orthogonal weights, spends the entire run trying to align them and produces erratic, unstable trajectories. Across weight-variance settings, network depths up to ten layers, dataset sizes from 100 to 50,000 samples, and four image benchmarks, the aligned start raises final accuracy and steers convergence to flatter minima with a compressed Hessian spectrum. A corollary the paper emphasizes is that the natural 'loosening' of alignment during fixed-feedback training gives IFA networks stronger resistance to FGSM, BIM, and PGD adversarial attacks than networks trained by exact backpropagation.","pith_inferences":["The natural next quantity is a bound on alignment drift: a proof that $\\|W_l - B_l^\\top\\|$ grows slowly relative to the learning rate would turn the paper's empirical stability claim into a predictive theorem, including its depth limits.","The initialization rule reads like a hardware recipe: mirror forward weights into the feedback pathway once at chip power-on, then update locally forever — a cheap, testable prescription for neuromorphic and analog platforms where per-iteration weight transport is precisely what is expensive.","The angle $\\theta_{\\mathrm{init}}$ is a tunable robustness knob that backpropagation lacks, since clean accuracy and adversarial robustness move in opposite directions with the alignment angle; a designer could set it to match the deployment threat level.","Because the paper's developmental story invokes noise-driven pretraining as the source of natural alignment, one could test that story directly: pretrain a network on random noise and measure whether the emergent alignment angle matches the empirically optimal $\\theta_{\\mathrm{init}}$."],"forward_implications":["Fixed random feedback plus a one-time aligned start reaches accuracy statistically indistinguishable from backpropagation on moderate feedforward networks (CIFAR-10, $P = 0.082$), with no weight transport during training.","Initial alignment widens the trainable region: IFA trains successfully across a much broader range of forward and backward weight variances, at greater depth, and with far fewer training samples than baseline FA.","Aligned starts converge to flatter minima, with lower Hessian trace and smaller maximum eigenvalue, and this tracks improved robustness across all fifteen CIFAR-10-C corruption types at multiple severity levels.","Misalignment is not pure loss: IFA-trained networks retain more accuracy under FGSM, BIM, and PGD adversarial attacks than backpropagation-trained networks at equal perturbation strength.","Soft alignment interpolates smoothly between these regimes, so a moderate initial angle (around $30^\\circ$ to $60^\\circ$) captures most of the accuracy gain while preserving some of the adversarial robustness."],"supporting_citations":[{"why":"Supplies feedback alignment, the baseline algorithm whose fixed random feedback IFA retains.","marker":"[23]"},{"why":"Defines backpropagation, the exact-symmetry algorithm whose weight transport IFA claims to avoid.","marker":"[1]"},{"why":"Shows noise pretraining induces forward-feedback alignment; provides the developmental mechanism for how one-time alignment could arise naturally.","marker":"[38]"},{"why":"Provides evidence that biologically motivated learning rules scale poorly in deep networks, the gap IFA is designed to close.","marker":"[26]"},{"why":"Supplies the Hessian-vector product and Lanczos methods used for all loss-landscape spectral claims.","marker":"[44]"},{"why":"Provides the corruption benchmark (CIFAR-10-C) used to measure robust generalization across fifteen corruption types.","marker":"[50]"},{"why":"Defines the FGSM attack used to demonstrate IFA's adversarial robustness advantage over backpropagation.","marker":"[51]"},{"why":"The sign-symmetry method that still requires sign transmission during learning, contrasted with IFA's one-time-only requirement.","marker":"[27]"}],"fun_headline_variants":["Soft-start weight alignment matches backprop without transport","One-time alignment makes fixed feedback rival exact backprop","Initial soft alignment: feedback learning hits backprop accuracy","Align feedback once at start for robust, transport-free learning","Weight alignment at init yields robust learning sans transport"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the one-time alignment stays informative for the whole training run: forward and feedback weights must not drift back out of alignment faster than learning can use the error signal, and the paper supports this only with empirical trajectories (Fig. 4b, Fig. S8), not an analytical bound. Its own Table S1 shows IFA still trails backpropagation by 6 to 8 points on deeper convolutional variants, indicating the premise weakens with depth.","fun_headline_variants_meta":{"raw":{"variants":["Soft-start weight alignment matches backprop without transport","One-time alignment makes fixed feedback rival exact backprop","Initial soft alignment: feedback learning hits backprop accuracy","Align feedback once at start for robust, transport-free learning","Weight alignment at init yields robust learning sans transport"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1475,"prompt_tokens":901,"completion_tokens":574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":499}},"tokens_in":517,"tokens_out":574,"duration_ms":6568,"temperature":1.0,"reasoning_tokens":499,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:45:02.729912+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Track the per-layer alignment angle between forward and feedback weights during IFA training on networks of depth 2 through 10: if the angle returns toward 90 degrees within the first few hundred update steps on the deeper nets while their final accuracy still trails backpropagation, the one-time alignment has stopped being load-bearing and the method collapses to baseline feedback alignment. A sharper version is to measure the early-training decay rate of alignment, $\\mathrm{d}\\theta/\\mathrm{d}t$, and check whether it grows with depth in a way that predicts the performance gap already visible in Table S1.","supporting_citations":[{"cited_title":"Rumelhart, Geoffrey E","cited_arxiv_id":null,"evidence_quote":"Defines backpropagation, the exact-symmetry algorithm whose weight transport IFA claims to avoid."},{"cited_title":"Pretraining with random noise for fast and robust learning without weight transport","cited_arxiv_id":null,"evidence_quote":"Shows noise pretraining induces forward-feedback alignment; provides the developmental mechanism for how one-time alignment could arise naturally."},{"cited_title":"Assessing the scalability of biologically-motivated deep learning algorithms and architectures","cited_arxiv_id":null,"evidence_quote":"Provides evidence that biologically motivated learning rules scale poorly in deep networks, the gap IFA is designed to close."},{"cited_title":"Pyhessian: Neural networks through the lens of the hessian","cited_arxiv_id":null,"evidence_quote":"Supplies the Hessian-vector product and Lanczos methods used for all loss-landscape spectral claims."},{"cited_title":"How important is weight symmetry in backpropagation? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016","cited_arxiv_id":null,"evidence_quote":"The sign-symmetry method that still requires sign transmission during learning, contrasted with IFA's one-time-only requirement."}],"review_version":1}