{"id":"aed456d3-150b-4568-b81e-2834b9f84cf7","arxiv_id":"2411.19285","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BPQP reformulates the backward pass of differentiable convex optimization layers as an equality-constrained quadratic program, allowing fast ADMM-based solvers to compute gradients.","lead":"This paper introduces BPQP, a method that speeds up training of neural networks that contain convex optimization problems as layers, by turning the gradient computation into a simpler quadratic program solvable by fast first-order solvers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's exact gradient claim needs strict complementarity and a correctly recovered active set; OSQP's finite-tolerance solutions do not guarantee either.","rationale":"The reader's verdict already identifies active-set correctness and strict complementarity as the weakest assumption, and my read agrees. The entire efficiency argument rests on replacing the KKT linear system Eq. (3) with the equality-constrained QP Eq. (6); if that replacement is not guaranteed exact, the backward gradients are approximate and the claimed generality of the method is not established. The example above shows that the proof's equivalence claim fails without strict complementarity: the two linear systems impose different constraints on tilde z, so the gradient is not uniquely determined by Eq. (3) alone. This does not mean BPQP is useless; under generic strict complementarity and a well-chosen active set, the reformulation is sound and the speedups are plausible. The right response is to require the authors to state the missing condition, prove Theorem 1 under it, and include a degenerate-instance experiment demonstrating what happens when the condition is violated. The unbenchmarked ADMM differentiable QP baseline and the unsupported non-convex remarks are secondary; the active-set condition is the gate through which the central exactness claim must pass.","tokens_in":18839,"tokens_out":11433,"duration_ms":103888,"concrete_test":"Construct the degenerate QP minimize 0.5||z||^2 + q^T z subject to z_1 <= 0, z_2 <= 0 at q=0, with loss L=z_1+z_2. Compute the BPQP gradient with (a) both constraints placed in the active set and (b) both treated as inactive. The two outputs for grad_q L are (0,0) and (-1,-1), and the original KKT system Eq. (3) is singular and admits both. This settles that Theorem 1 needs strict complementarity or an explicit selection rule for weakly active constraints. As a computational complement, run a random ensemble of QPs with P=10^{-6}I and near-degenerate constraints, comparing BPQP gradients at OSQP tolerances 1e-4 and 1e-8 against an exact active-set solve; a cos-similarity drop below the 0.992 reported in Table 3 would confirm that finite-tolerance active-set misclassification affects gradient quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the backward-pass gradient vector [tilde z, tilde lambda, tilde nu] solving Eq. (3) can be obtained as the optimal solution of the equality-constrained QP Eq. (6). The proof in Appendix A.2 replaces the complementarity block D(lambda*)G(z*) in Eq. (3) with the equality block G_+(z*) in Eq. (22). This replacement is exact only when every constraint i with g_i(z*)=0 has lambda*_i > 0 and every constraint with lambda*_i=0 has g_i(z*) < 0, i.e., strict complementarity. Without this condition, the i-th complementarity row of Eq. (3) is identically 0=0, so Eq. (3) imposes no constraint on tilde z, while Eq. (6) enforces G_{i+} tilde z = 0. The two linear systems are then not equivalent and can select different gradients. For example, minimize 0.5||z||^2 + q^T z subject to z_1 <= 0, z_2 <= 0 at q=0, with loss L=z_1+z_2. Eq. (3) admits both tilde z=(-1,-1) and tilde z=(0,0) with suitable dual variables, producing different parameter gradients. The problem is nondifferentiable at q=0, but this is precisely the degenerate regime Theorem 1 does not exclude by only assuming that the Jacobian vector exists. Moreover, even under strict complementarity with a small margin, OSQP returns z*, lambda* only to finite tolerance (Section 4.2), so the active set is inferred approximately; a near-active constraint that is misclassified changes Eq. (6) and hence the gradient. This is not a numerical footnote: the LP experiments use P=10^{-6}I in Eq. (17), making the Hessian nearly singular and near-degenerate constraints plausible. The theorem needs an explicit strict-complementarity/active-set-correctness hypothesis, or the exact-gradient claim should be weakened to an approximation with error controlled by the forward tolerance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BPQP, a differentiable convex optimization layer that reformulates the backward pass of implicit differentiation as an equality-constrained quadratic program (Eq. 6) whose KKT system coincides with the linear system obtained by differentiating the KKT conditions of the forward problem (Eq. 3), under an active-set replacement. The backward QP is then solved with OSQP, decoupling the forward and backward solvers. Experiments compare runtime and gradient accuracy against CVXPY, qpth/OptNet, Alt-Diff, JAXOpt, and an Exact matrix-inverse baseline on randomly generated QPs, LPs, and SOCPs, and on a CSI 500 portfolio-optimization task, reporting large speedups and competitive or improved decision metrics.","tokens_in":19269,"tokens_out":18853,"duration_ms":156698,"significance":"If the equivalence in Theorem 1 is established under the right assumptions, the central idea is practically useful: it turns the backward linear solve into a convex QP that can leverage mature first-order solvers, and the decoupling of forward and backward passes is attractive for large-scale end-to-end learning. The paper provides a proof in Appendix A.2, releases code through Qlib, and reports extensive runtime and accuracy experiments. Its value, however, depends on the exactness of the backward gradient under the stated conditions, which are currently too weak, and on the credibility of the LP and SOCP experimental claims. The theoretical contribution is a reformulation with empirical support rather than a new mathematical theory.","major_comments":[{"comment":"The proof of Theorem 1 replaces the complementarity block D(λ*)G(z*) and D(g(z*)) in Eq. (3) with the equality block G_+ in Eqs. (20)-(21). This replacement is exact only when the active set is known exactly and every active inequality satisfies λ*_i > 0 (strict complementarity). The stated assumptions—'not primal infeasible' and 'the corresponding Jacobian vector ∇yL exists'—do not imply strict complementarity. When a constraint is weakly active (g_i(z*)=0, λ*_i=0), the i-th complementarity row of Eq. (3) is identically 0=0 and imposes no constraint on \\tilde z, while Eq. (6) enforces G_{i+}\\tilde z=0; the two systems can then produce different gradients. A concrete degenerate case is min_z 0.5||z||^2 + q^T z subject to z_i ≤ 0 at q=0 with loss L=z_1+z_2, where Eq. (3) admits both \\tilde z=(-1,-1) and \\tilde z=(0,0), but Eq. (6) forces \\tilde z=0. In addition, OSQP returns solutions only to finite tolerance (Section 4.2), so the active set recovered from the forward pass is approximate; a near-active constraint that is misclassified changes Eq. (6) and hence the gradient. Theorem 1 should state the strict-complementarity and exact-active-set assumptions, or be reformulated as an approximate-gradient statement with a perturbation bound.","section":"Theorem 1 / Appendix A.2"},{"comment":"The experiments labeled 'LP' solve minimize θ^T z + ε||z||^2_2 subject to Az=b, Gz≤h with ε=10^{-6}. This is a strictly convex quadratic program, not a linear program. For a genuine LP (P=0), the KKT matrix in Eq. (3) can be singular and the backward solution nonunique; the paper provides no analysis of this case. Consequently, the claimed 21.02× speedup on LP and the statement that BPQP handles LP layers are not demonstrated for standard LPs. The experiment should be reported as 'regularized LP' or, preferably, true LP instances should be tested with a discussion of degeneracy and uniqueness.","section":"Section 5.1, Eq. (17)"},{"comment":"Table 1 gives absolute times with row labels '(scale 1.0e-04)' for Exact, CVXPY, qpth/OptNet, Alt-Diff, and JAXOpt, but '(scale 1.0e+00)' for BPQP. As printed, the BPQP entries are in different units from the other entries, and the quoted speedups in the abstract and Section 1 are not recoverable unless all rows are read with the same implicit scale. For example, the QP 100×20 total time for Exact is 484.2×10^{-4} s and for BPQP is 35.1×10^{-4} s if the same scale is used, giving the ~13.5× speedup; with the printed scale labels, BPQP would appear to be orders of magnitude slower. Please restate the table with one explicit unit for all rows or report raw times in a common unit.","section":"Table 1"},{"comment":"The claim that BPQP 'is still equipped to reformulate the backward pass as a QP' for non-convex problems is not supported. For a non-convex objective, the matrix P' in Eq. (6) need not be positive semidefinite, so the backward problem is not a convex QP and OSQP's convergence guarantees (Section 4.2) do not apply. The 'General Gradients' paragraph defines gradients that preserve the KKT norm at intermediate iterates, but Appendix A.4 does not show that these surrogate vectors equal or approximate the true loss gradient; it only states an identity dr(k)=0 and concludes norm preservation. This is a heuristic, not a theorem. Either provide conditions under which the KKT-norm-preserving update yields the correct gradient, or remove the non-convex claims from the discussion.","section":"Section 6 and 'General Gradients' in Section 4.1"}],"minor_comments":[{"comment":"There are dimension errors in the notation: h should map R^d to R^m and g should map R^d to R^n, and the backward variables in Eq. (3) should be \\tilde λ ∈ R^n and \\tilde ν ∈ R^m, not the reverse as printed.","section":"Definition 1 and Section 3.2"},{"comment":"The condition list for equivalence of Eq. (5) and Eq. (3) is confusing: D(\\tilde λ)G' = D(λ*)G(z*) is a matrix equation involving the unknown \\tilde λ, not a condition that can be checked a priori. The exposition would benefit from stating that the active-set replacement is used precisely to avoid this issue.","section":"Section 4.1"},{"comment":"There is a typo: 'the original optimization problem in can be reformulated' should read 'in Definition 1 can be reformulated'.","section":"Appendix A.2, Eq. (19)"},{"comment":"The text says 'The CosSim. of all methods are small enough for SOCP' but the table shows values of 1.00; the intended wording is probably 'close to one' or 'sufficiently high'.","section":"Table 3"},{"comment":"The sentence 'While its hard to perform experiments on non-convex problem due to the lack of baselines' contains grammar errors and should be rephrased.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional assessment matches my reading: the central equivalence is elegant and the empirical story is potentially interesting, but the paper overreaches in three places—the theorem's unstated regularity assumptions, the LP experiment that is actually a QP, and the non-convex discussion. All of these are fixable within the manuscript's scope, so I recommend major revision rather than rejection. I would also suggest that the editor ask the authors to tighten the 'General Gradients' subsection or remove it if it cannot be made rigorous."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"BPQP is worth your time if you work on differentiable optimization layers. The core idea is simple and, as far as I can tell, new: instead of solving the KKT linear system in the backward pass directly, the paper shows that the gradient vector can be obtained by solving an equality-constrained QP whose data comes from the active set of the forward solution. That means the backward pass can be handed to any QP solver — OSQP in this case — which decouples the forward and backward passes and opens the door to first-order methods. The decoupling is real, not cosmetic: it lets BPQP reuse the forward solver's active-set information and exploit sparsity and polishing. I buy the central claim, with a caveat.\n\nThe caveat is Theorem 1. The proof in Appendix A.2 replaces the complementarity block D(λ*)G with the active-set equality block G_+. That step is exact only when the active set is known exactly and strict complementarity holds. OSQP returns solutions to finite tolerance, so in practice the active set is inferred approximately; near-degenerate constraints can flip a row and change the backward QP. The paper states the theorem without these conditions. The stress-test example (q=0, z1≤0, z2≤0) shows that without strict complementarity the original KKT system and the active-set QP can select different gradients. This is a genuine gap, but it is patchable: state the assumptions, or weaken the exact-gradient claim to an approximation with error controlled by the forward tolerance. In the generic case the experiments suggest the error is small; CosSim around 0.99 is evidence, not proof.\n\nThe experiments are decent but have warts. The 'LP' benchmark is a regularized LP with ε=1e-6, which matters for the degeneracy concern. The scaling in Table 1 is confusing (two different scale factors in one table). The comparison set omits the closest related work — Butler and Kwon's ADMM differentiable QP layer — which weakens the 'state-of-the-art' claim. Section 6's non-convex claims are hand-waving and should be cut or supported. The 'General Gradients' paragraph in Section 4.1 is also too vague to be useful. The portfolio experiment is fine: BPQP is 2.75x faster than OptNet with comparable portfolio metrics, and the Sharpe improvement over two-stage is real.\n\nOverall: this is a solid engineering contribution with a mathematically correct core. It belongs in the literature, and a serious referee should engage with it, but the revision should tighten the theorem statement and clean up the benchmarks. I would bring it to a reading group, mainly to discuss whether active-set errors are a practical problem. I would cite it if I worked in this area.\n\nRecommendation: send it to peer review. Not a desk reject, but expect heavy revision on the theory section.","headline":"A genuinely useful reformulation of the backward pass as an equality-constrained QP, with a correct-but-underspecified Theorem 1 and a strong but slightly messy experimental section.","tokens_in":19823,"tokens_out":3195,"would_cite":true,"duration_ms":27405,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the backward pass through a differentiable convex optimization layer can be rewritten as an equality-constrained quadratic program, making gradient computation a standard QP solve.","keywords":["differentiable optimization layers","convex optimization","implicit differentiation","quadratic programming","KKT conditions","ADMM","backpropagation","portfolio optimization"],"falsifier":"Construct a small QP whose optimum has an inequality constraint with a dual multiplier comparable to the solver's tolerance (for example $10^{-6}$), then compute the backward gradient both by the paper's Eq. (6) and by a high-precision solution of Eq. (3). If the two disagree beyond numerical noise, or if changing the solver tolerance flips the active set and produces a discontinuous gradient, the exact-equivalence premise of Theorem 1 fails in a practically reachable regime.","tokens_in":18639,"feed_emoji":"⚡","tokens_out":19373,"duration_ms":135494,"temperature":0.7,"pith_summary":"This paper claims that the backward pass through a differentiable convex optimization layer — normally a costly linear solve with the KKT matrix — can be rewritten as an equality-constrained quadratic program whose optimal solution is exactly the gradient vector needed for backpropagation (Theorem 1). The reformulation decouples the forward and backward passes, so any general-purpose QP solver, in practice a first-order operator-splitting method, can compute gradients. If correct, this makes end-to-end training with optimization layers markedly cheaper: the paper reports total runtime speedups of up to 13.54x, 21.02x, and 1.67x on QP, LP, and SOCP layers, and it demonstrates a 500-asset portfolio optimization task that other differentiable layers could not handle at scale. The sympathetic reading is that gradient computation through KKT conditions is itself a convex QP, and exploiting that structure is enough to turn differentiable optimization from a bottleneck into a practical layer.","feed_headline":"Backward pass as a QP cuts optimization-layer training time by 21x","feed_subtitle":"Backprop solves a small quadratic program instead of a costly linear system, so optimization layers train faster.","key_machinery":"The central object is the KKT matrix of the forward optimization problem, and the key move is to replace its complementarity block — the diagonal matrices $D(\\lambda^\\star)G(z^\\star)$ and $D(g(z^\\star))$ — with equality constraints on the active set, the set of inequality constraints that hold with equality at the optimum. Because the active set is known after the forward pass, the backward gradient vector solves the equality-constrained quadratic program in Eq. (6), whose KKT conditions reproduce the desired linear system exactly. This hands the backward pass to any QP solver; the paper uses a first-order ADMM solver with sparsity, solution polishing, and active-set machinery, plus a small regularization $\\delta I$ to handle redundant constraints.","core_discovery":"The central claim is Theorem 1. Once the forward pass has identified the active set of inequality constraints, the vector $[\\tilde z, \\tilde\\lambda, \\tilde\\nu]$ that solves the backward KKT system (Eq. 3) is exactly the optimal solution of the equality-constrained quadratic program (Eq. 6) with objective $\\tfrac12 \\tilde z^\\top P' \\tilde z + q'^\\top \\tilde z$ and constraints $A'\\tilde z = b'$, $G'_+\\tilde z = c'_+$. The complementarity block of the KKT matrix is replaced by the active-set equality rows, which turns the backward system into a plain convex QP that any QP solver can handle. To guard against a singular KKT matrix, the implementation adds a small regularization to the diagonal. On simulated problems the resulting gradients match a high-precision KKT solve with 0.992 cosine similarity for QP, while total runtime is reduced by up to 21x on LP layers.","pith_inferences":["Inference: the reported speedups should grow with the fraction of inactive constraints, since the backward QP keeps only the active rows; the paper does not isolate this dependence, but it follows directly from the reformulation.","Inference: because the active set is read off a finite-tolerance solver's output, the natural stress test is a degenerate problem where a dual multiplier is near zero; Theorem 1 is stated without explicitly requiring strictly positive multipliers on all active inequalities, and this is where the backward QP gradient could diverge from the exact KKT value.","Inference: the same active-set-to-QP trick could be applied to higher-order derivative computations, such as Hessian-vector products, or to non-convex forward maps near a local minimum, since the derivation only uses the KKT structure at the returned point; the paper hints at the non-convex extension but does not test it."],"forward_implications":["End-to-end training of networks with optimization layers becomes practical at problem scales where direct KKT inversion is prohibitively slow, such as the reported 500-asset portfolio optimization task.","Because the backward pass is a standard convex QP, future improvements in QP solvers translate directly into faster differentiable layers without changing the method.","The forward solver no longer needs to be differentiable or share factorizations with the backward pass, so each pass can use the best available algorithm for its structure.","The paper's measurements show total runtime speedups of up to 13.54x on quadratic programs, 21.02x on linear programs, and 1.67x on second-order cone programs, with backward gradient cosine similarity 0.992 on QP against a high-precision KKT solve.","The method scales to large sparse problems (up to 5000x2000 in the experiments), where the paper reports that competing differentiable layers fail to generate results."],"supporting_citations":[{"why":"Supplies the KKT-condition implicit differentiation setup that BPQP reformulates into a QP.","marker":"[14]"},{"why":"Supplies the ADMM-based first-order QP solver and solution-polishing machinery used in both forward and backward passes.","marker":"[16]"},{"why":"Frames the 'Exact' baseline that computes gradients by direct KKT matrix inversion within deep declarative networks.","marker":"[2]"},{"why":"Establishes the differentiable convex optimization layer paradigm and the implicit-differentiation baseline for general cone programs.","marker":"[1]"},{"why":"Provides the unrolling-based baseline whose efficiency the paper compares against on QP problems.","marker":"[12]"},{"why":"Offers a prior ADMM-based differentiable QP layer with a custom fixed-point mapping that BPQP generalizes.","marker":"[18]"}],"fun_headline_variants":["Backward pass as QP: 21x faster optimization-layer training","Turn backprop into a QP to cut training time 21x","BPQP: solve backward pass as QP, get 21x speedup","Optimization layers train 21x faster with BPQP's QP trick","Active-set QP backprop slashes training time by 21x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The equivalence in Theorem 1 rests on knowing the exact active set and on every active inequality having a strictly positive dual multiplier; a first-order solver returns only a finite-tolerance solution, so on weakly active or degenerate constraints the active set is approximate and the backward QP can return a different gradient from the exact KKT system.","fun_headline_variants_meta":{"raw":{"variants":["Backward pass as QP: 21x faster optimization-layer training","Turn backprop into a QP to cut training time 21x","BPQP: solve backward pass as QP, get 21x speedup","Optimization layers train 21x faster with BPQP's QP trick","Active-set QP backprop slashes training time by 21x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1382,"prompt_tokens":984,"completion_tokens":398,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":299}},"tokens_in":600,"tokens_out":398,"duration_ms":3618,"temperature":1.0,"reasoning_tokens":299,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:20:43.530768+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a small QP whose optimum has an inequality constraint with a dual multiplier comparable to the solver's tolerance (for example $10^{-6}$), then compute the backward gradient both by the paper's Eq. (6) and by a high-precision solution of Eq. (3). If the two disagree beyond numerical noise, or if changing the solver tolerance flips the active set and produces a discontinuous gradient, the exact-equivalence premise of Theorem 1 fails in a practically reachable regime.","supporting_citations":[{"cited_title":"Deep declarative networks","cited_arxiv_id":null,"evidence_quote":"Frames the 'Exact' baseline that computes gradients by direct KKT matrix inversion within deep declarative networks."},{"cited_title":"Differentiable convex optimization layers","cited_arxiv_id":null,"evidence_quote":"Establishes the differentiable convex optimization layer paradigm and the implicit-differentiation baseline for general cone programs."},{"cited_title":"Alternating differentiation for optimization layers","cited_arxiv_id":null,"evidence_quote":"Provides the unrolling-based baseline whose efficiency the paper compares against on QP problems."},{"cited_title":"Efficient differentiable quadratic programming layers: an admm approach","cited_arxiv_id":null,"evidence_quote":"Offers a prior ADMM-based differentiable QP layer with a custom fixed-point mapping that BPQP generalizes."}],"review_version":1}