{"id":"b756e144-6c6c-4a49-8e8f-fd73085ea5b6","arxiv_id":"2602.16137","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A binary-code experiment design with O(log n) assortments, plus a boost-factor algorithm, provably recovers substitution nests in Nested Logit models and improves choice prediction.","lead":"This paper designs a tiny set of product assortments—about log of the number of products—that reveal which products are close substitutes. It proves that under a standard choice model these assortments let a new algorithm recover hidden product groups, and tests this on over 70 million users at fantasy-sports platform Dream11.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Nest identification fails without Assumption 2; identical-parameter interleaved nests are observationally equivalent, so the unqualified 'any Nested Logit ground truth' guarantee is false.","rationale":"The reader's weakest-assumption identification is exactly right: Assumption 2 is the load-bearing condition for nest identification. The paper's abstract overstates the result by claiming correct identification under 'any Nested Logit ground truth,' omitting both Assumption 1 and Assumption 2. While Assumption 1 is a WLOG reparameterization, Assumption 2 is a genuine identifiability restriction: without it, distinct nests can be observationally equivalent, and no experiment can recover the true partition. My counterexample with equal item weights and equal dissimilarity parameters shows the failure is not a pathological measure-zero event but can arise from natural symmetric nesting structures. This does not invalidate the conditional theorem, but it strengthens the case for a CONDITIONAL verdict requiring the claims to be qualified and Assumption 2 to be stated prominently in the abstract. The reader already reached CONDITIONAL, so no verdict change is needed.","tokens_in":46291,"tokens_out":13016,"duration_ms":130813,"concrete_test":"Implement the n=4 instance above with exact choice probabilities, run Algorithm 1 as specified, and confirm it outputs one nest instead of the true two-nest partition. Also enumerate all 2^4 assortments and verify that the two label-swapped partitions yield identical choice vectors, proving non-identifiability. This check directly demonstrates that Assumption 2 is load-bearing and that the unqualified guarantee in the abstract fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 4.5's correctness is conditional on Assumption 2, but Assumption 2 is a substantive identifiability condition, not a harmless general-position clause. It fails whenever two distinct nests have equal dissimilarity parameters and equal within-nest utility sums on every experimental assortment that partially removes both. A concrete violation: n=4, nests N1={1,2}, N2={3,4}, encodings 1=00, 2=11, 3=01, 4=10, all v_i=1, lambda_N1=lambda_N2=0.5, outside option weight 1. For every S in the design, both nests are partially removed and Mult(N1,S)=Mult(N2,S)=sqrt(2), so all offered items in S have identical boost factors. Algorithm 1 sets E[i,j]=1 for every pair that co-occurs in some S, and the transitivity steps complete the graph; the output is a single nest containing all four items. Moreover, the two label-swapped nest partitions yield identical choice probabilities on every subset, so no algorithm can identify the true partition from any data. Thus Assumption 2 is not merely a generic technicality; it is exactly what makes the partition identifiable, and the abstract's claim of 'guarantees correct identification under any Nested Logit ground truth' is false without it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a non-adaptive experiment design for discrete-choice data collection that requires only O(log n) distinct assortments, based on assigning items binary encodings and offering one assortment per digit value. It reports numerical gains over randomized and other designs for a range of choice models, and develops a Nested Logit nest-identification algorithm that, under Assumptions 1 and 2, recovers the exact nest partition from exact market shares (Theorems 4.5 and 4.8). A finite-sample guarantee (Theorem 4.6) and an Ω(log n) lower bound (Theorem B.1) are also given. The paper includes a large numerical study and a deployment at Dream11.","tokens_in":46655,"tokens_out":9889,"duration_ms":97613,"significance":"The O(log n) design is elegant, the graph-reconstruction formulation is novel, and the lower bound strengthens the contribution. The main proofs are detailed and appear correct under the stated assumptions. The finite-sample theorem and the empirical comparison, including the Dream11 deployment, are valuable. However, the abstract and parts of the introduction claim an unconditional guarantee of nest identification for any Nested Logit ground truth, which is not what the theorems establish. This is a substantive overstatement rather than a cosmetic issue, and it should be corrected before publication.","major_comments":[{"comment":"The abstract states that the design and algorithm 'guarantees correct identification of nests under any Nested Logit ground truth.' This is false as written. Theorem 4.5 is explicitly conditional on Assumption 2, and Assumption 2 is a substantive condition, not a harmless general-position clause. Consider n=4 with nests {1,2} and {3,4}, encodings 1=00, 2=11, 3=01, 4=10, all v_i=1, lambda=0.5 for both nests, outside weight 1. For every experimental assortment in the design, both nests are partially removed and Mult(N,S)=Mult(N',S)=sqrt(2), so all offered items have identical boost factors. Algorithm 1 then sets E[i,j]=1 for every pair that co-occurs in some assortment, and the transitivity steps merge all four items into one nest. The output is wrong, and since exact market shares are assumed, no amount of data fixes it. The abstract and the 'guaranteed for any Nested Logit ground truth'","section":"Abstract and §1.2, Theorem 4.5"},{"comment":"Assumption 2 is stated in terms of unobserved multipliers Mult(N,S) and is not implied by the Nested Logit functional form or by Assumption 1. The paper calls it a 'general position' assumption but provides no formal genericity statement (e.g., that the set of parameters violating it has measure zero under a natural prior), nor a data-based check. The counterexample in the previous comment shows that when it fails, a plausible and simple Nested Logit model yields incorrect output with no warning from the algorithm. For an identification claim, the paper should either prove a genericity result, or explicitly characterize Assumption 2 as part of the model class being identified, and state the consequences when it fails.","section":"§4.1, Assumption 2"},{"comment":"The no-outside-option theorem allows Algorithm 2 to output 1 for pairs of singleton nests, and the proof acknowledges that the resulting matrix may be non-transitive. This is acceptable for reconstructing the choice function if lambda for merged singletons is recovered as 1, but the paper's headline statements about 'correct nest identification' are again stronger than what is proved. The final paragraph of the proof says 'we can arbitrarily divide' the violating items into nests; this should be stated as part of the theorem's conclusion, and the abstract/introduction should not imply that the exact partition is always recovered in the no-outside-option case.","section":"§4.4, Theorem 4.8"}],"minor_comments":[{"comment":"The column headers are beverage icons that do not render in the text; label each column with its explicit binary encoding and item name.","section":"Table 2"},{"comment":"The z-statistic formula is typeset in a dense, hard-to-parse way. Rewrite it using standard two-proportion z-test notation with n_1, n_2, and pooled variance.","section":"Eq. (4)"},{"comment":"The text says '20 days in Spring 2025' but later specifies May 20, 2025 to June 10, 2025, which is 22 days. Please make the dates consistent.","section":"§7"},{"comment":"The constant C is said to be 'absolute' but the proof requires C>8+8√2; state the explicit condition when the constant is introduced.","section":"§B.3, proof of Theorem 4.6"}],"recommendation":"major_revision","confidential_remarks":"The proofs of the main theorems appear correct under their assumptions; the problem is that the paper's central advertised claim is materially stronger than what is proved. A revision that qualifies the guarantee and discusses Assumption 2 would make the contribution sound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is genuinely good: encode each item in binary, run one assortment per bit, and read nest structure off boost factors. The O(log n) non-adaptive design is new, the matching lower bound is clean, and the finite-sample bound looks correct. The Dream11 deployment is a rare and valuable proof-of-concept.\n\nBut the main theorem is not “any Nested Logit ground truth” as the abstract says. Theorem 4.5 is conditional on Assumption 2, and Assumption 2 is a substantive identifiability condition, not a harmless general-position clause. When two nests have identical dissimilarity parameters and identical sums of preference weights on every partial intersection, the boosts are identical and the data cannot distinguish the true partition from a label-swapped one. Concretely, take n=4, nests {1,2} and {3,4}, all v_i=1, lambda=0.5, encodings 1=00, 2=11, 3=01, 4=10. For every experimental assortment both nests are partially removed, Mult(N_1,S)=Mult(N_2,S)=sqrt(2), so all offered items share the same boost factor. Algorithm 1 outputs one nest of four, and no other algorithm can do better. The unqualified guarantee is false.\n\nThis is fixable. State the result as holding for generic ground truths satisfying Assumption 2, and either prove a genericity statement or discuss how to test the assumption from data. The simulations draw lambda uniformly from [0.3, 0.6], so exact ties are unlikely, but the paper never quantifies violation frequency. The d-level extension is only sketched, and no code or data are released, so the empirical claims are hard to verify independently.\n\nAll that said, the combinatorial design is a real contribution and the nest-identification perspective is new. With a careful rewrite of the abstract and a more honest treatment of Assumption 2, this can be a useful paper. It deserves a serious referee: yes, but the referee should push on the identifiability issue. I would bring it to a reading group and would cite it for the design idea, not for the unqualified recovery guarantee.","headline":"Clever O(log n) design for nest identification, but Assumption 2 is doing real identifiability work and the abstract overclaims.","tokens_in":47050,"tokens_out":2536,"would_cite":true,"duration_ms":24592,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62K05","62H30","91B42"],"pacs":[],"model":"deepseek-v4-flash","headline":"A deliberately chosen set of O(log n) assortments is enough to exactly recover the nest structure of any Nested Logit choice model, and the same design improves estimation across a broad benchmark.","keywords":["experiment design","choice models","Nested Logit","nest identification","boost factors","binary encoding","assortment optimization","market shares"],"falsifier":"Simulate a Nested Logit ground truth with two nests of equal total preference weight and equal dissimilarity parameter, using an encoding where both nests are partially removed by the same assortment and all boost factors coincide; with exact market shares, Algorithm 1 will output a single merged nest, showing the theorem's assumption is doing the work.","tokens_in":46167,"feed_emoji":"🧪","tokens_out":5869,"duration_ms":52975,"temperature":0.7,"pith_summary":"The paper tries to establish that the choice of experimental assortments—not just the estimation method—is a first-order determinant of how well a choice model can be learned from data. It proposes a combinatorial design that assigns each of n items a unique binary code and, for each digit position, offers two assortments: all items whose bit is 1 and all items whose bit is 0. With only 2⌈log2 n⌉ experimental assortments plus a control, the design guarantees exact recovery of the nest partition of any Nested Logit ground truth, provided exact market shares and a general-position condition on the model parameters. The same design consistently reduces estimation error relative to randomized assortments across a benchmark of choice models, and it was used in a large-scale field deployment on a fantasy sports platform.","feed_headline":"Just O(log n) menus can expose hidden choice nests","feed_subtitle":"A binary-encoding design plus boost-factor comparisons recovers Nested Logit substitution groups and beats randomized testing.","key_machinery":"The load-bearing object is the boost factor BF(i,S)=φ(i,S)/φ(i,[n]), which under Nested Logit factors as a nest-dependent multiplier Mult(N,S) times a common outside-option boost. Because the multiplier does not depend on which item inside a nest is considered, comparing boosts across items in an assortment yields same/different nest relations; the general-position assumption makes these comparisons two-way. Algorithm 1 stores these relations in an adjacency matrix of unknown entries and completes it with transitivity and clique-completion rules, yielding the exact partition.","core_discovery":"The central theorem is that, under Nested Logit with an outside option, exact observation of market shares for these O(log n) assortments lets Algorithm 1 return the true partition of items into nests. The argument compares, for each item in each experimental assortment, the ratio of its choice probability to its probability in the full assortment—the boost factor. Items in the same nest have identical boost factors; by a general-position assumption, unequal nests that are partially removed show unequal boosts, and items whose boost equals the outside option's boost are separated from all unavailable items. The paper proves correctness of the deduction steps, including one-hop transitivity a","pith_inferences":["If Assumption 2 fails on the true parameter values, the exact-recovery guarantee silently breaks: two nests with equal multipliers on every separating assortment are merged. The paper does not provide a test for this condition, so in deployments one must rely on the noisy-data version, whose community-detection step has no comparable guarantee.","The paper's numerical advantage for non-Nested-Logit choice models is not explained by theory; a plausible inference is that half-sized, balanced assortments maximize information per observation, but the authors do not prove this. A direct test would be to compare variance of estimates under balanced versus randomized designs at fixed budget and fixed number of menus.","Appendix D sketches deeper nesting trees but only proves the bottom two levels and defers higher levels to an omitted algebraic uniqueness argument; a reader applying this to multi-level nesting should treat that part as a roadmap rather than a theorem.","A testable deployment extension: use the learned nests to re-run the same 14-menu design on a different sports season; if nests are stable across seasons, the boost-factor signature of substitution is a persistent behavioral feature rather than an artifact of one product set."],"forward_implications":["A firm can identify substitution groups among n items using only O(log n) carefully balanced test menus rather than many random assortments, so experimentation can run in parallel or over short horizons.","Data-driven nests can replace ex-ante judgment and fixed nests in Nested Logit estimation; after nests are identified, the remaining parameters are identifiable from the same O(log n) data under a non-degeneracy condition.","The Ω(log n) lower bound means no adaptive procedure can do better in the worst case, so the design is order-optimal for this identification problem.","In well-specified settings, the design can beat even individualized random assortments, and in mis-specified settings it improves estimation for small data sizes.","The field deployment showed that Nested Logit with learned nests can beat both MNL and feature-based clustering out of sample."],"fun_headline_variants":["O(log n) menus expose hidden choice nests","Binary-encoded assortments reveal latent nests","Boost ratios in few menus uncover item nests","One O(log n) assay set identifies Nested Logit nests"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is the general-position assumption that two distinct nests never have exactly the same boost multiplier on any experimental assortment that partially removes both; if that coincidence occurs, the algorithm merges distinct nests.","fun_headline_variants_meta":{"raw":{"variants":["O(log n) menus expose hidden choice nests","Binary-encoded assortments reveal latent nests","Boost ratios in few menus uncover item nests","One O(log n) assay set identifies Nested Logit nests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000478,"raw_usage":{"total_tokens":2195,"prompt_tokens":728,"completion_tokens":1467,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":1406}},"tokens_in":472,"tokens_out":1467,"duration_ms":10195,"temperature":1.0,"reasoning_tokens":1406,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T22:38:13.277902+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate a Nested Logit ground truth with two nests of equal total preference weight and equal dissimilarity parameter, using an encoding where both nests are partially removed by the same assortment and all boost factors coincide; with exact market shares, Algorithm 1 will output a single merged nest, showing the theorem's assumption is doing the work.","supporting_citations":[],"review_version":1}