{"id":"6e01dbf5-e6fd-4648-85f7-afebb993c60b","arxiv_id":"2509.07789","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A systematic benchmark of filtered nearest-neighbor search algorithms shows no single winner: filter-then-search methods excel at containment and equality filters, while hybrid methods dominate overlap filters.","lead":"This paper benchmarks nine algorithms for filtered approximate nearest neighbor search, which combines vector similarity with metadata filtering. It proposes a tuning framework and finds that the best algorithm depends on filter type, query selectivity, and recall target.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tuning protocol: parameters selected on a small unvalidated sample and reused on the same query set for evaluation; if this overfits or fails to transfer to full indexes, reported QPS-recall frontiers are not intrinsic algorithm trade-offs.","rationale":"The reader's verdict is CONDITIONAL, and this stress-test agrees. The benchmark's novelty—unified tuning and standardized filtering—is also its point of maximum risk. The paper provides no evidence that parameters tuned on small samples are optimal for full indexes, and reusing the same query set for both tuning and evaluation can systematically favor configurations that overfit. Because the headline results (e.g., UNG's dominance in containment/equality, ACORN/DiskANN for overlap) drive the recommendations in Section 6, a failure of either assumption would change the paper's practical conclusions. The proposed check isolates this by comparing rankings under held-out queries and larger samples. If the rankings are stable, the concern is resolved and conditional acceptance stands; if not, the experimental conclusions need revision. The open-source code and datasets are valuable and should be credited, but they do not by themselves validate the transfer assumption.","tokens_in":17791,"tokens_out":4069,"duration_ms":44401,"concrete_test":"On arXiv and YFCC, for UNG, ACORN-1, Filtered-DiskANN, and Post-filter HNSW, split the query set into disjoint tuning and evaluation halves. Run Algorithm 1 with D' = 50% of the dataset and Q_tuning, then build full indexes with the selected parameters and measure QPS-recall on Q_eval at recall targets {0.8, 0.9, 0.95}. Also run a control with D' = 100% and Q_tuning. If the per-scenario Pareto ordering changes vs. the paper's Figure 9, or if the 50%-sample and 100%-sample rankings disagree, the tuning protocol does not support the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section 1.2) is that the framework 'isolates and accurately reflect[s] the fundamental performance characteristics and core trade-offs of each algorithmic paradigm.' This rests on Algorithm 1 producing representative parameters that generalize. Algorithm 1 samples a subset D' (line 5), evaluates every parameter θ on the full query set Q (line 10), interpolates QPS to fixed recall targets (line 13), and selects one representative per subspace; the selected θ is then used to build full indexes and evaluate on the same Q. Two load-bearing assumptions are unvalidated: (i) parameters optimized for a small random sub-index remain near-optimal for a full-size index—graph-based methods (HNSW M/ef, DiskANN R/L, ACORN γ) have size-dependent optimal values, and the paper reports no sample size or transfer checks; (ii) using the same Q for tuning and evaluation permits overfitting to the query distribution, inflating the QPS-recall frontier of the configuration that best matches Q. The paper also reports no variance/error bars, so stability of the ordering (UNG for containment/equality, ACORN/DiskANN for overlap) cannot be assessed. These issues are fixable, but until addressed the rankings should not be read as intrinsic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a benchmark and systematic experimental study of Filtered Approximate Nearest Neighbor Search (FANNS). It proposes a taxonomy (filter-then-search, search-then-filter, hybrid-search), a parameter tuning framework (Algorithm 1) that samples a subset of the dataset, sweeps parameter configurations, interpolates QPS at fixed recall targets, and selects representative parameters per subspace. It evaluates 9 algorithms on 6 real-world datasets across four filter scenarios (containment, equality, overlap, fixed-length equality), tests effects of query label length, selectivity, top-k, and dataset size, and provides recommendations. The central claim is that the framework isolates and accurately reflects fundamental performance characteristics and core trade-offs of each algorithmic paradigm. The paper also releases code and datasets.","tokens_in":18187,"tokens_out":1834,"duration_ms":17673,"significance":"If the central claim holds, this would be a valuable reference for FANNS algorithm selection and a standardized benchmark. The scope is substantial: 41,000 parameter combinations, ~1,300 full indexes, 6 datasets, multiple scenarios. The paper includes machine-checkable artifacts (open-source code), a clear taxonomy, and a structured tuning protocol, which are strengths. However, the significance depends on the validity of the tuning/evaluation protocol, which has potential issues (tuning on a sample and using the same query set for evaluation, no variance reporting).","major_comments":[{"comment":"The tuning protocol uses a randomly sampled subset D' to select parameters, but the selected parameters are then used to build full indexes and evaluate on the same query set Q with no validation on a held-out portion of the data. Since Algorithm 1 explicitly averages over the full query set Q (line 10) and selects the best-performing configuration per subspace (line 16), the reported QPS-recall frontiers may reflect overfitting to the tuning queries. The paper provides no transfer check (e.g., comparing tuned parameters on D' vs. full D, or evaluating on a disjoint query set) and no variance/error bars. This is load-bearing because the central claim in Section 1.2 is that the framework 'isolates and accurately reflect[s] the fundamental performance characteristics and core trade-offs of each algorithmic paradigm.' I request either (a) a held-out query set for final evaluation, (b) a dem","section":"Algorithm 1 (lines 5-16) and Section 5.3"},{"comment":"The transfer of parameters tuned on a small random sample D' to full-size indexes is assumed but not validated. Graph-based methods (HNSW M/ef, DiskANN R/L, ACORN γ) and UNG have size-dependent optimal parameters; the paper does not report the sample size |D'| for each dataset or any experiment showing that the relative ranking of configurations is preserved between D' and the full dataset. Without such a check, the comparisons may reflect tuning artifacts rather than intrinsic algorithm trade-offs. Please provide the sample sizes and a correlation/stability analysis, or justify the assumption with evidence from the 1,300 full-index builds.","section":"Algorithm 1 and Section 5.2"},{"comment":"No error bars, multiple runs, or statistical significance tests are reported for any QPS or recall measurement. Since the paper makes comparative claims (e.g., 'UNG is well-suited for containment and equality', 'DiskANN performs well in overlap'), the stability of the rankings is unknown. I am not asking for full repetition of all 41,000 configurations, but at least for a subset (e.g., main Pareto-frontier configurations on representative datasets) to assess run-to-run variance and establish that the observed margins are not noise.","section":"Section 5 and Figure 9"}],"minor_comments":[{"comment":"The table lists parameter-space dimensions, but the specific parameter names and ranges for each algorithm are not given in the text. A supplementary table or appendix listing the subspaces and parameter grids would improve reproducibility.","section":"Section 1.1, Table 1"},{"comment":"The recall targets R are defined as a list (e.g., 0.8, 0.9, 0.95), but the interpolation method (linear in what space?) is not fully specified. Clarify whether interpolation is on raw recall/QPS or log-transformed values, and how ties are broken in ranking.","section":"Section 4, Algorithm 1"},{"comment":"The dataset sizes for TripClick and LAION1M are listed as ~1M, but for YouTube-Audio the description says '5,000,000' while later Section 5.1 text mentions 'our largest dataset is limited to 500M vectors' — this appears to be a typo (likely 5M vs 500M). Please correct.","section":"Section 5.1, Table 3"},{"comment":"The fixed-length equality synthetic data generation says 'each position has three possible values, chosen with equal probability' and selectivity is 1/81, but with 4 positions and 3 values there are 3^4=81 combinations, each with probability (1/3)^4=1/81. This is consistent, but the sentence 'we followed the implementation in NHQ paper' should cite a specific section or hyperparameter (e.g., number of labels per point).","section":"Section 5.4"},{"comment":"The selectivity groups are described as '75th, 50th, 25th, and 1st percentiles' — this is ambiguous. Do these refer to percentiles of the selectivity distribution? Clarify the direction (e.g., 'low selectivity' = 1st percentile) and how the query groups are constructed.","section":"Section 5.6"},{"comment":"The recommendations are useful but some are not directly tied to the figures (e.g., 'avoid post-filter IVFPQ at high k' is supported by Figure 13, but the statement about ACORN-1 and stitch-diskann in overlap could cite Figure 11 or 12).","section":"Section 6, Recommendations"}],"recommendation":"minor_revision","confidential_remarks":"The paper's tuning protocol is the main risk; the lack of held-out validation and variance reporting is concerning but fixable. Also, UNG is a method from the authors' own prior work and is a focus of the benchmark; while the evaluation includes external algorithms and is not solely self-referential, the reader should be aware of a potential conflict of interest in the framing of recommendations. The scope and artifact release are strong positives."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the most substantial FANNS benchmark I've seen, and it's worth engaging with, but the evaluation protocol has a load-bearing weakness that the authors need to fix before the rankings are taken as definitive.\n\nWhat's good: The scale is real. 41,000 parameter combinations on small subsets, ~1,300 full indexes over six real-world datasets and four filter predicates—that's a serious amount of work. The taxonomy (filter-then-search, search-then-filter, hybrid) is not new, but the empirical grounding is. The concrete guidance (UNG for containment/equality, ACORN/DiskANN for overlap, UNG/CAPS for fixed-length equality) is useful, and the artifact release at GitHub is a plus.\n\nThe soft spot is exactly what the stress-test flags. Algorithm 1 samples a subset D' to tune parameters, evaluates each configuration on the full query set Q, then builds full indexes and evaluates on the same Q. So the query set used for parameter selection is the same one used for the final measurements. That's using the test data to pick parameters, and it can inflate the QPS-recall frontier for whichever configuration happens to match Q's distribution. On top of that, there are no error bars or repeated runs, so we have no idea whether the differences between algorithms are within noise. And the transfer from a small sub-index to the full index is never validated—graph-based methods often have size-dependent optimal parameters, and the paper doesn't report the sample size D' or any check that the tuned parameters remain near-optimal on the full index.\n\nThe authors overclaim in Section 1.2 when they say the framework 'isolates and accurately reflects fundamental performance characteristics.' That's not supported by the current protocol. But the flaws are fixable: hold out a separate query set for tuning, report variance across runs, and validate sample-to-full parameter transfer. The rankings are probably directionally right—the results are consistent across datasets and scenarios—but the precise frontiers are not reliable yet.\n\nThere's also a minor concern: the authors are benchmarking their own UNG, and they extended DiskANN implementations themselves. They didn't alter graph construction, and they include brute-force baselines, so this isn't a red flag, but it's another reason to want the protocol tightened.\n\nWho should read this: anyone choosing a FANNS algorithm in practice, and researchers working on vector search benchmarks. It deserves a serious referee, and I'd accept it with the expectation of major revisions addressing the tuning and variance issues.","headline":"The most substantial FANNS benchmark I've seen, worth citing and fixing, but the tuning protocol reuses test queries and lacks variance, so don't read the rankings as intrinsic yet.","tokens_in":18568,"tokens_out":4057,"would_cite":true,"duration_ms":43515,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a unified tuning and filtering protocol can make filtered nearest-neighbor comparisons fair—and that on that protocol, the best method depends on the filter type: UNG for containment/equality, ACORN/DiskANN for overla","keywords":["filtered approximate nearest neighbor search","benchmark","parameter tuning","filter-then-search","search-then-filter","hybrid-search","recall-QPS trade-off","graph-based index"],"falsifier":"Re-run the benchmark with a strict train/test split: tune parameters on one random half of each dataset's queries, then evaluate on the held-out half. If the reported rankings or Pareto frontiers change materially—for example, if a method the paper ranks low reaches the same recall at higher QPS under held-out tuning—the framework is not isolating algorithm merit. A simpler check: for one dataset, compare the sample-tuned parameters against a full grid search on the entire index; identical QPS-recall frontiers would support the protocol, divergence would refute it.","tokens_in":17753,"feed_emoji":"⚖️","tokens_out":8703,"duration_ms":85581,"temperature":0.7,"pith_summary":"The paper's thesis is that existing comparisons of Filtered Approximate Nearest Neighbor Search (FANNS) methods are unreliable, because each method is tuned differently, tested on different datasets and workloads, and reported with different experimental choices. It tries to establish that a common evaluation protocol—grouping methods into filter-then-search, search-then-filter, and hybrid-search, tuning all methods with one parameter-selection procedure, and using one bitset-based filtering implementation—can isolate each paradigm's genuine speed-accuracy trade-offs. Under that protocol, the paper reports that no single method dominates: UNG wins containment and equality queries, ACORN and Stitched DiskANN lead on overlap queries, and post-filtering methods stay competitive at low k but degrade at high k. The benchmark also finds that query-label length and filter selectivity move performance more than dataset scale does. If the protocol is sound, it gives practitioners a defensible basis for choosing an algorithm per workload instead of relying on isolated published numbers.","feed_headline":"UNG beats rivals on strict filters; DiskANN variants lead on overlap","feed_subtitle":"A 41,000-parameter sweep across six datasets shows method rankings flip with filter type and selectivity.","key_machinery":"The load-bearing mechanism is the paper's unified tuning protocol (Algorithm 1): it partitions the parameter space into subspaces, builds small indexes on a random sample, scores each configuration by interpolated QPS at fixed recall targets, and selects one representative configuration per subspace for full-size evaluation. Combined with a standardized bitset-filter implementation (bitwise AND/OR over precomputed label-to-vector bitsets) and the three-way taxonomy, this pipeline turns ad-hoc hyperparameter choices into a reproducible procedure, so differences in QPS-recall frontiers can be attributed to algorithm design rather than tuning luck.","core_discovery":"The paper's central claim is that a fair, parameter-aware evaluation framework can separate intrinsic algorithm trade-offs from tuning artifacts in filtered nearest neighbor search. It groups methods into filter-then-search (UNG, ACORN, brute-force), search-then-filter (post-filter HNSW, IVFPQ), and hybrid-search (Filtered-DiskANN, Stitched-DiskANN, NHQ, CAPS), and applies one tuning protocol plus one bitset-filtering implementation across six labeled datasets and four filter semantics. It reports that the best method depends on the filter constraint: UNG dominates containment and equality because its label graph never expands into invalid nodes; ACORN-1 and Stitched-DiskANN lead on overlap,","pith_inferences":["We infer the tuning protocol could become a community standard: future FANNS papers reporting only hand-tuned numbers would be hard to compare against this benchmark, though the authors stop short of saying so.","We infer the rankings generalize only as far as the six real-world label distributions represent production workloads; heavily skewed, correlated, or hierarchical labels could shift the rankings.","Because the paper tests only in-memory indexes, we infer rankings may change on disk-resident billion-scale data, where I/O access patterns and graph reordering become first-order factors.","The framework appears compatible with numerical range filters, which the paper explicitly leaves out; we infer the same unified tuning pipeline could apply to those methods, but that is untested."],"forward_implications":["For containment and equality workloads, UNG is the recommended default; its throughput stays high even as query labels lengthen, because the label graph restricts search to a small valid subgraph.","For overlap workloads, ACORN-1 and Stitched DiskANN offer the best balance; UNG degrades as label complexity grows.","Post-filter IVFPQ is a poor choice at high k, because valid items scatter across inverted lists; prefer UNG/ACORN for containment/equality and DiskANN/ACORN for overlap at recall@100.","Dataset size is not the main pressure: going from 10% to 100% of a dataset changes QPS-recall curves only slightly; label selectivity and query-label length matter more.","Methods that need base labels during construction (UNG, DiskANN variants, NHQ, CAPS) index more slowly; ACORN-1, HNSW, and IVFPQ build quickly and tolerate label changes, an operational constraint for dynamic data."],"supporting_citations":[{"why":"Supplies the UNG algorithm and its label-navigation graph design, the method the benchmark recommends for containment and equality queries.","marker":"[3]"},{"why":"Supplies Filtered-DiskANN and Stitched-DiskANN, the hybrid-search methods that lead the benchmark's overlap scenarios.","marker":"[14]"},{"why":"Supplies ACORN-1 and ACORN-gamma, the filter-then-search methods with predicate-agnostic graph traversal and bitset pre-filtering.","marker":"[37]"},{"why":"Supplies NHQ and the fixed-length equality synthetic-label setup used in the fixed-length experiments.","marker":"[47]"},{"why":"Supplies CAPS, the hybrid partition index compared in the fixed-length equality scenario.","marker":"[16]"},{"why":"Provides HNSW, the graph used by the post-filter search-then-filter baseline and by ACORN-1's index structure.","marker":"[29]"},{"why":"Provides product quantization and the inverted-file index behind the post-filter IVFPQ baseline.","marker":"[19]"},{"why":"Supplies the evidence that vector search parameters are coupled and dataset-dependent, motivating the unified tuning procedure.","marker":"[50]"},{"why":"Provides the YouTube-Audio and YouTube-Video datasets used in scalability experiments.","marker":"[1]"},{"why":"Provides the YFCC dataset and its metadata labels, a core benchmark workload.","marker":"[46]"}],"fun_headline_variants":["FANNS rankings flip with filter type","Filter selectivity decides best ANN method","UNG: strict filter king; DiskANN: overlap champ","No one best FANNS: constraint decides","Benchmark: filter shape, not size, picks winner"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The framework's fairness rests on the assumption that parameters selected on a small random sample of each dataset, then frozen, are still near-optimal for the full index; the evaluation does not hold out a separate query set to test that assumption.","fun_headline_variants_meta":{"raw":{"variants":["FANNS rankings flip with filter type","Filter selectivity decides best ANN method","UNG: strict filter king; DiskANN: overlap champ","No one best FANNS: constraint decides","Benchmark: filter shape, not size, picks winner"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000655,"raw_usage":{"total_tokens":2882,"prompt_tokens":837,"completion_tokens":2045,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1974}},"tokens_in":581,"tokens_out":2045,"duration_ms":18216,"temperature":1.0,"reasoning_tokens":1974,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T21:39:55.487247+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark with a strict train/test split: tune parameters on one random half of each dataset's queries, then evaluate on the held-out half. If the reported rankings or Pareto frontiers change materially—for example, if a method the paper ranks low reaches the same recall at higher QPS under held-out tuning—the framework is not isolating algorithm merit. A simpler check: for one dataset, compare the sample-tuned parameters against a full grid search on the entire index; identical QPS-recall frontiers would support the protocol, divergence would refute it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies NHQ and the fixed-length equality synthetic-label setup used in the fixed-length experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the YFCC dataset and its metadata labels, a core benchmark workload."}],"review_version":1}