REVIEW 4 major objections 5 minor 12 references
Efficient and Scalable Neural Symbolic Search for Knowledge Graph Complex Query Answering
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read NLISA achieves near-FIT accuracy at a tenth of the search cost by replacing full variable domains with top-k neural indices and solving the cyclic core by local search.
desk verdict Solid top-k pruning idea with strong empirical results; the recall assumption is worth probing, and the speedup claims need better support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Neural Logical Index, $I(x) = \operatorname{Topk}(h(G_S^x), k)$, a map from each query variable to a small candidate-entity set produced by applying a neural ranking function $h$ to the variable's constraint subgraph $G_S^x$. The local strategy computes the index from only the relations adjacent to the variable, using a hypernetwork-adapted relation-tail predictor; the global strategy uses a query embedding model over the whole query. A second mechanism is the local-search step, which fixes the free-variable candidate and greedily assigns the remaining variables by maximizing the t-norm truth value of their local constraints, giving $O(|D_x|^2+|D_x||D_y|)$ work per candidate. Together the index (the $CUTDOMAIN$ operation) and the local optimizer replace full-domain enumeration and carry the paper's complexity claims.
What would settle it
Construct a benchmark case where the correct intermediate entity is ranked just below $k$ by the neural index while the precise FIT method assigns it high fuzzy truth; run NLISA at that $k$ and check whether the final MRR drops specifically because that entity cannot be reached. If the loss appears and cannot be explained by the local search on a full domain, the top-$k$ cut is falsified as a universally safe reduction.
Extended reading notes
Core claim
The paper's claim is that precise search over the full entity set is unnecessary: every variable in an EFO1 query can be assigned a reduced domain $D_x = I(x) = \operatorname{Topk}(h(G_S^x), k)$, where $h$ ranks entities by how well they satisfy the constraints in the variable's neighborhood (local strategy) or the whole query graph (global strategy). Working inside these domains changes the cost of node removal from $O(|E|^2)$ to $O(|D_x|^2+|D_x||D_y|)$ per operation, and the remaining cyclic core is handled by greedily optimizing variable assignments from the free variable outward, a quadratic parallel local search rather than exponential enumeration. Empirically the paper shows NLISA reaches roughly 97% of FIT's MRR on tree queries with a $10\times$ speedup using 10% of the search space, and on cyclic Real EFO1 queries it keeps about 95% of FIT's performance. The named contribution is that general EFO1 complex query answering, including NP-hard cyclic instances, becomes tractable under data and query complexity that are both quadratic in the reduced domain.
Load-bearing premise
The argument depends on the unproven assumption that every entity that could contribute to a correct answer is already inside the short top-k candidate list the neural index returns, because anything outside that list is never reconsidered.
Editorial extensions
If this is right
- Symbolic complex query answering can be run on knowledge graphs with hundreds of thousands of entities, where exact methods like FIT exhaust memory.
- Cyclic EFO1 queries, previously exponential via variable enumeration, are answerable in quadratic time at near-baseline accuracy.
- The same precise node-removal pipeline can be kept for acyclic tree queries, so the method is an accelerator that preserves exactness where possible rather than a wholesale replacement.
- The search-space fraction $|D|/|E|$ becomes an explicit accuracy-efficiency knob: at 10% it gives roughly 97% relative MRR, and users can trade accuracy for speed by changing $k$.
- Because the reduced-domain operations are parallelizable and cache-friendly, the method's queries-per-second approaches query-embedding models while retaining symbolic interpretability.
Reading between the lines
- The top-k recall assumption could be tested directly: run FIT on a sample of queries, form its exact fuzzy vectors, and measure how often the true intermediate entity falls outside the neural index at various $k$; that would isolate pruning loss from search loss.
- The same domain-cutting recipe transfers to any learned-scoring constraint satisfaction problem: if a neural scorer can rank candidates for each variable, the search can be restricted to top-$k$ sets without changing the search algorithm.
- NLISA's global variant is only as strong as the query embedding model used to build the index; improving query embeddings would improve NLISA(Global) independently of the search machinery, a claim the paper does not test.
- The paper's appendix notes that the parallel caching scheme for the training-fact mask cannot mark observed edges in parallel and may cause a small performance drop; a fair reproduction should treat that as an implementation caveat when comparing against FIT's exact mask.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes NLISA, a neural-symbolic search framework for answering existential first-order logic (EFO1) queries over knowledge graphs. It computes neural logical indices that restrict each variable's search domain to a top-k candidate set, using either local constraints (relation-tail prediction via a hypernet) or global constraints (query embeddings, specifically LMPNN). For cyclic queries, it replaces the exact enumeration used by FIT with a greedy autoregressive local search. The authors claim this reduces data complexity from O(|E|^2) to O(|D|^2), makes cyclic queries tractable in quadratic time, cuts computational load by 90%, and maintains close to the MRR of the precise symbolic baseline FIT across tree-form and cyclic benchmarks, including on a 400K-entity graph.
Significance. If the claims hold, the paper offers a practical scalability recipe for neural-symbolic CQA: prune the search space aggressively with learned indices, then solve the remaining optimization with a parallel local search. The experiments are comprehensive in coverage (three KG sizes, tree-form and cyclic queries), the code is provided, and I do not see a direct scoring circularity in the design: final answer scores are computed from KG embedding truth values, not from the models used to build the reduced domains. The main value is conditional, however, on the recall of the top-k domains and on an honest accounting of the index-construction cost, neither of which is currently established. The approximate cyclic-query solver is also validated only through aggregate MRR, so its behavior on individual queries remains opaque.
major comments (4)
- [Section 3.1, Eq. (4)] The central premise is that I(x)=Topk(h(GS_x), k) contains the true entities for every variable, but this premise is never directly tested. Because Eq. (4) restricts the optimization to xi in Di, any true witness that falls outside the top-k set is irrecoverable in later search steps. The paper reports only final MRR, which can remain high even when existential witnesses are pruned: the t-norm aggregation in Eq. (4) tolerates low-probability partial assignments, and the free-variable ranking may still place the true answer near the top. This concern is most acute in Section 6.3, where |D| is only 2% of |E| on FB400K. Please report recall@k of the neural logical indices separately for existential and free variables, and separately for cyclic and acyclic queries, ideally restricted to hard answers, so the reader can verify that the reduced domains preserve the true witnesses.
- [Section 4.3, Algorithm 1] The stated time complexity O((|Dx||Dy| + |Ex|^2 + |Dx| + |Dy|)d) covers node removal and local optimization, but it omits the CUTDOMAIN step. Computing I(x) for each variable requires ranking entities by h(GS_x), which costs at least O(|E|) per variable under the scoring functions described; the paper does not describe an approximate-nearest-neighbor or precomputed-index scheme. If indices are built per query, the end-to-end complexity still contains an |E| term, and if they are precomputed offline, the storage and update costs must be stated. The symbol |Ex| is undefined. Please provide an end-to-end complexity statement and an empirical runtime breakdown that includes index construction, since this is load-bearing for the scalability claim.
- [Section 6.1, Table 2] The headline speedups in the abstract (10x) and introduction (13x) are not supported by the efficiency table. From Table 2, the QPS ratios relative to FIT are approximately 3.7x/2.8x on FB15k-237 (local/global), 5.5x/4.0x on FB15K, and 11.7x/6.7x on NELL; no configuration reaches 13x, and only NELL-local reaches roughly 10x. Please state exactly which configuration and experimental setup produce the claimed 10x/13x speedups, and report total end-to-end time including CUTDOMAIN rather than only the search phase. This matters because the abstract's '90% computational load reduction' is not by itself a wall-clock speedup claim.
- [Section 4.2, Eq. (5)] The greedy autoregressive local search is the only mechanism for escaping the NP-hard enumeration of cyclic queries, yet its approximation quality is validated only indirectly through aggregate MRR. I recommend a direct validation: on a subset of cyclic queries with small reduced domains, compare the greedy assignment's truth value and final answer rank against exact enumeration, and report the fraction of queries where the greedy solution matches the exact optimum. This would substantiate the claim that cyclic queries are approximately solved in quadratic time rather than merely reporting comparable average MRR.
minor comments (5)
- [Section 6.2] The text contains an unresolved reference 'As shown in Fig. ??'; the efficiency figure for Real EFO1 queries is missing and should be included.
- [Sections 4.1 and 4.3] The notation contains typos and undefined symbols: |Ex| appears in the complexity formula without definition, and the expression |Dy|×|D_y| in Section 4.1 appears malformed. Also, 'QSP' in Section 6.1 should be 'QPS'.
- [Appendix C] The global constraints strategy is described in only two short paragraphs; please provide the actual algorithm for treating existential variables as free variables with LMPNN, including how negated edges and cyclic query graphs are handled.
- [Table 5] The hypernet MRR values of 0.99-1.0 on relation tail prediction are surprisingly high; please describe the evaluation protocol (filtered vs unfiltered, candidate set size) and explain why the task is nearly saturated, since this is important for interpreting the quality of the local-constraint indices.
- [Section 7] The conclusion says 'with a 10% reduced search domain, our method achieves 90% performance'; this phrasing conflates search-space reduction with performance retention. Please reword to distinguish 10% of the search space from the reported relative MRR values, and reconcile the 97% claim in the introduction with the 90% wording here.
Circularity Check
No significant circularity: the answer set is produced by symbolic search over pruned domains and is evaluated against external hard-answer labels; the pruning is an approximation, not a by-construction restatement of the output.
full rationale
The derivation chain is self-contained and non-circular. NLISA's candidate domains are computed by separate predictors (a hypernet-based relation-tail predictor for local constraints, LMPNN for global constraints), while the final truth values and answer rankings are obtained by FIT-style symbolic node removal and local optimization over those domains using KG-embedding scores. Equation (3) is the exact optimization over all entities; Equation (4) restricts it to D_i, where D_i = Topk(h(GS_x),k) is an explicit approximation. The paper never asserts that the restricted optimum equals the unrestricted optimum; it measures final MRR against benchmark hard answers, so the central performance claim is empirically falsifiable. The efficiency claim (10% domain, roughly 90% load reduction) is a direct consequence of the chosen domain size and the stated O(|D|^2) complexity, but it is presented as a design choice ('we set |Dx| = |Dy| as approximately 10%'), not as a predicted outcome derived from the model. The only genuine concern—that NLI top-k recall is never directly measured and could silently drop true witnesses—is a correctness and verification risk, not circularity: no output variable is defined in terms of the target answer set, and no fitted parameter is renamed as a prediction. Self-citations to FIT/Yin et al. (2024) for node-removal proofs and to LMPNN/Wang et al. (2023b) for global constraints are citations to separate published systems; the present paper's contribution (domain pruning plus parallel local search with quadratic complexity) is evaluated against those baselines on external benchmarks rather than reduced to them.
Assumptions & free parameters
free parameters (2)
- Domain size k (|Dx|, |Dy|) =
2000 for FB15k-237 and FB15k, 6000 for NELL, 8000 for FB400K
- Hypernet hidden dimension =
Searched over [100, 200, 400]
assumptions (4)
- domain assumption Fuzzy logic t-norms (product, Godel) correctly model conjunction, disjunction, and existential quantification over neural truth values.
- ad hoc to paper Top-k pruning preserves recall: the highest-scoring entities from the Neural Logical Index contain the true answers.
- ad hoc to paper Greedy autoregressive local search finds near-optimal joint assignments for cyclic queries.
- domain assumption Knowledge graph embeddings (ComplEx) generalize to unobserved triples, so Pr(a,b) is meaningful for missing facts.
Cite this review
Pith. "Pith review of Efficient and Scalable Neural Symbolic Search for Knowledge Graph Complex Query Answering." pith.science (2026). https://pith.science/paper/YDE2UOAX
@misc{pith2026250508155,
author = {Pith},
title = {Pith review of: Efficient and Scalable Neural Symbolic Search for Knowledge Graph Complex Query Answering},
year = {2026},
howpublished = {\url{https://pith.science/paper/YDE2UOAX}},
note = {Machine review of arXiv:2505.08155}
}
abstract
Complex Query Answering (CQA) is a crucial reasoning task over Knowledge Graphs (KGs), which aims to answer first-order logical queries from incomplete KGs. While existing neural-symbolic methods achieve strong performance, they face significant complexity bottlenecks: quadratic data complexity scaling with the number of entities, and NP-hard query complexity for cyclic queries. Consequently, these approaches struggle to scale effectively to large knowledge graphs and complex queries. To address these limitations, we propose an efficient and scalable symbolic search method comprising two key components: (1) constraint strategies that drastically reduce the variable search domain, lowering data complexity; and (2) a local search algorithm that approximately solves NP-hard cyclic queries. Experiments on various CQA benchmarks demonstrate that, for tree-form queries, our method achieves 97% relative MRR with a 10$\times$ speedup using only 10% of the search space. Furthermore, it demonstrates robust performance on complex cyclic queries and large-scale KGs, effectively alleviating efficiency and scalability challenges. Our code is provided in https://github.com/HKUST-KnowComp/NLISA_KDD2026.
Figures
Reference graph
Works this paper leans on
-
[1]
Arakelyan, E., Daza, D., Minervini, P., and Cochez, M. (2020). Complex Query Answering with Neural Link Predictors. In International Conference on Learning Representations. Arakelyan, E., Minervini, P., and Augenstein, I. (2023). Adapting Neural Link Predictors for Complex Query Answering. arXiv:2301.12313 [cs]. Bai, J., Liu, X., Wang, W., Luo, C., and So...
arXiv 2020
-
[3]
Xiong, B., Potyka, N., Tran, T.-K., Nayyeri, M., and Staab, S. (2022). Faithful Embeddings for $$$\backslash$mathcal ${$e$}$$\backslash$mathcal ${$l$}$^${$++$}$ $$ Knowledge Bases. In International Semantic Web Conference, pages 22–38. Springer. Yang, D., Qing, P., Li, Y ., Lu, H., and Lin, X. (2022). GammaE: Gamma Embeddings for Logical Queries on Knowle...
arXiv 2022
-
[4]
Chen, Y ., Minervini, P., Riedel, S., and Stenetorp, P. (2021). Relation prediction as an auxiliary training objective for improving multi-relational graph representations. In 3rd Conference on Automated Knowledge Base Construction. Choudhary, N., Rao, N., Katariya, S., Subbian, K., and Reddy, C. (2021). Probabilistic entity representation model for reaso...
work page 2021
-
[5]
Wang, Z., Yin, H., and Song, Y . (2022). Logical Queries on Knowledge Graphs: Emerging Interface of Incomplete Relational Data. Data Engineering, page
work page 2022
-
[11]
for addressing a given cyclic query, while the bottom section illustrates the pipeline of NLISA tackling the same cyclic query. NLISA maintains a smaller fuzzy vector by preserving only the indices within the relevant domains, resulting in greater flexibility. Additionally, NLISA employs Local Optimization to search for assignments for each candidate of f...
work page 2024
-
[12]
in Fig. 3 and Fig. 4, where the visualization of BetaE benchmark is taken from Wang et al. (2023a) and the visualization of Real EFO1 benchmark is taken from Yin et al. (2024). 14 3P1P2P 2I3I 2IN3ININPPNIPIN PI IP 2U UPPIUN Figure 3: 14 query types propsed in BetaE benchmark (Ren and Leskovec, 2020). These query types are modeled by the operator tree. Fig...
work page 2023
-
[26]
Chen, H., Dalmau, V ., and Grußien, B
Curran Associates, Inc. Chen, H., Dalmau, V ., and Grußien, B. (2011). Arc consistency and friends. J. Log. Comput. , 23:87–108. 9 Chen, X., Hu, Z., and Sun, Y . (2022). Fuzzy logic based logical query answering on knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 3939–3948. Issue:
work page 2011
-
[31]
Hu, W., Fey, M., Zitnik, M., Dong, Y ., Ren, H., Liu, B., Catasta, M., and Leskovec, J. (2020). Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118–22133. Ji, S., Pan, S., Cambria, E., Marttinen, P., and Philip, S. Y . (2021). A survey on knowledge graphs: Representation, acquisition, ...
arXiv 2020
Show all 12 references
-
[2018]
Recently, Tsang et al
to Existential Positive First-Order (EPFO) queries (Ren et al., 2020), Existential First- Order (EFO) queries (Ren and Leskovec, 2020), real Existential First-Order queries (Yin et al., 2024). Recently, Tsang et al. (2025) developed query embeddings on knowledge hypergraphs. T...
2025
-
[2022]
In particular, the neural symbolic search method combining the knowledge graph embedding with symbolic search is of particular interest
represents the intermediate variable as a fuzzy vector, and simultaneously adapts graph neural network from KG completion to execute relation projection and models the logical operations with fuzzy logic. In particular, the neural symbolic search method combining the knowledge...
2020
-
[2023]
CQD-CO uses gradient optimization to estimate the embedding of existential variables (Arakelyan et al., 2020)
uses beam search during the execution of operator tree, maintaining only a beam width of entities for intermediate variables. CQD-CO uses gradient optimization to estimate the embedding of existential variables (Arakelyan et al., 2020). Unlike previous approximate search metho...
2020
-
[2024]
However, the complexity is|E|n (Yin et al., 2024), where n is the variable number of query
extends its scope to general EFO1 by using the enumeration to handle cyclic queries. However, the complexity is|E|n (Yin et al., 2024), where n is the variable number of query. In general, the above two precise methods constantly remove nodes and preserve results with fuzzy ve...
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.