{"id":"114e1b68-71de-445d-bcf5-e6e9d1a1feab","arxiv_id":"2505.01810","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper applies split conformal prediction to indoor positioning on the UJIIndoLoc dataset, reporting approximate coverage, but the contributions are restatements and the experimental reporting is inconsistent.","lead":"This paper applies conformal prediction, a standard method that converts model uncertainty into prediction sets with coverage guarantees, to deep learning based indoor positioning. It reports approximate coverage on a public benchmark, but its new claims about navigation risk control and p-values are unsupported by the experiments.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The coverage guarantee in Eq. (1) is conditional on exchangeability, and the paper's own §3.1.2 admits that the targeted indoor/path-navigation settings violate it; the random-split experiments cannot validate the claimed guarantee.","rationale":"Both the reader and this pass identify the same load-bearing point: the only rigorous guarantee in the paper is conditional on exchangeability, and the paper itself concedes that exchangeability is questionable in the target scenarios. This is not a disagreement with the conformal prediction literature; it is a scope mismatch between the advertised guarantee and the conditions under which it can be applied. The random-split experiment is a valid demonstration of split conformal prediction on a fixed dataset, but it cannot certify real deployments where calibration and test come from different temporal periods, paths, or devices. The paper's own §3.1.2 note is important: no revised coverage bound is supplied when its mitigations are used. Other issues (the §3.2 risk-control proof, the absence of any FDR/FNR experiments, and Table 1's sub-unit 'set sizes') reinforce the rejection, but they are secondary to the central coverage claim. Since the reader already recommends REJECT and this concern does not change that recommendation, the verdict is unchanged.","tokens_in":12460,"tokens_out":13477,"duration_ms":140754,"concrete_test":"Re-run the UJIIndoorLoc experiments with a temporal split instead of the random 70/10/20 split in §4.1.1: order samples by collection time (or by building/floor/device for heterogeneity), calibrate on the first 10% and test on the last 20% of the time-ordered data. Compute empirical coverage of the split-conformal sets at α = 0.1. If the observed coverage is below 0.9 by more than 1/(n+1), the acknowledged drift/sequence/device violation is not just theoretical: it invalidates Eq. (1) in the claimed deployment setting. If coverage remains ≥ 0.9, the test would show the heuristics suffice on this data, but the unconditional guarantee is still not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline guarantee, Eq. (1), is a finite-sample statement that holds only if the calibration and test data are exchangeable. Section 3.1.2 explicitly identifies temporal dynamics, path-sequence correlation, and device heterogeneity as factors that can violate exchangeability, and then states that the proposed fixes (recent calibration data, windowed path segments, signal normalization) 'don't eliminate exchangeability concerns.' This is an admission that the central condition for the guarantee is not expected to hold in the very settings the paper is about: multi-device indoor deployments and path navigation. Because the mitigation strategies are heuristic, there is no quantitative bound or alternative theorem showing that coverage remains near 1−α when the calibration/test distributions differ. The experiments in §4.1.1 split the dataset randomly into 70/10/20, which by construction preserves exchangeability, so they do not test the guarantee under the acknowledged violations. Consequently the claim that the framework provides correctness coverage guarantees for practical indoor positioning is not supported; at best the paper shows coverage under a random split, which is the easy case.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies split conformal prediction (Split CP) to deep learning-based fingerprint indoor positioning. It claims a finite-sample coverage guarantee for prediction sets, proposes conformal risk control to manage FDR and FNR in path navigation, and introduces a conformal p-value framework for pointwise reliability assessment. Experiments on UJIIndoLoc with MobileNetV1, VGG19, MobileNetV2, ResNet50, and EfficientNet compare empirical coverage with target coverage and report average prediction set sizes. The basic coverage argument in Section 3.1 is a standard Split CP derivation, but the paper's extensions and experiments contain several load-bearing gaps: the exchangeability condition is acknowledged to be violated in the motivating settings but is not tested there, the FDR/FNR loss definitions do not match the quantities being controlled, the p-value proof is not rigorous with tied scores, and several quantitative claims in the abstract and contributions are not supported by the reported experiments.","tokens_in":12704,"tokens_out":7215,"duration_ms":74454,"significance":"If the stated guarantees held, applying conformal prediction to indoor positioning would be practically valuable, especially for safety-critical location-based services. The paper usefully demonstrates that a standard Split CP wrapper can be placed on top of off-the-shelf CNN classifiers and that empirical coverage tracks nominal coverage on a random split of UJIIndoLoc. However, the claimed contributions for path navigation and pointwise reliability are not established: the risk-control losses do not implement FDR/FNR as defined, the p-value proof is not valid with tied scores, and the experiments do not test the settings in which the paper itself says exchangeability fails. The manuscript therefore does not currently support its headline promise of correctness coverage guarantees in practical indoor deployments.","major_comments":[{"comment":"The paper explicitly identifies temporal dynamics, path-sequence correlation, and device heterogeneity as factors violating exchangeability, and it concedes that the proposed mitigations 'don't eliminate exchangeability concerns.' Yet the guarantee in Eq. (1) is a finite-sample statement that holds only under exchangeability. The random 70/10/20 split in §4.1.1 preserves exchangeability by construction, so the experiments do not validate the guarantee in the multi-device, time-varying path-navigation settings that motivate the paper. The authors should either restrict the central claims to static exchangeable fingerprinting or provide quantitative tests under distribution shift (e.g., time-based splits, device-held-out splits) and a robustness analysis showing how coverage degrades when exchangeability fails.","section":"§3.1.2, §4.1.1"},{"comment":"The FDR/FNR instantiations do not satisfy the conformal risk control theorem as stated. The theorem controls E[L_{n+1}(λhat)] ≤ β for the chosen loss, but the FDR defined in Eq. (13) is an expectation of a ratio over path points, not a sum of individual losses. Eq. (12) uses squared error rather than the false-discovery indicator 1{||Y_i − Ŷ_i||² > λ and P_i = 1}, and it does not include the denominator used in Eq. (13). The same mismatch appears in Eqs. (14)-(15) for FNR. Consequently, the claimed guarantees FDR(λhat_FDR) ≤ β_FDR and FNR(λhat_FNR) ≤ β_FNR do not follow from Eq. (11). The loss functions and the target risks must be aligned, or the claims must be restricted to the quantity actually controlled.","section":"§3.2.1, Eqs. (12)-(15)"},{"comment":"The p-value proof is not valid as written. The equality P(1 + Σ_i 1{s_i ≥ s(x_test, y)} ≤ ⌊(n+1)α⌋) = ⌊(n+1)α⌋/(n+1) assumes that the rank of s(x_test, y) among the calibration scores is exactly uniform, which fails when scores are tied (e.g., duplicated softmax probabilities or identical Euclidean error distances). The standard argument requires care with the direction of the inequality or randomized tie-breaking. In addition, the calibration score definition s_i = max_{y_j∈[m]_i} s(x_i, y_j) does not match the score construction used elsewhere in the paper; for the null hypothesis y = Y_test, the comparison should use the calibration score s(x_i, Y_i).","section":"§3.3, Eqs. (19)-(21)"},{"comment":"Average prediction set sizes below 1 (e.g., 0.135 for EfficientNet at α = 1) are not interpretable with the given problem setup. If a prediction set is a subset of K discrete locations, its cardinality is integer-valued; if it is a region in continuous coordinates, the paper must define its 'size' (e.g., area, number of grid cells) and explain how candidate positions are enumerated for thresholding by Euclidean distance. As presented, Table 1 and Figure 6 do not provide a well-defined uncertainty-efficiency comparison.","section":"§4.1.3, Table 1"},{"comment":"The quantile formula in Eq. (2) is inconsistent with the proof and with standard split conformal prediction. It compares the empirical count over n calibration scores to a threshold with denominator n while the left-hand side has denominator n+1; the standard finite-sample corrected quantile is the ⌈(n+1)(1−α)⌉-th smallest calibration score, which is what the proof in Eq. (7) uses. The displayed formula should be corrected, otherwise the implementation and the theoretical derivation do not match.","section":"Eq. (2), §3.1.1"},{"comment":"The abstract and contribution list contain quantitative claims — 'up to 15% improvement in positioning reliability,' 'reducing misleading navigation instructions by 23%,' and 'outperforms traditional confidence scoring methods' — that are not supported by any experiment, baseline, or comparison in the paper. The experiments only compare empirical coverage to target coverage and report prediction set sizes; there is no evaluation against Bayesian neural networks, ensembles, MC-dropout, or heuristic confidence scores, despite Section 2.3 motivating the work through those baselines.","section":"§1, §4"}],"minor_comments":[{"comment":"The text 'VGG19p Simonyan and Zisserman' appears to contain a typo; it should read 'VGG19' followed by the citation.","section":"§4.1.2"},{"comment":"The coverage proof states an exact equality P(Y_test ∈ C(X_test)) = ⌈(n+1)(1−α)⌉/(n+1). This equality, and the upper bound in Eq. (1), hold only under a continuity/no-ties condition; the lower bound is the robust statement and should be emphasized as such.","section":"§3.1, proof after Eq. (7)"},{"comment":"The notation y_j∈[m]_i is undefined; the candidate label set for each calibration point should be formally defined before it is used in the score definition.","section":"§3.3"},{"comment":"Foundational conformal prediction and conformal risk control results are cited primarily through a narrow set of recent papers by Wang et al.; the primary literature (e.g., Vovk et al., Shafer and Vovk, and Angelopoulos et al. for risk control) should be cited so that readers can verify the standard results.","section":"References"},{"comment":"The architectural analysis in Section 4.2 is speculative and not supported by ablations; statements such as 'EfficientNet... allows the model to focus on the most discriminative features' should be presented as hypotheses rather than conclusions.","section":"§4.3"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as an early draft. The basic split-CP experiment is standard and appears sound, but the paper overclaims in several directions: the risk-control losses do not control the stated FDR/FNR quantities, the p-value proof ignores ties, the exchangeability failure is acknowledged but never tested, and the abstract's quantitative improvements have no experimental support. The repeated citation of a small set of recent papers for foundational CP results, while omitting the primary literature, is also a concern. I would not encourage resubmission in the current form; substantial repair of the risk-control definitions, addition of distribution-shift experiments, and recalibration of the claims would be needed before the paper could be considered again."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Zhiyi Zhou et al. apply split conformal prediction to fingerprint-based indoor positioning, with distance-based non-conformity scores and an extension to conformal risk control for path navigation. The core coverage guarantee is correct but completely standard; the real content is the application and the empirical study.\n\nWhat is genuinely useful: the distance-based score is a sensible adaptation for spatial errors, and the comparison of MobileNetV1, VGG19, MobileNetV2, ResNet50, and EfficientNet on UJIIndoLoc gives a concrete picture of how model quality translates into prediction-set size at fixed coverage. Section 3.1.2 deserves credit for honestly stating that temporal drift, path correlation, and device heterogeneity can break exchangeability.\n\nThe soft spots are consequential. First, the FDR and FNR instantiations do not match the theory. The risk-control theorem in Section 3.2 bounds the expected value of the loss functions, but those losses are squared Euclidean errors, not the 0/1 indicators in the FDR/FNR definitions (13) and (15). The theorem therefore certifies a different quantity than the paper claims. Second, the experiments are internally inconsistent: the abstract reports training/test accuracy (100%/85%), but the method section uses MSE for coordinate prediction, and the prediction set sizes in Table 1 are not defined for continuous outputs—values below 1 suggest something else is being tracked. Third, the introduction claims a 15% reliability improvement and a 23% reduction in misleading navigation instructions, but no uncertainty-quantification baselines (BNN, MC dropout, ensembles) appear in the experiments. Fourth, the conformal-prediction references are almost entirely self-citations; Vovk, Shafer, and Angelopoulos are missing, which is noticeable in a methods paper.\n\nThe stress-test concern is on point: the random 70/10/20 split preserves exchangeability by construction, so the experiments only validate the easy case. The paper admits its proposed mitigations do not eliminate exchangeability concerns, which makes the title's \"correctness coverage guarantees\" too strong for the stated deployment scenarios.\n\nWho is this for? A researcher wanting a first, nontechnical look at how CP behaves with distance scores on a standard indoor-positioning dataset. It does not deserve a desk rejection—the application is reasonable and the flaws are fixable—but it needs major revision before it can make its claims honestly. I would send it to peer review with the expectation that the loss-function mismatch and the missing baselines are addressed.","headline":"Split conformal prediction applied to indoor positioning: the coverage proof is standard and fine, but the risk-control losses don't match the stated FDR/FNR, and the experiments don't test the paper's own exchangeability caveat.","tokens_in":13176,"tokens_out":4517,"would_cite":false,"duration_ms":43307,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under exchangeable calibration data, conformal prediction guarantees the true position is in the predicted set with probability at least $1-\\alpha$.","keywords":["conformal prediction","indoor positioning","uncertainty quantification","coverage guarantee","fingerprint localization","risk control","path navigation","deep learning"],"falsifier":"Collect calibration fingerprints at one time or with one device, then test on fingerprints taken later or with a different device; if the empirical coverage over many test points falls materially below $1-\\alpha$, the exchangeability-based guarantee has failed.","tokens_in":12298,"feed_emoji":"📍","tokens_out":6143,"duration_ms":55732,"temperature":0.7,"pith_summary":"This paper applies conformal prediction to deep-learning-based indoor positioning and claims a statistical guarantee: for any user-chosen error rate $\\alpha$, the prediction set built from the model's non-conformity scores contains the true position with probability at least $1-\\alpha$. The guarantee is distribution-free and works with any positioning model, provided calibration and test data are exchangeable. The paper extends the same idea to path navigation by controlling false discovery and false negative rates, and adds a conformal p-value scheme for flagging unreliable positioning points. Experiments on a Wi-Fi fingerprint dataset show that empirical coverage tracks the target coverage across five lightweight models, with the prediction set size serving as an interpretable uncertainty measure.","feed_headline":"Indoor positioning now has a statistical coverage guarantee","feed_subtitle":"Conformal prediction turns model error into a set that contains the true position with a user-set probability.","key_machinery":"The load-bearing object is the non-conformity score and its quantile threshold. The paper computes scores on calibration data, sets $\\hat{q}$ as the empirical quantile of those scores, and builds prediction sets by thresholding the test score at $\\hat{q}$. Exchangeability of calibration and test scores makes the rank of the test score uniform, and that uniformity is the entire source of the coverage guarantee. For risk control, the same quantile logic is applied to loss functions that are non-increasing in the threshold parameter $\\lambda$, yielding bounds on expected loss for false discoveries and false negatives.","core_discovery":"The paper's central claim is that uncertainty in indoor positioning can be made statistically accountable. Using split conformal prediction, the authors convert the model's positioning error into a non-conformity score, typically the Euclidean distance between predicted and true coordinates, $s_i = \\|Y_i - \\hat{Y}_i\\|_2$. They then set a threshold $\\hat{q}$ as the smallest calibration score such that the fraction of scores below it reaches $\\lceil (n+1)(1-\\alpha)\\rceil/(n+1)$, and the prediction set for a test point is $\\{y : s(x_{\\text{test}}, y) \\le \\hat{q}\\}$. Under exchangeability, this set contains the true position with probability between $1-\\alpha$ and $1-\\alpha + 1/(n+1)$. The same machinery, with monotone loss functions defined on path points, controls expected false discovery and false negative rates in navigation, and the conformal p-value bounds the chance of wrongly rejecting the true position.","pith_inferences":["The paper's guarantee is conditional on exchangeability; in deployment, stale calibration data from signal drift or device change would likely push empirical coverage below $1-\\alpha$, which can be tested by splitting the dataset by time or device.","The conformal p-value filter could double as an online distribution-shift detector: a sustained drop in accepted points would signal that the environment has changed and recalibration is due.","Because the coverage proof is distribution-free, the same framework should extend to other fingerprint modalities, such as geomagnetic or Bluetooth signals, with no change to the guarantee."],"forward_implications":["System designers can choose $\\alpha$ and receive a prediction set that contains the true position with probability at least $1-\\alpha$, independent of which deep model produced the scores.","Prediction set size becomes an interpretable uncertainty measure: at the same $\\alpha$, EfficientNet gives smaller sets than VGG19, indicating higher confidence.","Path navigation can be operated under explicit false-discovery and false-negative budgets, so the rate of misleading instructions is bounded rather than merely hoped for.","The conformal p-value filter lets an application suppress positioning points whose reliability is low, keeping the proportion of position-error points below a specified significance level."],"supporting_citations":[{"why":"Supplies the conformal prediction with correctness coverage framework that the paper adapts to indoor positioning.","marker":"Wang et al. [2024]"},{"why":"Provides the conformal risk control formulation used for FDR and FNR control in path navigation.","marker":"Wang et al. [2025a]"},{"why":"Basis for the split conformal prediction approach and the coverage guarantee statement.","marker":"Wang et al. [2025c]"},{"why":"The UJIIndoLoc Wi-Fi fingerprint dataset on which all experiments are run.","marker":"Torres-Sospedra et al. [2014]"},{"why":"Source of the MobileNetV1 architecture used as one of the five positioning models.","marker":"Howard [2017]"},{"why":"Source of the EfficientNet architecture, the model reported to give the smallest prediction sets.","marker":"Tan and Le [2019]"}],"fun_headline_variants":["Indoor positioning gets statistical coverage guarantees","Conformal prediction gives indoor locating certified accuracy","Guaranteed correctness sets for indoor positioning","Statistical safety net for indoor position estimates","Certified indoor positioning via conformal prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee collapses if calibration and test data are not exchangeable; the paper concedes that temporal signal drift, path-sequence correlation, and device heterogeneity can break exchangeability, and its fixes are heuristic rather than part of the proof.","fun_headline_variants_meta":{"raw":{"variants":["Indoor positioning gets statistical coverage guarantees","Conformal prediction gives indoor locating certified accuracy","Guaranteed correctness sets for indoor positioning","Statistical safety net for indoor position estimates","Certified indoor positioning via conformal prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1224,"prompt_tokens":971,"completion_tokens":253,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":189}},"tokens_in":587,"tokens_out":253,"duration_ms":2725,"temperature":1.0,"reasoning_tokens":189,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:09:47.269340+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect calibration fingerprints at one time or with one device, then test on fingerprints taken later or with a different device; if the empirical coverage over many test points falls materially below $1-\\alpha$, the exchangeability-based guarantee has failed.","supporting_citations":[],"review_version":1}