{"id":"fd19f573-9ec0-4dd4-9810-765af0927271","arxiv_id":"2505.02815","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A SetTransformer that uses a probe embedding and its nearest gallery neighbors to decide whether to enroll a gait sample as known or new, tested on CASIA-B and PsyMo.","lead":"This paper trains a transformer model to decide whether a new walking sample matches someone already in a gait database or is a stranger who should be added. It tests the idea on two lab datasets using three existing gait recognition models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'database-agnostic' claim is contradicted by the paper's own cross-dataset results (Table 4: MCCs between -0.007 and 0.242); without a successful held-out dataset evaluation, the central claim is unsupported.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the representativeness of CASIA-B and PsyMo enrollment scenarios for unseen databases. My stress-test review of the full text confirms that this assumption is the one on which the 'database-agnostic' headline claim rests, and the paper's own Table 4 directly tests it with poor results. This is not an external disagreement or a matter of style; the mismatch is internal to the paper: the Discussion asserts generalization across datasets while the reported cross-dataset evaluation shows near-chance performance in several configurations. The within-dataset experiments, especially Table 3's comparison with logistic-regression baselines, are real evidence that the SetTransformer can be competitive in a fixed embedding space, and the paper deserves credit for reporting the cross-dataset results rather than omitting them. However, those same results prevent the strongest claim from being accepted as stated. A third-dataset held-out evaluation is the natural decisive check because it tests the actual deployment scenario claimed by the title and abstract. If such an evaluation were to succeed, the main concern would be resolved; if not, the paper would remain a useful within-dataset enrollment study with a scaled-back claim. Since the reader already reached CONDITIONAL for essentially this reason, my stress-test does not move the verdict.","tokens_in":10573,"tokens_out":4492,"duration_ms":51344,"concrete_test":"Train the enrollment SetTransformer on the union of the CASIA-B and PsyMo training scenarios (the actual proposed training setup), then evaluate zero-shot on a third dataset never used in any training step, e.g., OU-MVLP or WildGait, using a single gait embedding model (GaitPT, 512-d), K = 8, and the same fixed 0.5 decision threshold. Report MCC and ROC-AUC with confidence intervals. If MCC remains below roughly 0.3 or is statistically indistinguishable from Table 4's cross-dataset values, the 'database-agnostic' claim should be withdrawn or weakened to 'works within the datasets used for training.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the SetTransformer enrollment model 'generalizes across different settings of identities, samples per identities, gait recognition architectures, and datasets' (Discussion). For this to hold, the enrollment scenarios constructed in Section 3.1 from CASIA-B and PsyMo must be representative of unseen databases, and the decision function must transfer without retraining or threshold tuning. Table 4 is the paper's own direct test of this assumption: training on one dataset and testing on the other yields MCCs of 0.242, 0.216, 0.138, 0.112, 0.178, and -0.007 across the three embedding models. The chance-level MCC for GaitGraph PsyMo→CASIA-B and the uniformly low values do not demonstrate dataset-agnostic behavior; they demonstrate a substantial domain gap. Within-dataset results in Figures 3-6 and Table 3 may support flexibility inside a fixed embedding space, but they cannot carry the 'database-agnostic' claim because the gallery/probe construction and embedding statistics are shared within each experiment. The paper also does not report the minimal positive evidence for the claimed property: training on the union of CASIA-B and PsyMo and testing on a third database. Without that, the load-bearing assumption is not merely unverified; it is contradicted by the reported cross-dataset numbers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses open-set gait enrollment: deciding whether a probe gait embedding belongs to a known identity in a gallery or is a newcomer. The authors propose a SetTransformer-based binary classifier whose input is a probe embedding, its K nearest gallery neighbors, and per-identity average embeddings, with three variants for encoding neighbor-identity correspondence. They train and evaluate this model on enrollment scenarios constructed from CASIA-B and PsyMo at different identity-to-walk ratios, using embeddings from GaitPT, GaitFormer, and GaitGraph. The paper claims the method is database-agnostic and recognition-architecture-agnostic, requiring no task-specific thresholds or retraining, and reports within-dataset results, comparisons with logistic-regression baselines, and cross-dataset and cross-embedding transfer experiments.","tokens_in":10867,"tokens_out":3268,"duration_ms":32630,"significance":"If the central claims were established, the method would address a genuine deployment gap: automatic enrollment decisions in gait recognition systems, decoupled from the recognition backbone and applicable across databases without threshold tuning. The paper has strengths: it systematically varies id:walk ratios, compares against nontrivial baselines, and promises to release code and dataset scenarios. However, the headline claims of database-agnostic and architecture-agnostic behavior are not supported by the paper's own transfer experiments, and the within-dataset advantage over a simple baseline is inconsistent across ratios. The contribution is therefore potentially useful as a within-dataset enrollment model with known transfer limitations, but the current framing considerably overstates the evidence.","major_comments":[{"comment":"The cross-dataset results directly contradict the 'database-agnostic' claim in the Abstract and Discussion. Table 4 reports MCCs between -0.007 and 0.242 across all transfer directions and embedding models, with GaitGraph PsyMo→CASIA-B at -0.007 (chance-level). The paper's own text in §4 acknowledges 'a clear domain gap.' Since the method is claimed to generalize 'across different settings of identities, samples per identities, gait recognition architectures, and datasets' (Discussion), the load-bearing assumption that enrollment scenarios from CASIA-B and PsyMo transfer to other databases is not merely unverified but contradicted by the reported numbers. To substantiate the claim, the authors would need a successful held-out third-dataset experiment (e.g., training on the union of CASIA-B and PsyMo and testing on a different gait database) or a substantial reframing of the contribution as within-dataset enrollment only.","section":"§4, Table 4"},{"comment":"The claim that the SetTransformer 'scales better with data compared to traditional approaches' is only supported at ID:Walk ratio 0.75. At ratio 0.25, the logistic regression on neighbor distances achieves MCC 0.844 versus the SetTransformer's 0.615; at ratio 0.5, the baseline MCC is 0.832 versus 0.830, essentially a tie. The proposed method clearly wins only in the 0.75 setting. The comparison should be reported across all ratios and datasets, with error bars or significance tests, before claiming a consistent scaling advantage. As written, Table 3 shows the baseline is competitive or superior in two of three conditions.","section":"§4, Table 3"},{"comment":"The 'model-agnostic' claim is also not supported by the cross-embedding transfer results. Table 5 reports near-zero MCCs in several scenarios, e.g., GaitGraph→GaitFormer on PsyMo (MCC 0.000), GaitFormer→GaitPT on PsyMo (MCC 0.023), and GaitGraph→GaitPT on PsyMo (MCC 0.025). If the enrollment model were truly independent of the recognition architecture, one would expect more consistent transfer across embedding spaces; the large variability (MCC from 0.000 to 0.400 in the same dataset) indicates strong sensitivity to the embedding source. The Discussion's claim that the method generalizes 'across ... gait recognition architectures' should be qualified accordingly.","section":"§4, Table 5 and §5"}],"minor_comments":[{"comment":"In the cross-dataset evaluation paragraph, the text states 'GaitGraph ones nearly collapse in the opposite direction (MCC of 0.007)', but Table 4 reports -0.007 for this entry. The sign is important because -0.007 indicates chance or slightly inverted prediction; please correct the inconsistency.","section":"§4"},{"comment":"Both Table 4 and Table 5 have the caption 'Cross-embedding enrollment performance', but Table 4 crosses datasets (train dataset → test dataset) while Table 5 crosses embedding models (train embedding → test embedding). Rename Table 4 to 'Cross-dataset enrollment performance' to avoid confusion.","section":"Tables 4 and 5 captions"},{"comment":"The construction of training and test scenarios should clarify how probe walks from 'new IDs' are chosen so that no identity overlaps between the gallery and probe subsets within an enrollment scenario; otherwise, the measured F1/MCC could be inflated by trivial identity leakage.","section":"§3.1"},{"comment":"The figures report MCC without confidence intervals or repeated-seed variability. Given the large differences attributed to K, id:walk ratio, and training database variability, error bars or multiple-seed results would strengthen the claims and help readers assess stability.","section":"Figures 3-6"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is framed as 'database-agnostic' and 'architecture-agnostic', but the transfer experiments in Tables 4 and 5 are the weakest part of the evaluation. A successful revision would need to either add a third-dataset held-out evaluation that shows the method works without retraining, or thoroughly rewrite the title, abstract, and Discussion to present the method as a within-dataset enrollment model with documented transfer limitations. The baseline comparison also needs to be extended to all ratios and datasets to substantiate any scaling claim. I would lean toward major_revision rather than reject because the within-dataset framework and evaluation protocol may still have value, but the current claims are not defensible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful problem framing and a clean experimental protocol, but the headline claim of database-agnostic enrollment is not supported by the paper's own cross-dataset numbers.\n\nWhat's actually new: the application of SetTransformer to open-set gait enrollment, the id:walk ratio evaluation protocol, and the per-identity positional encoding trick. Decoupling enrollment from the recognition backbone is sensible, and the baselines (logistic regression on embeddings and on neighbor distances) are fair. The paper deserves credit for showing the cross-dataset results even though they hurt the story.\n\nWhere it falls short: Table 4 is the core problem. Training on CASIA-B and testing on PsyMo gives MCCs between about 0.11 and 0.24, and the reverse direction goes as low as -0.007. Those numbers do not support \"generalizes across datasets\" in the Discussion. The within-dataset results in Figures 3-6 are fine as evidence of flexibility inside a fixed embedding space, but they can't carry the dataset-agnostic claim. Also, at the lowest id:walk ratio (0.25), a simple logistic regression on neighbor distances beats the SetTransformer (MCC 0.844 vs 0.615). That's not fatal—at 0.5 they're tied and at 0.75 the transformer wins—but it does temper the \"scales better\" message. No code or hyperparameters are released, despite the promise, and Table 2 shows the underlying recognition models are quite weak (best CASIA-B accuracy about 0.64), which casts doubt on how much the conclusions generalize.\n\nThe paper is not a waste of time. The problem is real and the id:walk protocol is a genuinely useful way to evaluate enrollment. A revised version that tones down the claims, tests on a third dataset, and releases code would be a solid contribution. As is, I'd send it to a serious venue for peer review, but with the expectation of major revisions. The stress-test note is right on target: the central claim is contradicted by Table 4, and that needs to be fixed before publication.","headline":"A useful problem framing and a clean evaluation protocol for open-set gait enrollment, but the paper's own cross-dataset numbers contradict the headline claim of database-agnostic performance.","tokens_in":11374,"tokens_out":2130,"would_cite":false,"duration_ms":23004,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a SetTransformer that sees a probe embedding and its nearest gallery neighbors can decide known-vs-new enrollment without thresholds or retraining, transferring across datasets, recognition models, and gallery sizes.","keywords":["open-set gait enrollment","SetTransformer","gait recognition embeddings","database-agnostic","threshold-free open-set recognition","skeleton-based gait","gallery context","id:walk ratio"],"falsifier":"Run the enrollment model trained only on CASIA-B and PsyMo scenarios on a third gait database with different identities, capture conditions, and id:walk ratios; if MCC there is near zero, the 'database-agnostic' claim is false. The paper's Table 4 already provides a partial test, with cross-dataset MCC between $-0.007$ and $0.242$.","tokens_in":10384,"feed_emoji":"🚶","tokens_out":9635,"duration_ms":90858,"temperature":0.7,"pith_summary":"Open-set gait enrollment asks a surveillance or identification system to decide whether a new walking sample belongs to someone already in its gallery or to a person who should be enrolled as a new identity. The paper argues that this decision can be learned once and reused everywhere: it trains a SetTransformer on the probe embedding plus a small context set of nearest-neighbor gallery embeddings and their identity-average embeddings, and the model outputs an enrollment decision without any threshold that an operator must retune per database. This matters because classical approaches require manually tuned thresholds or per-dataset retraining, which does not scale when galleries and identity sets change. The paper tests the idea with skeleton-based embeddings from three recognition models on two datasets, across gallery sizes and id:walk ratios, and reports that the model outperforms logistic-regression baselines and improves with more neighbors, identity-linked positional encodings, and variable training databases.","feed_headline":"SetTransformer tells a known person from a newcomer in gait databases","feed_subtitle":"Trained once on varied gallery scenarios, it transfers across datasets and embedding models without retraining.","key_machinery":"The machinery is a SetTransformer, an attention-based permutation-invariant neural network that consumes an unordered set of embeddings. For a probe embedding $p$, the method selects its $K$ nearest gallery neighbors by embedding distance, and for each neighbor $g_k$ it forms an identity-average embedding $\\overline{\\mathrm{id}}_{g_k}=\\frac{1}{|G_{g_k}|}\\sum_{g_i\\in G_{g_k}} g_i$, where $G_{g_k}$ gathers all gallery embeddings of that identity. The input set $\\{p, g_k, \\overline{\\mathrm{id}}_{g_k}\\}$ is passed through self-attention, and a two-layer MLP reads the output at the probe's position to produce the enrollment decision. The paper compares additive pairing, per-instance positional encodings, and per-identity positional encodings as ways to link each neighbor to its identity average, and finds that one shared position code per identity works best, because it lets the model attend to identity groups rather than isolated samples.","core_discovery":"The central claim is that gait enrollment can be decoupled from gait recognition: a recognition model produces embeddings, and a separate permutation-invariant SetTransformer makes the open-set decision by attending to the probe embedding together with its $K$ nearest gallery neighbors and the per-identity mean embedding of each neighbor. The paper presents this as a single framework that is agnostic to the recognition architecture and to the database, because it learns from the gallery context itself rather than from a fixed gallery size or identity set. On CASIA-B and PsyMo, using embeddings from three skeleton-based recognition models, the paper reports that the method achieves high enrollment accuracy in balanced and well-sampled scenarios, scales better with data than the logistic-regression baselines, and benefits from training on variable gallery configurations, larger $K$, and per-identity positional encodings. The paper also reports cross-dataset and cross-model transfer experiments; in those runs the numbers are much lower, with cross-dataset MCC between $-0.007$ and $0.242$.","pith_inferences":["A direct extension would be to apply the same context-set formulation to open-set enrollment for other biometric embeddings, such as face or person re-identification, since the model only consumes embeddings and gallery structure; the paper does not test those modalities.","The cross-dataset and cross-model tables suggest the honest scope is transfer within the distribution of controlled laboratory gait trips rather than transfer to arbitrary databases; a sharper name for the property would be gallery-context generalization rather than database agnosticism.","A testable extension is to feed a small amount of unlabeled walks from the target database into the gallery context at inference time, letting the SetTransformer adapt its context statistics before enrollment rather than freezing the model entirely.","The method could be compared against a train-free baseline that uses the ratio of the probe-to-nearest-neighbor distance to the probe-to-second-nearest-neighbor distance, which is a standard open-set heuristic; the paper only compares trained logistic baselines."],"forward_implications":["A deployed gait system could run recognition and enrollment with one shared embedding space: the recognition model identifies who the person is, and the SetTransformer decides whether that person is already known.","Operator overhead falls because the enrollment decision no longer requires manually tuned similarity thresholds that need recalibration when the gallery changes.","Performance increases with the number of neighbors $K$ and with per-identity context, so systems with more gallery evidence can buy accuracy by widening the context set.","Training on variable gallery configurations, with different identity counts and walks per identity, is the mechanism behind generalization; fixed-gallery training is shown to be worse.","Scenarios with many identities but very few walks per identity remain the hard regime, which tells data collectors that more walks per identity matter for enrollment."],"supporting_citations":[{"why":"Supplies the SetTransformer architecture, the attention-based permutation-invariant model the method is built on.","marker":"[13]"},{"why":"Produces the GaitPT skeleton embeddings used in enrollment experiments.","marker":"[2]"},{"why":"Produces the GaitFormer skeleton embeddings used in enrollment experiments.","marker":"[5]"},{"why":"Produces the GaitGraph skeleton embeddings used in enrollment experiments.","marker":"[24]"},{"why":"Provides CASIA-B, one of the two datasets from which enrollment scenarios are built.","marker":"[30]"},{"why":"Provides PsyMo, the second dataset from which enrollment scenarios are built.","marker":"[6]"},{"why":"Represents the threshold-based open-set gait approach that the paper positions its method against.","marker":"[18]"},{"why":"Represents another threshold-based open-set gait approach that the paper positions its method against.","marker":"[17]"},{"why":"Supplies the argument that threshold-based open-set recognition is limited, motivating a learned enrollment model.","marker":"[12]"},{"why":"Defines the Matthews correlation coefficient, the main balanced metric used to report enrollment performance.","marker":"[16]"}],"fun_headline_variants":["SetTransformer enables open-set gait enrollment without retraining","Gait enrollment learns to generalize with SetTransformer context","Open-set gait ID made dataset-agnostic with SetTransformer","SetTransformer decouples gait enrollment from recognition","A SetTransformer learns open-set gait enrollment across datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that enrollment scenarios cut from CASIA-B and PsyMo represent other gait databases, so a model trained on those two transfers with no retraining; the paper's Table 4 cross-dataset MCCs, as low as $-0.007$ and at most $0.242$, show this premise is the point most likely to give way.","fun_headline_variants_meta":{"raw":{"variants":["SetTransformer enables open-set gait enrollment without retraining","Gait enrollment learns to generalize with SetTransformer context","Open-set gait ID made dataset-agnostic with SetTransformer","SetTransformer decouples gait enrollment from recognition","A SetTransformer learns open-set gait enrollment across datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000675,"raw_usage":{"total_tokens":3096,"prompt_tokens":994,"completion_tokens":2102,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":2025}},"tokens_in":610,"tokens_out":2102,"duration_ms":15677,"temperature":1.0,"reasoning_tokens":2025,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:39:54.813234+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the enrollment model trained only on CASIA-B and PsyMo scenarios on a third gait database with different identities, capture conditions, and id:walk ratios; if MCC there is near zero, the 'database-agnostic' claim is false. The paper's Table 4 already provides a partial test, with cross-dataset MCC between $-0.007$ and $0.242$.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SetTransformer architecture, the attention-based permutation-invariant model the method is built on."},{"cited_title":"Catruna, A","cited_arxiv_id":null,"evidence_quote":"Produces the GaitPT skeleton embeddings used in enrollment experiments."},{"cited_title":"Cosma and E","cited_arxiv_id":null,"evidence_quote":"Produces the GaitFormer skeleton embeddings used in enrollment experiments."},{"cited_title":"Teepe, A","cited_arxiv_id":null,"evidence_quote":"Produces the GaitGraph skeleton embeddings used in enrollment experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides CASIA-B, one of the two datasets from which enrollment scenarios are built."},{"cited_title":"Cosma and E","cited_arxiv_id":null,"evidence_quote":"Provides PsyMo, the second dataset from which enrollment scenarios are built."},{"cited_title":"Ni and B","cited_arxiv_id":null,"evidence_quote":"Represents the threshold-based open-set gait approach that the paper positions its method against."},{"cited_title":"Open-Set Gait Recognition from Sparse mmWave Radar Point Clouds","cited_arxiv_id":"2503.07435","evidence_quote":"Represents another threshold-based open-set gait approach that the paper positions its method against."},{"cited_title":"Gunther, S","cited_arxiv_id":null,"evidence_quote":"Supplies the argument that threshold-based open-set recognition is limited, motivating a learned enrollment model."},{"cited_title":"Matthews","cited_arxiv_id":null,"evidence_quote":"Defines the Matthews correlation coefficient, the main balanced metric used to report enrollment performance."}],"review_version":1}