{"id":"a1d9a6a6-aac2-494d-b2a3-dd28e6e4afdc","arxiv_id":"2502.03783","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An automated CT-to-ultrasound alignment pipeline produced a 100k-image ultrasound bone dataset and a segmentation model whose reported edge over manual labeling depends on a circular evaluation design.","lead":"Researchers built UltraBones100k, a 100,000-image ultrasound dataset of human lower limbs with bone labels generated automatically by aligning CT scans with tracked ultrasound, instead of manual expert labeling. The method could enable larger training sets for bone segmentation in computer-assisted orthopedic surgery, but the headline claim that the model beats manual labeling rests on a circular benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline model-vs-manual result is largely predetermined: Section 3.2 evaluates against the same CT-derived labels used for training, and the manual-labeling protocol explicitly excluded low-intensity regions, so the 320% completeness gain needs an independent ground-truth check before it can…","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the evaluation uses CT-derived labels from the paper's own pipeline as ground truth for both training and comparison, while the manual-labeling protocol excludes the very low-intensity regions where the claimed superiority is largest. My stress-test adds a mechanistic point: in anechoic/low-intensity regions, Eq. (2) has no bright image evidence to optimize against, so those labels are essentially unvalidated tracked-CT projections. The manual labeler was told not to annotate those regions, so the reported 320% completeness gain is close to a consequence of the annotation protocol rather than an empirical discovery about segmentation quality. This does not invalidate the dataset's value or the alignment-quality rating study; the released data and code permit external checks, and the CT-derived labels may well be accurate. But the specific claim that the model outperforms manual labeling in low-intensity regions requires an independent geometric reference before it can be accepted. The reader's CONDITIONAL verdict already reflects this uncertainty, so I recommend keeping it unchanged: keep the paper's acceptance conditional on a non-circular validation of the low-intensity labels, for example via physical digitization of the bone surface.","tokens_in":15401,"tokens_out":4545,"duration_ms":50318,"concrete_test":"After imaging a subset of test specimens, dissect the overlying soft tissue and digitize the exposed bone surfaces with a calibrated tracked pointer (with K-wires keeping the bones fixed), producing an independent point-cloud reference in CT space. Recompute accuracy, completeness, and F1 at a 0.5 mm threshold for CT-derived labels, model predictions, and manual labels against this physical reference, separately for high- and low-intensity regions. If the model's low-intensity completeness advantage over manual labeling does not persist under this neutral reference, the headline claim should be withdrawn or reworded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim depends on the CT-derived labels being correct in low-intensity and anechoic regions. Section 3.2 states 'our evaluation is based on the CT-derived labels', and those labels are produced by the same tracked-CT projection and Eq. (2) intensity optimization that generated the training data. In low-intensity regions there is no bright ultrasound signal to constrain Eq. (2); the label remains essentially the tracked CT projection, whose accuracy is governed by tracking/calibration error rather than image evidence. The manual-annotation protocol compounds the problem: the surgeon was instructed to label the hyperechoic bone surface layer and its acoustic shadow, i.e., not to label low-intensity or anechoic bone regions. The Discussion even concedes 'Manual labels mostly include only high-intensity bone regions.' Consequently, the reported low-intensity completeness of manual labeling (0.190 vs 0.801) is close to zero by protocol, not by impartial measurement. The model can 'outperform' the surgeon simply by emitting a line wherever the CT projection says one exists; whether that line is anatomically correct in anechoic regions is untested. The 18.7% frame discard after optimization further selects frames where the intensity-based correction is well-behaved. The magnitude of the reported improvement is therefore not a valid estimate of superiority without a neutral geometric reference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces UltraBones100k, a public dataset of about 100k ex-vivo ultrasound images of human lower limbs with automated bone-surface labels. Labels are obtained by optically tracking the ultrasound probe and a cadaver marker, projecting a CT-derived bone model into each image, and then refining per-frame pose via an intensity-based optimization (Eq. 2). Bone-shadow and incidence-angle masks are applied. The authors report a two-rater clinical evaluation showing that the refined alignment improves over the initial tracking alignment and over a global CT-US registration method. They then train a U-Net on the generated labels and compare its predictions with manual annotations by an expert surgeon on 700 test images, reporting that the model outperforms manual labeling on accuracy, completeness, and F1-score, especially in low-intensity regions (e.g., 320% completeness improvement at 0.5 mm).","tokens_in":1632,"tokens_out":2141,"duration_ms":40481,"significance":"If the results hold, the paper would make a substantial contribution: it provides the largest public ultrasound bone-segmentation dataset to date, releases code and pretrained model weights, and demonstrates a practical pipeline that avoids costly expert labeling. The per-frame alignment refinement with a clinical rating study is a useful methodological step over previous global registration approaches. The claim that a model trained on automatically generated labels can segment bone surfaces more completely than an expert, particularly in low-intensity regions, is scientifically important but currently rests on an internal evaluation design that is not fully independent. The dataset release itself is valuable regardless of the outcome of the model-versus-manual comparison.","major_comments":[{"comment":"The central quantitative comparison uses the CT-derived labels as ground truth for evaluating both the trained model and the manual labels. The text states that the evaluation is based on the CT-derived labels (Section 3.2), and these labels are generated by the same tracked-CT projection and Eq. (2) optimization used to produce the training data. Consequently, the reported accuracy, completeness, and F1 scores measure agreement with the pipeline's own output, not anatomical correctness. In low-intensity and anechoic regions, where the image provides little or no bright echo, the optimization in Eq. (2) has no intensity evidence to constrain the solution, so the label reduces essentially to the tracked CT projection. The reported 320% completeness improvement in low-intensity regions therefore requires an independent geometric or manual reference before it can be interpreted as superiority of the model over human labeling.","section":"Section 3.2 (Evaluation Metrics), Section 3.1 (Automatic labeling)"},{"comment":"The manual labeling protocol instructed the expert surgeon to identify the hyperechoic bone surface layer and its acoustic shadow, explicitly omitting low-intensity and anechoic bone regions. The Discussion concedes that manual labels mostly include only high-intensity bone regions. The dramatic completeness gap in low-intensity regions (0.190 manual vs. 0.801 model) is thus largely created by the labeling protocol rather than by an inherent limitation of human performance. A fair comparison requires either manual labels that also cover low-intensity regions (with a protocol allowing anatomical inference) or a separate evaluation restricted to regions where both manual and automatic labels are meaningful.","section":"Section 3.2 (Manual annotation)"},{"comment":"Approximately 18.7% of frames are discarded after optimization because the mean fiducial correction exceeds 1 mm. This selection step removes frames where the tracking-based alignment and the intensity correction disagree, and the test set is drawn from the same post-filtered pipeline. The paper does not analyze the discarded frames or report how the model behaves on them. If the discarded frames are systematically different (e.g., curved bone regions, poor acoustic coupling), the reported segmentation performance may not generalize to the full acquisition stream, and the low-intensity-region claim could be affected by this selection bias.","section":"Section 3.1 (Validation step, frame discard)"},{"comment":"The clinical evaluation on 119 frames provides useful evidence that the optimized alignment is visually preferred over the initial and global-registration alignments, and the Wilcoxon test with Bonferroni correction supports that specific claim. However, this evaluation was designed to rate alignment quality using visible errors, and it does not independently validate the correctness of labels in anechoic or low-intensity regions, where the expert cannot directly see the bone surface. The p<0.001 result therefore does not by itself establish that the low-intensity labels used as ground truth are anatomically correct.","section":"Section 3.1 (Clinical Evaluation)"}],"minor_comments":[{"comment":"Typo: 'tow expert physician' should be 'two expert physicians.'","section":"Section 3.1 (Clinical Evaluation)"},{"comment":"The sentence 'all ultrasound scanning was conducted on fully thawed, all ultrasound scanning was performed on fully thawed' contains a duplicated and incomplete phrase; it should read 'all ultrasound scanning was performed on fully thawed, fresh-frozen human tissue.'","section":"Section 3.1 (Dataset collection)"},{"comment":"The description of the manual labeling protocol would benefit from reporting the actual time taken per image and any intra- or inter-rater variability, as the Limitations section itself notes that multiple-expert annotations are valuable.","section":"Section 3.2 (Manual annotation)"},{"comment":"The optimization parameter lambda is selected from a finite set by testing on randomly sampled frames, but the paper does not state how many frames were used for this selection or whether the selected lambda was fixed before the clinical evaluation. A brief description of the selection procedure would improve reproducibility.","section":"Section 3.1 (Eq. 2)"},{"comment":"The generalizability figures (Figs. 8 and 9) are qualitative only; a sentence clarifying that these are illustrative and not quantitatively evaluated would prevent overinterpretation.","section":"Section 3.2 (Generalizability)"}],"recommendation":"major_revision","confidential_remarks":"The paper has a valuable dataset and a plausible labeling pipeline, and the clinical alignment rating is a genuinely useful contribution. The main concern is the circularity of the model-versus-manual evaluation: the ground truth is produced by the same pipeline that generated the training labels, and the manual protocol excludes exactly the regions where the model is claimed to excel. This is not a fatal flaw in the dataset itself, but the headline claim needs independent validation before it can be accepted as stated. I would ask the authors to provide at least one of the following in revision: (1) a comparison on a subset of frames where manual labels were obtained with a protocol that includes anatomical inference for low-intensity regions; (2) an independent geometric reference (e.g., a second imaging modality or a physically measured surface) for a sample of the low-intensity labels; or (3) a clear separation of the evaluation into high-intensity and low-intensity regions with the acknowledgment that the low-intensity comparison depends on the CT-derived reference. The 18.7% frame discard should also be analyzed or at least discussed as a potential selection bias."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a resource paper, and the resource is real. The 100k-image public dataset with CT-derived bone labels and pretrained weights is the kind of thing the ultrasound-bone-segmentation community needs; no prior public dataset of this scale exists. The per-frame intensity-based refinement of tracked CT-US alignment is a concrete advance over [12], and the two-rater clinical evaluation on 119 frames supports that the optimization improves alignment relative to both the initial tracking and a global CT-US registration.\n\nWhere the paper overreaches is the automatic-vs-manual comparison. Section 3.2 explicitly says evaluation is based on CT-derived labels, and the same pipeline made the training labels. Manual labeling asked the surgeon to outline the hyperechoic surface and shadow, so low-intensity bone was excluded by instruction; the Discussion even concedes manual labels mostly include high-intensity regions. Under that protocol, low manual completeness in low-intensity regions is close to by design, and model completeness there measures consistency with the CT projection, not anatomical correctness. The 320% completeness gain is therefore not a valid estimate of superiority. It could be a real gain, but the current experiment cannot show it.\n\nA few smaller cautions: 18.7% of frames are discarded after optimization; the released dataset presumably contains only kept frames, which is acceptable but should be stated in the evaluation narrative. The model comparison uses one expert annotator, so inter-rater variability is unknown; the authors acknowledge this. Free parameters (gamma, lambda, incidence cutoff, discard threshold) are partly empirical, and sensitivity of the labels to these choices is not reported. Citation pattern is appropriate, and the comparison to [12] is fair.\n\nWho gets value: anyone training or benchmarking ultrasound bone segmentation models will use this dataset. The central claim should be rephrased or re-evaluated, but the artifact deserves serious peer review rather than desk rejection. I would send it out, with a request for either a neutral geometric reference or a manual-label protocol that includes low-intensity regions, and a revised discussion that does not present the model-vs-surgeon comparison as established.","headline":"The dataset and labeling method are real resources; the headline superiority-over-surgeon claim is not established because the evaluation uses the pipeline's own CT-derived labels as ground truth and the manual protocol excludes low-intensity regions.","tokens_in":16249,"tokens_out":2302,"would_cite":true,"duration_ms":24638,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tracked CT models can label ultrasound bone surfaces at 100k-image scale, producing labels that beat manual labeling in low-intensity regions.","keywords":["ultrasound bone segmentation","bone surface extraction","automated labeling","CT-ultrasound registration","optical tracking","deep learning","ultrasound dataset","computer-assisted orthopedic surgery"],"falsifier":"On the same 700 test frames, have several ultrasound-trained surgeons label all visible bone including low-intensity and anechoic regions following a protocol without the instruction to restrict to hyperechoic layers, and evaluate all labels against an independent ground truth such as micro-CT of the scanned specimen. If the model's completeness gain over manual labels at the 0.5 mm threshold drops substantially below the reported 320%, the central claim of superiority over manual labeling fails; alternatively, if an independent CT-ultrasound alignment on held-out specimens yields a mean surface distance larger than 2 mm for this pipeline's labels, the optimization's core error assumption is violated.","tokens_in":15141,"feed_emoji":"🦴","tokens_out":4663,"duration_ms":44025,"temperature":0.7,"pith_summary":"This paper claims that bone surfaces in ultrasound can be labeled automatically and at large scale by superimposing a tracked CT bone model onto tracked ultrasound frames, without relying on costly expert manual annotation. The authors collected UltraBones100k, the largest known public dataset of over 100,000 ex-vivo human lower-limb ultrasound images with bone labels. A U-Net trained on these labels outperforms an expert surgeon on every reported metric, especially in low-intensity and anechoic bone regions where manual labeling is incomplete or impossible. If the claim holds, it would remove the main bottleneck to training and benchmarking ultrasound bone-segmentation models for computer-assisted orthopedic surgery.","feed_headline":"Automated CT-tracked labels beat manual ultrasound bone labeling","feed_subtitle":"A 100k-image dataset trained a model that segments bone surfaces 320% more completely in low-intensity regions.","key_machinery":"The load-bearing mechanism is the per-frame pose refinement in Eq. (2): small perturbations ($\\boldsymbol{\\theta}_\\epsilon$, $\\mathbf{d}_\\epsilon$) are added to each ultrasound image pose so that the mean intensity of pixels lying on the CT-derived bone intersection is maximized, while a regularization term keeps the corrected pose close to the original optical tracking data. The optimization is solved with differential evolution, and failed optimizations are detected and discarded via a fiducial-correction limit. Bone-shadow regions are removed, and only bone pixels with incidence angle $\\alpha \\leq 85^\\circ$ are retained, reflecting when ultrasound can actually return a surface echo. Evaluation uses distance-thresholded accuracy, completeness, and F1 score (Eqs. (3)-(6)), with high- and low-intensity regions separated by Otsu thresholding.","core_discovery":"The paper claims that automatically generated CT-derived labels, refined per frame by a local intensity-based optimization, are accurate enough to serve as ground truth for bone-surface segmentation and as training labels for deep models. After optical tracking and ultrasound calibration place each ultrasound pixel in CT space, the pipeline samples the CT bone surface, computes its intersection with the ultrasound image plane, and perturbs each frame's six-degree-of-freedom pose to maximize the mean intensity of intersected pixels while penalizing deviation from the recorded fiducial positions. After removing frames whose fiducial correction exceeds 1 mm (about 18.7% of frames), the remaining labels form UltraBones100k. A U-Net trained on this dataset achieves, at a 0.5 mm distance threshold, accuracy 0.900, completeness 0.895, and F1 0.893 across all bone regions; in low-intensity regions it improves completeness by 320% (0.190 to 0.801), accuracy by 27.4%, and F1 by 197% relative to an expert surgeon's manual labels.","pith_inferences":["If the per-frame optimization generalizes to other transducers and setups, the same tracking-plus-refinement recipe could generate labeled ultrasound datasets for anatomies like the spine or skull where manual labeling is even harder, while the paper's own limitation section warns that very different bone-soft-tissue structures may require fine-tuning.","The 18.7% frame-rejection rule acts as a quality filter; one testable extension is whether keeping only frames with small fiducial corrections is itself sufficient for training, without the intensity-maximization term.","The reported advantage over the surgeon may shrink if a neutral ground truth, such as an independent micro-CT scan of the same specimen, is used instead of the pipeline's own CT-derived labels, because the current evaluation compares manual labels against those same derived labels."],"forward_implications":["A segmentation model can be trained without expert annotation on a dataset roughly two orders of magnitude larger than typical manual-label sets, so model scale is no longer gated by labeling cost.","Because the model recovers bone contours in low-intensity and anechoic regions, downstream tasks such as 3D bone-surface reconstruction and CT-ultrasound registration can start from more complete 2D segmentations.","The public dataset and pretrained weights give the community a common benchmark, helping to standardize the more than 18 evaluation metrics currently used in ultrasound bone segmentation.","Qualitative results on pelvis, femur, and in-vivo fibula and tibia suggest the learned features transfer beyond the lower-leg ex-vivo training distribution, though the paper presents these results as qualitative rather than quantitative."],"supporting_citations":[{"why":"Reports a prior tracked CT-ultrasound labeling attempt that fell back to manual labeling because of 1-2 mm alignment errors, defining the problem this paper claims to solve.","marker":"[12]"},{"why":"Provides the global CT-ultrasound registration algorithm used as a state-of-the-art comparison that, the paper argues, converges to local minima and yields poorer annotations.","marker":"[23]"},{"why":"Supplies the iterative closest point algorithm used to register virtual spheres to fiducials in the CT model.","marker":"[29]"},{"why":"Establishes the freehand 3D ultrasound convention and previous calibration approach from which the authors adapt the image-origin definition and calibration procedure.","marker":"[30]"},{"why":"Describes the Z-phantom calibration method used for simultaneous spatial-temporal calibration, with the approximately 1.5 mm localization error that this paper seeks to improve on.","marker":"[31]"},{"why":"Defines the differential evolution optimizer used to solve the per-frame intensity maximization in Eq. (2).","marker":"[32]"},{"why":"Introduces the U-Net architecture that the authors train on UltraBones100k for bone-surface segmentation.","marker":"[33]"},{"why":"Provides the Otsu thresholding method used to split bone pixels into high- and low-intensity regions for the reported metrics.","marker":"[35]"}],"fun_headline_variants":["CT-tracked auto labels beat manual bone labeling","Auto labels from CT beat expert manual labels","320% boost: auto bone labels outperform manual","Auto bone labeling outshines manual on 100k images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats the CT-derived labels produced by the paper's own pipeline as ground truth for both training and comparison, so the claim that the model beats manual labeling depends on those derived labels actually being the true bone surface.","fun_headline_variants_meta":{"raw":{"variants":["CT-tracked auto labels beat manual bone labeling","Auto labels from CT beat expert manual labels","320% boost: auto bone labels outperform manual","Auto bone labeling outshines manual on 100k images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000459,"raw_usage":{"total_tokens":2363,"prompt_tokens":1072,"completion_tokens":1291,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":1230}},"tokens_in":688,"tokens_out":1291,"duration_ms":8353,"temperature":1.0,"reasoning_tokens":1230,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T00:44:42.030117+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same 700 test frames, have several ultrasound-trained surgeons label all visible bone including low-intensity and anechoic regions following a protocol without the instruction to restrict to hyperechoic layers, and evaluate all labels against an independent ground truth such as micro-CT of the scanned specimen. If the model's completeness gain over manual labels at the 0.5 mm threshold drops substantially below the reported 320%, the central claim of superiority over manual labeling fails; alternatively, if an independent CT-ultrasound alignment on held-out specimens yields a mean surface distance larger than 2 mm for this pipeline's labels, the optimization's core error assumption is violated.","supporting_citations":[{"cited_title":"van der Zee, M","cited_arxiv_id":null,"evidence_quote":"Reports a prior tracked CT-ultrasound labeling attempt that fell back to manual labeling because of 1-2 mm alignment errors, defining the problem this paper claims to solve."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the global CT-ultrasound registration algorithm used as a state-of-the-art comparison that, the paper argues, converges to local minima and yields poorer annotations."},{"cited_title":"Besl, N.D","cited_arxiv_id":null,"evidence_quote":"Supplies the iterative closest point algorithm used to register virtual spheres to fiducials in the CT model."},{"cited_title":"Mozaffari, W.-S","cited_arxiv_id":null,"evidence_quote":"Establishes the freehand 3D ultrasound convention and previous calibration approach from which the authors adapt the image-origin definition and calibration procedure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the Z-phantom calibration method used for simultaneous spatial-temporal calibration, with the approximately 1.5 mm localization error that this paper seeks to improve on."},{"cited_title":"Ronneberger, P","cited_arxiv_id":null,"evidence_quote":"Introduces the U-Net architecture that the authors train on UltraBones100k for bone-surface segmentation."},{"cited_title":"Otsu, et al., A threshold selection method from gray-level histograms, Autom","cited_arxiv_id":null,"evidence_quote":"Provides the Otsu thresholding method used to split bone pixels into high- and low-intensity regions for the reported metrics."}],"review_version":1}