Image-based marker tracking and registration for intraoperative 3D image-guided interventions using augmented reality

Ali Dhanaliwala; Andong Cao; Brian Park; Jianbo Shi; Terence Gade

REVIEW 3 major objections 6 minor 11 references

Image-based marker tracking and registration for intraoperative 3D image-guided interventions using augmented reality

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Coded radio-opaque stickers can auto-align a CT scan to a patient in about a second.

desk verdict Sensible marker-based AR registration pipeline, but the headline 5±2 mm accuracy rests on a 2D overlay proxy, not a 3D target registration error. read the letter →

arxiv 1908.03237 v1 pith:LYVZIX2L submitted 2019-08-08 cs.CV

Andong Cao , Ali Dhanaliwala , Jianbo Shi , Terence Gade , Brian Park This is my paper · ORCID

classification cs.CV

keywords AugmentedRealityMixedInterventionalRadiologyComputerVisionRegistrationRadio-opaquefiducialmarkersCTsegmentationTrianglematching

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a practical shortcut for augmented-reality surgery: place a few special stickers on the patient before the CT scan, then let the same stickers guide an automatic alignment of the scanned 3D model onto the body. The stickers are radio-opaque optical codes, so they show up both in the CT volume and to the camera of a head-mounted display. The paper reports that marker extraction and 3D-model conversion take under five seconds, that aligning three markers takes $0.9 \pm 0.2$ seconds, and that the resulting overlay lands within $5 \pm 2$ mm of the intended position. If this accuracy transfers to real patients, the method would remove the tedious manual refinement step that currently separates a scan from a usable surgical overlay.

What carries the argument

The load-bearing object is the radio-opaque optical fiducial: a code printed on a radio-opaque sticker, which is simultaneously detectable by a camera and by CT, giving two corresponding point sets with no manual picking. The matching machinery is triangle-ratio registration. Every triple of marker centroids forms a triangle; the triangle is encoded by the ratio of its side lengths (longest to shortest), which is invariant to rotation, translation, and scale, and the set of all candidate triangles is indexed in a k-d tree. The algorithm finds the detected triangle most similar to the triangle from the CT, aligns them in closed form, and flips the normal if the first alignment points the wrong way. This design is what lets registration succeed with only three correspondences and tolerate noisy marker positions.

What would settle it

Run the full pipeline on a CT phantom with several radio-opaque markers and internal bead targets, then measure where the hologram places an internal bead against its known physical position; if average 3D target error is substantially larger than the reported $5 \pm 2$ mm, the accuracy claim does not transfer to real image-guided intervention.

Watch

Extended reading notes

Core claim

The central claim is that a combined radio-opaque optical marker—a sticker that carries a machine-readable visual code and is also dense enough to appear on CT—can be placed on a patient before imaging, located in the CT scan afterward, and then used to register the CT-derived 3D object to the patient in real time. The paper's stated conclusion is that this makes it feasible to align an acquired CT scan with the patient automatically. The registration itself is done by matching triangles: each set of three marker centroids defines a triangle, and the algorithm encodes every triangle by the ratio of its sorted side lengths, stores them in a k-d tree, and rigidly aligns the most similar pair. In the tested extreme of only three visible markers, alignment took $0.9 \pm 0.2$ seconds with an accuracy of $5 \pm 2$ mm, and accuracy improved to $2 \pm 2$ mm when ten markers were visible.

Load-bearing premise

The reported $5 \pm 2$ mm accuracy is measured by overlaying a hologram of a flat reference image on a printed copy and comparing traced centers; if that 2D overlay distance does not match the true 3D alignment error on real anatomy, the accuracy claim is not established.

Editorial extensions

If this is right

The whole registration pipeline—CT marker extraction, 3D conversion, and hologram alignment—can run in under six seconds, making it practical to do at the start of a procedure.
Registration succeeds with as few as three markers, and accuracy improves with more: $0.5 \pm 0.2$ cm with four markers, $0.2 \pm 0.2$ cm with ten.
Against ICP and Vuforia, the algorithm is either faster or more accurate and is less sensitive to noise when point correspondences are sparse.
Accuracy is best at about 51.5 cm from the markers, which is about an arm's length and a natural pose for a surgeon.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the accuracy measurement is a 2D overlay test on a printed image, a natural next experiment is to put markers on a phantom with internal CT-visible targets and measure 3D target registration error; that would tell whether 5 mm is the true surgical accuracy.
The triangle-ratio matching core is agnostic to what the points are, so the same algorithm could register MRI or ultrasound volumes if a marker or anatomical landmark can be detected in both spaces.
The distance-dependence result suggests a practical workflow rule: hold the headset about an arm's length from the markers, or have the system warn when the user moves outside that comfortable band, since accuracy degrades with distance.
With only three markers needed, a plausible extension is marker-free registration on anatomical triples such as bony landmarks, reusing the triangle-ratio k-d tree without any stickers.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper is a proof-of-concept for a sensible workflow: put radio-opaque AprilTag markers on the patient before the CT, segment those markers automatically, then register the CT-derived 3D model to the live markers via triangle matching on the HoloLens. The integration is new even though each piece is established. What it does well: it tackles the practical registration problem with sparse fiducials, uses a k-d tree on triangle-edge ratios for correspondences, and reports a complete pipeline from CT to hologram. The bench experiment is understandable and the authors are clear about wanting fast automatic registration.

The problem is the accuracy evidence. The 5±2 mm number—and every comparison behind Figure 6—is not a 3D target registration error. It is the 2D distance between the traced center of a hologram and the traced center of a printed reference image (Sec. 3.2, Fig. 5). That measures lateral overlay error in one plane; it does not measure depth error, rotational error, or misalignment of internal anatomy. For image-guided procedures, depth error matters, and none of the numerical claims touch it. This is load-bearing because all quantitative claims rest on this proxy. The stress-test note is right.

The speed evidence is also shaky. Registration time is "elapsed time during which CPU utilization stays above idle baseline" from the Windows Device Portal, which is a rough proxy and not a direct timing measurement. The claim that CT marker extraction and 3D conversion take under 5 seconds appears in the abstract without a corresponding measurement in the methods. The Vuforia comparison is borrowed from Park et al. with a different marker, so it is not a head-to-head comparison; there are no trial counts, and no code or data to inspect.

None of this kills the idea. A triangle-matching registration using embedded radiopaque AprilTags is plausible and could well run fast with millimeter-level lateral accuracy. But as written, the paper does not support the abstract's quantitative claim of 0.9±0.2 s and 5±2 mm accuracy against a patient. The fixes are straightforward: measure 3D target registration error on a phantom with CT-visible targets, use direct timing, run repeated trials, report counts, and make code/data available.

Who should read this: anyone working on AR-based surgical navigation who wants a compact example of marker-based registration and a cautionary tale about validation metrics. It deserves a serious referee, but not acceptance as is; conditional acceptance after a 3D TRE evaluation is the right call.

Referee Report

3 major / 6 minor

Summary. This manuscript proposes an end-to-end workflow for automatic registration of a preoperative CT scan to a patient for HoloLens-based AR surgical guidance. Radio-opaque VisiMarkers with coded patterns are placed on the subject before CT, detected in the CT volume, and then detected optically by the HoloLens camera; a triangle-based matching algorithm aligns the corresponding marker configurations. The paper reports that CT marker extraction and conversion to a 3D surface take less than 5 seconds, that alignment with three markers takes 0.9 ± 0.2 seconds, and that this yields an accuracy of 5 ± 2 mm. It also compares the proposed method's speed and accuracy with ICP and Vuforia, claiming advantages in the sparse-correspondence setting.

Significance. If the claimed speed and accuracy are established, the contribution is practically meaningful: coded radio-opaque fiducials address a real sterility and visibility bottleneck in AR-guided interventions, and the triangle-matching registration is designed for the clinically relevant case of only three fiducials. The paper also makes a useful attempt to compare against standard alternatives and to report both speed and accuracy. However, the quantitative evidence is currently tied to a single 2D overlay proxy, the speed measurement is indirect, and the Vuforia comparison uses an external benchmark with a different marker. The feasibility claim is therefore plausible and worth testing more rigorously, but the present evidence does not yet establish it.

major comments (3)

[Section 3.2, Figure 5] The central accuracy claim of 5 ± 2 mm is measured as the 2D distance between the manually traced center of a hologram of a reference image and the center of a printed copy of that image at a single viewpoint. This is not a 3D target registration error of the CT-to-patient alignment: it does not measure depth error, rotational misalignment, or misregistration of internal anatomy, and the traced centers introduce human subjectivity. Because every quantitative accuracy claim in the paper relies on this proxy, the abstract's feasibility conclusion is not yet supported. The authors should report a true 3D target registration error using a phantom or cadaver with additional independent fiducials, measuring physical coordinates against CT-derived coordinates at multiple viewpoints and depths.
[Section 3.1] The reported registration time of 0.9 ± 0.2 s is inferred from the elapsed period during which CPU utilization stays above an idle baseline measured on the HoloLens. This is not a direct timing of the registration routine; it can be confounded by rendering, other processes, and the device's variable CPU governor, and it is not clear how start and end points are detected. Please report wall-clock time around the registration call (or an OS-level timestamp measurement), and state the number of trials and the definition of the uncertainty interval.
[Sections 3.1 and 3.2, Figure 4] The comparative claims against Vuforia are not controlled. The Vuforia speed data are taken from Park et al. (reference [10]) rather than measured on the same device and setup, and the Vuforia accuracy measurement uses a different marker than the one used for the proposed method, explicitly because the reference image has too little feature content for Vuforia to track. As a result, the statement in the introduction that the algorithm can offer 'either faster registration speed or higher registration accuracy' than Vuforia is not supported by the presented experiments. The comparison should either be performed under identical conditions with the same markers, or the claims should be restricted to a transparent benchmark-comparison limitation.

minor comments (6)

[Section 3.2, Figure 6] The caption says 'distance from marker,' but the experiment varies the distance of the marker from the HoloLens; please make the wording consistent.
[References] Reference [8] has a spelling error: 'Muoz-Salinas' should be 'Muñoz-Salinas.'
[Section 3.2] The text reports accuracy decreasing from 0.5 ± 0.2 cm with 4 markers to 0.2 ± 0.2 cm with 10 markers, while the abstract highlights 5 ± 2 mm for 3 markers; please clarify the relationship between the 3-marker and 4-marker results and ensure Figure 6 clearly indicates the number of trials per marker-count condition.
[Section 2.3] The registration algorithm is described only at a high level: the similarity metric for triangles, the encoding used in the k-d tree, and the disambiguation of a wrong normal are not specified in enough detail for reproduction. Please add the exact matching criterion and pseudocode or a more precise algorithmic description.
[Throughout] The marker name is written inconsistently as 'VisiMarkers' and 'Visimarkers'; please standardize the capitalization.
[Section 3.2] Please state the number of repeated measurements used to compute the quoted standard-deviation values and specify what the ± intervals represent.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the registration accuracy and speed are empirical measurements, and the only self-citation is a non-load-bearing comparison baseline.

full rationale

The paper's derivation chain is self-contained: radio-opaque markers are segmented in CT, marker centroids are extracted, triangles are formed and matched via a k-d tree, and the resulting alignment is evaluated by directly measuring the distance between the hologram center and a physical reference-image center (Sec. 3.2, Fig. 5). No equation defines the target quantity in terms of the input; the 5 ± 2 mm accuracy is a measured residual, not a prediction recovered from a fitted parameter. The only self-citation is [10] Park et al., used to supply Vuforia speed and accuracy data for comparison. That citation is not load-bearing for the central feasibility claim, which rests on the authors' own measurements of their triangle-matching algorithm. The HoloLens calibration (ChARUCO plus a modelled position-shift correction) is a calibration step; the paper does not state that the accuracy metric is the same fit used for that correction, so no self-definitional or fitted-input-as-prediction pattern can be exhibited. Concerns about the 2D overlay proxy being an imperfect measure of 3D target registration error are correctness/validity concerns, not circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on engineering assumptions rather than on a new mathematical derivation. Marker centroids from CT connected components must correspond to true physical marker centers, and the HoloLens calibration and display-shift correction must be accurate enough that the 2D overlay distance reflects 3D registration error. The markers themselves are a commercial product, so no new physical entity is introduced. Two free parameters are effectively hidden: the CT size filter range and the display-shift correction model, neither of which has a disclosed value.

free parameters (2)

CT marker size filter range = not reported
Section 2.1 filters connected components by 'sizes different from the expected size of the marker'. The exact range is not given and must be re-tuned for each scan.
HoloLens display shift correction model = not reported
Section 2.2 says the position shift of the hologram is measured, modelled and corrected, but the correction parameters are not disclosed. They directly affect overlay accuracy.

assumptions (4)

domain assumption Radio-opaque Visimarkers produce high-contrast, connected components in CT whose centroids correspond to physical marker centers.
The registration pipeline treats the CT centroid of each marker as the ground-truth 3D location to align. Section 2.1 describes segmentation by connected components and size filtering without validating centroid accuracy.
standard math A pinhole camera model with standard camera calibration and a ChARUCO board accurately models HoloLens image formation.
Used in Section 2.2 to estimate intrinsic parameters; this is a standard imaging assumption, but calibration errors likely contribute to the measured registration error.
domain assumption Triangle edge-ratio matching uniquely identifies correct correspondences among sparse markers.
Section 2.3 selects the most similar triangle in a k-d tree. The method assumes there are no near-degenerate or ambiguous triangle configurations in the surgical field.
domain assumption The error measured as the distance between the traced center of a projected hologram and the center of a real reference image is a valid proxy for CT-to-patient registration error.
Section 3.2 and Figure 5 define registration accuracy this way. It is a 2D overlay and manual-trace measurement, not a 3D target registration error on anatomy.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Image-based marker tracking and registration for intraoperative 3D image-guided interventions using augmented reality." pith.science (2026). https://pith.science/paper/LYVZIX2L

@misc{pith2026190803237,
  author       = {Pith},
  title        = {Pith review of: Image-based marker tracking and registration for intraoperative 3D image-guided interventions using augmented reality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LYVZIX2L}},
  note         = {Machine review of arXiv:1908.03237}
}

abstract

Augmented reality has the potential to improve operating room workflow by allowing physicians to "see" inside a patient through the projection of imaging directly onto the surgical field. For this to be useful the acquired imaging must be quickly and accurately registered with patient and the registration must be maintained. Here we describe a method for projecting a CT scan with Microsoft Hololens and then aligning that projection to a set of fiduciary markers. Radio-opaque stickers with unique QR-codes are placed on an object prior to acquiring a CT scan. The location of the markers in the CT scan are extracted and the CT scan is converted into a 3D surface object. The 3D object is then projected using the Hololens onto a table on which the same markers are placed. We designed an algorithm that aligns the markers on the 3D object with the markers on the table. To extract the markers and convert the CT into a 3D object took less than 5 seconds. To align three markers, it took $0.9 \pm 0.2$ seconds to achieve an accuracy of $5 \pm 2$ mm. These findings show that it is feasible to use a combined radio-opaque optical marker, placed on a patient prior to a CT scan, to subsequently align the acquired CT scan with the patient.

Figures

Figures reproduced from arXiv: 1908.03237 by the authors.

**Figure 2.** The procedure to segment out the marker (April Tags). First, apply the markers onto the patient’s body. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Overview of the registration algorithm. Left: when a new marker appears, record all new triangles that [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Left: average registration time of various registration methods. Right: average registration accuracy of various [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]

**Figure 5.** Figure 5: Left: real reference image and the hologram of the reference image. Middle: mark the center of the hologram [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: Left: registration accuracy vs. number of detected markers. Right: registration accuracy vs. distance from [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 10 canonical work pages

[10]

Park, B. J., Hunt, S., Nadolski, G., and Gade, T., ``Registration methods to enable augmented reality-assisted 3d image-guided interventions,'' in [ 15th International Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine 0.1em ] , 11072 , 110721A, International Society for Optics and Photonics (2019)

work page 2019
[1]

Frantz, T., Jansen, B., Duerinck, J., and Vandemeulebroucke, J., ``Augmenting microsoft's hololens with vuforia tracking for neuronavigation,'' Healthcare technology letters 5 (5), 221--225 (2018)

work page 2018
[2]

Qian, L., Barthel, A., Johnson, A., Osgood, G., Kazanzides, P., Navab, N., and Fuerst, B., ``Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2d-display,'' International journal of computer assisted radiology and surgery 12 (6), 901--910 (2017)

work page 2017
[3]

K., Hilden, H

Horn, B. K., Hilden, H. M., and Negahdaripour, S., ``Closed-form solution of absolute orientation using orthonormal matrices,'' JOSA A 5 (7), 1127--1135 (1988)

work page 1988
[4]

L., ``Multidimensional binary search trees used for associative searching,'' Communications of the ACM 18 (9), 509--517 (1975)

Bentley, J. L., ``Multidimensional binary search trees used for associative searching,'' Communications of the ACM 18 (9), 509--517 (1975)

work page 1975
[5]

W., Bateman, D., Hauberg, S., and Wehbring, R., GNU Octave version 4.2.1 manual: a high-level interactive language for numerical computations (2017)

Eaton, J. W., Bateman, D., Hauberg, S., and Wehbring, R., GNU Octave version 4.2.1 manual: a high-level interactive language for numerical computations (2017)

work page 2017
[6]

Schindelin, J., Arganda-Carreras, I., Frise, E., Kaynig, V., Longair, M., Pietzsch, T., Preibisch, S., Rueden, C., Saalfeld, S., Schmid, B., et al., ``Fiji: an open-source platform for biological-image analysis,'' Nature methods 9 (7), 676 (2012)

work page 2012
[7]

Zhang, Z., ``A flexible new technique for camera calibration,'' Pattern Analysis and Machine Intelligence, IEEE Transactions on 22 , 1330 -- 1334 (12 2000)

work page 2000

Show all 11 references

[8]

Romero Ramirez, F., Muñoz-Salinas, R., and Medina-Carnicer, R., ``Speeded up detection of squared fiducial markers,'' Image and Vision Computing 76 (06 2018)

2018
[9]

Garrido-Jurado, S., Muñoz-Salinas, R., Madrid-Cuevas, F., and Medina-Carnicer, R., ``Generation of fiducial marker dictionaries using mixed integer linear programming,'' Pattern Recognition 51 (10 2015)

2015
[11]

write newline

" write newline "" before.all 'output.state := FUNCTION blank.sep after.quote 'output.state := FUNCTION fin.entry output.state after.quoted.block = 'skip 'add.period if write newline FUNCTION new.block output.state before.all = 'skip output.state after.quote = after.quoted.blo...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [10]

Park, B. J., Hunt, S., Nadolski, G., and Gade, T., ``Registration methods to enable augmented reality-assisted 3d image-guided interventions,'' in [ 15th International Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine 0.1em ] , 11072 , 110721A, International Society for Optics and Photonics (2019)

work page 2019

[2] [1]

Frantz, T., Jansen, B., Duerinck, J., and Vandemeulebroucke, J., ``Augmenting microsoft's hololens with vuforia tracking for neuronavigation,'' Healthcare technology letters 5 (5), 221--225 (2018)

work page 2018

[3] [2]

Qian, L., Barthel, A., Johnson, A., Osgood, G., Kazanzides, P., Navab, N., and Fuerst, B., ``Comparison of optical see-through head-mounted displays for surgical interventions with object-anchored 2d-display,'' International journal of computer assisted radiology and surgery 12 (6), 901--910 (2017)

work page 2017

[4] [3]

K., Hilden, H

Horn, B. K., Hilden, H. M., and Negahdaripour, S., ``Closed-form solution of absolute orientation using orthonormal matrices,'' JOSA A 5 (7), 1127--1135 (1988)

work page 1988

[5] [4]

L., ``Multidimensional binary search trees used for associative searching,'' Communications of the ACM 18 (9), 509--517 (1975)

Bentley, J. L., ``Multidimensional binary search trees used for associative searching,'' Communications of the ACM 18 (9), 509--517 (1975)

work page 1975

[6] [5]

W., Bateman, D., Hauberg, S., and Wehbring, R., GNU Octave version 4.2.1 manual: a high-level interactive language for numerical computations (2017)

Eaton, J. W., Bateman, D., Hauberg, S., and Wehbring, R., GNU Octave version 4.2.1 manual: a high-level interactive language for numerical computations (2017)

work page 2017

[7] [6]

Schindelin, J., Arganda-Carreras, I., Frise, E., Kaynig, V., Longair, M., Pietzsch, T., Preibisch, S., Rueden, C., Saalfeld, S., Schmid, B., et al., ``Fiji: an open-source platform for biological-image analysis,'' Nature methods 9 (7), 676 (2012)

work page 2012

[8] [7]

Zhang, Z., ``A flexible new technique for camera calibration,'' Pattern Analysis and Machine Intelligence, IEEE Transactions on 22 , 1330 -- 1334 (12 2000)

work page 2000