REVIEW 4 major objections 5 minor 20 references
Boosting Generalizable Depth Estimation in Endoscopy by Mixture of Lightweight Experts and Intrinsic Image Alignment
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Routed low-rank experts and intrinsic-image alignment set new endoscopic depth records.
desk verdict A plausible but under-validated incremental contribution to endoscopic depth estimation; the MiLoRE part is solid, the IIA part rests on an untested decomposition, and the 'most accurate' claim is factually wrong on one number. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
MiLoRE is the adapting mechanism. In each Transformer block, three modules (QKV linear and two FFN MLPs) are fine-tuned by routing the input through a top-k selection of LoRA experts, where LoRA means low-rank adaptation: small trainable low-rank updates attached to frozen weights. A router predicts a distribution over experts with an injected noise term, and the output is W0 x plus the weighted sum of selected low-rank updates B_i A_i x. IIA is the lighting-robustness mechanism. A decomposition network with two style MLPs and AdaIN-controlled decoders predicts reflectance R and shading S for source and target frames; the illumination-free loss is the masked photometric error between the war
What would settle it
Shine light from two different positions on a fixed endoscopic scene with known depth and relight the same frames synthetically. If the predicted reflectance images change with lighting, or if the illumination-free alignment loss L_if is not near zero when geometry is unchanged, then the intrinsic decomposition is not providing the claimed lighting invariance. A second check: if removing IIA leaves depth accuracy unchanged on well-lit, low-reflectance frames, the mechanism is not needed there.
Extended reading notes
Core claim
On its own terms, the central discovery is that two mechanisms push self-supervised endoscopic depth estimation past prior art: MiLoRE replaces a single low-rank adapter with a gated mixture of low-rank experts, and IIA adds an illumination-free alignment loss from a self-supervised, style-controlled intrinsic decomposition. Trained on SCARED with a pretrained ViT encoder, the model achieves the best supervised depth on SCARED (RelAbs 0.047), best zero-shot depth on Hamlyn (0.140) and SERV-CT (0.083), and camera-intrinsics errors under about 1% (fx 0.51, fy 0.31, cx 1.02, cy 0.98 percent), with 3.6M trainable parameters. Ablations tie the gains to the expert count/top-k and to IIA; replacing
Load-bearing premise
The load-bearing premise is that the intrinsic image decomposition cleanly separates reflectance from shading, so the illumination-free alignment removes lighting effects without throwing away depth-relevant detail.
Editorial extensions
If this is right
- A single fine-tuned depth foundation model can serve multiple endoscopic settings: the reported zero-shot gains on Hamlyn and SERV-CT suggest less per-hospital retraining.
- Depth and ego-motion can be estimated when camera intrinsics are unknown, since the network predicts intrinsics to roughly one percent error, removing a practical calibration requirement.
- The trainable budget stays small (3.6M parameters) while accuracy improves, so the approach fits into a single-GPU fine-tuning budget.
- Illumination-free reflectance alignment is the proposed route to lighting robustness; if the decomposition stays stable, depth signal should survive light-source motion and specular reflection.
- The mixture-of-experts routing can be adjusted by changing expert count and top-k, giving a practical knob for balancing capacity and generalization.
Reading between the lines
- The paper leaves decomposition quality unevaluated; a direct test of whether the predicted reflectance is truly illumination-invariant, e.g. on synthetic relighting, would strengthen the mechanism it relies on.
- The router's expert selection could be read as a soft taxonomy of endoscopic scene appearances; tracking which experts fire on different tissue types might give a free domain-shift or novelty signal during surgery.
- The same two components could transfer to other non-Lambertian or specular imaging settings, such as cystoscopy, arthroscopy, or open surgical video, where reflectance confounds photometric depth losses.
- Because IIA is additive to existing appearance-flow and optical-flow alignments, it may compose with other depth backbones or parameter-efficient fine-tuning methods, not only the ViT encoder used here.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes EndoMINI, a self-supervised depth/ego-motion estimation framework for endoscopic scenes. It introduces MiLoRE, a mixture-of-experts variant in which LoRA adapters serve as experts with a top-k router, and an intrinsic image alignment (IIA) loss built on a self-supervised intrinsic image decomposition network. The model is fine-tuned on SCARED and evaluated on SCARED, Hamlyn, and SERV-CT, with comparisons to recent baselines including Depth Anything, EndoDAC, MonoPCC, and IID-SfMLearner. The authors report state-of-the-art depth metrics, lower trainable parameters (3.6M), and improved ego-motion/camera-intrinsics prediction. The central claims are that MiLoRE improves cross-scene adaptation and that IIA removes illumination effects.
Significance. The problem is timely and the ingredients are reasonable. MiLoRE is a clean, parameter-efficient combination of MoE and LoRA; the reported 3.6M trainable parameters is a concrete practical advantage. The use of intrinsic decomposition for illumination-robust alignment is plausible, but the paper does not yet demonstrate that the decomposition actually separates reflectance from shading. Because the empirical gains (e.g., SCARED RelAbs 0.047 vs 0.051) are small and no uncertainty estimates are reported, the strength of the evidence is limited. If the decomposition and statistical robustness are validated, the contribution would be useful to endoscopic 3D perception.
major comments (4)
- [§3.2, Table 1] The claim of 'most accurate zero-shot depth estimation on both Hamlyn and SERV-CT' is not supported by Table 1. On SERV-CT, EndoMINI's RelAbs (0.083) is worse than Depth Anything's (0.082). On Hamlyn, EndoMINI is best on RelAbs, but differences across methods are small (e.g., 0.140 vs 0.143). No error bars, confidence intervals, or multiple-run statistics are reported, so it is unclear whether the reported gains are significant. Please qualify the claim and add statistical evaluation or at least per-metric consistency analysis.
- [§2.3–2.4, Eq. (3)–(5)] The IIA loss is the main novelty for illumination handling, and it relies on the intrinsic decomposition recovering true reflectance. Eq. (3) only enforces R⊗S ≈ I plus style-vector consistency; this is underconstrained and admits a degenerate solution R≈I, S≈1, or a solution where shading remains in R. No visualizations, no quantitative separation metrics, and no relighting-invariance test are provided. Therefore, the improvement of ablation F (IIA on vs. off) could come from the added photometric constraint or extra network capacity rather than from illumination-free alignment. Please include direct evaluation of the decomposition, such as synthetic relighting experiments or reflectance-shading separation metrics.
- [§3.4, Table 3] The statement that EndoMINI provides 'the most accurate ego-motion estimation' is contradicted by Table 3: on Seq. 1, EndoMINI's ATE (0.0302) is worse than IID-SfMLearner (0.0296) and EndoDAC (0.0290). No average or standard deviation over the five sequences is given, so the aggregate claim is unsupported. Please report a summary statistic across sequences and clarify whether the camera-intrinsics results are from the same model used for depth evaluation.
- [§2.3, Fig. 4] The intrinsic image decomposition network is only sketched. The figure and text do not specify the encoder and decoder architecture, the dimension of the style vectors, whether source and target share weights, how the decomposition is initialized or trained relative to the depth/pose network, or how the gradients from Eq. (5)-(8) flow into the decomposition network. This is a reproducibility concern because the IIA contribution is central to the paper. Please provide architectural details and the exact training schedule.
minor comments (5)
- [§3.1] The text says 'five metrics' but then lists only RelAbs, RelSq, RMSE, and RMSElog; δ (threshold accuracy) is used in the tables but omitted from the list.
- [Table 1, Hamlyn row] The row for EndoDAC on Hamlyn is marked with citation [13] (AF-SfMLearner) but should cite EndoDAC [4].
- [§3.1] Typo: 'all 92672 frames from 21 rectified videos [12] are selected for for evaluation' -- duplicate 'for'.
- [Eq. (8)] Eq. (8) writes L = λ1(...) + λ2 Lif + λ3 Lsa + Lrec, but λ4, λ5, λ6 are only defined in Eq. (6). Please restate the full loss with all six weights defined in one place.
- [Abstract / §3.2] The term 'supervised depth estimation' is used for SCARED, while the method is described as self-supervised. Please clarify whether 'supervised' refers to fine-tuning with stereo/self-supervision or to the evaluation protocol.
Circularity Check
No significant circularity: MiLoRE and IIA are empirically validated; the IIA illumination-free claim rests on an untested decomposition assumption, which is a correctness risk rather than a circular step.
full rationale
The paper does not exhibit a circular derivation. The central contributions are (1) MiLoRE, a mixture-of-experts LoRA fine-tuning scheme, and (2) IIA, an intrinsic-image alignment loss. Neither reduces to its own input by construction. MiLoRE is a standard parameter-efficient fine-tuning module whose benefit is tested directly in ablations D and E against MoE and DV-LoRA baselines. IIA is an auxiliary photometric loss on predicted reflectance images (Eq. 5); it is a training objective, not a fitted constant renamed as a prediction, and ablation F provides an on/off comparison. The only self-citation, EndoDAC [4], is used as an architecture source and as a baseline with public weights, and the paper's improvements over it are empirically demonstrated, so this citation is independent support rather than load-bearing circularity. The IIA claim that 'the illumination condition will not influence the reflectance image' depends on an unvalidated assumption that the intrinsic decomposition truly separates reflectance from shading (Eq. 3 is under-constrained and permits degenerate R≈I, S≈1 solutions). This is a genuine limitation and correctness risk, but it is not circularity: the paper does not define reflectance in terms of the depth output, and the IIA loss is not equivalent to the standard photometric loss by construction unless one assumes the degenerate solution, which the paper does not assert. Therefore no specific equation reduces to another by construction, and no fitted parameter is presented as a prediction. The evaluation against SCARED, Hamlyn, and SERV-CT provides external validation.
Assumptions & free parameters
free parameters (10)
- lambda_1 (IID loss weight) =
0.01
- lambda_2 (illumination-free alignment weight) =
0.02
- lambda_3 (smoothness weight) =
0.001
- lambda_4 (optical flow reconstruction weight) =
0.01
- lambda_5 (depth reconstruction weight) =
0.01
- lambda_6 (intrinsic reconstruction weight) =
0.1
- LoRA rank r =
4
- Number of FFN experts =
5
- Number of QKV experts =
3
- Top-k =
2
assumptions (4)
- domain assumption An endoscopic image I can be decomposed as the pixel-wise product of reflectance R and shading S (I = R ⊗ S).
- domain assumption Photometric consistency holds between frames: pixel intensity differences are caused by viewpoint change and illumination, and depth/ego-motion can explain them.
- domain assumption Optical flow and appearance flow are available and accurate enough to compute visibility masks and to warp images and intrinsic components.
- domain assumption A pretrained ViT encoder (Depth Anything) can be effectively adapted to endoscopy with a small number of trainable low-rank adapters and a router.
Cite this review
Pith. "Pith review of Boosting Generalizable Depth Estimation in Endoscopy by Mixture of Lightweight Experts and Intrinsic Image Alignment." pith.science (2026). https://pith.science/paper/DB2V4EDV
@misc{pith2026260800415,
author = {Pith},
title = {Pith review of: Boosting Generalizable Depth Estimation in Endoscopy by Mixture of Lightweight Experts and Intrinsic Image Alignment},
year = {2026},
howpublished = {\url{https://pith.science/paper/DB2V4EDV}},
note = {Machine review of arXiv:2608.00415}
}
read the original abstract
Depth estimation is a significant task for 3D perception in endoscopic surgeries. However, illumination interference and feature diversity in various endoscopic scenes are still challenges for generalizable depth estimation and ego-motion estimation. Based on this, a novel self-supervised framework, EndoMINI, is proposed for depth estimation in endoscopic scenes. Specifically, mixture of low-rank experts (MiLoRE) is proposed to perform parameter-efficient fine-tuning, which can also boost the model adaptation to scenes with different characteristics. Meanwhile, an intrinsic image alignment (IIA) is introduced into the training loss to alleviate the influence of light reflectance in endoscopy with a novel intrinsic image decomposition network. The proposed method is evaluated on SCARED datasets for supervised depth estimation, and two endoscopic datasets, Hamlyn and SERV-CT, for zero-shot depth estimation, compared with state-of-the-art works as well. The experimental results demonstrate outstanding performance of the proposed model and the effects of the main contributions.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[13]
Medical image analysis77, 102338 (2022)
Shao, S., Pei, Z., Chen, W., Zhu, W., Wu, X., Sun, D., Zhang, B.: Self-supervised monocular depth and ego-motion estimation in endoscopy: Appearance flow to the rescue. Medical image analysis77, 102338 (2022)
2022
-
[4]
In: International Conference on Medical Image Computing and Computer-Assisted Intervention
Cui, B., Islam, M., Bai, L., Wang, A., Ren, H.: Endodac: Efficient adapting foun- dation model for self-supervised depth estimation from any endoscopic camera. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 208–218. Springer (2024)
2024
-
[1]
arXiv preprint arXiv:2101.01133 (2021)
Allan, M., Mcleod, J., Wang, C., Rosenthal, J.C., Hu, Z., Gard, N., Eisert, P., Fu, K.X., Zeffiro, T., Xia, W., et al.: Stereo correspondence and reconstruction of endoscopic data challenge. arXiv preprint arXiv:2101.01133 (2021)
arXiv 2021
-
[2]
arXiv preprint arXiv:2407.06204 (2024)
Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., Huang, J.: A survey on mixture of experts. arXiv preprint arXiv:2407.06204 (2024)
arXiv 2024
-
[3]
IEEE Transactions on Medical Imaging40(1), 371–380 (2020)
Collins, T., Pizarro, D., Gasparini, S., Bourdel, N., Chauvet, P., Canis, M., Calvet, L., Bartoli, A.: Augmented reality guided laparoscopic surgery of the uterus. IEEE Transactions on Medical Imaging40(1), 371–380 (2020)
work page 2020
-
[5]
Medical image analysis76, 102302 (2022)
Edwards, P.E., Psychogyios, D., Speidel, S., Maier-Hein, L., Stoyanov, D.: Serv-ct: A disparity dataset from cone-beam ct for validation of endoscopic 3d reconstruc- tion. Medical image analysis76, 102302 (2022)
work page 2022
-
[6]
Giannarou, S., Stoyanov, D., Noonan, D., Mylonas, G., Clark, J., Visentini- Scarzanella, M., Mountney, P., Yang, G.Z.: Hamlyn centre laparoscopic / endo- scopic video datasets, https://hamlyn.doc.ic.ac.uk/vision/
-
[7]
In: Proceedings of the IEEE/CVF in- ternational conference on computer vision
Godard, C., Mac Aodha, O., Firman, M., Brostow, G.J.: Digging into self- supervised monocular depth estimation. In: Proceedings of the IEEE/CVF in- ternational conference on computer vision. pp. 3828–3838 (2019)
2019
Show all 20 references
-
[8]
In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=nZeVKeeFYf9
Hu, E.J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=nZeVKeeFYf9
2022
-
[9]
IEEE Journal of Biomedical and Health Informatics pp
Li, B., Liu, B., Zhu, M., Luo, X., Zhou, F.: Image intrinsic-based unsupervised monocular depth estimation in endoscopy. IEEE Journal of Biomedical and Health Informatics pp. 1–11 (2024) 10 L. Shao et al
2024
-
[10]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Liu, Y., Li, Y., You, S., Lu, F.: Unsupervised learning for intrinsic image decom- position from a single image. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3248–3257 (2020)
2020
-
[11]
Medical image analysis71, 102058 (2021)
Ozyoruk, K.B., Gokceler, G.I., Bobrow, T.L., Coskun, G., Incetan, K., Almalioglu, Y., Mahmood, F., Curto, E., Perdigoto, L., Oliveira, M., et al.: Endoslam dataset and an unsupervised monocular visual odometry and depth estimation approach for endoscopic videos. Medical image ...
2021
-
[12]
IEEE Robotics and Automation Letters6(4), 7225– 7232 (2021)
Recasens, D., Lamarca, J., Fácil, J.M., Montiel, J., Civera, J.: Endo-depth-and- motion: Reconstruction and tracking in endoscopic videos using depth networks and photometric constraints. IEEE Robotics and Automation Letters6(4), 7225– 7232 (2021)
2021
-
[14]
Medical Image Analysis102, 103534 (2025)
Wang, Z., Zhou, Y., He, S., Li, T., Huang, F., Ding, Q., Feng, X., Liu, M., Li, Q.: Monopcc: Photometric-invariant cycle constraint for monocular depth estimation of endoscopic images. Medical Image Analysis102, 103534 (2025)
2025
-
[15]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., Zhao, H.: Depth anything: Un- leashing the power of large-scale unlabeled data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10371–10381 (2024)
2024
-
[16]
Advances in Neural Information Processing Systems37, 21875–21911 (2024)
Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., Zhao, H.: Depth anything v2. Advances in Neural Information Processing Systems37, 21875–21911 (2024)
2024
-
[17]
IEEE Transactions on Medical Imaging43(5), 1934–1944 (2024)
Yang, Z., Pan, J., Dai, J., Sun, Z., Xiao, Y.: Self-supervised lightweight depth estimation in endoscopy combining cnn and transformer. IEEE Transactions on Medical Imaging43(5), 1934–1944 (2024)
1934
-
[18]
In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion
Zhang, N., Nex, F., Vosselman, G., Kerle, N.: Lite-mono: A lightweight cnn and transformer architecture for self-supervised monocular depth estimation. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion. pp. 18537–18546 (2023)
2023
-
[19]
In: 2022 international conference on 3D vision (3DV)
Zhao, C., Zhang, Y., Poggi, M., Tosi, F., Guo, X., Zhu, Z., Huang, G., Tang, Y., Mattoccia, S.: Monovit: Self-supervised monocular depth estimation with a vision transformer. In: 2022 international conference on 3D vision (3DV). pp. 668–678. IEEE (2022)
2022
-
[20]
In: 2024 IEEE International Conference on Bioin- formatics and Biomedicine (BIBM)
Zhou, Y., He, S., Wang, H., Huang, F., Liu, M., Li, Q., Wang, Z.: Improved self-supervised monocular endoscopic depth estimation based on pose alignment- friendly dynamic view selection. In: 2024 IEEE International Conference on Bioin- formatics and Biomedicine (BIBM). pp. 300...
2024
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.