REVIEW 4 major objections 5 minor 23 references
CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgery
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A new cataract surgery scene graph dataset, CAT-SG, holds over 1.8 million annotated relations, and the CatSGG+ model outperforms the prior state of the art on generating such graphs.
desk verdict CAT-SG is a genuinely useful surgical scene graph dataset with real annotation effort, but the 'Close to' relation likely derived from pseudo-masks makes the headline scale and top-line F1 gains unreliable until clarified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery that carries the argument is the scene graph plus the CatSGG+ architecture. Nodes are surgical instruments or anatomical structures grounded by position, size, and bounding box; edges are relations, with semantic relations predicted from pair proposals built by concatenating two instance query vectors from Mask2Former, $pair_n = [q_i; q_j]$, and the geometric relation 'Close to' derived from adjacent predicted masks. CatSGG+ adds temporal context by max-pooling same-class query vectors across the eight frames in a chunk before building pair embeddings, so the relation predictor sees motion and appearance change, not just a single frame. The relation existence model uses a 2-layer network with sigmoid, and the relation type model is a 3-layer multi-label classifier, trained with binary cross-entropy.
What would settle it
Manually segment a random sample of CAT-SG frames and compare the masks with the released pseudo-masks; if overlap is low, or if recomputing 'Close to' edges from the manual masks materially changes the relation distribution or downstream F1, the geometric grounding claim would be falsified.
Extended reading notes
Core claim
The central claim is that dynamic scene graphs can represent cataract surgery at scale and that this representation enables fine-grained, temporally aware surgical workflow understanding. CAT-SG annotates 29 object classes and eight semantic relations—holding, activation, pushing, pulling, cutting, inserting, retracting—plus the geometric relation 'Close to', totaling 1,811,252 relations on 164,162 frames sampled at 5 fps from 50 CATARACTS videos. The authors report that CatSGG+, their scene graph generation model built on Mask2Former with a VideoSwin backbone pretrained on surgical videos and with max-pooled query embeddings across 8-frame chunks, reaches macro F1 43.11, outperforming ORacle's 34.65. They further report that using semantic relations in a GATv2 phase recognizer raises 30-frame-window accuracy from 73.77 to 78.63, and that their technique recognition baseline reaches 68.75 accuracy for distinguishing 'Stop and Chop' from 'Divide and Conquer' using 10-second windows at 5 fps.
Load-bearing premise
The load-bearing premise is that the pseudo-labelled segmentation masks based on CaDIS correctly ground every object, since node positions, sizes, bounding boxes, and the 'Close to' relations computed from mask adjacency all depend on those masks.
Editorial extensions
If this is right
- CAT-SG provides 1.8 million relations over 164,162 frames, making it the largest surgical scene graph dataset among comparable efforts and a benchmark for tool–tissue interaction modeling.
- The CatSGG+ result of 43.11 macro F1 against ORacle's 34.65 indicates that a segmentation-based, query-pooling approach can outperform a large vision-language model for surgical scene graph generation.
- Semantic relations improve phase recognition: GATv2 with CAT-SG relations reaches 78.63 accuracy and 70.15 F1 on a 30-frame (90-second) window, beating the Holm et al. baseline of 73.77 accuracy and 64.93 F1.
- The new surgical technique recognition task is learnable at 5 fps: the GATv2 baseline reaches 68.75 accuracy and 48.40 F1, with finer temporal sampling outperforming lower-resolution windows.
- Because relations are human-readable, downstream phase and technique predictions built on these graphs can be traced to specific interactions such as 'phaco handpiece activating' or 'cystotome pulling'.
Reading between the lines
- Editorial inference: because 'Close to' accounts for about 93% of all relations, reported scene graph F1 is dominated by a binary adjacency task; a useful stress test would be to evaluate CatSGG+ on the seven semantic relations alone, excluding 'Close to'.
- Editorial inference: the geometric grounding depends on pseudo-labelled CaDIS masks, so the dataset's spatial fidelity is only as good as those masks; comparing a manually segmented subset against the pseudo-masks would quantify how much mask error propagates into node grounding and 'Close to' relations.
- Editorial inference: the same annotation recipe could transfer to other surgical video datasets that have phase labels but no interaction annotations, since scene graph structure is not specific to cataract surgery.
- Editorial inference: CatSGG+ max-pools same-class queries across an 8-frame chunk; attention-based pooling or variable-length chunks might better capture short-lived interactions such as 'cutting' (1,925 samples), which the current model predicts with lower F1.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CAT-SG, a dynamic scene graph dataset for cataract surgery built on the CATARACTS corpus. It contains 29 object classes and 1,811,252 relation instances over 164,162 frames, including eight relation types, a dominant geometric 'Close to' relation, and per-video surgical technique labels. The authors also propose CatSGG/CatSGG+, a Mask2Former-based scene graph generator that uses query embeddings for pairwise relation prediction, and evaluate it against the ORacle baseline. The central claims are that CAT-SG is the first and largest cataract surgery scene graph dataset and that CatSGG+ outperforms prior state of the art on scene graph generation, with additional benchmarks for phase recognition and surgical technique recognition.
Significance. If the annotations are trustworthy, CAT-SG addresses a genuine gap: existing surgical video datasets focus on tools or phases, and a fine-grained dynamic scene graph dataset could support workflow analysis, training feedback, and explainable intraoperative decision support. The scale is notable, and the introduction of surgical technique recognition as a downstream task is a useful addition. The model design is reasonable and the comparison to ORacle is appropriate in spirit. However, the paper's main value depends on the reliability of the pseudo-mask grounding and on the provenance of the 'Close to' relation, which together account for the majority of the dataset and for the reported performance gap. These issues must be resolved before the dataset and benchmark can be considered scientifically sound.
major comments (4)
- [Section 2.2, Table 1, Section 3] The manuscript never specifies how the 1,677,724 'Close to' ground-truth relations were produced. Section 2.2 says that pseudo-labelled CaDIS masks provide grounding (position, size, bounding box), while Section 3 defines the predicted 'Close to' relation as mask adjacency from M2F predictions. If the ground-truth 'Close to' labels were also derived from the same CaDIS pseudo-masks, then the evaluation of this relation is circular because M2F is trained on those masks; the reported Close-to F1 of 91.63 and the macro-F1 gap over ORacle (43.11 vs 34.65) would be inflated, especially since 'Close to' constitutes roughly 93% of all relations. Please state the exact generation procedure for 'Close to' labels, and if they are mask-derived, validate the pseudo-masks against manual segmentation (e.g., IoU) and re-evaluate the benchmark under a non-circular protocol.
- [Section 2.2] The annotation process is described only in terms of annotator counts and total hours; no inter-annotator agreement measure (e.g., Cohen's kappa or pairwise F1) is reported for relation labels, object presence, or technique labels. For a dataset paper whose central contribution is fine-grained manual annotation, the absence of any agreement measure makes it impossible to assess label noise and the practical upper bound of the downstream benchmarks.
- [Sections 4 and 5, Table 5] The surgical technique recognition task is described as a new downstream task, but the evaluation protocol is missing. With 50 videos and two technique classes, the paper does not state the train/validation/test split, the number of test videos, the cross-validation scheme, or how the mean and standard deviation in Table 5 were computed. As written, the technique recognition results are not reproducible and do not support the claim that CAT-SG enables this task.
- [Section 4, Table 3] The comparison with ORacle is not sufficiently controlled. The paper says that ORacle's single-view variant was trained and evaluated, but it does not report the used temporal windows, number of training chunks, learning rate, or whether the same splits and frames were used as for CatSGG/CatSGG+. Because the headline claim is that CatSGG+ outperforms ORacle, these training and evaluation details must be specified for the comparison to be meaningful.
minor comments (5)
- [Table 3] CatSGG+ reports Pushing F1 = 0.00 while CatSGG reports 4.32, yet the text states that CatSGG+ 'further enhances temporal consistency.' This failure on one relation class should be discussed, because it tempers the temporal-consistency claim and may point to a sampling or class-imbalance issue.
- [Section 2.2] The paper says '8 categories' of interactions and Table 1 lists eight relation rows, but the text in Section 3 and later refers to 'semantic relations' and 'geometric relations' without a clear formal distinction; please define which relations are semantic and which are geometric in the dataset itself.
- [Section 4] The 'none' class in Table 3 is not defined in the dataset description; please state how negative pairs were sampled and how the 'none' class was constructed for evaluation.
- [General] The paper does not provide a dataset release URL, code repository, or license information; for a dataset and benchmark paper these should be included.
- [Table 5] The technique recognition experiments include only GATv2; the text says GNNs and LLMs are both baselines, but no LLM result is reported for this task. Please either add the result or remove the claim.
Circularity Check
No demonstrated circularity; the geometric-relation pipeline is model-vs-labels (not a fitted derivation), and the unvalidated CaDIS pseudo-mask grounding is a data-quality risk rather than a circular step.
full rationale
This is a dataset and supervised-model paper, not a derivation from first principles. The central claims are the CAT-SG annotations and the CatSGG benchmark against ORacle. I searched for self-definitional or fitted-input circularity. The strongest candidate is the 'Close to' relation: Section 3 says it 'is determined based on M2F segmentation predictions' via mask adjacency, while Section 2.2 describes the scene graph annotations as created 'through a manual labeling process' and only uses pseudo-labelled CaDIS masks to provide object grounding (position, size, bounding box). The paper never states that the ground-truth 'Close to' labels were algorithmically generated from the same pseudo-masks used to train M2F, so I cannot exhibit the specific reduction that would make the evaluation circular. If the ground-truth 'Close to' relations were in fact mask-derived, the 91.63 F1 would partly measure segmentation consistency rather than independent relation understanding, but that is an unclarified provenance point, not an asserted equivalence. The self-citations ([10] for mask-adjacency design, [12] for graph optimization) are modeling and baseline choices, not load-bearing uniqueness claims. The ORacle comparison may be uneven because ORacle lacks segmentation grounding, but uneven benchmarking is not circularity. Overall, no circular step is demonstrated; the pseudo-mask validation gap is a correctness/data-quality concern.
Assumptions & free parameters
free parameters (3)
- Relation existence threshold =
0.5
- Temporal chunk size =
8 frames
- Training chunks per video =
18
assumptions (3)
- domain assumption CATARACTS phase and tool annotations are accurate
- domain assumption Pseudo-labelled masks from CaDIS are accurate enough for object grounding
- domain assumption The 8 relation categories and 29 objects sufficiently capture relevant tool-tissue interactions
Cite this review
Pith. "Pith review of CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgery." pith.science (2026). https://pith.science/paper/2KUGNQ6Y
@misc{pith2026250621813,
author = {Pith},
title = {Pith review of: CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgery},
year = {2026},
howpublished = {\url{https://pith.science/paper/2KUGNQ6Y}},
note = {Machine review of arXiv:2506.21813}
}
read the original abstract
Understanding the intricate workflows of cataract surgery requires modeling complex interactions between surgical tools, anatomical structures, and procedural techniques. Existing datasets primarily address isolated aspects of surgical analysis, such as tool detection or phase segmentation, but lack comprehensive representations that capture the semantic relationships between entities over time. This paper introduces the Cataract Surgery Scene Graph (CAT-SG) dataset, the first to provide structured annotations of tool-tissue interactions, procedural variations, and temporal dependencies. By incorporating detailed semantic relations, CAT-SG offers a holistic view of surgical workflows, enabling more accurate recognition of surgical phases and techniques. Additionally, we present a novel scene graph generation model, CatSGG, which outperforms current methods in generating structured surgical representations. The CAT-SG dataset is designed to enhance AI-driven surgical training, real-time decision support, and workflow analysis, paving the way for more intelligent, context-aware systems in clinical practice.
Figures
Reference graph
Works this paper leans on
-
[1]
Medical Image Analysis 52, 24–41 (2019)
Al Hajj, H., Lamard, M., Conze, P.H., Roychowdhury, S., Hu, X., Maršalkait˙ e, G., Zisimopoulos, O., Dedmari, M.A., Zhao, F., Prellberg, J., Sahu, M., Galdran, A., Araújo, T., Vo, D.M., Panda, C., Dahiya, N., Kondo, S., Bian, Z., Vahdat, A., Bialopetravičius, J., Flouty, E., Qiu, C., Dill, S., Mukhopadhyay, A., Costa, P., Aresta, G., Ramamurthy, S., Lee, ...
work page 2019
-
[2]
Brody, S., Alon, U., Yahav, E.: How attentive are graph attention networks? In: International Conference on Learning Representations (2022), https://openreview. net/forum?id=F72ximsx7C1
work page 2022
-
[3]
arXiv preprint arXiv:2304.08345 (2023)
Chen, S., He, X., Guo, L., Zhu, X., Wang, W., Tang, J., Liu, J.: Valor: Vision- audio-language omni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345 (2023)
arXiv 2023
-
[4]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1290–1299 (2022)
2022
-
[5]
arXiv preprint arXiv:2305.14314 (2023) 10 F
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient finetun- ing of quantized llms. arXiv preprint arXiv:2305.14314 (2023) 10 F. Holm et al
arXiv 2023
-
[6]
In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidi- rectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). pp. 4171–4186 (2019)
2019
-
[7]
Funke, I., Rivoir, D., Speidel, S.: Metrics matter in surgical phase recognition (2023), https://arxiv.org/abs/2305.13961
arXiv 2023
-
[8]
Watch and Learn: Leveraging Expert Knowledge and Language for Surgical Video Understanding
Gastager, D., Ghazaei, G., Patsch, C.: Watch and learn: Leveraging expert knowl- edge and language for surgical video understanding. ArXiv abs/2503.11392 (2025), https://api.semanticscholar.org/CorpusID:277044026
work page Pith review arXiv 2025
Show all 23 references
-
[9]
arXiv preprint arXiv:1906.11586 (2019)
Grammatikopoulou, M., Flouty, E., Kadkhodamohammadi, A., Quellec, G., Chow, A., Nehme, J., Luengo, I., Stoyanov, D.: Cadis: Cataract dataset for image seg- mentation. arXiv preprint arXiv:1906.11586 (2019)
2019 arXiv
-
[10]
In: Proceedings of the IEEE/CVF international conference on computer vision
Holm, F., Ghazaei, G., Czempiel, T., Özsoy, E., Saur, S., Navab, N.: Dynamic scene graph representation for surgical video. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 81–87 (2023)
2023
-
[11]
Ji, J., Krishna, R., Fei-Fei, L., Niebles, J.C.: Action genome: Actions as composi- tionsofspatio-temporalscenegraphs.In:ProceedingsoftheIEEE/CVFconference on computer vision and pattern recognition. pp. 10236–10247 (2020)
2020
-
[12]
arXiv preprint arXiv:2407.20214 (2024)
Köksal, Ç., Ghazaei, G., Holm, F., Farshad, A., Navab, N.: Sangria: Surgical video scene graph optimization for surgical workflow prediction. arXiv preprint arXiv:2407.20214 (2024)
2024 arXiv
-
[13]
In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin trans- former. In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition. pp. 3202–3211 (2022)
2022
-
[14]
In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2022), https: //openreview.net/forum?id=eJhc_CPXQIT
Luo, Z., Durante, Z., Li, L., Xie, W., Liu, R., Jin, E., Huang, Z., Li, L.Y., Wu, J., Niebles, J.C., Adeli, E., Fei-Fei, L.: MOMA-LRG: Language-refined graphs for multi-object multi-actor activity parsing. In: Thirty-sixth Conference on Neural Information Processing Systems Da...
2022
-
[15]
In: International Conference on Medi- cal Image Computing and Computer-Assisted Intervention
Murali, A., Alapatt, D., Mascagni, P., Vardazaryan, A., Garcia, A., Okamoto, N., Mutter, D., Padoy, N.: Encoding surgical videos as latent spatiotemporal graphs for object and anatomy-driven reasoning. In: International Conference on Medi- cal Image Computing and Computer-Assi...
2023
-
[16]
Rodin, I., Furnari, A., Min, K., Tripathi, S., Farinella, G.M.: Action scene graphs forlong-formunderstandingofegocentricvideos.In:ProceedingsoftheIEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18622–18632 (2024)
2024
-
[17]
In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition
Wang, G., Li, Z., Chen, Q., Liu, Y.: Oed: towards one-stage end-to-end dynamic scene graph generation. In: Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition. pp. 27938–27947 (2024)
2024
-
[18]
IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)
Wang, J., Wen, Z., Li, X., Guo, Z., Yang, J., Liu, Z.: Pair then relation: Pair-net for panoptic scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)
2024
-
[19]
In: European Conference on Computer Vision
Yang, J., Ang, Y.Z., Guo, Z., Zhou, K., Zhang, W., Liu, Z.: Panoptic scene graph generation. In: European Conference on Computer Vision. pp. 178–196. Springer (2022)
2022
-
[20]
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Yang, J., Peng, W., Li, X., Guo, Z., Chen, L., Li, B., Ma, Z., Zhou, K., Zhang, W., Loy, C.C., et al.: Panoptic video scene graph generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18675– 18685 (2023) CAT-SG 11
2023
-
[21]
In: Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision
Zhou, Z., Shi, M., Caesar, H.: Hilo: Exploiting high low frequency relations for unbiased panoptic scene graph generation. In: Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision. pp. 21637–21648 (2023)
2023
-
[22]
Advances in Neural Information Processing Systems37, 107633–107661 (2025)
Zhu, F., Yang, J., Jiang, H.: Towards flexible visual relationship segmentation. Advances in Neural Information Processing Systems37, 107633–107661 (2025)
2025
-
[23]
In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention
Özsoy, E., Pellegrini, C., Keicher, M., Navab, N.: Oracle: Large vision-language models for knowledge-guided holistic or domain modeling. In: International Con- ference on Medical Image Computing and Computer-Assisted Intervention. pp. 455–465. Springer (2024)
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.