{"id":"415e6eb4-3cde-4e85-98d3-250dfe343bea","arxiv_id":"2505.23763","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"By distilling a large teacher into a MobileNetV2 student and adding a reinforcement-learned canvas selector, the authors reduce FG-SBIR compute from 40.18G to 0.254G FLOPs with accuracy 32.77% vs 33.03%.","lead":"Sketch retrieval usually needs a large neural network, which is too slow for phones. This paper cuts the computation by 99.37 percent while keeping accuracy nearly unchanged, using knowledge distillation and a smart canvas-size selector that shrinks each sketch just enough.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy parity in the headline claim is not statistically established: 33.03 vs 32.77 on 679 sketches is a 0.26-point difference (about 2 queries) with no variance or significance reported.","rationale":"The paper makes a very strong efficiency claim, and the experiments are broad. The RL canvas-selector circularity, which the reader identifies as the weakest assumption, is partially discharged by the paper's own cross-model agreement numbers: four selectors trained with different critics agree on 89.21% of test sketches, teacher-vs-student critics agree on 94.21%, and cross-model plug-in is close. Those checks make the model-dependence concern less load-bearing than it first appears, because the selected canvas sizes are not tightly coupled to one student's blind spots. I therefore did not choose that as the central attack. Instead, the most load-bearing gap is statistical: the headline parity is a 0.26-point difference on 679 test sketches (about two correct retrievals), and no variance, seed variation, or significance test is reported. On multiple other datasets the point drops are larger. Since the paper's value proposition is 'equal accuracy at 99.37% fewer FLOPs', the lack of any uncertainty quantification means the central claim is not yet established. This is a testable reporting issue, not an unsupported allegation: a bootstrap or multi-seed rerun would settle it. The reader's verdict of CONDITIONAL is appropriate, so I do not change it; the concern reinforces the need for released code and variance estimates, which CONDITIONAL already requires.","tokens_in":21859,"tokens_out":22428,"duration_ms":215915,"concrete_test":"Run the full B-VGG16-SN and SketchyNetV2 pipelines for 10 random seeds on QMUL-ShoeV2 and report per-seed Top-1 accuracies. Then apply a paired bootstrap over the 679 test queries (or report the 95% CI for the Top-1 difference). If the upper confidence bound for the accuracy drop exceeds -1.0 point, the 'retaining accuracy' phrasing is not supported and the headline should be weakened; if the CI is tight around -0.26, the central claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim 'retaining accuracy (33.03% vs 32.77%)' rests on point estimates in Table 2 (B-VGG16-SN row) for QMUL-ShoeV2, which has 679 evaluation sketches. A 0.26 percentage-point difference corresponds to fewer than two correctly retrieved sketches. The paper reports no standard deviations, no seed variation, and no test of whether the Top-1 drop is distinguishable from noise. The other datasets in Table 2 show larger point drops (ChairV2: 52.16 to 50.75, -1.41; Sketchy: 18.61 to 16.55, -2.06; FS-COCO: 5.16 to 4.69, -0.47), but none have error bars either. Because the contribution is expressly 'same accuracy at 160x fewer FLOPs', a drop of only 2-3 points would change the qualitative claim from 'retaining accuracy' to 'modest accuracy loss for efficiency'. The suggested circularity of the RL canvas selector is substantially mitigated by the paper's cross-model agreement experiments (89.21%, 94.21%, 95.64%), which show the selected canvas sizes are not idiosyncratic to one critic. The remaining unaddressed weakness is statistical: with n=679, the standard error of the accuracy difference is around 2.5 points, so the reported parity cannot be distinguished from a real several-point drop. No code, per-query outputs, or variance estimates are released, so one cannot bootstrap this from the paper alone.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two components for efficient fine-grained sketch-based image retrieval (FG-SBIR): SketchyNetV1, a MobileNetV2 student trained from a VGG-16 teacher via relational knowledge distillation on pairwise embedding distances, and SketchyNetV2, which adds a reinforcement-learning-based canvas-size selector that rasterizes each vector sketch at a predicted resolution to reduce FLOPs further. The headline result is a reduction from 40.18G to 0.254G FLOPs (99.37%) with reported Top-1 accuracy of 33.03% vs. 32.77% on QMUL-ShoeV2, plus analogous results on ChairV2, Sketchy, and FS-COCO. The paper includes pilot studies motivating the backbone choice and dynamic canvas-sizing, ablations of reward components and backbones, and cross-model and cross-dataset generalization tests of the canvas selector.","tokens_in":22194,"tokens_out":15244,"duration_ms":129359,"significance":"If the claims hold, this is a useful and timely contribution: it is, to my knowledge, the first systematic study of efficient inference specifically for FG-SBIR, and the two modules are generic enough to be attached to different teacher-student pairs. The paper's strengths are its breadth of evaluation (four FG-SBIR datasets plus categorical SBIR and sketch recognition), the explicit pilot studies that motivate the design, and the cross-model agreement experiments (89.21%, 94.21%, 95.64%) that partially discharge concerns about the RL selector fitting a single critic. The FLOPs reductions are consistent across all tested teachers, and the ablations in Table 3 justify the main design choices. However, the central claim of 'retaining accuracy' needs stronger statistical support before it can be taken as established.","major_comments":[{"comment":"The headline claim of 'retaining the accuracy (33.03% vs 32.77%)' rests on single point estimates on 679 evaluation sketches in QMUL-ShoeV2; the 0.26-point difference is less than two correctly retrieved queries. No standard deviations, multiple seeds, or significance tests are reported anywhere in the paper. At n=679, the standard error of this accuracy difference is on the order of 2-3 percentage points, so the observed parity is compatible with a nontrivial true accuracy drop. The same issue affects all datasets in Table 2. I recommend reporting mean±std across at least three seeds, or releasing per-query outputs so that paired bootstrap confidence intervals can be computed; without this, the central 'retaining accuracy' claim is not statistically established.","section":"Table 2 and abstract"},{"comment":"Even ignoring statistical noise, the point estimates on two of the four datasets show non-negligible relative drops: Top-1 falls from 18.61% to 16.55% on Sketchy (-2.06 points, about 11% relative) and from 5.16% to 4.69% on FS-COCO (-0.47 points, about 9% relative). The abstract and conclusion describe the result as 'retaining the accuracy,' which is accurate only for ShoeV2 (and arguably ChairV2). The paper should either present these as an explicit accuracy-efficiency trade-off or add a qualifier such as 'within a small margin on most datasets.' This is load-bearing because the contribution is framed as accuracy-neutral compression.","section":"Table 2, Sketchy and FS-COCO rows"},{"comment":"The RL reward for the canvas-size selector is computed from the student's own rank and triplet loss, and the final evaluation of SketchyNetV2 uses the same student. This creates a risk that the selector is tuned to the student's resolution-dependent blind spots rather than to the sketch's intrinsic abstraction level. The cross-model agreement tests in Section 5.3 substantially mitigate this risk, and I credit them; however, agreement on discrete canvas sizes does not fully guarantee that the chosen sizes preserve accuracy on an independent critic. I ask for one additional experiment targeting the headline model: train the canvas-size selector for the B-VGG16-SN-based SketchyNetV2 setup using the teacher FT as the critic (instead of the student), then report the student's Top-1 and FLOPs. The existing teacher-as-critic experiment in Section 5.3 is for the Triplet-SN setting, not for the B-VGG16-SN model whose accuracy retention is the headline claim.","section":"Section 4.2, Eqs. (4)-(6)"}],"minor_comments":[{"comment":"The abstract reports a FLOPs reduction of '97.96% percent' and the conclusion reports '97.92 (84.9)%'; the two numbers should be reconciled. Also, '97.96% percent' contains a redundant 'percent.'","section":"Abstract vs. Conclusion"},{"comment":"The sentence comparing offset-coordinate and absolute vector formats says 'Turns out the former is better for encoding,' but Table 3 shows the absolute-coordinate version (Ours) achieving 32.77% Top-1 versus 32.61% for offset-coordinate; the text should say 'the latter' or correct the table.","section":"Section 5.3"},{"comment":"The caption does not state the input resolution at which FLOPs, parameters, and per-sample time are measured; given the paper's focus on 256×256 as the full resolution, the table should specify this explicitly.","section":"Table 1"},{"comment":"The right panel's bar heights are described in the text as percentages of sketches achieving perfect retrieval at each canvas size, normalized by those achieving perfect retrieval at 256×256, but the axis label '% sketches' is ambiguous; the caption should explain this normalization.","section":"Figure 2"},{"comment":"The reported full-sketch inference times for the SoTAs (37, 38, 36, 42 ms) are not directly comparable to the per-sample feature extraction times in Table 1 (e.g., VGG-16 at 84 ms), since the SoTAs use different backbones and possibly different hardware; please clarify what is being measured and on which hardware.","section":"Section 5.3, Comparing Inference Times"},{"comment":"No code or pre-trained models are released. Given the number of hyperparameters (λ, λF, λr, λTri, β, Tmax) and the RL training procedure, releasing code would substantially strengthen reproducibility and would also allow readers to compute bootstrap confidence intervals for the headline accuracy claim.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"To the editor: this is a solid empirical paper with a clear contribution and extensive experiments. The main risk is statistical support for the headline 'retaining accuracy' claim; the point estimates alone do not establish parity, and the drops on Sketchy and FS-COCO are larger than the narrative suggests. If the authors add variance estimates or per-query outputs and qualify the cross-dataset claims, I would be willing to support acceptance after a revision. I see no novelty or integrity concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know upfront: this is a solid, applied efficiency paper, and if the numbers hold, it is the first efficient inference pipeline for FG-SBIR. The reviewer's conditional verdict and the stress-test note both land in the right place, but I think the paper deserves a serious referee despite the statistical weakness.\n\nWhat is actually new: nobody has attacked FG-SBIR compute before. The paper shows photo-efficient backbones (MobileNetV2) lose ~37% relative accuracy off-the-shelf, then fixes that with a relational knowledge distillation that transfers teacher pairwise distances rather than logits or features. That is a sensible adaptation to cross-modal retrieval. The second component, an RL canvas-size selector that operates on vector sketches and picks a resolution per query, is also new, and the idea of exploiting sketch sparsity to dynamically reduce input resolution is well motivated. The experiments are broad: four datasets, several SoTA teachers, pruning/quantization baselines, dynamic-resolution baselines, ablations, cross-model agreement (89-95%), cross-dataset transfer, and a QuickDraw recognition side experiment. That is a lot of evidence.\n\nThe soft spots. The biggest is the headline claim \"retaining accuracy\" — 33.03 vs 32.77 on 679 sketches is a 0.26-point difference, roughly two queries, with no standard deviation, seeds, or significance test. The paper cannot distinguish parity from a real 2-3 point drop. This is load-bearing because the contribution is explicitly \"same accuracy at 160x fewer FLOPs.\" I would not call it fatal; the accuracy drops on the other datasets (1-2 points) still leave the efficiency story intact, but the abstract should be tempered. Second, no code or models are released, so the exact FLOPs numbers and the on-device timing cannot be independently checked. Third, the RL canvas selector is trained with a reward from the same student it is evaluated on, which is mild circularity, but the paper's cross-model agreement experiments (89-95%) substantially soften this — the selected resolutions are not idiosyncratic to one critic. Fourth, the main text omits a direct discussion of the closely related [56], though the supplementary covers it.\n\nBottom line: this is a well-executed applied paper, honest about its limitations in the supplementary, with a plausible central claim. A serious referee should see it, mainly to push for variance estimates, significance testing, and code release. I'd take it to a reading group. I would probably cite it if I worked in sketch retrieval.\n\nRecommendation: send to peer review.","headline":"A solid first efficiency paper for FG-SBIR whose headline 'retaining accuracy' rests on point estimates without variance; the method is sound and the paper deserves peer review.","tokens_in":22764,"tokens_out":2036,"would_cite":true,"duration_ms":19387,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that efficient networks for human sketch are achievable by combining cross-modal knowledge distillation with a reinforcement-learned canvas-size selector, cutting the FLOPs of a VGG-16 FG-SBIR model from 40.18G to 0.254G…","keywords":["efficient inference","human sketch","fine-grained sketch-based image retrieval","knowledge distillation","reinforcement learning","canvas-size selection","FLOPs reduction","MobileNetV2"],"falsifier":"A reader could render every test sketch at all four canvas sizes, run the full teacher network on each, and record the size giving the best retrieval rank; if the learned selector agrees with that per-sketch argmax far less often than the reported cross-model agreement rate, or if its choices correlate with student-specific errors, the claim that it selects the content-optimal canvas is refuted.","tokens_in":21658,"feed_emoji":"✏️","tokens_out":7869,"duration_ms":73043,"temperature":0.7,"pith_summary":"Human sketches are sparse, vector inputs, yet no efficient-inference method has been designed for them, and photo-efficient backbones such as MobileNetV2 lose about 37% relative accuracy when used off-the-shelf for fine-grained sketch-based image retrieval (FG-SBIR). The paper claims this gap can be closed by two plug-n-play components: a cross-modal knowledge distillation that transfers the pairwise distance structure of a large teacher (VGG-16) into a MobileNetV2 student, and a reinforcement-learned canvas-size selector that renders each vector sketch at the smallest resolution that preserves retrieval. Together they reduce FLOPs from 40.18G to 0.254G (99.37%) on the ShoeV2 benchmark while keeping Top-1 accuracy essentially unchanged (33.03% vs 32.77%). If correct, this makes FG-SBIR, a task with direct commercial value, deployable on edge devices for the first time.","feed_headline":"99.37% fewer FLOPs without losing sketch retrieval accuracy","feed_subtitle":"A distilled student plus a per-sketch resolution selector cut FLOPs from 40.18G to 0.254G at equal accuracy.","key_machinery":"Two components carry the argument. SketchyNetV1 is a relational knowledge-distillation scheme: a MobileNetV2 student is trained with a triplet loss plus a Huber loss on the pairwise $\\ell^2$ distances ($d_{sp}$, $d_{sn}$, $d_{pn}$) between sketch, positive-photo, and negative-photo embeddings of a frozen VGG-16 teacher, so the student inherits the teacher's embedding geometry rather than a fragile feature regression. SketchyNetV2 adds a canvas-size selector: a one-layer GRU that reads the vector sketch (sequences of $x,y$ and pen-state coordinates, capped by Douglas-Peucker simplification) and samples a canvas size from $\\{32,64,128,256\\}$; the retrieval network acts as a critic, and policy-gradient training maximizes a reward that combines inverse rank and negative triplet loss with a normalized FLOP penalty. The selector is what exploits sketch abstraction, since sketches rendered at $32\\times32$ retain far more retrieval accuracy than photos do at the same resolution.","core_discovery":"The paper's central discovery is that the two properties that make sketches hard for photo models—their fine-grained detail and their abstraction—are exactly the properties an efficient sketch network should exploit. A small student trained to reproduce the teacher's inter-feature distances for sketch-photo-negative triplets, rather than its raw features, can match a VGG-16 teacher's retrieval accuracy at a fraction of the FLOPs; and because sketches are vector graphics that can be rendered at any size, a policy trained to pick the smallest canvas that keeps the retrieved rank high cuts the remaining compute by another factor of 3.28. The paper verifies the core numbers on four FG-SBIR datasets and shows the canvas selector transfers across teachers and datasets.","pith_inferences":["Editorial extension: the selector operates entirely in vector space, so the same abstraction-aware resolution scheduling should transfer to other vector line inputs such as handwriting, technical drawings, and floor plans, where rendering cost and retrieval or recognition accuracy trade off similarly.","Editorial extension: the paper's future-work suggestion could be turned into a concrete test—use the per-sketch chosen canvas to label how much visual information a sketch truly needs, then train a photo resizer to shrink photos to that budget, extending the FLOPs savings to the photo gallery side of retrieval.","Editorial extension: because the reward uses the student's own rank and triplet loss, the selector may inherit the student's blind spots; a useful control experiment would be training the selector with rewards from the full teacher network and comparing the selected canvas sizes.","Editorial extension: the method composes with quantization and pruning—the paper reports 0.249G with pruning and about 9.1ms with INT8 on an iPhone 13—so the reported 160x reduction is not an upper bound, and an INT8-quantized SketchyNetV2 is a plausible sub-0.1G target."],"forward_implications":["Off-the-shelf efficient photo models are not a shortcut for sketch retrieval: the paper's pilot shows MobileNetV2 at 0.833G reaches only 20.85% Top-1 on ShoeV2, versus 33.03% for VGG-16, and fixed low resolution does not close that gap.","The two components compose: SketchyNetV1 alone reduces FLOPs by roughly 48x with a marginal accuracy drop, and the canvas selector then cuts another factor of 3.28 while keeping accuracy at 32.77% versus 33.03%.","Accuracy retention holds across different teachers and datasets: SketchyNetV2 variants of existing FG-SBIR models stay within about 1% of their teachers while using far fewer FLOPs, and a canvas selector trained on one dataset and teacher transfers to another with a small loss (e.g., 53.14% vs 55.32% on ChairV2).","The canvas selector is not tied to FG-SBIR: plugged into a standard sketch-recognition pipeline on QuickDraw it drops FLOPs from 5.28G to 0.261G while keeping accuracy at 68.14% versus 68.71%.","Because the selector is a separate module, accuracy-focused and transformer-based retrieval models can be fitted with it; the supplementary material shows transformer teachers maintain near-teacher accuracy in their SketchyNetV2 variants."],"supporting_citations":[{"why":"Supplies the FG-SBIR triplet-Siamese baseline and the ShoeV2/ChairV2 benchmark datasets that the whole comparison rests on.","marker":"[79]"},{"why":"Defines the VGG-16 architecture used as the full-resolution teacher and as the 40.18G FLOPs reference point being compressed.","marker":"[60]"},{"why":"Supplies MobileNetV2, the lightweight student backbone whose off-the-shelf failure motivates both proposed components.","marker":"[57]"},{"why":"Provides the knowledge-distillation paradigm that the paper adapts into cross-modal relational distillation between teacher and student embedding spaces.","marker":"[51]"},{"why":"Gives the policy-gradient optimization method used to train the canvas-size selector against the non-differentiable rasterization step.","marker":"[64]"},{"why":"Provides the per-sample FLOPs computation and dynamic low-resolution distillation ideas used to quantify and reward compute savings.","marker":"[13]"},{"why":"Defines the vector-sketch representation as a sequence of points with pen states, which the canvas-size selector consumes.","marker":"[23]"},{"why":"Supplies the Douglas-Peucker simplification that caps vector sequence length for the selector without losing visual content.","marker":"[69]"}],"fun_headline_variants":["Sketch retrieval: 99.37% fewer FLOPs, accuracy intact","Distillation + canvas selector cut sketch FLOPs 99.37%","40.18G to 0.254G: efficient sketch retrieval at exact accuracy","Sketch nets: 99.37% FLOP savings from smarter canvas choice","Photo nets fail sketches? This fix cuts FLOPs 99.37%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the student's own retrieval score is a trustworthy proxy for the true minimal canvas size of a sketch; the paper reports cross-model agreement of 89.21% on selected sizes, but if the student systematically favors small canvases that hide hard negatives, the FLOPs savings would be tuned to the student's blind spots rather than to sketch content.","fun_headline_variants_meta":{"raw":{"variants":["Sketch retrieval: 99.37% fewer FLOPs, accuracy intact","Distillation + canvas selector cut sketch FLOPs 99.37%","40.18G to 0.254G: efficient sketch retrieval at exact accuracy","Sketch nets: 99.37% FLOP savings from smarter canvas choice","Photo nets fail sketches? This fix cuts FLOPs 99.37%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1561,"prompt_tokens":963,"completion_tokens":598,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":492}},"tokens_in":579,"tokens_out":598,"duration_ms":6084,"temperature":1.0,"reasoning_tokens":492,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:38:39.115126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could render every test sketch at all four canvas sizes, run the full teacher network on each, and record the size giving the best retrieval rank; if the learned selector agrees with that per-sketch argmax far less often than the reported cross-model agreement rate, or if its choices correlate with student-specific errors, the claim that it selects the content-optimal canvas is refuted.","supporting_citations":[{"cited_title":"Sketch me that shoe","cited_arxiv_id":null,"evidence_quote":"Supplies the FG-SBIR triplet-Siamese baseline and the ShoeV2/ChairV2 benchmark datasets that the whole comparison rests on."},{"cited_title":"Mobilenetv2: Inverted residuals and linear bottlenecks","cited_arxiv_id":null,"evidence_quote":"Supplies MobileNetV2, the lightweight student backbone whose off-the-shelf failure motivates both proposed components."},{"cited_title":"Fitnets: Hints for thin deep nets","cited_arxiv_id":null,"evidence_quote":"Provides the knowledge-distillation paradigm that the paper adapts into cross-modal relational distillation between teacher and student embedding spaces."},{"cited_title":"Policy gradient methods for reinforcement learning with function approximation","cited_arxiv_id":null,"evidence_quote":"Gives the policy-gradient optimization method used to train the canvas-size selector against the non-differentiable rasterization step."},{"cited_title":"Dynamic low-resolution distillation for cost- efficient end-to-end text spotting","cited_arxiv_id":null,"evidence_quote":"Provides the per-sample FLOPs computation and dynamic low-resolution distillation ideas used to quantify and reward compute savings."},{"cited_title":"A neural representation of sketch drawings","cited_arxiv_id":null,"evidence_quote":"Defines the vector-sketch representation as a sequence of points with pen states, which the canvas-size selector consumes."},{"cited_title":"The douglas- peucker algorithm for line simplification: re-evaluation through visualization","cited_arxiv_id":null,"evidence_quote":"Supplies the Douglas-Peucker simplification that caps vector sequence length for the selector without losing visual content."}],"review_version":1}