DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding

Chenggang Yan; Guanyu Chen; Haochen Shi; Jinkai Zheng; Shuqiang Jiang; Weiqing Min; Yilin Wang

DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FEAST, a reinforcement-learning-guided segmentation decoder, sets a new state of the art on a novel 24k-image dish benchmark, beating Mask2Former by 3.21 mIoU.

desk verdict The dish dataset is a real contribution; the RL-decoder story is mostly MoE, and the headline +3.21% needs an honest accounting of the pretrained reference policy before it can be trusted. read the letter →

arxiv 2607.23070 v2 pith:NKRKDHPU submitted 2026-07-25 cs.CV

Yilin Wang , Haochen Shi , Guanyu Chen , Weiqing Min , Jinkai Zheng , Chenggang Yan , Shuqiang Jiang This is my paper · ORCID

classification cs.CV

keywords foodsegmentationbenchmarkinstancelong-taildistributionmixture-of-expertsMarkovdecisionprocessreinforcementlearningDishSeg24k

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces DishSeg24k, a large-scale benchmark for dish-level food segmentation with 24,096 real-world dining images, 112,281 annotated instances across 278 fine-grained categories, built around dense inter-dish overlap, fine-grained class similarity, and a long-tail distribution. The authors argue that existing benchmarks oversimplify real dining scenes and that query-based segmentation decoders are too deterministic. They propose FEAST, which models query refinement as a Markov Decision Process where each decoder layer applies a continuous 'semantic shift' sampled from a policy, routed through a mixture of shared and specialized experts, and trained with two decoupled critics (task and structure rewards). On DishSeg24k, FEAST with ResNet-50 reaches 53.43% mIoU, outperforming the strongest baseline Mask2Former by +3.21% mIoU, and it also improves FoodSeg103 by +1.47% mIoU. If the reported numbers are accurate, FEAST is the best-performing food segmentation method on both benchmarks, and DishSeg24k becomes the most demanding public testbed for dish-level segmentation.

What carries the argument

The key mechanism is the stochastic MoE decoder: object queries are updated via Gaussian-sampled semantic shifts in a residual transition, routed through one shared expert plus E_r routed experts with a stop-gradient router, and trained with dual critics (task and structure) via policy gradient. A KL penalty anchors the learned policy to a frozen pretrained Mask2Former decoder to prevent catastrophic deviation, while the structure reward (routing entropy minus a load-balancing KL) prevents expert collapse under the long-tail distribution.

What would settle it

Take a random subset of DishSeg24k test images (e.g., 500) and have a second set of annotators re-label them under the same protocol; compute pixel-level agreement. If agreement is significantly lower in overlapping regions than in non-overlapping regions, or if the mIoU ranking of methods changes when evaluated on a high-agreement subset, then the reported differences between methods may be dominated by annotation noise rather than segmentation quality.

Watch

Extended reading notes

Core claim

The central claim is that modeling query-based mask decoding as a sequential stochastic decision process—rather than deterministic layer-by-layer refinement—improves segmentation in densely overlapping, long-tailed food scenes. FEAST treats each decoder layer as an action that shifts object queries in embedding space (q_{k+1}=q_k+a_k), with the action sampled from a Gaussian distribution produced by an MoE policy network. Two independent reward signals guide the training: a task reward measuring incremental mask and classification improvement, and a structure reward encouraging balanced expert routing. The authors show that the MoE decoder alone gives the largest gain (+3.03% mIoU over the M

Load-bearing premise

All method comparisons on DishSeg24k inherit the annotation rule that overlapping ingredients are assigned to the dominant dish, which presumes a single well-defined label for every pixel in dense overlap; no inter-annotator agreement is reported to verify that this rule is applied consistently.

Editorial extensions

If this is right

If the reported gains are accurate, FEAST establishes a new state of the art for food segmentation, beating Mask2Former by +3.21 mIoU on DishSeg24k and +1.47 mIoU on FoodSeg103.
DishSeg24k becomes the largest dish-level benchmark to date (24,096 images, 112,281 instances, 278 categories, 4.66 instances per image), offering a more demanding testbed for food segmentation and downstream tasks like volume estimation and dietary assessment.
The RL-guided MoE decoding framework is general: it can be applied to other segmentation tasks that exhibit dense overlap and long-tail distributions, not just food.
The decoupled dual-reward design—separating mask-quality improvement from routing-balance—provides a recipe for stabilizing MoE training under imbalanced data and may generalize to other long-tail recognition problems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The benchmark's annotation rule for overlapping ingredients ('unified masks based on dominant dish identity') presumes a single correct label for every pixel; without inter-annotator agreement statistics, the reported mIoU differences between methods could partly reflect label noise rather than pure segmentation ability.
At inference, FEAST sets the action to its mean (a_k = mu_k), making the decoder deterministic; the stochastic exploration benefits are realized only during training. An extension would test whether sampling multiple actions at test time and ensembling the resulting masks improves boundary accuracy in overlap regions.
A direct diagnostic experiment would be to re-annotate a random subset of DishSeg24k test images by a second set of annotators and compute pixel-level agreement; if agreement is low in overlapping regions, the method ranking may shift when evaluated on a high-agreement subset.
The dataset's additionally mentioned ingredient-level annotations could enable a transfer-learning test: pretraining on dish-level masks and then evaluating on ingredient-level benchmarks (or vice versa) would quantify how well the dish-level representation supports finer-grained understanding.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague —

Read this one for the dataset, not for the RL machinery. DishSeg24k — 24k images, 112k instance masks, 278 dish classes, 4.66 instances per image — is a genuine step up in scale and density for dish-level segmentation. The re-annotation of ZSFood plus new images under a unified taxonomy is valuable, assuming the masks are clean and the overlap rule doesn't inject too much label noise. If the data actually ships, it becomes a standard benchmark for food computing.

What the paper does well: the ablations are unusually honest. Table 3 shows the MoE decoder gives +3.03 mIoU over the baseline, while stochastic queries and the RL objective add +0.15 combined. Table 4 is even clearer: every RL component alone hurts, and only the full stack gives +0.18 over the MoE-only reference. The abstract and introduction oversell the RL story relative to those numbers, but the data is presented in a way you can check.

The larger problem is the reference policy. The paper calls it a "frozen pretrained Mask2Former decoder" but never says where that checkpoint came from. If it was pretrained on COCO or — worse — on DishSeg24k itself, then FEAST's decoder starts with weights the Table 2 Mask2Former baseline does not get, and the headline +3.21% is partly initialization, not architecture. The Table 3 ablation attributing +3.03 to the MoE decoder has the same issue if that row uses the same pretrained init. This is not necessarily deliberate cheating; it's incomplete reporting, and it is a load-bearing unknown. The annotation-noise concern from the reader is legitimate but secondary, since it affects methods roughly symmetrically.

Also missing: error bars or significance tests, inter-annotator agreement numbers, and a clear statement of which experiments use the pretrained decoder. All fixable, but they need to be fixed before the SOTA claim is credible.

Who is this for? People building food segmentation benchmarks and anyone working on MoE decoders. The dataset is the contribution. The method is worth a look but not a revelation. I would not take the +3.21% at face value until the pretraining question is answered.

Recommendation: send it to peer review. A serious referee can force the pretraining disclosure and get the dataset released in a verifiable form. That's where the value lies.

Referee Report

4 major / 5 minor

Summary. The paper introduces DishSeg24k, a large-scale dish-level food segmentation benchmark with 24,096 images, 112,281 instances, and 278 categories, plus a proposed method FEAST that formulates query-based decoding as an MDP with an RL-guided Mixture-of-Experts decoder. The authors report that FEAST outperforms Mask2Former by +3.21% mIoU on DishSeg24k with a ResNet-50 backbone and generalizes to FoodSeg103 (+1.47% mIoU). The central claim is that the stochastic expert-decoding mechanism is responsible for these gains, and that the new benchmark captures dense overlap, fine-grained similarity, and long-tail distributions.

Significance. If the benchmark is of high annotation quality and the method's gains are real, DishSeg24k would be a valuable resource for food segmentation research, and the RL-guided MoE decoder could be a useful architectural alternative. The paper includes a large-scale dataset with detailed statistics, extensive comparisons, code/model release planned, and a fairly complete ablation study. However, the significance is tempered by two unresolved issues: the marginal contribution of the RL components themselves (Table 4: R5 vs R0 is only +0.18% mIoU) and the undisclosed pretraining of the reference policy, which could confound the headline SOTA comparison.

major comments (4)

[Sec. 5.1, Eq. (6)] The reference policy π_ref is described only as 'the frozen pretrained Mask2Former decoder,' but the pretraining data is never stated. If π_ref was pretrained on DishSeg24k (or another food segmentation dataset), then FEAST receives a pretraining advantage that the Mask2Former baseline in Table 2 does not receive, since the baselines are 'trained from scratch' under the stated protocol. This would directly confound the +3.21% mIoU headline gain and the +3.03% MoE ablation gain in Table 3 (B vs A). Please disclose the pretraining data for π_ref and rerun the Mask2Former baseline initialized from the same checkpoint, or otherwise demonstrate that the comparison is fair.
[Sec. 4.4, Eq. (4), Table 4] The task reward in Eq. (4) is computed from the same ground-truth masks and the same final-layer bipartite matching used by the segmentation loss L_seg, making it closer to per-layer deep supervision than to an independent RL signal. The claimed central mechanism—RL-guided stochastic decoding—is weakly supported: Table 4 shows that each individual RL component (R1, R2, R3, R4) degrades performance below the MoE-only reference, and the full RL stack (R5) adds only +0.18% mIoU over MoE-only (R0). The paper should either provide direct evidence of the RL mechanism's effect (e.g., reward curves, query trajectory diversity, or a comparison against a non-RL stochastic MoE baseline) or temper the claim that RL decoding is the driving contribution.
[Sec. 3.2, Annotation Workflow] The annotation rule that 'overlapping ingredients are assigned unified masks based on dominant dish identity' presumes that a dominant-dish label is well-defined for every pixel in dense overlap. No inter-annotator agreement or label-noise study is reported for this critical rule. If the rule is applied inconsistently, the small metric differences between methods (including the +0.18% mIoU from the RL components) could be within annotation noise. Please report a label-quality study on a subset (e.g., pixel-level agreement on overlapping regions) to justify the benchmark's use for fine-grained method comparison.
[Sec. 5.4, Table 4] Table 4's mDice and mAcc columns appear to be swapped relative to Table 3 (Row B: 53.25 mIoU, 60.48 mDice, 60.68 mAcc, whereas Table 4 R0 shows 60.68 mDice and 60.48 mAcc). Please verify the reported numbers; if the columns are swapped, the ablation conclusions about mDice/mAcc need to be corrected.

minor comments (5)

[Abstract / Sec. 5.1] FoodSeg103 is described as having 104 categories in Sec. 5.5 but 103 in the abstract and elsewhere; please standardize the count.
[Fig. 4 caption] Typo: 'Doupled rewards' should be 'Decoupled rewards'.
[Table 5] CCNet appears twice with different reference numbers ([15] and [37]); the original CCNet is [15], while [37] is the FoodSeg103 benchmark paper. Please disambiguate.
[Table 1] The 'UECFoodPixCompomplete' entry has a typo; should be 'UECFoodPixComplete'.
[Sec. 4.2] The MDP is defined as a tuple (S,A,P,R,γ), but no transition probability distribution is specified for stochastic policies; since the transition is deterministic, this should be stated explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FEAST's performance claims are empirical benchmark results; the RL reward is a supervised training signal, not a fitted prediction, and the self-cited ZSFood source is reused only for raw images.

full rationale

The paper's central claims are empirical: FEAST achieves higher mIoU on a new benchmark and on the external FoodSeg103 benchmark. There is no derivation chain in which a predicted quantity is defined in terms of the fitted input. The task reward in Eq. (4) uses the same ground-truth masks and bipartite matching as the segmentation loss, but this is standard supervised training signal (deep supervision), not a separate prediction claimed to be derived from first principles. The method does not fit a parameter to a subset and then 'predict' a closely related quantity. The only self-citation of note is [45] ZSFood, whose raw images are reused for DishSeg24k after discarding the original bounding-box annotations and re-annotating pixels; this is a disclosed data source, not an unverified load-bearing theoretical premise. No uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled via self-citation. The undisclosed pretraining of the reference policy π_ref is a legitimate experimental-transparency and fairness concern, but it is not circularity: the paper does not claim to derive the reference policy, and the reported comparisons could be unfair without making the result definitionally equivalent to its inputs. The benchmark evaluation is external to the method, and the FoodSeg103 results provide an independent, externally falsifiable check. We therefore find no circular step requiring a score above zero.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on annotation-quality assumptions and unreported RL hyperparameters; no new physical entities are introduced. The benchmark's validation relies on a subjective taxonomy and a dominant-dish labeling rule that is not independently checked.

free parameters (5)

Reward weights λ_dice, λ_cls, λ_div, λ_bal
Introduced in Eqs. (4) and (5) to combine task and structure rewards; numerical values are not reported, and no sensitivity analysis is given, so the RL objective is under-specified.
KL penalty coefficient β = 0.01
Used in Eq. (6) to anchor the learned policy to a frozen Mask2Former decoder; chosen by hand with no ablation.
GAE λ = 0.95
Sec. 4.4 fixes λ_GAE=0.95 without sensitivity analysis.
Routing temperature τ schedule = 1.0 → 0.01 annealed linearly
Eq. (1); the schedule details and its effect on routing are not ablated.
Number of routed experts E_r = 4
Selected by the Fig. 6 ablation (2/4/8 experts); this is central to the MoE gain.

assumptions (5)

standard math Mask2Former's masked cross-attention, self-attention, bipartite matching, and per-query losses are a working base (Sec. 4.1, 4.4).
The method is built on the Mask2Former pipeline and assumes its query-based segmentation machinery is correct.
domain assumption The contextual MDP Markov property holds with pixel features as static context and query state s_k (Sec. 4.2).
The state omits the policy's stochastic history; this is a modeling choice, not proven.
ad hoc to paper A single forward pass is a complete K-step episode and one parameter update per batch yields a valid on-policy policy gradient (Sec. 4.5, 5.1).
REINFORCE/GAE with one episode per update can be high-variance; stability is assumed and not analyzed.
domain assumption Overlapping dishes have a well-defined 'dominant dish identity' mask (Sec. 3.2, Annotation Workflow).
No inter-annotator agreement is reported; label noise would directly affect all benchmark numbers.
domain assumption Images from ZSFood plus 3,493 collected web images represent real-world dining (Sec. 3.1).
No sampling or demographic description beyond Chinese/Western restaurant scenes is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding." pith.science (2026). https://pith.science/paper/NKRKDHPU

@misc{pith2026260723070,
  author       = {Pith},
  title        = {Pith review of: DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NKRKDHPU}},
  note         = {Machine review of arXiv:2607.23070}
}

read the original abstract

Food segmentation is essential for applications such as intelligent catering, dietary assessment, and recommendation. However, existing benchmarks fail to capture the complexity of real-world dining scenes. The challenges of dense inter-dish overlap, fine-grained class similarity, and extreme long-tail class distributions exceed the fidelity of current datasets. To fill this gap, we introduce \textbf{DishSeg24k}, a large-scale dish-level segmentation benchmark with 24,096 images, 112,281 instances, and 278 fine-grained categories in real-world dining environments. Based on DishSeg24k, we further propose \textbf{Food Expert-Adaptive Segmentation Transformers (FEAST)} to address these challenges. FEAST models query-based decoding as a Markov Decision Process (MDP), where each decoder layer update is treated as a sequential decision step that explores uncertainty along dish boundaries. We further redesign the decoder with a reinforcement learning (RL)-guided Mixture-of-Experts (MoE) module, in which a dual-critic decoupled optimization scheme separates task-oriented query refinement from structure-aware expert routing. This design promotes expert specialization and prevents expert collapse under long-tail category distributions. Finally, extensive experiments on DishSeg24k demonstrate the state-of-the-art performance of FEAST, which outperforms previous methods by {+3.21\%} mIoU, {+3.68\%} mDice, and {+4.00\%} mAcc, respectively. We further validate the effectiveness of FEAST on FoodSeg103. The dataset and code will be publicly released.