{"id":"a8698626-2252-4346-ad6c-accc026b1dd6","arxiv_id":"2607.08575","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A 0.88B-parameter VLA reaches 90.0% MT50 success, and a single-query conformal head returns a calibrated joint uncertainty set over the executed action prefix.","lead":"FabriVLA is a compact robot-control model that reaches 90.0% average success on the Meta-World MT50 benchmark with 0.88B parameters, and a companion calibration method turns one policy call into a statistically guaranteed uncertainty set over the actions about to be executed. It offers a path toward cheap, auditable robot policies that flag when their own outputs should not be trusted.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on unverified protocol equivalence of cross-paper baseline scores; the JCAC core is sound.","rationale":"The reader's weakest assumption is exactly the protocol equivalence of the cross-paper baseline comparison. I agree: the JCAC core is a textbook split-conformal construction with a sound max-of-normalized-residuals score, and the paper is careful to scope coverage to exchangeable expert states rather than closed-loop rollouts. The architecture ablations and the LIBERO-Safety ranking analyses are plausible. The single most load-bearing weak point is the SOTA claim in Table 1: it is a comparison against numbers the authors did not generate, with no uncertainty quantification and no released artifacts to verify protocol equivalence. A 2.5-point lead could easily be evaluation noise or a protocol artifact. This does not invalidate the paper's contributions, but it means the headline claim is conditional until the comparison is verified. Since the reader already issued CONDITIONAL for this reason, my stress-test does not change the verdict—it strengthens the reader's rationale with a concrete reproducibility test.","tokens_in":12957,"tokens_out":6070,"duration_ms":63692,"concrete_test":"Re-run LA4VLA, Evo-Depth, and Evo-1 under FabriVLA's exact protocol using the authors' released checkpoints and evaluation code (same 50 tasks, 10 episodes per task, 400-step horizon, H5 execution, success flag, and the same four-tier grouping). If no checkpoints are available, obtain the per-task success rates or per-tier evaluation code from the original papers and recompute the tier averages using FabriVLA's tier assignment. If the 2.5-point lead over LA4VLA does not persist under the matched protocol, the 'state-of-the-art' claim should be downgraded to 'competitive among compared methods'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FabriVLA achieves state-of-the-art 90.0% MT50 success depends entirely on Table 1's cross-paper baseline numbers, taken 'from the corresponding papers' without re-running them. The lead over LA4VLA is only 2.5 points (90.0 vs 87.5), and the comparison assumes the baselines used the same protocol: same 50 tasks, 10 episodes per task, 400-step horizon, fixed H5 execution, same success flag, and the same four difficulty-tier grouping averaged equally. Any mismatch—different episode count, longer execution horizon, different tier definitions, or a different success criterion—could erase or reverse this margin. The paper reports no confidence intervals for its own success rate, and with 10 binary episodes per task, the standard error on a task with 50% success is ~15.8 points, so the aggregate lead is within plausible evaluation noise. The JCAC contribution is mathematically sound (split conformal with a single max score over the executed prefix), but the headline SOTA claim is the load-bearing part: if the comparison is not protocol-equivalent, the abstract's central claim weakens substantially, even though the calibration method stands on its own.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FabriVLA, a 0.88B-parameter vision-language-action model that truncates InternVL3.5-1B to 14 LM layers, fuses layer 6 with layer 14, and uses gated self-attention in an 8-block flow-matching action head; it is trained end-to-end in one stage without robot-data pretraining. On Meta-World MT50 the authors report 90.0% average success, which they call state-of-the-art over seven recent VLAs. The second contribution, JCAC, trains a small residual-scale head on a frozen policy's residuals, then uses split conformal calibration with a max-normalized-residual score to build a joint prediction set over the executed He x d action prefix. Experiments report coverage and radius on held-out expert states, ranking AUROC for failure and constraint events on LIBERO-Safety, and a real-robot cube-to-tray deployment with a novelty diagnostic.","tokens_in":13281,"tokens_out":7084,"duration_ms":64051,"significance":"If the SOTA comparison held, the paper would make a useful empirical case that a compact VLA can match much larger models, and JCAC would provide a simple, inexpensive way to attach distribution-free prefix-level uncertainty to a frozen policy. The conformal construction is standard split conformal, and the paper deserves credit for explicitly limiting coverage claims to exchangeable data and for labeling closed-loop / cross-policy / real-robot results as ranking measurements rather than coverage guarantees. The residual scale head is trained on a split disjoint from calibration, so no circularity is apparent. However, the headline SOTA claim rests entirely on cross-paper baseline numbers with no confidence intervals and no unified protocol, which is a load-bearing weakness.","major_comments":[{"comment":"The abstract's 'state-of-the-art 90.0%' claim depends on Table 1 baseline numbers quoted 'from the corresponding papers.' The lead over LA4VLA is 2.5 points (90.0 vs 87.5), and no confidence intervals are reported for either FabriVLA or the baselines. With 10 episodes per task, a per-task binomial standard error at 50% success is ~15.8 points; the aggregate 2.5-point margin is within plausible evaluation noise unless the protocols are exactly matched. The paper does not re-run baselines and gives only a vague 'same formula' statement. Please report CIs, provide per-task/per-tier episode counts and success criterion for each baseline, and either re-run baselines under the identical H5/400-step/tier-averaging protocol or soften the SOTA claim to 'competitive.'","section":"§5.2, Table 1"},{"comment":"Several load-bearing details are deferred to unavailable appendices: the full MT50 metric definition, the exact split sizes for JCAC training/calibration/test, the choice of sigma_min, the head architecture dimensions, hyperparameters, and the LIBERO-Safety confirmation protocol (development gate, freeze criteria). Without these, I cannot verify that the reported numbers follow the stated protocol or that the conformal calibration used a data-disjoint split exactly as claimed. Please include the supplementary material or move the essential protocol details into the main body.","section":"§4.3, §5.2 (Appendices A–D)"},{"comment":"The coverage set in Eq. (11) is defined on the normalized/padded action space, while the executed prefix is denormalized before execution (Section 3.5). The paper does not state how the normalized set C_alpha maps to the physical action prefix that 'reaches the robot.' As written, the claim that the set covers 'the whole executed action prefix' is ambiguous. If the set is meant to cover the denormalized actions, the residual score in Eq. (10) must be computed on denormalized residuals (or the normalization must be affine and explicitly inverted); otherwise the coverage guarantee does not apply to the executed actions. Please clarify.","section":"§3.5, §4.2, Eq. (11)"}],"minor_comments":[{"comment":"The '3.3x tighter' claim refers to mean radius at 88.80% coverage versus the global radius at 89.33% coverage. The text notes this, but the abstract does not; please state the coverage levels next to the ratio.","section":"Abstract, Table 2"},{"comment":"The perturbation experiments report coverage on held-out expert states, but exchangeability is broken under perturbation; please label these results as empirical and outside the formal guarantee, as is already done for the closed-loop and cross-policy results.","section":"§5.3"},{"comment":"The value of sigma_min is not given in the main text; please specify it (or point to the appendix) so the scale head floor is clear.","section":"§4.1, Eq. (8)"},{"comment":"The sentence 'Baseline averages are computed by the same formula' would benefit from an explicit statement of the tier-averaging formula and a check that each baseline used the same tier grouping; per-tier definitions can differ across papers.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The JCAC contribution is methodologically sound and carefully scoped; the main risk is the unsupported SOTA comparison. If the authors cannot re-run baselines under a unified protocol, they should soften the headline claim. The paper would benefit from releasing code and the supplementary appendices to support reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of FabriVLA. The genuinely useful piece is JCAC: a frozen policy, one query, an elementwise scale head, and a split-conformal quantile that covers the whole executed prefix at once. That is clean, standard, and correctly scoped. They are honest that the coverage guarantee only holds for exchangeable expert-state data, and they label the closed-loop and cross-policy results as ranking measurements rather than coverage. That is the right way to present it.\n\nThe architecture also has sensible ideas: zero-initialized gated self-attention among action tokens and shallow/deep layer fusion. The ablations suggest both help, though the numbers come from a shorter schedule and a different horizon, so they are not directly comparable to the main results. That is stated, which I appreciate.\n\nThe soft spot is exactly what the stress-test flags. Table 1's 90.0% SOTA claim rests on per-tier scores taken from other papers, with no shared protocol and no confidence intervals. The gap over LA4VLA is 2.5 points, and with 10 episodes per task, the noise floor is high. If the baselines used different episode counts, horizons, or success criteria, the lead could vanish or reverse. The paper needs either a re-run under a common protocol or at least a clear statement of protocol equivalence and per-task CIs. This is not a fatal flaw for the JCAC contribution, which stands independently, but it does weaken the abstract's central selling point.\n\nA related practical issue: no code or data is linked, and nearly all implementation details live in unavailable appendices. That makes the work hard to verify. The authors should release artifacts and supply the appendices.\n\nWho benefits from this paper? Someone working on lightweight VLAs for edge deployment, and anyone interested in cheap, calibrated uncertainty for action chunks. The method is simple enough to apply to other policies, and the LIBERO-Safety ranking experiments are a nice demonstration, even if they are only ranking.\n\nMy recommendation: send it to peer review. The JCAC core is sound, the writing is honest, and the architecture is worth examining. But I would not accept it without the artifacts and without addressing the protocol-equivalence question for the SOTA comparison. If the authors can supply those, the paper is acceptable; otherwise, the result should be framed as a strong method paper rather than an SOTA claim.\n\nIn short: credit where due, but the headline needs backing before it is taken seriously.","headline":"JCAC's conformal construction is sound and the model has sensible design choices, but the headline SOTA claim leans on unverified cross-paper baselines and no released artifacts.","tokens_in":13726,"tokens_out":1346,"would_cite":true,"duration_ms":15307,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 0.88B-parameter action model hits 90.0% on Meta-World MT50 and produces calibrated uncertainty sets from the same single query.","keywords":["vision-language-action model","flow matching","conformal prediction","action chunk uncertainty","robot manipulation","Meta-World","calibration","lightweight policy"],"falsifier":"To test the performance claim, re-run all compared VLA models on Meta-World MT50 with an identical evaluation script (10 episodes per task, 400-step horizon, fixed 5-step execution window, same success criterion and tier averaging) and see whether the 90.0% average still leads. To test the uncertainty claim, draw two disjoint calibration and test sets from the same expert-state distribution, calibrate the single conformal quantile on the first, and measure the joint prefix coverage on the second; if empirical coverage falls below the nominal level for large n, the exchangeability assumption or","tokens_in":12902,"feed_emoji":"🤖","tokens_out":8110,"duration_ms":64835,"temperature":0.7,"pith_summary":"FabriVLA is a compact vision-language-action policy that the authors train end-to-end in a single stage, without a separate robot-data pretraining phase, and they report 90.0% average success on the 50-task Meta-World benchmark with only 0.88B parameters. The paper's second claim is that a post-training head, JCAC, can make a frozen policy auditable: one forward pass returns both the action prefix and an elementwise error scale, and a single conformal quantile then covers every executed step and dimension at a user-chosen confidence level. The reported set is 3.3 times tighter in mean radius than an unconditional conformal set. If both claims hold, compact robot policies need not trade accuracy for size or silence, because the same query that produces an action also produces a usable bound on its error.","feed_headline":"0.88B robot policy hits 90% and returns calibrated error sets","feed_subtitle":"A single-stage VLA matches larger models on Meta-World MT50, and a conformal head bounds the whole action prefix from the same query.","key_machinery":"The load-bearing mechanism for performance is the combination of shallow layer fusion and gated action self-attention. Layer fusion concatenates the token sequences of the final retained layer (layer 14) and an intermediate layer (layer 6), projects the concatenation back to token width with an identity-initialized matrix, and feeds the result to every cross-attention block, so the head receives spatial detail as well as semantics at no cost beyond one linear layer. Gated self-attention multiplies the self-attention output by a learnable scalar initialized to zero; at initialization each block is exactly a cross-attention block, and the gate opens only as gradients warrant. For uncertainty,","core_discovery":"On the paper's own terms, the central discovery is that a vision-language-action model can be made small and self-auditing at once. FabriVLA keeps the first 14 layers of a pretrained vision-language backbone and fuses a shallow layer (layer 6) into the deepest retained layer (layer 14) through a learned linear projection, preserving fine-grained spatial detail alongside task semantics. Its flow-matching action head uses zero-initialized gated self-attention among action tokens, so inter-step structure appears only to the extent training justifies. Trained with a single-stage recipe on Meta-World MT50, the 0.88B-parameter policy reaches 90.0% average success, ahead of all compared baselines i","pith_inferences":["Since the formal guarantee only holds for exchangeable expert agreement, the strongest defensible use of JCAC in real deployment is as a ranking and prioritization signal; turning the radii into hard safety alarms would require separate closed-loop validation, which the paper itself stops short of.","The 3.3x tightness gain comes from conditioning the radius on the query; a natural next test is whether a head trained with a proper scoring rule directly on the maximum residual, rather than a Gaussian NLL on per-element residuals, yields even tighter sets at the same coverage.","The zero-initialized gate is a general architectural trick: any generative head that has a safe cross-attention baseline could start from that baseline and open self-attention only when training warrants, which may be worth testing on diffusion or other action heads outside this paper.","The MT50 comparison relies on per-tier scores quoted from published papers; if all models were re-evaluated under a shared protocol, the claimed lead could shrink or widen, so a common evaluation harness would be the next testable step."],"forward_implications":["If the results hold, a compact single-stage VLA can match or beat much larger models on multi-task manipulation, lowering the compute and latency threshold for real-time deployment.","A single forward pass can serve both control and audit: the same query that produces an action chunk also yields a calibration set, so uncertainty costs no additional policy evaluation.","The conformal guarantee is finite-sample and distribution-free under exchangeability, meaning the returned set has marginal joint coverage at the user-chosen confidence on data that resembles the calibration distribution.","The uncertainty radii rank episodes by risk before execution, enabling selective review, replanning triggers, or deferral rules in safety-critical settings.","The training recipe transfers unchanged from simulation to a real robot arm, suggesting the method is not tied to a particular simulator or domain."],"fun_headline_variants":["Conformal action bounds: 0.88B VLA scores 90% on MT50","90% success from 0.88B params with conformal action bounds","Calibrated action chunks: tiny VLA hits 90% on MT50","Tiny model, tight bounds: 0.88B policy scores 90% with risk ranking","0.88B VLA: 90% Meta-World win and calibrated risk sets"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise for the performance claim is that the per-tier MT50 scores reported by the compared models were measured under the same evaluation protocol as FabriVLA's (episodes, horizon, execution length, success flag, tier averaging); for the uncertainty claim, the load-bearing premise is that calibration and test expert-state decisions are exchangeable.","fun_headline_variants_meta":{"raw":{"variants":["Conformal action bounds: 0.88B VLA scores 90% on MT50","90% success from 0.88B params with conformal action bounds","Calibrated action chunks: tiny VLA hits 90% on MT50","Tiny model, tight bounds: 0.88B policy scores 90% with risk ranking","0.88B VLA: 90% Meta-World win and calibrated risk sets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001763,"raw_usage":{"total_tokens":6812,"prompt_tokens":782,"completion_tokens":6030,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":5927}},"tokens_in":526,"tokens_out":6030,"duration_ms":38851,"temperature":1.0,"reasoning_tokens":5927,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T04:22:43.748868+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"To test the performance claim, re-run all compared VLA models on Meta-World MT50 with an identical evaluation script (10 episodes per task, 400-step horizon, fixed 5-step execution window, same success criterion and tier averaging) and see whether the 90.0% average still leads. To test the uncertainty claim, draw two disjoint calibration and test sets from the same expert-state distribution, calibrate the single conformal quantile on the first, and measure the joint prefix coverage on the second; if empirical coverage falls below the nominal level for large n, the exchangeability assumption or","supporting_citations":[],"review_version":3}