{"id":"1e13a538-3c0d-4256-b8c3-7848b69a402d","arxiv_id":"2412.17348","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ORIGAMI is a generative transformer with structure-preserving tokenization, key/value position encoding, and grammar-constrained decoding that matches or beats baselines on tabular, multi-label, and code-classification tasks.","lead":"Researchers present ORIGAMI, a transformer model that learns directly from JSON-like data without flattening it into tables. It adds structure-aware position encoding and grammar checks, and reports competitive or better accuracy on tabular, multi-label, and code-classification benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CodeNet Java250 claim of beating CNN/GNN is not yet supported: split and preprocessing protocol are unspecified, so the 0.6-point margin over GNN may be an artifact.","rationale":"The reader's weakest assumption—that the CodeNet Java250 comparison may rest on a mismatched test split or preprocessing protocol—is exactly the most load-bearing concern. The paper's other headline results are not equally fragile: the DDXPlus multi-label comparison is internally consistent in direction, and the 1.8–2.2% margins over strong baselines are larger than the apparent reporting inconsistencies. The synthetic Dungeons experiment is explicitly a designed probe and is framed as an ablation, so its artificiality weakens generality claims but not the core architectural claim. By contrast, the CodeNet result is the only direct support for the Abstract's assertion that ORIGAMI outperforms specialized CNN/GNN models, and it rests on a single point estimate with no error bars, no explicit split definition, and a nonstandard UNKNOWN relabeling of discarded instances. Because the reader's verdict is already CONDITIONAL and this concern is the reason for conditionality, the verdict should remain unchanged. The concrete test—rerunning ORIGAMI on the official Puri et al. split and verifying the baselines were computed under the same protocol—would settle whether the 0.6-point margin is real or an artifact, and should be a stated condition for accepting the paper's strongest claim.","tokens_in":22925,"tokens_out":7916,"duration_ms":80341,"concrete_test":"Obtain the exact CodeNet Java250 train/test split used by Puri et al. [29] from the Project CodeNet repository or the paper's accompanying code. Re-run ORIGAMI's full pipeline on that split with the same 4000-token cap and UNKNOWN relabeling, report the number of discarded test instances, and compute accuracy over 5 seeds with confidence intervals. Independently confirm that the baseline numbers (MLP 71.0%, CNN 89.5%, GNN 94.1%) were produced on that identical test set and without dropping the same long instances. If ORIGAMI's accuracy on the exact split does not exceed 94.1%, or if the baseline GNN number changes under the same protocol, the claim of beating specialized models in Section 4.3 and the Abstract is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that ORIGAMI 'outperform[s] ... specialized models such as convolutional and graph neural networks on a code classification task' rests almost entirely on Table 3: 94.7% vs 94.1% for the best GNN on CodeNet Java250. Section 4.3 does not state which train/test split was used, how the 75,000 submissions were divided, or whether that split matches the one behind the baseline numbers taken from Puri et al. [29]. The section also truncates sequences at 4000 tokens, discards roughly 1% of instances, and relabels them as UNKNOWN during evaluation. If Puri et al.'s baselines were evaluated on a different test set, or without the same truncation/discard policy, the comparison is not apples-to-apples. A 0.6-point difference with no reported variance, no seed count, and no number of test instances is well within the range of split- and preprocessing-induced noise. The missing split statement is therefore not cosmetic: it is the only evidence supporting the headline claim that ORIGAMI beats convolutional and graph neural networks on this task.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ORIGAMI is a decoder-only transformer designed for end-to-end supervised learning on semi-structured JSON data. It makes three contributions: (1) a tokenizer that treats keys and primitive values as atomic tokens and adds grammatical tokens for objects and arrays; (2) a Key/Value Position Encoding (KVPE) computed as the sum of embeddings of the stack symbols of a pushdown automaton parsing the token stream, making the encoding invariant to the order of key/value pairs and compositional over nested key paths; and (3) PDA-based guardrails that mask invalid next tokens both during training and inference. Classification is cast as next-token prediction, so the same architecture handles single-label and multi-label targets and can generate array-valued outputs. Experiments cover eight UCI tabular benchmarks converted to JSON, the DDXPlus multi-label medical diagnosis dataset, code classification on CodeNet Java250, and ablations on a synthetic Dungeons key-lookup task. The authors report that ORIGAMI is competitive with GBDTs on the tabular benchmarks, outperforms multi-output baselines on DDXPlus, and, on CodeNet, exceeds MLP, CNN and GNN baselines (94.7% vs. 94.1% for the best GNN) while remaining below CodeBERT.","tokens_in":150,"tokens_out":9056,"duration_ms":106141,"significance":"Assuming the empirical claims hold, ORIGAMI is a useful and timely contribution: it is an end-to-end generative model for semi-structured data, with a principled position encoding (KVPE) grounded in a pushdown automaton and a clean mechanism for constrained training and inference. The evaluation is generally careful: 5-fold CV with hyperparameter search on the UCI benchmarks, 5-seed runs on an independent DDXPlus test set, and well-designed ablations (PE variants, guardrails, permutation upscaling) that provide falsifiable, machine-checkable evidence for each contribution. The main deficits are in the CodeNet experiment, where the test split and preprocessing protocol are unspecified and the 0.6-point margin over the best GNN is within the range of split and preprocessing noise, and in the Dungeons ablation, which is deliberately constructed to favor KVPE. These issues do not invalidate the architecture but currently prevent the paper from supporting its strongest headline claim.","major_comments":[{"comment":"The claim that ORIGAMI 'outperform[s] ... specialized models such as convolutional and graph neural networks on a code classification task' (Abstract) rests on Table 3, but the evaluation protocol is not specified. The text does not state how the 75,000 CodeNet Java250 submissions are partitioned into train/test, how many test instances ORIGAMI is evaluated on, or whether the split coincides with the one underlying the baseline numbers taken from Puri et al. [29]. In addition, ORIGAMI truncates sequences and vocabulary at 4000 tokens, discards roughly 1% of instances, and, per the text, classifies discarded instances as UNKNOWN, whereas the baselines' preprocessing is not described. Given a 0.6 percentage-point margin over the best GNN with no variance, no seed count, and no test-set size, the comparison is not yet apples-to-apples. Please report the split, the number of test instances, the standard deviation over several seeds, and either rerun the baselines under identical preprocessing or demonstrate that Puri et al.'s published split and preprocessing match.","section":"4.3, Table 3"},{"comment":"The Dungeons experiment is constructed in a way that strongly favors ORIGAMI: the target depends on a key-path lookup (door_no then key_color), while the flattened representation given to tabular baselines uses positional column names such as corridor.4.blue_key, which are not invariant to the shuffling of door objects. Hence the tabular baselines cannot solve the task regardless of their capacity; the 100%-vs-32-41% result is a mechanism check for KVPE, not evidence that ORIGAMI generally outperforms tabular models on semi-structured data. I recommend stating this explicitly and, if feasible, adding a variant where tabular baselines are given the correctly aligned key paths, so the comparison isolates the effect of structure preservation.","section":"4.4.2, Figures 5 and 7"}],"minor_comments":[{"comment":"Five independent runs with different seeds are reported with 0.0% standard deviation for F1 on every model, while precision and recall show 0.1-0.2% standard deviations; please clarify whether F1 (and the other metrics) are computed on pooled predictions per run or whether the 0.0% values are rounding artifacts. Also state whether F1 is micro- or macro-averaged.","section":"Table 2"},{"comment":"The KVPE encoding drops the array-position symbol for the last element of an array: for an array of length n, the stack contains Array(n) for the first element, Array(n-1) for the second, ..., and no array symbol for the last element, so the last element is encoded identically to a scalar value under the same key path. The paper acknowledges this in a footnote but does not explain why this boundary case does not affect the benchmarks; please either fix the encoding (e.g., with a dedicated LAST_ARRAY_ITEM symbol) or provide evidence that this quirk does not impact the reported results.","section":"3.4, footnote 3"},{"comment":"The decoding procedure for multi-label outputs is underspecified: when the model emits an Array(n) token, the paper says it 'greedily extracts n additional tokens', but it should state how n is determined and how invalid or repeated labels are handled. Also, the MOC baselines are fitted on a binary matrix while ORIGAMI directly predicts the label list; this is a fair difference, but the paper should explicitly note that the baselines do not receive the label co-occurrence structure.","section":"4.2"},{"comment":"The CodeNet hyperparameters are listed with a dagger indicating they were excluded from the hyperparameter search, but no justification or selection procedure is given; please state how these values were chosen (e.g., from prior experiments or a small validation sweep).","section":"4.3, Table 6"},{"comment":"The abstract and Section 3.1 describe the tokenization as 'reversible', but reversibility holds only for sequences that represent valid objects, as the footnote in Section 3.3 notes. Please qualify the claim, for instance, as 'reversible on the language of valid token sequences'.","section":"3.1, Abstract"},{"comment":"The figures would be easier to interpret with error bars or shaded confidence bands around the averaged curves, particularly Figure 4 and Figure 8, and with larger axis labels. As printed, the 'O' markers for ordered sequences and the many thin individual-run lines are hard to distinguish.","section":"Figures 4, 6-8"},{"comment":"The novelty statement 'to the best of our knowledge this work is the first to utilize the same approach during training to accelerate convergence' should be softened or supported by a broader comparison with the guided-decoding literature, since training-time masking with grammars has been explored in other settings.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The CodeNet experiment is the main risk to the paper's headline claim: the test split is unspecified and the 0.6-point margin over the best GNN is narrow. I would ask the editor to require the split definition and an uncertainty estimate before publication. The Dungeons experiment should be reframed as a mechanism check rather than a general superiority result. The rest of the empirical work is solid and the ablations are well designed; the paper is a strong candidate after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper. First, the architecture is a credible piece of engineering: a tokenizer that keeps JSON keys and values atomic, a position encoding (KVPE) derived from the stack of a pushdown automaton, and grammar masking applied during training and inference. The ablations are careful and mostly support the component choices. Second, the headline claim that ORIGAMI beats convolutional and graph neural networks on CodeNet Java250 is not yet supported, because the test split and preprocessing protocol are unspecified. The 0.6-point margin over the GNN could easily be split- or preprocessing-induced noise.\n\nWhat is genuinely new is the combination: KVPE is a sensible way to encode hierarchical key paths and array positions in a permutation-invariant manner, and using the PDA both for position encoding and for masking invalid next tokens during training is a clean idea. The order-permutation upscaling is a reasonable adaptation of XLNet-style factorization to JSON, and the Dungeons synthetic probe is a good way to isolate the value of the position encoding. The DDXPlus multi-label results are strong and appear well controlled, with multiple seeds and a clear advantage for the object-structured representation.\n\nNow the soft spots, in proportion. The CodeNet issue is real: Section 4.3 does not state which split of the 75,000 submissions was used, how it relates to Puri et al.'s baselines, or how many test instances were evaluated. The paper also truncates sequences at 4000 tokens and discards about 1% of instances, relabeling them as UNKNOWN during evaluation; if the baselines were not subject to the same policy, the comparison is not apples-to-apples. That should be fixed before the claim is taken at face value. Second, Table 2 shows zero-variance F1 for all models while precision and recall have non-zero variance; that inconsistency needs an explanation or a correction. Third, the footnote in Section 3.4 admitting that the last array element has no array symbol on the stack is waved away with 'does not affect model performance' but no evidence is given. Fourth, there is no code release, which limits reproducibility. Finally, the Dungeons dataset is a designed probe, not a real-world benchmark; the 'only ORIGAMI generalizes' framing is fine as an ablation result, but should not be read as evidence of general superiority.\n\nThe central argument holds up: the architecture does what it claims, and the ablations support the importance of the components. This deserves a serious referee. I would send it to review and ask the authors to clarify the CodeNet split, report variance or seeds on that comparison, and fix the Table 2 inconsistency.","headline":"Well-engineered transformer for JSON with a solid ablation story, but the CodeNet comparison needs a split statement before the headline claim is supported.","tokens_in":23706,"tokens_out":3701,"would_cite":true,"duration_ms":31851,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68Q45"],"pacs":[],"model":"deepseek-v4-flash","headline":"A generative transformer that consumes JSON directly as key/value token sequences with stack-based position encodings matches or beats tabular and specialized baselines, and reformulates classification as next-token prediction.","keywords":["semi-structured data","JSON","generative transformer","key/value position encoding","pushdown automaton","constrained decoding","next-token prediction","multi-label classification"],"falsifier":"Re-run CodeNet Java250 using the exact official train/test split and preprocessing of the published baselines, applying the same UNKNOWN relabeling of discarded long instances to every method; if ORIGAMI's accuracy no longer exceeds the best GNN, the headline advantage is an evaluation artifact. In a second check, enumerate all key paths of a dataset and test whether any two distinct pushdown-automaton stack states produce identical KVPE embedding sums; a collision would falsify the paper's 'unique' encoding claim.","tokens_in":2030,"feed_emoji":"📦","tokens_out":2761,"duration_ms":91375,"temperature":0.7,"pith_summary":"ORIGAMI is a decoder-only transformer that learns directly from JSON objects by turning each object into a token sequence and training the model to predict the next token; classification becomes a generation problem, so single-label and multi-label predictions need no architectural change. The paper's claim is that this works only if the architecture respects the data's structure, which it does through three components: a reversible tokenizer that keeps keys and values atomic, a position encoding computed from the stack of a pushdown automaton that parses the token stream, and grammar-based masking that forbids invalid tokens during both training and inference. On tabular benchmarks converted to JSON, ORIGAMI matches gradient-boosted trees on average; on the DDXPlus medical-diagnosis dataset it beats multi-output baselines; and on CodeNet Java250 code classification it reports 94.7% accuracy, above an MLP, CNN, and the best GNN, though below a much larger pre-trained CodeBERT. A synthetic 'Dungeons' task, where the correct label requires looking up a key path in a shuffled array, is solved to 100% test accuracy by ORIGAMI while flattened baselines stay near chance. The upshot is that flattening semi-structured data may be unnecessary: a structural inductive bias plus grammar constraints can let a small generative model outperform specialized architectures.","feed_headline":"A transformer that reads JSON natively beats graph nets on code","feed_subtitle":"Stack-based position encoding plus grammar guardrails let a generative model handle single- and multi-label tasks.","key_machinery":"The load-bearing mechanism is the key/value position encoding (KVPE). For each input token, a deterministic pushdown automaton -- the same automaton that recognizes the language of valid JSON token sequences -- records its stack state, and KVPE encodes that state as the sum of the embeddings of the stack symbols; the sum of the root marker, the current key, and the current array position represents the token's full key path independently of where it appears in the linear sequence. This is what makes sibling order irrelevant, enables sampling random permutations of key/value pairs during training, and lets the model answer queries about any key at inference time. The second mechanism is grammar-constrained decoding: the same automaton's transition masks set invalid next-token logits to $-\\infty$, so the model never wastes capacity on learning grammar and convergence on the Dungeons task is reached in 303 training steps on average versus 495 without guardrails.","core_discovery":"Stated on its own terms, the paper's discovery is a recipe for end-to-end supervised learning on JSON: represent every object as a depth-first token sequence of key, value, and structural tokens; add to each position a key/value position embedding formed by summing the embeddings of the pushdown-automaton stack symbols that are active while parsing that token; and use the automaton to mask out next tokens that would violate JSON grammar, during training as well as inference. Because the position encoding depends only on the stack, not on absolute token order, sibling key/value pairs can be shuffled without changing the model's input representation, which the paper exploits as a regularization and data-upscaling technique. The reformulation of classification as next-token prediction then lets a single model output a single label, an array of labels, or even a nested object, and the paper's experiments claim this beats the json2vec baselines on 7 of 8 JSON-ified benchmark datasets, outperforms multi-label baselines on DDXPlus, and outperforms an MLP, CNN, and GNN on the CodeNet Java250 classification task while trailing only the far larger pre-trained CodeBERT.","pith_inferences":["Beyond the paper, the summed-stack-state encoding is not shown to be collision-free: two different key paths could in principle yield the same KVPE vector, which would break the 'unique position embedding' claim; counting collisions on real datasets would settle this.","Beyond the paper, ORIGAMI's recipe is portable: KVPE could be attached to any transformer decoder as a drop-in positional encoding for structured tokens, potentially improving models that currently flatten JSON before embedding.","Beyond the paper, the CodeNet gap to CodeBERT (94.7% vs 97.4%) is attributed by the authors to scale and pre-training, which implies that a pre-trained or larger ORIGAMI-style model -- or one fine-tuned from a language model checkpoint -- could close that gap while retaining native JSON structure.","Beyond the paper, the same generative setup could support unsupervised tasks the paper only names, such as cardinality estimation for document databases or synthetic JSON generation, since the model already approximates the full joint distribution over object tokens."],"forward_implications":["If ORIGAMI's results hold, nothing about a prediction task requires flattening semi-structured data first: deeply nested objects such as ASTs can be fed to a transformer in native form, avoiding feature matrices with over a million columns.","Multi-label and variable-size outputs become free: an $\\text{Array}(n)$ token tells the model to keep sampling labels, so one architecture handles single-label, multi-label, and structured outputs.","Order-invariant position encoding plus key permutation upscaling is a data-efficient regularizer: on the small contraceptive dataset mean accuracy rises from 50.85% to 54.0%, and the gains are largest where data is scarce.","Grammar masking during training accelerates convergence by about 39% on the synthetic task, suggesting the same trick will speed up any structured-output model where outputs must obey a formal grammar.","Because predictions for any key of an object are available from one trained model, the same framework can auto-complete partial documents or predict missing fields, applications the paper explicitly flags as next steps."],"supporting_citations":[{"why":"Defines JSON's unordered key/value semantics, which motivates the order-invariant KVPE and the shuffling objective.","marker":"[4]"},{"why":"Provides the json2vec baselines and the JSON-ification procedure for the eight UCI benchmarks ORIGAMI is compared against.","marker":"[45]"},{"why":"Supplies the CodeNet Java250 dataset and the published MLP/CNN/CodeBERT/GNN accuracy numbers that the code-classification comparison depends on.","marker":"[29]"},{"why":"The standard decoder-only transformer architecture that ORIGAMI modifies with its tokenizer, KVPE, and guardrails.","marker":"[40]"},{"why":"Establishes grammar-constrained decoding with pushdown automata during inference, which ORIGAMI extends by applying the same masking during training.","marker":"[43]"},{"why":"Shows permutation-based factorization-order training for autoregressive models, which ORIGAMI adapts via sampled key/value orderings.","marker":"[48]"},{"why":"The DDXPlus dataset used for the multi-label diagnosis experiment where ORIGAMI reports higher F1 than multi-output baselines.","marker":"[8]"}],"fun_headline_variants":["JSON-native transformer beats graph nets on code classification","ORIGAMI transformer reads JSON natively, outperforms GNNs on code","Generative transformer for JSON hierarchy beats graph networks on code","End-to-end JSON transformer wins on multi-label and code classification","ORIGAMI: stack-based encodings enable JSON-native transformer, beats GNNs"],"cache_read_input_tokens":25728,"weakest_assumption_plain":"The headline comparison against convolutional and graph networks depends on baseline accuracy numbers taken from the CodeNet paper that may come from a different test split, and only ORIGAMI's dropped long examples are counted as errors, so the lead over the best graph network could come from evaluation mismatch.","fun_headline_variants_meta":{"raw":{"variants":["JSON-native transformer beats graph nets on code classification","ORIGAMI transformer reads JSON natively, outperforms GNNs on code","Generative transformer for JSON hierarchy beats graph networks on code","End-to-end JSON transformer wins on multi-label and code classification","ORIGAMI: stack-based encodings enable JSON-native transformer, beats GNNs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000656,"raw_usage":{"total_tokens":3030,"prompt_tokens":1001,"completion_tokens":2029,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":1937}},"tokens_in":617,"tokens_out":2029,"duration_ms":13786,"temperature":1.0,"reasoning_tokens":1937,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:34:26.682614+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run CodeNet Java250 using the exact official train/test split and preprocessing of the published baselines, applying the same UNKNOWN relabeling of discarded long instances to every method; if ORIGAMI's accuracy no longer exceeds the best GNN, the headline advantage is an evaluation artifact. In a second check, enumerate all key paths of a dataset and test whether any two distinct pushdown-automaton stack states produce identical KVPE embedding sums; a collision would falsify the paper's 'unique' encoding claim.","supporting_citations":[{"cited_title":"The JavaScript Object Notation (JSON) Data Interchange Format","cited_arxiv_id":null,"evidence_quote":"Defines JSON's unordered key/value semantics, which motivates the order-invariant KVPE and the shuffling objective."},{"cited_title":"A Framework for End-to-End Learning on Semantic Tree-Structured Data","cited_arxiv_id":"2002.05707","evidence_quote":"Provides the json2vec baselines and the JSON-ification procedure for the eight UCI benchmarks ORIGAMI is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CodeNet Java250 dataset and the published MLP/CNN/CodeBERT/GNN accuracy numbers that the code-classification comparison depends on."},{"cited_title":"Willard and Rémi Louf","cited_arxiv_id":null,"evidence_quote":"Establishes grammar-constrained decoding with pushdown automata during inference, which ORIGAMI extends by applying the same masking during training."},{"cited_title":"DDXPlus: A new dataset for automatic medical diagnosis","cited_arxiv_id":null,"evidence_quote":"The DDXPlus dataset used for the multi-label diagnosis experiment where ORIGAMI reports higher F1 than multi-output baselines."}],"review_version":1}