{"id":"e9f37022-5feb-403d-a000-7fe1d0b2ce0b","arxiv_id":"2508.11121","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Tafo, a neuro-symbolic system, predicts spreadsheet conditional-formatting rules including colors with no user input, and its authors report it matches user-applied formatting better than all tested baselines.","lead":"Researchers built Tafo, a system that guesses which cells in a spreadsheet you want to highlight and how, without requiring a rule or explanation from you. It combines an enumeration-based rule engine with a large language model, and was tested against formatting in 1.8 million real Excel workbooks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Format learner can retrieve the ground-truth rule/table from the same corpus, so end-to-end gains may reflect memorization rather than generalization.","rationale":"The reader's verdict is CONDITIONAL, and I agree with the overall verdict, but I elevate a different, more precise risk. The reader's weakest assumption (noisy oracle) is real but applies symmetrically to all baselines and to both condition and format metrics; it speaks to external validity. The format-retrieval leakage is an internal-validity threat: if the query table is in the retrieval pool, the format learner's reported advantage may be an artifact of retrieving the ground-truth answer. This is strongly suggested by the ablation in Table 7, where removing corpus retrieval costs 9-19 points, the largest ablation effect; that is exactly what one would expect if retrieval is returning the answer key. The paper never specifies a sanitization step, and the missing appendix and artifact release mean no code is available to check. The central claim that Tafo can automate over 50% of tasks with 3 suggestions and the 15.6-26.5% advantage over baselines should therefore not be accepted as evidence of generalization until the sanitized retrieval test is run. If leakage is confirmed, the verdict should move toward REJECT or the claim should be re-scoped to 'retrieval-augmented format suggestion' rather than predictive learning. If the test shows stable performance, the CONDITIONAL verdict can be upgraded to ACCEPT. The paper has genuine value: a new task formulation, a three-generator neuro-symbolic architecture, and a large-scale benchmark; those contributions can stand even if the format-learner claim needs re-scoping. My partial agreement with the reader reflects that we both flag the benchmark, but my concern is more specific and more directly damaging to the internal comparison.","tokens_in":20338,"tokens_out":4385,"duration_ms":45838,"concrete_test":"Re-run the format-learning and end-to-end evaluations (Table 4, Figure 11) with retrieval sanitized: for each test task, exclude from the corpus (i) the exact workbook containing the query table and (ii) any workbook whose header/formula/rule similarity exceeds the mining threshold λ_T, as measured by the same weighted Hamming distance used in Section 4.4. Then recompute exact/property match and end-to-end execution match. Also report the percentage of test tasks for which the ground-truth format was present in the retrieved set f_r before sanitization. If exact/property match or the end-to-end number drops materially (e.g., more than 5 points), or if the gap to GPT-4/FormaT5 closes, the headline results are inflated by leakage; if the numbers are stable, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing weakness is in the format learner, not in the oracle assumption. Section 4.4 mines candidate formats f_r from 'similar tables and rules in the corpus', ranking rule-table pairs by weighted Hamming distance over headers, formulas, predicates, and constants. The benchmark (Section 5.1) is drawn from the same 1.8M-workbook corpus, with 25K held-out tasks. The paper never states that the query workbook itself, or workbook-level near-duplicates, are removed from the retrieval pool. Since the query table is by construction the closest match to itself, its user-applied format can enter f_r directly. The format learner would then be reading the answer key: Table 4's exact/property match and Figure 11's end-to-end 50% would reflect retrieval of the ground-truth format rather than generalization from similar tables. This is load-bearing because the headline 15.6-26.5% improvement and the '3 suggestions automate 50%' claim depend on format quality, and it is the one component where the evaluation protocol is explicitly vulnerable to self-retrieval. The noisy-oracle concern raised by the reader affects external validity equally for all systems; this leakage would invalidate the internal comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Tafo, a neuro-symbolic system for predicting conditional formatting (CF) suggestions for spreadsheet tables without any user specification. Tafo generates rule conditions through three complementary mechanisms — a purely symbolic enumerator, an LLM-based multi-step reasoning generator, and a neuro-symbolic combination — then ranks candidates using execution-based clustering and learns visual formats by retrieving similar (table, rule) pairs from a corpus. The system is evaluated on 25K CF tasks and 100K manually formatted column tasks extracted from 1.8 million public Excel workbooks. The authors report that Tafo outperforms a range of symbolic and neural baselines by 15.6%–26.5% on execution match, achieves 50% end-to-end execution match with three suggestions, and attains higher format exact/property match and condition diversity. They also present ablations of the condition generator, the ranker, and the format learner, as well as robustness checks on sample size and data type.","tokens_in":20621,"tokens_out":4417,"duration_ms":46482,"significance":"The paper introduces a practically important task — fully predictive formatting suggestion — and a technically plausible architecture that combines symbolic synthesis, LLM semantic knowledge, and corpus-based format retrieval. The evaluation is broad: it compares against six symbolic and seven neural baselines, uses multiple matching metrics (exact, sketch, execution; color and property match), and includes diversity, coverage, complexity, latency, and ablation analyses. If the reported gains are reproducible after addressing the methodological concerns below, Tafo would be a meaningful advance for spreadsheet automation and for neuro-symbolic recommenders more generally. The strengths are the clear problem formalization, the systematic baseline selection, and the careful construction of the CF and manual-formatting benchmarks from a large real-world corpus.","major_comments":[{"comment":"The format learner retrieves candidate (table, rule) pairs from the same 1.8M-workbook corpus that also defines the evaluation ground truth, and the paper does not state that the query workbook or its near-duplicates are excluded from the retrieval pool. Since a table is by definition the most similar item to itself, the format learner can directly copy the user's own format for that table, which would inflate the format-match numbers in Table 4 and the end-to-end numbers in Figure 11. This is load-bearing for the headline claims of 50% automation and 15.6%–26.5% relative gains. Please state explicitly whether the target workbook is removed from the retrieval set, and if not, re-run the format learner with a retrieval pool that excludes the target workbook and near-duplicates (e.g., by workbook ID or min-hash deduplication) and report the resulting Table 4 and Figure 11 numbers.","section":"Section 4.4 and Section 5.1"},{"comment":"The headline comparisons are reported as point estimates with no confidence intervals or significance tests. For example, Table 3 shows a top-5 execution match of 64.3% for Tafo versus 58.3% for GPT-4, and Table 4 shows top-1 exact format match of 13.2% versus 9.5% for FormaT5; with 25K tasks these differences may be significant, but without noise estimates the reader cannot assess reliability. Please report the number of task-level replicates, compute bootstrap confidence intervals or paired significance tests for the main metrics, and clarify what the 'Error Bounds' in Figure 10 represent (currently undefined).","section":"Section 6, Tables 3–4, Figures 10–11"},{"comment":"The benchmark treats user-applied formatting as the ground-truth target with no check for formatting that is stylistic, idiosyncratic, or erroneous. This assumptions affects the external-validity interpretation of 'automating over 50% of user formatting tasks' — a system might match arbitrary user color choices rather than semantically meaningful highlighting. I recommend reporting a human evaluation on a sample of tasks to measure whether the top-ranked suggestions are judged useful, and/or an analysis on a subset of tasks where user formatting follows common semantic patterns (e.g., green for positive, red for negative, bold for maxima). At minimum, a limitation paragraph acknowledging this issue should be added.","section":"Section 5.1 and Section 6.1.1"}],"minor_comments":[{"comment":"The word 'accross' should be 'across'.","section":"Table 4 caption"},{"comment":"The system name is written as both 'TaFo' and 'Tafo'; please use one consistent spelling throughout.","section":"Abstract and body"},{"comment":"The y-axis label 'TabuTran' appears to be a typo; it should likely read 'Tafo'.","section":"Figure 15"},{"comment":"The manual-formatting automation numbers are inconsistent: Section 1 says 'over 50%', Figure 10 shows 55% at top-3, and Section 7 says 'over 60%'. Please reconcile these claims.","section":"Section 6.1.1 and Section 7"},{"comment":"There are unresolved placeholders: 'Implementation details are in Appendix??' and 'Figure?? shows...' must be replaced with actual references.","section":"Section 5.3 and Section 6.3"},{"comment":"The CCS Concepts and Keywords fields contain placeholder text ('Do Not Use This Code', 'Do, Not, Us, This, Code, Put, the, Correct, Terms, for, Your, Paper') and must be replaced.","section":"CCS Concepts and Keywords"},{"comment":"The phrase 'After the conidition' contains a typo; it should be 'After the condition'.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The retrieval-leakage concern in Section 4.4 is the primary gate for acceptance; without an explicit exclusion of the query workbook from the format-retrieval pool, the format and end-to-end results are not interpretable. The paper also needs statistical rigor before it can be accepted. The placeholder text and inconsistent naming/numbers suggest this is an early preprint; a thorough revision is needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the Tafo paper. The new thing here is real: fully predictive conditional formatting — no examples, no NL — with a neuro-symbolic pipeline that combines symbolic predicate enumeration, LLM-generated semantic hints, and execution-based diversity ranking. The evaluation is large-scale (1.8M workbooks, 25K held-out CF tasks plus 100K manual-format tasks), and the condition-learning results in Table 3 look internally consistent: Tafo beats the best neural baseline by roughly 5-7 points on execution match at top-1. The format learner that mines similar tables for candidate colors and shades is a sensible idea, and the ablation showing corpus retrieval matters is informative. Full credit for tackling a task nobody else has defined cleanly.\n\nBut there are two soft spots that matter.\n\nFirst, the format learner may be reading the answer key. Section 4.4 retrieves candidate formats from similar tables and rules in the corpus using weighted Hamming distance on headers, formulas, predicates, and constants. The evaluation tasks are drawn from that same corpus. The paper never states that the target workbook or its near-duplicates are excluded from the retrieval pool. Since a table is most similar to itself, the user's own format can enter the candidate set directly. That would inflate format quality numbers and, in turn, the end-to-end 50% claim — and it would do so specifically for Tafo, since the baselines do not do corpus retrieval. This is not a minor nit; if the authors cannot rule out self-retrieval, the central comparison in Table 4 is compromised.\n\nSecond, the numbers do not quite add up. Table 3 gives Tafo top-3 execution match for conditions at 46.8%. Table 4 gives top-3 exact format match at 18.9% and property match at 47.2%. Yet Figure 11 and the text claim 50% end-to-end execution match with three suggestions. Since end-to-end is defined as condition execution match AND format color match, 50% is suspiciously high. Either the metric is defined differently (e.g., measured only over tasks where the condition matched), or there is an evaluation-set mismatch. The paper needs to clarify this.\n\nMinor issues: no error bars or significance tests; several dangling references (Appendix??, Figure??); no artifact release mentioned; and the manual-formatting benchmark lacks a simple baseline like repeating the most common existing format in the column. The noisy-oracle concern — user formatting is not always semantically meaningful — is real but applies equally to all systems, so it is an external-validity caveat rather than a fatal flaw.\n\nWho is this for? Researchers in spreadsheet automation and neuro-symbolic program synthesis. The task formulation is worth publishing, and the pipeline has reusable ideas, especially execution clustering and LLM-to-symbolic hints. But the paper should not be accepted until the retrieval leakage and the metric discrepancy are addressed.\n\nYes, it deserves a serious referee — the novelty is sufficient. My own read, though, is that the headline accuracy numbers are probably too high as stated.","headline":"New task and a well-engineered neuro-symbolic pipeline, but possible format-retrieval leakage and a suspiciously high end-to-end number need resolution before the headline claims hold.","tokens_in":790,"tokens_out":844,"would_cite":false,"duration_ms":39990,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Spreadsheet conditional-formatting rules can be predicted from the table alone, with no user examples or instructions.","keywords":["conditional formatting","predictive formatting","spreadsheet automation","program synthesis","neuro-symbolic systems","large language models","execution-guided ranking","table formatting"],"falsifier":"Take a fresh sample of spreadsheets with conditional-formatting rules, ask the users who wrote them to rate whether Tafo's top-three suggestions capture what they intended, and compare that acceptance rate with execution match; if accepted suggestions are far rarer than execution match, the benchmark's ground truth overstates practical automation.","tokens_in":20147,"feed_emoji":"📊","tokens_out":11613,"duration_ms":112780,"temperature":0.7,"pith_summary":"The paper sets out to show that spreadsheet conditional formatting can be made fully predictive: given a table and a target column, a system called Tafo suggests both the condition that triggers a style change and the visual format to apply, without the user supplying examples, natural-language instructions, or any other intent. The authors argue this matters because most users either do not know which rules to create or find the rule interface difficult, and prior automation required some form of specification. On a benchmark of 105K conditional-formatting tasks and 100K manually formatted columns drawn from 1.8 million public workbooks, Tafo matches the user's own formatting on over half of the tasks with just three suggestions. It also reports execution-match accuracy of 36.7%, 46.8%, and 64.3% at top-1, top-3, and top-5 suggestions, with 15.6 to 26.5 percentage points higher execution-match accuracy than the adapted neural and symbolic baselines.","feed_headline":"Three suggestions match user formatting on half of spreadsheet tasks","feed_subtitle":"No examples or instructions: it predicts both the rule and its colors from the table alone.","key_machinery":"The mechanism that carries the argument is a three-way neuro-symbolic generator followed by execution-based ranking. The symbolic arm performs component-based synthesis over typed static properties, including counts of blanks, errors, and formulas, most common values, statistical summaries, and text and date predicates, with beam search driven by a learned ranker. The neural arm prompts a large language model in a fixed four-step chain of thought: pick relevant columns, propose predicates and functions, list constants, then write rules; Tafo parses these generations into predicate components that re-enter the symbolic enumerator with an extra 10% score boost. The ranking step executes every candidate rule on the table, clusters rules whose boolean output vectors are identical into execution-equivalence classes, scores each cluster by the average member score, and samples round-robin across clusters, so the final suggestions differ in the cells they highlight. The format learner is a separate corpus-retrieval step that finds similar table-rule pairs by comparing headers, formulas, and rule structure with a weighted Hamming distance, mines their format identifiers, and grounds the chosen color to the shade used by at least 75% of the sheet's formatted cells.","core_discovery":"The central discovery, on the paper's own terms, is that the two halves of a conditional-formatting rule, the trigger and the format, can be learned together from the data itself. Tafo does this by pooling candidates from three generators: a symbolic enumerator that builds predicates from typed static properties, a large-language-model generator that reasons about the table in four steps, and a neuro-symbolic generator that feeds the LLM's outputs back into the symbolic search with boosted weight. All candidates are executed on the table and clustered by the set of cells they highlight; clusters are scored and sampled round-robin, which is what makes the final shortlist both accurate and diverse. The format half is learned by retrieving similar table-rule pairs from a large corpus, mining the format identifiers used there, weighting formats already present in the current sheet double, and grounding colors to the sheet's dominant shade. Evaluated on 105K conditional-formatting tasks, Tafo reaches 64.3% execution match at top-5, 46.8% at top-3, and about 50% end-to-end condition-plus-format match with three suggestions.","pith_inferences":["Inference: the execution-match metric makes the rule itself interchangeable with any other rule that highlights the same cells, so a production system could present highlighting patterns directly to users and only generate a symbolic rule when a user asks to see or edit it.","Inference: the same corpus-retrieval machinery that finds similar tables for formats could be turned into a recommender for other data-dependent artifacts such as charts, pivot tables, or data-validation rules, and the paper's own reliance on similar-table sharing gives a route to test that.","Inference: the shade-grounding heuristic, which uses a shade if 75% of the sheet's formatted cells use it, is a simple personalization rule; replacing it with a per-user or per-document learned color model is a direct way to test whether top-1 format match can move beyond the reported 13.2% exact match.","Inference: because the ground truth is formatting users actually applied in public workbooks, the headline automation rate is an upper bound on what users would accept; a deployment study measuring whether users keep the suggestions would be a sharper test of practical value."],"forward_implications":["If the accuracy numbers hold, spreadsheet users can receive useful conditional-formatting suggestions without writing anything: three suggestions reproduce the user's own formatting on more than half of both rule-based and manually formatted columns.","Because suggestions are ranked by execution, the shortlist is not a set of near-duplicates: two suggestions already cover over 80% of the column's cells, so a user scanning three options sees rules that highlight different parts of the data.","The two halves of formatting are separable but coupled: a rule trigger can be suggested even when the user never wrote a rule, since the same pipeline matches manually highlighted cells, and the format can be suggested even when the condition is simple.","The ablations imply the design's main levers: removing symbolic property extraction costs about 19.5 points of top-1 execution match, and removing execution-guided ranking costs about 20.4 points at top-5.","The corpus-driven format learner implies that format suggestions are not fixed templates but adapt to the table's existing style, so the same condition can be suggested with the user's shade of green rather than a default color."],"supporting_citations":[{"why":"Supplies the 1.8M-workbook corpus and the de-duplicated 105K-task benchmark, and the example-based rule-learning system that is a principal baseline to beat.","marker":"[40]"},{"why":"The natural-language-plus-examples formatting system used as the strongest prior baseline, plus the evidence that users struggle to give complete formatting specifications.","marker":"[36]"},{"why":"The chain-of-thought prompting technique that the neural generator uses to produce predicates, constants, and rule suggestions.","marker":"[49]"},{"why":"The large language model used both as a neural baseline and as the engine behind Tafo's neural and neuro-symbolic generators.","marker":"[31]"},{"why":"An open code language model adapted as a neural baseline for generating both conditions and formats from table prompts.","marker":"[47]"},{"why":"Supplies the embeddings used to measure condition diversity and to retrieve few-shot examples for the LLM prompt.","marker":"[9]"},{"why":"The finding that similar tables share artifacts like formulas, which the format learner extends by mining similar tables for format properties.","marker":"[53]"}],"fun_headline_variants":["Tafo auto-formats tables: 50% match with three suggestions","Conditional formatting predicted without examples or prompts","Spreadsheet rules and styles learned from data alone","Neuro-symbolic model suggests table formats, no user input","Three format suggestions match half of user rules"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy numbers treat the formatting users actually applied in 1.8 million public workbooks, their conditional-formatting rules and manually highlighted cells, as the correct target, so the claim assumes this real-world formatting encodes shared, desirable semantics rather than idiosyncratic taste, habit, or mistakes.","fun_headline_variants_meta":{"raw":{"variants":["Tafo auto-formats tables: 50% match with three suggestions","Conditional formatting predicted without examples or prompts","Spreadsheet rules and styles learned from data alone","Neuro-symbolic model suggests table formats, no user input","Three format suggestions match half of user rules"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000546,"raw_usage":{"total_tokens":2640,"prompt_tokens":1003,"completion_tokens":1637,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":1559}},"tokens_in":619,"tokens_out":1637,"duration_ms":11355,"temperature":1.0,"reasoning_tokens":1559,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:28:11.677797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh sample of spreadsheets with conditional-formatting rules, ask the users who wrote them to rate whether Tafo's top-three suggestions capture what they intended, and compare that acceptance rate with execution match; if accepted suggestions are far rarer than execution match, the benchmark's ground truth overstates practical automation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The natural-language-plus-examples formatting system used as the strongest prior baseline, plus the evidence that users struggle to give complete formatting specifications."},{"cited_title":"Chi, Quoc V Le, and Denny Zhou","cited_arxiv_id":null,"evidence_quote":"The chain-of-thought prompting technique that the neural generator uses to produce predicates, constants, and rule suggestions."}],"review_version":2}