{"id":"308f87db-974d-464f-8984-7de10976ea44","arxiv_id":"2507.02877","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"AuraGenome combines LLM agents, D3.js code templates, and a visual interface to generate, refine, and reuse circular genome visualizations from natural language.","lead":"This paper introduces AuraGenome, a system that uses seven cooperating AI agents to turn plain-language requests and genomic data files into circular genome charts. It claims the tool is faster and easier than the standard Circos workflow, and that completed charts can be reused and modified without writing code.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 34-vs-112-minute and 89%-vs-76% claims rest on a 12-person, single-task crossover with no order-effect analysis and no blinded, reliability-checked accuracy scoring, so the measured advantage may be an artifact of task practice.","rationale":"The reader identified the user study and expert rubric as the weakest assumption; this stress-test sharpens that concern by showing the problem is not merely representativeness but internal validity. Even within the single task, the identical-task crossover design confounds tool comparison with practice transfer, and the unblinded, reliability-unchecked expert scoring confounds accuracy with expectation. The central quantitative claims therefore remain unverified. The framework and case studies are plausible, and no evidence suggests misconduct, but the reported means cannot yet support the 'rapid' and 'scalable' contribution as stated. The verdict remains conditional pending artifact release and reanalysis, so no change from the reader's verdict is needed.","tokens_in":13319,"tokens_out":3965,"duration_ms":44304,"concrete_test":"Request the per-participant, per-condition completion times and accuracy scores and compute the order effect: compare participants who used AuraGenome first (Group B) with those who used it after Circos (Group A). If the AuraGenome time advantage shrinks by more than half in the Aura-first group, the 69% reduction is largely task transfer; additionally, have two new raters blindly re-score the generated charts with the stated rubric and report Cohen's kappa. If kappa is below 0.6 or the blind accuracy margin reverses, the accuracy claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that AuraGenome is 'rapid, reusable, and scalable' is carried by the User Study section under Quantitative Results. The evidence is a within-subject crossover where all 12 participants perform the same multi-layered circular chart task in both conditions. Because the task is identical, Group A (Circos first) gets a full practice pass on the dataset and target chart before touching AuraGenome; Group B (AuraGenome first) later faces Circos with fatigue and a known target. No order-effect statistics are reported, so the reported 34 minutes (SD 9.16) versus 112 minutes (SD 16.5) and 89% versus 76% accuracy means cannot be separated from learning transfer. The accuracy scores are even less secure: they come from 'two independent genomics experts' with no rubric described, no blinding to tool, and no inter-rater reliability. One Circos participant did not finish in 150 minutes; the paper does not state whether that observation was capped, excluded, or treated as 150 minutes, which materially changes the Circos mean. No significance test is reported for time or accuracy; only a Mann-Whitney U test on Likert items is described. The comparison is also solely against Circos; no LLM baseline or IntelliCircos is included, so the gain cannot be attributed specifically to the seven-agent architecture. None of the evaluation artifacts (GenoVis-300 prompts, task dataset, scores, logs) are released despite the GitHub link, making the quantitative results currently unreproducible. The system itself is coherent and the COLO-829 case provides some external grounding, but the headline numbers are not yet demonstrated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AuraGenome, a framework that combines seven specialized LLM-driven agents with an interactive visual analytics system to generate multi-layered circular genome visualizations from natural language and tabular genomic data. The system supports ring, radial, and chord layouts, offers a sequence-log-based reuse mechanism, and includes RAG-grounded code generation with modular code interfaces. The authors evaluate AuraGenome through two expert-led case studies (AML translocation/expression profiling and reproduction of a COLO-829 Nature figure) and a within-subject user study (n=12) comparing AuraGenome with Circos. The reported quantitative results are a 34-minute (SD 9.16) versus 112-minute (SD 16.5) completion time, 89% (SD 4.61) versus 76% (SD 18.31) accuracy, and statistically significant Likert-scale differences on five of six questions.","tokens_in":13588,"tokens_out":3912,"duration_ms":39129,"significance":"If the results hold, AuraGenome would meaningfully lower the technical barrier for genomics researchers to create publication-ready circular plots, and the paper's system design is a substantial engineering contribution: the requirement analysis with eight domain experts, the GenoVis-300 pre-study for model-to-agent assignment, the structured prompt architecture, and the two case studies (including an external anchor in the COLO-829 reproduction) all give useful evidence that the approach is workable. However, the user study as reported does not yet support the headline quantitative claims. The missing order-effect analysis, lack of paired significance tests, absent accuracy-scoring protocol, and unclear handling of an incomplete Circos observation mean that the measured 69% time reduction and 13-point accuracy gap may be artifacts of the evaluation design. The framework itself is plausible and well-motivated, but the evidence needs strengthening before the paper's central claims can be accepted.","major_comments":[{"comment":"The within-subject crossover uses the identical task in both conditions, but the paper reports no order-effect analysis. Group A performs the task with Circos first, giving participants a full practice pass on the dataset and target chart before they use AuraGenome; Group B uses AuraGenome first and then faces Circos with a known target and possible fatigue. Under this design, the 34- versus 112-minute difference may partly or entirely reflect learning transfer rather than tool effectiveness. The authors should report per-group means for time and accuracy, or restructure the study with two matched but different tasks, and include an order term in the statistical analysis.","section":"User Study, Study Design and Implementation"},{"comment":"Accuracy scores are the only output-quality measure, but the paper does not describe the scoring rubric, whether the two experts were blinded to tool identity, or the inter-rater reliability (e.g., Cohen's kappa). Without a rubric and reliability statistics, the 89% versus 76% difference cannot be interpreted. Please provide the scoring protocol and an inter-rater agreement measure.","section":"User Study, Quantitative Results (Figure 8-a)"},{"comment":"No significance test is reported for task completion time or accuracy; the only inferential statistics in the section are Mann-Whitney U tests on Likert items. The statement that 'AuraGenome significantly outperformed Circos' is not supported without paired tests appropriate to the within-subject design (e.g., Wilcoxon signed-rank) for time and accuracy, along with effect sizes.","section":"User Study, Quantitative Results"},{"comment":"The paper states that one participant could not complete the Circos task within the 150-minute maximum but does not say how this observation entered the analysis. If that participant's time was capped at 150 minutes, excluded, or imputed, the Circos mean of 112 minutes changes materially. The handling of this participant must be stated explicitly.","section":"User Study, Quantitative Results"},{"comment":"The evaluation compares AuraGenome only with Circos; there is no LLM-based baseline (e.g., IntelliCircos, or direct prompting of GPT-4o/DeepSeek-R1 without the multi-agent workflow). Because the user study cannot separate the contribution of the seven-agent architecture from the simple fact of using an LLM, the claim that the multi-agent design specifically drives the gains is not evidenced. At minimum, the authors should acknowledge this as a limitation or provide an ablation.","section":"User Study and Case Studies"},{"comment":"The paper cites GenoVis-300, the user-study task dataset, accuracy scores, and interaction logs, but none of these artifacts are available at the provided GitHub repository. Without the benchmark prompts, the task specification, and the anonymized scores, the quantitative results cannot be independently checked. Please release these artifacts or clearly mark the repository as containing the evaluation data.","section":"Availability and Reproducibility"}],"minor_comments":[{"comment":"The phrase 'Performance was evaluated quatitative analysis' contains a typo; 'quatitative' should be 'quantitative'. In addition, 'We summarized them for futher thematic analysis' should read 'further'.","section":"User Study, Study Design and Implementation"},{"comment":"In the GenoVis-300 pre-study, 'condidates' should be 'candidates', and in requirement R2 'suiteable' should be 'suitable'.","section":"AuraGenome Multi-Agent Workflow, Pre-Study"},{"comment":"The text later refers to 'Parameter Panel (Figure 4-e)' and 'Sequence Log View (Figure 4-f)', but in the System Overview the Track Parameter Panel is labeled (f) and the Sequence Log View is labeled (g). The figure callouts need to be reconciled.","section":"AuraGenome Visual Analytics System, System Overview"},{"comment":"Case 1 describes steps A1-A2, A3, A5, and A6-A7, but there is no A4 in the description or in Figure 6. Either add A4 to the workflow or renumber the panels.","section":"Case Studies, Case 1"},{"comment":"The pseudocode contains 'Y ou are an expert' with an extra space in 'You'; this should be corrected.","section":"Prompt Strategy 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution with a credible design and two useful case studies, but the user-study evidence is currently not rigorous enough to support the abstract's quantitative claims. The missing order-effect analysis, the absence of paired significance tests for time and accuracy, the lack of an accuracy-scoring protocol, and the unstated handling of the incomplete Circos observation are all fixable within the manuscript's scope. I would like to see these points addressed before acceptance; the paper should not be rejected, but it also should not be accepted in its current form. I also note that none of the evaluation artifacts are released, which is a reproducibility concern for a journal that emphasizes verification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely new: instead of asking an LLM to produce a whole Circos-style plot from scratch, the authors decompose the job into seven role-specific agents, ground them with D3 templates and a RAG knowledge base, and tie the whole thing together with a visual analytics system whose sequence log lets users branch and reuse earlier steps. The GenoVis-300 pre-study that maps LLM strengths to agent roles is a sensible design move, and the COLO-829 case, where the system reproduces a published Nature figure, gives the work some external anchor. The framework is coherent, well described, and the case studies read as honest demonstrations of what the system can do.\n\nThe soft spot is the user study, and it is the load-bearing evidence for the paper's 'rapid, reusable, scalable' claim. Twelve participants, one task, a within-subject crossover with no order-effect analysis. Because the task is identical in both conditions, the group that did Circos first gets a practice pass on the exact same chart before touching AuraGenome; the paper does not address how much of the 34-vs-112-minute gap is learning transfer. The one participant who did not finish in 150 minutes is mentioned but not handled explicitly, which materially changes the Circos mean. Accuracy is reported as 89% vs 76%, but the two 'independent experts' had no described rubric, no blinding, and no inter-rater reliability. No significance test is given for time or accuracy at all, only Mann-Whitney U on Likert items. And no evaluation artifacts are released despite the GitHub link, so the quantitative results are not reproducible. The comparison is only against Circos, so the specific benefit of the seven-agent architecture over a simpler LLM baseline is untested.\n\nThat said, the direction of the effect is plausible, and the qualitative feedback plus the reuse logic give me real confidence that the system helps. The paper's own limitations section is honest about linear/hybrid layouts not being supported, though it underplays the statistical weaknesses. This is a solid systems paper with a weak evaluation section, not a paper with a broken core.\n\nI would send this to peer review and ask for major revision: report order-effect statistics, state how the incomplete participant was handled, spell out the accuracy rubric and inter-rater reliability, release the prompts and task materials, and soften the superiority claims until the evidence is solid. For visualization practitioners and bioinformatics tool builders, this is worth reading now.","headline":"AuraGenome is a thoughtful and potentially useful LLM-powered workflow for circular genome plots, but the headline time and accuracy gains rest on a small, methodologically under-reported user study that needs major revision before the quantitative claims can be taken at face value.","tokens_in":14194,"tokens_out":1684,"would_cite":true,"duration_ms":19935,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an LLM-powered multi-agent system lets biologists build circular genome charts in roughly a third of the time, with higher accuracy.","keywords":["circular genome visualization","multi-agent workflow","large language models","visual analytics","D3.js","natural language interface","visualization reuse","user study"],"falsifier":"Re-run the comparison on a task and accuracy rubric designed by independent visualization researchers, with a larger cohort and blinded scoring of outputs; the central claim would be undermined if the completion-time gap narrows to non-significance or if AuraGenome's accuracy advantage disappears under blinded scoring.","tokens_in":13092,"feed_emoji":"🧬","tokens_out":5793,"duration_ms":59338,"temperature":0.7,"pith_summary":"The paper tries to establish that an LLM-powered multi-agent framework can replace hand-scripted configuration for circular genome visualization, producing results faster and with comparable or better quality. If true, domain scientists can explore structural variants and gene regulation without writing complex configuration files. The paper supports this with a seven-agent pipeline that parses intent, plans layout, generates and validates D3.js code, and a visual interface that makes each generation step reusable. In a controlled comparison with 12 biology researchers, the same multi-layered circular chart took about 34 minutes with AuraGenome versus 112 minutes with the standard script-based tool Circos, with mean output accuracy 89 percent versus 76 percent.","feed_headline":"Seven-agent LLM pipeline cuts circular genome chart time by 69%","feed_subtitle":"A user study reports 34-minute chart builds versus 112 minutes with scripting, at 89% accuracy.","key_machinery":"The load-bearing machinery is the seven-agent workflow plus the layer-aware Sequence Log View. The workflow decomposes a visualization request into modular steps handled by specialized LLM agents, supported by structured prompt blocks, retrieval-augmented generation from a curated template library, and modular D3.js utility functions that keep code generation at the level of content and parameters rather than low-level syntax. The Sequence Log View encodes each generation step as a node with layout type and status, letting users branch from prior states and reuse configurations; this is what makes generation reusable and traceable.","core_discovery":"On its own terms, the central claim is that a semantically decomposed multi-agent workflow can convert raw genomic data and natural-language requests into publication-ready multi-layered circular visualizations without manual scripting. The system pairs seven LLM agents—intent recognition, layout recommendation, D3.js code generation, code modification, validation, explanation, and operational suggestion—with an interactive visual analytics interface whose Sequence Log View records each generation step as a reusable node. In the user study, the same circular chart task took an average of 34 minutes with AuraGenome versus 112 minutes with Circos, and expert-scored output accuracy averaged 89 percent versus 76 percent. Two case studies, one on AML chromosomal translocations and gene expression and one on the COLO-829 somatic mutation landscape, are presented as evidence that the workflow supports real analysis and cross-task reuse.","pith_inferences":["If the time savings hold beyond the single studied task, a likely mechanism is that LLM-generated code scaffolds replace the slow assembly of configuration blocks; a direct test would compare AuraGenome against a template-based authoring tool rather than manual scripting.","The Sequence Log View is effectively a provenance and reuse graph; this design could generalize to collaborative or reproducible visualization workflows, where sharing a branch of generation steps becomes sharing an analysis recipe.","The paper leaves open whether its accuracy rubric rewards chart completeness rather than biological correctness; a follow-up with domain-expert verification of the resulting charts would separate the two."],"forward_implications":["Biologists without programming expertise can independently produce publication-ready circular genome charts, lowering the barrier to entry for exploratory genomic analysis.","Visualization workflows become reusable: a user can inherit layout logic and code from a previous project, as demonstrated by branching from the AML case to build the COLO-829 chart.","Natural-language refinement with real-time visual feedback can replace iterative configuration-file editing, cutting the time spent on trial-and-error styling.","The same structured multi-agent design, if extended, could support other layout families, but the current system is explicitly limited to circular layouts."],"supporting_citations":[{"why":"Supplies the baseline tool for the user study, in which the same circular chart task took 112 minutes.","marker":"[1]"},{"why":"The closest prior LLM-assisted Circos authoring work; positions AuraGenome's D3-based multi-agent approach against it.","marker":"[20]"},{"why":"Cited for the pitfalls of LLMs in generating visualizations, such as context loss and erroneous code, motivating the multi-agent and validation design.","marker":"[2]"},{"why":"An interactive JavaScript reimplementation of Circos; represents the prior state of interactive circular visualization.","marker":"[7]"},{"why":"Prior automation of genomic visualization parameter optimization; context for automated genomics visualization.","marker":"[13]"},{"why":"The COLO-829 somatic mutation catalogue used as the data source and fidelity target of the second case study.","marker":"[18]"},{"why":"A grammar-based toolkit for genomics visualization that frames the syntactic approach AuraGenome contrasts with LLM generation.","marker":"[12]"}],"fun_headline_variants":["Seven-agent LLM cuts genome chart build time by 70%","From words to circular genomes: 34-min visualizations with seven LLM agents","LLM agents slash genome visualization time and boost accuracy","AuraGenome: 7-agent LLM workflow makes circular charts in 34 minutes","Reusable circular genome charts: 70% faster with LLM agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The effectiveness claim rests on the assumption that the 12-participant single-task user study and the two expert-led case studies are representative of real circular-genome visualization work, and that the author-designed accuracy rubric captures output quality.","fun_headline_variants_meta":{"raw":{"variants":["Seven-agent LLM cuts genome chart build time by 70%","From words to circular genomes: 34-min visualizations with seven LLM agents","LLM agents slash genome visualization time and boost accuracy","AuraGenome: 7-agent LLM workflow makes circular charts in 34 minutes","Reusable circular genome charts: 70% faster with LLM agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000636,"raw_usage":{"total_tokens":2917,"prompt_tokens":912,"completion_tokens":2005,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":1908}},"tokens_in":528,"tokens_out":2005,"duration_ms":13027,"temperature":1.0,"reasoning_tokens":1908,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:42:54.891943+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison on a task and accuracy rubric designed by independent visualization researchers, with a larger cohort and blinded scoring of outputs; the central claim would be undermined if the completion-time gap narrows to non-significance or if AuraGenome's accuracy advantage disappears under blinded scoring.","supporting_citations":[{"cited_title":"Circos: an information aesthetic for compar- ative genomics,","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline tool for the user study, in which the same circular chart task took 112 minutes."},{"cited_title":"IntelliCircos: A Data-driven and AI-powered Authoring Tool for Circos Plots","cited_arxiv_id":"2503.24021","evidence_quote":"The closest prior LLM-assisted Circos authoring work; positions AuraGenome's D3-based multi-agent approach against it."},{"cited_title":"Promises and pitfalls: using large language models to generate visualization items,","cited_arxiv_id":null,"evidence_quote":"Cited for the pitfalls of LLMs in generating visualizations, such as context loss and erroneous code, motivating the multi-agent and validation design."},{"cited_title":"BioCircos. js: an interactive Circos JavaScript library for biological data visualization on web applications,","cited_arxiv_id":null,"evidence_quote":"An interactive JavaScript reimplementation of Circos; represents the prior state of interactive circular visualization."},{"cited_title":"Enabling multimodal user interactions for genomics visualization creation,","cited_arxiv_id":null,"evidence_quote":"Prior automation of genomic visualization parameter optimization; context for automated genomics visualization."},{"cited_title":"A comprehensive catalogue of somatic mutations from a human cancer genome,","cited_arxiv_id":null,"evidence_quote":"The COLO-829 somatic mutation catalogue used as the data source and fidelity target of the second case study."},{"cited_title":"Gosling: a grammar-based toolkit for scalable and interactive genomics data visualization,","cited_arxiv_id":null,"evidence_quote":"A grammar-based toolkit for genomics visualization that frames the syntactic approach AuraGenome contrasts with LLM generation."}],"review_version":2}