Generalizing Large Language Model Usability Across Resource-Constrained

Yun-Da Tsai

REVIEW 4 major objections 5 minor 248 references

Generalizing Large Language Model Usability Across Resource-Constrained

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This dissertation claims that text-centric representation, inference-time optimization, and correct-by-construction synthetic data — not parameter scale — can make LLMs generalize across modalities, resource constraints, and low-resource…

desk verdict A solid thesis-style compilation of prior wins; the headline Verilog SOTA needs artifacts and a contamination audit before I'd bank on it. read the letter →

arxiv 2505.17040 v1 pith:AGROHPCK submitted 2025-05-13 cs.LG cs.CL

Yun-Da Tsai This is my paper · ORCID

classification cs.LGcs.CL

keywords largelanguagemodelsmultimodalalignmentmodalitymismatchpromptoptimizationuncertaintyquantificationRTLcodegenerationVerilogcorrect-by-constructionsyntheticdata

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This dissertation argues that the practical ceiling on large language models in constrained settings is set less by model scale than by how inputs are represented and how training data is built. Its central demonstration is that translating every input modality — images, tables, waveforms — into natural language lets a single frozen LLM handle unseen, noisy, or missing modality combinations without retraining, and that this text-centric pipeline (TAMML) outperforms embedding-based cross-modality translation baselines on three datasets. A second thread shows LLMs can act as optimizers through iterative prompting, and that standard uncertainty metrics measure answer diversity rather than correctness, making them unfit to guide such search. A third thread addresses Verilog, a data-scarce hardware language, with correct-by-construction synthetic problems (Karnaugh maps, finite-state machines, waveforms) and targeted code-repair data, lifting a fine-tuned Starcoder2-15B above prior state-of-the-art pass@1 by 3.8%, 10.9%, and 6.6% on VerilogEval-Machine, VerilogEval-Human, and RTLLM. If these results hold, users facing limited data or compute can buy LLM capability with data curation and prompt engineering instead of scaling models.

What carries the argument

The load-bearing device is the modality-to-text transformation $F_m(x)$ of Eq. (3.1): a function that converts any modality input — image, table, waveform, or text — into a natural-language summary. Together with the three stages that follow it (text-style translation across modalities, cross-modality summarization, and chain-of-thought reasoning augmentation), this transformation collapses all input heterogeneity into one linguistic space, so a frozen LLM can do the reasoning in-context and the model never needs new weights for new modalities. The same 'turn heterogeneous input into a uniformly verifiable format' move recurs on the code side in the correct-by-construction generators: programs that sample random Boolean minterms, finite-state-machine transition graphs, and waveform timing patterns, then emit problem statements and Verilog solutions whose consistency is guaranteed by construction rather than by the LLM's own verification. A companion repair-data loop writes detailed LLM-generated error reports for the model's own mistakes and injects those same errors into open-source code, producing training pairs aimed at the 'minor' bugs that cause pass-rate volatility.

What would settle it

Run the TAMML pipeline on a dataset whose label depends on a precise continuous table value (a price, a timing delay) and corrupt the serialization so that value is rounded or dropped; if accuracy collapses toward a constant predictor while an embedding-based model trained on raw values holds its performance, the 'text preserves task-relevant information' premise is falsified. A complementary check for the Verilog claim: score Starcoder2-15B-CC-Repair on freshly written, uncontaminated non-textual RTL problems; if pass@1 falls back to the base model's level, the reported gains came from benchmark overlap rather than new capability.

Watch

Extended reading notes

Core claim

The dissertation's central claim is that principled methods in three areas — alignment, optimization, and synthetic data generation — can significantly broaden LLM usability across modalities, resource regimes, and application domains without large-scale retraining or parameter scaling. Concretely, it claims that a frozen LLM prompted with structured natural-language summaries of each input modality beats embedding-based zero-shot cross-modality translation methods (SDEdit, DDRM, Idinvert) under train/test modality mismatch, with roughly 21% relative accuracy gain over the strongest baseline on PetFinder and 54% lower mean squared error on Airbnb, and stays competitive even when train and test modality sets match. It further claims that prompt-space adversarial perturbation, rather than gradient-based robust training, is the effective way to harden this pipeline against noisy, missing, or reordered modalities. On the code side, it claims that LLM-generated synthetic Verilog data is unreliable precisely where hardware problems are hardest — non-textual representations like Karnaugh maps and waveforms — and that replacing it with generator-verified correct-by-construction data plus LLM-injected repair data is what allows a 15B model to surpass prior state of the art on VerilogEval and RTLLM. Finally, it claims that the uncertainty metrics commonly used for LLM reasoning track answer diversity, not correctness uncertainty, so they should not guide prompt optimization; a correctness-aligned metric would hit 50% accuracy at maximum uncertainty on binary tasks.

Load-bearing premise

The whole multimodal argument rests on one premise: that turning each input modality into a written description via the transformation $F_m(x)$ at Eq. (3.1) loses none of the information the task depends on, because if the captioner or table serializer drops exact values, timing relations, or visual details, the frozen text-only LLM has no way to recover them and the claimed advantage over embedding-based systems collapses; the premise is tested on only three datasets (PetFinder, Airbnb, Avito).

Editorial extensions

If this is right

A deployment that adopts text-centric alignment can add a new modality (audio, sensor streams, video) by writing one captioner-stage prompt and rerunning inference, with no weight updates and no paired data for the new modality.
Prompt-space adversarial perturbation becomes a viable alternative to gradient-based robust training for multimodal inputs, with the extra property that every perturbation is human-readable and attributable.
Correct-by-construction and repair data make fine-tuning effective in low-resource symbolic domains where LLM self-verification is unreliable, because solution correctness is guaranteed by the generator, not by the model.
Because standard uncertainty metrics do not track correctness, iterative prompting guided by a correctness-aligned uncertainty signal is the paper's prescribed — and benchmarked — direction for inference-time optimization.
Extreme data pruning (1% of the MBPP training set retains near-full-data pass@1) implies fine-tuning budgets can be cut by orders of magnitude when clustering plus diversity metrics select the examples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the text-centric premise holds, the framework's ceiling tracks the quality of the captioners and serializers it leans on, so the pipeline should improve automatically as off-the-shelf captioning improves; the paper's own four-captioner comparison suggests the downstream model is not the bottleneck.
The correct-by-construction recipe should transfer to any machine-checkable low-resource language — VHDL, SystemVerilog assertions, assembly, formal specifications — since it only needs an invertible generator that samples problems with certified solutions; testing it on VHDL would be a direct check of the generality claim.
The paper stops short of building the optimizer its uncertainty analysis points to; a natural next step is an iterative search that asks the LLM for its own correctness probability (via self-consistency over perturbed questions) and uses that to decide when to stop, with the 50%-accuracy-at-max-uncertainty criterion as the acceptance test.
The same adversarial-prompting machinery could double as an automated robustness test-suite for multimodal systems: let the LLM propose semantically plausible corruptions and use the drop in a correctness-aligned uncertainty metric as the pass/fail gate.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

You should know: this is a PhD dissertation that repackages the author's published papers — TAMML, RTLFixer, the correct-by-construction Verilog data work, plus data pruning and a short RL chapter — under a "resource-constrained generalization" umbrella. If you're expecting a single new technical result, you won't find one. But if you treat it as a unified record of a coherent research program, it is a reasonably careful one.

What it does well: the empirical backbone is broad and mostly uses external benchmarks (PetFinder, Airbnb, Avito, VerilogEval, RTLLM, GSM8K, HumanEval/MBPP). TAMML's text-centric alignment is tested against diffusion/GAN translation baselines and against fine-tuned LanguageBind-style alignment, which is a fair comparison set. The Verilog synthetic-data chapters give a clear analysis of why naive SDG fails — non-textual representations and checkpoint variability — and the correct-by-construction pipeline (KMaps, FSMs, waveforms) plus the targeted repair data is a sensible, believable fix. The author also states limitations in the conclusion, which I respect.

Soft spots, in order of importance. The headline SOTA numbers for Starcoder2-15B on VerilogEval and RTLLM rest on decontamination and re-evaluation claims that are not verifiable from the preprint. The appendix that supposedly details re-evaluation of baselines is not fully included, and no code or data is released. The Repair-data generation (Figure 5.7) starts from error reports on benchmark problems; the decontamination sentence in Section 5.3.5 is about the CC data, not the Repair data. If any benchmark-derived prompt or testbench leaks into the fine-tuning set, the 3.8/10.9/6.6 pass@1 gains are inflated. Second, many headline tables report single-point estimates with no error bars or significance tests. Third, the "significantly broaden usability" thesis is a meta-claim that is never directly tested — it is an arrangement of results across separate domains, not a single experiment.

None of this kills the paper. The individual components were peer-reviewed separately, and the framing is honest. But the dissertation as submitted doesn't yet support the strongest quantitative claim in its abstract.

Who's it for: someone entering RTL code generation or modality-mismatch robustness who wants a map of what worked and what didn't. It deserves a serious referee if the artifacts are provided; without them, I'd condition acceptance on releasing data and doing an explicit contamination audit of the Repair set. I'd send it out, but I'd ask for those revisions.

Referee Report

4 major / 5 minor

Summary. This doctoral dissertation argues that principled methods for alignment, optimization, and synthetic data generation can broadly improve LLM usability under resource constraints. It develops a text-centric multimodal alignment pipeline (TAMML) that converts images, tables, and text into natural language and uses in-context learning to handle modality mismatch and robustness, with adversarial prompting as a robustness mechanism. It also studies inference-time optimization with uncertainty quantification, and addresses Verilog code generation through agent-based repair (RTLFixer), correct-by-construction synthetic data, targeted code-repair data, and data pruning. The most specific load-bearing empirical claims are that TAMML outperforms embedding-based cross-modality baselines under modality mismatch (Tables 3.2 and 3.3), and that a fine-tuned Starcoder2-15B model surpasses prior state-of-the-art pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM by 3.8%, 10.9%, and 6.6% respectively (Section 5.3.1).

Significance. If the claims hold, the dissertation makes several useful contributions. The correct-by-construction synthetic data generation for Karnaugh maps, FSMs, and waveforms is a genuinely valuable idea for low-resource hardware-language modeling, and the use of external benchmarks such as VerilogEval, RTLLM, GSM8K, and StrategyQA means the central comparisons are not definitionally circular. The TAMML framework's modular text-centric design, with its interpretable intermediate representations, is also a sensible alternative to embedding-fusion approaches in dynamic modality settings. However, the manuscript is a compilation of separate studies, and the unified 'usability across resource-constrained settings' claim is only partially tested because the chapters use different tasks, datasets, and baselines. The strongest quantitative result, the Verilog SOTA claim, currently rests on decontamination and re-evaluation procedures that are described only by reference to an appendix that is not present in the submitted text, and no code or data artifacts are provided. Reproducibility and decontamination evidence are therefore essential before the headline claims can be accepted.

major comments (4)

[Section 5.3.1 and Tables 5.12-5.13] The Verilog state-of-the-art claim depends on two conditions that are not verifiable from the submitted manuscript. First, Section 5.3.5 says that CC data excludes entries duplicating benchmark data representations, but the Repair data is explicitly constructed from error reports on benchmark problems (Figure 5.7), and the validation step in Appendix I.49 states that 'the generated code fix will be evaluated for functional correctness' without specifying the oracle or whether any benchmark-derived function is used as an oracle. If any benchmark problem statement, reference solution, or testbench survives into the fine-tuning data or validation filter, the reported pass@1 gains of 3.8%, 10.9%, and 6.6% could be inflated. The table captions state 'All models are re-evaluated (see Appendix I.1)', but Appendix I.1 is not included in the submitted text, and no code or dataset is released. Please provide the full re-evaluation protocol, a precise decontamination description covering the Repair pipeline, and release the fine-tuning data and evaluation harness.
[Section 3.4.2, Tables 3.2-3.3] The headline cross-modality comparisons are not fully controlled. The embedding-based baselines (SDEdit, DDRM, Idinvert) involve trained generative translation plus fine-tuned downstream models, while TAMML uses a frozen LLM, so the comparison conflates representation choice with training protocol. In addition, the reported numbers are single-point estimates without error bars or significance tests, even though the text states that LLM outputs can vary under identical prompts and temperature (Section 5.2.4). The claimed 21% accuracy improvement on PetFinder and 54% MSE reduction on Airbnb are computed from these single averages. Reporting variance across at least several repeated evaluation runs, and ideally matching the downstream fine-tuning budget or reporting it explicitly, is necessary to support the comparative claims.
[Section 3.3.1, Eq. (3.2)] The robustness claims for adversarial prompting are partially self-referential. The adversarial perturbations are generated by an LLM of the same general class used both for the text-centric alignment and for the downstream reasoning, guided by labels and instructions (LLM(x', inst, label, T)). Demonstrating robustness against self-generated perturbations does not necessarily imply robustness against independently constructed or human-authored adversarial inputs. This is a correctness-risk concern rather than a circularity claim about the benchmarks themselves; a concrete test would be to evaluate the trained text-centric pipeline on perturbations produced by a different model family (or by human annotators) and show that the robustness advantage persists.
[Section 3.2.3, Eq. (3.1)] The central premise that modality-to-text transformation preserves task-relevant information is tested only on PetFinder, Airbnb, and Avito, which are all social-adoption or e-commerce datasets with tabular, image, and text inputs. The abstract's claim of extending to 'any modalities' and the inclusion of waveforms and FSM diagrams in the pipeline are not supported by experimental evidence in Chapter 3. At minimum, the text should restrict the generalization claim to the modality types and datasets actually evaluated, or add experiments on at least one structurally different modality, such as audio or time-series, where captioning/serialization loss is known to be more severe.

minor comments (5)

[Section 1.4] The chapter outline skips Chapter 4 entirely, moving from Chapter 3 to Chapter 5; please correct the outline to include the inference-time optimization chapter.
[Sections 3.4.6 and 3.4.7] The two sections both carry the title 'Modality Robustness Baselines' and contain overlapping text about MLLMs, robust training, and text-centric strategies; merge them or rename one to avoid duplication.
[Tables 3.2 and 3.3] The row labels such as 'text+image | tabular' are visually ambiguous in the typeset table; clarifying the notation (e.g., 'train: text+image, test: tabular') would substantially improve readability.
[References and Appendix I.1] The full text references Appendices I.1 and Appendix H for evaluation details, but these appendices are not included in the submitted manuscript; since the SOTA claim depends on them, at least a complete evaluation-protocol appendix should be part of the manuscript.
[Notation throughout] The model name is written inconsistently as 'Starcoder2', 'StarCoder2', and 'starcoder2-15B'; please unify the spelling and the hyphenation in all tables and text.

Circularity Check

1 steps flagged · score 6.0 of 10

Verilog SOTA claim is partially circular: the 'Repair' fine-tuning data is generated from the same VerilogEval/RTLLM benchmark problems on which state-of-the-art pass@1 is reported.

fitted input called prediction [Section 5.3.1 (contribution bullet), Section 5.3.5 / Figure 5.7, and Appendix I.3 examples]
"The development of an automated framework that utilizes LLMs to generate error reports from benchmark problems at various checkpoints, which are then injected into open-source code to create a fine-tuning dataset targeted at correcting the model's specific 'minor' mistakes. ... Another example based on multi_booth_8bit from RTLLM."

The 'benchmark problems' used for Repair data are VerilogEval-Human/Machine and RTLLM, exactly the benchmarks on which Section 5.3.1 claims SOTA pass@1 gains of 3.8%, 10.9%, and 6.6%. Error reports are extracted from the model's correct and erroneous code on these problems and transformed into fine-tuning data, so the evaluation set directly informs the training distribution. The paper's decontamination statement ('entries that duplicate the data representations of benchmark problems were excluded') is made only for the CC data, not for the Repair pipeline, and Appendix I.3 openly labels examples as derived from VerilogEval-Human and RTLLM problems.

full rationale

The bulk of this dissertation is empirically self-contained: TAMML is evaluated on PetFinder, Airbnb, and Avito with external labels; RTLFixer is assessed on VerilogEval/RTLLM using compiler feedback and does not train on those benchmarks; the correct-by-construction CC data is explicitly decontaminated; data pruning is validated on MBPP and HumanEval; and the uncertainty analysis uses GSM8K/StrategyQA ground truths. Self-citations to TAMML [138], RTLFixer [132], and CraftRTL [174] are normal and not load-bearing because the dissertation re-derives the methods in place. The reader-flagged loops (LLM-generated adversarial perturbations, LLM self-verification of synthetic data) are self-referential quality-control steps, but they do not make the reported metrics equal to their inputs by definition. The one genuine circular element is the Repair-data pipeline: fine-tuning data is constructed from error reports on the very VerilogEval/RTLLM problems later used for the SOTA pass@1 claim. This makes the headline Verilog result partially a fit to the benchmark rather than an independent prediction, so I assign partial circularity (6) rather than a full collapse. Additionally, no code or dataset is released and Appendix I.1 appears incomplete, so the decontamination and re-evaluation protocols cannot be independently verified; that is a correctness/verifiability risk distinct from the circularity finding itself.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims of the dissertation are empirical. They do not depend on fitted mathematical constants. The main assumptions are about the sufficiency of text-based representations and the reliability of LLM-based verification, both domain assumptions rather than standard mathematical axioms.

assumptions (3)

domain assumption Modality-to-text transformation preserves task-relevant information for downstream prediction.
The entire TAMML pipeline in Chapter 3.2 depends on this. Table 3.8 shows only small differences across captioners, but this does not prove preservation for all tasks or modalities.
domain assumption LLM self-verification and syntax checkers are sufficient filters for synthetic Verilog data quality.
The SDG pipeline in Section 5.3.2 uses self-verification to keep synthetic data. The authors themselves note this is insufficient for non-textual representations, which motivates the correct-by-construction data.
domain assumption VerilogEval and RTLLM benchmarks measure the intended functional correctness.
All state-of-the-art claims in Chapter 5 use these benchmarks as ground truth. They are constructed by humans or LLMs and are not machine-checked in a formal system.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Generalizing Large Language Model Usability Across Resource-Constrained." pith.science (2026). https://pith.science/paper/AGROHPCK

@misc{pith2026250517040,
  author       = {Pith},
  title        = {Pith review of: Generalizing Large Language Model Usability Across Resource-Constrained},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AGROHPCK}},
  note         = {Machine review of arXiv:2505.17040}
}

read the original abstract

Large Language Models (LLMs) have achieved remarkable success across a wide range of natural language tasks, and recent efforts have sought to extend their capabilities to multimodal domains and resource-constrained environments. However, existing approaches often rely on costly supervised fine-tuning or assume fixed training conditions, limiting their generalization when facing unseen modalities, limited data, or restricted compute resources. This dissertation presents a systematic study toward generalizing LLM usability under real-world constraints. First, it introduces a robust text-centric alignment framework that enables LLMs to seamlessly integrate diverse modalities-including text, images, tables, and any modalities - via natural language interfaces. This approach supports in-context adaptation to unseen or dynamically changing modalities without requiring retraining. To enhance robustness against noisy and missing modalities, an adversarial prompting technique is proposed, generating semantically challenging perturbations at the prompt level to stress-test model reliability. Beyond multimodal setting, the dissertation investigates inference-time optimization strategies for LLMs, leveraging prompt search and uncertainty quantification to improve performance without additional model training. This perspective offers an efficient alternative to scaling model parameters or retraining from scratch. Additionally, the work addresses low-resource domains such as Verilog code generation by designing correct-by-construction synthetic data pipelines and logic-enhanced reasoning models, achieving state-of-the-art performance with minimal data. Together, these contributions form a unified effort to enhance the adaptability, scalability, and efficiency of large language models under practical constraints.

Figures

Figures reproduced from arXiv: 2505.17040 by the authors.

**Figure 2.1.** Three axes of generalization considered in this dissertation: modality, resource, and symbolic domain. LLMs are increasingly deployed in diverse real-world applications that require them to generalize across different input modalities, tasks, and resource conditions. This dissertation explores generalization across three complementary axes: modality, 9 [PITH_FULL_IMAGE:figures/full_fig_p039_2_1.png] view at source ↗

**Figure 3.1.** Illustration of the modality mismatch problem. During training, the model observes a fixed modality configuration. At inference, it must generalize across all combinations, including fully seen, fully unseen, and mixture setups. • Fully Seen: MI = MT . The inference modality set exactly matches the training configuration. This serves as the baseline. • Fully Unseen (Disjoint Modality): MT ∩ MI = ∅. Inference uses en… view at source ↗

**Figure 3.2.** Architectural comparison between embedding-based and text-centric multimodal alignment approaches. Text-centric methods transform all inputs into a common linguistic representation, enabling more flexible processing and fusion. nerabilities when confronted with missing, noisy, or dynamically composed inputs. Furthermore, fusion methods necessitate substantial retraining overhead whenever new modalities are introduce… view at source ↗

Figures from the paper (29 more)

**Figure 3.3.** Figure 3.3: Text-centric multimodal alignment pipeline architecture. Each modality undergoes transformation into text via specialized summarization functions, followed by text-style translation, multimodal summarization, and reasoning augmentation to produce a unified prompt for…

**Figure 3.4.** Figure 3.4: Prompt templates for different pipeline modules, illustrating the text transformation, translation, summarization, and reasoning processes employed in the multimodal alignment framework. 3.2.7 Discussion This pipeline offers several compelling advantages: • Modular e…

**Figure 3.5.** Figure 3.5: Transformation pipeline for multimodal alignment. Raw inputs are converted to text representations using foundation models, then processed through parallel modality summarization and LLM reasoning. The resulting text outputs are concatenated for downstream prediction…

**Figure 3.6.** Figure 3.6: Example prompt templates for each module in the adversarial training pipeline. 3.4 Experimental Evaluation 3.4.1 Evaluation Setup Evaluation Metrics The evaluation uses two primary metrics: • Absolute performance: Measured via accuracy (for classification) or mean sq…

**Figure 3.7.** Figure 3.7: Comparison between traditional and proposed approaches for multimodal learning. Traditional downstream training (top) relies on embeddings from modality-specific foundation models, limiting adaptation to unseen modalities without complete retraining. Previous resea…

**Figure 3.8.** Figure 3.8: Visualization of embedding distributions before and after applying the proposed approach. The left panel shows the original distinct clustering of image and text embeddings, while the right panel demonstrates how the method brings these distributions closer in the se…

**Figure 3.9.** Figure 3.9: Visualization of the cross-modality translation approach, showing how source modality data (orange) is effectively transformed (purple) to align closely with the target modality distribution (blue) in semantic space [PITH_FULL_IMAGE:figures/full_fig_p080_3_9.png]

**Figure 3.10.** Figure 3.10: Visualization of SDEdit-based cross-modality translation on the PetFinder dataset, demonstrating less effective alignment between transformed source modality (purple) and target modality distribution (blue) compared to the proposed approach. 3.4.6 Modality Robustnes…

**Figure 3.11.** Figure 3.11: Robustness evaluation under noisy conditions across three datasets. The top row displays relative accuracy while the bottom row shows effective robustness (performance drop ratio). Text-centric methods consistently exhibit superior resilience to noise compared to b…

**Figure 3.12.** Figure 3.12: Modality-specific noise impact on accuracy (top) and performance drop ratio (bottom) for Image (left), Table (center), and Text (right). Results reveal varying degrees of sensitivity to noise across different modalities, suggesting opportunities for targeted robustn…

**Figure 4.1.** Figure 4.1: Relationship between LLM correctness uncertainty and response accuracy in the GSM8K dataset. A reliable uncertainty quantification metric exhibits 50% accuracy at maximum uncertainty, while standard confidence-based metrics fail to provide sufficient guidance for pr…

**Figure 4.2.** Figure 4.2: Prompting strategy for optimization with LLMs. The process involves: (1) loss function formulation from samples, (2) solution generation following algorithm instructions, (3) loss calculation and result tracking, and (4) iteration until stopping criteria are met. ou…

**Figure 4.3.** Figure 4.3: Benchmark workflow for evaluating uncertainty in prompt optimization. The three-step process at each level includes: 1) question perturbation, 2) model output sampling with temperature adjustments, and 3) uncertainty measurement using different metrics. answer sampli…

**Figure 4.4.** Figure 4.4: Performance metrics for GPT-3.5-Turbo across optimization tasks. Goal Metric values (left) show optimization capability, while Policy Metric values (right) demonstrate alignment with ground truth. (a) Goal Metric - performance degrades with dimension increase (b) Unc…

**Figure 4.5.** Figure 4.5: Black-Box optimization performance for GPT-3.5-Turbo and GPT-4. Performance degrades as dimensions increase, with significant drop-off beyond dimension 6 in both optimization capability (left) and solution stability (right). LLMs can function effectively as black-bo…

**Figure 4.6.** Figure 4.6: Gradient Descent performance metrics showing consistent optimization capability across problem dimensions. Near-zero Policy Metric values indicate strong alignment with ground truth, while high Goal Metric values demonstrate effective optimization. The policy metric …

**Figure 4.7.** Figure 4.7: Correlation between uncertainty metrics and uncertainty types across models and datasets. The maps reveal that existing metrics correlate strongly with answer uncertainty (confidence/diversity) rather than correctness uncertainty, making them unsuitable for prompt o…

**Figure 4.8.** Figure 4.8: Uncertainty Metric results across optimization tasks and problem dimensions. Higher uncertainty values for smaller dimensions suggest LLMs possess a richer solution space for small-scale problems, consistent with their better performance on these problems. in these…

**Figure 4.9.** Figure 4.9: Performance improvement with self-consistency in Gradient Descent optimization. Narrower confidence intervals indicate enhanced stability, while negative Policy Metric values with high Goal Metric values demonstrate significant outperformance compared to ground tru…

**Figure 5.1.** Figure 5.1: Architectural overview of the RTLFixer framework illustrating the seamless integration of language model reasoning with hardware verification tools to create a robust iterative debugging pipeline. 5.2.3 RTLFixer System Overview: Resolving Syntax Error with LLM Agen…

**Figure 5.2.** Figure 5.2: Frequency distribution of error categories in the RTL benchmark dataset, highlighting the relative prevalence of different error types encountered in practical hardware designs rules (e.g., type mismatches) 3. Timing Errors: Issues related to clock domain crossings, …

**Figure 5.3.** Figure 5.3: Comparative visualization of VerilogEval pass@1 results before (inner circle) and after (outer circle) syntax error correction with RTLFixer, demonstrating substantial improvement in successful compilation Remarkably, even without access to explicit compiler diagnost…

**Figure 5.4.** Figure 5.4: Frequency distribution of iterations required by ReAct to successfully resolve syntax errors, demonstrating the efficiency of the approach with most errors fixed in a single revision Challenges in Debugging Simulation Errors: While this framework is readily adaptabl…

**Figure 5.5.** Figure 5.5: The proposed methods reduce pass rate variability during training: SDG (left) shows high volatility with significant degradation on many problems, while SDG-CC-Repair (right) stabilizes learning outcomes on solvable problems (details in Appendix I.1.10). 5.3.4 Variab…

**Figure 5.6.** Figure 5.6: State transition logic [PITH_FULL_IMAGE:figures/full_fig_p143_5_6.png]

**Figure 5.7.** Figure 5.7: Overview of the approach for generating targeted code repair data: (1) prompting the LLM to generate detailed error reports from correct and erroneous code, (2) validating error report quality by ensuring the LLM can debug the errors based on the report, and (3) leve…

**Figure 5.8.** Figure 5.8: pass@1 on non-textual problems with total number of CC data with temperature 0.8 [PITH_FULL_IMAGE:figures/full_fig_p151_5_8.png]

**Figure 5.9.** Figure 5.9: Comprehensive pipeline for efficient data pruning in LLM fine-tuning. The process consists of three key stages: (1) embedding instruction-following data and applying dimensionality reduction to the feature representations, (2) clustering to identify and group semanti…

**Figure 5.10.** Figure 5.10: Performance comparison under extreme data pruning on the MBPP (left) and HumanEval (right) benchmarks. Using only 1% of the training data, our pruning method retains performance close to that of full-data training on MBPP, achieving a 4.1% gain over the base model. …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

248 extracted references · 7 canonical work pages

[1]

Accessed on: 10 September, 2023

Inside airbnb : Hawaii, 2023. Accessed on: 10 September, 2023

2023
[2]

Achiam, S

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 techni- cal report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023
[3]

Alayrac, J

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A.Mensch, K.Millican, M.Reynolds, etal. Flamingo: avisuallanguagemodel for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022

2022
[4]

Claude 3.5 and claude 3.7 models.https://www.anthropic.com/ news/claude-3-5-and-3-7, 2024

Anthropic. Claude 3.5 and claude 3.7 models.https://www.anthropic.com/ news/claude-3-5-and-3-7, 2024. Accessed: 2025-04-27

2024
[5]

Austin, A

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

arXiv 2021
[6]

J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 163

arXiv 2023
[7]

Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

arXiv 2022
[8]

Batten, N

C. Batten, N. Pinckney, M. Liu, H. Ren, and B. Khailany. Pyhdl-eval: An llm evaluation framework for hardware design using python-embedded dsls. In Proceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD, MLCAD ’24, New York, NY, USA, 2024. Association for Computing Machinery

2024

Show all 248 references

[9]

Beltagy, M

I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

2004 arXiv
[10]

Bhandari, J

J. Bhandari, J. Knechtel, R. Narayanaswamy, S. Garg, and R. Karri. Llm- aided testbench generation and bug detection for finite-state machines, 2024

2024
[11]

Blocklove, S

J. Blocklove, S. Garg, R. Karri, and H. Pearce. Chip-chat: Chal- lenges and opportunities in conversational hardware design.arXiv preprint arXiv:2305.13243, 2023

2023 arXiv
[12]

Brown, B

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Nee- lakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

1901
[13]

Cassano, J

F. Cassano, J. Gouwar, F. Lucchetti, C. Schlesinger, A. Freeman, C. J. An- derson, M. Q. Feldman, M. Greenberg, A. Jangda, and A. Guha. Knowledge 164 transfer from high-resource to low-resource programming languages for code llms, 2024

2024
[14]

Cassano, J

F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H.Yee, Y.Zi, C.J.Anderson, M.Q.Feldman, A.Guha, M.Greenberg, and A. Jangda. Multipl-e: A scalable and extensible approach to benchmarking neural code generation, 2022

2022
[15]

Chang, Z

K. Chang, Z. Chen, Y. Zhou, W. Zhu, kun wang, H. Xu, C. Li, M. Wang, S. Liang, H. Li, Y. Han, and Y. Wang. Natural language is not enough: Benchmarking multi-modal generative ai for verilog generation, 2024

2024
[16]

Chaudhary

S. Chaudhary. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca, 2023

2023
[17]

B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen. Codet: Code generation with generated tests, 2022

2022
[18]

C. Chen, B. Cui, J. Ma, R. Wu, J. Guo, and W. Liu. A systematic review of fuzzing techniques.Computers & Security, 75:118–137, 2018

2018
[19]

L. Chen, S. Li, J. Yan, H. Wang, K. Gunaratna, V. Yadav, Z. Tang, V. Srini- vasan, T. Zhou, H. Huang, and H. Jin. Alpagasus: Training a better alpaca with fewer data, 2024

2024
[20]

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Ed- wards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

2021 arXiv
[21]

H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, 165 M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024

2024
[22]

F. Cui, C. Yin, K. Zhou, Y. Xiao, G. Sun, Q. Xu, Q. Guo, D. Song, D. Lin, X. Zhang, et al. Origen: Enhancing rtl code generation with code-to-code augmentation and self-reflection.arXiv preprint arXiv:2407.16237, 2024

2024 arXiv
[23]

Da Tsai and S

Y. Da Tsai and S. De Lin. Fast online inference for nonlinear contextual bandit based on generative adversarial network. arXiv preprint arXiv:2202.08867, 2022

2022 arXiv
[24]

Das and V

D. Das and V. Khetan. Deft: Data eﬀicient fine-tuning for large language models via unsupervised core-set selection.arXiv preprint arXiv:2310.16776, 2023

2023 arXiv
[25]

DeepMind

G. DeepMind. Introducing gemini 2.0: Our next-generation ai models. https://blog.google/technology/google-deepmind/google-gemini-ai- update-december-2024/#gemini-2-0-flash, 2024. Accessed: 2025-04-27

2024
[26]

DeepSeek-AI, Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y. Wu, Y. Li, H. Gao, S. Ma, W. Zeng, X. Bi, Z. Gu, H. Xu, D. Dai, K. Dong, L. Zhang, Y. Piao, Z. Gou, Z. Xie, Z. Hao, B. Wang, J. Song, D. Chen, X. Xie, K. Guan, Y. You, A. Liu, Q. Du, W. Gao, X. Lu, Q. Chen, Y. W...

2024
[27]

DeLorenzo, A

M. DeLorenzo, A. B. Chowdhury, V. Gohil, S. Thakur, R. Karri, S. Garg, and J. Rajendran. Make every move count: Llm-based high-quality rtl code generation using mcts, 2024. 166

2024
[28]

S. Diao, P. Wang, Y. Lin, and T. Zhang. Active prompting with chain-of- thought for large language models.arXiv preprint arXiv:2302.12246, 2023

2023 arXiv
[29]

X. Dong, Y. He, Z. Zhu, and J. Caverlee. Promptattack: Probing dialogue state trackers with adversarial prompts. arXiv preprint arXiv:2306.04535, 2023

2023 arXiv
[30]

Dosovitskiy, L

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Un- terthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arxiv 2020. arXiv preprint arXiv:2010.11929, 2010

2020 arXiv
[31]

Dubey, A

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Ta...

2024
[32]

Kto: Model alignmentasprospecttheoreticoptimization

K.Ethayarajh, W.Xu, N.Muennighoff, D.Jurafsky, andD.Kiela. Kto: Model alignmentasprospecttheoreticoptimization. arXivpreprintarXiv:2402.01306, 2024

2024 arXiv
[33]

X. Feng, Z. Wan, M. Wen, Y. Wen, W. Zhang, and J. Wang. Alphazero- like tree-search can guide large language model decoding and training.arXiv preprint arXiv:2309.17179, 2023. 170

2023 arXiv
[34]

Friedman

N. Friedman. Introducing github copilot: your ai pair programmer. 2021

2021
[35]

Z. Fu, H. Yang, A. M.-C. So, W. Lam, L. Bing, and N. Collier. On the effectiveness of parameter-eﬀicient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12799–12807, 2023

2023
[36]

Cao, et al

W.Gao, Z.Deng, Z.Niu, F.Rong, C.Chen, Z.Gong, W.Zhang, D.Xiao, F.Li, Z. Cao, et al. Ophglm: Training an ophthalmology large language-and-vision assistant based on instructions and dialogue.arXiv preprintarXiv:2306.12174, 2023

2023 arXiv
[37]

Girdhar, A

R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. Imagebind: One embedding space to bind them all. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180–15190, 2023

2023
[38]

Gorishniy, I

Y. Gorishniy, I. Rubachev, V. Khrulkov, and A. Babenko. Revisiting deep learning models for tabular data.Advancesin Neural Information Processing Systems, 34:18932–18943, 2021

2021
[39]

Bi, et al

D.Guo, D.Yang, H.Zhang, J.Song, R.Zhang, R.Xu, Q.Zhu, S.Ma, P.Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via rein- forcement learning.arXiv preprint arXiv:2501.12948, 2025

2025 arXiv
[40]

D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence.arXiv preprint arXiv:2401.14196, 2024. 171

2024 arXiv
[41]

D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024

2024
[42]

Guo, Y.-H

P.-F. Guo, Y.-H. Chen, Y.-D. Tsai, and S.-D. Lin. Towards optimizing with large language models.arXiv preprint arXiv:2310.05204, 2023

2023 arXiv
[43]

Guo, Y.-D

P.-F. Guo, Y.-D. Tsai, and S.-D. Lin. Benchmarking large language model uncertainty for prompt optimization.arXiv preprint arXiv:2409.10044, 2024

2024 arXiv
[44]

I. Guz, J. Elliott, M. Konstantin, S. Dane, V. Kassym, and W. Kan. Avito de- mand prediction challenge. https://kaggle.com/competitions/avito-demand- prediction, 2018. Accessed: 2025-02-01

2018
[45]

Henighan, J

T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020

2010 arXiv
[46]

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advancesin neural information processing systems, 33:6840–6851, 2020

2020
[47]

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

2021 arXiv
[48]

B. Hui, H. Yuan, N. Gong, P. Burlina, and Y. Cao. Pleak: Prompt leaking attacks against large language model applications. arXiv preprint arXiv:2405.06823, 2024

2024 arXiv
[49]

Jaech, A

A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, 172 A. Madry, A. Beutel, A. Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

2024 arXiv
[50]

R. Just, D. Jalali, and M. D. Ernst. Defects4j: a database of existing faults to enable controlled testing studies for java programs. InProceedings of the 2014 InternationalSymposium on SoftwareTestingand Analysis, ISSTA 2014, page 437–440, New York, NY, USA, 2014. Association...

2014
[51]

Kanungo, D

T. Kanungo, D. M. Mount, N. S. Netanyahu, C. D. Piatko, R. Silverman, and A. Y. Wu. An eﬀicient k-means clustering algorithm: Analysis and imple- mentation. IEEE transactions on pattern analysis and machine intelligence, 24(7):881–892, 2002

2002
[52]

Kaplan, S

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

2001 arXiv
[53]

Karras, S

T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019

2019
[54]

Kawar, M

B. Kawar, M. Elad, S. Ermon, and J. Song. Denoising diffusion restoration models. Advancesin Neural Information Processing Systems, 35:23593–23606, 2022

2022
[55]

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017

2017
[56]

Kocetkov, R

D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferrandis, Y. Jernite, M. Mitchell, S. Hughes, T. Wolf, D. Bahdanau, L. von Werra, and H. de Vries. The stack: 3 tb of permissively licensed source code, 2022. 173

2022
[57]

W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica. Eﬀicient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

2023
[58]

H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning, 2022

2022
[59]

B. Lei, Y. Li, and Q. Chen. Autocoder: Enhancing code large language model with AIEV-Instruct, 2024

2024
[60]

Lewis, E

P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval- augmented generation for knowledge-intensive nlp tasks, 2021

2021
[61]

K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, and Y. Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023

2023 arXiv
[62]

M. Li, Y. Zhang, S. He, Z. Li, H. Zhao, J. Wang, N. Cheng, and T. Zhou. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning.arXiv preprint arXiv:2402.00530, 2024

2024 arXiv
[63]

M. Li, Y. Zhang, Z. Li, J. Chen, L. Chen, N. Cheng, J. Wang, T. Zhou, and J. Xiao. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning.arXiv preprint arXiv:2308.12032, 2023

2023 arXiv
[64]

R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y. Zhuo, T. Wang, 174 O. Dehaene, M. Davaadorj, J. Lamy-Poirier, J. Monteiro, O. Shliazhko, N. Gontier, N. Meade, A. Zebaze, M.-H. Yee, L. K. Umap...

2023
[65]

Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022

2022
[66]

P. P. Liang, Y. Lyu, X. Fan, Z. Wu, Y. Cheng, J. Wu, L. Chen, P. Wu, M. A. Lee, Y. Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning.arXiv preprint arXiv:2107.07502, 2021

2021 arXiv
[67]

Lightman, V

H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe. Let’s verify step by step.arXiv preprint arXiv:2305.20050, 2023

2023 arXiv
[68]

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning, 2023

2023
[69]

J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code genera- 175 tion. InThirty-seventhConference on Neural Information Processing Systems, 2023

2023
[70]

Liu, T.-D

M. Liu, T.-D. Ene, R. Kirby, C. Cheng, N. Pinckney, R. Liang, J. Al- ben, H. Anand, S. Banerjee, I. Bayraktaroglu, B. Bhaskaran, B. Catan- zaro, A. Chaudhuri, S. Clay, B. Dally, L. Dang, P. Deshpande, S. Dhodhi, S. Halepete, E. Hill, J. Hu, S. Jain, A. Jindal, B. Khailany, G. ...

2024
[71]

M. Liu, N. Pinckney, B. Khailany, and H. Ren. Verilogeval: Evaluating large language models for verilog code generation.arXiv preprint arXiv:2309.07544, 2023

2023 arXiv
[72]

Liu, Y.-D

M. Liu, Y.-D. Tsai, W. Zhou, and H. Ren. Craftrtl: High-quality syn- thetic data generation for verilog code models with correct-by-construction non-textual representations and targeted code repair. arXiv preprint arXiv:2409.12993, 2024

2024 arXiv
[73]

S. Liu, W. Fang, Y. Lu, Q. Zhang, H. Zhang, and Z. Xie. Rtlcoder: Out- performing gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution.arXiv preprint arXiv:2312.08617, 2023

2023 arXiv
[74]

W. Liu, W. Zeng, K. He, Y. Jiang, and J. He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312.15685, 2023

2023 arXiv
[75]

Y. Liu, G. Deng, Y. Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y. Liu, 176 H. Wang, Y. Zheng, et al. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499, 2023

2023 arXiv
[76]

Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, K. Wang, and Y. Liu. Jailbreaking chatgpt via prompt engineering: An em- pirical study.arXiv preprint arXiv:2305.13860, 2023

2023 arXiv
[77]

Lozhkov, R

A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

2024 arXiv
[78]

S. Lu, N. Duan, H. Han, D. Guo, S. won Hwang, and A. Svyatkovskiy. Reacc: A retrieval-augmented code completion framework, 2022

2022
[79]

Y. Lu, S. Liu, Q. Zhang, and Z. Xie. Rtllm: An open-source bench- mark for design rtl generation with large language model. arXiv preprint arXiv:2308.05345, 2023

2023 arXiv
[80]

Y. Lu, S. Liu, Q. Zhang, and Z. Xie. Rtllm: An open-source benchmark for design rtl generation with large language model. In2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), pages 722–727. IEEE, 2024

2024
[81]

Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024

2024
[82]

J. Ma, A. Cao, Z. Xiao, J. Zhang, C. Ye, and J. Zhao. Jailbreaking prompt at- 177 tack: Acontrollableadversarialattackagainstdiffusionmodels. arXivpreprint arXiv:2404.02928, 2024

2024 arXiv
[83]

M. Ma, J. Ren, L. Zhao, D. Testuggine, and X. Peng. Are multimodal transformers robust to missing modality? InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18177–18186, 2022

2022
[84]

Maćkiewicz and W

A. Maćkiewicz and W. Ratajczak. Principal components analysis (pca). Computers & Geosciences, 19(3):303–342, 1993

1993
[85]

Madry, A

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. To- wards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083, 2017

2017 arXiv
[86]

McInnes, J

L. McInnes, J. Healy, and J. Melville. Umap: Uniform manifold approxima- tion and projection for dimension reduction.arXiv preprint arXiv:1802.03426, 2018

2018 arXiv
[87]

Meding, L

K. Meding, L. M. S. Buschoff, R. Geirhos, and F. A. Wichmann. Trivial or impossible–dichotomous data diﬀiculty masks model differences (on imagenet and beyond). arXiv preprint arXiv:2110.05922, 2021

2021 arXiv
[88]

C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. Sdedit:guided image synthesis and editing with stochastic differential equa- tions. International Conference on Learning Representations, 2022

2022
[89]

Introducing meta llama 3: The most capable openly available llm to date, 2024

Meta AI. Introducing meta llama 3: The most capable openly available llm to date, 2024. Accessed: 2024-09-10. 178

2024
[90]

B. B. Moser, F. Raue, and A. Dengel. A study in dataset pruning for image super-resolution. arXiv preprint arXiv:2403.17083, 2024

2024 arXiv
[91]

Muennighoff, Q

N. Muennighoff, Q. Liu, A. Zebaze, Q. Zheng, B. Hui, T. Y. Zhuo, S. Singh, X. Tang, L. von Werra, and S. Longpre. Octopack: Instruction tuning code large language models.arXiv preprint arXiv:2308.07124, 2023

2023 arXiv
[92]

D. Müllner. Modern hierarchical, agglomerative clustering algorithms.arXiv preprint arXiv:1109.2378, 2011

2011 arXiv
[93]

A. Naik. On the limitations of embedding based methods for measuring func- tional correctness for code generation.arXiv preprint arXiv:2405.01580, 2024

2024 arXiv
[94]

Nakano, J

R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. Webgpt: Browser-assisted question- answering with human feedback.arXiv preprint arXiv:2112.09332, 2021

2021 arXiv
[95]

Nichols, J

D. Nichols, J. H. Davis, Z. Xie, A. Rajaram, and A. Bhatele. Can large language models write parallel code? InProceedings of the 33rd International Symposium onHigh-PerformanceParallelandDistributedComputing, HPDC ’24, page 281–294, New York, NY, USA, 2024. Association for Com...

2024
[96]

Adler, N

Nvidia, :, B. Adler, N. Agarwal, A. Aithal, D. H. Anh, P. Bhat- tacharya, A. Brundyn, J. Casper, B. Catanzaro, S. Clay, J. Cohen, S. Das, A. Dattagupta, O. Delalleau, L. Derczynski, Y. Dong, D. Egert, E. Evans, A. Ficek, D. Fridman, S. Ghosh, B. Ginsburg, I. Gitman, T. Grzegor...

2024
[97]

Openai models api

OpenAI. Openai models api. 2023

2023
[98]

Pearce, B

H. Pearce, B. Tan, and R. Karri. Dave: Deriving automatically verilog from en- glish. InProceedings of the 2020 ACM/IEEE Workshopon Machine Learning for CAD, pages 27–32, 2020

2020
[99]

Pedregosa, G

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Pas- sos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python.Journal of Machine Learnin...

2011
[100]

Pei, H.-L

Z. Pei, H.-L. Zhen, M. Yuan, Y. Huang, and B. Yu. Betterv: Controlled verilog generation with discriminative guidance. arXiv preprint arXiv:2402.03375, 2024

2024 arXiv
[101]

Penedo, H

G. Penedo, H. Kydlíček, L. von Werra, and T. Wolf. Fineweb, April 2024

2024
[102]

B. Peng, C. Li, P. He, M. Galley, and J. Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023. 180

2023 arXiv
[103]

Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, and F. Wei. Kosmos-2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

2023 arXiv
[104]

Perez and I

F. Perez and I. Ribeiro. Ignore previous prompt: Attack techniques for lan- guage models. arXiv preprint arXiv:2211.09527, 2022

2022 arXiv
[105]

Pruthi, F

G. Pruthi, F. Liu, S. Kale, and M. Sundararajan. Estimating training data influence by tracing gradient descent.Advances in Neural Information Processing Systems, 33:19920–19930, 2020

2020
[106]

Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023

2023 arXiv
[107]

R. Qiu, G. L. Zhang, R. Drechsler, U. Schlichtmann, and B. Li. Autobench: Automatic testbench generation and evaluation using llms for hdl design, 2024

2024
[108]

Rafailov, A

R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advancesin Neural Information Processing Systems, 36, 2024

2024
[109]

M. F. Rahman, W. Liu, S. B. Suhaim, S. Thirumuruganathan, N. Zhang, and G. Das. Hdbscan: Density based clustering over location based services.arXiv preprint arXiv:1602.03730, 2016

2016 arXiv
[110]

S. N. Roy. On a heuristic method of test construction and its use in multivari- ate analysis. The Annals of Mathematical Statistics, 24(2):220–238, 1953

1953
[111]

Roziere, J

B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, 181 J. Liu, T. Remez, J. Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

2023 arXiv
[112]

Schick, J

T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach them- selves to use tools.arXiv preprint arXiv:2302.04761, 2023

2023 arXiv
[113]

Schoch, R

S. Schoch, R. Mishra, and Y. Ji. Data selection for fine-tuning large language models using transferred shapley values. arXiv preprint arXiv:2306.10165, 2023

2023 arXiv
[114]

D. W. Scott. Scott’s rule. Wiley Interdisciplinary Reviews: Computational Statistics, 2(4):497–502, 2010

2010
[115]

Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

2024 arXiv
[116]

Singh, J

A. Singh, J. D. Co-Reyes, R. Agarwal, A. Anand, P. Patil, P. J. Liu, J. Harri- son, J. Lee, K. Xu, A. Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models.arXiv preprint arXiv:2312.06585, 2023

2023 arXiv
[117]

Snell, J

C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling llm test-time compute opti- mally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

2024 arXiv
[118]

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv:2010.02502, October 2020. 182

2010 arXiv
[119]

Y. Song, C. Lothritz, D. Tang, T. F. Bissyandé, and J. Klein. Revisiting code similarity evaluation with abstract syntax tree edit distance, 2024

2024
[120]

Sorscher, R

B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning.Advancesin Neural Information Processing Systems, 35:19523–19536, 2022

2022
[121]

Sleep-dependentmemory consolidation

R.Stickgold. Sleep-dependentmemory consolidation. Nature, 437(7063):1272– 1278, 2005

2005
[122]

H. Su, J. Kasai, C. H. Wu, W. Shi, T. Wang, J. Xin, R. Zhang, M. Osten- dorf, L. Zettlemoyer, N. A. Smith, et al. Selective annotation makes language models better few-shot learners.arXiv preprint arXiv:2209.01975, 2022

2022 arXiv
[123]

Lab: Large-scale alignment for chatbots, 2024

S.Sudalairaj, A.Bhandwaldar, A.Pareja, K.Xu, D.D.Cox, andA.Srivastava. Lab: Large-scale alignment for chatbots, 2024

2024
[124]

Takamaeda-Yamazaki

S. Takamaeda-Yamazaki. Pyverilog: A python-based hardware design process- ing toolkit for verilog hdl. InApplied Reconfigurable Computing, volume 9040 of Lecture Notes in Computer Science, pages 451–460. Springer International Publishing, Apr 2015

2015
[125]

Q. Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

2025
[126]

TehraniJamsaz, A

A. TehraniJamsaz, A. Bhattacharjee, L. Chen, N. K. Ahmed, A. Yazdan- bakhsh, and A. Jannesari. Coderosetta: Pushing the boundaries of unsuper- vised code translation for parallel programming, 2024

2024
[127]

Thakur, B

S. Thakur, B. Ahmad, H. Pearce, B. Tan, B. Dolan-Gavitt, R. Karri, and 183 S. Garg. Verigen: A large language model for verilog code generation.arXiv preprint arXiv:2308.00708, 2023

2023 arXiv
[128]

Touvron, T

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and eﬀicient foundation language models.arXiv preprint arXiv:2302.13971, 2023

2023 arXiv
[129]

Touvron, L

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bash- lykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

2023 arXiv
[130]

T. H. Trinh, Y. Wu, Q. V. Le, H. He, and T. Luong. Solving olympiad geometry without human demonstrations.Nature, 625(7995):476–482, 2024

2024
[131]

Tsai, Y.-D

T.-H. Tsai, Y.-D. Tsai, and S.-D. Lin. lil＇hdoc: an algorithm for good arm iden- tification under small threshold gap. InPacific-AsiaConference on Knowledge Discovery and Data Mining, pages 78–89. Springer, 2024

2024
[132]

Y. Tsai, M. Liu, and H. Ren. Rtlfixer: Automatically fixing rtl syntax errors with large language models.arXiv preprint arXiv:2311.16543, 2023

2023 arXiv
[133]

Tsai and S.-D

Y.-D. Tsai and S.-D. Lin. Handling concept drift in non-stationary bandit through predicting future rewards. InPacific-Asia Conference on Knowledge Discovery and Data Mining, pages 161–173. Springer, 2024

2024
[134]

Y.-D. Tsai, C. Liow, Y. S. Siang, and S.-D. Lin. Toward more generalized malicious url detection models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21628–21636, 2024. 184

2024
[135]

Y.-D. Tsai, M. Liu, and H. Ren. Code less, align more: Eﬀicient llm fine-tuning for code generation with data pruning.arXiv preprint arXiv:2407.05040, 2024

2024 arXiv
[136]

Tsai, T.-H

Y.-D. Tsai, T.-H. Tsai, and S.-D. Lin. Differential good arm identification. arXiv preprint arXiv:2303.07154, 2023

2023 arXiv
[137]

Tsai, Y.-C

Y.-D. Tsai, Y.-C. Tsai, B.-W. Huang, C.-P. Yang, and S.-D. Lin. Automl-gpt: Large language model for automl.arXiv preprint arXiv:2309.01125, 2023

2023 arXiv
[138]

Tsai, T.-Y

Y.-D. Tsai, T.-Y. Yen, P.-F. Guo, Z.-Y. Li, and S.-D. Lin. Text-centric align- ment for multi-modality learning.arXiv preprint arXiv:2402.08086, 2024

2024 arXiv
[139]

Tsai, T.-Y

Y.-D. Tsai, T.-Y. Yen, K.-T. Liao, and S.-D. Lin. Enhance modality robust- ness in text-centric multimodal alignment with adversarial prompting.arXiv preprint arXiv:2408.09798, 2024

2024 arXiv
[140]

Tufano, C

M. Tufano, C. Watson, G. Bavota, M. D. Penta, M. White, and D. Poshy- vanyk. An empirical study on learning bug-fixing patches in the wild via neural machine translation.ACM Trans. Softw. Eng. Methodol., 28(4), Sept. 2019

2019
[141]

Tzeng, J

E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial discriminative domain adaptation. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7167–7176, 2017

2017
[142]

Vinyals, A

O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2015. 185

2015
[143]

M. P. Walker and R. Stickgold. Sleep-dependent learning and memory consol- idation. Neuron, 44(1):121–133, 2004

2004
[144]

A. J. Wang, K. Q. Lin, D. J. Zhang, S. W. Lei, and M. Z. Shou. Too large; data reduction for vision-language pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 3147–3157, 2023

2023
[145]

Anoverviewofimagecaptiongenerationmeth- ods

H.Wang, Y.Zhang, andX.Yu. Anoverviewofimagecaptiongenerationmeth- ods. Computational intelligence and neuroscience, 2020(1):3062706, 2020

2020
[146]

S. Wang, Z. Zhao, X. Ouyang, Q. Wang, and D. Shen. Chatcad: Interactive computer-aideddiagnosisonmedicalimageusinglargelanguagemodels. arXiv preprint arXiv:2302.07257, 2023

2023 arXiv
[147]

Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Ha- jishirzi. Self-instruct: Aligning language models with self-generated instruc- tions. arXiv preprint arXiv:2212.10560, 2022

2022 arXiv
[148]

J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652, 2021

2021 arXiv
[149]

J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

2023
[150]

Chain-of-thought prompting elicits reasoning in large language models

J.Wei, X.Wang, D.Schuurmans, M.Bosma, F.Xia, E.Chi, Q.V.Le, D.Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advancesin neural information processing systems, 35:24824–24837, 2022. 186

2022
[151]

Y. Wei, O. Duchenne, J. Copet, Q. Carbonneaux, L. Zhang, D. Fried, G. Syn- naeve, R. Singh, and S. I. Wang. Swe-rl: Advancing llm reasoning via reinforce- ment learning on open software evolution.arXiv preprint arXiv:2502.18449, 2025

2025 arXiv
[152]

Y. Wei, Z. Wang, J. Liu, Y. Ding, and L. Zhang. Magicoder: Source code is all you need.arXiv preprint arXiv:2312.02120, 2023

2023 arXiv
[153]

Y. Weng, M. Zhu, F. Xia, B. Li, S. He, S. Liu, B. Sun, K. Liu, and J. Zhao. Large language models are better reasoners with self-verification, 2023

2023
[154]

Williams and M

S. Williams and M. Baxter. Icarus verilog: open-source verilog more than a year later.Linux Journal, 2002(99):3, 2002

2002
[155]

Wortsman, P

M. Wortsman, P. J. Liu, L. Xiao, K. Everett, A. Alemi, B. Adlam, J. D. Co- Reyes, I. Gur, A. Kumar, R. Novak, J. Pennington, J. Sohl-dickstein, K. Xu, J. Lee, J. Gilmer, and S. Kornblith. Small-scale proxies for large-scale trans- former training instabilities, 2023

2023
[156]

Y. Wu, D. Huang, W. Shi, W. Wang, L. Gao, S. Liu, Z. Nan, K. Yuan, R. Zhang, X. Zhang, Z. Du, Q. Guo, Y. Pu, D. Yin, X. Hu, and Y. Chen. In- versecoder: Unleashing the power of instruction-tuned code llms with inverse- instruct, 2024

2024
[157]

Wu, Y.-D

Y.-A. Wu, Y.-D. Tsai, and S.-D. Lin. Linearapt: An adaptive algorithm for the fixed-budget thresholding linear bandit problem. arXiv preprint arXiv:2403.06230, 2024

2024 arXiv
[158]

C. S. Xia, Y. Wei, and L. Zhang. Automated program repair in the era of 187 large pre-trained language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1482–1494, 2023

2023
[159]

M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen. Less: Selecting in- fluential data for targeted instruction tuning.arXivpreprintarXiv:2402.04333, 2024

2024 arXiv
[160]

T. Xie, Z. Gao, Q. Ren, H. Luo, Y. Hong, B. Dai, J. Zhou, K. Qiu, Z. Wu, and C. Luo. Logic-rl: Unleashing llm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14768, 2025

2025 arXiv
[161]

C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, and D. Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

2023 arXiv
[162]

Xu and W

Y. Xu and W. Wang. Linkprompt: Natural and universal adversarial attacks on prompt-based language models. InProceedings of the 2024 Conference of theNorthAmericanChapteroftheAssociation forComputationalLinguistics: Human Language Technologies (Volume 1: Long Papers), pages 647...

2024
[163]

Y. Xu, Y. Yao, Y. Huang, M. Qi, M. Wang, B. Gu, and N. Sundaresan. Rethinking the instruction quality: Lift is what you need, 2023

2023
[164]

Yang, P.Huang, J

Y. Yang, P.Huang, J. Cao, J. Li, Y. Lin, and F. Ma. A prompt-based approach to adversarial example generation and robustness enhancement.Frontiers of Computer Science, 18(4):184318, 2024

2024
[165]

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. Re- 188 act: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022

2022 arXiv
[166]

Yen, Y.-D

T.-Y. Yen, Y.-D. Tsai, K.-T. Liao, and S.-D. Lin. Enhance the robustness of text-centric multimodal alignments.arXiv preprint arXiv:2407.05036, 2024

2024 arXiv
[167]

Young, A

P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014

2014
[168]

Z. Yu, X. Zhang, N. Shang, Y. Huang, C. Xu, Y. Zhao, W. Hu, and Q. Yin. Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning, 2024

2024
[169]

Zhang, Z

B. Zhang, Z. Liu, C. Cherry, and O. Firat. When scaling meets llm fine- tuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193, 2024

2024 arXiv
[170]

Zhang, S

D. Zhang, S. Zhoubian, Y. Yue, Y. Dong, and J. Tang. Rest-mcts*: Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.03816, 2024

2024 arXiv
[171]

Zhang, B

F. Zhang, B. Chen, Y. Zhang, J. Liu, D. Zan, Y. Mao, J.-G. Lou, and W. Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570, 2023

2023 arXiv
[172]

Zhang, P.-N

H. Zhang, P.-N. Kung, M. Yoshida, G. Van den Broeck, and N. Peng. Adapt- able logical control for large language models.Advancesin Neural Information Processing Systems, 37:115563–115587, 2024. 189

2024
[173]

Zhang, G

K. Zhang, G. Li, Y. Dong, J. Xu, J. Zhang, J. Su, Y. Liu, and Z. Jin. Codedpo: Aligning code models with self generated and verified source code, 2024

2024
[174]

Y. Zhao, D. Huang, C. Li, P. Jin, Z. Nan, T. Ma, L. Qi, Y. Pan, Z. Zhang, R. Zhang, et al. Codev: Empowering llms for verilog generation through multi-level summarization.arXiv preprint arXiv:2407.10424, 2024

2024 arXiv
[175]

C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy. Lima: Less is more for alignment, 2023

2023
[176]

B. Zhu, B. Lin, M. Ning, Y. Yan, J. Cui, H. Wang, Y. Pang, W. Jiang, J. Zhang, Z. Li, et al. Languagebind: Extending video-language pretrain- ing to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852, 2023

arXiv
[177]

J. Zhu, Y. Shen, D. Zhao, and B. Zhou. In-domain gan inversion for real image editing. InEuropean conference on computer vision, pages 592–608. Springer, 2020. 190 Appendix A — Multimodal Mismatch Ex- periment Detail Setup A.1 Model Checkpoints We conduct all experiments GPT-4...

2020
[179]

The tabluar column has : [host_response_time, host_response_rate,…]
[180]

The following is the meaning of each column, brackets represent the options, you must in- clude each column

Please start with ”The host’s response time to booking inquiries is”. The following is the meaning of each column, brackets represent the options, you must in- clude each column. If there are brackets, you must select one of the options, if there are no brackets and for numeri...
[181]

If the original table contains columns that are marked as ”Unknown,” the conversion into natural text will omit any mention of those particular attributes
[182]

Avoid fabricating or inventing any content!
[183]

All information should naturally blend together from different modalities, just like the contents of the same paragraph, and the order should be switched and blended, but not missing any information
[184]

Examples Input: The host’s response time to booking inquiries is within an hour

Please start with ”This homestay is” and summarize into 1 to 2 paragraph. Examples Input: The host’s response time to booking inquiries is within an hour. The host’s response rate to booking inquiries is 96%.… Answer: This homestay is a modern and inviting entire condo located...
[185]

Each modality is first processed by a modality-specific encoder 198
[186]

Representations are aligned to a common embedding space
[187]

The LLM performs cross-modal attention to integrate information
[188]

A text-centric decoder generates the final output C.1.3 Perturbation Methods For robustness testing, we applied various types of perturbations: • Text: Character/word/sentence deletion, substitution, and reordering • Images: Gaussian noise, blur, cropping, and color distortion...
[189]

Potential adopters might be hesitant to take on a pet that requires special care, even if the injury is minor

**Health Condition**: Jack Jack has a minor injury and is recovering. Potential adopters might be hesitant to take on a pet that requires special care, even if the injury is minor. Figure D.5:Alignment module refining noisy text input into coherent and structured descriptions....
[190]

This is a positive factor that could help in his adoption

**Breed and Size**: Jack Jack is a Chihuahua, a breed that is generally popular due to their small size and suitability for apartment living. This is a positive factor that could help in his adoption
[191]

Puppies tend to get adopted faster than adult dogs, which may slow down his adoption rate

**Age**: At 36 months (3 years old), Jack Jack is no longer a puppy. Puppies tend to get adopted faster than adult dogs, which may slow down his adoption rate
[192]

Potential adopters might be hesitant to take on a pet that requires special care, even if the injury is minor

**Health Condition**: Jack Jack has a minor injury and is recovering. Potential adopters might be hesitant to take on a pet that requires special care, even if the injury is minor
[193]

**Vaccination and Deworming**: The fact that Jack Jack is vaccinated and dewormed is a positive aspect and can reassure potential adopters about his health
[194]

Many adopters prefer pets that are already neutered to avoid the cost and re- sponsibility of the procedure

**Sterilisation Status**: Jack Jack is not sterilized, which could be a concern for some adopters. Many adopters prefer pets that are already neutered to avoid the cost and re- sponsibility of the procedure
[195]

A well-written profile can significantly impact adoption rates

**Profile Write-Up**: The profile write-up for Jack Jack is somewhat unclear and lacks detailed information that could appeal to potential adopters. A well-written profile can significantly impact adoption rates
[196]

More photos and possibly videos could help showcase Jack Jack’s personality and increase interest

**Photos**: There are only 3 photos uploaded. More photos and possibly videos could help showcase Jack Jack’s personality and increase interest
[197]

However, the adoption rate can still vary based on local preferences and the number of available pets

**Location**: Jack Jack is located in Selangor, Malaysia, which has a significant popu- lation and potentially a larger pool of adopters. However, the adoption rate can still vary based on local preferences and the number of available pets. Figure D.6: LLM extracting semantic ...
[203]

Figure D.7: Cross-modal information synthesis for reconstructing missing tabular data

**Location and Accessibility**: The profile does not specify the exact location beyond being at a vet clinic, which might make it less accessible for potential adopters who prefer to know more about where the pet is currently staying. Figure D.7: Cross-modal information synthe...
[204]

Potential adopters might be hesitant to take on a pet that requires special care, even if the injury is minor

**Health Condition**: Jack Jack has a minor injury and is recovering. Potential adopters might be hesitant to take on a pet that requires special care, even if the injury is minor. Figure E.9: Noise compensation in textual data. The alignment module transforms highly fragmente...
[205]

Younger pets, especially puppies and kittens, generally have higher adoption rates because they are perceived as more adaptable and have a longer potential lifespan

**Age**: At 78 months (6.5 years old), Filo is significantly older than the other pets listed. Younger pets, especially puppies and kittens, generally have higher adoption rates because they are perceived as more adaptable and have a longer potential lifespan
[206]

**Health and Background**: While Filo has been treated for his external wounds, his background as a stray and his extended stay in a cage might raise concerns for potential adopters about his health and behavior. The write-up mentions his gratefulness and calm demeanor, but it...
[207]

The lan- guage used to describe Filo’s situation is more factual and less engaging compared to de- scriptions of other pets, which emphasize their cuteness and playful nature

**Emotional Connection**: The profile write-up is heartfelt and provides a touching backstory, but it lacks the emotional appeal seen in the profiles of younger pets. The lan- guage used to describe Filo’s situation is more factual and less engaging compared to de- scriptions ...
[208]

Potential adopters often respond more positively to images that show the pet’s personality and energy

**Visual Appeal**: Although there are 5 photos, the description of the image shows Filo in a resting position, which may not be as engaging as images of playful or interactive behavior. Potential adopters often respond more positively to images that show the pet’s personality ...
[209]

His profile needs to stand out more to attract potential adopters who are specifically looking for an older, more mature dog

**Competition**: Filo is competing with younger, more visually appealing pets that are often adopted faster. His profile needs to stand out more to attract potential adopters who are specifically looking for an older, more mature dog
[210]

Figure F.10:Cross-modal information recovery for tabular data

**Location and Accessibility**: The profile does not specify the exact location beyond being at a vet clinic, which might make it less accessible for potential adopters who prefer to know more about where the pet is currently staying. Figure F.10:Cross-modal information recove...
[211]

population count

**Health Condition**: Jack Jack has a minor injury and is recovering. Potential adopters might be hesitant to take on a pet that requires special care, even if the injury is minor. Figure F.12: Compensation for noisy text input through language model reasoning. Despite severel...
[212]

2 error(s) during elaboration

begin 7 out[i] <= in[99 - i]; 8 end 9 end 10 endmodule compiler error message iverilog vector100r.sv:5: error: Unable to bind wire/reg/memory `clk' in `top_module' vector100r.sv:5: error: Failed to evaluate event expression 'posedge clk'. 2 error(s) during elaboration. ModelSi...
[213]

Concatenating the two 1 bits at the beginning of the concatenated vector
[214]

Problem Description You are given a Verilog module that demonstrates the use of bit slicing and concatenation in a loop

Assign the output vectors from the concatenated vector in the correct order and bit ranges For example: assign {w, x, y, z} = {2'b11 , a, b, c , d, e , f }; To correct the implementation, you should concatenate the last bit as ‘2’b11’ on the right, as shown in the correct impl...
[215]

Instead of accessing ‘data[8*(3-i) +: 8]’, you should access ‘data[8*i +: 8]’ to display the correct slices of the register

**Correct the loop:** The loop currently reverses the order of the 8-bit slices. Instead of accessing ‘data[8*(3-i) +: 8]’, you should access ‘data[8*i +: 8]’ to display the correct slices of the register
[216]

Correct the slicing order for accurate display

**Fix the slicing outside of the loop:** Ensure that the LSB (‘data[7:0]’) corresponds to the lower bits of the ‘data’ register, and the MSB (‘data[31:24]’) corresponds to the higher bits. Correct the slicing order for accurate display. The corrected implementation should ensu...
[217]

Identify the line of code that performs the shifting operation
[218]

Notice that the least significant bit (q[0]) is connected to the most significant bit (q[3]) instead of a zero
[219]

Change the line of code to connect a zero to the most significant bit and shift the re- maining bits to the right
[220]

248 LLM constructed Repair data from open-source code Input: You are provided with a Verilog module that implements a shift register

Test the implementation to ensure that it behaves as expected. 248 LLM constructed Repair data from open-source code Input: You are provided with a Verilog module that implements a shift register. However, the implementation contains a common error related to shifting operatio...
[221]

**Identify the shifting operation error:** The line ‘shreg = {shreg[WIDTH-2:0], shreg[WIDTH-1]};’ causes a circular shift because the MSB is incorrectly fed back into the LSB position
[222]

The correct line should be ‘shreg = {shreg[WIDTH-2:0], SI};’ to shift in the new input bit from the ‘SI‘ input and ensure the bits shift to the right

**Fix the shifting logic:** Replace the erroneous MSB assignment with the correct new input bit (‘SI’). The correct line should be ‘shreg = {shreg[WIDTH-2:0], SI};’ to shift in the new input bit from the ‘SI‘ input and ensure the bits shift to the right. By fixing the shifting...
[223]

In this case, it’s the block that starts with always @(posedge clk or posedge reset)

Locate the always block that handles the reset and clock edge. In this case, it’s the block that starts with always @(posedge clk or posedge reset)
[224]

251 LLM constructed Repair data from open-source code Input: Problem Description: You are given a Verilog module that implements a 4-bit up counter

Within the if (reset) block, add a line to initialize the p register to zero: p <= 0;. 251 LLM constructed Repair data from open-source code Input: Problem Description: You are given a Verilog module that implements a 4-bit up counter. The counter should reset to zero when the...
[225]

This causes incorrect counting behavior after a reset

Identify the initialization issue: The counter should reset to zero, but currently, it is being set to 4’d3 during reset. This causes incorrect counting behavior after a reset
[226]

Change counter_up <= 4’d3; to counter_up <= 4’d0

Fix the reset logic: Modify the if (reset) block in the always block to properly initialize the counter to zero. Change counter_up <= 4’d3; to counter_up <= 4’d0
[227]

Ensure proper reset behavior: After fixing the initialization, the counter will correctly reset to zero when the reset signal is active and increment correctly on each clock cycle thereafter. Output: module up_counter( input clk , input reset , output [3:0] counter ); reg [3:0...
[232]

Now, Please use your creativity to create a brand new high-quality Verilog problem

The problem description section should be enclosed within <PROBLEM> </PROB- LEM> tags. Now, Please use your creativity to create a brand new high-quality Verilog problem. Figure I.40: Prompt used to generate initial 50 seed problems for Self-Instruct. 266 Your goal is to creat...
[236]

The problem should be desinged for the programmers to solve it with one verilog mod- ule
[237]

Below shows some examples: <PROBLEM> {seed problems} </PROBLEM> Now, Please use your creativity to create a brand new high-quality Verilog problem

The problem description section should be enclosed within <PROBLEM> </PROB- LEM> tags. Below shows some examples: <PROBLEM> {seed problems} </PROBLEM> Now, Please use your creativity to create a brand new high-quality Verilog problem. Figure I.41: Prompt used for Self-Instruct...
[241]

* Guidelines for the problem description format: The problem description section should be enclosed within <PROBLEM> </PROBLEM> tags

The problem should be designed for the programmers to solve it with one Verilog mod- ule. * Guidelines for the problem description format: The problem description section should be enclosed within <PROBLEM> </PROBLEM> tags. Please increase the diﬀiculty of the given programmin...
[242]

Your new problem should not be directly solved by the original code snippet
[243]

If you do have a reset method that is synchronous to a clock, make sure to add the clock signal to the problem module input

You can also change the bit-width requiremnt, how to reset internal signals (if applica- ble), and whether the solution needs a clock signal (combinatorial versus sequential logic). If you do have a reset method that is synchronous to a clock, make sure to add the clock signal...
[244]

Add new constraints and requirements to the original problem, adding approximately 10 additional words
[245]

Replace a commonly used requirement in the programming task with a less common and more specific one
[246]

Now, Please gain inspiration from the following random code snippet to create a high- quality Verilog problem

If the original problem can be solved with only a few logical steps, please add more rea- soning steps. Now, Please gain inspiration from the following random code snippet to create a high- quality Verilog problem. Code snippet for inspiration: ‘‘‘ {code snippet} ‘‘‘ Output: 2...
[247]

This should be **completely self-contained**, providing all the contextual information one needs to understand and solve the problem
[248]

Assume common verilog knowledge, but ensure that any specific context, variables, or code snippets pertinent to this problem are explicitly included
[249]

Do not include the code snippet in the problem
[250]

Yes” if you are certain that the{Wikipedia title} is related to hardware design or Verilog coding language. Your answer should start with “Yes

The problem should be designed for the programmers to solve it with one Verilog mod- ule. * Guidelines for the problem description format: The problem description section should be enclosed within <PROBLEM> </PROBLEM> tags. Now, Please gain inspiration from the following textb...
[251]

The module should be **completely self-contained**, fulfilling all the requirements needed to solve the problem
[252]

Ensure the correctness of the syntax and functionality
[253]

top_module

The module name must be “top_module”
[254]

Do not add blackslash in front of underscore symbol
[255]

The solution must be valid Verilog code
[256]

The solution section should be enclosed within <SOLUTION> </SOLUTION> tags
[257]

The reasoning section should be enclosed within <REASON> </REASON> tags
[258]

Try to embed the reasoning in comments

Write comments in the solution section. Try to embed the reasoning in comments. Always try to write the corresponding comments before the code
[259]

True” or “False

The module should be **completely self-contained**, do not include or import outside the module and define everything inside the module. Below shows an example: Problem description: Build a counter that counts from 0 to 999, inclusive , with a period of 1000 cycles . The reset...
[260]

Erroneous implementation
[261]

However, there is a bug in the code which causes incorrect results

Hints for fixing Here is an example: <EXAMPLE> The following Verilog module is intended to implement the specification below. However, there is a bug in the code which causes incorrect results. Please fix the bug to make the module work as intended. Erroneous Implementation: /...
[262]

Verify the bit-width of the counter and the increment operation
[263]

Check the initialization and wrapping condition of the counter
[264]

</EXAMPLE> Now, here is the commonly made error: ‘‘‘ {error report} ‘‘‘ Inject the above error into the following module and create an error repair practice problem

Ensure that the addition operation correctly handles the 4-bit counter. </EXAMPLE> Now, here is the commonly made error: ‘‘‘ {error report} ‘‘‘ Inject the above error into the following module and create an error repair practice problem. Check if it is possible to inject the e...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Accessed on: 10 September, 2023

Inside airbnb : Hawaii, 2023. Accessed on: 10 September, 2023

2023

[2] [2]

Achiam, S

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 techni- cal report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

[3] [3]

Alayrac, J

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A.Mensch, K.Millican, M.Reynolds, etal. Flamingo: avisuallanguagemodel for few-shot learning. Advances in Neural Information Processing Systems, 35:23716–23736, 2022

2022

[4] [4]

Claude 3.5 and claude 3.7 models.https://www.anthropic.com/ news/claude-3-5-and-3-7, 2024

Anthropic. Claude 3.5 and claude 3.7 models.https://www.anthropic.com/ news/claude-3-5-and-3-7, 2024. Accessed: 2025-04-27

2024

[5] [5]

Austin, A

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

arXiv 2021

[6] [6]

J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 163

arXiv 2023

[7] [7]

Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

arXiv 2022

[8] [8]

Batten, N

C. Batten, N. Pinckney, M. Liu, H. Ren, and B. Khailany. Pyhdl-eval: An llm evaluation framework for hardware design using python-embedded dsls. In Proceedings of the 2024 ACM/IEEE International Symposium on Machine Learning for CAD, MLCAD ’24, New York, NY, USA, 2024. Association for Computing Machinery

2024