LLM-Supported Natural Language to Bash Translation

Erik Hemberg; Finnian Westenfelder; Miguel Tulla; Silviu Chiricescu; Stephen Moskal; Una-May O'Reilly

REVIEW 4 major objections 4 minor 55 references

LLM-Supported Natural Language to Bash Translation

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that executing both a candidate Bash command and a ground truth command, then asking an LLM to judge whether the outputs satisfy the original prompt, can determine functional equivalence with 95% accuracy, a 16-point gain…

desk verdict Useful dataset + benchmark paper, but the 95% FEH accuracy is measured on rotation-based negatives and the downstream model comparisons use a different, less accurate FEH. read the letter →

arxiv 2502.06858 v1 pith:4MWQKP34 submitted 2025-02-07 cs.CL cs.AI

Finnian Westenfelder , Erik Hemberg , Miguel Tulla , Stephen Moskal , Una-May O'Reilly , Silviu Chiricescu This is my paper · ORCID

classification cs.CLcs.AI

keywords naturallanguagetoBashtranslationfunctionalequivalenceheuristicLLMevaluationcommandexecutionbenchmarkNL2SHdatasetin-contextlearningLoRAfine-tuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to make natural-language-to-Bash (NL2SH) evaluation trustworthy. The authors find that over half of the pairs in the widely used InterCode-Bash test set are erroneous, and that existing equivalence heuristics miss many correct answers. They respond with a manually verified test set of 600 instruction-command pairs and a training set of 40,939 pairs, plus a functional equivalence heuristic that executes both the ground truth and the candidate command and asks an LLM to judge whether the candidate's output accomplishes the task. On their test set this heuristic reaches 95% accuracy, 16 percentage points above the previous best. They also show that parsing, in-context learning, constrained decoding, and fine-tuning each improve model accuracy, with the best evaluated model at 74%.

What carries the argument

The functional equivalence heuristic (FEH) is the load-bearing mechanism. It combines dynamic execution with an LLM judge: the candidate command and ground truth are each run in a fresh Docker container; the resulting standard outputs, together with the original natural-language task and the two commands, are passed to an LLM prompted to output only 'true' or 'false'. This replaces the TF-IDF output comparison used by InterCode with semantic judgment conditioned on the task, which is what lets commands such as 'du -s .' and 'du -d 0 -h' be recognized as equivalent despite different output text.

What would settle it

Have a set of independent Bash experts re-label a random sample of the 600 test pairs without seeing the heuristic's answers, then compare; if expert agreement with the supplied labels is well below 95%, or if the rotated negatives turn out to be much easier to reject than actual model near-misses, the reported accuracy would not generalize.

Watch

Extended reading notes

Core claim

The central claim is that command execution paired with LLM evaluation of command outputs can determine the functional equivalence of two Bash commands with 95% accuracy. The proposed heuristic runs the ground-truth command and the model's command in identical Docker containers, captures standard output, and feeds the task text, both commands, and both outputs to an LLM that returns 'true' or 'false'. This design repairs the two failure modes the paper identifies in prior work: execution-based heuristics that compare outputs with TF-IDF miss semantically equivalent but syntactically different outputs, while pure LLM judgment without execution is nearly useless (recall 0.05 for Llama-3.1-8B). With execution, the same LLM's recall jumps to 0.74, and GPT-4 reaches 0.91 recall with 0.95 accuracy. The paper also reports a manual audit showing 50.4% of InterCode-Bash pairs contain prompt, command, or environment errors, motivating the corrected and expanded dataset.

Load-bearing premise

The 95% accuracy claim assumes the authors' manually assigned equivalence labels are correct, and that the negative examples created by rotating the third column of the dataset by ten positions are representative of the mistakes real models make.

Editorial extensions

If this is right

If the heuristic holds up, NL2SH benchmarks can move from parse- and string-based similarity to execution-plus-judgment, so valid alternative commands count as correct.
The finding that half of InterCode-Bash pairs are erroneous implies that earlier model comparisons run on that benchmark need to be revisited with cleaned data.
Output parsing, in-context learning, constrained decoding, and LoRA fine-tuning each buy accuracy gains of up to 32%, but the paper finds gains shrink as model size grows.
At 74% best-case accuracy, NL2SH translation remains too unreliable for unverified use; the paper's own safety recommendation is sandboxing and human review.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper evaluates its headline 95% FEH with GPT-4 as the judge, but evaluates model translation accuracy with a different judge (execution plus the mxbai embedding model); a direct comparison of the two judges on identical model outputs would clarify which one is responsible for the reported gains.
The negative-example construction (rotating one column of the test set) may understate the difficulty of distinguishing near-miss commands, so the 95% could be an upper bound for real-world use; a harder negative set built from common model errors would be a sharper test.
The same execution-plus-LLM pattern could extend to other side-effect-heavy code-generation tasks, but the paper only compares standard output, not filesystem state or process side effects, so the method's scope beyond stdout remains untested.
Because the FEH relies on a strong LLM and an execution sandbox, its cost and stochasticity likely confine it to offline benchmarking; interactive or large-scale evaluation may still need cheaper approximations.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take: this is a useful, well-scoped empirical contribution, but the paper's headline number is not as solid as the abstract makes it sound.

The genuinely valuable parts are the manual audit of InterCode, which found over half of the pairs erroneous, and the resulting corrected, expanded test set with two equivalent commands per instruction. Releasing the code and benchmark configuration also makes this a practical resource. The FEH idea, execute both commands then have an LLM judge the outputs against the prompt, is reasonable, and on their own test set it does beat the prior heuristics they compare against. The model evaluation section is workmanlike and the finding that parsing and ICL reliably help is plausible.

The soft spots are concentrated in the FEH evaluation. The 300 negative pairs are created by rotating the second command column by ten positions, so each negative is a command written for a completely different task. Real model failures tend to be near-misses, an extra flag, a different utility, a missing argument, and those are exactly the cases that are harder to judge. So the reported 95% accuracy is probably optimistic for the use case that matters. Also, that number has no confidence intervals or repeated-run variance, even though the FEH uses a stochastic LLM; calling it '95% confidence' in the abstract is sloppy language. On top of that, the downstream model comparisons in Table 5 use the exec + mxbai-embed FEH, which scored 90% accuracy, not the exec + GPT-4 FEH that achieved 95%. That mismatch doesn't invalidate either result, but it means the advertised 95% heuristic isn't actually the one used to rank the models, and the practical benefit of the more expensive heuristic is left untested. The manual equivalence labels are also author-assigned without an independent agreement check, which is less critical but worth noting.

None of this is fatal. The dataset and benchmark are real contributions, and the central direction, execution-derived outputs plus LLM judgment, is sound. I'd send it to review, with the expectation that the authors add a more realistic FEH evaluation, report variance, and correct the confidence wording.

Who this is for: anyone working on NL2SH or execution-based code evaluation. It's a solid read for a reading group, and I'd cite it if I were working in that space.

Referee Report

4 major / 4 minor

Summary. The paper introduces NL2SH-ALFA, a manually verified test dataset of 600 instruction-command pairs and a training dataset of 40,939 pairs, alongside a functional equivalence heuristic (FEH) that combines command execution with LLM evaluation of the resulting outputs. The authors report that this FEH determines functional equivalence with 95% accuracy (a 16% improvement over prior heuristics) and use it (in a variant with a different embedding model) to evaluate the impact of parsing, constrained decoding, in-context learning, and in-weight learning on Llama, Qwen, and GPT models, claiming accuracy improvements of up to 32%.

Significance. If the 95% FEH accuracy claim holds, the paper would provide a substantially more reliable automatic evaluator for NL2SH translation than the existing execution-based or LLM-only heuristics, and the released dataset and code would be valuable community resources. The paper is honest about several limitations (test-set size, unverified training data, LLM stochasticity) and it compares against multiple baselines. However, the central FEH claim rests on a synthetic negative set and single-annotator labels, and the model evaluations in Section 5.2 actually use a different, lower-accuracy FEH variant, so the headline result is not directly used in the translation-method comparison. These issues are load-bearing for the paper's central claims.

major comments (4)

[§5.1, Table 4] The FEH accuracy of 95% is measured on non-equivalent pairs created by rotating the third column of the test dataset by ten positions. This produces pairs where the model command is associated with a different natural-language task, which are likely far easier to reject than realistic model-generated errors that differ by a single flag or utility and produce partially overlapping outputs. The paper does not evaluate the FEH on any of the model outputs generated in Section 5.2, so the reported precision, recall, and accuracy may overstate the heuristic's performance in the actual benchmark use case. I recommend testing on model-generated commands or on adversarially perturbed ground-truth commands (e.g., single-flag changes) to demonstrate robustness.
[§5.2, Table 5] All translation-method accuracies in Table 5 are computed using the exec + mxbai-embed FEH (accuracy 0.90 in Table 4), not the headline exec + gpt-4-0613 FEH (accuracy 0.95). The manuscript does not explain this choice or report how model rankings would change under the 95%-accurate FEH. Since the paper's third contribution is to evaluate translation methods using the proposed benchmark and FEH, the mismatch between the evaluated FEH and the one used for model comparison is a significant gap that should be addressed.
[§4.1, §5.1] The 300 functionally equivalent pairs used in the FEH evaluation are labeled solely by the authors; no inter-annotator agreement or second-rater verification is reported. Because the 95% accuracy figure is measured against these manual labels, systematic labeling errors would directly distort the central claim. An independent annotation pass or at least a randomized audit with agreement statistics is needed.
[Abstract, §6.2] The abstract claims the FEH can determine equivalence 'with 95% confidence,' while Section 6.2 phrases the same result as 'with 95% accuracy.' A point estimate of accuracy on a fixed test set is not a confidence statement. The manuscript should report a confidence interval or other uncertainty measure, especially given the acknowledged stochasticity of the LLM-based FEH and the single seed/temperature-0 evaluation.

minor comments (4)

[§4.1] The text states that fixing the InterCode errors results in 193 verified pairs and that 117 additional pairs are created, which totals 310, yet the following sentence refers to 'our 300 verified pairs.' Please clarify the discrepancy (e.g., 10 held-out pairs or a typo).
[§5.1] The thresholds for bleu, nl2cmd, tfidf, and mxbai-embed are all fixed at 0.75 with no sensitivity analysis; reporting results across a range of thresholds would strengthen the comparison.
[§4.3.2] The phrase 'average relative probability of the first token' is undefined; please specify the exact metric used (e.g., probability relative to the second token, or normalized log-probability).
[§5.1] The use of temperature 0 and a static seed does not guarantee deterministic outputs for API-based models (e.g., gpt-4-0613); running multiple seeds and reporting variance would better quantify the FEH's variability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the FEH is validated against externally assigned manual labels, and downstream model evaluations use a fixed FEH benchmarked on a separate evaluation task.

full rationale

The paper's derivation chain is not circular in the equation-level sense. The test dataset (Section 4.1) is constructed by manual verification of instruction-command pairs and by manually authoring a second command per prompt; these manual labels are external ground truth, not outputs of the FEH. The FEH comparison (Section 5.1, Table 4) measures each heuristic against those manual labels, and no FEH parameter is fitted to the model outputs it later scores. The downstream translation-method evaluation (Section 5.2, Table 5) uses the exec + mxbai-embed FEH as a fixed scorer after that FEH's 0.90 accuracy was established on the separate FEH-evaluation task; model evaluations do not feed back into the FEH construction. The rotated-column negative set is an evaluation-design choice that may make negatives easier than real model mistakes, and there is a consistency concern that the headline 95% accuracy corresponds to exec + GPT-4 while Table 5 uses the 90% exec + mxbai-embed FEH, but these are correctness and robustness limitations rather than circularity. No load-bearing self-citation, imported uniqueness theorem, or ansatz smuggled in via citation is present; citations to prior work are external. The claimed improvements are therefore self-contained empirical measurements, not reductions of predictions to inputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the authors' manual labels, an ad hoc negative-pair construction, and the execution environment; these are listed as axioms and free parameters above. No new physical or formal entities are postulated.

free parameters (4)

cosine_similarity_threshold = 0.75
Used in FEH comparisons for bleu, nl2cmd, tfidf, and mxbai-embed variants; chosen by hand with no sensitivity analysis, and it directly affects precision and recall in Table 4.
icl_example_count = 25
Number of in-context examples selected as optimal by sweeping 1 to 50 on Llama3.1-8b-Instruct against the new test set; this tunes the method to the evaluation data (Section 4.3.3).
dedup_cosine_threshold = 0.9
Threshold for removing training rows semantically similar to test rows; chosen ad hoc, affects contamination risk between training and test (Section 4.1).
lora_hyperparameters = rank=64, alpha=32, dropout=0.1, batch=32, lr=1e-5, epochs=10
Chosen by limited experimentation under hardware constraints; affects all in-weight learning results in Table 5.

assumptions (4)

domain assumption Manually assigned equivalence labels in NL2SH-ALFA are correct.
Section 4.1 states 100 hours of manual verification produced the 600 pairs, with no inter-annotator agreement or external audit; all FEH and model accuracy numbers inherit this labeling.
ad hoc to paper Rotating the third column by ten positions creates a valid set of non-equivalent command pairs.
Section 5.1 constructs the negative evaluation set this way; if rotated pairs are accidentally equivalent or unrepresentative, the FEH accuracy is overstated.
domain assumption Docker execution captures all functionally relevant behavior, including side effects, through git-diff and MD5 checks plus stdout.
Section 4.2 keeps the InterCode execution environment and replaces only the stdout comparison; side-effect differences outside these checks are not measured.
domain assumption bashlex parsing is a sufficient filter for command validity in the training data.
Section 4.1 removes unparsable commands from the 40,939 training pairs, but parseability does not guarantee that a command accomplishes the stated task.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LLM-Supported Natural Language to Bash Translation." pith.science (2026). https://pith.science/paper/4MWQKP34

@misc{pith2026250206858,
  author       = {Pith},
  title        = {Pith review of: LLM-Supported Natural Language to Bash Translation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4MWQKP34}},
  note         = {Machine review of arXiv:2502.06858}
}

read the original abstract

The Bourne-Again Shell (Bash) command-line interface for Linux systems has complex syntax and requires extensive specialized knowledge. Using the natural language to Bash command (NL2SH) translation capabilities of large language models (LLMs) for command composition circumvents these issues. However, the NL2SH performance of LLMs is difficult to assess due to inaccurate test data and unreliable heuristics for determining the functional equivalence of Bash commands. We present a manually verified test dataset of 600 instruction-command pairs and a training dataset of 40,939 pairs, increasing the size of previous datasets by 441% and 135%, respectively. Further, we present a novel functional equivalence heuristic that combines command execution with LLM evaluation of command outputs. Our heuristic can determine the functional equivalence of two Bash commands with 95% confidence, a 16% increase over previous heuristics. Evaluation of popular LLMs using our test dataset and heuristic demonstrates that parsing, in-context learning, in-weight learning, and constrained decoding can improve NL2SH accuracy by up to 32%. Our findings emphasize the importance of dataset quality, execution-based evaluation and translation method for advancing NL2SH translation. Our code is available at https://github.com/westenfelder/NL2SH

Figures

Figures reproduced from arXiv: 2502.06858 by the authors.

**Figure 2.** A diagram of NL2SH translation with a comparison of functional equivalence heuristics. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Relationships between NL2SH datasets. 4.2 Functional Equivalence Heuristic (FEH) Our evaluation of related work in Section 3 reveals the InterCode benchmark is more accurate than previous NL2SH benchmarks because its FEH uses execution-based evaluation. However, its TFIDF method for comparing command outputs may fail to determine functional equivalence because syntactically different outputs may convey the same in… view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Dangerous translation observed in testing. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Prompt for evaluating the functional equivalence of Bash commands. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: Prompt for evaluating the functional equivalence of Bash commands after execution. Note the addition of [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

**Figure 7.** Figure 7: NL2SH translation prompt used in the baseline evaluation. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8: NL2SH translation prompt used in the parsing, constrained decoding and in-weight learning evaluations. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

**Figure 9.** Figure 9: NL2SH translation prompt used in the in-context learning evaluation. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 36 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Mayank Agarwal, Tathagata Chakraborti, Quchen Fu, David Gros, Xi Victoria Lin, Jaron Maene, Kartik Talamadupula, Zhongwei Teng, and Jules White. 2021. https://proceedings.mlr.press/v133/agarwal21b.html Neurips 2020 nlc2cmd competition: Translating natural language to bash commands . In Proceedings of the NeurIPS 2020 Competition and Demonstration Track, v...

work page 2021
[4]

Pooja Aggarwal, Oishik Chatterjee, Ting Dai, Prateeti Mohapatra, Brent Paulovicks, Brad Blancett, and Arthur De Magalhaes. 2024. https://doi.org/10.1109/CLOUD62652.2024.00052 CodeSift: An LLM-Based Reference-Less Framework for Automatic Code Validation . In 2024 IEEE 17th International Conference on Cloud Computing (CLOUD), pages 404--410, Los Alamitos, C...

work page arXiv 2024
[5]

Duarte Alves, Nuno Guerreiro, Jo \ a o Alves, Jos \'e Pombal, Ricardo Rei, Jos \'e de Souza, Pierre Colombo, and Andre Martins. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.744 Steering large language models for machine translation with finetuning and in-context learning . In Findings of the Association for Computational Linguistics: EMNLP 2023, ...

doi:10.18653/v1/2023.findings-emnlp.744 2023
[6]

Shikhar Bharadwaj, Shirish Shevade, and Marine Carpuat. 2022. https://doi.org/10.18653/v1/2022.naacl-main.230 Efficient constituency tree based encoding for natural language to bash translation . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3159--31...

work page doi:10.18653/v1/2022.naacl-main.230 2022
[7]

Richard Blum and Christine Bresnahan. 2021. Linux Command Line and Shell Scripting Bible. Wiley

work page 2021
[8]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

2020

Show all 55 references

[9]

Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2023. https://doi.org/10.1109/TSE.2023.3267446 MultiPL-E: A Sc...

2023
[10]

Oishik Chatterjee, Pooja Aggarwal, Suranjana Samanta, Ting Dai, Prateeti Mohapatra, Debanjana Kar, Ruchi Mahindru, Steve Barbieri, Eugen Postea, Brad Blancett, and Arthur De Magalhaes. 2024. https://arxiv.org/abs/2409.17166 Scriptsmith: A unified llm framework for enhancing it...

2024 arXiv
[11]

Liguo Chen, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang, Yijiang Xu, Jian Wu, Yidong Wang, Qing Gao, Jindong Wang, Wei Ye, and Shikun Zhang. 2024. https://arxiv.org/abs/2408.16498 A survey on evaluating large language models in code generation tasks . Preprint, arXiv:2408.16498

2024 arXiv
[12]

Heejae Chon, Seonghyeon Lee, Jinyoung Yeo, and Dongha Lee. 2024. https://arxiv.org/abs/2408.14504 Is functional correctness enough to evaluate code language models? exploring diversity of generated codes . Preprint, arXiv:2408.14504

2024 arXiv
[13]

Berkeley Churchill, Oded Padon, Rahul Sharma, and Alex Aiken. 2019. https://doi.org/10.1145/3314221.3314596 Semantic program alignment for equivalence checking . In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2019, pag...

2019
[14]

Quchen Fu, Zhongwei Teng, Marco Georgaklis, Jules White, and Douglas Schmidt. 2023. https://doi.org/10.13052/jmltapissn.2023.002 Nl2cmd: An updated workflow for natural language to bash commands translation . Journal of Machine Learning Theory, Applications and Practice, 1

2023 doi
[15]

Quchen Fu, Zhongwei Teng, Jules White, and Douglas C. Schmidt. 2021. https://doi.org/10.1109/ICMLA52953.2021.00202 A transformer-based approach for translating natural language to bash commands . In 2021 20th IEEE International Conference on Machine Learning and Applications (...

2021
[16]

Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.674 Grammar-constrained decoding for structured NLP tasks without finetuning . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proces...

2023 doi
[17]

David Gros. 2019. http://www.cs.utexas.edu/users/ai-labpub-view.php?PubID=127814 Ainix: An open platform for natural language interfaces to shell commands . Undergraduate Honors Thesis, Computer Science Department, University of Texas at Austin

2019
[18]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

2021 arXiv
[19]

Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin Clement, and Nan Duan. 2022. https://aclanthology.org/2022.dash-1.5/ Execution-based evaluation for data science code generation models . In Proceedings of the Fourth Workshop on Data...

2022
[20]

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186

2024 arXiv
[21]

Daniel Jenson and Yingxiao Liu. 2022. Translating natural language to bash commands using deep neural networks. https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1224/reports/custom_116997097.pdf. Accessed: 2024-10-09

2022
[22]

Sathvik Joel, Jie JW Wu, and Fatemeh H. Fard. 2024. https://arxiv.org/abs/2410.03981 A survey on llm-based code generation for low-resource and domain-specific programming languages . Preprint, arXiv:2410.03981

2024
[23]

Anish Joshi. 2024. Bash scripting assistant. https://github.com/AnishJoshi13/Bash-Scripting-Assistant. Accessed: 2024-10-09

2024
[24]

Idan Kamara. 2016. bashlex: Python parser for bash. https://github.com/idank/bashlex. Accessed: 2024-09-09

2016
[25]

Michael Kerrisk. 2024. Linux manual pages. https://man7.org/linux/man-pages/index.html. Accessed: 2024-09-09

2024
[26]

K.B.Dharun Krishna, Sebastiaan Speck, Owen Voke, and Darío Herenu. 2024. Tldr: Collaborative cheatsheets for console commands. https://github.com/tldr-pages/tldr. Accessed: 2024-10-09

2024
[27]

Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lipp. 2024. https://www.mixedbread.ai/blog/mxbai-embed-large-v1 Open source strikes bread - new fluffy embeddings model

2024
[28]

Xi Victoria Lin. 2017. https://api.semanticscholar.org/CorpusID:3809743 Program synthesis from natural language using recurrent neural networks

2017
[29]

Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. 2018. https://aclanthology.org/L18-1491/ NL 2 B ash: A corpus and semantic parser for natural language interface to the linux operating system . In Proceedings of the Eleventh International Conference on ...

2018
[30]

Meta Llama. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

2024 arXiv
[31]

Zach Lloyd, Michelle Lim, and Aloke Desai. 2024. Warp: Your terminal, reimagined. https://www.warp.dev/. Accessed: 2024-09-09

2024
[32]

Yogesh Mali. 2023. https://huggingface.co/datasets/yogeshm/text_to_bash text\_to\_bash

2023
[33]

Nickil Maveli, Antonio Vergari, and Shay B. Cohen. 2024. https://arxiv.org/abs/2408.11081 What can large language models capture about code functional equivalence? Preprint, arXiv:2408.11081

2024 arXiv
[34]

Dirk Merkel. 2014. Docker: lightweight linux containers for consistent development and deployment. Linux journal, 2014(239):2

2014
[35]

Microsoft. 2024. Microsoft ai shell. https://learn.microsoft.com/en-us/powershell/utility-modules/aishell/overview?view=ps-modules

2024
[36]

Atharva Naik. 2024. https://arxiv.org/abs/2405.01580 On the limitations of embedding based methods for measuring functional correctness for code generation . Preprint, arXiv:2405.01580

2024 arXiv
[37]

OpenAI. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

2024 arXiv
[38]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pages 31...

2002
[39]

T. Ramesh. 2022. https://huggingface.co/datasets/TRamesh2/NL2CMD Nl2cmd . Accessed: 2024-08-20

2022
[40]

Chet Ramey and Brian Fox. 2024. Bash: GNU Project's Shell. Accessed: 2024-09-09

2024
[41]

Romit. 2024. https://huggingface.co/datasets/Romit2004/LinuxCommands Linuxcommands . Accessed: 2024-08-20

2024
[42]

Johan Rosenkilde, Matt Rothenberg, and Andy Feller. 2024. Github copilot for cli. https://githubnext.com/projects/copilot-cli/

2024
[43]

Farkhod Sadykov. 2024. Shellgpt: A command-line productivity tool powered by ai large language models like gpt-4. https://github.com/TheR1D/shell_gpt. Accessed: 2024-09-09

2024
[44]

Jean E. Sammet. 1966. https://doi.org/10.1145/365230.365274 The use of english as a programming language . Commun. ACM, 9(3):228--230

1966
[45]

Amazon Web Services. 2024. Amazon codewhisperer natural language to bash translation. https://docs.aws.amazon.com/codewhisperer/latest/userguide/command-line-conversation.html

2024
[46]

Danil Shaikhelislamov, Mikhail Drobyshevskiy, and Andrey Belevantsev. 2024. https://genai-evaluation-kdd2024.github.io/genai-evalution-kdd2024/assets/papers/GenAI_Evaluation_KDD2024_paper_25.pdf Codepatchllm: Configuring code generation using a static analyzer

2024
[47]

Jie Shi, Sihang Jiang, Bo Xu, Jiaqing Liang, Yanghua Xiao, and Wei Wang. 2023. https://doi.org/10.1109/ISSRE59848.2023.00082 Shellgpt: Generative pre-trained transformer model for shell language understanding . In 2023 IEEE 34th International Symposium on Software Reliability ...

2023
[48]

William E Shotts. 2019. The Linux Command Line: A Complete Introduction. No Starch Press

2019
[49]

Yewei Song, Cedric Lothritz, Xunzhu Tang, Tegawend \'e Bissyand \'e , and Jacques Klein. 2024. https://doi.org/10.18653/v1/2024.acl-short.3 Revisiting code similarity evaluation with abstract syntax tree edit distance . In Proceedings of the 62nd Annual Meeting of the Associat...

2024 doi
[50]

Karen Sparck Jones. 1988. A statistical interpretation of term specificity and its application in retrieval, pages 132--142. Taylor Graham Publishing, GBR

1988
[51]

Prashanth Vijayaraghavan, Luyao Shi, Stefano Ambrogio, Charles Mackin, Apoorva Nitsure, David Beymer, and Ehsan Degan. 2024. https://arxiv.org/abs/2406.04379 Vhdl-eval: A framework for evaluating large language models in vhdl code generation . Preprint, arXiv:2406.04379

2024 arXiv
[52]

Ngoc Phuoc An Vo, Brent Paulovicks, and Vadim Sheinin. 2024. https://arxiv.org/abs/2405.06807 Execution-based evaluation of natural language to bash and powershell for incident remediation . Preprint, arXiv:2405.06807

2024 arXiv
[53]

John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. Intercode: standardizing and benchmarking interactive coding with execution feedback. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, U...

2023
[54]

Ezri Zhu, Georgios Liargkovas, Michael Greenberg, and Konstantinos Kallas. 2024 a . try utility. https://github.com/binpash/try. Accessed: 2024-09-25

2024
[55]

Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024 b . https://doi.org/10.18653/v1/2024.findings-naacl.176 Multilingual machine translation with large language models: Empirical results and analysis . In Findings of t...

2024 doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Mayank Agarwal, Tathagata Chakraborti, Quchen Fu, David Gros, Xi Victoria Lin, Jaron Maene, Kartik Talamadupula, Zhongwei Teng, and Jules White. 2021. https://proceedings.mlr.press/v133/agarwal21b.html Neurips 2020 nlc2cmd competition: Translating natural language to bash commands . In Proceedings of the NeurIPS 2020 Competition and Demonstration Track, v...

work page 2021

[4] [4]

Pooja Aggarwal, Oishik Chatterjee, Ting Dai, Prateeti Mohapatra, Brent Paulovicks, Brad Blancett, and Arthur De Magalhaes. 2024. https://doi.org/10.1109/CLOUD62652.2024.00052 CodeSift: An LLM-Based Reference-Less Framework for Automatic Code Validation . In 2024 IEEE 17th International Conference on Cloud Computing (CLOUD), pages 404--410, Los Alamitos, C...

work page arXiv 2024

[5] [5]

Duarte Alves, Nuno Guerreiro, Jo \ a o Alves, Jos \'e Pombal, Ricardo Rei, Jos \'e de Souza, Pierre Colombo, and Andre Martins. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.744 Steering large language models for machine translation with finetuning and in-context learning . In Findings of the Association for Computational Linguistics: EMNLP 2023, ...

doi:10.18653/v1/2023.findings-emnlp.744 2023

[6] [6]

Shikhar Bharadwaj, Shirish Shevade, and Marine Carpuat. 2022. https://doi.org/10.18653/v1/2022.naacl-main.230 Efficient constituency tree based encoding for natural language to bash translation . In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3159--31...

work page doi:10.18653/v1/2022.naacl-main.230 2022

[7] [7]

Richard Blum and Christine Bresnahan. 2021. Linux Command Line and Shell Scripting Bible. Wiley

work page 2021

[8] [8]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

2020