Pith. sign in

REVIEW 4 major objections 5 minor 68 references

Memory-Augmented Agent Training for Business Document Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Iteratively distilling an LLM agent's own successes and failures into a long-term memory lifts transport-reference extraction on held-out UBL invoices from 14.2% for chain-of-thought prompting to 55.8%.

desk verdict A credible engineering extension of memory-based agent learning, but the headline gains rest on a single split of a private dataset and an unvalidated LLM judge, so the strong claims should be treated as conditional. read the letter →

arxiv 2412.15274 v1 pith:QZMZQ3TP submitted 2024-12-17 cs.CL cs.AI

Jiale Liu , Yifan Zeng , Malte H{o}jmark-Bertelsen , Marie Normann Gadeberg , Huazheng Wang , Qingyun Wu This is my paper · ORCID
classification cs.CLcs.AI
keywords Matrixmemory-augmentedagentLLMtrainingbusinessdocumentunderstandingtransportreferenceextractionUBLinvoicesiterativeself-refinementexperience-drivenmemory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Matrix, a training paradigm that turns a general-purpose LLM agent into a specialized business-document processor without fine-tuning. The agent solves extraction tasks on a batch of training invoices, a reflector labels each trajectory as correct or incorrect, and a meta-optimizer rewrites a long-term memory prompt that encodes the reusable patterns. On a private dataset of 704 real UBL invoices from a logistics company, the memory-augmented agent reaches 55.82% exact-match success with a gpt-4o-mini backbone, beating chain-of-thought prompting (14.20%), a vanilla two-agent system (8.39%), and a verbal-reinforcement baseline (18.03%), while using fewer API calls and lower cost after optimization. The authors claim this shows domain expertise for agents can be built from experience-driven memory refinement rather than from hand-coded rules or model updates.

What carries the argument

The load-bearing object is the long-term memory module $M$, a natural-language instruction that the optimizer rewrites after each epoch. It is what converts scattered trajectories into generalizable heuristics: the memory is passed to the optimizer along with the evaluated trajectories, so each new version encodes patterns that worked and warnings about patterns that failed, and the same memory then guides all future task-solving attempts.

What would settle it

Run the same optimization loop with a training set of 60 invoices drawn from a different invoice layout or a different logistics provider than the test set; if the memory does not improve success beyond the no-memory baseline, the reported gains come from pattern memorization rather than transferable domain expertise. The paper's own appendix is an early pointer: with eight training documents Matrix stops beating the verbal-reinforcement baseline, so a systematic sweep over training-set size would settle whether the gains are robust.

Watch

Extended reading notes

Core claim

The central claim is that iterative batch-level memory refinement is what makes an LLM agent effective on structured business documents. Matrix samples a mini-batch of training tasks, records the agent's trajectories, has a reflector compare each outcome to the ground truth, and feeds trajectories plus current memory into an optimizer LLM that produces an updated memory; the loop repeats for several epochs and the final memory is used to steer inference on held-out documents. The paper reports that this raises success from a no-memory baseline to 55.82% (gpt-4o-mini) and 48.15% (gpt-4o), that the weaker backbone ends up outperforming the stronger one after optimization, and that the optimized system cuts average API calls by up to 21.3% while shifting the distribution of successfully analyzed documents toward longer inputs.

Load-bearing premise

The whole pipeline assumes that the 60 training invoices given to the optimizer represent the range of invoice structures and transport-reference formats that appear in the 704 held-out invoices, since the memory is distilled from those training trajectories alone.

Editorial extensions

If this is right

  • If Matrix works as reported, an LLM agent can be specialized to a document task purely through prompt-level memory updates, without any weight fine-tuning.
  • The optimized memory reduces average API calls per solved task by 8.12% (gpt-4o) and 21.3% (gpt-4o-mini), so the agent becomes cheaper to run as well as more accurate.
  • The distribution of successfully solved document lengths shifts toward longer documents after optimization, suggesting memory-guided agents can handle inputs beyond what the raw model manages.
  • A weaker backbone (gpt-4o-mini) can outperform a stronger one (gpt-4o) once both are equipped with the same optimized memory, indicating that the memory carries most of the task expertise.
  • The method's gains require enough representative training data; with only eight training invoices the advantage over baselines largely disappears, as the paper's own anonymized-data experiments show.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 30–35% margins are measured against baselines that do not use any learned memory; a more informative comparison would pit Matrix against retrieval-based memory or example-selection methods under a fixed total token budget.
  • Because the memory is plain natural language, the same optimization loop could be re-run on a small batch of new document types to adapt the agent to format drift, which would test whether the memory transfers across invoice structures rather than only within one.
  • The paper mentions coreset selection as future work; a direct test is whether a carefully chosen subset of 60 invoices reproduces the full-data gains, which would confirm the memory is capturing distributional patterns rather than memorizing the training batch.
  • The surprising finding that gpt-4o-mini with memory beats gpt-4o with memory suggests the optimizer, itself a gpt-4o, may over-fit its own style of reasoning; future work could vary the optimizer model to see how much the memory's quality depends on the optimizer's strength.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Matrix (Memory-Augmented agent Training through Reasoning and Iterative eXploration), a framework that iteratively refines a natural-language long-term memory for an LLM-based two-agent document-processing system. On a private UBL invoice dataset of 764 documents, with transport-reference extraction as the task, the authors report success rates of 55.82% (gpt-4o-mini backbone) and 48.15% (gpt-4o backbone) on 704 held-out documents after training on 60 documents. They report average absolute gains of roughly 30 percentage points over chain-of-thought prompting, 35 points over a vanilla two-agent system, and 27 points over Reflexion, along with reduced API calls and cost. The paper also releases a small anonymized subset of the dataset and presents results on it in an appendix.

Significance. If the results hold, Matrix is a practically useful and cost-efficient way to specialize LLM agents to business-document extraction, and the released anonymized benchmark is a valuable resource for the community. The paper's strengths include a real-world industrial dataset, a held-out evaluation protocol, an explicit comparison against several baselines, and an honest appendix that shows the method's sensitivity to training-set size. However, the central quantitative claims rest on a single split of a private dataset and on an LLM judge whose accuracy is not assessed, so the magnitude of the reported gains is not yet established.

major comments (4)
  1. [Section 4.2, Figure 2] The headline comparison rests on one random 60/704 split of a private dataset, with no repeated splits, no seeds, and no confidence intervals. Because the learned memory is distilled from 60 trajectories, the reported margins over Reflexion (16.76 percentage points for gpt-4o and 37.79 for gpt-4o-mini) could be split-specific; the paper provides no evidence that 60 invoices are representative of the format variation in the 704 test documents. Please report results over multiple training splits or bootstrap resamples of the test set, and characterize the training-set composition.
  2. [Appendix B, Figure 8, and Section D] The released anonymized experiment is directly relevant to the generalizability claim. With 8 training documents, Matrix does not consistently outperform Reflexion (38.09 vs. 42.85 for gpt-4o), and the text concedes that the optimizer "struggles to capture a comprehensive pattern" and that "Matrix requires a substantial amount of training data." This limitation should be reflected in the abstract and conclusion; as written, the claim that Matrix outperforms all baselines by a large margin is not qualified by the training-data requirement.
  3. [Section 4.1, Evaluation Protocol] All success rates are determined by an LLM judge, but the judge's accuracy is never assessed. If the judge is lenient toward plausible-looking but incorrect references, the absolute success rates and the margins over baselines could be inflated. Please validate the judge against a human-annotated sample or a deterministic normalization procedure, and state explicitly how the judge's decision is combined with the exact-match criterion introduced in Eq. (3).
  4. [Figures 3-5 and Section 4.3] The optimization curves and the cost/API-call analyses are single-run trajectories with no variance estimates. Because both the agent and the meta-optimizer are stochastic LLMs, repeated runs are needed to establish that the reported decreases in API calls and cost (8.12% for gpt-4o and 21.3% for gpt-4o-mini) are systematic rather than run-specific. At minimum, report the number of runs and standard deviations.
minor comments (5)
  1. [Abstract and Section 1] The figures 30.3%, 35.2%, and 27.28% are consistent with averaging absolute percentage-point differences across the two backbones (for example, (41.62 + 18.89)/2 = 30.26 for the comparison with chain-of-thought), not with relative improvements. Please state this definition explicitly; otherwise the numbers appear inconsistent with Figure 2, where the relative gain for gpt-4o-mini over chain-of-thought is about 293%.
  2. [Section 4.2] The sentence "The performance nearly doubles and outperforms all other methods" is accurate for gpt-4o-mini (18.03 to 55.82) but not for gpt-4o (31.39 to 48.15); please qualify the statement.
  3. [Section 4.4 and Appendix A.4] There are typos in the text, including "reuable" (Section 4.4), "specifcally" (Appendix A.4), and "we enforce a upper limit" (Section 3.2); these should be corrected.
  4. [Figure 5] The two y-axis labels ("1e-2 USD" and "1e-3 USD") are not tied to the two backbones in the caption; please clarify which curve uses which axis.
  5. [Appendix C and Section 4.2] The optimizer and reflector prompts, sampling temperatures, and the exact batch-truncation rule are not provided; adding these details would materially improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Matrix is evaluated with a standard train/test split and its reported gains are held-out measurements, not fitted predictions.

full rationale

The paper's claimed derivation chain is empirical rather than formal, and the central evaluation does not reduce to its inputs. Equation (3) defines the objective as minimizing a loss over the data distribution, Equation (5) updates memory from training trajectories and ground-truth labels, and Section 4.2 explicitly states that 60 samples are randomly selected for training while the remaining 704 are reserved for testing. The success rates reported in Figure 2 are therefore measured on a held-out split, not re-statements of the training objective. The LLM judge compares the agent output against the ground-truth label, so the evaluation target is not defined by the method itself. The self-citations, including AG2/AutoGen (Wu et al. 2023), are implementation and related-work references; they are not load-bearing justifications for the Matrix result, and the cited codebase is public and reproducible. The paper's own Appendix B shows weaker performance with only 8 training documents and states that the optimizer 'struggles to capture a comprehensive pattern'; this is an acknowledged data-sufficiency and generalization limitation, not evidence of circularity, because that test split is also held out and the optimizer does not see test labels. No equation or fitted parameter is equivalent by construction to the headline improvement percentages. The empirical claims may have robustness concerns, but no circular step is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 3 invented entities

The central claim depends on standard training hyperparameters, on the representativeness of a 60-document split, and on several unvalidated domain assumptions about the reliability of LLM-generated reflections and the LLM judge. The memory, reflector, and meta-optimizer are new system components with no independent falsifiable evidence outside this paper.

free parameters (5)
  • training set size = 60 of 764 documents
    A single random 60/704 split; the headline generalization claim depends on this subset representing the test distribution.
  • number of optimization epochs = 7
    Final reported performance is after 7 epochs; stopping point chosen without evidence of convergence.
  • batch size = 14 tasks per epoch
    Mini-batch size for memory optimization; truncated when trajectories exceed context length.
  • maximum conversation turns = 5
    Caps trajectory length and influences success rate and API cost.
  • Reflexion max trials = 7
    Sets the baseline budget for Reflexion comparisons.
assumptions (4)
  • domain assumption The 60-document training split is representative of the 704-document test distribution.
    Section 4.2 uses a single 60/704 split; Appendix B shows performance collapses when training set is 8 documents, so the claim is sensitive to this premise.
  • domain assumption LLM judge accurately determines whether an extracted reference matches the ground truth.
    Section 4.1 replaces exact-match comparison with an LLM judge and provides no reliability analysis.
  • domain assumption UBL invoice structure and transport-reference patterns are consistent enough for learned memory to transfer across documents.
    Section 3.1 formulates the task as learning f: D x Q -> A; Section 4.1 argues UBL documents have 'significant variability', which this assumption must overcome.
  • domain assumption LLM reflections on trajectories plus ground truth yield correct error attributions for memory optimization.
    Section 3.2 Reflection relies on LMreflect to label correct/incorrect steps without validation.
invented entities (3)
  • Reflector module
    purpose: Labels trajectories as correct/incorrect and identifies key reasoning steps for memory updates.
    Evidence is only the paper's own experiments; no separate falsifiable handle.
  • Meta-optimizer LMoptim
    purpose: Updates the long-term memory M from trajectories, evaluations, and current memory.
    Introduced and evaluated only inside Matrix; no external benchmark.
  • Long-term memory module M
    purpose: Stores distilled actionable insights that guide the agent's future extraction attempts.
    The contents are not released, so the memory's effect can only be inferred from reported aggregate results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Memory-Augmented Agent Training for Business Document Understanding." pith.science (2026). https://pith.science/paper/QZMZQ3TP

@misc{pith2026241215274,
  author       = {Pith},
  title        = {Pith review of: Memory-Augmented Agent Training for Business Document Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QZMZQ3TP}},
  note         = {Machine review of arXiv:2412.15274}
}
read the original abstract

Traditional enterprises face significant challenges in processing business documents, where tasks like extracting transport references from invoices remain largely manual despite their crucial role in logistics operations. While Large Language Models offer potential automation, their direct application to specialized business domains often yields unsatisfactory results. We introduce Matrix (Memory-Augmented agent Training through Reasoning and Iterative eXploration), a novel paradigm that enables LLM agents to progressively build domain expertise through experience-driven memory refinement and iterative learning. To validate this approach, we collaborate with one of the world's largest logistics companies to create a dataset of Universal Business Language format invoice documents, focusing on the task of transport reference extraction. Experiments demonstrate that Matrix outperforms prompting a single LLM by 30.3%, vanilla LLM agent by 35.2%. We further analyze the metrics of the optimized systems and observe that the agent system requires less API calls, fewer costs and can analyze longer documents on average. Our methods establish a new approach to transform general-purpose LLMs into specialized business tools through systematic memory enhancement in document processing tasks.

Figures

Figures reproduced from arXiv: 2412.15274 by the authors.

Figure 1
Figure 1. The training and inference pipeline of Matrix. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison between Matrix and baselines with [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Success rate comparison between agent with [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 6
Figure 6. Figure 6: Distribution of successfully analyzed document [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 4
Figure 4. Figure 4: Comparison of average number of API calls it [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Average cost of API calls after each epoch. The [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 7
Figure 7. Figure 7: Performance of the optimized agent across itera [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Performance comparison of baselines and Matrix on anonymized dataset. Reflexion performs better than Matrix on [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 20 canonical work pages

  1. [1]

    Abujabal, A.; Saha Roy, R.; Yahya, M.; and Weikum, G. 2019. C om QA : A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters. In Burstein, J.; Doran, C.; and Solorio, T., eds., Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies...

    work page 2019

  2. [2]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  3. [3]

    Antonacopoulos, A.; Bridson, D.; Papadopoulos, C.; and Pletschacher, S. 2009. A realistic dataset for performance evaluation of document layout analysis. In 2009 10th International Conference on Document Analysis and Recognition, 296--300. IEEE

    work page 2009

  4. [4]

    Bajaj, P.; Campos, D.; Craswell, N.; Deng, L.; Gao, J.; Liu, X.; Majumder, R.; McNamara, A.; Mitra, B.; Nguyen, T.; et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268

    arXiv 2016

  5. [5]

    Brown, T. B. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  6. [6]

    Cai, T.; Wang, X.; Ma, T.; Chen, X.; and Zhou, D. 2023. Large language models as tool makers. arXiv preprint arXiv:2305.17126

    arXiv 2023

  7. [7]

    Cheng, J.; Liu, X.; Zheng, K.; Ke, P.; Wang, H.; Dong, Y.; Tang, J.; and Huang, M. 2023. Black-box prompt optimization: Aligning large language models without model training. arXiv preprint arXiv:2311.04155

    arXiv 2023

  8. [8]

    Dong, Q.; Li, L.; Dai, D.; Zheng, C.; Wu, Z.; Chang, B.; Sun, X.; Xu, J.; and Sui, Z. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234

    arXiv 2022

Show all 68 references
  1. [9]

    Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    L.; and Wang, M

    Guo, X.; Huang, K.; Liu, J.; Fan, W.; V \'e lez, N.; Wu, Q.; Wang, H.; Griffiths, T. L.; and Wang, M. 2024. Embodied llm agents learn to cooperate in organized teams. arXiv preprint arXiv:2403.12482

    2024 arXiv

  3. [11]

    Hamdi, A.; Carel, E.; Joseph, A.; Coustaty, M.; and Doucet, A. 2021. Information extraction from invoices. In International Conference on Document Analysis and Recognition, 699--714. Springer

    2021

  4. [12]

    W.; Ufkes, A.; and Derpanis, K

    Harley, A. W.; Ufkes, A.; and Derpanis, K. G. 2015. Evaluation of deep convolutional nets for document image classification and retrieval. In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), 991--995. IEEE

    2015

  5. [13]

    Hu, S.; Lu, C.; and Clune, J. 2024. Automated design of agentic systems. arXiv preprint arXiv:2408.08435

    2024 arXiv

  6. [14]

    S.; Yu, A

    Huang, J.; Chen, X.; Mishra, S.; Zheng, H. S.; Yu, A. W.; Song, X.; and Zhou, D. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798

    2023 arXiv

  7. [15]

    Ji, Z.; Yu, T.; Xu, Y.; Lee, N.; Ishii, E.; and Fung, P. 2023. Towards mitigating LLM hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, 1827--1843

    2023

  8. [16]

    Jiang, D.; Zhang, J.; Weller, O.; Weir, N.; Van Durme, B.; and Khashabi, D. 2024. Self-[in] correct: Llms struggle with refining self-generated responses. arXiv preprint arXiv:2404.04298

    2024 arXiv

  9. [17]

    Joshi, M.; Choi, E.; Weld, D.; and Zettlemoyer, L. 2017. T rivia QA : A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Barzilay, R.; and Kan, M.-Y., eds., Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ...

    2017

  10. [18]

    Kamoi, R.; Zhang, Y.; Zhang, N.; Han, J.; and Zhang, R. 2024. When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs. arXiv preprint arXiv:2406.01297

    2024 arXiv

  11. [19]

    Krieger, F.; Drews, P.; Funk, B.; and Wobbe, T. 2021. Information extraction from invoices: a graph neural network approach for datasets with high layout variety. In Innovation Through Information Systems: Volume II: A Collection of Latest Research on Technology Issues, 5--20....

    2021

  12. [20]

    Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K.; et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7: 453--466

    2019

  13. [21]

    Liu, J.; Shen, D.; Zhang, Y.; Dolan, B.; Carin, L.; and Chen, W. 2021. What Makes Good In-Context Examples for GPT- 3 ? arXiv preprint arXiv:2101.06804

    2021 arXiv

  14. [22]

    Lu, Y.; Bartolo, M.; Moore, A.; Riedel, S.; and Stenetorp, P. 2021. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786

    2021 arXiv

  15. [23]

    Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36

    2024

  16. [24]

    Min, S.; Lewis, M.; Zettlemoyer, L.; and Hajishirzi, H. 2021. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943

    2021 arXiv

  17. [25]

    B.; Winther, O.; and Laws, F

    Palm, R. B.; Winther, O.; and Laws, F. 2017. Cloudscan-a configuration-free invoice analysis system using recurrent neural networks. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 1, 406--413. IEEE

    2017

  18. [26]

    Pan, L.; Saxon, M.; Xu, W.; Nathani, D.; Wang, X.; and Wang, W. Y. 2023. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188

    2023 arXiv

  19. [27]

    Prasad, A.; Hase, P.; Zhou, X.; and Bansal, M. 2022. Grips: Gradient-free, edit-based instruction search for prompting large language models. arXiv preprint arXiv:2203.07281

    2022 arXiv

  20. [28]

    gradient descent

    Pryzant, R.; Iter, D.; Li, J.; Lee, Y. T.; Zhu, C.; and Zeng, M. 2023. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495

    2023 arXiv

  21. [29]

    R.; Qin, Y.; Liu, Z.; and Ji, H

    Qian, C.; Han, C.; Fung, Y. R.; Qin, Y.; Liu, Z.; and Ji, H. 2023. Creator: Tool creation for disentangling abstract and concrete reasoning of large language models. arXiv preprint arXiv:2305.14318

    2023 arXiv

  22. [30]

    Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016. SQ u AD : 100,000+ Questions for Machine Comprehension of Text. In Su, J.; Duh, K.; and Carreras, X., eds., Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2383--2392. Austin, Te...

    2016

  23. [31]

    Riba, P.; Dutta, A.; Goldmann, L.; Forn \'e s, A.; Ramos, O.; and Llad \'o s, J. 2019. Table detection in invoice documents by graph neural networks. In 2019 International Conference on Document Analysis and Recognition (ICDAR), 122--127. IEEE

    2019

  24. [32]

    Shinn, N.; Cassano, F.; Gopinath, A.; Narasimhan, K.; and Yao, S. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36

    2024

  25. [33]

    Song, L.; Liu, J.; Zhang, J.; Zhang, S.; Luo, A.; Wang, S.; Wu, Q.; and Wang, C. 2024. Adaptive In-conversation Team Building for Language Model Agents. arXiv preprint arXiv:2405.19425

    2024 arXiv

  26. [34]

    H.; Shi, W.; Wang, T.; Xin, J.; Zhang, R.; Ostendorf, M.; Zettlemoyer, L.; Smith, N

    Su, H.; Kasai, J.; Wu, C. H.; Shi, W.; Wang, T.; Xin, J.; Zhang, R.; Ostendorf, M.; Zettlemoyer, L.; Smith, N. A.; et al. 2022. Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975

    2022 arXiv

  27. [35]

    R.; Yao, S.; Narasimhan, K.; and Griffiths, T

    Sumers, T. R.; Yao, S.; Narasimhan, K.; and Griffiths, T. L. 2023. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427

    2023 arXiv

  28. [36]

    Talmor, A.; and Berant, J. 2018. The Web as a Knowledge-base for Answering Complex Questions. In North American Association for Computational Linguistics (NAACL)

    2018

  29. [37]

    S.; Hassanat, A

    Tarawneh, A. S.; Hassanat, A. B.; Chetverikov, D.; Lendak, I.; and Verma, C. 2019. Invoice classification using deep features and machine learning techniques. In 2019 IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT), 855--...

    2019

  30. [38]

    Wang, C.; Liu, X.; Yue, Y.; Tang, X.; Zhang, T.; Jiayang, C.; Yao, Y.; Gao, W.; Hu, X.; Qi, Z.; et al. 2023. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity. arXiv preprint arXiv:2310.07521

    2023 arXiv

  31. [39]

    Wang, C.; Wu, Q.; Weimer, M.; and Zhu, E. 2021. Flaml: A fast and lightweight automl library. Proceedings of Machine Learning and Systems, 3: 434--447

    2021

  32. [40]

    Z.; Mao, J.; Fried, D.; and Neubig, G

    Wang, Z. Z.; Mao, J.; Fried, D.; and Neubig, G. 2024. Agent workflow memory. arXiv preprint arXiv:2409.07429

    2024 arXiv

  33. [41]

    V.; Zhou, D.; et al

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824--24837

    2022

  34. [42]

    Wu, Q.; Bansal, G.; Zhang, J.; Wu, Y.; Zhang, S.; Zhu, E.; Li, B.; Jiang, L.; Zhang, X.; and Wang, C. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155

    2023 arXiv

  35. [43]

    T.; Peng, R.; Wu, Q.; and Wang, C

    Wu, Y.; Jia, F.; Zhang, S.; Li, H.; Zhu, E.; Wang, Y.; Lee, Y. T.; Peng, R.; Wu, Q.; and Wang, C. 2024 a . MathChat: Converse to Tackle Challenging Math Problems with LLM Agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents

    2024

  36. [44]

    Wu, Y.; Yue, T.; Zhang, S.; Wang, C.; and Wu, Q. 2024 b . StateFlow: Enhancing LLM Task-Solving through State-Driven Workflows. arXiv preprint arXiv:2403.11322

    2024 arXiv

  37. [45]

    Wu, Z.; Wang, Y.; Ye, J.; and Kong, L. 2022. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering. arXiv preprint arXiv:2212.10375

    2022 arXiv

  38. [46]

    Xia, X.; Liu, J.; Yu, J.; Shen, X.; Han, B.; and Liu, T. 2022. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In The Eleventh International Conference on Learning Representations

    2022

  39. [47]

    Xia, X.; Liu, J.; Zhang, S.; Wu, Q.; Wei, H.; and Liu, T. 2024. Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraints. In Forty-first International Conference on Machine Learning

    2024

  40. [48]

    Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  41. [49]

    V.; Zhou, D.; and Chen, X

    Yang, C.; Wang, X.; Lu, Y.; Liu, H.; Le, Q. V.; Zhou, D.; and Chen, X. 2023. Large language models as optimizers. arXiv preprint arXiv:2309.03409

    2023 arXiv

  42. [50]

    W.; Salakhutdinov, R.; and Manning, C

    Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W. W.; Salakhutdinov, R.; and Manning, C. D. 2018. HotpotQA : A Dataset for Diverse, Explainable Multi-hop Question Answering. In Conference on Empirical Methods in Natural Language Processing ( EMNLP )

    2018

  43. [51]

    Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

    2024

  44. [52]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. ReAct : Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR)

    2023

  45. [53]

    R.; Peng, H.; and Ji, H

    Yuan, L.; Chen, Y.; Wang, X.; Fung, Y. R.; Peng, H.; and Ji, H. 2023. Craft: Customizing llms by creating and retrieving from specialized toolsets. arXiv preprint arXiv:2309.17428

    2023 arXiv

  46. [54]

    Zhang, J.; Xiang, J.; Yu, Z.; Teng, F.; Chen, X.; Chen, J.; Zhuge, M.; Cheng, X.; Hong, S.; Wang, J.; et al. 2024 a . AFlow: Automating Agentic Workflow Generation. arXiv preprint arXiv:2410.10762

    2024 arXiv

  47. [55]

    Zhang, S.; Jia, F.; Wang, C.; and Wu, Q. 2023 a . Targeted hyperparameter optimization with lexicographic preferences over multiple objectives. In The Eleventh international conference on learning representations

    2023

  48. [56]

    Zhang, S.; Wu, Y.; Zheng, Z.; Wu, Q.; and Wang, C. 2024 b . Hypertime: Hyperparameter optimization for combating temporal distribution shifts. In Proceedings of the 32nd ACM International Conference on Multimedia, 4610--4619

    2024

  49. [57]

    Zhang, S.; Xia, X.; Wang, Z.; Chen, L.-H.; Liu, J.; Wu, Q.; and Liu, T. 2023 b . Ideal: Influence-driven selective annotations empower in-context learners in large language models. arXiv preprint arXiv:2310.10873

    2023 arXiv

  50. [58]

    Zhang, S.; Zhang, J.; Liu, J.; Song, L.; Wang, C.; Krishna, R.; and Wu, Q. 2024 c . Offline Training of Language Model Agents with Functions as Learnable Weights. In Forty-first International Conference on Machine Learning

    2024

  51. [59]

    Zhao, A.; Huang, D.; Xu, Q.; Lin, M.; Liu, Y.-J.; and Huang, G. 2024. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 19632--19642

    2024

  52. [60]

    Zhao, Z.; Wallace, E.; Feng, S.; Klein, D.; and Singh, S. 2021. Calibrate Before Use: Improving Few-shot Performance of Language Models. In Meila, M.; and Zhang, T., eds., Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machin...

    2021

  53. [61]

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023 a . Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36: 46595--46623

    2023

  54. [62]

    Zheng, X.; Yang, C.; Zhang, S.; Wang, Y.; Zhang, B.; Wu, Y.; Wu, Y.; Shao, L.; and Ji, R. 2023 b . Ddpnas: Efficient neural architecture search via dynamic distribution pruning. International Journal of Computer Vision, 131(5): 1234--1249

    2023

  55. [63]

    Zhong, X.; Tang, J.; and Yepes, A. J. 2019. Publaynet: largest dataset ever for document layout analysis. In 2019 International conference on document analysis and recognition (ICDAR), 1015--1022. IEEE

    2019

  56. [64]

    Zhou, D.; Sch \"a rli, N.; Hou, L.; Wei, J.; Scales, N.; Wang, X.; Schuurmans, D.; Cui, C.; Bousquet, O.; Le, Q.; et al. 2022 a . Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625

    2022 arXiv

  57. [65]

    I.; Han, Z.; Paster, K.; Pitis, S.; Chan, H.; and Ba, J

    Zhou, Y.; Muresanu, A. I.; Han, Z.; Paster, K.; Pitis, S.; Chan, H.; and Ba, J. 2022 b . Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910

    2022 arXiv

  58. [66]

    Zhuge, M.; Wang, W.; Kirsch, L.; Faccio, F.; Khizbullin, D.; and Schmidhuber, J. 2024. Language agents as optimizable graphs. arXiv preprint arXiv:2402.16823

    2024 arXiv

  59. [67]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  60. [68]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.