pith. sign in

arxiv: 2606.29437 · v1 · pith:JNBBI3CFnew · submitted 2026-06-28 · 💻 cs.HC · cs.AI· cs.CY

LLMography: Transforming Human-AI Conversations into Traceability, Oversight, and Auditability Indicators

Mohammed Bousmah This is my paper

Pith reviewed 2026-06-30 02:12 UTC · model grok-4.3

classification 💻 cs.HC cs.AIcs.CY
keywords LLMhuman-AI interactiontraceabilityauditabilityprovenanceconversation analysisAI ethicsKPI
0
0 comments X

The pith

LLMography converts human-AI conversation traces into quantitative scores for provenance, human direction, and auditability.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces LLMography as a framework that treats conversation histories between humans and large language models as structured records, similar to bibliographies, to quantify elements like prompt quality, human oversight, and AI reliance. It presents a prototype that processes these traces to produce KPI reports with specific metrics such as Human Direction Score and Auditability Score. An exploratory test on 19 student engineering reports found most interactions classified as human-AI co-production, with average scores in the 70-85 range across several indicators. The paper also applies the method to its own composition process, labeling it as human-originated and human-directed with AI assistance. This approach aims to shift transparency efforts from detecting AI-generated outputs alone to documenting the full interaction trajectory for better reproducibility and oversight.

Core claim

LLMography transforms Human-AI conversations into measurable indicators of provenance, human contribution, AI dependency, reproducibility, and auditability by parsing interaction traces into structured data. A prototype generates KPI reports that include Prompt Quality Score, Human Direction Score, AI Dependency Level, Auditability Score, Final Output Traceability, Privacy Risk Level, and a recommended label. In a preliminary evaluation on 19 anonymized student reports, most were classified as Human-AI co-produced, yielding average scores of 86.8/100 for Human Direction, 81.9/100 for Prompt Quality, 72.8/100 for Auditability, and 77.1/100 for Final Output Traceability. The framework was also

What carries the argument

The LLMography framework that parses raw Human-AI conversation traces to compute and report KPI scores including Prompt Quality Score, Human Direction Score, AI Dependency Level, and Auditability Score.

If this is right

  • Interactions can be classified as human-originated, human-directed, or AI-dominated based on the computed scores.
  • Audit reports in education and engineering can include standardized traceability labels derived from conversation history.
  • Transparency requirements can extend beyond final output detection to include documented interaction trajectories.
  • Privacy risk levels and reproducibility assessments become derivable directly from the trace data.
  • Self-application to the paper's own process demonstrates the framework's internal consistency for mixed human-AI authorship claims.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method could be integrated into collaborative writing platforms to generate real-time oversight dashboards during drafting.
  • Legal or academic integrity reviews might adopt these scores as supplementary evidence when determining contribution levels.
  • Testing the prototype on non-technical domains such as creative writing or policy drafting would reveal whether the KPI definitions generalize.

Load-bearing premise

Raw conversation traces contain sufficient structured information to allow reliable computation of the proposed KPI scores without external ground-truth validation or detailed scoring algorithms.

What would settle it

A side-by-side comparison where independent human raters score the same set of conversation traces for human direction and AI dependency, then measure statistical agreement with the prototype's computed scores.

Figures

Figures reproduced from arXiv: 2606.29437 by Mohammed Bousmah.

Figure 1
Figure 1. Figure 1: LLMography MVP input interface for submitting a Human–AI conversation link or [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Example of a LLMography KPI report generated by the MVP, including conversation [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example of the detailed LLMography audit interpretation, including human–AI con [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Average LLMography KPI scores across the 19 analyzed audit reports. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of LLMography labels across the 19 analyzed audit reports. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
read the original abstract

The growing use of Large Language Models (LLMs) in education, software engineering, academic writing, and technical documentation raises a key question: how can we evaluate not only AI-assisted outputs, but also the interaction process that produced them? Current debates often focus on detecting whether a final artifact was generated by AI, while overlooking the conversation history that reveals human direction, AI contribution, corrections, validation, and traceability. This paper introduces LLMography, a framework for transforming Human-AI conversations into measurable indicators of provenance, human contribution, AI dependency, reproducibility, and auditability. By analogy with bibliography and webography, LLMography documents the dynamic trajectory of interaction between a human and a Large Language Model as a structured trace of Human-AI co-production. We present a prototype that analyzes Human-AI conversation traces and generates KPI reports including Prompt Quality Score, Human Direction Score, AI Dependency Level, Auditability Score, Final Output Traceability, Privacy Risk Level, and a recommended LLMography label. A preliminary exploratory evaluation was conducted on 19 anonymized audit reports from engineering students. Most interactions were classified as Human-AI co-produced, with average scores of 86.8/100 for Human Direction, 81.9/100 for Prompt Quality, 72.8/100 for Auditability, and 77.1/100 for Final Output Traceability. The paper also applies LLMography to its own writing process, classified as human-originated, human-directed, AI-assisted co-production. The findings suggest that AI transparency should move beyond output detection toward documenting the history of interaction.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper introduces LLMography, a framework for converting Human-AI conversation traces into structured indicators of provenance, human contribution, AI dependency, reproducibility, and auditability. It describes a prototype that produces KPI reports with scores including Prompt Quality Score, Human Direction Score, AI Dependency Level, and Auditability Score. A preliminary evaluation on 19 anonymized student audit reports reports averages such as 86.8/100 Human Direction and 81.9/100 Prompt Quality, classifying most interactions as Human-AI co-produced; the framework is also applied to the paper's own writing process, classified as human-originated and AI-assisted co-production.

Significance. If the KPI methods can be specified and externally validated, the work could shift AI transparency efforts from post-hoc output detection toward documented interaction histories, offering a practical tool for oversight in education, software engineering, and technical writing.

major comments (2)
  1. [Abstract] Abstract and prototype description: the central claim that raw conversation traces enable reliable, low-bias computation of KPIs (Prompt Quality Score, Human Direction Score, etc.) is unsupported because no scoring functions, decision rules, thresholds, or weighting schemes are provided; the reported averages from 19 reports therefore cannot be reproduced or assessed for validity.
  2. [Evaluation] Evaluation on 19 reports: no inter-rater reliability, ground-truth labels, baseline comparisons, or sensitivity analysis are reported for the KPI calculations, leaving open whether the scores reflect reproducible measurement or ad-hoc assignment.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive comments. We address each major point below, acknowledging where the current manuscript is limited and outlining specific revisions to improve reproducibility and transparency.

read point-by-point responses

  1. Referee: [Abstract] Abstract and prototype description: the central claim that raw conversation traces enable reliable, low-bias computation of KPIs (Prompt Quality Score, Human Direction Score, etc.) is unsupported because no scoring functions, decision rules, thresholds, or weighting schemes are provided; the reported averages from 19 reports therefore cannot be reproduced or assessed for validity.

    Authors: The referee correctly notes that the manuscript does not provide explicit scoring functions, decision rules, thresholds, or weighting schemes. The prototype is described conceptually in the text, with KPI results reported from the 19 reports but without the underlying computation details. We will revise by adding a new subsection to the prototype description that specifies the exact scoring mechanisms, rules, thresholds, and any weighting used for each KPI. This will directly support reproducibility of the reported averages. revision: yes

  2. Referee: [Evaluation] Evaluation on 19 reports: no inter-rater reliability, ground-truth labels, baseline comparisons, or sensitivity analysis are reported for the KPI calculations, leaving open whether the scores reflect reproducible measurement or ad-hoc assignment.

    Authors: We agree that these elements are absent. The evaluation is presented as preliminary and exploratory, with scores produced automatically by the prototype on anonymized student reports. No human raters were used, so inter-rater reliability metrics were not applicable or collected. We will revise the evaluation section to explicitly state these limitations, clarify that the scoring follows defined rules (detailed per the first comment), and add a forward-looking discussion of planned validation studies including baselines and sensitivity analysis. revision: partial

standing simulated objections not resolved
  • Absence of ground-truth labels and inter-rater reliability for the 19 reports, which cannot be retroactively generated without new data collection.

Circularity Check

0 steps flagged

No significant circularity; framework proposal lacks internal reduction to inputs

full rationale

The manuscript introduces LLMography as a new framework and prototype for generating KPI scores (Prompt Quality Score, Human Direction Score, etc.) from conversation traces, reports averages from an exploratory evaluation on 19 external student reports, and notes self-application to its own writing process. No equations, scoring algorithms, or definitions appear in the provided text that would make any claimed indicator equivalent to its inputs by construction, nor is there a self-citation chain or fitted parameter renamed as prediction. The central contribution is the definition of the framework itself, which remains independent of the reported scores and does not reduce to a closed loop in the given material.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 2 invented entities

The framework rests on the untested assumption that conversation logs yield objective contribution metrics and introduces several new scoring entities without independent evidence of their validity or calibration.

free parameters (1)
  • KPI scoring thresholds and weights
    Cutoffs and relative importance of Prompt Quality, Human Direction, and other scores are not derived from data or theory and must be chosen by the implementer.
axioms (1)
  • domain assumption Human-AI conversation traces contain extractable, quantifiable signals of human versus AI contribution
    Invoked as the basis for all KPI generation but not demonstrated or justified in the abstract.
invented entities (2)
  • LLMography label no independent evidence
    purpose: Classification of interaction type (human-originated, co-produced, etc.)
    New categorical output introduced by the framework with no external validation.
  • Prompt Quality Score no independent evidence
    purpose: Numerical measure of prompt effectiveness
    Invented metric whose computation method is unspecified.

pith-pipeline@v0.9.1-grok · 5825 in / 1392 out tokens · 43294 ms · 2026-06-30T02:12:22.839760+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 5 canonical work pages

  1. [1]

    Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. Guidelines for human-ai interaction. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–13, 2019

    2019

  2. [2]

    How to cite chatgpt, April 2023

    APA Style. How to cite chatgpt, April 2023

    2023

  3. [3]

    Citing generative ai in apa style: Part 1—reference formats, September 2025

    APA Style. Citing generative ai in apa style: Part 1—reference formats, September 2025

    2025

  4. [4]

    Cesare G. Ardito. Contra generative ai detection in higher education assessments.arXiv preprint arXiv:2312.05241, 2023. 14

    work page arXiv 2023

  5. [5]

    K. Bittle. Generative ai and academic integrity in higher education.Information, 16(4):296, 2025

    2025

  6. [6]

    The ai assessment scale (aias) in action: A pilot implementation of genai supported assessment.arXiv preprint arXiv:2403.14692, 2024

    Leon Furze, Mike Perkins, Jasper Roe, and Jason MacVaugh. The ai assessment scale (aias) in action: A pilot implementation of genai supported assessment.arXiv preprint arXiv:2403.14692, 2024

    work page arXiv 2024

  7. [7]

    Kofinas, Chih-Hua Tsay, and David Pike

    Alexander K. Kofinas, Chih-Hua Tsay, and David Pike. The impact of generative ai on academic integrity of authentic assessments within a higher education context.British Journal of Educational Technology, 56:2522–2549, 2025

    2025

  8. [8]

    The w3c prov family of specifications for modelling provenance metadata

    Paolo Missier, Khalid Belhajjame, and James Cheney. The w3c prov family of specifications for modelling provenance metadata. InProceedings of the 16th International Conference on Extending Database Technology, pages 773–776, 2013

    2013

  9. [9]

    Higher education assessment practice in the era of generative ai tools

    Bayode Ogunleye, Kudirat Ibilola Zakariyyah, Oluwaseun Ajao, Olakunle Olayinka, and Hemlata Sharma. Higher education assessment practice in the era of generative ai tools. arXiv preprint arXiv:2404.01036, 2024

    work page arXiv 2024

  10. [10]

    Shaping integrity: Why generative artificial intelligence does not have to undermine education.arXiv preprint arXiv:2407.19088, 2024

    Myles Joshua Toledo Tan and Nicholle Mae Amor Tan Maravilla. Shaping integrity: Why generative artificial intelligence does not have to undermine education.arXiv preprint arXiv:2407.19088, 2024

    work page arXiv 2024

  11. [11]

    Generative ai in higher education: Seeing chatgpt through universities’ policies, resources, and guidelines.arXiv preprint arXiv:2312.05235, 2023

    Hui Wang, Anh Dang, Zihao Wu, and Son Mac. Generative ai in higher education: Seeing chatgpt through universities’ policies, resources, and guidelines.arXiv preprint arXiv:2312.05235, 2023

    work page arXiv 2023

  12. [12]

    PROV-DM: The PROV Data Model

    World Wide Web Consortium. PROV-DM: The PROV Data Model. W3C Recommenda- tion, April 2013

    2013

  13. [13]

    PROV-Overview: An Overview of the PROV Family of Documents

    World Wide Web Consortium. PROV-Overview: An Overview of the PROV Family of Documents. W3C Working Group Note, April 2013. 15

    2013