The reviewed record of science sign in
Pith

arxiv: 2605.23956 · v1 · pith:WKKF2CQP · submitted 2026-05-11 · cs.AI · cs.LG· cs.MA

QUIVER: A Formal Framework for Quantifying Perturbation Propagation and Bifurcation in Compound AI Systems

Prashanti Nilayam , Sankalp Nayak This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-30 22:17 UTCgrok-4.3pith:WKKF2CQPrecord.jsonopen to challenge →

classification cs.AI cs.LGcs.MA
keywords compound AI systemsperturbation propagationtrajectory divergencebifurcation thresholdsLLM pipelinessensitivity matrixevaluation faithfulnesscascade patterns
0
0 comments X

The pith

QUIVER defines sensitivity matrices and trajectory divergence measures to track perturbation propagation in graph-structured LLM pipelines.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces QUIVER to quantify how small input changes affect chained LLM systems whose nodes produce mixed outputs and can take different execution paths. It supplies a sensitivity matrix that labels edges by amplification or absorption behavior plus a decomposition of observed differences into value drift, path changes, and iteration shifts. Bifurcation thresholds locate the minimal change that alters the structural path, while a faithfulness check flags when node-level test data no longer match live traffic. Validation across more than eight thousand traces from three distinct production and research pipelines shows the measures can separate mechanically different cascades that produce identical overall divergence and can flag nodes likely to bifurcate from data alone. These capabilities matter because compound AI systems are now standard in deployed settings yet lack tools for diagnosing where and why behavior diverges under variation.

Core claim

QUIVER defines a sensitivity matrix with type-dispatched distance metrics that classifies edges as amplifiers, absorbers, or threshold-sensitive, complemented by occurrence-lift. Trajectory divergence is decomposed into value drift, structural path divergence, and iteration count divergence. Bifurcation thresholds identify the smallest perturbation that causes structural execution path changes. Distribution faithfulness quantifies when per-node evaluation datasets diverge from production distributions. On two enterprise pipelines and one DSPy multihop QA pipeline, 8200+ instrumented traces demonstrate that the framework reveals distinct sensitivity profiles, distinguishes mechanistically dif

What carries the argument

The QUIVER framework's sensitivity matrix with type-dispatched distance metrics together with its decomposition of trajectory divergence into value drift, structural path divergence, and iteration count divergence.

If this is right

  • Distinct sensitivity profiles can be recovered for structurally different compound AI architectures.
  • Mechanistically different cascade patterns that yield identical aggregate divergence rates become distinguishable.
  • Nodes prone to trajectory bifurcation become predictable from observational traces without explicit perturbation experiments.
  • Stale evaluation artifacts localize to particular node-field categories that standard aggregate metrics leave hidden.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • System designers could use the sensitivity matrix to prioritize hardening or monitoring of high-amplifier nodes during architecture iteration.
  • The same decomposition might be applied to other stochastic directed-graph computation systems whose nodes produce mixed outputs.
  • Continuous collection of the three divergence components could support early detection of distribution shift in live deployments.

Load-bearing premise

The type-dispatched distance metrics and decomposition of trajectory divergence into value drift, structural path divergence, and iteration count divergence accurately capture perturbation propagation and bifurcation behavior in stochastic LLM pipelines without requiring external ground-truth perturbation experiments.

What would settle it

Controlled injection of known perturbations into the same pipelines followed by direct measurement of whether the predicted bifurcation thresholds and edge classifications match the observed structural path changes and divergence components.

Figures

Figures reproduced from arXiv: 2605.23956 by Prashanti Nilayam, Sankalp Nayak.

Figure 1
Figure 1. Figure 1: Comparative architectures of Systems P and Q. System P uses a parallel-intake first wave (rewriter, signal analysis, context sufficiency) feeding a planner loop with conditional replanning and tool execution. System Q uses a dual-planner entry point with a winner-selection mechanism that routes to either a fast path or a slow path through retrieval, reranking, and generation. framework quantifies how pertu… view at source ↗
Figure 2
Figure 2. Figure 2: σˆ heatmap, BM25 vs ColBERTv2. Rows: upstream vi ; columns: downstream vj . Color encodes σˆij on a diverging scale centered at σˆ=1 (white). Greyed cells are temporally infeasible (i fires after j) or have insufficient data (n(di>ϵ)<30). Strong amplifier and absorber cells (deep red, deep blue) are stable across retrievers; only near-unity cells shift color — the same three edges marked flip in [PITH_FUL… view at source ↗
Figure 3
Figure 3. Figure 3: Per-pair (di , dj ) scatter for the most pronounced flipping edge. Each dot is one same￾input pair. Dashed diagonal is dj=di (σˆ=1); pairs above amplify, pairs below absorb. Grey dots on the y-axis (di<ϵ) carry the intrinsic-noise term — both retrievers cluster these tightly at dj = 0, so the intrinsic noise is exactly 0.000 in both cases. The cloud of drifted pairs (di>ϵ) sits well below the diagonal unde… view at source ↗
read the original abstract

Compound AI systems that chain multiple LLM calls into directed computation graphs are now the dominant architecture for production AI. Although these architectures leverage heterogeneous nodes with mixed-mode outputs, no existing framework quantifies how perturbations propagate through such pipelines, where nodes are stochastic and execution paths can diverge structurally. We introduce QUIVER, a formal framework for measuring perturbation propagation in graph-structured LLM pipelines. The framework defines: (1) a sensitivity matrix with type-dispatched distance metrics that classifies edges as amplifiers, absorbers, or threshold-sensitive, complemented by occurrence-lift; (2) trajectory divergence decomposing variation into value drift, structural path divergence, and iteration count divergence; (3) bifurcation thresholds identifying the smallest perturbation that causes structural execution path changes; and (4) distribution faithfulness, quantifying when per node evaluation datasets diverge from production distributions. We validate on two production enterprise pipelines and a public DSPy multihop QA pipeline, three structurally distinct architectures. Across 8,200+ instrumented traces (32,000+ pair comparisons), we demonstrate that QUIVER reveals distinct sensitivity profiles across architectures, distinguishes mechanistically different cascade patterns producing identical divergence rates, predicts nodes prone to trajectory bifurcation from observational data alone, and localizes stale evaluation artifacts to specific node-field categories that aggregate metrics cannot surface.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper introduces QUIVER, a formal framework for quantifying perturbation propagation and bifurcation in compound AI systems consisting of chained LLM calls into directed graphs. It defines (1) a sensitivity matrix using type-dispatched distance metrics that classify edges as amplifiers, absorbers, or threshold-sensitive (with occurrence-lift), (2) trajectory divergence decomposed into value drift, structural path divergence, and iteration count divergence, (3) bifurcation thresholds for the smallest perturbation causing structural path changes, and (4) distribution faithfulness for per-node evaluation datasets. Validation on 8,200+ instrumented traces (32,000+ pair comparisons) from two enterprise pipelines and one DSPy multihop QA graph claims to show distinct sensitivity profiles across architectures, distinguish mechanistically different cascades with identical divergence rates, predict bifurcation-prone nodes from observational data alone, and localize stale artifacts to specific node-field categories.

Significance. If the metrics hold, the framework supplies a structured, decomposable approach to analyzing stochastic LLM pipelines that aggregate metrics cannot address, with explicit type-dispatched distances and divergence components that could aid debugging of production systems. The application to real enterprise traces and the distinction of cascade mechanisms constitute concrete strengths; the paper also supplies explicit metric definitions that support reproducibility.

major comments (1)
  1. [Validation] Validation section: the central claims (distinct profiles, distinguishing cascades with identical rates, predicting bifurcation nodes, localizing artifacts) rest on the type-dispatched distance metrics and the value-drift / structural-path / iteration-count decomposition applied to post-hoc observational traces. No controlled perturbation-injection experiments at known nodes are reported, so it remains possible that alternative distance functions or decompositions would yield the same patterns; this directly affects whether the metrics correctly identify amplifiers, bifurcation points, or stale artifacts.
minor comments (1)
  1. Clarify how the 32,000+ pair comparisons are constructed from the 8,200+ traces (e.g., which pairs are compared and under what sampling) to ensure the reported counts are unambiguous.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback and for recognizing the framework's potential strengths in analyzing real enterprise traces and distinguishing cascade mechanisms. We address the major comment on validation below.

read point-by-point responses

  1. Referee: [Validation] Validation section: the central claims (distinct profiles, distinguishing cascades with identical rates, predicting bifurcation nodes, localizing artifacts) rest on the type-dispatched distance metrics and the value-drift / structural-path / iteration-count decomposition applied to post-hoc observational traces. No controlled perturbation-injection experiments at known nodes are reported, so it remains possible that alternative distance functions or decompositions would yield the same patterns; this directly affects whether the metrics correctly identify amplifiers, bifurcation points, or stale artifacts.

    Authors: We agree that the absence of controlled perturbation-injection experiments at known nodes is a limitation of the current validation, which relies on post-hoc observational traces. This design choice was driven by the goal of demonstrating applicability to production systems where deliberate perturbations risk operational disruption. To address the concern that alternative metrics or decompositions might produce similar patterns, we will revise the manuscript to add (1) a new limitations subsection explicitly discussing the observational nature of the validation and (2) a sensitivity analysis on the public DSPy multihop QA pipeline that applies controlled perturbations at selected nodes and compares outputs against the type-dispatched metrics and divergence decomposition. This will provide direct evidence on whether the metrics correctly surface amplifiers and bifurcation points. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper defines its core components (sensitivity matrix with type-dispatched metrics, trajectory divergence decomposition into value/structural/iteration components, bifurcation thresholds, and distribution faithfulness) as explicit framework primitives and then applies them to 8200+ observational traces from three pipelines. No equations, parameter fits, or self-citations are exhibited that reduce any claimed prediction or classification back to the inputs by construction. The reported distinctions and localizations are presented as empirical outcomes of applying the independently defined metrics to the data; the derivation chain does not collapse into tautology or fitted-input renaming. This is the normal case of a self-contained definitional framework evaluated on external traces.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; no equations or sections available to identify free parameters, axioms, or invented entities. No free parameters, axioms, or invented entities can be extracted.

pith-pipeline@v0.9.1-grok · 5767 in / 1088 out tokens · 21385 ms · 2026-06-30T22:17:32.835499+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Detection Without Correction: A Two-Parameter Decomposition of Multi-Stage LLM Pipelines

    cs.MA 2026-05 unverdicted novelty 6.0

    LLM pipelines fail mainly via detection-without-correction, with conditional miscorrection rates of 53-94% dominating across models, benchmarks, and methods while detection thresholds vary widely.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    This is valid for small perturbations around the current operating point

    Local linearity:The sensitivity of each edge is independent of perturbation magnitude. This is valid for small perturbations around the current operating point. For large perturbations, the bifurcation analysis (Definitions 9–10) provides the appropriate tool

    work page

  2. [2]

    Under current operating conditions, which node is causing structural instability?

    Path independence:The path-product gives per-path sensitivity, not total sensitivity when multiple paths connect two nodes. For node pairs connected by k paths, we report both maxp σ(p) and the empirically measured transitive sensitivity ˆσ(i, j). Agreement between the two validates the multiplicative approximation; divergence indicates significant intera...

    work page 2025

  3. [3]

    Measured i on a held-out probe set before and after the change

    work page

  4. [4]

    For each downstream edge (vi, vj), compare di against the drift budget τij at the team’s chosen significance levelα

    work page

  5. [5]

    Ifd i > τ ij, re-run golden dataset evaluation for nodev j

    work page

  6. [6]

    Ifd i ≤τ ij, skip re-evaluation ofv j — the perturbation is within the safe budget. This replaces the current industry practice of either re-evaluating everything (expensive and slow) or re-evaluating nothing downstream (risky) with a quantitatively grounded policy derived from the pipeline’s measured sensitivity profile. For pipelines with conditional lo...

    work page

  7. [7]

    Within each group, form all pairwise combinations

    Pair formation.Group traces by input similarity (same seed, same intent class, or similar query embedding). Within each group, form all pairwise combinations. For a corpus of N traces withSseeds andRrepeats per seed, this yieldsS R 2 pairs

    work page

  8. [8]

    Store the per-field distances alongside the aggregate for later analysis

    Distance computation.For each pair and each node vi, compute the type-dispatched distance di using Definition 2. Store the per-field distances alongside the aggregate for later analysis. 17

    work page

  9. [9]

    Sensitivity estimation.For each edge (vi, vj)∈E , collect all pairs where di > ϵ and computeˆσij =E[d j/di]over those pairs

    work page

  10. [10]

    Partial regression (for multi-parent nodes).When node vj has multiple parents, the simple ratio dj/di confounds the contributions of different parents. We estimate partial sensitivities via multivariate regression: dj = X k∈pa(j) αk dk + X k<l∈pa(j) γkl dk dl +ε(16) The coefficient αi is the partial sensitivity of vj to vi, controlling for co-variation ac...

    work page

  11. [11]

    Heuristics include: gap analysis that returned marginal results, confidence scores near decision thresholds, or categorical classifications with low model confidence

    Boundary identification.From the observational corpus, identify traces near the bifurcation boundary — traces where the loop controller nearly chose a different action or a conditional branch was marginal. Heuristics include: gap analysis that returned marginal results, confidence scores near decision thresholds, or categorical classifications with low mo...

    work page

  12. [12]

    Perturbation strategies depend on the node’s output type: • Text outputs:paraphrase at varying temperatures, keyword addition/removal, prompt variant substitution

    Perturbation application.For each near-boundary trace and each target upstream node vi, apply controlled perturbations of known magnitude. Perturbation strategies depend on the node’s output type: • Text outputs:paraphrase at varying temperatures, keyword addition/removal, prompt variant substitution. •Categorical outputs:flip to next-most-likely class. •...

    work page

  13. [13]

    Re-execution.Re-run the pipeline from the perturbed node forward, holding all other inputs fixed. 18

    work page

  14. [14]

    Threshold recording.Record the smallest perturbation magnitude di at which Dshape >0 orD iter >0

    work page

  15. [15]

    always upstream-dirty

    Aggregation.Report ˆβshape(vi) as the minimum observed threshold across all near- boundary traces, with sample size and interquartile range. Stratification.For perturbations that are binary (categorical flips, boolean overrides), the perturba- tion is either effective (the baseline had a different value) or a no-op (the baseline already matched the pertur...

    work page arXiv 1950