Pith. sign in

REVIEW 3 major objections 6 minor 36 references

A Unifying Information-theoretic Perspective on Evaluating Generative Models

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that precision and recall metrics for generative models unify as divergence estimates and proposes three cross-entropy scores that separately measure fidelity, inter-class diversity, and intra-class diversity.

desk verdict A useful unification of kNN-based precision/recall metrics, but the proposed RE component cannot carry its claimed intra-class diversity load. read the letter →

arxiv 2412.14340 v3 pith:XOIXYTDY submitted 2024-12-18 cs.LG cs.AIcs.CV

Alexis Fox , Samarth Swarup , Abhijin Adiga This is my paper · ORCID
classification cs.LGcs.AIcs.CV MSC 62B1062G0768T07
keywords generativemodelevaluationprecisionandrecallmodecollapseshrinkageinformationtheoryk-nearestneighborscross-entropydensityestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a class of k-nearest-neighbor-based precision and recall metrics for generative models are all estimating the same underlying information-theoretic quantity—a divergence between the real and generated distributions—with different choices of loss function. It further claims that no existing metric can separately measure the two kinds of diversity loss, and it introduces three scores, Precision Cross-Entropy, Recall Cross-Entropy, and Recall Entropy, that it argues separate fidelity, inter-class diversity, and intra-class diversity. This matters because mode invention, mode dropping, and mode shrinkage call for different fixes, and most current metrics cannot tell them apart.

What carries the argument

The carrying mechanism is kNN density estimation. The entropy estimator $\hat{H}_k(X)$ and cross-entropy estimator $\hat{CE}_k(X,Y)$ are taken from Leonenko, Pronzato, and Savani (2008) and are used to define the three scores; the Rényi divergence estimator of Noshad et al. (2017), built from counts of points from each set inside a kNN ball, supplies the bias analysis that unifies PRC and Density & Coverage. The three scores subtract $H(R)$ as a baseline so identical real and generated distributions give zero for all three, and per-sample additivity lets each score be decomposed to individual generated or real points.

What would settle it

Generate real samples from two well-separated Gaussian modes with equal intra-class variance and generated samples from only one of those modes with the same intra-class variance; compute RE on large sample sets. If RE moves substantially away from zero while PCE stays near zero, the intra-class-diversity interpretation of RE fails.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the empirical definitions of Precision Recall Cover and Density & Coverage rearrange into nearest-neighbor-ratio estimators of a general divergence of the form $D(X\|Y)=E_{x\sim X}[g(f_Y(x)/f_X(x))]$, where PRC and Coverage use a threshold indicator $g(z)=1[z\ge \eta/C]$, Density uses $g(z)=z$, and KL divergence uses $g(z)=-\log z$. On that basis it defines $PCE=CE(G,R)-H(R)$, $RCE=CE(R,G)-H(R)$, and $RE=H(G)-H(R)$, and claims these three components separately detect mode invention, mode dropping, and mode shrinkage. Its experiments on ImageNet and CIFAR-10 are offered as evidence that PCE correlates with human realism judgment, that RCE responds to class removal, and that RE responds to class-conditional variance reduction.

Load-bearing premise

The central claim rests on the assumption that Recall Entropy, $H(G)-H(R)$, measures only intra-class diversity, so deleting an entire real mode leaves it unchanged.

Editorial extensions

If this is right

  • A single triplet of scores can flag mode invention, mode dropping, and mode shrinkage separately, something one-dimensional metrics such as Fréchet distance cannot do.
  • Precision Recall Cover and Density & Coverage reduce to special cases of a common divergence estimator, so their biases and sample-size behavior can be analyzed through one shared lens.
  • Density converges to 1 as sample size grows, so it stops measuring fidelity for large generated sets; this is a concrete warning for practitioners using Density as a fidelity score.
  • Of the recall-oriented measures, only Recall Entropy responds strongly to mode shrinkage, while Recall Cross-Entropy, Coverage, and Recall Coverage respond to mode dropping.
  • Because the estimators are additive per sample, PCE can be decomposed to flag memorized generated samples that sit nearly on top of real training points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper does not take is a class-conditional version of Recall Entropy, $H(G|C)-H(R|C)$, which would remove the inter-class entropy contribution and likely give a cleaner intra-class diversity signal.
  • If the unification is right, then designing a new precision/recall metric is equivalent to choosing a loss function $g$ for the density ratio; choosing a bounded or Lipschitz $g$ could reduce outlier sensitivity in a principled way.
  • The paper's sample-size result for Density implies that published comparisons using Density on large generated sets may need to be re-checked for sample-size artifacts; this is an implication beyond the paper's own experiments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an information-theoretic unification of kNN-based precision/recall metrics, focusing on Precision Recall Cover and Density & Coverage, by rewriting them as estimators of Rényi-type divergence functionals. It then introduces a three-component metric, PCE, RCE, and RE, which are claimed to separately quantify fidelity, inter-class diversity (mode dropping), and intra-class diversity (mode shrinkage). The authors derive bias expressions, state desiderata for generative-model metrics, and report experiments on CIFAR-10 and ImageNet, including correlation with human error rates, a CFG-based mode-shrinkage study, and a class-dropping mode-dropping study.

Significance. If the three-way separation held, the proposed metric would be a practically useful diagnostic tool, since most existing precision/recall measures conflate mode dropping with mode shrinkage. The paper has genuine strengths: it provides explicit algebraic connections between PRC/Density-Coverage and divergence estimators; it releases code and uses public benchmarks; and the observation that Density can become non-discriminative at large sample sizes is a useful caution. However, the central claim that RE isolates intra-class diversity is not currently established, and several supporting statements in the experimental sections are not quantitatively supported.

major comments (3)
  1. [Section 5.2, Eq. (25), and Figure 4] The claim that RE = H(G) - H(R) measures intra-class diversity is not supported by its definition. For a generated mixture G with class variable C and well-separated modes, H(G) ≈ H(C) + Σ_c w_c H(G_c), so deleting a mode reduces H(C) and thereby H(G); since H(R) is a constant baseline, RE must decrease under mode dropping as well as under mode shrinkage. Figure 4 asserts that RE remains relatively constant when classes are dropped from 1000 to 100, but no numerical values or error bars are reported, and the statement conflicts with Eq. (25). Please also specify whether the real set was held fixed in this experiment; if real classes were dropped together with generated ones, the experiment does not measure mode dropping against a fixed target. Without a conditional-entropy formulation (e.g., H(G|C) - H(R|C)) or equivalent mode-level analysis, the central claim that RE isolates intra-class diversity is not established.
  2. [Section 5.2, Eq. (24), and Figure 3] RCE = CE(R,G) - H(R) = D_KL(R||G), and KL divergence is sensitive to any mismatch between the real and generated densities, including within-mode variance mismatch. For Gaussian modes with the same mean and σ_G < σ_R, D_KL(R||G) = d/2[(σ_R²/σ_G²) - 1 + log(σ_G²/σ_R²)] > 0, so RCE should increase under mode shrinkage. The statement in Section 6 that RCE shows only a slight change as CFG is increased is therefore theoretically unexpected and needs quantitative support; if the effect is real, the paper should explain why the KL term is dominated by other factors. This matters because the claimed separation between RCE as an inter-class measure and RE as an intra-class measure is a core contribution.
  3. [Section 4.2, Eqs. (19)-(20)] The statement that J1(G,R) = 1 and hence Density is asymptotically 1 is only valid under a support condition. By definition, J1(G,R) = ∫ f_R(x) (f_G(x)/f_R(x)) dx over the support of R, which equals ∫_{supp(R)} f_G(x) dx; this can be strictly less than 1 when G places mass outside the support of R, and it is 0 for disjoint supports. The Gaussian example in Appendix C has overlapping supports, but the general claim in Section 4.2 should be qualified. This is a secondary result relative to the proposed metric, but it should be corrected because it is used to argue that Density is unreliable.
minor comments (6)
  1. [Section 6, Mode Dropping] The text says 'Figure 3(d) shows that the one-dimensional F D cannot distinguish diversity loss from precision,' but the referenced panel is in Figure 4; please correct the cross-reference.
  2. [Appendix A] There are several typos and spacing errors, including 'institition', 'enviroments', and 'V AEs'; the manuscript should be proofread before resubmission.
  3. [Section 4.3] The symbol D is used both for Density and for a general divergence (e.g., D(G||R)); please introduce a distinct notation, such as Div, to avoid ambiguity.
  4. [Section 4.1 and Appendix B] The transition from the combined-set balls B_{k,U} to the per-set balls B_{k,G}, and the replacement of the indicator function by a sigmoid, are asserted rather than derived. Please state these approximations explicitly and, if possible, provide a numerical check of their effect on the bias expression.
  5. [Section 5.2] The phrase 'normalized versions of the two components of D_KL(G||R)' is imprecise: PCE - RE = D_KL(G||R), but each term also contains the baseline H(R); consider saying 'shifted components' or defining the decomposition explicitly.
  6. [Figures 3 and 4] No raw values or confidence intervals are reported, and the text states that error bars are 'visually negligible'; the reader cannot assess the stability of RE and RCE from the figures alone, especially for the mode-dropping claim.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the proposed metrics are defined from entropy/cross-entropy and validated against external benchmarks, with no fitted-input-as-prediction or self-citation chain.

full rationale

The paper's derivation chain has two parts. First, Section 4 rewrites Precision Recall Cover and Density & Coverage in terms of the kNN ratio estimators of Noshad et al. (2017). This is an explicit rearrangement of the published definitions of those metrics: Equation 10 rewrites PC using the kNN ratio estimator, and the paper itself identifies the two structural differences (neighbor set G versus U, and the added 1 in the denominator) and derives the associated bias terms in Appendix B. This is not circular: the reformulation does not assume the conclusion it claims, and the estimator it builds on is an external, published mathematical result, not a self-citation. Second, the proposed metric components PCE, RCE, and RE are defined directly from Shannon entropy and cross-entropy (Eqs. 23-25). They are not fitted to experimental outcomes, and the experiments in Section 6 use external evidence: human error-rate rankings from Stein et al. (2024), a classifier-free guidance perturbation implemented with DiT-XL-2, and an ImageNet mode-dropping protocol. These are outside the metric's fitted values. The desiderata in Section 5.1 were chosen to match the intended semantics, so the sensitivity experiments partly re-test design choices, but that is construction validation rather than circular reasoning. There is no load-bearing self-citation chain: the authors do not justify any central premise by citing their own prior work. The skeptical concern that RE = H(G) - H(R), being a marginal-entropy difference, may also respond to mode dropping is a correctness and validity question about whether Figure 4's claim is supported by the definition and reported numbers. Per the review rules, that concern belongs under correctness risk rather than circularity, and it does not by itself establish that any derivation reduces to its own inputs. No concrete circular step of the enumerated kinds can be quoted from the paper.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The metric components are mathematical constructs. The main unstated premise is that RE=H(G)-H(R) behaves as an intra-class diversity measure, which is not naturally implied by the definition.

free parameters (2)
  • k = 5 in all experiments
    k is the number of nearest neighbors used by the entropy and cross-entropy estimators and by PRC/Density/Coverage; all scores depend on it, and no sensitivity analysis is reported.
  • k' = 15 in Section 6
    k' is the coverage threshold in PRC. The setting k=5, k'=15 conflicts with the definition k=Ck' for integer C in Section 4.1, so the experimental configuration is ambiguous.
assumptions (4)
  • domain assumption Real and generated densities belong to a Holder smoothness class, as required by the kNN estimator bias bounds in Noshad et al. (2017).
    Invoked in Section 4 and Appendix B to transfer consistency and bias results from the divergence estimator to PRC and Density/Coverage.
  • domain assumption Counts of points inside a kNN ball estimate the local density ratio f_R/f_G (and vice versa).
    This is the core premise of the unification, used around Eqns 10-13; it is standard for kNN density estimation but not proven in this paper.
  • domain assumption DINOv2 embeddings provide a feature space in which Euclidean distances correspond to perceptual similarity.
    Stated as a requirement in Section 7 and used in all experiments following Stein et al. (2024); the metric's meaning depends on this embedding.
  • ad hoc to paper The sigmoid approximation to the indicator function preserves the asymptotic bias results.
    Introduced in Section 4.1 after Eqn 13 to satisfy the Lipschitz condition of Lemma 3.2 of Noshad et al. (2017); the bias analysis in Appendix B relies on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Unifying Information-theoretic Perspective on Evaluating Generative Models." pith.science (2026). https://pith.science/paper/XOIXYTDY

@misc{pith2026241214340,
  author       = {Pith},
  title        = {Pith review of: A Unifying Information-theoretic Perspective on Evaluating Generative Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XOIXYTDY}},
  note         = {Machine review of arXiv:2412.14340}
}
read the original abstract

Considering the difficulty of interpreting generative model output, there is significant current research focused on determining meaningful evaluation metrics. Several recent approaches utilize "precision" and "recall," borrowed from the classification domain, to individually quantify the output fidelity (realism) and output diversity (representation of the real data variation), respectively. With the increase in metric proposals, there is a need for a unifying perspective, allowing for easier comparison and clearer explanation of their benefits and drawbacks. To this end, we unify a class of kth-nearest-neighbors (kNN)-based metrics under an information-theoretic lens using approaches from kNN density estimation. Additionally, we propose a tri-dimensional metric composed of Precision Cross-Entropy (PCE), Recall Cross-Entropy (RCE), and Recall Entropy (RE), which separately measure fidelity and two distinct aspects of diversity, inter- and intra-class. Our domain-agnostic metric, derived from the information-theoretic concepts of entropy and cross-entropy, can be dissected for both sample- and mode-level analysis. Our detailed experimental results demonstrate the sensitivity of our metric components to their respective qualities and reveal undesirable behaviors of other metrics.

Figures

Figures reproduced from arXiv: 2412.14340 by the authors.

Figure 1
Figure 1. We visualize the various failure modes (a)-(c) for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pearson correlation heatmap between the precision [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Behavior of metrics at different CFG values for [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Two-dimensional visualization of the sample set of [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Behavior of Density and P C when sample size is substantially increased, from 1 × 104 to 1.5 × 106 . The horizontal dashed line at y = 1.0 signifies what our analysis claims density converges too. Memorization. As noted in Section 5.3, our measures can be dissected for…
Figure 7
Figure 7. Figure 7: A sample of the “most memorized” images from [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 18 canonical work pages

  1. [1]

    S.; and van der Schaar, M

    Alaa, A.; Van Breugel, B.; Saveliev, E. S.; and van der Schaar, M. 2022. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In International Conference on Machine Learning, 290--306. PMLR

    work page 2022

  2. [2]

    Borji, A. 2018. Pros and Cons of GAN Evaluation Measures. arXiv:1802.03446

    arXiv 2018

  3. [3]

    Borji, A. 2022. Pros and cons of GAN evaluation measures: New developments. Computer Vision and Image Understanding, 215: 103329

    work page 2022

  4. [4]

    Cheema, F.; and Urner, R. 2023. Precision Recall Cover: A Method For Assessing Generative Models. In International Conference on Artificial Intelligence and Statistics, 6571--6594. PMLR

    work page 2023

  5. [5]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248--255. Ieee

    2009

  6. [6]

    Dhariwal, P.; and Nichol, A. 2021. Diffusion models beat GANs on image synthesis. Advances in neural information processing systems, 34: 8780--8794

    work page 2021

  7. [7]

    Djolonga, J.; Lucic, M.; Cuturi, M.; Bachem, O.; Bousquet, O.; and Gelly, S. 2020. Precision-recall curves using information divergence frontiers. In International Conference on Artificial Intelligence and Statistics, 2550--2559. PMLR

    work page 2020

  8. [8]

    J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y

    Goodfellow, I. J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative Adversarial Networks. arXiv:1406.2661

    arXiv 2014

Show all 36 references
  1. [9]

    Han, J.; Choi, H.; Choi, Y.; Kim, J.; Ha, J.-W.; and Choi, J. 2022. Rarity Score : A New Metric to Evaluate the Uncommonness of Synthesized Images. arXiv:2206.08549

    2022 arXiv

  2. [10]

    Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30

    2017

  3. [11]

    Ho, J.; and Salimans, T. 2022. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598

    2022 arXiv

  4. [12]

    Huang, X.; Khetan, A.; Cvitkovic, M.; and Karnin, Z. 2020. TabTransformer: Tabular Data Modeling Using Contextual Embeddings. arXiv:2012.06678

    2020 arXiv

  5. [13]

    P.; and Welling, M

    Kingma, D. P.; and Welling, M. 2022. Auto-Encoding Variational Bayes. arXiv:1312.6114

    2022 arXiv

  6. [14]

    A.; Zhou, S.; Zhao, S.; Gordon, M.; and Ermon, S

    Kolchinski, Y. A.; Zhou, S.; Zhao, S.; Gordon, M.; and Ermon, S. 2019. Approximating human judgment of generated image quality. arXiv preprint arXiv:1912.12121

    2019 arXiv

  7. [15]

    Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images

    2009

  8. [16]

    Kynk\" a \" a nniemi, T.; Karras, T.; Laine, S.; Lehtinen, J.; and Aila, T. 2019. Improved Precision and Recall Metric for Assessing Generative Models. In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d Alch\' e -Buc, F.; Fox, E.; and Garnett, R., eds., Advances in Neural Info...

    2019

  9. [17]

    Leonenko, N.; Pronzato, L.; and Savani, V. 2008. A class of R \'e nyi information estimators for multidimensional densities

    2008

  10. [18]

    Liu, L.; Pillutla, K.; Welleck, S.; Oh, S.; Choi, Y.; and Harchaoui, Z. 2021. Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals. In Ranzato, M.; Beygelzimer, A.; Dauphin, Y.; Liang, P.; and Vaughan, J. W., eds., Advance...

    2021

  11. [19]

    F.; Oh, S

    Naeem, M. F.; Oh, S. J.; Uh, Y.; Choi, Y.; and Yoo, J. 2020. Reliable Fidelity and Diversity Metrics for Generative Models. In III, H. D.; and Singh, A., eds., Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning R...

    2020

  12. [20]

    R.; Sekeh, S

    Noshad, M.; Moon, K. R.; Sekeh, S. Y.; and Hero, A. O. 2017. Direct estimation of information divergence using nearest neighbor ratios. In 2017 IEEE International Symposium on Information Theory (ISIT), 903--907. IEEE

    2017

  13. [21]

    Oquab, M.; Darcet, T.; Moutakanni, T.; Vo, H.; Szafraniec, M.; Khalidov, V.; Fernandez, P.; Haziza, D.; Massa, F.; El-Nouby, A.; Assran, M.; Ballas, N.; Galuba, W.; Howes, R.; Huang, P.-Y.; Li, S.-W.; Misra, I.; Rabbat, M.; Sharma, V.; Synnaeve, G.; Xu, H.; Jegou, H.; Mairal, ...

    2024 arXiv

  14. [22]

    Park, D.; and Kim, S. 2023. Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE

    2023

  15. [23]

    P \'o czos, B.; and Schneider, J. 2011. On the Estimation of -Divergences. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 609--617. JMLR Workshop and Conference Proceedings

    2011

  16. [24]

    Ravuri, S.; Rey, M.; Mohamed, S.; and Deisenroth, M. 2023. Understanding Deep Generative Models with Generalized Empirical Likelihoods. arXiv:2306.09780

    2023 arXiv

  17. [25]

    R \'e nyi, A. 1961. On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics, volume 4, 547--562. University of California Press

    1961

  18. [26]

    Sajjadi, M. S. M.; Bachem, O.; Lucic, M.; Bousquet, O.; and Gelly, S. 2018. Assessing Generative Models via Precision and Recall. In Bengio, S.; Wallach, H.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; and Garnett, R., eds., Advances in Neural Information Processing Systems...

    2018

  19. [27]

    Salimans, T.; Goodfellow, I.; Zaremba, W.; Cheung, V.; Radford, A.; and Chen, X. 2016. Improved techniques for training gans. Advances in neural information processing systems, 29

    2016

  20. [28]

    A.; Maheswaranathan, N.; and Ganguli, S

    Sohl-Dickstein, J.; Weiss, E. A.; Maheswaranathan, N.; and Ganguli, S. 2015. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. arXiv:1503.03585

    2015 arXiv

  21. [29]

    L.; Taylor, E.; and Loaiza-Ganem, G

    Stein, G.; Cresswell, J.; Hosseinzadeh, R.; Sui, Y.; Ross, B.; Villecroze, V.; Liu, Z.; Caterini, A. L.; Taylor, E.; and Loaiza-Ganem, G. 2024. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information ...

    2024

  22. [30]

    Theis, L.; Oord, A. v. d.; and Bethge, M. 2015. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844

    2015 arXiv

  23. [31]

    Unterthiner, T.; van Steenkiste, S.; Kurach, K.; Marinier, R.; Michalski, M.; and Gelly, S. 2019. Towards Accurate Generative Models of Video: A New Metric & Challenges. arXiv:1812.01717

    2019 arXiv

  24. [32]

    Xiang, J.; Liu, Y.; Cai, D.; Li, H.; Lian, D.; and Liu, L. 2021. Assessing Dialogue Systems with Distribution Distances. arXiv:2105.02573

    2021 arXiv

  25. [33]

    F.; Banchs, R.; and Rudnicky, A

    Zhang, C.; Sedoc, J.; D'Haro, L. F.; Banchs, R.; and Rudnicky, A. 2021. Automatic Evaluation and Moderation of Open-domain Dialogue Systems. arXiv:2111.02110

    2021 arXiv

  26. [34]

    F.; and Bernstein, M

    Zhou, S.; Gordon, M.; Krishna, R.; Narcomey, A.; Fei-Fei, L. F.; and Bernstein, M. 2019. HYPE : A benchmark for human eye perceptual evaluation of generative models. Advances in neural information processing systems, 32

    2019

  27. [35]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.