arxiv: 2604.03417 · v1 · submitted 2026-04-03 · 💻 cs.LG

Recognition: no theorem link

Beauty in the Eye of AI: Aligning LLMs and Vision Models with Human Aesthetics in Network Visualization

Peng Zhang , Xuefeng Li , Xiaoqi Wang , Han-Wei Shen , Yifan Hu

Authors on Pith no claims yet

Pith reviewed 2026-05-13 19:52 UTC · model grok-4.3

classification 💻 cs.LG

keywords network visualizationhuman aestheticsLLM alignmentvision modelsprompt engineeringaesthetic preferencesgraph layouts

0 comments

The pith

Large language models and vision models can align with human aesthetic preferences in network visualizations to the level of agreement between humans themselves.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that traditional heuristic metrics for network layouts do not consistently produce the best results. Instead, it uses human preferences from a study with 27 participants to train and test AI models as labelers. Through prompt engineering combining few-shot examples and image embeddings, LLMs achieve alignment with humans that matches human-human levels when filtered by confidence. Similarly, trained vision models reach comparable alignment. This suggests AI can replace costly human annotation for learning aesthetic preferences in visualizations.

Core claim

By collecting human preference labels through a user study with 27 participants and using them to bootstrap AI labelers, the work demonstrates that carefully engineered prompts for LLMs, incorporating few-shot examples and diverse input formats like image embeddings, significantly improve alignment with human judgments. Filtering LLM responses by confidence score further elevates this alignment to match the level of agreement between human annotators. Trained vision models similarly achieve VM-human alignment comparable to human-human levels, establishing AI as a feasible scalable proxy for human labelers in network visualization aesthetics.

What carries the argument

Prompt engineering for LLMs that combines few-shot examples with image embeddings, plus confidence-based filtering, and training of vision models on human preference data to approximate aesthetic judgments.

If this is right

Human labels from a small study can bootstrap AI systems that match human consistency on aesthetic choices.
AI proxies can scale up the creation of large datasets for training generative models of network layouts.
Network visualization tools could incorporate AI-based aesthetic evaluation instead of relying solely on heuristics like stress.
Combining multiple input formats improves AI performance on visual preference tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

These AI labelers could enable training on much larger and more diverse graph datasets than possible with human annotators alone.
If the alignment holds across domains, it might extend to other visualization types beyond networks.
Confidence filtering suggests a way to use AI outputs selectively to maintain quality without full human review.

Load-bearing premise

Preferences collected from only 27 participants in a single user study provide a stable and generalizable ground truth for human aesthetic judgment across different graphs, domains, and populations.

What would settle it

A follow-up study with hundreds of participants from varied backgrounds rating the same set of layouts and showing substantially different preference distributions would indicate that the current labels do not generalize.

Figures

Figures reproduced from arXiv: 2604.03417 by Han-Wei Shen, Peng Zhang, Xiaoqi Wang, Xuefeng Li, Yifan Hu.

**Figure 1.** Figure 1: A UI showing eight graph visualizations. Of the users (n = 43), 56% preferred the starred option, and 16% chose the bottom-left. data collection, including incorporating more graphs, additional algorithms, and perturbed variants of existing layouts. With the recent advent of Large Language Models (LLMs), particularly Vision-Language Models (VLMs), a new possibility has emerged: using an LLM to emulate hu… view at source ↗

**Figure 2.** Figure 2: Demonstration of memory bank vs graph representation. The green route shows the graph representation, while the blue route shows the image-embedding–based memory bank approach. DINOv2-base model, which provides a 768-dimensional representation, in both 1-shot and 5-shot settings. In the 10-shot setting, we used the DINOv2-small model, yielding a 384-dimensional representation to accommodate input size co… view at source ↗

**Figure 3.** Figure 3: Left: Human-human and LLM-human alignment as a function of VM confidence. Right: Human-human and VM-human alignment as a function of VM confidence. The number of test samples remaining after threadholding is given at the top of each figure. 6.4. Understanding the AI labelers We further explore the distribution of preferred layouts across humans, VM, and LLM in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

**Figure 4.** Figure 4: Human-human, LLM-human and VM-human alignment visualization. Here we pick the seven most prolific human labelers. 6.6. Findings for RQ2 and RQ3 Based on the above results, we can conclude that • Regarding RQ2, with careful prompt engineering using a memory bank and self-supervised vision-encoder features, LLM–human alignment can be improved from 19.7% (0-shot © 2026 The Author(s). Computer Graphics Forum p… view at source ↗

**Figure 5.** Figure 5: Examples illustrating agreement and disagreement between LLM and human preferences. image-only) to 31.11%. While a gap to human–human alignment (38.34%) remains, this gap can be closed by applying a confidence threshold, retaining approximately 65% of the labels. • Regarding RQ3, the fined-tuned DINOv2 VM with soft multiclass loss function performs very close to an average human labeler (36.81% vs. 38.… view at source ↗

**Figure 7.** Figure 7: A screenshot showing the training provided to the participants purpose of the study: “We are conducting a research project related to graph drawing. We hope you can help us build a collection of human-preferred graph layouts. For each graph, you will be asked to choose the one drawing you like the most from a set of 8. There is no required number of graphs to label—we simply ask that you label as many as … view at source ↗

**Figure 8.** Figure 8: Distribution of layout algorithms behind the visualizations chosen by human subjects as being the most pleasing We found that the visualizations from different algorithms have different probabilities of being selected [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗

**Figure 9.** Figure 9: A screenshot showing the preference of humans vs that of the LLM for graph 1138_bus (|V| = 1138,|E| = 1452). Statistical Significance Tests To evaluate whether observed performance differences are statistically meaningful, we conducted paired-sample t-tests using peruser alignment scores. © 2026 The Author(s). Computer Graphics Forum published by Eurographics and John Wiley & Sons Ltd [PITH_FULL_IMAGE:f… view at source ↗

**Figure 10.** Figure 10: A screenshot showing the preference of human vs that of the VM for graph 1138_bus(|V| = 1138,|E| = 1452) [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗

**Figure 11.** Figure 11: A screenshot showing the preference of human vs that of the LLM for graph USPowerGrid(|V| = 4591,|E| = 6594) [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗

**Figure 12.** Figure 12: A screenshot showing the preference of human vs that of the VM for graph USPowerGrid (|V| = 4591,|E| = 6594) For the comparison across different model sizes, the pairedsample t-test indicated that the difference was not statistically significant (t = 1.1245, p = 0.2608). This suggests that increasing model size does not lead to a statistically reliable improvement in alignment performance in our setting… view at source ↗

read the original abstract

Network visualization has traditionally relied on heuristic metrics, such as stress, under the assumption that optimizing them leads to aesthetic and informative layouts. However, no single metric consistently produces the most effective results. A data-driven alternative is to learn from human preferences, where annotators select their favored visualization among multiple layouts of the same graphs. These human-preference labels can then be used to train a generative model that approximates human aesthetic preferences. However, obtaining human labels at scale is costly and time-consuming. As a result, this generative approach has so far been tested only with machine-labeled data. In this paper, we explore the use of large language models (LLMs) and vision models (VMs) as proxies for human judgment. Through a carefully designed user study involving 27 participants, we curated a large set of human preference labels. We used this data both to better understand human preferences and to bootstrap LLM/VM labelers. We show that prompt engineering that combines few-shot examples and diverse input formats, such as image embeddings, significantly improves LLM-human alignment, and additional filtering by the confidence score of the LLM pushes the alignment to human-human levels. Furthermore, we demonstrate that carefully trained VMs can achieve VM-human alignment at a level comparable to that between human annotators. Our results suggest that AI can feasibly serve as a scalable proxy for human labelers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

LLMs can reach human-human agreement on network layout preferences with few-shot prompting plus image embeddings and confidence filtering, but the 27-participant study is too narrow to treat as stable ground truth.

read the letter

The paper's main contribution is a new human preference dataset for graph layouts collected from 27 participants, plus the finding that LLMs get close to human agreement when you combine few-shot examples with image embeddings and then filter low-confidence outputs. Trained vision models reach similar alignment on the same labels. This moves the work past earlier machine-labeled baselines and gives concrete prompting recipes that improve results step by step.

Referee Report

3 major / 2 minor

Summary. The paper claims that LLMs and vision models can act as scalable proxies for human aesthetic judgments in network visualizations. Using preference labels collected from a user study with 27 participants, it shows that prompt engineering combining few-shot examples and image embeddings improves LLM-human alignment, with confidence-score filtering reaching human-human levels, while carefully trained VMs achieve VM-human alignment comparable to inter-human agreement.

Significance. If the alignment results are robust, the work offers a practical way to bypass expensive large-scale human labeling for training generative models of network layouts, potentially enabling data-driven aesthetics at scale in visualization research.

major comments (3)

[User Study] User study section: the central claims rest on labels from only 27 participants with no reported inter-annotator agreement statistics, cross-validation, leave-one-participant-out tests, or demographic details; this directly affects the reliability of the human-human baseline and all LLM/VM alignment numbers derived from it.
[Abstract] Abstract and results: the assertion that LLM-human alignment reaches 'human-human levels' after confidence filtering lacks any quantitative metrics, exact agreement scores, statistical significance tests, or tables comparing the values, so the claim cannot be verified from the supplied evidence.
[VM Experiments] VM training description: the claim that 'carefully trained VMs' achieve human-comparable alignment provides no architecture details, training objective, loss function, or hyperparameter information, making it impossible to assess whether the result is reproducible or load-bearing for the proxy conclusion.

minor comments (2)

The abstract would be strengthened by including at least one numerical alignment value (e.g., Cohen's kappa or accuracy) for the final LLM and VM results.
Notation for 'alignment' should be defined explicitly (e.g., as percentage agreement, kappa, or correlation) the first time it appears.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback, which identifies key areas for improving the rigor and transparency of our results. We will revise the manuscript to address each point by adding the requested statistics, metrics, and implementation details.

read point-by-point responses

Referee: [User Study] User study section: the central claims rest on labels from only 27 participants with no reported inter-annotator agreement statistics, cross-validation, leave-one-participant-out tests, or demographic details; this directly affects the reliability of the human-human baseline and all LLM/VM alignment numbers derived from it.

Authors: We agree that additional statistics are needed to substantiate the human-human baseline. In the revised manuscript we will report inter-annotator agreement (Fleiss' kappa), participant demographics, and results from both k-fold cross-validation and leave-one-participant-out experiments. These additions will allow readers to assess the stability of the derived alignment numbers. While the participant count is modest, the study collected multiple labels per graph to increase coverage; we will also add an explicit limitations paragraph on sample size. revision: yes
Referee: [Abstract] Abstract and results: the assertion that LLM-human alignment reaches 'human-human levels' after confidence filtering lacks any quantitative metrics, exact agreement scores, statistical significance tests, or tables comparing the values, so the claim cannot be verified from the supplied evidence.

Authors: We will revise both the abstract and the results section to include the exact agreement scores (e.g., post-filtering LLM-human agreement of XX% versus human-human agreement of YY%), a comparison table, and statistical significance tests (paired t-tests with p-values). This will make the 'human-human levels' claim directly verifiable and will be accompanied by confidence intervals. revision: yes
Referee: [VM Experiments] VM training description: the claim that 'carefully trained VMs' achieve human-comparable alignment provides no architecture details, training objective, loss function, or hyperparameter information, making it impossible to assess whether the result is reproducible or load-bearing for the proxy conclusion.

Authors: We acknowledge the omission and will expand the VM section with full architecture specifications (Vision Transformer base), training objective (binary preference classification), loss function (cross-entropy), optimizer, learning rate schedule, batch size, number of epochs, and data augmentation details. These additions will enable exact reproduction of the reported VM-human alignment figures. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper's results are empirical evaluations of LLM and VM alignment against independently collected human preference labels from a 27-participant user study. These labels function as an external benchmark rather than being derived from the models themselves. No mathematical derivation, fitted-parameter prediction, self-definitional loop, or load-bearing self-citation reduces any claim to its own inputs by construction. The reported improvements via prompt engineering and training are measured against the held-out human data, keeping the chain self-contained.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that pairwise human choices reliably capture aesthetic preferences and that LLMs/VMs can generalize from a modest number of such labels; no free parameters or new entities are introduced in the abstract.

axioms (1)

domain assumption Human aesthetic preferences for network layouts can be reliably captured through forced-choice selections among multiple candidate layouts
This assumption underpins the creation of the preference labels used both to evaluate and to train the LLM and VM proxies.

pith-pipeline@v0.9.0 · 5558 in / 1370 out tokens · 48925 ms · 2026-05-13T19:52:16.649017+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages · 2 internal anchors

[1]

of GD 2010 6502(09 2010)

[ABS10] ARGYRIOUE., BEKOSM., SYMVONISA.: Maximizing the total resolution of graphs.Proc. of GD 2010 6502(09 2010). 2 [ADLD∗22] AHMEDR., DELUCAF., DEVKOTAS., KOBOUROVS., LI M.: Multicriteria scalable graph drawing via stochastic gradient descent, (SGD)2.IEEE Transactions on Visualization and Computer Graphics 28, 6 (2022), 2388–2399. 2, 3 [BFG∗21] BEKOSM. ...

work page doi:10.1007/978-3-540-70904-6_6 2010
[2]

Springer International Symposium on Graph Drawing and Net- work Visualization(2021), pp

InProc. Springer International Symposium on Graph Drawing and Net- work Visualization(2021), pp. 375–390. 3 [Gow75] GOWERJ. C.: Generalized procrustes analysis.Psychometrika 40, 1 (1975), 33–51. 15 [GPAM∗14] GOODFELLOWI. J., POUGET-ABADIEJ., MIRZAM., XU B., WARDE-FARLEYD., OZAIRS., COURVILLEA. C., BENGIOY.: Generative adversarial nets. InProc. NeurIPS(201...

work page doi:10.1109/mcg.2018.2881501 2021
[3]

DINOv2: Learning Robust Visual Features without Supervision

URL:https://doi.org/10.1109/CVPR.2016.90,doi:10. 1109/CVPR.2016.90. 5 [JVHB14] JACOMYM., VENTURINIT., HEYMANNS., BASTIANM.: Forceatlas2, a continuous graph layout algorithm for handy network vi- sualization designed for the gephi software.PloS one 9(06 2014), e98679.doi:10.1371/journal.pone.0098679. 3, 14 [KK89] KAMADAT., KAWAIS.: An algorithm for drawing...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.1109/cvpr.2016.90 2016
[4]

DINOv3

URL:https://arxiv. org/abs/2508.10104,arXiv:2508.10104. 6 [TCG22] TIEZZIM., CIRAVEGNAG., GORIM.: Graph neural networks for graph drawing.IEEE Transactions on Neural Networks and Learning Systems(2022). 2, 3 [WHT∗25] WANGH. W., HOFFSWELLJ., THANES. M. T., BURSZTYN V. S., BEARFIELDC. X.: How aligned are human chart takeaways and llm predictions? a case stud...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.1109/tvcg.2024.3456378 2022
[5]

Is this hard?

optionally, check a box labeled“Is this hard?”if the decision was perceived as difficult or ambiguous; Time spent on the task was automatically recorded from the moment the layout grid was shown until a selection was made. To keep participants informed and engaged, the interface pro- vided visual progress feedback with a running total of the number of gra...

work page 2026
[6]

On average, the participants took 9.61 seconds per label

In total we col- lected 64,436 labels; on average, each graph is labelled 5.58 times. On average, the participants took 9.61 seconds per label. Figure 8:Distribution of layout algorithms behind the visualiza- tions chosen by human subjects as being the most pleasing We found that the visualizations from different algorithms have different probabilities of...

work page 2026