REVIEW 6 cited by
Aligning Machine and Human Visual Representations across Abstraction Levels
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Deep neural networks have achieved success across a wide range of applications, including as models of human behavior and neural representations in vision tasks. However, neural network training and human learning differ in fundamental ways, and neural networks often fail to generalize as robustly as humans do raising questions regarding the similarity of their underlying representations. What is missing for modern learning systems to exhibit more human-aligned behavior? We highlight a key misalignment between vision models and humans: whereas human conceptual knowledge is hierarchically organized from fine- to coarse-scale distinctions, model representations do not accurately capture all these levels of abstraction. To address this misalignment, we first train a teacher model to imitate human judgments, then transfer human-aligned structure from its representations to refine the representations of pretrained state-of-the-art vision foundation models via finetuning. These human-aligned models more accurately approximate human behavior and uncertainty across a wide range of similarity tasks, including a new dataset of human judgments spanning multiple levels of semantic abstractions. They also perform better on a diverse set of machine learning tasks, increasing generalization and out-of-distribution robustness. Thus, infusing neural networks with additional human knowledge yields a best-of-both-worlds representation that is both more consistent with human cognitive judgments and more practically useful, thus paving the way toward more robust, interpretable, and human-aligned artificial intelligence systems.
Forward citations
Cited by 6 Pith papers
-
Towards Robust Foundation Models for Digital Pathology
PathoROB shows that all 20 evaluated pathology foundation models encode medical center information and that lower robustness correlates with larger downstream performance drops.
-
Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering Human Perceptual Variability on Facial Expressions
Faces generated on ANN decision boundaries raise inter-individual variability in emotion labeling, and fine-tuning on those labels improves both group-level and individual-level prediction.
-
BIRD: Behavior Induction via Representation-structure Distillation
BIRD transfers aligned behavior across models with different architectures, tasks, and data by minimizing linear CKA between teacher and student representations, and three teacher representation properties explain mos...
-
Beginning with You: Perceptual-Initialization Improves Vision-Language Representation and Alignment
Perceptually initializing a CLIP vision encoder with NIGHTS triplet judgments before YFCC15M contrastive training improves zero-shot accuracy and retrieval over an identical random-start baseline.
-
Mixing Configurations for Downstream Prediction
Mixing multiple resolution clusterings of an embedding, aligned between train and test, and fused by attention, improves downstream regression and classification over single-resolution baselines.
-
A Geometric Theory of Cognition for Machine Intelligence
Cognition is re-described as Riemannian gradient flow on a manifold; an anisotropic metric makes fast intuitive and slow deliberative timescales emerge from a single equation.
Discussion (0). Sign in to comment.