Pith. sign in

REVIEW 4 major objections 5 minor 76 references

Media Content Atlas: A Pipeline to Explore and Investigate Multidimensional Media Space using Multimodal LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper presents Media Content Atlas, a multimodal-LLM pipeline that maps 1.12 million real smartphone screenshots into content-based topic clusters, searchable images, and interactive views, and reports that experts rated the topic…

desk verdict A genuinely useful pipeline paper for large-scale screen content analysis; treat the self-rated accuracy numbers as preliminary, not as validated performance. read the letter →

arxiv 2504.16323 v1 pith:ARLQUIJ7 submitted 2025-04-22 cs.HC cs.SI

Merve Cerit , Eric Zelikman , Mu-Jung Cho , Thomas N. Robinson , Byron Reeves , Nilam Ram , Nick Haber This is my paper · ORCID
classification cs.HCcs.SI
keywords mediacontentanalysismultimodallargelanguagemodelssmartphonescreenshotscontent-basedclusteringtopicmodelingimageretrievalinteractivevisualization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most media research measures screens by app name or total minutes, which hides the actual content people see. The paper tries to change that with Media Content Atlas (MCA), a pipeline in which multimodal large language models (models that read an image and produce text) describe every screenshot, embed those descriptions in a semantic space, cluster similar content, assign each cluster a topic label, and allow natural-language image retrieval inside an interactive map. The authors run MCA on 1.12 million in-situ smartphone screenshots from 112 adults over one month, and report that four expert evaluators rated the generated topic clusters 96% relevant, the per-screenshot descriptions 83% accurate, within-cluster image similarity 89%, and retrieval results 79.5% relevant. Their central claim is that this is the first pipeline to combine moment-by-moment content analysis, content-based clustering, topic modeling, image retrieval, and interactive visualization at this scale, enabling both open-ended exploration and hypothesis-driven analysis. The practical point is that the bottleneck in screen-data research has shifted from data collection to making raw screen captures interpretable.

What carries the argument

The load-bearing object is the MCA pipeline itself, a four-stage processing chain. Stage one converts each screenshot twice: a contrastive vision-language model produces a vector embedding, and a vision-language assistant writes a plain-language description of the screen. Stage two embeds those descriptions with a text-embedding model, compresses the combined vectors with nonlinear dimensionality reduction, and groups them with a hierarchical density-based clusterer that marks low-density points as noise. Stage three passes representative descriptions and keywords to a language model that writes a short topic label for each cluster. Stage four renders the projected points on an interactive map colored by topic, with search, zoom, filtering by app or participant, and hover-over metadata, so exploration replaces manual browsing. Because every component can be swapped for another model or adapted by prompting and fine-tuning, the pipeline functions as a reusable instrument rather than a single fixed analysis.

What would settle it

Recruit five media researchers who were not involved in building the pipeline, show them the same five clusters and ten images per cluster with their topic labels and descriptions, and count how many they rate relevant or accurate on the same seven-point scale; if the independent rates fall substantially below 96% and 83%, the reported evaluation does not generalize beyond the author-evaluators.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a model-agnostic chain of existing multimodal models can organize a million private screenshots into interpretable content without manual labeling. The pipeline produced 2,086 clusters from 1.12 million screenshots, set aside 232,375 (20.75%) low-density images as noise, and named clusters across many granularities, from app-specific themes such as a particular game to cross-platform themes such as ASMR, political events, and gambling. The accompanying evaluation is the evidence that this organization matches human judgment: 96% of topic labels were rated relevant or highly relevant, 83% of descriptions were accurate or highly accurate, 89% of images within clusters were similar or highly similar, and 79.5% of retrieval results were relevant or highly relevant. The paper's authors state that, to their knowledge, MCA is the first pipeline to combine moment-by-moment screen content analysis, content-based clustering, topic modeling, image retrieval, and interactive visualization at this scale.

Load-bearing premise

The pipeline's reported usefulness depends entirely on the ratings of four expert researchers who are also co-authors of the paper; if their self-evaluations are systematically too generous, the 96% relevance and 83% accuracy figures do not prove that outside researchers will find the tool useful.

Editorial extensions

If this is right

  • Screen-media researchers can study content rather than containers, distinguishing, for example, specific uses inside an app such as kitchen videos on a social platform from the app as a whole.
  • Abstract research constructs such as depression, substance abuse, or political content can be queried directly against millions of raw screenshots, with the caveat that retrieval quality varies by how cleanly the construct maps to visual and textual cues.
  • Datasets too large to view manually—the paper estimates 190 eight-hour workdays to watch its 1.12 million screenshots at five seconds each—become explorable through a zoomable map that shows large themes first and finer clusters on closer inspection.
  • Because MCA is model-agnostic, future improvements in multimodal models can be dropped into the pipeline, so the method can improve without redesign.
  • The same environment supports both inductive discovery and hypothesis-driven checks, letting researchers alternate between open exploration and targeted measurement in one private and secure interface.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test this, but the strongest check of its central utility claim is to have independent media researchers who were not involved in building MCA rate the same clusters and descriptions; replication would show the reported numbers are not an artifact of the authors' own expertise.
  • A natural extension the paper leaves implicit is to use the cluster structure itself as a quantitative phenotype—for instance, the number, spread, and temporal movement of a person's content clusters could be linked to health or behavioral outcomes, going beyond the experts' suggested use cases.
  • The retrieval failures the paper lists (blood versus ketchup, brand-specific soda queries, influencer ambiguity) suggest that a privacy-preserving synthetic benchmark of screen images would be a useful testbed for separating model capability from the prevalence of content in real datasets; the paper calls for a synthetic benchmark but does not build one.
  • Because the pipeline works on whatever images it is given, the same chain could be pointed at desktop screen recordings, video frames, or television capture; the paper does not make that claim, but nothing in its architecture limits it to smartphone screenshots.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Media Content Atlas (MCA), a pipeline for analyzing large-scale smartphone screenshot data using multimodal large language models. The pipeline embeds screenshots with CLIP, generates image descriptions with LLaVA-OneVision, clusters and topic-labels the embeddings via UMAP/HDBSCAN and Llama-2-based labeling, supports semantic image retrieval with CLIP and GTE embeddings, and provides DataMapPlot-based interactive visualizations. It is evaluated on 1.12 million screenshots from 112 adults. The central claims are that MCA supports content-based clustering at scale, that expert evaluators rated its clusters 96% relevant and its descriptions 83% accurate, and that it is more informative and useful than app-based baselines. The paper also reports 89% within-cluster similarity and 79.5% retrieval relevance, and it discusses limitations including the fact that the expert evaluators are also co-authors.

Significance. If the utility claims were well supported, MCA would be a useful methodological contribution to media research and HCI: it combines moment-by-moment content analysis, content-based clustering, semantic retrieval, and interactive exploration at a scale that manual labeling cannot reach. The paper makes code available, documents implementation details and hyperparameters, and is transparent about many model failure modes, which is a strength. However, the evidence for the central claims is narrow: four expert co-authors, five clusters, small image samples, and no inferential statistics. The scale of the dataset is a strength, but scale alone does not establish accuracy or usability. The contribution is best read as a promising systems description with a preliminary evaluation rather than a fully validated tool, and the current claims outrun the evidence.

major comments (4)
  1. [§4.2 and §6] The headline figures—96% topic-image relevance, 83% description accuracy, 89% within-cluster similarity, and 79.5% retrieval relevance—are derived from ratings by four experts who are also co-authors, a limitation the paper itself acknowledges in Section 6 ('Expert evaluation bias is another limitation, as evaluators are also co-authors due to data privacy'). No inter-rater reliability, confidence intervals, or independent ground truth are reported. Because these percentages are repeated in the abstract and introduction as primary evidence of MCA's utility, the evaluation is too fragile to support the claims as stated. Please report the exact number of judgments per metric, per-expert score distributions, and agreement statistics (e.g., Fleiss' kappa or ICC), and either add an external-rater protocol or reframe the results as preliminary self-assessment with appropriate uncertainty.
  2. [§4.2 and Appendix 'More examples from Image Retrieval'] The evaluation sample is small and partially unspecified: five randomly selected clusters (out of 2,086) with ten images each for relevance and accuracy, five clusters of 25 images for similarity, and an unreported number of retrieval queries for the retrieval ratings. The appendix itself states that 'a more comprehensive assessment requires calculating recall and precision@k to quantitatively evaluate retrieval performance,' yet the paper reports only pooled relevance percentages. Please report all denominators, the number of retrieval queries, per-cluster and per-query breakdowns, and, where feasible, precision/recall or a related retrieval metric.
  3. [§7 and Abstract] The claim that MCA is 'the first pipeline' to combine the listed functions and that it operates at 'unprecedented scale' is not supported by a systematic comparison with prior screenome and content-analysis pipelines. The related-work section describes existing OCR-based, metadata-based, and qualitative approaches but does not demonstrate that no prior system provides any of these functions at comparable scale. Please either temper the abstract's 'unprecedented scale' phrasing or add a concrete comparison table of existing tools and their capabilities relative to MCA.
  4. [§4.2 and §5] The usability and adoption claims ('highly usable,' 'more informative,' 'very likely to be adopted') rest on Likert responses from the same four co-author evaluators, with no standardized usability instrument, no full response distributions per item, and no described qualitative analysis method for the think-aloud sessions. Please report the full distribution of responses for each survey item and state how the qualitative data were analyzed, or reduce these claims to 'initial qualitative feedback from a small self-selected expert group.'
minor comments (5)
  1. [§3 and Appendix] The model name is spelled inconsistently as 'Llava-OneVision' and 'Llava-OneVision+GTE-Large'; please standardize the naming throughout.
  2. [Figures 3 and 4] The captions say 'by Cluster and Expert,' but the left panels appear to aggregate responses across experts; please clarify what the bar lengths and colors encode.
  3. [Appendix 'More Examples of the Media Content Atlas Topics'] The statement that the complete list of clusters will be shared only after privacy review materially affects reproducibility and should be stated in the main text as well.
  4. [References] Reference [23] has a formatting error: 'J, A. Pater' should be 'Jessica A. Pater.'
  5. [§5] The phrase 'Structured questions from a nonexpert' is ambiguous; please specify who the nonexpert was and what role they played.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the pipeline is assembled from external pretrained components and the acknowledged co-author evaluation is a validity limitation, not a circularity.

full rationale

The paper makes no mathematical or derivation-level claim that would reduce to its inputs by construction. The Media Content Atlas is built from externally validated components (CLIP embeddings, LLaVA-OneVision descriptions, GTE embeddings, BERTopic, UMAP, HDBSCAN) applied to 1.12 million screenshots; the reported 96% relevance and 83% description accuracy are empirical survey ratings of sampled outputs, not fitted parameters later relabeled as predictions. The pipeline's construction does not use these ratings as inputs, so there is no self-definitional or fitted-input circularity. The most salient self-referential element is acknowledged directly in Section 6: 'Expert evaluation bias is another limitation, as evaluators are also co-authors due to data privacy.' This is an honest methodological limitation about evaluator independence and small-sample stability, but it is not circular reasoning in the derivation chain: the experts' assessments do not define the pipeline, nor are they invoked as an external theorem. The novelty claim is explicitly hedged ('To our knowledge, MCA is the first pipeline to...') and is not supported by a self-citation chain. No load-bearing self-citations, imported uniqueness theorems, or ansatz-smuggling citations are present. Accordingly, the paper's central contribution is self-contained as a pipeline demonstration, and the circularity score is 0.

Assumptions & free parameters 9 free parameters · 4 assumptions · 0 invented entities

The pipeline introduces no new theoretical entities. Its central claims rest on free hyperparameters chosen by hand and on domain assumptions about the reliability of pretrained models and the validity of self-evaluation by co-author experts.

free parameters (9)
  • UMAP n_neighbors = 50
    Set to balance cluster granularity; tuned by iterative qualitative assessment.
  • UMAP min_dist = 0.05
    Chosen to ensure tighter cluster formation; tuned qualitatively.
  • UMAP n_components = 20 then 10
    Dimensionality reduction settings selected by visual and clustering validity checks.
  • UMAP negative_sample_rate = 20
    Hyperparameter set for convergence; mentioned without systematic tuning.
  • UMAP n_epochs = 1000
    Set to improve convergence at computational cost.
  • HDBSCAN min_cluster_size = 400
    Selected to ensure medium-sized clusters; iterative process balancing granularity and interpretability.
  • HDBSCAN min_samples = 20
    Defines core point density; tuned with EOM selection.
  • LLaVA max_new_tokens = 128
    Limits description length; chosen by experimentation.
  • LLaVA temperature = 0
    Deterministic decoding for reproducible descriptions.
assumptions (4)
  • domain assumption CLIP embeddings capture the visual semantic similarity of screenshots as judged by humans.
    The clustering and retrieval depend on the pretrained CLIP model's representation space aligning with human notions of content similarity; the paper does not validate this for screenshot data.
  • domain assumption LLaVA-OneVision descriptions are accurate enough to support clustering and retrieval.
    Descriptions are used for topic modeling and retrieval; the paper documents hallucinated details and misinterpretations, so this assumption is only partially satisfied.
  • domain assumption HDBSCAN clusters correspond to meaningful media content categories.
    The pipeline treats clusters as topics without comparing to hand-labeled ground truth.
  • domain assumption The four expert co-authors' ratings are a valid measure of pipeline quality.
    The paper acknowledges this is a limitation due to evaluator bias; the assumption underlies all reported accuracy percentages.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Media Content Atlas: A Pipeline to Explore and Investigate Multidimensional Media Space using Multimodal LLMs." pith.science (2026). https://pith.science/paper/ARLQUIJ7

@misc{pith2026250416323,
  author       = {Pith},
  title        = {Pith review of: Media Content Atlas: A Pipeline to Explore and Investigate Multidimensional Media Space using Multimodal LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ARLQUIJ7}},
  note         = {Machine review of arXiv:2504.16323}
}
read the original abstract

As digital media use continues to evolve and influence various aspects of life, developing flexible and scalable tools to study complex media experiences is essential. This study introduces the Media Content Atlas (MCA), a novel pipeline designed to help researchers investigate large-scale screen data beyond traditional screen-use metrics. Leveraging multimodal large language models (MLLMs), MCA enables moment-by-moment content analysis, content-based clustering, topic modeling, image retrieval, and interactive visualizations. Evaluated on 1.12 million smartphone screenshots continuously captured during screen use from 112 adults over an entire month, MCA facilitates open-ended exploration and hypothesis generation as well as hypothesis-driven investigations at an unprecedented scale. Expert evaluators underscored its usability and potential for research and intervention design, with clustering results rated 96% relevant and descriptions 83% accurate. By bridging methodological possibilities with domain-specific needs, MCA accelerates both inductive and deductive inquiry, presenting new opportunities for media and HCI research.

Figures

Figures reproduced from arXiv: 2504.16323 by the authors.

Figure 1
Figure 1. Media Content Atlas Pipeline Architecture Diagram (more information in Appendix). Preliminary Tests and Evaluation: Testing with Experts. To evaluate and iterate on our design, we structured our tests around the following key questions: • What types of clusters does this pipeline generate, and how do they vary across dimensions (e.g., app-based, UI elements (e.g., layouts), or higher-order themes (e.g., political co… view at source ↗
Figure 2
Figure 2. A screenshot of the interactive Media Content Atlas dashboard interface. 4.1 Overview of Clusters and Topic Labels Media Content Atlas generated (under default settings) 2,086 clusters from 1.12 million screenshot images, with 232,375 images (20.75%) classified as noise—data points in low-density regions that do not belong to any significant cluster. Cluster sizes varied significantly, from 24 screenshots in smaller… view at source ↗
Figure 3
Figure 3. Topic-Image Relevance(a) and Description Accuracy(b) Scores by Cluster and Expert. The bar plots on the left represent the count of responses for each randomly selected cluster, with varying bar lengths indicating the number of responses and colors representing different slider values. The boxplots with scatter points on the right illustrate the distribution of scores provided by each of the four experts. Topic-Imag… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Within-Cluster Similarity by Cluster and Expert. The bar plots on the left represent the count of responses for each randomly selected cluster, with varying bar lengths indicating the number of responses and colors representing different slider values. The boxplots wit…
Figure 5
Figure 5. Figure 5: Mean expert ratings for each query, comparing CLIP (green bars) and Llava [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 38 canonical work pages

  1. [1]

    Altheide and Christopher J

    David L. Altheide and Christopher J. Schneider. 2013. Qualitative Media Analysis . SAGE Publications, Thousand Oaks, CA. https://doi.org/10.4135/9781452270043

    work page doi:10.4135/9781452270043 2013

  2. [2]

    Bell, Catherine V

    Beth T. Bell, Catherine V. Talbot, and Nova Deighton-Smith. 2024. Following up on #fitspiration: A comparative content analysis and thematic analysis of social media content aiming to inspire fitness from 2014 and 2021. Psychology of Popular Media 13, 4: 666–676. https://doi.org/10.1037/ppm0000523

    work page doi:10.1037/ppm0000523 2024

  3. [3]

    E. K. Bozzelli. 2008. Subjective Definitions of Substance Abuse Problems: Does Age Matter? Retrieved from https://rave.ohiolink.edu/etdc/view?acc_num=miami1220005252

    work page 2008

  4. [4]

    Shyam Sundar, Thomas N

    Miriam Brinberg, Nilam Ram, Xiao Yang, Mu -Jung Cho, S. Shyam Sundar, Thomas N. Robinson, and Byron Reeves. 2021. The idiosyncrasies of everyday digital lives: Using the Human Screenome Project to study user behavior on smartphones. Computers in human behavior 114. https://doi.org/10.1016/j.chb.2020.106570

    arXiv 2021

  5. [5]

    Ricardo J. G. B. Campello, Davoud Moulavi, Arthur Zimek, and Jörg Sander. 2015. Hierarchical density estimates for data clust ering, visualization, and outlier detection. ACM transactions on knowledge discovery from data 10, 1: 1–51. https://doi.org/10.1145/2733381

    doi:10.1145/2733381 2015

  6. [6]

    Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, Tianshuo Zhou, Yue Yu, Chujie Gao, Qihui Zhang, Yi Gui, Zhen Li, Yao Wan, Pan Zhou, Jianfeng Gao, and Lichao Sun. 2024. GUI -WORLD: A dataset for GUI -oriented multimodal LLM-based agents. arXiv [cs.CV]. Retrieved from http://arxiv....

    arXiv 2024

  7. [7]

    Lee Giles

    Agnese Chiatti, Mu Jung Cho, Anupriya Gagneja, Xiao Yang, Miriam Brinberg, Katie Roehrick, Sagnik Ray Choudhury, Nilam Ram, Byron Reeves, and C. Lee Giles. 2018. Text extraction and retrieval from smartphone screenshots: building a repository for life in media. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing. https://doi.org/10.1145/3...

    arXiv 2018

  8. [8]

    Lee Giles

    Agnese Chiatti, Xiao Yang, Miriam Brinberg, Mu Jung Cho, Anupriya Gagneja, Nilam Ram, Byron Reeves, and C. Lee Giles. 2017. T ext Extraction from Smartphone Screenshots to Archive in situ Media Behavior. In Proceedings of the Knowledge Capture Conference. https://doi.org/10.1145/3148011.3154468

    work page arXiv 2017

Show all 76 references
  1. [9]

    Robinson

    Mu-Jung Cho, Byron Reeves, Nilam Ram, and Thomas N. Robinson. 2023. Balancing media selections over time: Emotional valence, informational content, and time intervals of use. Heliyon 9, 12: e22816. https://doi.org/10.1016/j.heliyon.2023.e22816

    2023 doi

  2. [10]

    Justin Cornelius and Daniel Muise. 2025. Screenlake Research Kit, maintained by the Accelerator at Princeton University [Software] . Retrieved from https://github.com/Screenlake/screenlake-research-kit

    2025

  3. [11]

    Cuthbert

    Bruce N. Cuthbert. 2022. Research Domain Criteria (RDoC): Progress and Potential. Current directions in psychological science 31, 2: 107 –114. https://doi.org/10.1177/09637214211051363

    2022 doi

  4. [12]

    Dawei Dai, Yuanhui Zhang, Long Xu, Qianlan Yang, Xiaojing Shen, Shuyin Xia, and Guoyin Wang. 2024. PA -LLaVA: A large language -vision assistant for human pathology image understanding. In 2024 IEEE International Conference on Bioinformatics and Biomedicine (BIBM) , 3138 –3143...

    2024

  5. [13]

    Bill Ferster. 2023. Interactive visualization: Insight through inquiry . MIT Press, London, England. Retrieved from https://books.google.com/books?hl=en&lr=&id=rG6vEAAAQBAJ&oi=fnd&pg=PR9&dq=discovery+interactive+visualization+large+scale+data&ots=tSncu5p P1Q&sig=k0ptP8nDE8TOPB...

    2023

  6. [14]

    Flanagan

    O. Flanagan. 2025. What is it like to be an addict?: Understanding substance abuse . Oxford University Press, London, England. Retrieved from https://books.google.com/books?hl=en&lr=&id=UREwEQAAQBAJ&oi=fnd&pg=PP1&dq=how+do+you+define+substance+abuse&ots=xvc9K6q- uB&sig=j3tajAN...

    2025

  7. [15]

    Gillan and Robb B

    Claire M. Gillan and Robb B. Rutledge. 2021. Smartphones and the neuroscience of mental health. Annual review of neuroscience 44, 1: 129 –151. https://doi.org/10.1146/annurev-neuro-101220-014053 9

    2021 doi

  8. [16]

    Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class -based TF -IDF procedure. arXiv [cs.CL] . Retrieved from http://arxiv.org/abs/2203.05794

    2022 arXiv

  9. [17]

    Smeaton, and Aiden R

    Cathal Gurrin, Alan F. Smeaton, and Aiden R. Doherty. 2014. LifeLogging: Personal big data. Foundations and Trends® in Information Retrieval 8, 1: 1–125. https://doi.org/10.1561/1500000033

    2014 doi

  10. [18]

    Jeff Hancock, Sunny Xun Liu, Mufan Luo, and Hannah Mieczkowski. 2022. Psychological Well-Being and Social Media Use: A Meta-Analysis of Associations between Social Media Use and Depression, Anxiety, Loneliness, Eudaimonic, Hedonic and Social Well-Being. https://doi.org/10.2139...

    2022 doi

  11. [19]

    Heer and D

    J. Heer and D. Boyd. 2005. Vizster: visualizing online social networks. In IEEE Symposium on Information Visualization, 2005. INFOVIS 2005 , 32 –39. https://doi.org/10.1109/infvis.2005.1532126

    2005

  12. [20]

    Clayton, and Justin Robert Keene

    Richard Huskey, Shelby Wilcox, Russell B. Clayton, and Justin Robert Keene. 2020. The limited capacity model of motivated med iated message processing: meta-analytically summarizing two decades of research. Annals of the International Communication Association 44, 4: 322 –349....

    2020

  13. [21]

    Thomas R. Insel. 2018. Digital phenotyping: a global tool for psychiatry. World psychiatry: official journal of the World Psychiatric Association (WPA) 17, 3: 276–277. https://doi.org/10.1002/wps.20550

    2018 doi

  14. [22]

    Jacobucci, B

    R. Jacobucci, B. Ammerman, and Nilam Ram. 2024. Examining passively collected smartphone -based data in the days prior to psychiatric hospitalization for a suicidal crisis: Comparative case analysis. JMIR formative research 8. https://doi.org/10.2196/55999

    2024 doi

  15. [23]

    Pater, Oliver Haimson, Nazanin Andalibi, and Elizabeth D

    J, A. Pater, Oliver Haimson, Nazanin Andalibi, and Elizabeth D. Mynatt. 2016. Characterizing the Presentation of Eating Disorders Online. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, San Francisco, CA USA, 1185–1200

    2016

  16. [24]

    screen time

    Linda K Kaye, Amy Orben, David A Ellis, Simon C Hunter, and Stephen Houghton. 2020. The conceptual and methodological mayhem of “screen time.” International journal of environmental research and public health 17, 10: 3661. https://doi.org/10.3390/ijerph17103661

    2020 doi

  17. [25]

    Leaning, Elena Idda, Faraz Hussain, Emma Ning, Tory A

    Loran Knol, Anisha Nagpal, Imogen E. Leaning, Elena Idda, Faraz Hussain, Emma Ning, Tory A. Eisenlohr-Moul, Christian F. Beckmann, Andre F. Marquand, and Alex Leow. 2024. Smartphone keyboard dynamics predict affect in suicidal ideation. npj digital medicine 7, 1: 54. https://d...

    2024 doi

  18. [26]

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po -Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. arXiv [cs.LG]. Retrieved from http://arxiv....

    2024 arXiv

  19. [27]

    Adam D. I. Kramer, Jamie E. Guillory, and Jeffrey T. Hancock. 2014. Experimental evidence of massive -scale emotional contagion through social networks. Proceedings of the National Academy of Sciences of the United States of America 111, 24: 8788–8790. https://doi.org/10.1073/...

    2014 doi

  20. [28]

    Gunther Kress and Jeff Bezemer. 2023. Multimodal discourse analysis. In The Routledge Handbook of Discourse Analysis . Routledge, London, 139 –155. https://doi.org/10.4324/9781003035244-12

    2023 doi

  21. [29]

    Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. SAGE Publications, Thousand Oaks, CA

    2018

  22. [30]

    Hamilton, Nilam Ram, Katherine Roehrick, and Byron Reeves

    Jihye Lee, James T. Hamilton, Nilam Ram, Katherine Roehrick, and Byron Reeves. 2023. The psychology of poverty and life online: natural experiments on the effects of smartphone payday loan ads on psychological stress. Information, Communication and Society: 1–22. https://doi.o...

    2023

  23. [31]

    Gyorda, George D

    Damien Lekkas, Joseph A. Gyorda, George D. Price, and Nicholas C. Jacobson. 2023. Depression deconstructed: Wearables and passive digital phenotyping for analyzing individual symptoms. Behaviour research and therapy 168, 104382: 104382. https://doi.org/10.1016/j.brat.2023.104382

    2023

  24. [32]

    Alex Leow, Jonathan Stange, John Zulueta, Olusola Ajilore, Faraz Hussain, Andrea Piscitello, Kelly Ryan, Jennifer Duffecy, Sc ott Langenecker, Peter Nelson, and Melvin McInnis. 2019. 247. BiAffect: Passive monitoring of psychomotor activity in mood disorders using mobile keyst...

    2019 doi

  25. [33]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. 2024. LLaVA-OneVision: Easy Visual Task Transfer. arXiv [cs.CV]. Retrieved from http://arxiv.org/abs/2408.03326

    2024 arXiv

  26. [34]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng G ao. 2023. LLaVA-Med: Training a Large Language -and-Vision Assistant for BioMedicine in one day. Neural Information Processing Systems abs/2306.00890: ...

    2023 doi arXiv

  27. [35]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023. Improved baselines with visual instruction tuning. arXiv [cs.CV] . Retrieved from http://arxiv.org/abs/2310.03744

    2023 arXiv

  28. [36]

    Qian Liu, Zequan Zheng, Jiabin Zheng, Qiuyi Chen, Guan Liu, Sihan Chen, Bojia Chu, Hongyu Zhu, Babatunde Akinwunmi, Jian Huan g, Casper J. P. Zhang, and Wai-Kit Ming. 2020. Health communication through news media during the early stage of the COVID-19 outbreak in China: Digita...

    2020 doi

  29. [37]

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv [cs.CL]. Retrieved from http://arxiv.org/abs/2308.03281

    2023 arXiv

  30. [38]

    Leland McInnes. 2024. DataMapPlot. Github. Retrieved from https://github.com/TutteInstitute/datamapplot

    2024

  31. [39]

    Leland McInnes, John Healy, and S. Astels. 2017. hdbscan: Hierarchical density based clustering. Journal of open source software 2: 205. https://doi.org/10.21105/JOSS.00205

    2017 doi

  32. [40]

    Leland McInnes, John Healy, and James Melville. 2018. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduc tion. arXiv [stat.ML]. Retrieved from http://arxiv.org/abs/1802.03426

    2018 arXiv

  33. [41]

    Palumbo, Colin G

    Giorgia Michelini, Isabella M. Palumbo, Colin G. DeYoung, Robert D. Latzman, and Roman Kotov. 2021. Linking RDoC and HiTOP: A new interface for advancing psychiatric nosology and neuroscience. Clinical psychology review 86, 102025: 102025. https://doi.org/10.1016/j.cpr.2021.102025

    2021

  34. [42]

    Morgan, Chareen Snelson, and Patt Elison -Bowers

    Elizabeth M. Morgan, Chareen Snelson, and Patt Elison -Bowers. 2010. Image and video disclosure of substance use on social media websites. Computers in human behavior 26, 6: 1405–1411. https://doi.org/10.1016/j.chb.2010.04.017

    2010 doi

  35. [43]

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022. MTEB: Massive Text Embedding Benchmark. arXiv [cs.CL] . Retrieved from http://arxiv.org/abs/2210.07316

    2022 arXiv

  36. [44]

    Hongyan Mu, Jianzhou Chen, Wenjie Huang, Gui Huang, Meiying Deng, Shimiao Hong, Peng Ai, Chuan Gao, and Huangkai Zhou. 2024. OmicShare tools: A zero-code interactive online platform for biological data analysis and visualization. iMeta 3, 5: e228. https://doi.org/10.1002/imt2.228

    2024 doi

  37. [45]

    O’Halloran

    Kay L. O’Halloran. 2011. Multimodal discourse analysis. The Bloomsbury handbook of discourse analysis : 249 –282. Retrieved from https://www.torrossa.com/gs/resourceProxy?an=5213037&publisher=FZ0661#page=266

    2011

  38. [46]

    O’Halloran, Gautam Pal, and Minhao Jin

    Kay L. O’Halloran, Gautam Pal, and Minhao Jin. 2021. Multimodal approach to analysing big social and news media data. Discourse Context & Media 40, 100467: 100467. https://doi.org/10.1016/j.dcm.2021.100467

    2021

  39. [47]

    Amy Orben, Adrian Meier, Tim Dalgleish, and Sarah -Jayne Blakemore. 2024. Mechanisms linking social media use to adolescent mental health vulnerability. 10 Nature reviews psychology 3, 6: 407–423. https://doi.org/10.1038/s44159-024-00307-y

    2024 doi

  40. [48]

    S. Papert. 1993. The children’s machine: rethinking school in the age of the computer. New York. https://doi.org/10.5860/choice.31-1648

    1993 doi

  41. [49]

    Pater, Lauren E

    Jessica A. Pater, Lauren E. Reining, Andrew D. Miller, Tammy Toscos, and Elizabeth D. Mynatt. 2019. Notjustgirls: Exploring m ale-related eating disordered content across social media platforms. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems . ...

    2019

  42. [50]

    Pettersen, Thomas D

    Eric F. Pettersen, Thomas D. Goddard, Conrad C. Huang, Elaine C. Meng, Gregory S. Couch, Tristan I. Croll, John H. Morris, and Thomas E. Ferrin. 2021. UCSF ChimeraX: Structure visualization for researchers, educators, and developers. Protein science: a publication of the Prote...

    2021 doi

  43. [51]

    Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and I

    Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and I. Sutskever. 2021. Learning transferable visual models from natural language supervision. International Confer...

    2021

  44. [52]

    Robinson

    Nilam Ram, Xiao Yang, Mu-Jung Cho, Miriam Brinberg, Fiona Muirhead, Byron Reeves, and Thomas N. Robinson. 2020. Screenomics: A New Approach for Observing and Studying Individuals’ Digital Lives. Journal of adolescent research 35, 1: 16–50. https://doi.org/10.1177/0743558419883362

    2020 doi

  45. [53]

    Robinson, James J

    Byron Reeves, Nilam Ram, Thomas N. Robinson, James J. Cummings, C. Lee Giles, Jennifer Pan, Agnese Chiatti, M. J. Cho, Katie Roehrick, Xiao Yang, Anupriya Gagneja, Miriam Brinberg, Daniel Muise, Yingdan Lu, Mufan Luo, Andrew Fitzgerald, and Leo Yeykelis. 2021. Screenomics: A F...

    2021

  46. [54]

    Byron Reeves, Thomas Robinson, and Nilam Ram. 2020. Time for the Human Screenome Project. Nature 577, 7790: 314–317. https://doi.org/10.1038/d41586- 020-00032-5

    2020 doi

  47. [55]

    Cummings

    Byron Reeves, Leo Yeykelis, and James J. Cummings. 2016. The Use of Media in Media Psychology. Media psychology 19, 1: 49 –71. https://doi.org/10.1080/15213269.2015.1030083

    2016

  48. [56]

    Ralf Schmälzle and Richard Huskey. 2023. Integrating media content analysis, reception analysis, and media effects studies. Frontiers in neuroscience 17: 1155750. https://doi.org/10.3389/fnins.2023.1155750

    2023

  49. [57]

    Frank Serafini and Stephanie F. Reid. 2023. Multimodal content analysis: expanding analytical approaches to content analysis. Visual communication 22, 4: 623–

    2023

  50. [58]

    Siddhant Bikram Shah, Shuvam Shiwakoti, Maheep Chaudhary, and Haohan Wang. 2024. MemeCLIP: Leveraging CLIP representations fo r multimodal meme classification. arXiv [cs.LG]. Retrieved from http://arxiv.org/abs/2409.14703

    2024 arXiv

  51. [59]

    Ben Shneiderman. 2000. Creating creativity: user interfaces for supporting innovation. ACM transactions on computer -human interaction: a publication of the Association for Computing Machinery 7, 1: 114–138. https://doi.org/10.1145/344949.345077

    2000

  52. [60]

    Ben Shneiderman. 2007. Creativity support tools: accelerating discovery and innovation. Communications of the ACM 50, 12: 20 –32. https://doi.org/10.1145/1323688.1323689

    2007

  53. [61]

    Shneiderman and C

    B. Shneiderman and C. Plaisant. 2010. Designing the user interface: strategies for effective human -computer interaction. Retrieved from https://www.academia.edu/download/41860700/UE-_Notes_7th_Feb_2015.pdf

    2010

  54. [62]

    Shashank Sheshar Singh, Samya Muhuri, Shivansh Mishra, Divya Srivastava, Harish Kumar Shakya, and Neeraj Kumar. 2024. Social network analysis: A survey on process, tools, and application. ACM computing surveys 56, 8: 1–39. https://doi.org/10.1145/3648470

    2024 doi

  55. [63]

    Snowflake Inc. 2019. Streamlit - Open-source app framework for Machine Learning and Data Science projects. Retrieved from https://streamlit.io

    2019

  56. [64]

    Songyan Teng, Simon D’Alfonso, and Vassilis Kostakos. 2024. A Tool for Capturing Smartphone Screen Text. In Proceedings of the CHI Conference on Human Factors in Computing Systems, 1–24. https://doi.org/10.1145/3613904.3642347

    2024

  57. [65]

    Larissa Terán, Heather Gahler, Jennifer Stevens Aubrey, and Kun Yan. 2024. #Grab, #touch, #drink: A content analysis of colle ge party culture in Instagram fraternity pictures. Psychology of Popular Media 13, 3: 495–500. https://doi.org/10.1037/ppm0000468

    2024 doi

  58. [66]

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? Exploring the visual sho rtcomings of multimodal LLMs. arXiv [cs.CV]. Retrieved from http://arxiv.org/abs/2401.06209

    2024 arXiv

  59. [67]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Praj jwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, ...

    2023 arXiv

  60. [68]

    Twenge, Gabrielle N

    Jean M. Twenge, Gabrielle N. Martin, and Brian H. Spitzberg. 2019. Trends in U.S. Adolescents’ media use, 1976–2016: The rise of digital media, the decline of TV, and the (near) demise of print. Psychology of popular media culture 8, 4: 329–345. https://doi.org/10.1037/ppm0000203

    2019 doi

  61. [69]

    Stange, Jennifer Duffecy, Faraz Hussain, Andrea Piscitello, John Bark, Scott A

    Claudia Vesel, Homa Rashidisabet, John Zulueta, Jonathan P. Stange, Jennifer Duffecy, Faraz Hussain, Andrea Piscitello, John Bark, Scott A. Langenecker, Shannon Young, Erin Mounts, Larsson Omberg, Peter C. Nelson, Raeanne C. Moore, Dave Koziol, Keith Bourne, Casey C. Bennett, ...

    2020

  62. [70]

    Andrew Z. H. Yee, Ryan Yu, Sun Sun Lim, Kwan Hui Lim, Tien Tuan Anh Dinh, Lionell Loh, Andre Hadianto, and Miguel Quizon. 2023. ScreenLife Capture: An open -source and user -friendly framework for collecting screenomes from Android smartphones. Behavior research methods 55, 8:...

    2023 doi

  63. [71]

    Shamma, and Elizabeth F

    Jude Yew, David A. Shamma, and Elizabeth F. Churchill. 2011. Knowing funny: genre perception and categorization in social vid eo sharing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/1978942.1978984

    2011

  64. [72]

    Umut Yildirim, Rohan Dutta, Burak Yildirim, and Atharva Vaidya. 2024. Experimentation in Content Moderation using RWKV. arXiv [cs.CL]. Retrieved from http://arxiv.org/abs/2409.03939

    2024 arXiv

  65. [73]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal Large Language M odels. arXiv [cs.CV]. Retrieved from http://arxiv.org/abs/2306.13549

    2023 arXiv

  66. [74]

    Lin Yue, Weitong Chen, Xue Li, Wanli Zuo, and Minghao Yin. 2019. A survey of sentiment analysis in social media. Knowledge and information systems 60, 2: 617–663. https://doi.org/10.1007/s10115-018-1236-4 11 A APPENDICES Smartphone Screen Data - Participant Demographics The da...

    2019 doi

  67. [76]

    Different prompts were explored, with

    with high visual question-answering capabilities across single-image, multi-image, and video scenarios. Specifically, the llava- onevision-qwen2-7b-ov-hf model is used to generate an image description of each screenshot, with the following hyperparameters: a max new token of 1...

  68. [649]

    https://doi.org/10.1177/1470357219864133

    doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.