Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks

Deb Roy; Hope Schroeder; Jad Kabbara

REVIEW 4 major objections 5 minor 18 references

Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Human-reviewed LLM labels still skew "ground truth" and inflate reported model performance, a 410-annotator experiment shows.

desk verdict Strong empirical evidence that LLM suggestions shift crowd labels and inflate reported F1, though the F1 inflation is partly mechanical and the crowd baseline is unvalidated. read the letter →

arxiv 2507.15821 v1 pith:R2YS4VJ5 submitted 2025-07-21 cs.CY

Hope Schroeder , Deb Roy , Jad Kabbara This is my paper · ORCID

classification cs.CY

keywords LLM-assistedannotationsubjectivegroundtruthanchoringbiashuman-in-the-loopcrowdmodelevaluationqualitativecoding

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether the common practice of having people review LLM-generated annotations protects subjective labeling from AI bias. In a pre-registered experiment with 410 crowd annotators and more than 7,000 annotations across two community-dialogue datasets, it finds that review does not protect the labels: annotators strongly adopted GPT-4 and Llama suggestions, even when the suggestions were disclosed as coming from AI. The label distribution shifted toward the models, and when those human-reviewed, LLM-assisted labels were used as ground truth, reported model $F_1$ scores jumped by about 0.3 for both models. The authors conclude that evaluation of subjective NLP tasks against this kind of "human-in-the-loop" gold data can substantially overstate model performance, and that the same contamination affects downstream social-science analyses that use the labels.

What carries the argument

The machinery is a controlled crowd-annotation experiment with three escalating forms of LLM assistance—plain text suggestions, text suggestions labeled as AI, and pre-highlighted suggested labels—compared against a no-assistance control, all run through the open-source Potato annotation interface. The load-bearing metric is the intersection ratio between the set of labels the LLM suggested and the set approved by a crowd of five annotators at thresholds of 3, 4, or 5 agreeing votes, together with per-label $F_1$ scores computed against either unassisted or assisted crowd ground truth. This design isolates the effect of the suggestion itself: any difference between conditions is attributable to exposure to the LLM's proposed labels.

What would settle it

Have a panel of domain experts, blind to the experimental conditions, label the same 200 NYC quotes with the same codebook and compute GPT-4's $F_1$ against their consensus. If that $F_1$ is no higher than the .47 obtained against unassisted crowd labels—or if the assisted crowd labels resemble the expert gold no better than the unassisted labels do—the claimed ground-truth inflation would fail to replicate against a stronger reference.

Watch

Extended reading notes

Core claim

The central claim, stated by the authors' Section 5.4 heading, is that using human-reviewed, LLM-assisted labels as ground truth significantly inflates reported model performance. GPT-4's weighted $F_1$ across the 20-label NYC codebook was .47 against unassisted 3/5-majority crowd labels but .79 against labels produced by annotators who had reviewed GPT-4's suggestions; Llama went from .44 to .79 against Llama-assisted labels. This happened despite every suggestion being individually vetted by five crowdworkers whose votes were aggregated by majority. The authors trace the mechanism to suggestion uptake: overlap between the crowd's labels and the LLM's suggested labels rose from 40 percent in the unassisted baseline to 81-87 percent with text suggestions, and the shift also altered substantive label-prevalence findings, such as a five-fold increase in the frequency of one intrinsic-motivation label.

Load-bearing premise

The load-bearing premise is that a 3/5 majority vote among five paid crowdworkers in the unassisted condition is a valid, unbiased reference for "true" labels on these subjective tasks, against which the assisted conditions are measured.

Editorial extensions

If this is right

Gold data built by having humans review LLM suggestions is biased toward the suggesting model, so any benchmark built on it will overstate that model's skill.
Switching the ground truth from unassisted crowd labels to assisted ones raised both models' weighted $F_1$ by roughly 0.3, with the largest label-level jump from .28 to .82.
Substantive conclusions change as well: one intrinsic-motivation label appeared in only 4 quotes in the unassisted baseline but in 19 quotes when annotators saw text suggestions.
Stronger visual presentation increases uptake: pre-highlighting suggestions raised full five-annotator consensus from 8 to 38 percent.
Improved inter-rater agreement under assistance can reflect convergence to a single LLM's viewpoint rather than more reliable independent judgment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If assisted-gold contamination feeds training data, iterative human-LLM annotation loops could drift further from unaided human judgment over successive rounds, a dynamic the paper does not simulate.
Because disclosing that suggestions came from AI barely changed uptake, interventions such as asking annotators to commit to labels before seeing suggestions, or showing model confidence, are testable next steps.
The paper's limitation section anticipates weaker anchoring among domain experts, so an extension could check whether the effect is a property of the interface rather than of worker inexperience.
The overlap metric divides by the number of AI-suggested labels, so conditions where the model proposes few labels could mechanically raise the ratio; a re-analysis using intersection-over-union would offer a more conservative estimate.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The headline: this paper gives us the strongest direct measurement yet that 'just put a human in the loop' does not neutralize LLM influence in subjective annotation. The label-distribution shift is real and replicated: with a 3/5 majority vote, overlap with GPT-4 suggestions goes from 40% in the unassisted control to 81-87% when suggestions are shown, and the shift is large enough to move a rare code from 4 to 19 instances. That is the solid core.

What's new: prior work showed analysts anchor on LLM topic lists, but this paper quantifies the magnitude of label shift and downstream F1 inflation in a crowd pipeline, across three interface conditions, two models, and two datasets, and releases the data. The design is careful—pre-registered, attention checks, Bonferroni corrections, a partial replication on a second corpus. The paper is also honest about its own limits: it flags the time-comparison confound, notes that using model-influenced labels to evaluate the model is not standard, and confines the main claim to crowdworkers.

The soft spots are real but not fatal. The F1 inflation (+.32/+.35) is partly mechanical: the assisted gold labels were created by annotators who saw the exact model suggestions being scored, so some gain is self-agreement. The paper acknowledges this in the conclusion, but the Section 5.4 heading overstates it as a standalone finding. Also, the unassisted 3/5 majority vote is the reference, yet low-to-medium Krippendorff's alpha means that baseline is noisy; without external expert or community-worker validation, part of the F1 gain could reflect reduced noise rather than bias. The label shift itself is direct and does not depend on that baseline. There is also no human-suggestion control, so the specific role of LLM suggestions versus any suggestion is not isolated. Minor issues: no confidence intervals on the overlap metric or F1 differences, the RTFC replication does not cover the F1 analysis, and the abstract's confidence claim is slightly broader than the RTFC result.

Bottom line: this paper deserves a serious referee. The core phenomenon is strong, the contribution is practically important, and the overstatements are fixable. I'd bring it to a reading group and expect to cite it when discussing LLM-assisted gold data. Recommend sending to peer review with expectations of major-but-clear revisions.

Referee Report

4 major / 5 minor

Summary. The paper reports a preregistered between-subjects crowdsourcing experiment in which 410 Prolific workers annotated quotes from two community conversation datasets (NYC and RTFC) using complex, subjective codebooks. Three LLM-assistance formats (text suggestions, text suggestions with an AI disclosure, and pre-highlighted labels) are compared against a no-assistance control, using GPT-4 suggestions on the main NYC data and Llama 3.1 70B as a replication, plus a GPT-4 replication on the RTFC data. The authors find that assistance did not reduce annotation time, increased self-reported confidence and understanding, led annotators to take up LLM suggestions strongly (raising overlap between crowd labels and LLM suggestions), and, when assisted crowd labels are aggregated into a 3/5 majority ground truth, raised the evaluated F1 of the suggestion-generating model from .47 to .79 for GPT-4 and from .44 to .79 for Llama. They conclude that human-reviewed, LLM-assisted labels can inflate reported model performance and homogenize label distributions in subjective tasks.

Significance. If the central claim holds, this is a valuable and timely result for NLP evaluation and computational social science. It provides empirical evidence that the common practice of having crowdworkers 'review' LLM suggestions can contaminate the gold labels used for evaluation, even when every label is individually vetted and aggregated by majority vote. The paper's strengths include a preregistered design, a large annotator sample, multiple assistance presentations, two models, two datasets, and a planned dataset release. The label-distribution shift and suggestion-uptake findings are robust and replicated. The main risk is that the strongest normative claim, that F1 is 'inflated' from .47 to .79, depends on the unassisted crowd majority being a valid gold standard, which the paper does not independently establish.

major comments (4)

[Section 5.4, Table 2] The headline claim that human-reviewed, LLM-assisted labels 'significantly inflate' reported F1 from .47 to .79 treats the unassisted 3/5 majority vote as the reference gold standard. The paper reports only low-to-medium Krippendorff's alpha for these codebooks (Section 4) and provides no external validation, such as expert annotations on a subset, for this baseline. If the unassisted crowd labels are noisier than the assisted labels, part of the F1 gain could reflect reduced measurement noise rather than bias toward the LLM's construct. Please either validate the baseline against an external reference or reframe the claim as 'reported F1 depends on the annotation protocol' and explicitly discuss the baseline-noise alternative.
[Section 5.1, Table 1] The time comparison is confounded: annotators in all assistance conditions answered an additional per-question rating of suggestion helpfulness, while annotators in the control condition did not. The paper acknowledges this in Section 5.1 but still presents 'did not make them faster' as a headline finding. Please report a time comparison that excludes or controls for the extra rating question, or state the finding as conditional on the presence of that question.
[Section 4, Conditions 1-3] All non-control conditions present LLM-generated suggestions, so the experiment cannot distinguish anchoring on LLM-specific content from anchoring on any plausible suggestion. The AI-disclosure condition tests whether naming the source matters, but it does not provide a non-LLM suggestion baseline. This limits the strength of the 'LLM' framing in the title and discussion; the data show that presenting suggestions changes labels, not that LLM suggestions are uniquely influential. Please add an explicit limitation or, ideally, a control condition with non-LLM suggestions.
[Section 5.4] The F1 increases of +.32 and +.35 are reported as averages across 20 labels, but no confidence intervals or paired statistical tests are provided for these label-level changes. The heading states that reported performance 'significantly inflates,' yet the only significance tests reported in this section concern the overlap metric in Section 5.3. Because label frequencies vary widely (Table 2), a paired test over label-level F1 scores is needed to support the word 'significantly' for the F1 claim.

minor comments (5)

[Conclusion] The phrase 'as much as +x.56' should be '+.56'.
[Section 5.3] The text reads 'at least 3 (of the 53) annotators'; the '53' appears to be a formatting artifact and should read 'of the 5'.
[Section 4] The sentence 'we replicated Condition 1, text-based label suggestions, with labels created by by Llama 3.1 70B' contains a duplicated 'by'.
[Abstract and Methods] The study is described as preregistered, but no registration identifier or repository link is provided. Please add the preregistration URL.
[Figure 1] The overlap figure would benefit from confidence intervals or error bars, since the paper reports significant differences but the figure shows only point estimates.

Circularity Check

0 steps flagged · score 0.0 of 10

The paper's central finding is an empirical measurement with an independent control baseline; no load-bearing argument reduces to its inputs or to a self-citation.

full rationale

The headline claim that LLM-assisted ground truth inflates reported F1 is not derived from the definition of the assisted labels alone. The paper measures, rather than assumes, that annotators uptake suggestions (Section 5.3 reports that overlap with LLM suggestions rises from 40% at the unassisted 3/5 crowd threshold to 81-87% with assistance), and then recomputes a standard F1 metric against two separately collected crowd aggregates: an unassisted control and an assisted condition. The +.32/+.35 F1 increases are quantities computed from the observed label distributions, not fitted parameters dressed as predictions; the comparison against the control condition is the external reference that makes the finding non-tautological. The paper's own conclusion, 'Obviously, using labels influenced by the model to evaluate the model is not standard or advisable in classic evaluation paradigms,' explicitly treats the evaluated setting as the object of study, not as an asserted theorem. Self-citations appear only as corpus provenance (Schroeder et al. 2024) and related-work context (Overney et al. 2024; Schroeder et al. 2025); none carries a load-bearing mathematical or uniqueness argument. Concerns about the unvalidated 3/5 majority baseline or low inter-annotator agreement are threats to construct validity, not circular derivation, and no quoted reduction shows Eq. X = Eq. Y by construction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted parameters: the 3/5 threshold is a pre-specified aggregation rule, and the helpfulness rating conversion is a coding choice. The main axioms are about the validity of majority-vote ground truth and annotator representativeness.

assumptions (4)

domain assumption Majority vote (3/5) over crowd annotations approximates ground truth for subjective labels.
Used throughout Section 4 and Section 5.4 to define ground truth and compute F1. If this is invalid, the F1 inflation could be an artifact of a noisy baseline.
domain assumption Paid Prolific crowdworkers with 95-100% approval, English as first language, and some college education are a representative annotation workforce.
Recruitment criteria in Appendix G; the paper generalizes findings to 'annotation' practices.
domain assumption Annotators answer honestly on self-reported measures.
Section 4.1 instructs them that answers will not affect performance; no external validation is provided.
domain assumption The codebooks and quote sampling are unbiased for the comparison.
Spanish quotes were removed, and samples of 200 quotes per corpus were used. If exclusion correlates with labels, effects could be biased.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks." pith.science (2026). https://pith.science/paper/R2YS4VJ5

@misc{pith2026250715821,
  author       = {Pith},
  title        = {Pith review of: Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R2YS4VJ5}},
  note         = {Machine review of arXiv:2507.15821}
}

read the original abstract

LLM use in annotation is becoming widespread, and given LLMs' overall promising performance and speed, simply "reviewing" LLM annotations in interpretive tasks can be tempting. In subjective annotation tasks with multiple plausible answers, reviewing LLM outputs can change the label distribution, impacting both the evaluation of LLM performance, and analysis using these labels in a social science task downstream. We conducted a pre-registered experiment with 410 unique annotators and over 7,000 annotations testing three AI assistance conditions against controls, using two models, and two datasets. We find that presenting crowdworkers with LLM-generated annotation suggestions did not make them faster, but did improve their self-reported confidence in the task. More importantly, annotators strongly took the LLM suggestions, significantly changing the label distribution compared to the baseline. When these labels created with LLM assistance are used to evaluate LLM performance, reported model performance significantly increases. We believe our work underlines the importance of understanding the impact of LLM-assisted annotation on subjective, qualitative tasks, on the creation of gold data for training and testing, and on the evaluation of NLP systems on subjective tasks.

Figures

Figures reproduced from arXiv: 2507.15821 by the authors.

**Figure 2.** Time (number of seconds) spent by annotators on the task, for Control and Conditions 1-3 of NYC [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

**Figure 3.** Four self-reported measures across conditions. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Condition 1: Text-based label suggestions in interface, with example and label suggestions from NYC corpus. Interactive tooltips gave extended code definitions, and the full codebook was linked in the header of the annotation task. Annotators viewed one additional ques…

**Figure 5.** Figure 5: Condition 3: Pre-highlighted label suggestions in interface, with example and label suggestions from NYC corpus showing highlighted suggestions included "Health Care Providers" and "Health Education & Support". Interactive tooltip with code definition for a different c…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 17 canonical work pages

[1]

My church helped me find alternative childcare during the pandemic

External motivations (Theme) • Civic Organizations: The speaker mentions civic organizations like non-profits, churches, NGOs, and community clubs, associations as a factor in health decisions – Example: "My church helped me find alternative childcare during the pandemic." • Family & Friends: Mentions family and friends as a factor in health decisions – E...

work page
[2]

My church helped me find alternative childcare during the pandemic

Intrinsic Motivations (Theme) • Fear of Virus: Mentions not wanting to become ill from COVID as a factor in health decisions – Example: "My church helped me find alternative childcare during the pandemic." • Getting Back to Normal: Mentions a desire for a return to activities and social routines as a factor in health decisions – Example: "I thought you kn...

work page
[3]

A community health person came to my school and explained what was going on in the pandemic and the latest research on masking. So that’s when we started masking

Role of Community Organizations (Theme) • Health Education & Support: Mentions that community health educators were a factor in health decisions – Example: "A community health person came to my school and explained what was going on in the pandemic and the latest research on masking. So that’s when we started masking." • Support Incentives: Mentions that ...

work page
[4]

We would have been so lost without the food bank that restocked each week

Resources (Theme) • Significant Impact Resources: Mentions that impact resources like food, financial assistance, rent moratorium, student loan suspension, employment helped during COVID – Example: "We would have been so lost without the food bank that restocked each week." • Unmet community needs: Mentions that a resource is needed – Example: "We just ne...

work page 2020
[5]

I just couldn’t get over how scary it was that my sister had this reaction to the vaccine. I know it could happen to me. So no I did not go through with the vaccine

Vaccine Hesitancy (Theme) • Did not vaccinate: Mentions that the speaker did not choose to vaccinate – Example: "I just couldn’t get over how scary it was that my sister had this reaction to the vaccine. I know it could happen to me. So no I did not go through with the vaccine." • Mistrust or Skepticism: Mentions that the speaker has/had mistrust or skept...

work page
[6]

Helping at my church made me feel like I was making a difference even though the world was going crazy

Personal COVID-19 Experience (Theme) • Resilience, Connection, & Hope: Mentions agency, control, or feeling empowered during the pandemic period – Example: "Helping at my church made me feel like I was making a difference even though the world was going crazy." • Stress, Fear, & Uncertainty: Mentions stress, fear, or uncertainty during the pandemic period...

work page
[7]

Talking about this has made me remember how hard that period was for our family

Future Visions & Takeaways • Conversation Reflections: Mentions reflections on the conversation – Example: "Talking about this has made me remember how hard that period was for our family." • Post-pandemic future: Mentions a future vision for their life or community after the pandemic – Example: "I just can’t wait for the schools to go back to normal and ...

work page
[8]

Economic Opportunity (Theme) • Jobs: References to a person’s ability to provide for themselves and their families. Can include statements about working multiple jobs; working in a particular industry; facing unemployment; job satisfaction; difficulties in finding a job; observations about the job market; discrimination within a job or during a job search...

work page

Show all 18 references

[9]

will tell you anything just to get your vote

Government and Institutions (Theme) • Expectations References to the expectations and aspirations that the public has of elected officials, city government, and/or civic institutions. • Processes: References to processes through which the public interfaces with government, suc...
[10]

The responses and resources intended to support healing from those traumas

Public Health (Theme) • Mental Health: Those who struggle with mental health; systemic issues of mental health; responses to those with mental health issues; resources and isntitutions that support mental health • Drugs and Drug Use Disorder: Addiction, systemic issues of drug...
[11]

• Street Violence: Refers to situations like street fighting, assaults on the street, unintentional harm of bystanders, etc

Safety (Theme) • Sense of Safety: Refers to feeling unsafe within daily life routines at home, in one’s neighbor- hood, and throughout the city. • Street Violence: Refers to situations like street fighting, assaults on the street, unintentional harm of bystanders, etc. • Gun V...
[12]

• Transportation: Public transportation like the buses and trains, quality of transportation, affordability and accessibility of transportation, safety of public transit

Infrastructure (Theme) • Climate Impacts: Climate change, impact of climate change on the community, actions to address climate change, fears around climate change. • Transportation: Public transportation like the buses and trains, quality of transportation, affordability and ...
[13]

• Housing Instability: Difficulty paying rent, having frequent moves, living in overcrowded conditions, or doubling up with friends and relatives

Housing (Theme) • Gentrification and displacement: Displacement of lower income residents; physical transfor- mation and change of the cultural character of the neighborhood. • Housing Instability: Difficulty paying rent, having frequent moves, living in overcrowded conditions...
[14]

Quality and nature of those relationships

Community Life (Theme) • Community Relationships: Relationships between community members, across generations, and across communities. Quality and nature of those relationships. • Community Values: Values instilled throughout the community, values differences within and across...
[15]

Education (Theme) • Quality of Education: Education that leads to empowerment as a process of strengthening individuals and communities to get more control over their own situations and environments; education systems that focus on the importance of quality learners, quality l...
[17]

annotation_schemes

Inequality (Theme) • Race: Defined as lack of jobs, services, goods, based on skin color, ethnicity, language. • Class: Refers to socioeconomic status, education, and types of disparities, including neighbors re-entering society. • Gender: Discrimination based on (anatomy) fem...
[18]

Community Life: Community Resources

Overall, responses were very well-formed according to this prompt, both for Llama and GPT-4. One single label that was not in our codebook was hallucinated 3 times across the generation process for the NYC corpus: External Motivations: Government. Qualitatively, we note with i...
[2023]

con- trol

ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks. Proceedings of the Na- tional Academy of Sciences, 120(30):e2305016120. ArXiv:2303.15056 [cs]. Zeyu He, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Shaurya Rohatgi, and Ting-Hao Kenneth Huang. 2024. If in a Crowdsou...

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

My church helped me find alternative childcare during the pandemic

External motivations (Theme) • Civic Organizations: The speaker mentions civic organizations like non-profits, churches, NGOs, and community clubs, associations as a factor in health decisions – Example: "My church helped me find alternative childcare during the pandemic." • Family & Friends: Mentions family and friends as a factor in health decisions – E...

work page

[2] [2]

My church helped me find alternative childcare during the pandemic

Intrinsic Motivations (Theme) • Fear of Virus: Mentions not wanting to become ill from COVID as a factor in health decisions – Example: "My church helped me find alternative childcare during the pandemic." • Getting Back to Normal: Mentions a desire for a return to activities and social routines as a factor in health decisions – Example: "I thought you kn...

work page

[3] [3]

A community health person came to my school and explained what was going on in the pandemic and the latest research on masking. So that’s when we started masking

Role of Community Organizations (Theme) • Health Education & Support: Mentions that community health educators were a factor in health decisions – Example: "A community health person came to my school and explained what was going on in the pandemic and the latest research on masking. So that’s when we started masking." • Support Incentives: Mentions that ...

work page

[4] [4]

We would have been so lost without the food bank that restocked each week

Resources (Theme) • Significant Impact Resources: Mentions that impact resources like food, financial assistance, rent moratorium, student loan suspension, employment helped during COVID – Example: "We would have been so lost without the food bank that restocked each week." • Unmet community needs: Mentions that a resource is needed – Example: "We just ne...

work page 2020

[5] [5]

I just couldn’t get over how scary it was that my sister had this reaction to the vaccine. I know it could happen to me. So no I did not go through with the vaccine

Vaccine Hesitancy (Theme) • Did not vaccinate: Mentions that the speaker did not choose to vaccinate – Example: "I just couldn’t get over how scary it was that my sister had this reaction to the vaccine. I know it could happen to me. So no I did not go through with the vaccine." • Mistrust or Skepticism: Mentions that the speaker has/had mistrust or skept...

work page

[6] [6]

Helping at my church made me feel like I was making a difference even though the world was going crazy

Personal COVID-19 Experience (Theme) • Resilience, Connection, & Hope: Mentions agency, control, or feeling empowered during the pandemic period – Example: "Helping at my church made me feel like I was making a difference even though the world was going crazy." • Stress, Fear, & Uncertainty: Mentions stress, fear, or uncertainty during the pandemic period...

work page

[7] [7]

Talking about this has made me remember how hard that period was for our family

Future Visions & Takeaways • Conversation Reflections: Mentions reflections on the conversation – Example: "Talking about this has made me remember how hard that period was for our family." • Post-pandemic future: Mentions a future vision for their life or community after the pandemic – Example: "I just can’t wait for the schools to go back to normal and ...

work page

[8] [8]

Economic Opportunity (Theme) • Jobs: References to a person’s ability to provide for themselves and their families. Can include statements about working multiple jobs; working in a particular industry; facing unemployment; job satisfaction; difficulties in finding a job; observations about the job market; discrimination within a job or during a job search...

work page