Rating for Parents: Predicting Children Suitability Rating for Movies Based on Language of the Movies

Mahsa Shafaei; Niloofar Safi Samghabadi; Sudipta Kar; Thamar Solorio

REVIEW 3 major objections 6 minor 1 cited by

Rating for Parents: Predicting Children Suitability Rating for Movies Based on Language of the Movies

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A neural model reads movie scripts and predicts the MPAA rating with 78 percent weighted F1, beating an SVM baseline by 6.1 percent.

desk verdict New dataset and task for MPAA rating prediction from scripts, but the 78% weighted F1 hides zero correct predictions on G and NC-17, undercutting the suitability claim. read the letter →

arxiv 1908.07819 v2 pith:YEIJ4N2O submitted 2019-08-21 cs.CL cs.LG

Mahsa Shafaei , Niloofar Safi Samghabadi , Sudipta Kar , Thamar Solorio This is my paper · ORCID

classification cs.CLcs.LG

keywords MPAAratingpredictionmoviescriptstextclassificationRNNwithattentionemotionanalysisgenremodelingcontentsuitabilityparentalguidance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Can a movie's age-suitability rating be predicted before production, just from its script? This paper argues yes: a recurrent network with attention that reads character dialogue and folds in the movie's genre and emotional tone assigns MPAA ratings (G, PG, PG-13, R, NC-17) with a weighted F1 of 78.03 percent, beating an SVM baseline by 6.1 percent. The authors contribute the first script corpus paired with MPAA ratings, about 7,000 movies across 24 genres, and show that context-sensitive modeling beats counting bad words. If the approach holds, parents could screen unrated films, streaming services could auto-apply age filters, and producers could adjust content before costly post-production changes. The paper also reports that its model predicts zero G and zero NC-17 films correctly, so the headline score does not yet translate into a complete suitability checker.

What carries the argument

The load-bearing mechanism is an attention-augmented recurrent reader: a 300-dimensional GloVe embedding layer feeds an LSTM whose hidden states are combined by Bahdanau-style attention, so the model learns which words in a conversation matter for the rating. Two context vectors are concatenated to the attention output before the dense and softmax layers: a normalized emotion vector (the per-movie percentage of words in each of eight NRC emotion categories plus two sentiments) and a 24-way multi-hot genre vector. The emotion vector supplies the dominant affective tone; the genre vector supplies the interpretive frame that decides whether, for example, a violent phrase is routine in an action movie or alarming in an animated family film. The ablation comparisons (L&A, +genre, +emotion, +emotion+genre) are what attribute the measured gain to each signal.

What would settle it

Run the same architecture on a class-balanced test set or report per-class recall for G and NC-17; if the model cannot identify a single all-ages or adult-only film even when those classes are oversampled, the claim that it predicts suitability collapses. Alternatively, have parents rate a held-out set of unrated scripts and compare their judgments to the model's output; if script-based predictions disagree with human suitability judgments more often than human raters disagree with each other, the task framing fails.

Watch

Extended reading notes

Core claim

The central claim is that the MPAA rating of a movie is predictable from the movie's script alone, provided the model sees not just words but also the genre and the emotional content of conversations. Jointly modeling genre and emotion improves a plain LSTM-with-attention reader: emotion alone adds about 1.9 points and genre alone about 1.0 point of weighted F1, and their combination reaches 78.03 percent, outperforming the threshold-based bad-word baseline by 14.13 percent, an SVM with lexical and sentiment features by 6.1 percent, and a CNN analog by 3.29 percent. The system uses 300-dimensional GloVe embeddings, an LSTM layer, Bahdanau attention, a normalized emotion vector built from the NRC lexicon, and a 24-way multi-hot genre vector before a softmax over five ratings. The error analysis is interpreted as evidence that context, not raw profanity counts, drives suitability distinctions: R-rated 'disgust' words denote violent acts while PG-rated ones denote milder discomfort. However, the confusion matrix shows no correct predictions for G or NC-17, and most R-rated errors fall into the adjacent PG-13 class.

Load-bearing premise

The load-bearing premise is that a single weighted-F1 score on an imbalanced test split measures whether a model can actually judge children's suitability; with 4,030 of 7,040 movies rated R and zero correct predictions for G and NC-17, the 78 percent score says little about the extremes that matter most for child safety.

Editorial extensions

If this is right

If script-only prediction is reliable, producers can estimate a film's rating before shooting wraps and adjust content to hit a target audience.
Streaming services and parental-control tools can assign provisional age filters to the large back-catalog of unrated online films.
Emotion and genre features are measurable gains, indicating that suitability is an affective-context phenomenon rather than a profanity-counting problem.
The same architecture transfers as a baseline to other content-suitability axes, such as violence intensity or substance-use depictions.
The error profile suggests that for practical use, a two-stage design (first separate R and PG-13 from family-friendly films, then refine) may serve better than five-way classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A practical parental-control deployment would need the model to recover G and NC-17, which it currently never does; this implies either cost-sensitive training, oversampling, or a two-tier cascade rather than the reported five-way classifier.
Because the emotion signal is computed from a lexicon's word counts rather than a learned representation, the reported emotion gains may understate what a trained affect model could achieve; testing a supervised emotion encoder on the same architecture is a natural next experiment.
The paper's under-18-audience analysis suggests MPAA ratings shape viewing behavior; if automated ratings become widespread, they could shift what children watch, making the reliability of extreme-class predictions a policy matter rather than just a benchmark number.
The 6-point gap over the SVM is measured on a single split; an adversarial check would compare against a stronger contextual encoder to see whether the recurrent-attention advantage survives.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing to know: this is the first paper to frame MPAA rating prediction from movie scripts as a task, and it contributes a ~7k-script dataset with ratings. Second: the 78.03% weighted F1 headline is mostly a measure of how well the model predicts R-rated movies, because the test set is dominated by R (4,030 out of 7,040) and the model predicts zero correct for G and NC-17. The authors show this in the confusion matrix and admit it, so there is no deception, but the central claim that they are predicting children's suitability is undercut by total failure on the two classes that matter most for that claim.

What's genuinely good: the dataset is new and, once released, will be a resource. The task is well motivated. The ablation design (adding genre and emotion to a plain LSTM with attention) is clean and shows both features help. The error analysis is more honest than usual—they examine attention weights, bad word ratios, and emotion patterns, and they avoid overclaiming in the text. The baseline set includes a threshold model, SVM, and CNN, which is fair. The related work is appropriate.

The soft spots are real. The evaluation is a single stratified split with no repeated runs, no confidence intervals, and no significance tests. The weighting of F1 masks the extreme classes, and the paper offers only "low instance count" as the explanation. That is plausible, but per-class precision/recall and at least a few random seeds would be needed to know whether the failure is a small-sample artifact or a systematic inability to detect G or NC-17 content. The paper also only promises data availability; no code or data is shipped. Minor point: the threshold baseline with a hand-compiled bad-word list is a strawman, though it serves the rhetorical purpose of showing bad words alone don't suffice.

Overall, I'd send this to peer review. The dataset and task are worth publishing, but the evaluation needs to be strengthened before the suitability claim can be taken at face value. If I were working on content moderation or movie analytics, I'd want to cite the dataset. I'd bring it to a reading group as a case study in metric choice on imbalanced data.

Referee Report

3 major / 6 minor

Summary. The paper introduces a new task, automatic prediction of MPAA ratings from movie scripts, together with a corpus of about 7K movies (Table 1) and a neural architecture. The proposed model is an LSTM with attention over dialogue, combined with a movie-level NRC emotion vector and a multi-hot genre vector (Section 4). On a single stratified 80/10/10 split, the full model achieves a weighted F1 of 78.03%, which is 6.1 points above an SVM baseline and 3.29 points above a CNN baseline (Table 3). The paper also presents threshold, SVM, and CNN baselines, per-genre results, an emotion analysis, an attention-weight analysis, and a discussion of MPAA ratings' practical effect on under-18 audiences.

Significance. The dataset release and the systematic comparison of threshold, SVM, CNN, and LSTM-attention models are genuinely useful first steps for a task that appears not to have been studied before. The ablation results (Table 3) give credible evidence that emotion and genre features each contribute to weighted F1 on this corpus, and the reported 78.03% is internally consistent with the described pipeline. However, the significance of the headline claim is currently limited by the evaluation metric: because the test set contains 4,030 R-rated movies out of 7,040, weighted F1 is dominated by the majority classes, and the model identifies no G or NC-17 movies correctly (Figure 4). For a system whose stated purpose is predicting children's suitability, that is not a mere presentation issue; it affects whether the 78.03% figure supports the central claim. With per-class results, repeated splits, or a reframed claim about predicting restricted ratings, the contribution could become solid.

major comments (3)

[§7, Fig. 4; §3, Table 1; §6, Table 3] The model achieves zero true-positive predictions for G and NC-17, as the confusion matrix in Figure 4 shows and as the paper acknowledges in Section 7, while weighted F1 (78.03%) is dominated by R and PG-13 because of the class distribution in Table 1. Since the paper's stated goal is to predict suitability for children, the G and NC-17 classes are the most decision-relevant ones: G is the only rating with unrestricted admission for all ages, and NC-17 is the only rating that excludes all viewers under 17. The single weighted-F1 number therefore does not, by itself, support the suitability-prediction claim; the authors should report per-class precision/recall/F1 and a macro-F1 or balanced evaluation, and temper the abstract and conclusion accordingly.
[§5–§6, Table 3] All results are based on one stratified split, with no repeated runs, confidence intervals, or significance tests; the gap between the best model and the CNN baseline is 3.29 F1 points, which could plausibly be within run-to-run variance for LSTM training. The authors should report results over multiple seeds (and ideally multiple splits) with standard deviations and a paired significance test, or at least show that the ranking of models is stable across restarts.
[§5.2] The development set is used both for hyperparameter selection (learning rate, LSTM hidden units, dropout) and for selecting the 'best weighted-F1' model over 200 iterations, with no statement of how many configurations were tried; this makes the validation-based model selection part of the reported test result. The paper should state the number of configurations, confirm that the test set was used only once, and ideally include a small study of sensitivity to the reported hyperparameters.

minor comments (6)

[§4.2] The phrase 'spatial dependency of the words' should be 'sequential dependency' for an LSTM.
[§5.1] In the threshold baseline description, 'NRC-17' should be 'NC-17'.
[§6] The text says 'MPPA rating' in the paragraph after Table 4; should be 'MPAA'.
[§7.1] The term 'miss-classiﬁed' should be 'misclassiﬁed' throughout Section 7.
[§3] 'It should bee noted' contains a typo; should be 'be noted'.
[Table 5] The column header 'Rate' is ambiguous; use 'Rating' and format the example sentences so their source rating is clear.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the classifier's inputs and labels are independent, and the stated result is a held-out benchmark rather than a fitted quantity.

full rationale

This paper is a supervised classification study, not a derivation, so there are no equations in which the predicted quantity is defined in terms of the fitted parameters or vice versa. The MPAA rating labels are crawled from IMDB and are external to the model inputs; the features (GloVe embeddings, NRC emotion counts, and genre vectors) are also external and are not constructed from the MPAA ratings. The 78.03% weighted F1 result is reported on a held-out test split created by stratified random sampling, so the central claim is an empirical benchmark rather than a re-statement of a fitted value. The only self-citations are to Shafaei et al. (2019) for the raw script corpus and for the SVM feature recipe; neither of these imports the target MPAA ratings or the paper's conclusion. The threshold baseline does fit thresholds on the training set, but it is explicitly a baseline and is not the source of the main claim. The paper's own error analysis states that 'our model is not able to correctly predict any instances of G and NC-17' and attributes this to the low number of instances; this is a disclosed limitation of the evaluation, not a circular step. No pattern from the enumerated circularity kinds applies, so the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central result depends on external resources (GloVe, NRC lexicon, IMDB metadata and ratings) and on a small set of hyperparameters chosen on the validation set. No new entities are introduced. The most consequential assumptions are the sufficiency of dialogue-only scripts and the meaningfulness of weighted F1 on an imbalanced test set.

free parameters (5)

learning_rate = 0.00001
Selected by grid search on the validation set (Section 5.2).
lstm_hidden_units = 256
Selected by grid search over {32, 64, 128, 256} on validation (Section 5.2).
dropout_rate = 0.3
Selected by grid search over {0.3, 0.4, 0.5} on validation (Section 5.2).
svm_regularization_C = not reported (tuned over {1, 10, 100, 1000})
Chosen by grid search for the SVM baseline (Section 5.1).
threshold_baseline_boundaries = t1..t5, values not reported
Trained on the training set for the bad-word threshold baseline (Section 5.1).

assumptions (4)

domain assumption Dialogue-only movie scripts are sufficient to predict MPAA ratings.
Section 3 states scripts contain only conversations; the system ignores visual and audio content that CARA members use to rate films.
domain assumption MPAA ratings scraped from IMDB are accurate.
Section 3: 'we crawl the IMDB website to extract MPAA rating.' No manual verification or inter-annotator agreement is reported.
domain assumption Normalized NRC lexicon counts per emotion capture the emotional context relevant to suitability.
Section 4.3 computes a 10-dimensional emotion vector from the whole movie; this assumes global lexicon counts are a meaningful signal.
ad hoc to paper Keeping the dataset imbalanced is appropriate for the evaluation.
Section 3 says imbalance was intentionally kept for representativeness; combined with weighted F1 this choice hides poor performance on rare classes.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Rating for Parents: Predicting Children Suitability Rating for Movies Based on Language of the Movies." pith.science (2026). https://pith.science/paper/YEIJ4N2O

@misc{pith2026190807819,
  author       = {Pith},
  title        = {Pith review of: Rating for Parents: Predicting Children Suitability Rating for Movies Based on Language of the Movies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YEIJ4N2O}},
  note         = {Machine review of arXiv:1908.07819}
}

read the original abstract

The film culture has grown tremendously in recent years. The large number of streaming services put films as one of the most convenient forms of entertainment in today's world. Films can help us learn and inspire societal change. But they can also negatively affect viewers. In this paper, our goal is to predict the suitability of the movie content for children and young adults based on scripts. The criterion that we use to measure suitability is the MPAA rating that is specifically designed for this purpose. We propose an RNN based architecture with attention that jointly models the genre and the emotions in the script to predict the MPAA rating. We achieve 78% weighted F1-score for the classification model that outperforms the traditional machine learning method by 6%.

Figures

Figures reproduced from arXiv: 1908.07819 by the authors.

**Figure 2.** MPAA rating distribution per genre. Although genre can help the model to improve [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 4.** Confusion matrix of the best model for pre [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

Figures from the paper (3 more)

**Figure 3.** Figure 3: MPAA rating distribution per emotion. The [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]

**Figure 5.** Figure 5: Average emotion score of correctly and incor [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Percentage of under 18 audiences in each [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Video-Based MPAA Rating Prediction: An Attention-Driven Hybrid Architecture Using Contrastive Learning
cs.CV 2025-09 conditional novelty 4.0 of 10

A CNN+LSTM+attention model with contrastive learning predicts MPAA ratings from short video clips with 88% accuracy on a custom 323-clip dataset.

Reference graph

Works this paper leans on

24 extracted references · 19 canonical work pages · cited by 1 Pith paper

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

MPAA Motion Picture Association of America. 2010. Classification and rating rules

work page 2010
[4]

Segun Taofeek Aroyehun and Alexander Gelbukh. 2018. Aggression detection in social media: Using deep neural networks, data augmentation, and pseudo labeling. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 90--97

work page 2018
[5]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

arXiv 2014
[6]

Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Eleventh International AAAI Conference on Web and Social Media

work page 2017
[7]

Theodoros Giannakopoulos, Alexandros Makris, Dimitrios Kosmopoulos, Stavros Perantonis, and Sergios Theodoridis. 2010. Audio-visual fusion for detecting violent scenes in videos. In Hellenic Conference on Artificial Intelligence, pages 91--100. Springer

work page 2010
[8]

Gabin Gninkoun and Mohammad Soleymani. 2011. Automatic violence scenes detection: A multi-modal approach

work page 2011

Show all 24 references

[9]

Homa Hosseinmardi, Rick Han, Qin Lv, Shivakant Mishra, and Amir Ghasemianlangroodi. 2014. Towards understanding cyberbullying behavior in a semi-anonymous social network. In Advances in Social Networks Analysis and Mining (ASONAM), 2014 IEEE/ACM International Conference on, pa...

2014
[10]

Timothy Jay and Kristin Janschewitz. 2008. The pragmatics of swearing. Journal of Politeness Research. Language, Behaviour, Culture, 4(2):267--288

2008
[11]

Lucille Jenkins, Theresa Webb, Nick Browne, Abdelmonem A Afifi, and Jess Kraus. 2005. An evaluation of the motion picture association of america’s treatment of violence in pg-, pg-13--, and r-rated films. Pediatrics, 115(5):e512--e517

2005
[12]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

2014 arXiv
[13]

Victor R Martinez, Krishna Somandepalli, Karan Singla, Anil Ramakrishna, Yalda T Uhls, and Shrikanth Narayanan. 2019. Violence rating prediction from movie scripts

2019
[14]

Puneet Mathur, Ramit Sawhney, Meghna Ayyar, and Rajiv Shah. 2018. Did you offend me? classification of offensive tweets in hinglish language. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 138--148

2018
[15]

Saif Mohammad. 2011. From once upon a time to happily ever after: Tracking emotions in novels and fairy tales. In Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 105--114. Association for Computationa...

2011
[16]

Chikashi Nobata, Joel Tetreault, Achint Thomas, Yashar Mehdad, and Yi Chang. 2016. Abusive language detection in online user content. In Proceedings of the 25th international conference on world wide web, pages 145--153. International World Wide Web Conferences Steering Committee

2016
[17]

Ji Ho Park and Pascale Fung. 2017. One-step and two-step classification for abusive language detection on twitter. arXiv preprint arXiv:1706.01206

2017 arXiv
[18]

AAP American Academy of Pediatrics. 2001. Media violence. Pediatrics, 108(5):1222--1226

2001
[19]

James D Sargent, Thomas A Wills, Mike Stoolmiller, Jennifer Gibson, and Frederick X Gibbons. 2006. Alcohol use in motion pictures and its relation with early-onset teen drinking. Journal of studies on alcohol, 67(1):54--65

2006
[20]

Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1--10

2017
[21]

Mahsa Shafaei, Adrian Pastor Lopez-Monroy, and Thamar Solorio. 2019. Exploiting textual, visual and product features for predicting the likeability of movies. In The 32nd International FLAIRS Conference

2019
[22]

Vinay Singh, Aman Varshney, Syed Sarfaraz Akhtar, Deepanshu Vijay, and Manish Shrivastava. 2018. Aggression detection on social media text using deep neural networks. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2), pages 43--50

2018
[23]

Theresa Webb, Lucille Jenkins, Nickolas Browne, Abdelmonen A Afifi, and Jess Kraus. 2007. Violent entertainment pitched to adolescents: an analysis of pg-13 films. Pediatrics, 119(6):e1219--e1229

2007
[24]

Ziqi Zhang, David Robinson, and Jonathan Tepper. 2018. Detecting hate speech on twitter using a convolution-gru based deep neural network. In European Semantic Web Conference, pages 745--760. Springer

2018

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

MPAA Motion Picture Association of America. 2010. Classification and rating rules

work page 2010

[4] [4]

Segun Taofeek Aroyehun and Alexander Gelbukh. 2018. Aggression detection in social media: Using deep neural networks, data augmentation, and pseudo labeling. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), pages 90--97

work page 2018

[5] [5]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

arXiv 2014

[6] [6]

Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Eleventh International AAAI Conference on Web and Social Media

work page 2017

[7] [7]

Theodoros Giannakopoulos, Alexandros Makris, Dimitrios Kosmopoulos, Stavros Perantonis, and Sergios Theodoridis. 2010. Audio-visual fusion for detecting violent scenes in videos. In Hellenic Conference on Artificial Intelligence, pages 91--100. Springer

work page 2010

[8] [8]

Gabin Gninkoun and Mohammad Soleymani. 2011. Automatic violence scenes detection: A multi-modal approach

work page 2011