{"id":"aa3cb87d-3f14-4f7b-8d37-980ddcc058ac","arxiv_id":"1907.06292","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Releases TWEETQA dataset for QA on social media tweets and shows neural models including fine-tuned BERT lag substantially behind humans.","lead":"This paper introduces TWEETQA, the first large-scale question answering dataset built from tweets that journalists used in news articles, with human-written abstractive answers. A smart generalist might read it to understand why current AI systems that work on Wikipedia or news still fail on informal, real-time social media text.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Journalist-sourced tweet filter selects for coherent/news-like content, weakening claim of general social media QA coverage","rationale":"The reader already flagged the exact collection criterion as the weakest assumption; the full abstract supplies no additional evidence that mitigates the bias, so the concern stands as the primary internal risk to the central claim.","tokens_in":1675,"tokens_out":317,"duration_ms":12004,"concrete_test":"Draw a matched-size sample of tweets from the same time window using only temporal and language filters (no journalist citation requirement); repeat the human QA annotation protocol on this sample; fine-tune the same BERT model and compare F1 / human agreement on the two sets. A >15-point drop in model performance or human agreement on the unfiltered sample would confirm the selection bias materially affects the headline claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The dataset is built exclusively from tweets that journalists cited when writing news articles. This curation step (explicit in the abstract) preferentially retains factual, well-formed, and event-oriented tweets while excluding the noisy, abbreviated, opinion-heavy, or context-dependent posts that dominate typical social-media streams. Because the central claim is that TWEETQA enables QA “over social media data,” the representativeness assumption is load-bearing: if the filtered distribution differs systematically from uncurated social media, both the “first large-scale social-media QA dataset” framing and the reported model-vs-human gap become statements about a narrower, easier subdomain rather than social media in general.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript presents TWEETQA as the first large-scale QA dataset for social media, constructed by collecting tweets cited by journalists in news articles and having annotators generate questions with abstractive answers. It evaluates neural QA models and BERT on the dataset, finding substantial gaps relative to human performance, and argues for the development of improved systems targeting social media text.","tokens_in":1781,"tokens_out":317,"duration_ms":67825,"significance":"Should the dataset prove representative of social media QA challenges, the work would be significant as it introduces a new benchmark in an important but under-served domain of informal, real-time text. The reported model-human performance gap provides concrete evidence of current limitations and could motivate targeted research. The provision of a dataset with abstractive answers over tweets is a notable contribution compared to extractive QA datasets like SQuAD.","major_comments":[{"comment":"The collection method restricts to tweets used by journalists to write news articles. This curation step preferentially selects for coherent and factual tweets, which may not represent the full distribution of social media content including noisy or opinion-based posts. Since the central claim is that this enables QA 'over social media data,' this representativeness assumption is load-bearing and requires explicit discussion or validation in the manuscript.","section":"Abstract"}],"minor_comments":[{"comment":"Typo: 'pop-ular' should read 'popular'. Typo: 'eventsare' should read 'events are'.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on dataset construction and representativeness. We address the major comment below and will revise the manuscript to incorporate an explicit discussion of the curation approach and its implications.","responses":[{"response":"We agree that restricting to tweets cited by journalists introduces a curation bias toward more coherent and factual content, as opposed to the full range of noisy or opinion-based social media posts. This step was deliberate to ensure the collected tweets contain substantive information suitable for QA, as noted in the abstract and methods. The tweets nonetheless originate from Twitter and exhibit social-media-specific traits including informal language, abbreviations, and real-time context. In the revised manuscript we will add a dedicated limitations subsection that explicitly discusses the curation rationale, acknowledges the resulting deviation from the broader social-media distribution, and clarifies the scope of our central claim. Full empirical validation against the entire Twitter distribution is not feasible within the scope of this work due to the scale and ephemerality of social media data, but the added discussion will make the assumptions transparent.","revision_made":"yes","referee_comment":"[Abstract] The collection method restricts to tweets used by journalists to write news articles. This curation step preferentially selects for coherent and factual tweets, which may not represent the full distribution of social media content including noisy or opinion-based posts. Since the central claim is that this enables QA 'over social media data,' this representativeness assumption is load-bearing and requires explicit discussion or validation in the manuscript."}],"tokens_in":1247,"tokens_out":322,"duration_ms":11101,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that this paper releases TWEETQA, a collection of questions and abstractive answers written over tweets that journalists had already used in news articles, and reports that fine-tuned BERT still trails human performance by a noticeable margin while models tuned on formal text do even worse here.","headline":"TWEETQA is a usable dataset release for QA on journalist-cited tweets, but the curation step makes the social-media claim narrower than the abstract suggests.","tokens_in":2259,"tokens_out":138,"would_cite":true,"duration_ms":14295,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Foundation/RealityFromDistinction.lean","rs_theorem":"reality_from_one_distinction","paper_passage":"we only gather tweets used by journalists to write news articles... first large-scale dataset for QA over social media data"},{"relation":"unclear","rs_module":"IndisputableMonolith/Cost/FunctionalEquation.lean","rs_theorem":"washburn_uniqueness_aczel","paper_passage":"even the fine-tuned BERT model is still lagging behind human performance with a large margin"}],"headline":"NLP social-media QA dataset curation is orthogonal to RS forcing chain","alignment":"orthogonal","rationale":"Paper centers on journalist-filtered tweet collection, abstractive QA annotation, and baseline evaluation (BERT etc.) on informal text; no contact with distinction axioms, J-cost, φ-ladder, 8-tick periodicity, or spacetime emergence. Matches the 'survey/measurement of a known process' example of orthogonality.","tokens_in":49502,"confidence":"high","tokens_out":255,"duration_ms":5064,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"TweetQA is the first large-scale dataset for question answering over tweets, revealing that even fine-tuned BERT lags human performance significantly.","keywords":["question answering","social media","TweetQA","abstractive QA","BERT","neural models","dataset"],"falsifier":"Demonstrating a model that matches or exceeds human performance on the TweetQA dataset using standard techniques would undermine the claim that social media text presents distinct difficulties.","tokens_in":2579,"feed_emoji":"🐦","tokens_out":556,"duration_ms":19123,"temperature":0.7,"pith_summary":"The authors build a dataset of questions and abstractive answers from tweets that journalists used in news articles. They test neural models known to work on formal text and find they perform much worse on this social media data. This matters because many real-time events and news are reported on social media, so effective QA systems there could support better applications. The large gap with human performance suggests current methods need adaptation for informal text.","feed_headline":"Dataset shows AI lags on questions about tweets","feed_subtitle":"TweetQA finds fine-tuned BERT still far behind humans, indicating need for social media QA improvements.","key_machinery":"The TweetQA dataset, built from journalist-sourced tweets with abstractive QA pairs, used as a benchmark to demonstrate limitations of existing QA models on social media text.","core_discovery":"We present the first large-scale dataset for QA over social media data by collecting tweets used by journalists to write news articles and having annotators create questions and abstractive answers on them. Two recently proposed neural models perform poorly on this dataset compared to formal text, and even fine-tuned BERT lags behind human performance with a large margin.","pith_inferences":["Extending the dataset to other social media platforms could reveal if Twitter-specific features drive the difficulty.","Training models with more social media data might close the performance gap with humans.","Applications like automated news summarization or event detection could use such QA systems if improved."],"forward_implications":["QA systems for real-time knowledge from social media will require new approaches beyond those for news and Wikipedia.","Models must handle abstractive answers rather than just extractive spans.","The dataset provides a testbed to develop and evaluate social media specific QA techniques.","Performance gaps indicate that informal language and noise in tweets pose unique challenges."],"fun_headline_variants":["TweetQA introduces first large QA dataset from social media","Neural models struggle with question answering on tweets","Fine-tuned BERT lags humans on tweet question answering","New dataset tests QA on journalist-used tweets","TweetQA shows models limited on informal social text"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That tweets selected because journalists used them to write news articles form a representative and useful sample for general social media QA.","fun_headline_variants_meta":{"raw":{"variants":["TweetQA introduces first large QA dataset from social media","Neural models struggle with question answering on tweets","Fine-tuned BERT lags humans on tweet question answering","New dataset tests QA on journalist-used tweets","TweetQA shows models limited on informal social text"]},"model":"grok-4.3","cost_usd":0.003854,"raw_usage":{"total_tokens":1957,"prompt_tokens":616,"num_sources_used":0,"completion_tokens":68,"cost_in_usd_ticks":38537000,"prompt_tokens_details":{"text_tokens":616,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1273,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":616,"tokens_out":68,"duration_ms":7585,"temperature":1.0,"reasoning_tokens":1273,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-24T21:22:30.782673+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Demonstrating a model that matches or exceeds human performance on the TweetQA dataset using standard techniques would undermine the claim that social media text presents distinct difficulties.","supporting_citations":[],"review_version":1}