{"id":"4cbdbd2b-328f-4c97-bc74-bdb3b18384f9","arxiv_id":"2504.19489","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Community search algorithms produce structurally dense but psychologically non-cohesive communities, and psychological and structural cohesion are uncorrelated.","lead":"This paper evaluates eight community search algorithms on Twitter data using five new measures of 'psychological cohesion' adapted from social psychology. It finds that algorithms optimized for structural density do not find communities that feel cohesive to people, and that the two types of cohesion do not correlate.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Negative conclusion hinges on unvalidated proxy measures for psychological cohesion; without evidence that EI/SIT/CED/GIP/GID track self-reported GEQ cohesion, the claim that no algorithm identifies psychologically cohesive communities may be an artifact of the operationalization.","rationale":"The reader's weakest assumption is exactly the construct validity of the five psychology-informed measures, and I agree that this is the load-bearing point. The paper's headline findings are negative empirical statements about community search algorithms: structural cohesion does not correlate with psychological cohesion, and no algorithm retrieves psychologically cohesive communities. Both statements are only as strong as the measures used to define 'psychological cohesion.' The measures are derived from the GEQ by a plausible but unvalidated mapping: self-report items about enjoyment, friendship, group preference, and interaction frequency are translated into graph formulas over interaction counts, timestamps, and LLM sentiment labels. A community could score high on these proxies while its members do not perceive themselves as cohesive (e.g., frequent but shallow positive exchanges), and a genuinely cohesive group could interact sparsely outside the observed window. Because no external criterion—human GEQ responses, expert ratings, or behavioral validation—is provided, the negative results are vulnerable to the objection that they describe the behavior of the formulas, not the psychology of the users. The paper itself concedes in §8 that no ground-truth communities based on cohesion theories exist and that the measures are constrained by sentiment analysis tools, which supports this concern. Other weaknesses, such as missing error bars, unreleased datasets, and the unresolved placeholder reference in §7, are real but secondary; they do not directly threaten the semantic validity of the central claim. The proposed human-subject validation is a single, feasible check that would settle whether the concern lands: if the proxies correlate strongly with self-reported GEQ cohesion and preserve algorithm rankings, the central claim gains independent support; if not, the paper's conclusion should be weakened substantially. For these reasons I maintain the reader's CONDITIONAL verdict rather than moving to accept or reject.","tokens_in":20508,"tokens_out":4487,"duration_ms":51430,"concrete_test":"Run a criterion-validity study on a sample of communities from the datasets/algorithms used in the paper: for 30–50 returned communities, have actual members (or, failing access, trained annotators given anonymized interaction transcripts) complete the adapted GEQ from Table 1; compute Spearman correlations between per-community mean GEQ responses and each of EI, SIT, CED, GIP, GID at the settings used in §6.4 (default λ=0.0001, plus at least one alternative decay rate). Also compare the algorithms' rank order by each proxy with their rank order by human-rated cohesion. If correlations are weak or non-significant, the operationalization lacks criterion validity and the central claim is unsubstantiated; if they are strong and rankings agree, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'no algorithm effectively identifies psychologically cohesive communities in online social networks' (Abstract; §8)—depends on the five measures introduced in §5 being valid operationalizations of the adapted GEQ items in Table 1. The paper does not establish this. EI (§5.1, Def. 5.2) equates enjoyment with a decayed, sentiment-excitation-weighted sum of interaction polarities produced by an LLM; SIT (Def. 5.3) equates best-friendship with summed mutual sentiment; CED (Def. 5.4) equates group importance with a difference of within/outside sentiment sums; GIP (Def. 5.5) and GID (Def. 5.6) are edge-count ratios. These are plausible behavioral proxies, but the GEQ is a self-report instrument about members' perceptions; no evidence connects the proxies to those perceptions. The paper contains no human-rated cohesion data, no correlation with actual GEQ responses, and no expert validation; §8 explicitly lists the absence of theory-grounded ground truth as a limitation. Section 7.5's 'no explicit correlation' is also only a one-query case study (Figures 10–11), so it cannot independently support the general no-correlation claim. If the measures misrepresent what members feel, then the negative findings are an artifact of the proxy definitions, not a property of the algorithms.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents CHASE, a framework for evaluating community search algorithms using five newly proposed \"psychology-informed\" cohesiveness measures derived from an adaptation of the Group Environment Questionnaire. The authors evaluate eight representative community search algorithms—three k-core-based, three k-truss-based, and two learning-based variants of the same framework—on four Twitter/X datasets with LLM-assigned sentiment labels. Their headline findings are that different algorithms return widely divergent communities for the same query, that learning-based methods produce weak structural results or fail outright, that structural and psychological cohesiveness show no clear correlation, and that no algorithm effectively identifies psychologically cohesive communities. A codebase is provided.","tokens_in":20910,"tokens_out":6941,"duration_ms":67292,"significance":"If the negative findings hold, the paper would provide a valuable challenge to the community-search field by showing that structural density metrics do not automatically capture group cohesion as conceptualized in social psychology. The manuscript's strengths include a reproducible codebase, a concrete adaptation workflow from a recognized psychological instrument, multiple real-world datasets, and robustness checks with a second sentiment tool (VADER) and two decay families. However, the central conclusions depend on the unvalidated construct validity of the five proposed measures and on a very narrow evidential basis for the general no-correlation claim; both need to be addressed before the findings can be accepted as stated.","major_comments":[{"comment":"Definitions 5.2–5.6 introduce EI, SIT, CED, GIP, and GID as quantitative proxies for the adapted GEQ items in Table 1, but the paper provides no evidence that these proxies correspond to the psychological construct they claim to measure. The GEQ is a self-report instrument about members' perceptions, whereas the proposed measures are graph formulas over LLM sentiment labels, interaction counts, and decay parameters; the mapping is asserted rather than validated. The authors themselves note in §8 that validation of the measures would require ground-truth datasets grounded in cohesion theories and acknowledge that no such datasets are used. Because the central claim that 'no algorithm effectively identifies psychologically cohesive communities' presupposes that the five measures faithfully operationalize group cohesion, the negative result may be an artifact of the operationalization. I recommend adding a validation component—for example, collecting adapted-GEQ ratings from human annotators for a sample of returned communities and reporting correlations with the five measures, or at minimum an expert face-validity assessment—or, absent that, substantially softening the claims.","section":"§5, Definitions 5.2–5.6; §8"},{"comment":"The general claim in the Abstract and §8 that 'there is no clear correlation between structural and psychological cohesiveness' is supported only by the case study in §7.5, which examines a single query node ('158') and five returned communities (Figures 10–11). That is too narrow an evidential basis for a blanket conclusion across algorithms, queries, datasets, and parameter settings. The paper should report a quantitative analysis across all 100 queries per dataset and, where applicable, across parameter combinations—for example, Spearman rank correlations between the structural metrics (diameter, size, minimum degree, k-core/k-truss values) and each of the five psychology-informed measures, with significance tests or confidence intervals—rather than relying on one visual comparison.","section":"§7.5, Figures 10–11; Abstract; §8"},{"comment":"The blanket statement that 'no algorithm effectively identifies psychologically cohesive communities' is not fully supported by the paper's own reported results. Section 7.2 states that 'communities identified by CSD mostly exhibit positive EI values across all datasets' and that 'users within communities from CSD, ST-Exa, and I2ACSM generally engage more with each other' on the GID measure (Figure 7). The paper never defines a threshold or criterion for 'effectively identifies' (e.g., a minimum fraction of queries with positive EI/CED/SIT, a minimum GIP/GID value, or robustness across decay parameters), so the conclusion conflates 'no algorithm succeeds on every measure' with 'no algorithm succeeds on any measure.' The authors should either define and justify a performance criterion or revise the conclusion to reflect the mixed evidence.","section":"§7.2; §8"},{"comment":"The second key finding—that 'recent learning-based algorithms tend to produce communities with low structural cohesiveness or fail to identify valid communities' (Section 1)—is based on only two variants of the same unsupervised framework, TransZero-LS and TransZero-GS, with TransZero-GS returning disconnected node sets in all datasets. Two variants from one framework are too narrow a sample to support a generalization about learning-based community search methods, especially given the authors' own observation that insufficient node features may explain the poor performance. The claim should be restricted to the tested methods or supported with additional learning-based baselines.","section":"§6.2, Table 4; §7.1, Table 6"}],"minor_comments":[{"comment":"The sentence 'The reader may refer to [?] for detailed results' contains an unresolved reference that must be fixed.","section":"§7"},{"comment":"The abbreviation 'CC' is used for the dataset without being defined; it should be introduced as Chicago_COVID at first use.","section":"§7.4–§7.5"},{"comment":"The stray '13.' before 'While strong topological connections may suggest users’ enjoyment...' appears to be a numbering error and should be removed.","section":"§4.2"},{"comment":"The claim that 'the cohesiveness scores are largely unaffected by sentiment analysis techniques' is followed by examples of polarity switches and obvious changes (ST-Exa's EI on BTW, CED values on C144); the sentence should be qualified to indicate which measures and datasets are stable.","section":"§7.3"},{"comment":"Table 2 marks Item 13 as fully captured ('✓') by k-core, k-truss, k-clique, and k-ECC, but the GEQ item refers to frequency of interaction; degree-based measures do not in themselves capture interaction frequency or temporality, so the '✓' appears overgenerous.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"This is a genuinely useful experimental paper with a novel social-psychology framing, but the central conclusions outrun the evidence as currently presented. The most important fix is a validation of the five measures against human judgments or a clear statement of their status as proxies; the second is to broaden the no-correlation analysis and soften the blanket claims. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that it is the first to take the Group Environment Questionnaire from social psychology and turn it into computable graph measures for evaluating community search algorithms. That is a real contribution. The five proposed measures (EI, SIT, CED, GIP, GID) are clearly defined, the CHASE framework is sensible, the code is released, and the authors evaluate eight representative algorithms on four Twitter-derived datasets with genuine robustness checks (two sentiment analyzers, two decay functions). The negative finding — that structural cohesiveness does not track psychology-based cohesion, and that no tested algorithm retrieves psychologically cohesive communities — is interesting and worth taking seriously, especially because the authors openly list the lack of theory-grounded ground truth as a limitation in Section 8. They do not overclaim the mechanism; they say current methods simply ignore the psychology. The soft spots are mostly about how far the central claim can travel. The five measures are plausible behavioral proxies, but the GEQ is a self-report instrument about members' perceptions, and the paper provides no evidence connecting these graph formulas to those perceptions — no human-annotated cohesion data, no correlation with actual GEQ responses, no expert validation. That matters, because if the measures misrepresent what members feel, the negative result becomes an artifact of the operationalization. Also, the 'no clear correlation' claim in the abstract is supported in the body mainly by a single case study (Figures 10–11), so the general statement is stronger than the evidence. Minor issues: a leftover placeholder reference in Section 7, two datasets appear not to be released, and most plots lack error bars. None of these are fatal; they are fixable with a validation study or by softening the wording. Who gets value from this: community search researchers who care about evaluation design, and social computing folks who want psychology-informed benchmarks. It deserves a serious referee. I would send it to peer review with the expectation of major revision: the authors should either add a small human-validation experiment (even crowdsourced ratings on a few communities would help) or explicitly reframe the contribution as a proposed evaluation framework plus preliminary evidence, rather than a conclusive negative result. The core idea is good and the empirical work is substantial; the gap between the measures and the psychological construct is the one thing that needs real attention.","headline":"A genuinely interdisciplinary evaluation paper that brings social-psychology cohesion measures into community search, with a thought-provoking negative result that is, however, somewhat stronger than the validation behind it.","tokens_in":647,"tokens_out":809,"would_cite":true,"duration_ms":19205,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Community search algorithms fail to return psychologically cohesive groups, an experimental evaluation finds.","keywords":["community search","group cohesion","social psychology","cohesiveness measures","online social networks","sentiment analysis","experimental evaluation","graph algorithms"],"falsifier":"Collect self-reported cohesion ratings from the members of communities returned by these algorithms on the same datasets, then correlate those ratings with the five measures and with k-core or k-truss density. A strong positive correlation between human ratings and structural density would directly contradict the no-correlation claim, while a strong correlation with the proposed measures would support them as operationalizations.","tokens_in":20325,"feed_emoji":"👥","tokens_out":4421,"duration_ms":44728,"temperature":0.7,"pith_summary":"This paper experimentally tests whether community search algorithms—methods that retrieve a dense subgraph around query nodes—actually return groups that hang together in the psychological sense. Drawing on group-cohesion theory, it builds five questionnaire-inspired measures of enjoyment, friendship, comparative enjoyment, interaction preference, and interaction density, then evaluates eight representative algorithms on four Twitter-derived networks under a framework called CHASE. The central finding is negative: structural metrics such as k-core and k-truss do not clearly correlate with psychological cohesion, and none of the tested algorithms consistently recovers communities that score well on the psychology-informed measures. If the findings hold, they matter because community search is used in applications where human group dynamics are the point, yet the field's core measure of success may be measuring the wrong thing.","feed_headline":"No community search algorithm finds psychologically cohesive groups","feed_subtitle":"Eight algorithms tested on four social networks: structural density does not predict group cohesion.","key_machinery":"The load-bearing object is a set of five psychology-informed cohesiveness measures derived from adapted items of the Group Environment Questionnaire: Enjoyment Index (cumulative sentiment-weighted interactions), Sentimental Interaction Tendency (reciprocal sentiment exchange between mutual interaction partners), Comparative Enjoyment Degree (inside-minus-outside enjoyment), Group Interaction Preference (share of activities that are interactions), and Group Interaction Density (interaction frequency per pair per time unit). The formulas embed a sentiment-aware excitation function with time decay, so each measure turns the questionnaire's wording into a computable graph statistic. The measures do the work of translating psychological cohesion into an evaluation yardstick that existing structural metrics cannot provide.","core_discovery":"The paper claims that no current community search algorithm effectively identifies psychologically cohesive communities in online social networks, and that there is no clear correlation between structural cohesiveness and psychological cohesiveness. To reach this conclusion, it adapts the social-psychology construct of group cohesion into five quantitative measures and applies them to communities returned by eight algorithms for the same query nodes across four real-world networks. The evaluation shows that algorithms differ sharply in what they retrieve for one query, that learning-based methods often return weak or disconnected results, and that high structural density does not translate into high psychological cohesion.","pith_inferences":["The five measures could be validated against human ratings: ask members of retrieved communities how cohesive they feel, then compare those ratings with EI, SIT, CED, GIP, and GID; this would tell whether the negative result is real or an artifact of the mapping.","Because the measures depend on sentiment labels, switching the sentiment model changes some polarity scores; using richer sentiment models or emotion dimensions might alter which algorithms look cohesive.","The same psychology-informed measures could be repurposed as supervision signals for community detection, not just evaluation, turning the finding into a design principle.","The absence of ground truth built on cohesion theory is itself a finding: existing benchmark communities are structural, which may systematically bias algorithm comparisons."],"forward_implications":["If structural and psychological cohesion are uncorrelated, then optimizing k-core, k-truss, or similar density objectives will not by itself produce communities people experience as cohesive.","Evaluation frameworks for community search should include psychology-grounded measures in addition to structural quality and ground-truth overlap.","Learning-based community search needs richer features than user identifiers; current representations lack the interaction and sentiment signals these measures rely on.","Future algorithms could treat the five measures as optimization objectives or constraints when searching for human-centered communities.","Creating ground-truth communities annotated using group-cohesion theory would allow direct validation of both the measures and the algorithms."],"supporting_citations":[{"why":"Supplies the community search problem definition, the structural cohesiveness metrics, and the evaluation conventions being challenged.","marker":"[34]"},{"why":"Provides the definition of cohesion as a dynamic process reflected in a group's tendency to stick together, which anchors the new measures.","marker":"[17]"},{"why":"Supplies the original Group Environment Questionnaire items that are adapted into the five psychology-informed measures.","marker":"[20]"},{"why":"Provides ALS, one of the evaluated temporal community search algorithms.","marker":"[63]"},{"why":"Provides WCF-CRC, a dynamic-network community search algorithm included in the evaluation.","marker":"[92]"},{"why":"Provides ST-Exa, a size-constrained algorithm that performs well structurally and is used in the parameter-sensitivity analysis.","marker":"[107]"},{"why":"Provides TransZero-LS and TransZero-GS, the learning-based community search algorithms whose weak results drive part of the negative finding.","marker":"[94]"},{"why":"Supplies the lexicon-based sentiment analysis baseline used to check whether sentiment labeling choices affect the cohesiveness measures.","marker":"[48]"}],"fun_headline_variants":["Structural density fails to predict group cohesion","No algorithm finds psychologically cohesive groups","Community search: density isn't cohesion","Psychological cohesion eludes all community search","Structural and psychological cohesion don't align"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusion assumes that its five formulas faithfully capture what people mean when they say a group feels cohesive; if the questionnaire items are not actually represented by sentiment-weighted interaction counts, then the failure belongs to the measures rather than to the algorithms.","fun_headline_variants_meta":{"raw":{"variants":["Structural density fails to predict group cohesion","No algorithm finds psychologically cohesive groups","Community search: density isn't cohesion","Psychological cohesion eludes all community search","Structural and psychological cohesion don't align"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000303,"raw_usage":{"total_tokens":1674,"prompt_tokens":807,"completion_tokens":867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":423,"completion_tokens_details":{"reasoning_tokens":807}},"tokens_in":423,"tokens_out":867,"duration_ms":7848,"temperature":1.0,"reasoning_tokens":807,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:50:38.550535+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect self-reported cohesion ratings from the members of communities returned by these algorithms on the same datasets, then correlate those ratings with the five measures and with k-core or k-truss density. A strong positive correlation between human ratings and structural density would directly contradict the no-correlation claim, while a strong correlation with the proposed measures would support them as operationalizations.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides ALS, one of the evaluated temporal community search algorithms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides ST-Exa, a size-constrained algorithm that performs well structurally and is used in the parameter-sensitivity analysis."}],"review_version":1}