{"id":"71149003-8c9c-4aa5-ae4e-081c8f71a1d8","arxiv_id":"2606.21647","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"ConcernBERT is a BERT embedding model trained with triplet loss on class membership to encode concern-level semantics in Java entities, evaluated by recovering original classes from merged unlabeled groups on a new dataset of over 2M files, outperforming existing models.","lead":"ConcernBERT trains a BERT model on Java code entities using triplet loss and class membership labels to embed methods and attributes that share responsibilities closer together. A smart generalist might read it to see whether AI techniques can turn abstract software design principles into measurable, automatable tasks like refactoring.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Synthetic class-merging recovery may not validate concern-level semantics in unmodified code","rationale":"The reader's weakest_assumption matches the load-bearing evaluation assumption exactly; the abstract provides no counter-evidence, so the concern stands and the UNVERDICTED verdict is appropriate.","tokens_in":1731,"tokens_out":258,"duration_ms":12867,"concrete_test":"Construct a held-out test set of 200 manually annotated cohesive method groups from 10 open-source Java projects (groups chosen by developers without reference to class boundaries); measure ConcernBERT's clustering F1 against the class-merging baseline and report whether the reported advantage persists.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (higher performance demonstrates encoding of concern-level semantics) depends on the evaluation proxy: artificially merging methods from ≥2 classes into unlabeled groups and measuring recovery of original memberships. Because training also uses class-membership context via triplet loss, strong results on this task could arise from learning class-boundary signals rather than responsibility semantics; the proxy never tests unmodified real-world code where concerns may cross classes or require non-class cues. No independent validation (e.g., human-annotated concerns or downstream refactoring tasks) is described in the abstract.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes ConcernBERT, a BERT-based embedding model trained at the entity level using triplet loss with class-membership context to learn responsibilities and concerns in Java code. It contributes a large-scale dataset of over two million Java files from more than six thousand repositories. Evaluation consists of merging methods from two or more classes into unlabeled groups and testing recovery of the original class memberships; the paper claims ConcernBERT achieves significantly higher performance than existing models, demonstrating effectiveness at encoding concern-level semantics for downstream tasks including architecture recovery, extract class refactoring, and cohesion measurement.","tokens_in":1841,"tokens_out":402,"duration_ms":19997,"significance":"If the performance gains are substantiated with quantitative metrics and the class-recovery proxy is shown to measure concern semantics rather than class boundaries, the work could operationalize separation of concerns and single-responsibility principles in an actionable, data-driven manner. The contributed large-scale dataset would be a reusable asset for the software engineering community.","major_comments":[{"comment":"Abstract: the assertion that ConcernBERT 'achieves significantly higher performance than existing models' supplies no quantitative metrics, baseline descriptions, statistical significance tests, error bars, or dataset construction details, making it impossible to assess whether the data support the central claim.","section":"Abstract"},{"comment":"Evaluation (class recovery task, as described): the central claim that higher recovery performance demonstrates encoding of concern-level semantics depends on the proxy of artificially merging methods from ≥2 classes and recovering original memberships. Because training also uses class-membership context via triplet loss, strong results on this task could arise from learning class-boundary signals rather than responsibility semantics; the proxy never tests unmodified real-world code where concerns may cross classes or require non-class cues. No independent validation (human-annotated concerns or downstream refactoring tasks) is described.","section":"Evaluation"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our manuscript. The comments highlight important issues regarding the abstract's claims and the evaluation design. We respond to each major comment below and indicate the revisions we will make.","responses":[{"response":"We agree that the abstract should include quantitative support for its claims rather than relying on a qualitative statement. In the revised version, we will update the abstract to report key performance metrics (such as accuracy or F1 improvements over baselines), reference the statistical significance of results, and briefly note the scale of the dataset and evaluation setup. Full experimental details remain in the body of the paper.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the assertion that ConcernBERT 'achieves significantly higher performance than existing models' supplies no quantitative metrics, baseline descriptions, statistical significance tests, error bars, or dataset construction details, making it impossible to assess whether the data support the central claim."},{"response":"We acknowledge that the class-recovery proxy is closely aligned with the class-membership signal used during training, which raises a legitimate question about whether the model is primarily capturing class boundaries rather than broader concern semantics. The task is intended to evaluate the model's ability to group entities by learned responsibility in an unlabeled setting, consistent with the single-responsibility principle. However, we agree this does not constitute fully independent validation on unmodified code or human-annotated concerns. In the revision, we will add an explicit discussion of this limitation in the evaluation section, clarify the assumptions of the proxy, and outline directions for future validation using downstream tasks such as refactoring.","revision_made":"partial","referee_comment":"[Evaluation] Evaluation (class recovery task, as described): the central claim that higher recovery performance demonstrates encoding of concern-level semantics depends on the proxy of artificially merging methods from ≥2 classes and recovering original memberships. Because training also uses class-membership context via triplet loss, strong results on this task could arise from learning class-boundary signals rather than responsibility semantics; the proxy never tests unmodified real-world code where concerns may cross classes or require non-class cues. No independent validation (human-annotated concerns or downstream refactoring tasks) is described."}],"tokens_in":1394,"tokens_out":473,"duration_ms":18146,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"ConcernBERT trains a BERT model at the entity level with triplet loss, using class membership to pull methods and attributes from the same class closer in embedding space. The authors also release a dataset of over two million Java files from more than six thousand repositories.\n\nThe practical motivation is sound. Tasks like god-class splitting and cohesion measurement need better ways to group entities that share a responsibility, and a public large-scale dataset helps the subfield.\n\nThe evaluation design is the main soft spot. The model is trained on class labels and then tested by merging methods from different classes and measuring recovery of those same labels. This setup can reward learning class-specific cues rather than responsibility semantics that would apply to unmodified code where concerns often cross class lines. The abstract supplies no numbers, baselines, or statistical details, so the claim of significantly higher performance cannot be assessed yet.\n\nThe paper is aimed at software engineering researchers who build or evaluate tools for refactoring and architecture recovery. Anyone working on code embeddings or concern identification will find the dataset and training approach worth examining.\n\nIt should go to peer review. The dataset is a concrete contribution and the method is clearly described; the authors will need to strengthen the evaluation and report full results, but the work is grounded enough to merit referee time.","headline":"ConcernBERT adds a large Java dataset and an entity-level BERT+triplet-loss setup, but the class-recovery evaluation does not clearly separate concern semantics from class-boundary signals.","tokens_in":2385,"tokens_out":338,"would_cite":false,"duration_ms":11516,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"ConcernBERT learns to group code entities by shared responsibilities using class membership and triplet loss.","keywords":["concern identification","class membership","software cohesion","BERT embeddings","triplet loss","extract class refactoring","architecture recovery","responsibility grouping"],"falsifier":"If ConcernBERT fails to match or exceed baselines when tested on human-labeled cohesive groups drawn from unaltered production code, the claim that it encodes concern-level semantics would not hold.","tokens_in":2620,"feed_emoji":"","tokens_out":608,"duration_ms":14543,"temperature":0.7,"pith_summary":"The paper introduces ConcernBERT to turn the abstract idea of separation of concerns into a learnable property of code. It trains a BERT model on methods and attributes so that entities belonging to the same class end up close in embedding space, treating class membership as a signal for common responsibility. Evaluation creates artificial mixed groups by merging methods from different classes and measures how well the model recovers the original memberships. The model outperforms prior approaches on a dataset of millions of Java files, positioning the embeddings as a basis for identifying cohesive concern groups.","feed_headline":"ConcernBERT recovers original class groups from merged methods","feed_subtitle":"A triplet-loss BERT model trained on class labels outperforms priors at assigning methods back to their responsibility groups on a dataset o","key_machinery":"ConcernBERT, a BERT-based embedding model trained with triplet loss on class-membership context to position methods and attributes by shared concerns.","core_discovery":"ConcernBERT is a BERT-based embedding model trained at the entity level that uses triplet loss to directly optimize the relative positioning of methods and attributes in the embedding space, and uses class-membership context to learn responsibilities and concerns, recovering original class memberships from merged groups with significantly higher performance than existing models.","pith_inferences":["The class-membership signal might allow detection of god classes by identifying entities that do not cluster tightly with any single concern.","Similar triplet-loss training on other labeled groupings, such as package or module boundaries, could extend the method to additional design principles.","If the embeddings prove stable, they could be integrated into static analysis tools to flag low-cohesion areas during development."],"forward_implications":["Embeddings from ConcernBERT can support architecture recovery by clustering entities according to learned concerns.","Extract class refactoring can use the model to propose splits based on responsibility groupings rather than manual review.","Cohesion metrics can be derived from distances in the learned embedding space.","The large-scale Java dataset enables training models that generalize across many repositories for concern detection."],"fun_headline_variants":["ConcernBERT embeds methods with triplet loss on class context","Triplet-loss BERT recovers class memberships from mixed methods","BERT model learns concerns by positioning methods and attributes","ConcernBERT beats priors in assigning methods to responsibility groups"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Artificially merging methods from two or more classes and testing recovery of original memberships is a valid proxy for identifying naturally cohesive responsibility groups in real unmodified code.","fun_headline_variants_meta":{"raw":{"variants":["ConcernBERT embeds methods with triplet loss on class context","Triplet-loss BERT recovers class memberships from mixed methods","BERT model learns concerns by positioning methods and attributes","ConcernBERT beats priors in assigning methods to responsibility groups"]},"model":"grok-4.3","cost_usd":0.004537,"raw_usage":{"total_tokens":2161,"prompt_tokens":639,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":45365500,"prompt_tokens_details":{"text_tokens":639,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1462,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":639,"tokens_out":60,"duration_ms":8170,"temperature":1.0,"reasoning_tokens":1462,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T13:25:09.577904+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If ConcernBERT fails to match or exceed baselines when tested on human-labeled cohesive groups drawn from unaltered production code, the claim that it encodes concern-level semantics would not hold.","supporting_citations":[],"review_version":1}