{"id":"3e071c65-7bce-44e0-9b18-86c2d116c961","arxiv_id":"2508.01492","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"OpenLambdaVerse presents a filtered, current dataset of open-source Serverless Framework applications with AWS Lambda functions, plus a characterization of their architecture and security posture.","lead":"This paper builds a new dataset of GitHub repositories that use the Serverless Framework with AWS Lambda functions, and analyzes their size, languages, triggers, and security practices. It updates the earlier Wonderless dataset with fresh filtering steps and current repositories, giving researchers and practitioners a snapshot of how serverless applications are actually built.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset representativeness rests on filtering steps the abstract does not describe; an audit of precision and recall is needed before the characterization can be trusted.","rationale":"This stress-test is based on the abstract only, as the full text was not available. In good faith, the paper's goal is clear: construct a dataset and use it to characterize current serverless application usage. The most load-bearing condition for that goal is that the final dataset is a faithful sample of repositories that genuinely use the Serverless Framework with AWS Lambda. The reader's weakest-assumption analysis identified exactly this: the filtering steps must not systematically bias the sample. I agree. The abstract provides no evidence about the filter's precision or recall, nor about thresholds or deduplication, so the validity of the dataset is unverified. However, the absence of evidence does not make the claim false; it makes the correctness risk unknown. The proposed concrete test—a manual audit with inter-rater agreement—would resolve this directly. If the audit shows high precision and recall, the dataset is credible; if not, the analytical findings become suspect. Since no new information beyond the abstract is available, the appropriate verdict remains UNVERDICTED, matching the reader's low-confidence assessment. I am not raising an objection to the methodology's internal logic, only to the unsubstantiated representativeness assumption that the entire empirical contribution depends on. The paper's contribution is potentially valuable—an up-to-date open dataset of serverless apps is useful—but the value can only be assessed once the filtering pipeline is transparent and validated.","tokens_in":710,"tokens_out":2214,"duration_ms":29254,"concrete_test":"Take a random sample of 200 repositories from the final dataset and 200 repositories from the broader GitHub population that contain serverless.yml or mention 'Serverless Framework', then have two independent human raters classify each as 'true positive' or 'false positive' for the stated inclusion criteria (Serverless Framework, one or more AWS Lambda functions). Compute precision on the included set and recall on the broader set. If either falls below 90%, the filter is systematically biased and the reported distributions should be recomputed or reported as conditional on the filter's error profile.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that OpenLambdaVerse contains current GitHub repositories using the Serverless Framework with AWS Lambda, and that the subsequent analysis characterizes real-world serverless applications. Every reported distribution—size, complexity, runtimes, triggers, maturity, security—inherits the sampling validity of the multi-step filter. The abstract says the pipeline builds on Wonderless and applies 'multiple new filtering steps,' but gives no details on those steps, no exclusion criteria, and no validation against a labeled ground truth. This is a missing-evidence risk, not an internal inconsistency. A filter that relies on serverless.yml presence could miss repositories using other Infrastructure-as-Code tools or include unrelated repos with coincidental YAML; a filter that requires AWS Lambda specifically may exclude multi-provider or non-AWS serverless projects, biasing the characterization toward a narrow subset. Without precision/recall numbers or a manual audit, the analysis cannot be distinguished from artifacts of the filter. Since the full text is unavailable, this is not an assertion of error, but the load-bearing assumption is not yet supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces OpenLambdaVerse, a dataset of GitHub repositories that use the Serverless Framework with at least one AWS Lambda function, constructed by extending the Wonderless methodology with several new filtering steps. The paper then uses this dataset to characterize serverless application size, complexity, programming languages, runtimes, triggers, project maturity, and security practices. The abstract presents the dataset and the analysis as contributions but provides no details of the filtering procedure or any validation of its accuracy.","tokens_in":1004,"tokens_out":1663,"duration_ms":20409,"significance":"If the dataset is representative of real-world serverless applications, OpenLambdaVerse would be a timely and useful resource for both practitioners and researchers, addressing a genuine gap in up-to-date empirical data on serverless workloads. The authors are to be credited for targeting a concrete and relevant problem and for building on an established methodology (Wonderless). However, the significance hinges entirely on the validity of the filtering pipeline, which the abstract does not describe or validate; the claimed value cannot be assessed from the abstract alone.","major_comments":[{"comment":"The abstract states that the methodology builds on Wonderless and applies 'multiple new filtering steps,' but it does not specify what these steps are, what inclusion/exclusion criteria they use, or how their accuracy is verified. Every subsequent characterization (size, complexity, runtimes, triggers, maturity, security) is a distribution over the resulting sample, so any systematic selection bias in these filters would propagate directly into the conclusions. Without a precision/recall evaluation against labeled ground truth or a documented manual audit, the analysis cannot be distinguished from artifacts of the filter. This is the load-bearing assumption of the paper and it is currently unsupported.","section":"Abstract, first paragraph"},{"comment":"The phrase 'current GitHub repositories' is undefined: there is no snapshot date, no definition of 'current' (e.g., last commit within a time window), and no statement about how forks, archived repositories, or inactive projects are treated. This matters directly for the claimed insights on 'project maturity' and on the 'state-of-the-art,' since a repository that is current by one criterion may be stale by another. The omission is a completeness gap in the dataset-defining criteria.","section":"Abstract, first paragraph"},{"comment":"The abstract claims insights into 'security practices (or lack of)' and the 'state-of-the-art,' but the sample is restricted to repositories using the Serverless Framework and AWS Lambda. Security practices observed in this subset may reflect the Serverless Framework's defaults, templates, or documentation rather than general serverless practice, and multi-provider or non-AWS serverless applications are excluded by design. The paper should explicitly bound the scope of its conclusions to this stack and discuss how the single-framework/single-provider filter could bias the characterization.","section":"Abstract, second paragraph"}],"minor_comments":[{"comment":"The phrase 'how are the functions triggered' is grammatically awkward in the context of a declarative list; consider 'how functions are triggered.'","section":"Abstract, second paragraph"},{"comment":"The abstract gives no quantitative information about the dataset, such as the number of repositories, functions, or unique applications, which would help readers gauge the scale of the resource.","section":"Abstract"},{"comment":"The abstract does not mention where the dataset will be hosted or how it can be accessed, which is important for a contribution that is intended as a resource for practitioners and researchers.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":"This review is based on the abstract only because the full text was not available. The central concern—the unvalidated filtering pipeline—may be fully addressed in the body of the paper, so I cannot recommend rejection, but I also cannot recommend acceptance without seeing the methods and validation. A full review with the complete manuscript is necessary to judge whether the dataset construction and analysis are sound. If the paper is being considered based on the abstract alone, I would advise the editor to request the full text and specifically ask the authors to provide precision/recall or a manual audit of the filtering steps."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate dataset paper that honestly builds on Wonderless rather than pretending to be brand-new. The new thing is the 2025 GitHub sample and the characterization of it, which is exactly the kind of updated empirical resource the serverless community can use.\n\nWhat the paper does well: it identifies a real gap (stale datasets), credits the prior methodology, and promises a multi-dimensional view of serverless applications—size, complexity, languages, runtimes, triggers, maturity, security. If the full text delivers that, the dataset will be a useful reference.\n\nThe soft spot is the one in the stress-test note: the abstract says the pipeline applies 'multiple new filtering steps' but does not describe them. Every distribution in the analysis inherits the sampling validity of those steps. That is a missing-evidence problem, not a proven error. The reader's soundness score of 3 is fair for an abstract-only review.\n\nI would not desk-reject this. A serious referee needs to see the full methodology: exactly how the filters work, how many repos they exclude, and any manual audit or precision/recall check. If those details are in the paper, it is solid. If not, the fix is to add them. I would not cite the dataset myself until the filtering is shown to be sound, but the paper is the right kind of contribution for the subfield and deserves a full review.\n\nRecommendation: send to peer review, and have the referees push for explicit validation of the filtering pipeline.","headline":"Incremental dataset paper, honestly built on Wonderless; value hinges on filter validation not visible in the abstract—worth review.","tokens_in":1329,"tokens_out":2925,"would_cite":false,"duration_ms":32059,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenLambdaVerse builds a dataset of current GitHub repositories that use the Serverless Framework with AWS Lambda functions and analyzes it to characterize real-world serverless application size, complexity, language and runtime choices…","keywords":["serverless computing","Function-as-a-Service","AWS Lambda","Serverless Framework","dataset","GitHub repository mining","empirical software engineering","Infrastructure-as-Code"],"falsifier":"One could re-run the pipeline on a short time window using independent search terms or a manual audit of a random sample of candidate repositories, and check whether the resulting set closely matches OpenLambdaVerse. If many repositories that the pipeline excludes actually contain AWS Lambda functions and Serverless Framework configurations, or if many included repositories are misclassified, the characterization of sizes, languages, triggers, and security practices would not reliably reflect real serverless applications.","tokens_in":537,"feed_emoji":"⚡","tokens_out":3383,"duration_ms":33088,"temperature":0.7,"pith_summary":"This paper tries to fill a gap in empirical knowledge about serverless computing: there is little up-to-date, directly observable information about how developers actually build serverless applications. The authors create OpenLambdaVerse, a dataset of current GitHub repositories that use the Serverless Framework and contain at least one AWS Lambda function, and then characterize those applications along several dimensions. The value of the contribution, if it works, is an evidence-based snapshot of real-world serverless workloads that practitioners and researchers can use instead of relying on toy examples or vendor documentation.","feed_headline":"New dataset maps real-world AWS Lambda serverless apps","feed_subtitle":"OpenLambdaVerse catalogs Serverless Framework repositories to reveal sizes, languages, triggers, and security gaps.","key_machinery":"The machinery is the dataset construction pipeline: searching GitHub for repositories that use the Serverless Framework, applying multiple filtering steps to keep only applications containing one or more AWS Lambda functions, and then extracting metadata about each repository to quantify size, complexity, language and runtime choices, trigger types, project maturity, and security-related configuration. This pipeline is the central object because every conclusion in the paper rests on the sample it produces.","core_discovery":"The central claim is that OpenLambdaVerse provides a current, curated sample of real-world serverless applications and that analyzing this sample yields a reliable picture of the state of the art in this stack. Specifically, the paper argues that its multi-step filtering methodology identifies repositories that genuinely use the Serverless Framework with AWS Lambda, and that the resulting dataset can be used to describe application size and complexity, the languages and runtimes in use, how functions are triggered, project maturity, and the security practices — or lack of them — in these projects. The discovery, in other words, is the dataset itself plus the characterization it enables, positioned as a successor to earlier serverless datasets.","pith_inferences":["If the dataset is representative, it likely overrepresents small and hobby projects, since public GitHub repositories skew that way; the paper's size and maturity findings should be read as describing public GitHub rather than serverless usage inside companies.","The dataset could be used to train or benchmark automated tools that detect misconfigurations or security smells in serverless configurations, since it provides real-world examples at scale.","A natural extension would be to pair the GitHub metadata with cloud deployment logs or package-level dependency data to connect repository characteristics with runtime behavior and cost, something the paper does not attempt."],"forward_implications":["Researchers get a ready-made, up-to-date dataset for studying real serverless workloads rather than synthetic or outdated examples.","Practitioners can use the distribution of languages, runtimes, and triggers to inform their own architectural choices and migration decisions.","The characterization of security practices reveals how widespread (or rare) certain security configurations are in this stack, highlighting a target for tooling and training.","The dataset provides a baseline for future comparisons as the serverless ecosystem evolves, enabling longitudinal studies.","The filtering methodology can be adapted to other Infrastructure-as-Code frameworks and cloud providers."],"supporting_citations":[],"fun_headline_variants":["Dataset reveals real-world AWS Lambda app patterns","OpenLambdaVerse: real-world serverless app insights","Serverless app analysis: sizes, triggers, and security gaps","New open dataset for serverless architecture research","OpenLambdaVerse: security gaps in serverless apps exposed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the repository search and filtering steps correctly identify the intended population — current GitHub projects that use the Serverless Framework with at least one AWS Lambda function — without systematically under- or over-selecting certain kinds of projects, because any bias in the sample would skew every characterization that follows.","fun_headline_variants_meta":{"raw":{"variants":["Dataset reveals real-world AWS Lambda app patterns","OpenLambdaVerse: real-world serverless app insights","Serverless app analysis: sizes, triggers, and security gaps","New open dataset for serverless architecture research","OpenLambdaVerse: security gaps in serverless apps exposed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1532,"prompt_tokens":896,"completion_tokens":636,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":561}},"tokens_in":512,"tokens_out":636,"duration_ms":7278,"temperature":1.0,"reasoning_tokens":561,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:32:49.922514+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One could re-run the pipeline on a short time window using independent search terms or a manual audit of a random sample of candidate repositories, and check whether the resulting set closely matches OpenLambdaVerse. If many repositories that the pipeline excludes actually contain AWS Lambda functions and Serverless Framework configurations, or if many included repositories are misclassified, the characterization of sizes, languages, triggers, and security practices would not reliably reflect real serverless applications.","supporting_citations":[],"review_version":1}