{"id":"ca958aef-e39f-4b98-a79c-190c710cc79f","arxiv_id":"1909.01866","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"A tutorial showing, through a synthetic hiring example, that covariate shift, sample selection bias, and imbalance bias in training data can be learned by a neural network and visualized interactively.","lead":"This paper is a tutorial that explains how bias enters machine learning models through training data, using a synthetic job-recruiting example. It demonstrates three types of bias with interactive parallel-coordinate and relevance-visualization plots.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The interactive demonstrations rely on an unspecified neural network and LRP implementation; the illustrated bias effects could be artifacts, though the central claim is independently supported.","rationale":"The reader identified the unspecified neural network and LRP attributions as the weakest assumption, and I agree that this is the point where the paper's demonstration is least secure. However, I do not see this as a fatal flaw in the central claim, because the paper is explicitly an expository tutorial, not a research contribution claiming a new empirical result. The general statement that biased data can propagate into model decisions is supported by standard references and is not in dispute. The demonstration's lack of reproducibility is a genuine limitation for an educational artifact that asks readers to draw conclusions from interactive plots, but it does not invalidate the paper's core message. Therefore, the appropriate verdict remains UNVERDICTED as an educational piece rather than a research contribution, and no change to the reader's verdict is needed.","tokens_in":7212,"tokens_out":4483,"duration_ms":51342,"concrete_test":"Obtain the associated Python code or reconstruct the model from an explicit specification, then train the network on the synthetic covariate-shift dataset with 10 different random seeds and recompute the LRP attributions using a second propagation rule (e.g., epsilon with epsilon=0.01 instead of the default). If the relevance ranking of Pytorch relative to Python is not consistently lower across seeds and LRP variants, the Section 3.1 demonstration is not robust; similarly, check whether the university relevance pattern in Section 3.2 survives these variations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's stated goal is to illustrate how biased data affects a machine learning model, and the interactive plots in Sections 3.1–3.3 are the primary vehicle for this illustration. The load-bearing assumption is that the displayed model behavior and LRP attributions faithfully represent what a typical trained model would do on the synthetic hiring task. This assumption is not secured: the paper never specifies the network architecture, activation functions, layer sizes, training procedure, hyperparameters, or the exact LRP propagation rule used. Without these details, a reader cannot tell whether the observed patterns—e.g., Pytorch being less relevant under covariate shift, or university being highly relevant under sample selection bias—are robust properties of models trained on biased data or idiosyncratic to one arbitrary configuration. The paper even includes a side note in Section 3.1 admitting that the network failed to learn the intended two-skill rule for some in-distribution applicants, which suggests the chosen model may be poorly tuned or misspecified; if so, the attribution differences could reflect optimization artifacts rather than the injected bias. That said, the central claim that biased training data can bias a model is well-established and cited in Section 5; the demonstration is illustrative, not the sole support. Thus the concern weakens the paper's pedagogical credibility but does not overturn its main message.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This short paper argues—correctly and in an accessible way—that machine learning models can inherit bias from biased training data. The authors define three types of bias (covariate shift, sample selection bias, and imbalance bias), introduce a synthetic hiring scenario in which the ground-truth label is based on a simple two-of-four-skills rule, and present interactive parallel-coordinate and LRP-relevance visualizations intended to show how each bias type distorts the predictions and feature attributions of a trained neural network. The paper closes with a discussion of countermeasures and a list of further reading.","tokens_in":7503,"tokens_out":4755,"duration_ms":53072,"significance":"The paper addresses a topic of clear importance: understanding and detecting bias in machine learning systems. Its accessible framing and the use of interactive visualizations as a pedagogical device are commendable, and the central qualitative claim is well supported by prior literature cited in Section 5. However, the manuscript's own demonstration is not independently verifiable: it omits the data-generation details, the network architecture, the training procedure, and the exact LRP rule, and it provides no quantitative evaluation. If these gaps are filled, the paper could serve as a useful educational resource; as written, it reads more like an extended abstract for a workshop demonstration than a standalone journal article. The authors should be credited for clearly separating the three bias types and for explicitly stating that a model may or may not pick up a given bias, a nuance that is often missing in popular treatments.","major_comments":[{"comment":"The demonstration's load-bearing assumption is that the displayed behavior of the neural network and its LRP attributions faithfully represent what a typical model trained on the biased synthetic data would do. The paper does not specify the network architecture, activation functions, layer sizes, training procedure, hyperparameters, the exact LRP propagation rule, or the data-generation parameters (e.g., the number of samples, score distributions, or the university–skill correlation). Without these details, the plots in Sections 3.1–3.3 cannot be reproduced or independently assessed, and the observed relevance patterns could be idiosyncratic to one arbitrary configuration. Please provide a complete description of the model and data in an appendix, and include a stable URL or DOI for the code and interactive plots.","section":"§3.1–3.3, Figures 1–7"},{"comment":"The side note in Section 3.1 states that the neural network 'did not pick up' the intended two-skill rule even for some in-distribution applicants. This admission is important: if the network fails to learn the ground-truth rule in the unbiased case, then the differences in LRP relevance attributed to covariate shift in Figures 2 and 3 could instead be effects of underfitting, misoptimization, or an unsuitable LRP rule. The paper should report the model's training and test accuracy on the unbiased data, a confusion matrix or quantitative comparison of model decisions against the ground-truth HR rule for each bias scenario, and a sanity check of the LRP attributions on correctly classified examples.","section":"§3.1, side note"},{"comment":"The conclusion states that 'we have shown that if the data is biased, the resulting model may be biased, too.' As a scientific claim, this is not established by the preceding demonstration: the bias is injected into the synthetic data by design, and the observed model behavior is described only qualitatively. The claim is independently supported by the literature cited in Section 5, so it is not incorrect, but the paper should explicitly position the synthetic experiments as illustrative rather than as a proof or systematic study. Adding a small quantitative experiment with multiple random seeds and architectures would also strengthen the robustness claims.","section":"§4, Conclusions"}],"minor_comments":[{"comment":"The sentence 'we develop interactive plots to visualizing the bias' should read 'to visualize the bias.'","section":"Abstract"},{"comment":"The definition of covariate shift as 'one of the features is not covered uniformly in the dataset' is imprecise and could be confused with missing data. Please define it as a change in the marginal distribution P(X) between the training and test/deployment populations, while P(Y|X) remains the same.","section":"§2, first bullet"},{"comment":"Several places use 'Lets' instead of 'Let's' (e.g., 'Lets take a simplified...', 'Lets make this more concrete'), and 'HRs decision' should be 'HR's decision.'","section":"§3, Figures and text"},{"comment":"The caption of Figure 7 reads 'Interactive Visualization of Relevance of each Feature,' but the figure actually shows pie charts of invite/reject ratios and correct/incorrect decisions. Please correct the caption to match the content.","section":"Figure 7 caption"},{"comment":"The interactive plots are not self-contained in a static paper: the reader is directed to a workshop homepage or to 'run our python code' without a direct URL or repository identifier. Please include a stable link and, ideally, a supplementary PDF with screenshots or static versions of the key states of the plots.","section":"§3"},{"comment":"Reference [7] is incomplete ('In CVPR.' with no year or venue details), and some other entries lack pages or years. Please complete all references for reproducibility of the literature survey.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is essentially a tutorial/demonstration paper. Its contribution is pedagogical, and the interactive visualizations may be valuable for teaching, but the technical content is currently too thin for a full-length journal article: no model details, no quantitative results, and no accessible artifact. The gaps are fixable within the manuscript's scope, so I recommend major revision rather than rejection. The authors should also consider whether a workshop venue or a short/tutorial format would be more appropriate than a regular research paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an expository tutorial, not a research paper. It explains three common dataset biases, illustrates each with a synthetic hiring example and interactive plots, and uses LRP to show how a network's feature relevance shifts under each bias. There is no new method or empirical result. The interactive artifact itself is the contribution, and as a teaching aid it is reasonably effective.\n\nWhat it does well: the prose is plain and approachable, the three-way distinction between covariate shift, sample selection bias, and class imbalance is cleanly drawn, and the authors are honest about the limits — they explicitly label the article a gentle introduction in Section 5, and the side note in Section 3.1 admits the network did not pick up the intended two-skill rule for some in-distribution applicants. That kind of transparency deserves credit. The citation list is appropriate and includes the standard fairness and explainability references.\n\nSoft spots: the central demonstration is not independently verifiable from the text. The paper never specifies the network architecture, training procedure, hyperparameters, or the exact LRP variant, so the reader cannot tell whether the observed relevance patterns are robust or idiosyncratic to one arbitrary model. The interactive plots are not embedded in the paper; they live behind a footnote to a workshop page. On arXiv, the paper is incomplete as a standalone artifact. The stress test's concern about the network being poorly tuned is plausible given the side note, and it does undercut the pedagogical credibility slightly, but not the main message: biased training data can bias a model, and that claim is well-cited prior knowledge. The demo is illustrative, not load-bearing.\n\nOne more thing: the paper makes a claim in the conclusion that the model may be biased, and says countermeasures depend on domain-expert collaboration. That is fine. But the introduction's framing that ML could help overcome bias is a bit straw-man; the 'hope' is not a serious position. Minor.\n\nBottom line: for a research venue this is not a contribution, but for a workshop or an educational venue it is a solid teaching artifact. It deserves a serious referee if the interactive material is reviewed along with the text, and the authors should be asked to document the model and make the demos reproducible.","headline":"A clear, honest tutorial on dataset bias with interactive demos, but the demos are not reproducible from the text and the paper adds no research contribution beyond the artifact.","tokens_in":7961,"tokens_out":2653,"would_cite":false,"duration_ms":25203,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bias enters a machine learning model through the data it is trained on, and it can be made visible with interactive visualizations.","keywords":["bias in machine learning","Visualization in Explainable AI","dataset bias","covariate shift","sample selection bias","imbalance bias","parallel coordinates","layer-wise relevance propagation"],"falsifier":"Train the same synthetic hiring task with several different network architectures and random seeds, hold the biased data fixed, and check whether the LRP relevance patterns (for example, university importance under sample selection bias) still appear in every run; if the patterns vanish in most runs, the paper's demonstration does not support its general conclusion.","tokens_in":7048,"feed_emoji":"🤖","tokens_out":4856,"duration_ms":47620,"temperature":0.7,"pith_summary":"This paper makes a conceptual point: a machine learning model can be biased because the data used to train it is biased. Using a simplified hiring scenario with four skill scores and a university attribute, the paper shows how three types of bias—covariate shift, sample selection bias, and imbalance bias—change both the model's predictions and the features it relies on. Interactive parallel-coordinates plots let a reader see the biased patterns in the data, and layer-wise relevance propagation shows which features drive individual decisions. The takeaway is that biased data can propagate into model behavior, so inspecting data and models before deployment is a necessary safeguard.","feed_headline":"Bias enters machine learning models via training data","feed_subtitle":"Interactive demos show three types of data bias distorting model decisions and explanations.","key_machinery":"The demonstration is built on two visual inspection tools. A parallel-coordinates plot (each feature is a vertical axis; each applicant is a polyline connecting their score on each axis) lets the reader brush value ranges and see which combinations of features occur together in the training and test data. Layer-wise relevance propagation (LRP) computes, by a backward pass through the network, how much each input feature contributed to the model's decision for a given applicant, displayed as a relative relevance bar chart. These two tools carry the argument: the plots show the bias in the data, and the LRP attributions show the bias in the learned model.","core_discovery":"The paper's central claim is that bias enters a machine learning model through the data used to build it, and that this is not a rare corner case: any of three identifiable dataset distortions can be picked up by the model. In the synthetic hiring example, when a skill (Pytorch) was never high in the training data, the model became unreliable for applicants with high Pytorch scores; when a university was spuriously correlated with the invite label, the model treated university as a relevant feature; and when the reject class dominated the training set, the model's predictions skewed toward rejection even on a balanced test set. The paper argues that such effects can be made visible and understandable through interactive visualization of the data and explainability analysis of the model, making it possible for domain experts to recognize and counter the bias.","pith_inferences":["The paper's demonstration uses one unspecified network; the three biases should appear for any model class, but the exact relevance patterns are likely architecture-dependent, so the qualitative claim is more robust than the specific numbers.","In practice, the three bias types often compound, so the clean separation used in the demos may not hold; detection tools would need to disentangle confounded sources.","A testable extension is to train a deliberately debiased model (for example, resampling to balance classes or removing the university feature) and check whether LRP attributions drop the spurious feature, turning the visual explanation into a diagnostic for mitigation."],"forward_implications":["If training data is biased, the resulting model's predictions can be biased even when the test population has changed; the Pytorch example shows a model failing on a pattern it never saw.","Models can pick up spurious correlations, such as university identity, and treat them as relevant indicators rather than the semantically meaningful skills.","Class imbalance in the training set can shift the model's decision boundary so that its rejection rate is distorted relative to the true label distribution.","The same inspection techniques (parallel coordinates plus LRP) can be used to detect bias in the training data and in the model before deployment."],"supporting_citations":[{"why":"Supplies the parallel-coordinates plot used to visualize the training and test data in all three bias scenarios.","marker":"[1]"},{"why":"Supplies the layer-wise relevance propagation algorithm that produces the per-feature relevance scores in the interactive explanations.","marker":"[3]"},{"why":"Documents real-world dataset bias (gender classification) that motivates the claim that biased training data causes unfair model behavior.","marker":"[4]"},{"why":"Shows that even popular benchmark datasets are biased, supporting the claim that dataset bias is a general problem rather than an artifact of the synthetic example.","marker":"[7]"}],"fun_headline_variants":["Bias in ML? It's baked in by your training data","Three ways skewed data skews machine learning","Interactive plots reveal how data bias poisons models","Your training data is the sneaky source of ML bias","The data trap that makes AI decisions unfair"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The demonstrations assume that the neural network and the explanation scores shown in the plots are representative of machine learning models generally.","fun_headline_variants_meta":{"raw":{"variants":["Bias in ML? It's baked in by your training data","Three ways skewed data skews machine learning","Interactive plots reveal how data bias poisons models","Your training data is the sneaky source of ML bias","The data trap that makes AI decisions unfair"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1253,"prompt_tokens":785,"completion_tokens":468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":401,"completion_tokens_details":{"reasoning_tokens":393}},"tokens_in":401,"tokens_out":468,"duration_ms":5602,"temperature":1.0,"reasoning_tokens":393,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:33:32.225971+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same synthetic hiring task with several different network architectures and random seeds, hold the biased data fixed, and check whether the LRP relevance patterns (for example, university importance under sample selection bias) still appear in every run; if the patterns vanish in most runs, the paper's demonstration does not support its general conclusion.","supporting_citations":[{"cited_title":"Parallel coordinates: a tool for visualizing multi-dimensional geometry","cited_arxiv_id":null,"evidence_quote":"Supplies the parallel-coordinates plot used to visualize the training and test data in all three bias scenarios."},{"cited_title":"On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation","cited_arxiv_id":null,"evidence_quote":"Supplies the layer-wise relevance propagation algorithm that produces the per-feature relevance scores in the interactive explanations."},{"cited_title":"Gender shades: Intersectional accuracy dispar- ities in commercial gender classification","cited_arxiv_id":null,"evidence_quote":"Documents real-world dataset bias (gender classification) that motivates the claim that biased training data causes unfair model behavior."},{"cited_title":"Unbiased look at dataset bias","cited_arxiv_id":null,"evidence_quote":"Shows that even popular benchmark datasets are biased, supporting the claim that dataset bias is a general problem rather than an artifact of the synthetic example."}],"review_version":1}