{"id":"a8314c74-329e-4ab4-a5d5-582e43a73a82","arxiv_id":"2412.13734","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Text2Relight learns to re-light portrait photos from text prompts using a synthetic dataset generated by a three-stage pipeline.","lead":"This paper introduces a system that re-lights a portrait photo according to a text description, such as 'warm candlelight' or 'eerie green glow'. It builds a large synthetic training dataset by generating text prompts, creating matching lighting images, and transferring those lighting conditions to portraits.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on unvalidated text-to-lighting targets: the synthetic 'lighting image' from a few-step text-to-image model is assumed to encode the prompt's lighting, and all quantitative tests use targets from the same pipeline.","rationale":"I read the paper in good faith: the method is clearly described, the data pipeline is ambitious, and the qualitative results are plausible. The strongest claim is that a single image plus a text prompt can convincingly relight both foreground and background while preserving identity. For that claim to hold, the synthesized training targets must actually encode the lighting described by the text. That condition is least secure at the step where a text-to-image model produces an RGB 'lighting image' that is then transferred through learned relighters. No validation of text-to-lighting fidelity is provided, and the quantitative benchmark is generated by the same pipeline, so a model that learns pipeline-specific biases can score well without learning a semantically correct text-to-lighting mapping. This is not an accusation of methodological fraud; it is a standard train/test leakage problem that the paper does not address. The limitation paragraph openly admits spatial and localization failures, which is consistent with this concern. I do not recommend rejection because the user study and qualitative examples provide some independent evidence, but the central claim remains conditional on a real-data validation with known lighting. Therefore my read does not change the verdict: it remains CONDITIONAL, which I encode as UNCHANGED.","tokens_in":17920,"tokens_out":5568,"duration_ms":59904,"concrete_test":"Hold out a set of lightstage-captured identities and their corresponding OLAT/HDR environment maps that were not used to build the 1.5M pairs; for each held-out portrait, install the known environment map as ground-truth relighting, and generate text prompts from the environment map's dominant light directions and colors (e.g., 'warm light from the left'). Run Text2Relight against IP2P, MGIE, and a prompt-agnostic relighting baseline conditioned on the same environment map; compare outputs with the ground truth using LPIPS and an estimated lighting direction/color error (for example, by rendering a chrome ball under the estimated lights). If Text2Relight does not beat the prompt-agnostic relighting baseline on these held-out real relights, the claimed generalization to unconstrained scenes and the text-to-lighting mapping are not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest link is the synthesis target, not the diffusion training itself. In the 'Lighting Image Generation' section, an RGB image is produced from the prompt by a four-step latent consistency model and is used directly as the lighting image; there is no check that this image, or the point lights optimized from it in Eq. S1, actually has the direction, color, or intensity described by the text. The foreground relighter and the background point-light transfer then produce training targets whose lighting is inferred from that image, so text-to-target alignment can be wrong in a way that is invisible to the training loss. This is especially risky for abstract prompts such as 'joyful lighting' or 'sweet cotton candy', where the generated RGB image may be a stylized scene rather than a physically meaningful illumination field. Compounding this, the quantitative evaluation in 'Experiments/Datasets' is built from the same synthesis pipeline: the ground-truth relighting is generated with the same foreground/background relighting components, so Table 1 can reward reproducing artifacts of those components rather than true text-contingent relighting. The limitation paragraph independently confirms that the model is 'sometimes confused to localize the text-specified lighting,' which is exactly what a mislabeled text-lighting training distribution would produce. The user study on real images is suggestive, but preference scores can reflect appealing content changes rather than correctness of the lighting itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Text2Relight, a diffusion-model-based method for single-image portrait relighting guided by a text prompt. To address the lack of paired text-relighting data, the authors build a synthetic dataset of 1.5M pairs by (i) generating diverse text prompts with an LLM under a hand-crafted category hierarchy, (ii) generating a lighting image (RGB or HDR panorama) from the prompt, and (iii) transferring the lighting to portrait foregrounds and backgrounds using image-based relighting and point-light optimization. A pretrained InstructPix2Pix model is fine-tuned on this data with auxiliary tasks (shadow removal, light positioning). Experiments compare the model with IP2P, GLIDE, and MGIE on synthetic test pairs and a user study.","tokens_in":18219,"tokens_out":5522,"duration_ms":45468,"significance":"If validated, the paper would be a significant step toward open-vocabulary, text-controllable relighting of both foreground and background in arbitrary portraits. The data synthesis pipeline is a substantial engineering contribution, and the qualitative results, particularly on real images, are visually convincing. The user study provides independent evidence that users prefer the method over three text-editing baselines. The work also demonstrates useful applications such as shadow removal, light positioning, and background harmonization. However, the quantitative claims are weakened by evaluation on targets generated with the same pipeline used for training, and the text-to-lighting alignment of the synthetic targets is not verified.","major_comments":[{"comment":"The ground-truth relighting targets are synthesized with the same foreground and background relighting components used to create the training data, so Table 1's SSIM, LPIPS, CVS, and FIS measure how well the model reproduces the outputs of those components rather than how well it performs true text-contingent relighting; because the baselines were not trained on this distribution, the comparison is biased. The user study on real images (Table 2) is a useful independent check, but preference scores can reflect appealing content changes rather than lighting correctness, so it does not fully resolve the circularity.","section":"Experiments/Datasets and Table 1"},{"comment":"The RGB lighting image produced by a four-step latent consistency model is used without validation as the conditioning target for foreground relighting and as the source of the point lights optimized in Eq. (S1); for abstract prompts such as \"joyful lighting\" or \"sweet cotton candy\", this image is likely a stylized scene rather than a physically interpretable illumination field, so the text-to-lighting mapping in the training targets can be systematically wrong. The limitation paragraph's admission that the model is \"sometimes confused to localize the text-specified lighting\" is consistent with this concern. I recommend a validation study that checks whether the optimized light positions and colors match the prompt's stated attributes on a held-out set.","section":"Lighting Image Generation"},{"comment":"The comparison to IP2P, GLIDE, and MGIE is not controlled: the baselines are used off-the-shelf with an instruction template, while the proposed model is fine-tuned on 1.5M synthetic relighting pairs and additionally benefits from auxiliary-task data; thus the performance gaps in Table 1 may largely reflect the advantage of in-domain training data. To support the claim of \"outperforms existing text-guided image generation models\", the authors should either fine-tune the baselines on a matched subset of the synthetic data or compare against the same architecture trained without the relighting-specific data, since an ablation is shown for the proposed model but not for the baselines.","section":"Baselines and Experiments"}],"minor_comments":[{"comment":"The FIS value for GLIDE (-0.969) is far below zero and should be explained; also, no error bars or statistical significance tests are provided for any metric, so it is unclear whether the differences in Table 1 and Table 3 are meaningful.","section":"Table 1"},{"comment":"The LLaVA score (LS) is obtained with a model fine-tuned on ChatGPT-generated ratings and a single subjective \"Lighting Adjustment\" question; the calibration and variance of this metric should be reported, and it should be interpreted cautiously.","section":"Metrics"},{"comment":"The point-light transfer uses a \"relative distance\" heuristic to account for depth differences between the lighting image and the target background, but this heuristic is not validated against physical relighting ground truth; a brief analysis of its effect on the synthesized targets would strengthen the pipeline's credibility.","section":"Supplementary, Point Light Reconstruction and Transfer"},{"comment":"The dataset statistics report only the total number of pairs (1.5M); a breakdown by source type (OLAT vs. single image), number of identities, number of unique text prompts, and background diversity would help assess the risk of overfitting and the actual text diversity.","section":"Dataset Summary"},{"comment":"There are minor typographical and formatting issues, such as \"models does not generalize\" in the abstract and inconsistent typesetting of model names like LLaVA; these should be cleaned up.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's quantitative evaluation is the main weakness. The authors are at Adobe Research and have access to internal lightstage data and relighting models, so adding a non-circular evaluation should be feasible. The paper does not release code or data, which hampers reproducibility. The related work section cites several recent relighting methods, but the comparison omits them; while text-guided methods are the direct competitors, a comparison with state-of-the-art single-image relighting (without text) would help contextualize the trade-offs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, novel system for text-guided portrait relighting, but the headline numbers are weaker than they look because the evaluation is partly circular.\n\nWhat's genuinely new is the end-to-end pipeline: hierarchical LLM prompt generation, a text-to-lighting image (RGB or HDR panorama), and separate relighting of foreground and background (image-based model or OLAT rendering, point-light optimization for background). Combining these to build 1.5M synthetic text-relight pairs and fine-tuning InstructPix2Pix with mask conditioning and auxiliary tasks (shadow removal, light positioning) is a real contribution. The qualitative results look good, and the user study on real images shows a strong preference for their output over IP2P, GLIDE, and MGIE. The method is described clearly enough to reproduce, and the supplementary is thorough.\n\nThe soft spots are the evaluation. The photometric metrics in Table 1 compare against targets generated by the same synthesis pipeline used for training, so high SSIM/LPIPS can just mean the model reproduces the pipeline's artifacts. Baselines are not fine-tuned on relighting data, which is an unfair comparison, and there are no error bars or significance tests. More fundamentally, the lighting image from the four-step latent consistency model is assumed to actually encode the prompt's lighting, but no check verifies direction, color, or intensity. The paper's own limitation statement admits the model 'sometimes confused to localize the text-specified lighting,' which is consistent with mislabeled text-lighting training pairs.\n\nStill, the central idea holds up. The user study is independent of the synthesis pipeline and shows real preference. The auxiliary tasks and ablations are sensible. The concern about text-to-lighting alignment is legitimate but addressable—e.g., evaluate on lightstage-captured ground truth or use human judgments on physical correctness.\n\nI'd send this to peer review. It deserves referee time and probably a revision that strengthens the evaluation: external test data, error bars, relighting-specific baselines, and a sanity check on the lighting image. If the authors release code and data, that would help a lot.\n\nWho is it for: anyone working on portrait relighting, text-guided editing, or synthetic data pipelines. I'd read it and would cite it for the data synthesis approach.","headline":"Novel text-driven relighting pipeline with impressive qualitative results, but the quantitative evaluation is partly circular and needs firmer external validation.","tokens_in":18733,"tokens_out":2097,"would_cite":true,"duration_ms":19164,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single portrait photo and a text prompt can relight both the person and the background while preserving identity, using a diffusion model trained on 1.5 million synthetic text-relighting pairs.","keywords":["portrait relighting","text-guided image editing","diffusion models","data synthesis pipeline","point light optimization","hierarchical text generation","shadow removal","light positioning"],"falsifier":"Take portraits captured under known physical lighting, for example lightstage OLAT data, generate prompts that accurately describe that lighting and also prompts that describe incorrect lighting, run the model, and compare the shadow directions, shadow softness, intensity, and color casts in the outputs against the measured ground truth. If the model scores well on its own synthetic test set but cannot reproduce the correct physical lighting on these held-out captures, or cannot distinguish correct from incorrect descriptions, the claim that it has learned a genuine text-to-lighting mapping is falsified.","tokens_in":17754,"feed_emoji":"💡","tokens_out":9813,"duration_ms":76075,"temperature":0.7,"pith_summary":"This paper tries to establish that natural-language text can serve as a direct control signal for portrait lighting, including lighting described by emotion, time, smell, or mood rather than by physical parameters. It introduces Text2Relight, a diffusion model that takes one portrait image, a foreground mask, and a text prompt, and outputs a version of the same scene in which both the person and the background are relighted to match the prompt while identity and content are preserved. Because no scalable dataset of text-to-relighting pairs existed, the paper's central contribution is a synthesis pipeline that manufactures such pairs: a large language model generates diverse light-aware prompts under a hand-crafted category hierarchy, a text-to-image model renders a lighting image for each prompt, and image-based relighting transfers that lighting to the foreground and background separately. Training on 1.5 million pairs, the model reports better structural, perceptual, identity, and text-alignment scores than general text-guided editing baselines, and users prefer it in roughly two-thirds of choices. If the claim holds, creative relighting becomes possible from a single photo with no lightstage equipment.","feed_headline":"A text prompt alone relights a portrait and its background","feed_subtitle":"Trained on 1.5 million synthetic pairs, it changes light and mood without breaking identity.","key_machinery":"The load-bearing mechanism is the synthetic data pipeline, because without it there is no ground truth for the text-to-lighting mapping. Its first stage is hierarchical prompt generation: the authors define 19 high-level categories (atmosphere, color, temperature, directionality, emotions, intensity, light location, smell, sound, taste, time, weather, universe, and others), an LLM expands each into at least 30 sub-words, and two to six randomly selected words are fed back to the LLM as constraints for composing a scene description. The second stage converts the prompt into a lighting image, either an RGB image produced by a fast latent consistency model or an HDR panorama produced by a fine-tuned text-guided diffusion model. The third stage transfers this lighting onto portrait data: for the foreground, an image-based relighting model operates on a single image, and when one-light-at-a-time (OLAT) captures from a lightstage are available, HDR rendering from the panorama is used instead; for the background, the lighting image is decomposed into about 20 point lights whose 3D position, color, intensity, ellipsoid ratio, and diffusion parameter are optimized to minimize photometric error, then those lights are re-rendered against the target background's albedo, depth, and normals. The final model is a diffusion UNet initialized from a text-guided image-editing model, trained with a foreground-mask condition and jointly on auxiliary delighting and light-positioning tasks.","core_discovery":"The central discovery claim is that the text-to-lighting mapping can be learned rather than hand-designed, provided the training signal is generated with lighting fully decomposed from content. The paper's formulation is $\\tilde{I} = f_\\theta(I, M, T)$: a single function that ingests a source image, a foreground mask, and a text prompt and produces a relighted image. The training targets come from three pipeline stages: hierarchical LLM prompt generation; text-conditioned lighting image generation as an RGB image or an HDR panorama; and separate image-based relighting of the foreground and background, where the background lighting is represented as twenty optimized point lights and transferred via $\\tilde{I} = \\hat{A} * \\sum_i s(L_i, \\{\\hat{D}, \\hat{N}\\})$. On synthetic test data built by the same pipeline, the model outperforms general text-guided editors on SSIM, LPIPS, CLIP vision similarity, face identity similarity, and a fine-tuned vision-language score, and in a 30-participant user study it is preferred 66.17% of the time. The paper also claims that the hand-crafted hierarchy substantially broadens prompt diversity relative to vision-language captioning, and that auxiliary tasks of shadow removal and light positioning improve the geometry and intrinsic-appearance modeling that relighting requires.","pith_inferences":["Because the training targets are themselves generated by a text-to-image model, the text-to-lighting mapping can only be as faithful as that generator; a direct check would be to run the pipeline on lightstage captures with known physical lighting and compare shadow directions and color casts against ground truth.","The background representation factorizes light into point sources, so the same optimized point-light transfer could be reused for other editing tasks such as object insertion, multi-image lighting transfer, or relighting arbitrary scenes, not just portraits.","The model's apparent ability to interpret prompts about smell, taste, and emotion suggests it learns a culturally shared visual association rather than a physical model of light; a cross-cultural user study on such prompts would show how much of the mapping is learned convention.","The reported limitation that the model sometimes places strong unnatural point lights in the background and confuses text-specified light positions points to a concrete extension: conditioning the network on explicit 3D light-position tokens alongside the text could give users direct positional control."],"forward_implications":["A single in-the-wild portrait photo is enough: users can specify lighting by mood, time of day, weather, color, or even smell and taste, and get a relighted scene with identity preserved.","The same model can be prompted for shadow removal and light positioning, making it a multi-purpose lighting editor rather than a single-task relighter.","Because the model is initialized from a general editing model and trained on diverse synthetic pairs, it also handles background harmonization of composited portraits and, per the supplementary results, generalizes to non-portrait objects and frame-by-frame video.","Any improvement to the three pipeline components, better text-to-image lighting rendering, better intrinsic decomposition, or better point-light optimization, should improve the downstream relighting model, since the pipeline produces its training data."],"supporting_citations":[{"why":"Supplies the base text-guided editing model architecture and pretrained weights that Text2Relight repurposes, and serves as the main baseline.","marker":"Brooks, Holynski, and Efros 2023"},{"why":"Provides the image-based portrait relighting model used to synthesize foreground relighting targets from single images.","marker":"Ren et al. 2023"},{"why":"Supplies the lightstage OLAT data and HDR rendering approach used for foreground relighting when multi-light captures are available.","marker":"Pandey et al. 2021"},{"why":"Provides the intrinsic image diffusion and point-light representation that the background relighting builds on, and is the comparison for the lighting optimization.","marker":"Kocsis, Sitzmann, and Nießner 2023"},{"why":"Generates the RGB lighting image from a text prompt in four denoising steps, forming half of the lighting-image generation stage.","marker":"Luo et al. 2023"},{"why":"The latent diffusion model fine-tuned for text-to-HDR-panorama generation and used for background inpainting and source augmentation.","marker":"Rombach et al. 2022"},{"why":"The large language model used in the hierarchical prompt generation stage and for the vision-language score training data.","marker":"OpenAI 2022"},{"why":"Supplies the real-world portrait images used to test generalization and as source images for real-data augmentation.","marker":"Kvanchiani et al. 2023"}],"fun_headline_variants":["Text prompt alone relights portraits and backgrounds","Learned text-to-lighting mapping from synthetic pairs","Synthetic data pipeline teaches AI to relight from text","Portrait relighting driven by text, no manual edits needed","Lighting changes via text prompts, preserving identity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The synthetic targets are trustworthy: the text-to-image model really renders lighting that matches the prompt, and the foreground and background relighting steps really transfer that lighting onto real portraits; if any of these components mislabels the training data, the model learns the wrong text-to-lighting mapping and the gains measured on synthetic test data will not transfer to real relighting.","fun_headline_variants_meta":{"raw":{"variants":["Text prompt alone relights portraits and backgrounds","Learned text-to-lighting mapping from synthetic pairs","Synthetic data pipeline teaches AI to relight from text","Portrait relighting driven by text, no manual edits needed","Lighting changes via text prompts, preserving identity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000452,"raw_usage":{"total_tokens":2350,"prompt_tokens":1093,"completion_tokens":1257,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":709,"completion_tokens_details":{"reasoning_tokens":1181}},"tokens_in":709,"tokens_out":1257,"duration_ms":8412,"temperature":1.0,"reasoning_tokens":1181,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:49:57.798158+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take portraits captured under known physical lighting, for example lightstage OLAT data, generate prompts that accurately describe that lighting and also prompts that describe incorrect lighting, run the model, and compare the shadow directions, shadow softness, intensity, and color casts in the outputs against the measured ground truth. If the model scores well on its own synthetic test set but cannot reproduce the correct physical lighting on these held-out captures, or cannot distinguish correct from incorrect descriptions, the claim that it has learned a genuine text-to-lighting mapping is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the base text-guided editing model architecture and pretrained weights that Text2Relight repurposes, and serves as the main baseline."},{"cited_title":"Relightful Harmonization: Lighting-aware Portrait Background Replacement","cited_arxiv_id":"2312.06886","evidence_quote":"Provides the image-based portrait relighting model used to synthesize foreground relighting targets from single images."},{"cited_title":"E.; and Fanello, S","cited_arxiv_id":null,"evidence_quote":"Supplies the lightstage OLAT data and HDR rendering approach used for foreground relighting when multi-light captures are available."}],"review_version":1}