REVIEW 4 major objections 6 minor 84 references
Inferring Questions from Programming Screenshots
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read State-of-the-art vision-language models, especially GPT-4o and Gemini-1.5-Pro, can reconstruct plausible Stack Overflow questions from code or IDE screenshots alone for a substantial fraction of cases, with GPT-4o exceeding 60% similarity…
desk verdict A genuinely new benchmarking task with usable data, but the headline numbers overstate the models' ability to reconstruct the original Stack Overflow question: 0.69 is relevance to the screenshot, not to the original question (Q2 tops out at 0.43). read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a benchmark that feeds each of 143 single-image Stack Overflow posts to an LLM through one of three prompt templates (in-context learning, chain-of-thought, and few-shot learning) and then compares the generated title and body with the original post. Comparison runs through two channels: cosine similarity between text embeddings produced by the all-MiniLM-L6-v2 sentence-embedding model, and developer Likert ratings for three questions (Q1 relevance of the generated question to the screenshot, Q2 relevance to the original Stack Overflow question, and Q3 relevance of the posted image to the posted question). This design separates model capability, prompting style, and image-content difficulty, and it is what lets the paper attribute performance gaps to the content of the screenshots.
What would settle it
Have human developers who see only the same screenshots write questions, then score LLM-generated questions against human-generated ones using the same embedding and Likert measures. If the humans substantially outperform GPT-4o and Gemini on the same 143 images, the 'promising capability' claim is not about screenshot understanding but about a lower bar. Conversely, if humans match the LLMs, the benchmark is image-limited, not model-limited.
Extended reading notes
Core claim
On its own terms, the paper claims that state-of-the-art multimodal LLMs, specifically Gemini and GPT-4o, show promising capability in interpreting screenshot-based programming queries. That capability is quantified two ways: embedding-based similarity between generated and original Stack Overflow text reached 0.59, and developers rated the relevance of generated questions to the screenshot at 0.67–0.68. The headline result is that GPT-4o generated questions with over 60% similarity to the original posted question for 51.75% of the 143 images. The paper also finds that the link between screenshot content and the posted question is imperfect, with average image relevance to the original question at 0.69, and that model-image alignment correlates only moderately with the screenshot's relevance to the original question. These numbers support the conclusion that screenshot-only questioning is feasible for simple, clear images but not yet reliable for complex or cluttered ones.
Load-bearing premise
The evaluation treats the original Stack Overflow title plus body as the gold-standard target for a screenshot-only input, even though images and text are only loosely aligned in many posts (average relevance 0.69), so every similarity score measures agreement with an imperfect standard.
Editorial extensions
If this is right
- If correct, developer forums could offer an image-to-question assistant that turns a code or IDE screenshot into a formatted Stack Overflow-style post, lowering the barrier for users who struggle to articulate their problem.
- Screenshot-only debugging copilots become a realistic near-term goal: the models already reconstruct the core problem from clear images, and further fine-tuning on programming screenshots should improve accuracy on complex cases.
- The strong dependence on image content implies that practical tools should guide users to capture explicit error messages, code, and annotations rather than full cluttered IDE windows.
- With temperature zero, in-context learning and chain-of-thought prompting give more consistent results than few-shot prompting, so deployment choices about prompting matter as much as model choice.
- The moderate correlation between screenshot relevance and generated-question alignment suggests that improving screenshot-to-question reconstruction requires improving the input, not just the model.
Reading between the lines
- Beyond the paper, matching the original post may be the wrong success criterion for a screenshot-only feature; a generated question that is more accurate than the original text would be penalized by the embedding metric. An outcome-based evaluation, such as whether the generated question receives a correct and accepted answer, would be a stronger test of usefulness.
- A testable extension suggested by the data is to pair each screenshot not with the original post but with a human-written question created by a developer who sees only the screenshot. If humans beat GPT-4o by a wide margin, the bottleneck is model vision; if humans do not, the benchmark is measuring image information content rather than model capability.
- Because few-shot learning was consistently worst, the two manually selected examples may have steered generation toward those posts' style. Varying few-shot examples by language or framework would separate prompt-example effects from model limits.
- The paper's results imply that an interactive assistant could classify screenshot complexity first and ask for a text clarification when the image is cluttered, reserving fully automatic conversion for high-confidence screenshots.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper investigates whether vision-language LLMs can generate Stack Overflow questions from code or IDE screenshots, comparing Llama-3.2, Gemini-1.5-Pro, and GPT-4o under in-context learning, chain-of-thought, and few-shot prompting. The dataset consists of 143 Stack Overflow posts with a single code/IDE screenshot. The authors evaluate generated questions against the original post using all-MiniLM-L6-v2 embedding cosine similarity and developer ratings on three Likert questions: Q1 (relevance of the generated question to the screenshot), Q2 (relevance of the generated question to the original post), and Q3 (relevance of the original post to the screenshot). The reported headline results are embedding similarities up to 0.59 for GPT-4o and Gemini, a developer-perceived relevance of up to 0.69, and the claim that GPT-4o exceeds 60% similarity with the baseline for 51.75% of images. The paper concludes that state-of-the-art LLMs show promising capabilities for inferring questions from programming screenshots, while also acknowledging limitations around screenshot relevance, small sample size, and possible training-data exposure.
Significance. If the central claim were fully supported, the paper would provide useful early evidence on whether multimodal LLMs could enable screenshot-first technical Q&A, with practical implications for developer forums and debugging copilots. The study has notable strengths: it uses a manually filtered, recent dataset; it makes data and code available in a GitHub repository; the developer evaluation shows high inter-rater reliability (Cohen's kappa 0.86); and the authors are transparent about dataset size and the imperfect alignment between Stack Overflow images and their associated text posts. However, the paper's headline numbers as written conflate two different developer-rated measures (Q1 and Q2) and the embedding similarity is not calibrated against any null or upper bound, so the 'promising capabilities' conclusion outruns the evidence. The substantive results—particularly the Q2 alignment near 0.43 and the moderate Q3-Q2 correlation near 0.53—are more modest than the abstract and conclusion suggest.
major comments (4)
- [Abstract, Section VII] The claim that 'perceived relevance scores from developers reach up to 0.69' is based on Q1 from Table II, which measures the relevance of the generated question to the screenshot, not Q2, which measures relevance to the original Stack Overflow question. The Q2 maximum is 0.43 (GPT-4o with in-context learning), and all Q2 values are in the 0.34–0.43 range. The abstract and Section VII therefore overstate what the developer evaluation supports; the statement that models can 'infer' the original posted question rests on the uncalibrated embedding similarity and the 51.75% statistic, not on the perceived-relevance measure cited. Please revise the abstract and conclusion to report Q1 and Q2 separately and to qualify any claim about reconstructing the original question.
- [Section V-A, Table I] The embedding-based similarity scores are presented without calibration. A cosine similarity of 0.59 between a generated question and the original post has no established interpretation as a measure of question-reconstruction quality, and the same is true for the derived '51.75% of images above 60% similarity' statistic. Without a null baseline (for example, questions generated by the same model with no image or with an unrelated image) and an upper bound (for example, the original question embedded against itself or a paraphrase oracle), the absolute value 0.59 cannot be read as 'promising'. The authors should either add such controls or reinterpret the scores relative to them, and they should state which all-MiniLM-L6-v2 similarities correspond to clearly relevant versus irrelevant programming questions.
- [Sections V-A, V-B, VI] The paper reports no confidence intervals, standard errors, or significance tests for any of the key comparisons. In Table I, the differences between GPT-4o and Gemini are often 0.01–0.02 (for example, 0.59 versus 0.58 in the Combined row), and in Table II the Q2 differences between models are also small (0.43 versus 0.41, 0.40, 0.39). With n=143 for the embedding evaluation and n=50 for the developer evaluation, these effect sizes are plausibly within sampling noise. The paper should report uncertainty estimates and, where relevant, significance tests or effect sizes before asserting that one model or one prompting technique is 'best' or that one is 'significantly' better than another.
- [Sections IV-A, IV-C, VI-A] The ground-truth assumption—that the original Stack Overflow post is the correct target that a model should produce from the screenshot—is load-bearing and is only partially addressed. Section VI-A reports Q3 (image relevance to the original post) averaged at 0.69 with a moderate Q3-Q2 correlation of 0.53, which means that for many posts the screenshot alone does not determine the original question. The conclusions in Section VII do not qualify the 'promising capabilities' statement accordingly. In addition, the two few-shot examples in Section IV-C (references [42] and [43]) are drawn from the same October 2024 dataset used for the evaluation, which contaminates the few-shot condition even though that condition underperformed; this should be disclosed as a threat to validity in Section VI.
minor comments (6)
- [Section I] The sentence 'In particular, we answer:' introduces research questions that are never explicitly enumerated; either list the RQs or remove this dangling sentence.
- [Figure 1] The prompt figure is labeled 'Made a little short due to space constraint' and the text is visibly truncated; the full prompt templates should be included, either in the figure or in an appendix, for reproducibility.
- [Section IV-B] The bulleted list of metrics uses '0:' and '�' as bullet symbols, which appear to be formatting artifacts; these should be replaced with standard bullet points or numbered list items.
- [Tables I and II] The asterisks attached to 0.59 and 0.68 are never explained in the captions or the text; please define the asterisk or remove it.
- [Section VI-B] The 'manual analysis' of the ten best- and worst-performing images does not report who performed the analysis, how the set was selected, or whether a second annotator was involved; this makes the qualitative finding hard to verify.
- [Section IV-B] There is a typo in the text: 'H UGGING FACE' should be 'Hugging Face'.
Circularity Check
No circularity: the paper measures LLM outputs against an external human-written baseline; no fitted parameter, definitional identity, or load-bearing self-citation makes the results equivalent to their inputs.
full rationale
The paper's derivation chain is an empirical measurement, not an analytic derivation. Section IV establishes a baseline from original Stack Overflow titles and bodies and compares LLM-generated questions to that baseline using a fixed, off-the-shelf embedding model (all-MiniLM-L6-v2) plus developer ratings Q1-Q3. No parameter is fitted to the target result; the embedding model is used as a pre-trained tool, and the developer ratings are independent judgments. The reported 0.59 is the maximum observed cosine similarity, and the 51.75% figure is a threshold count over the same similarity scores; these are descriptive statistics over observed outputs, not quantities constructed from the values they claim to validate. The 0.69 developer figure in Section VII is best read as the average Q3 image-to-original relevance reported in Section VI-A rather than a model performance score; while this creates a reporting ambiguity, it is not a circular reduction because Q3 is an annotation of the dataset, not a function of the model outputs. The few-shot examples are drawn from the same dataset, which is a potential contamination concern for the few-shot condition, but it is not circular: the examples are fixed input exemplars, and the evaluated output is generated for other screenshots rather than recovered from a fitted target. The extensive self-citations, including the self-referential reference [57], appear in related-work and reference-list material and are not invoked as the justification for the paper's central empirical claim. Accordingly, no load-bearing step reduces by definition to its own inputs, and the recorded circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Original Stack Overflow title and body constitute the gold-standard question for each screenshot.
- domain assumption all-MiniLM-L6-v2 embeddings preserve semantic similarity sufficient for evaluating question generation.
- domain assumption The two manually selected few-shot examples are representative of the dataset.
Cite this review
Pith. "Pith review of Inferring Questions from Programming Screenshots." pith.science (2026). https://pith.science/paper/NHKJSVRI
@misc{pith2026250418912,
author = {Pith},
title = {Pith review of: Inferring Questions from Programming Screenshots},
year = {2026},
howpublished = {\url{https://pith.science/paper/NHKJSVRI}},
note = {Machine review of arXiv:2504.18912}
}
read the original abstract
The integration of generative AI into developer forums like Stack Overflow presents an opportunity to enhance problem-solving by allowing users to post screenshots of code or Integrated Development Environments (IDEs) instead of traditional text-based queries. This study evaluates the effectiveness of various large language models (LLMs), specifically LLAMA, GEMINI, and GPT-4o in interpreting such visual inputs. We employ prompt engineering techniques, including in-context learning, chain-of-thought prompting, and few-shot learning, to assess each model's responsiveness and accuracy. Our findings show that while GPT-4o shows promising capabilities, achieving over 60% similarity to baseline questions for 51.75% of the tested images, challenges remain in obtaining consistent and accurate interpretations for more complex images. This research advances our understanding of the feasibility of using generative AI for image-centric problem-solving in developer communities, highlighting both the potential benefits and current limitations of this approach while envisioning a future where visual-based debugging copilot tools become a reality.
Figures
Reference graph
Works this paper leans on
-
[42]
G. AI, “Gemini 1.5 pro model card,” 2024. Accessed: 2024-11-10
work page 2024
-
[43]
S. Overflow, “Few-shot example 2.” https://stackoverflow.com/questions/ 79084406/trying-to-stack-2-columns-into-one-excel, 2024. Accessed: 2024-11-05
work page 2024
-
[1]
Americans’ social media use,
Pew Research Center, “Americans’ social media use,” 2024. Accessed: 2024-11-05
2024
-
[2]
53 social media statistics to inform your 2024 social strategy,
Hootsuite, “53 social media statistics to inform your 2024 social strategy,” 2024. Accessed: 2024-11-05
2024
-
[3]
Eye of the mind: Image processing for social coding,
M. Nayebi, “Eye of the mind: Image processing for social coding,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results , pp. 49 –52, 2020
2020
-
[4]
Research update: Improving the question - asking experience
L. Park, “Research update: Improving the question - asking experience.” https://stackoverflow.blog/2019/09/26/ research-update-improving-the-question-asking-experience/, 2019. Accessed: 2024-12-15
work page 2019
-
[5]
Then, we measured both the image’s relevance to the LLM -generated question and the alignment between LLM-generated responses and the original question on Stack Overflow
Question Formulation - Clear title - Technical details - Answer- able format Output Format TITLE: Generated Title BODY: Generated Body question on Stack Overflow. Then, we measured both the image’s relevance to the LLM -generated question and the alignment between LLM-generated responses and the original question on Stack Overflow. To guide these evaluati...
-
[6]
Policy on posting code / error message etc as images,
batFINGER, “Policy on posting code / error message etc as images,”
Show all 84 references
-
[7]
Discourage screenshots of code and/or errors,
apaul, “Discourage screenshots of code and/or errors,” 2015. Accessed: 2024-11-05
2015
-
[8]
Why should i not upload images of code/data/errors?,
C. Gray, “Why should i not upload images of code/data/errors?,” 2015. Accessed: 2024-11-05
2015
-
[9]
Zero -shot text -to-image generation,
A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero -shot text -to-image generation,” arXiv preprint arXiv:2102.12092, 2021
2021 arXiv
-
[10]
High - resolution image synthesis with latent diffusion models,
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High - resolution image synthesis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni - tion, pp. 10684–10695, 2022
2022
-
[11]
Git: A generative image- to-text transformer for vision and language,
J. Wang, Z. Yang, X. Hu, L. Li, K. Lin, Z. Gan, Z. Liu, C. Liu, and L. Wang, “Git: A generative image- to-text transformer for vision and language,” arXiv preprint arXiv:2205.14100, 2022
2022 arXiv
-
[12]
Generating diverse high-fidelity images with vq-vae-2,
A. Razavi, A. van den Oord, and O. Vinyals, “Generating diverse high-fidelity images with vq-vae-2,” in Advances in Neural Information Processing Systems, vol. 32, 2019
2019
-
[13]
Gui testing using computer vision,
T.-H. Chang, T. Yeh, and R. C. Miller, “Gui testing using computer vision,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1535–1544, 2010
2010
-
[14]
Generative image ai using design sketches as input: Opportunities and challenges,
C. Zhang, W. Wang, P. Pangaro, N. Martelaro, and D. Byrne, “Generative image ai using design sketches as input: Opportunities and challenges,” in Proceedings of the 15th Conference on Creativity and Cognition, pp. 254–261, 2023
2023
-
[15]
Automated system testing using visual gui testing tools: A comparative study in industry,
E. Borjesson and R. Feldt, “Automated system testing using visual gui testing tools: A comparative study in industry,” in 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pp. 350–359, IEEE, 2012
2012
-
[16]
Improving random gui testing with image -based widget detection,
T. D. White, G. Fraser, and G. J. Brown, “Improving random gui testing with image -based widget detection,” in Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis, pp. 307–317, 2019
2019
-
[17]
Understanding the characteristics of visual contents in open source issue discussions: a case study of jupyter notebook,
V. Agrawal, Y.-H. Lin, and J. Cheng, “Understanding the characteristics of visual contents in open source issue discussions: a case study of jupyter notebook,” in Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering , pp. 249 –254, 2022
2022
-
[18]
Image -based communication on social coding platforms,
M. Nayebi and B. Adams, “Image -based communication on social coding platforms,” Journal of Software: Evolution and Process, vol. 36, no. 5, p. e2609, 2024
2024
-
[19]
Negative results of image process- ing for identifying duplicate questions on stack overflow,
F. Ahmed, S. Datta, and M. Nayebi, “Negative results of image process- ing for identifying duplicate questions on stack overflow,” arXiv preprint arXiv:2407.05523, 2024
2024 arXiv
-
[20]
Understanding the characteristics and the role of visual issue reports,
H. Kuramoto, D. Wang, M. Kondo, Y. Kashiwa, Y. Kamei, and N. Ubayashi, “Understanding the characteristics and the role of visual issue reports,” Empirical Software Engineering , vol. 29, no. 4, p. 89, 2024
2024
-
[21]
psc2code: Denoising code extraction from programming screencasts,
L. Bao, Z. Xing, X. Xia, D. Lo, M. Wu, and X. Yang, “psc2code: Denoising code extraction from programming screencasts,” ACM Trans- actions on Software Engineering and Methodology (TOSEM) , vol. 29, no. 3, pp. 1–38, 2020
2020
-
[22]
pix2code: Generating code from a graphical user in - terface screenshot,
T. Beltramelli, “pix2code: Generating code from a graphical user in - terface screenshot,” in Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems, pp. 1–6, 2018
2018
-
[23]
Learning transferable visual models from natural language supervision,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning, pp. 8748–8763, PMLR, 2021
2021
-
[24]
Visualbert: A simple and performant baseline for vision and language,
L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang, “Visualbert: A simple and performant baseline for vision and language,” arXiv preprint arXiv:1908.03557, 2019
1908 arXiv
-
[25]
Blip: Bootstrapping language -image pre-training for unified vision -language understanding and generation,
J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language -image pre-training for unified vision -language understanding and generation,” in International conference on machine learning , pp. 12888– 12900, PMLR, 2022
2022
-
[26]
Scaling up visual and vision -language representation learning with noisy text supervision,
C. Jia, Y. Yang, Y. Xia, Y.- T. Chen, Z. Parekh, H. Pham, Q. Le, Y.- H. Sung, Z. Li, and T. Duerig, “Scaling up visual and vision -language representation learning with noisy text supervision,” in International conference on machine learning, pp. 4904–4916, PMLR, 2021
2021
-
[27]
Chain -of-thought prompting elicits reasoning in large language models,
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou, “Chain -of-thought prompting elicits reasoning in large language models,” arXiv preprint arXiv:2201.11903, 2022
2022 arXiv
-
[28]
Language mod- els are few -shot learners,
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language mod- els are few -shot learners,” Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020
1901
-
[29]
A systematic survey of prompt engineering on vision-language foundation models,
J. Gu, Z. Han, S. Chen, A. Beirami, B. He, G. Zhang, R. Liao, Y. Qin, V. Tresp, and P. Torr, “A systematic survey of prompt engineering on vision-language foundation models,” arXiv preprint arXiv:2307.12980 , 2023
2023 arXiv
-
[30]
Review of large vision models and visual prompt engineering,
J. Wang, Z. Liu, L. Zhao, Z. Wu, C. Ma, S. Yu, H. Dai, Q. Yang, Y. Liu, S. Zhang, et al. , “Review of large vision models and visual prompt engineering,” Meta-Radiology, p. 100047, 2023
2023
-
[31]
Llama: Open and efficient foundation language models,
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozie`re, N. Goyal, E. Hambro, F. Azhar, et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023
2023 arXiv
-
[32]
A survey on in -context learning,
Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, T. Liu, et al., “A survey on in -context learning,” arXiv preprint arXiv:2301.00234, 2022
2022 arXiv
-
[33]
Gpt-4o system card,
A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024
2024 arXiv
-
[34]
Gemini: a family of highly capable multimodal models,
G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023
2023 arXiv
-
[35]
Generative artificial intelligence for software engineering –a research agenda,
A. Nguyen-Duc, B. Cabrero-Daniel, A. Przybylek, C. Arora, D. Khanna, T. Herda, U. Rafiq, J. Melegati, E. Guerra, K.- K. Kemell, et al. , “Generative artificial intelligence for software engineering –a research agenda,” arXiv preprint arXiv:2310.18648, 2023
-
[36]
Re - thinking software engineering in the foundation model era: From task - driven ai copilots to goal -driven ai pair programmers,
A. E. Hassan, G. A. Oliva, D. Lin, B. Chen, Z. Ming, et al. , “Re - thinking software engineering in the foundation model era: From task - driven ai copilots to goal -driven ai pair programmers,” arXiv preprint arXiv:2404.10225, 2024
2024 arXiv
-
[37]
model card
Hugging Face, “model card.” https://huggingface.co/ sentence-transformers/all-MiniLM-L6-v2, 2024. Taken from the official website of Hugging Face
2024
-
[38]
Stack exchange data explorer
S. Exchange, “Stack exchange data explorer.” https://data.stackexchange. com/. Accessed: 2024-11-07
2024
-
[39]
Question inferring tool
S. App, “Question inferring tool.” https://question -inferring.streamlit. app/, 2024. Accessed: 2024-11-05
2024
-
[40]
Semantic cosine simi - larity,
F. Rahutomo, T. Kitasuka, M. Aritsugi, et al. , “Semantic cosine simi - larity,” in The 7th international student conference on advanced science and technology ICAST , vol. 4, p. 1, University of Seoul South Korea, 2012
2012
-
[41]
Llama 3.2 model card,
M. AI, “Llama 3.2 model card,” 2024. Accessed: 2024-11-10
2024
-
[44]
Few-shot example 1
S. Overflow, “Few-shot example 1.” https://stackoverflow.com/questions/ 79044080/is-getenv-s-not-part-of-cstdlib/, 2024. Accessed: 2024-11-05
2024
-
[45]
Examining ownership models in software teams,
U. A. Koana, Q. H. Le, S. Raman, C. Carlson, F. Chew, and M. Nayebi, “Examining ownership models in software teams,” Empirical Software Engineering, vol. 29, no. 6, pp. 1–43, 2024
2024
-
[46]
Github repository
Github, “Github repository.” https://github.com/Research -Purpose/ MSR-2025, 2024. Accessed: 2024-11-05
2025
-
[47]
Developers’ perception of github actions: A survey analysis,
S. G. Saroar and M. Nayebi, “Developers’ perception of github actions: A survey analysis,” in Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering , pp. 121 –130, 2023
2023
-
[48]
Github marketplace for automation and innovation in software production,
S. G. Saroar, W. Ahmed, E. Onagh, and M. Nayebi, “Github marketplace for automation and innovation in software production,” Information and Software Technology, vol. 175, p. 107522, 2024
2024
-
[49]
Release practices for mobile apps– what do users and developers think?,
M. Nayebi, B. Adams, and G. Ruhe, “Release practices for mobile apps– what do users and developers think?,” in 2016 ieee 23rd international conference on software analysis, evolution, and reengineering (saner) , vol. 1, pp. 552–562, IEEE, 2016
2016
-
[50]
Ownership in the hands of accountability at brightsquid: A case study and a developer survey,
U. A. Koana, F. Chew, C. Carlson, and M. Nayebi, “Ownership in the hands of accountability at brightsquid: A case study and a developer survey,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineeri...
2008
-
[51]
Essmart way to manage customer requests,
M. Nayebi, L. Dicke, R. Ittyipe, C. Carlson, and G. Ruhe, “Essmart way to manage customer requests,” Empirical Software Engineering, vol. 24, pp. 3755–3789, 2019
2019
-
[52]
Documentation of machine learning software,
Y. Hashemi, M. Nayebi, and G. Antoniol, “Documentation of machine learning software,” in 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 666–667, IEEE, 2020
2020
-
[53]
Nayebi, Analytical Release Management for Mobile Apps
M. Nayebi, Analytical Release Management for Mobile Apps . PhD thesis, PhD thesis, University of Calgary, 2018
2018
-
[54]
Mining treatment -outcome constructs from sequential software engineering data,
M. Nayebi, G. Ruhe, and T. Zimmermann, “Mining treatment -outcome constructs from sequential software engineering data,” IEEE Transac - tions on Software Engineering, vol. 47, no. 2, pp. 393–411, 2019
2019
-
[55]
Github mar - ketplace: Driving automation and fostering innovation in software de- velopment,
S. G. Saroar, W. Ahmed, E. Onagh, and M. Nayebi, “Github mar - ketplace: Driving automation and fostering innovation in software de- velopment,” in 2025 IEEE 32nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Journal First, 2025
2025
-
[56]
Data driven requirements engineering: Implications for the community,
M. Nayebi, “Data driven requirements engineering: Implications for the community,” in 2018 IEEE 26th International Requirements Engineering Conference (RE), pp. 439–441, IEEE, 2018
2018
-
[57]
Inferring questions from programming screenshots,
F. Ahmed, X. Tan, O. Adewole, and M. Nayebi, “Inferring questions from programming screenshots,” in 22nd International Conference on Mining Software Repositories (MSR), 2025
2025
-
[58]
How do papers make into machine learning frameworks: A preliminary study on tensorflow,
F. Pepe, C. Farkas, M. Nayebi, G. Antoniol, and M. Di Penta, “How do papers make into machine learning frameworks: A preliminary study on tensorflow,” in 33rd IEEE/ACM International Conference on Program Comprehension (ICPC 2025), 2025
2025
-
[59]
User driven functionality deletion for mobile apps,
M. Nayebi, K. Kuznetsov, A. Zeller, and G. Ruhe, “User driven functionality deletion for mobile apps,” in 2023 IEEE 31st International Requirements Engineering Conference (RE), pp. 6–16, IEEE, 2023
2023
-
[60]
Recommending and release planning of user -driven functionality deletion for mobile apps,
M. Nayebi, K. Kuznetsov, A. Zeller, and G. Ruhe, “Recommending and release planning of user -driven functionality deletion for mobile apps,” Requirements Engineering, vol. 29, no. 4, pp. 459–480, 2024
2024
-
[61]
Crowd - sourced exploration of mobile app features: A case study of the fort mcmurray wildfire,
M. Nayebi, M. Marbouti, R. Quapp, F. Maurer, and G. Ruhe, “Crowd - sourced exploration of mobile app features: A case study of the fort mcmurray wildfire,” in 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Society Track (ICSE- SEIS...
2017
-
[62]
App store mining is not enough for app improvement,
M. Nayebi, H. Cho, and G. Ruhe, “App store mining is not enough for app improvement,” Empirical Software Engineering, vol. 23, pp. 2764 – 2794, 2018
2018
-
[63]
Which version should be released to app store?,
M. Nayebi, H. Farahi, and G. Ruhe, “Which version should be released to app store?,” in 2017 ACM/IEEE International Symposium on Em - pirical Software Engineering and Measurement (ESEM) , pp. 324 –333, IEEE, 2017
2017
-
[64]
Analytical product release planning,
M. Nayebi and G. Ruhe, “Analytical product release planning,” in The art and science of analyzing software data, pp. 555–589, Elsevier, 2015
2015
-
[65]
App store mining is not enough,
M. Nayebi, H. Cho, H. Farrahi, and G. Ruhe, “App store mining is not enough,” in 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C), pp. 152–154, IEEE, 2017
2017
-
[66]
A longitudinal study of identifying and paying down ar - chitecture debt,
M. Nayebi, Y. Cai, R. Kazman, G. Ruhe, Q. Feng, C. Carlson, and F. Chew, “A longitudinal study of identifying and paying down ar - chitecture debt,” in 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 171–18...
2019
-
[67]
Data- driven requirements engineering-an update,
W. Maalej, M. Nayebi, and G. Ruhe, “Data- driven requirements engineering-an update,” in 2019 IEEE/ACM 41st International Confer - ence on Software Engineering: Software Engineering in Practice (ICSE- SEIP), pp. 289–290, IEEE, 2019
2019
-
[68]
Asymmetric release planning: Compromising satisfaction against dissatisfaction,
M. Nayebi and G. Ruhe, “Asymmetric release planning: Compromising satisfaction against dissatisfaction,” IEEE Transactions on Software Engineering, vol. 45, no. 9, pp. 839–857, 2018
2018
-
[69]
Optimized functionality for super mobile apps,
M. Nayebi and G. Ruhe, “Optimized functionality for super mobile apps,” in 2017 IEEE 25th international requirements engineering con - ference (RE), pp. 388–393, IEEE, 2017
2017
-
[70]
Anatomy of functionality deletion: an exploratory study on mobile apps,
M. Nayebi, K. Kuznetsov, P. Chen, A. Zeller, and G. Ruhe, “Anatomy of functionality deletion: an exploratory study on mobile apps,” in Proceedings of the 15th International Conference on Mining Software Repositories, pp. 243–253, 2018
2018
-
[71]
Analytics for software project management –where are we and where do we go?,
M. Nayebi, G. Ruhe, R. C. Mota, and M. Mufti, “Analytics for software project management –where are we and where do we go?,” in 2015 30th IEEE/ACM International Conference on Automated Software Engineering Workshop (ASEW), pp. 18–21, IEEE, 2015
2015
-
[72]
The vision: Requirements en - gineering in society,
G. Ruhe, M. Nayebi, and C. Ebert, “The vision: Requirements en - gineering in society,” in 2017 IEEE 25th International Requirements Engineering Conference (RE), pp. 478–479, IEEE, 2017
2017
-
[73]
Analysis of marketed versus not - marketed mobile app releases,
M. Nayebi, H. Farrahi, and G. Ruhe, “Analysis of marketed versus not - marketed mobile app releases,” in Proceedings of the 4th International Workshop on Release Engineering, pp. 1–4, 2016
2016
-
[74]
An open innovation approach in support of product release decisions,
M. Nayebi and G. Ruhe, “An open innovation approach in support of product release decisions,” in Proceedings of the 7th International Workshop on Cooperative and Human Aspects of Software Engineering, pp. 64–71, 2014
2014
-
[75]
What counts is decisions, not num- bers—toward an analytics design sheet,
G. Ruhe and M. Nayebi, “What counts is decisions, not num- bers—toward an analytics design sheet,” in Perspectives on Data Science for Software Engineering, pp. 111–114, Elsevier, 2016
2016
-
[76]
Hybrid labels are the new measure!,
M. Nayebi, S. J. Kabeer, G. Ruhe, C. Carlson, and F. Chew, “Hybrid labels are the new measure!,” IEEE Software, vol. 35, no. 1, pp. 54 –57, 2017
2017
-
[77]
More insight from being more focused: analysis of clustered market apps,
M. Nayebi, H. Farrahi, A. Lee, H. Cho, and G. Ruhe, “More insight from being more focused: analysis of clustered market apps,” in Proceedings of the International Workshop on App Market Analytics , pp. 30 –36, 2016
2016
-
[78]
Analytical open innovation for value - optimized service portfolio planning,
M. Nayebi and G. Ruhe, “Analytical open innovation for value - optimized service portfolio planning,” in Software Business. Towards Continuous Value Delivery: 5th International Conference, ICSOB 2014, Paphos, Cyprus, June 16-18, 2014. Proceedings 5, pp. 273–288, Springer, 2014
2014
-
[79]
Trade -off service portfolio planning –a case study on mining the android app market,
M. Nayebi and G. Ruhe, “Trade -off service portfolio planning –a case study on mining the android app market,” tech. rep., PeerJ PrePrints, 2015
2015
-
[80]
Predicting the vector impact of change-an industrial case study at brightsquid,
S. J. Kabeer, M. Nayebi, G. Ruhe, C. Carlson, and F. Chew, “Predicting the vector impact of change-an industrial case study at brightsquid,” in 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 131–140, IEEE, 2017
2017
-
[81]
Investigating answerability of llms for long-form question answering,
M. M. Bhat, R. Meng, Y. Liu, Y. Zhou, and S. Yavuz, “Investigating answerability of llms for long-form question answering,” arXiv preprint arXiv:2309.08210, 2023
2023 arXiv
-
[82]
Evaluating open - domain question answering in the era of large language models,
E. Kamalloo, N. Dziri, C. L. Clarke, and D. Rafiei, “Evaluating open - domain question answering in the era of large language models,” arXiv preprint arXiv:2305.06984, 2023
2023 arXiv
-
[84]
Example analysis dashboard
“Example analysis dashboard.” https://example-analysis-msr2025. streamlit.app/, 2024. Accessed: 2024-01-31
2024
-
[2021]
Accessed: 2024-11-05
2024
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.