arXiv:2611.18575v1 Announce Type: new Abstract: A large fraction of software intent is expressed visually before it is implemented. A designer hands a screenshot or a Figma mockup to a front-end engineer, an architect sketches a UML diagram or a flowchart, and a bug report ships with a picture of what went wrong. Turning such visual artifacts directly into executable software has therefore become an active problem at the intersection of computer vision and software engineering. The present survey organizes the literature on image-to-code generation for software engineering, defined as the family of methods whose input contains a visual modality and whose output is software an engineer would otherwise write by hand. We deliberately scope the survey to software-engineering code, namely user-interface and front-end code, code recovered from software diagrams such as UML and flowcharts, program repair and test code driven by visual signals, and general software-code targets, and we exclude adjacent code-generation domains whose home communities lie elsewhere, such as chart plotting scripts, vector-graphics programs, hardware description languages, and robot policies. Rather than cataloguing systems in isolation, we introduce a unified processing pipeline of visual understanding, component recognition, layout reasoning, code planning, code generation, and verification, and organize the survey around its stages. We trace five overlapping eras from rule-based reverse engineering to agentic generation, survey each pipeline stage and each software-engineering task in depth grounded in what individual systems actually do, consolidate the datasets, benchmarks, and metrics, and lay out the open challenges of design-system generalization, long-horizon and interactive generation, cross-framework deployment, accessibility, hallucinated components, and unreliable evaluation.
Continue to arXiv Software Engineering →You are being forwarded in 6s — stay on this page