AI Learns to Paint with Code: A Breakthrough in Creative RL

From Prompts to Paintbrushes: A New Paradigm in AI Art

The conventional AI image generation workflow is a one-way street: you type a prompt, the model returns an image, and if you want to change anything, you start over with a new prompt. This fundamental limitation sparked a unique project by designer-developer Surya Narreddi and collaborator Cameron Franz, who trained a language model to create images by writing code—specifically, p5.brush JavaScript sketches. The result is an AI that doesn't just generate a static image but produces an editable, code-based artifact.

The project, detailed on Narreddi's website, represents a significant departure from typical text-to-image systems. Instead of a black box that outputs pixels, the model generates a complete program that renders the artwork. This means the output is inherently editable at a granular level—you can tweak the code to adjust a brushstroke, change a color, or modify the composition without re-prompting the model.

More importantly, the research tackles a deeper question: how to apply reinforcement learning (RL) to creative and design tasks. RL thrives on verifiable rewards—a math problem is right or wrong, a game is won or lost. Aesthetic quality, however, is subjective. The project's core challenge was designing a reward function that could guide the model toward producing beautiful, watercolor-style paintings without causing it to converge on a single, bland output.

The Training Loop: Code, Render, Judge, Update

The system operates on a four-step loop, repeated thousands of times during training. First, the model receives a prompt like "draw a peach hibiscus in watercolour" and writes a complete p5.brush JavaScript sketch. This sketch is then rendered in a sandboxed Puppeteer environment, producing a PNG image. The PNG is judged against two random reference paintings from a hand-rated pool, with a separate judge model choosing the better watercolor. This judgment is converted into a reward signal, which is used to update the model via GRPO, and the loop repeats.

The non-obvious choices live in what is being judged, how the judgment is made, what is in the reference pool, and how the system prompt is written. Each element proved critical to the project's success and failure.

The Reward Function Puzzle

The initial rubric was a complex beast, containing nine separate signals: a compilation gate, a check that the code used p5.brush, a code length ramp targeting around 3,000 tokens, HPSv3 (a human preference model), prompt adherence judged by a council of GPT-5.4 and Gemini, and four additional quality judges for recognizability, aesthetics, technique, and depth.

Despite this complex setup, the model plateaued at around 0.65 reward and stayed there. Every rollout looked the same—a flat, clip-art flower with five rounded petals. The reward kept increasing, but capabilities didn't improve. The diagnosis came from analyzing the sub-rewards in isolation. The four quality judges plus prompt adherence were correlated with each other at 0.85 to 0.95; they were measuring the same thing five times. Code length, contributing roughly a third of the total reward, had saturated by step thirty and was producing zero gradient afterward. HPSv3, the one signal showing real variance, was weighted at a mere 0.10.

The fix had two halves. First, they replaced absolute scoring with pairwise judgment. Instead of asking a judge to score a rollout from zero to ten (which resulted in scores compressed near zero), they showed the judge the rollout, two references from the pool, and asked a single question: "which of these is the better hibiscus watercolour?" This relative question opened up the dynamic range and proved more reliable for the judge model.

Second, they built a reference pool of hand-rated examples. From 1,664 images rated into "love," "okay," and "nope" tiers, the 117 "love-tier" examples seeded the comparison pool. Every subsequent rollout was judged against these hand-curated examples of what the team considered good.

The new rubric collapsed everything into four components: a binary compile-and-uses-brush gate (0.05 weight), a binary length check (0.05), HPSv3 (0.30), and the pairwise judge against the reference pool (0.60). The results were dramatic: the same base model and training data reached the previous plateau three times faster, kept climbing past it, and produced code that compressed from 13,500 tokens to under 2,000. The model learned that winning compositions did not need verbose code.

The Reference Pool: Hand-Rated Quality at Scale

The reference pool itself was a critical component. It contained 581 reference paintings, all hand-rated from 1,664 generations. Of these, 117 were "love-tier," 266 were "okay," and 198 were supplements from a separate generation run used to widen the comparison set in colors where hand-rated examples were thin. Every image in the pool was model output, as sourcing enough human-made examples was difficult given the niche nature of the p5.brush library.

The generation work ran through two pipelines: AutoResearch with Opus 4.6, GPT-5.4, and Gemini 3.1 Pro iterating against reference photographs under a VLM judge, and a larger batch run on Gemini 3.1 Pro. Both pipelines fed a system prompt that had itself been evolved through GEPA, a prompt-optimization library.

System Prompt Evolution: Less is More

The system prompt also required significant work. Early versions included a 400-line p5.brush API reference, which led the model to produce confident, well-formatted code that invented APIs that didn't exist. The fix involved using GEPA to evolve the prompt against a scoring function over 200 iterations. The optimization converged on a prompt with a strict allowlist of eight brush methods, no API documentation, and no examples. The first time three out of three generations produced visible hibiscus blobs was on the version written after throwing out the 400-line reference entirely.

The finding generalizes: long reference documentation in a system prompt makes models hallucinate APIs, while a short, opinionated allowlist constrains output better than the original spec.

Why This Matters: The Shift from Prompt to Code

This project is part of a broader movement in digital art. As Refik Anadol's installations at MoMA and DATALAND demonstrate, modern artists are increasingly swapping oil and bronze for algorithms. The machine isn't just a brush; it's a collaborator. Narreddi's work takes this a step further by making the code itself the artifact, allowing for direct manipulation and a more participatory creative process.

While Narreddi admits this is not a better way to make images—it's much slower—it addresses a fundamental frustration: the inability to directly participate in AI image creation beyond the prompt. By putting attention and effort across the prompt, the model, and the artifact, this project opens new possibilities for how we interact with AI-generated art.

The project is ongoing, with a final training run aimed at fixing discovered issues and a full technical report slated for publication in June 2026. For those interested in the intersection of AI, creativity, and code, this is a project worth watching.