RL shifts the perception of beauty from human to model

When the reward isn't a correct result, but taste

Current RL models are most often trained on math or logic, where the result is verifiable (a test passes or fails). Engineer Sergio Paniego has now published an open replication of an earlier viral experiment with p5.brush, in which he demonstrates that Reinforcement Learning can successfully optimize even subjective aesthetics.

While the original model learned to paint isolated flowers from feedback on text prompts, this open procedure trains the Qwen3.5-35B model to directly generate 150 lines of JavaScript for an entire composition. The trick lies in the reward function: the model doesn't get points for precision, but for style. The score consists of 60% pairwise evaluation by another vision model against a hand-curated set of "beautiful" images and 30% from the HPSv3 preference model.

Code generation evolves from a function into a full-fledged craft

This is a significant shift for the use of open-source tools, namely the TRL platform and OpenEnv. Code is no longer used here just as a helper for data retrieval, but as the artistic tool itself. By restricting the model to just ten library methods (e.g., color bleeding, strokes), they forced it to focus on style instead of technical complexity. All components, including the reference dataset and the reward model, are now publicly available on Hugging Face, opening the way to cheap and replicable RL fine-tuning for aesthetics to anyone with GPU access.

Fast visual demo still hits the fragility of infrastructure

Opening the procedure hits the reality of operations. During training, the engineer discovered that a network layer error must not return a zero score. If it did, the model would start incorrectly punishing valid but slowly rendered code, which would completely destroy the learning process. It was the stability of the infrastructure, not the algorithm itself, that constituted the biggest obstacle on the way to the result.

Expert architecture will require adjustments

Since the default Qwen/Qwen3.5-35B-A3B is built on Mixture of Experts (MoE), it turned out that a standard LoRA applied only to key layers doesn't reach the routed experts. The solution was to use an "all-linear" approach that encompasses every linear layer in the network. Even though the experts themselves are frozen, the adapters thus gain enough influence. For future deployment of similar visual tasks on MoE models, this approach must be accounted for in the design of the training pipeline itself.

Lilith's verdict

Aesthetics are now handled not by an artist with a brush, but by an engineer with a reward function.