Project Description

Together with Francesco Scarel and Alessandro Fontana via Liminal Research ETS, I developed a web app that allowed students from Istituto BEM Staranzano (IT) to brainstorm design proposals for a new wetland habitat, leveraging the power of SemSeg ControlNet in ComfyUI.

Users draw on a browser canvas by picking from a curated palette of colors, each semantically associated with a landscape element: sand, trees, water, dirt, and bushes. The drawing is their design intent — no 3D software or technical skill required.

ComfyUI receives each drawing via network tunneling and feeds it into a ControlNet SemSeg preprocessor, which interprets the color regions as semantic scene segments and drives a Stable Diffusion 1.5 generation at 1280×720 pixels.

Each generation is then automatically composited onto the base aerial photograph of the site and color-corrected, entirely within ComfyUI — giving students an immediate, photorealistic preview of their proposal in context.

To maximize generation speed during the workshop sessions, the workflow was deliberately kept lightweight: no IPAdapter, no custom LoRAs. Speed and accessibility over fine-tuned fidelity.

The project grew out of a rapid environment-prototyping concept I developed in ComfyUI, using SemSeg sketching as a design tool. The results from the students were genuinely surprising.

More information about the educational project: lnkd.in/dhmpvh_e