Research: sketch/constraint-guided layout generation for templates (arXiv:2510.27632)

[ajianaz]

Context

Sketch-to-Layout (arXiv:2510.27632): sketch-guided multimodal layout generation — users give a rough sketch instead of complex constraint specifications, and the model generates aesthetic layouts. Addresses usability of constraint input.

Why relevant

Cosy templates are constraint-driven; this offers a lower-friction authoring path: designer sketches placement → layout constraints generated → template created.

Proposal

• Explore sketch/image input as an alternative to manual constraint authoring for new templates
• Keep it optional; existing structured constraints remain the source of truth

References

• Paper: http://arxiv.org/abs/2510.27632

[ajianaz]

Research findings — Sketch-to-Layout (arXiv:2510.27632)

Paper: "Sketch-to-Layout: Sketch-Guided Multimodal Layout Generation" — Brioschi et al., Google DeepMind, Oct 2025, accepted at ICCV 2025 Workshop (HiGen). Code/data: https://github.com/google-deepmind/sketch_to_layout (Apache-2.0 code, CC-BY data).

What the paper does

• Problem: constraint-based layout generation (element sizes, spatial relations, grids) is powerful but high-friction. Sketches are the natural design input. Empirical comparison (few-shot Gemini 1.5 Pro) shows sketches dominate the time-vs-quality trade-off: best mIoU with the least input time vs textual constraints (types / types+sizes / spatial relations).
• Method: fine-tuned PaLI-Gemma 3B (open VLM, Gemma code-pretrained backbone). Inputs = ink sketch + image assets + text prompt (canvas size, asset names, text contents) → output = structured protobuf string (code-generation formulation), convertible to SVG/image. Content-aware: asset contents materially improve placement.
• Key data trick: hand-drawing full-layout sketches at scale is prohibitively expensive, so they define wireframe primitives (crossed rectangle = image; horizontal lines = text block), had 10 annotators draw ~470 primitives (≈50 min total), then compose full synthetic sketches by KD-tree nearest-neighbor matching of primitives to each asset's geometry (width/aspect for images; bbox + font size for text). Result: ~235k sketch/layout pairs (175k PubLayNet, 33k DocLayNet, 27k SlideVQA).
• Results: fine-tuned model beats all constraint-based baselines by >40% mIoU (PubLayNet mIoU 0.76 vs 0.29–0.33); synthetic sketches ≈ human sketches at test time (validates the proxy); performance scales with sketch coverage (partial sketches still work as soft priors). Also introduces COS (Content Ordering Score, Levenshtein-based reading-order metric).

Applicability to Cosy

Cosy differs from the paper's setting: we generate repeatable templates (SVG + schema + defaults), not one-off layouts. The paper's output (structured layout) maps well onto our authoring flow, not our runtime. Three adoption paths:

Path	Effort	Fit
A. VLM-assisted authoring (recommended now) — designer sketch + few-shot prompt (existing cosy templates as examples) → draft SVG layout → human templatizes (extract minijinja tokens + schema). Uses the paper's insight without any ML training; sketch-guided Gemini alone already gives rough-correct placement (mIoU ~0.32).	1–2 days prototype	High
B. Sketch → field mapping for users — user uploads a wireframe; system returns the closest existing cosy template + suggested field mapping. Lower value; template library already covers common shapes.	3–5 days	Medium
C. Fine-tune a PaLI-Gemma-class model on cosy's own templates — compose synthetic sketches over our 148 templates using their primitive method (need ~1 h primitive annotation + data augmentation by rendering template variants; 148 base samples is far too few raw). Training + serving infra.	2–4 weeks + GPU	Low today — revisit if template count grows 5–10× or we ship an end-user sketch→card feature in cosy serve

Recommendation

1. Prototype Path A as an offline authoring accelerator: sketch + brand.json → draft template.svg via a VLM with cosy-specific few-shot examples; human review + templatize. Structured schema stays the source of truth (as the issue proposal requires — sketch is an input aid, not the runtime).
2. Borrow two ideas into tooling regardless: (a) the primitive vocabulary (crossed-rect = image, lines = text) as the canonical sketch spec for cosy authoring docs; (b) COS-style ordering check as a future CI metric for template field ordering.
3. Defer Path C. Keep Research: design-quality evaluation via designer-annotated preference (TASTE / PosterReward) #60's TASTE/PosterReward idea in mind as the evaluation half of the same future feature.

Risks / notes

• Domain gap: paper trained on documents/slides; cosy cards use decorative layers (gradients, dot grids, ascii-art) the primitive vocabulary doesn't cover — the paper explicitly notes extensibility to more primitive types.
• VLM output must always pass cosy schema validation; treat model output strictly as a draft.
• Licensing is friendly (Apache-2.0 / CC-BY); dataset TFRecords are for their document domains, not directly reusable for social cards.

Exploration checkbox in the issue body ticked. Keeping this issue open for the Path A go/no-go decision.

[ajianaz]

Follow-up: local, CPU-only, lightweight VLM options for Path A/C

Question raised: can the sketch→layout draft step run fully local on CPU with a small model? Yes — 2026 has a healthy sub-4B ecosystem. Summary of verified options:

Tier 1 — ultra-tiny (<1B), detection-grade

• Florence-2 (Microsoft, 0.2B base / 0.77B large, MIT): natively emits structured detection dicts (bboxes + labels) that parse straight to JSON — ideal for a sketch-parsing stage. Model card, structured output format
• SmolVLM-256M/500M (HF, Apache-2.0): <1 GB memory (paper) but VQA/caption-focused, no native box output — would need a paired detector (pattern reference)

Tier 2 — sweet spot 1.6–4B, can generate structured drafts

• Qwen3-VL-2B-Instruct (Apache-2.0): 4-bit GGUF confirmed working on CPU via llama.cpp; pre-quantized MLX 4-bit checkpoints also exist. Best small model for code-style structured output.
• LFM2.5-VL-1.6B / LFM2.5-VL-3B (Liquid AI, edge-first): claims competitive vision performance vs models 2× its size with faster CPU deployment; LFM2-VL outperforms Gemma-3-4B per the LFM2 tech report / LFM2.5-VL-3B blog; Unsloth fine-tune supported (guide)
• Moondream2 (1.9B) / Moondream3 (9B MoE, ~2B active) (Apache-2.0): frequently cited ultra-light option
• PaliGemma-3B: the paper's exact base, available GGUF in llama.cpp

Tier 3 — upper CPU-feasible (7–8B Q4)

Qwen2.5-VL-7B / Qwen3-VL-4B/8B: best quality, but single-digit tokens/s on CPU — acceptable only for offline authoring, not interactive.

Runtime

• llama.cpp (GGUF + mmproj) — true CPU-only, or Ollama as a wrapper
• On Apple Silicon: MLX (still 100% local, uses unified memory) is noticeably faster than CPU-only llama.cpp

Recommended architecture (updates Path A → fully local)

Two-stage, so each model does what it's best at:

1. Florence-2 parses the wireframe sketch → JSON boxes/labels (deterministic, <1 GB, instant)
2. Deterministic Rust rules (or Qwen3-VL-2B when text reasoning is needed) map boxes → cosy SVG template skeleton → human templatizes

Single-model alternative: Qwen3-VL-2B end-to-end (sketch + assets → draft SVG), simplest to prototype.

Path C impact

Fine-tuning no longer requires cloud GPUs: mlx-vlm LoRA/QLoRA runs 2B-class VLM LoRA directly on Apple Silicon (works with Qwen3-VL-2B), and Unsloth supports LFM2.5-VL. Combined with the paper's synthetic-sketch composition over our 148 templates, a LoRA prototype is now a Mac-weekend-scale experiment rather than a GPU-project. Raw sample count (148) still mandates the augmentation trick first.

Updated recommendation: prototype Path A fully local with Qwen3-VL-2B GGUF (or the Florence-2 two-stage), zero API cost, zero data egress. Keeps structured schema as source of truth.

[ajianaz]

Follow-up 2: mobile direction — covecto as the image-to-SVG layer, Flutter + Rust FFI

New constraint from maintainer: the end target is mobile (Flutter + Rust FFI), and we already own covecto (dual-engine raster→SVG vectorizer, Rust, MIT). Re-ran the feasibility with that in mind, including actual compile checks on this machine:

Repo	aarch64-apple-ios	aarch64-linux-android
covecto (core + api + cli)	✅ clean	✅ clean
cosy (resvg 0.48 / minijinja)	✅ clean	✅ clean

All deps in the hot path (vtracer, pngtosvg, image, roxmltree, usvg, resvg, minijinja) are pure Rust — mobile-ready as-is.

What covecto can and cannot do for sketch→layout

Covecto is a pixel tracer (color quantization + spline fitting). It vectorizes faithfully but has no semantic understanding: a hand-drawn wireframe comes out as bezier strokes, not "title region / image region / 3 text blocks". So it cannot replace the sketch-parsing step alone — but it slots into a mobile-first pipeline as preprocessing:

sketch photo (phone camera / finger drawing)
  → binarize + covecto B/W vectorize      (clean strokes → polygons)
  → geometric primitive analysis (pure Rust: rect fitting, crossed-rect = image,
    h-line clusters = text, per the paper's primitive vocabulary)
  → layout boxes JSON
  → cosy template scaffold + schema

This is the no-ML path: zero model download, milliseconds, fully offline — ideal on phone, where Qwen3-VL-2B Q4 (~1.5 GB RAM + load time) is impractical and even Florence-2 ONNX is heavy. The VLM paths from the previous comment stay as the desktop/cloud option for messy real-world sketches.

Covecto also enables a second, independent mobile feature: photo/art → vector → embedded as a cosy template asset (its natural product fit).

Proposed architecture

Flutter app (Dart)
  └── flutter_rust_bridge (Dart ⇄ Rust codegen)
        └── one glue crate (new repo)
              ├── cosy render: JSON → template → SVG → resvg PNG (fonts bundled as assets for fontdb)
              ├── covecto-core: vectorize(bytes, preset) → svg string   [B/W preset for sketches]
              └── sketch-geom (new): vector primitives → layout boxes JSON

• Prefer resvg rasterization → PNG bytes over handing SVG to flutter_svg (its SVG feature coverage can't be trusted with our decorative templates).
• Drop covecto's HTTP API on mobile; call covecto-core directly via FFI. Binary size fine (~5–10 MB/arch with the existing lto = "fat" + strip profile).
• Prereq: cosy is currently a bin-only crate — needs a small src/lib.rs refactor to expose the render pipeline as a library.

Phased plan

1. Phase 1 (1–2 wk): lib target in cosy + glue crate with flutter_rust_bridge → render cosy templates and vectorize images on-device (iOS + Android demo).
2. Phase 2 (2–3 wk): geometric sketch parser (covecto B/W + rectangle/line fitting → layout JSON → template scaffold). Needs a small corpus of real hand sketches for validation.
3. Phase 3 (optional): on-device Florence-2 (ONNX int8) or cloud VLM fallback if the geometric path struggles with messy sketches.

Covecto repo note: healthy (v0.1.1, default branch develop, matches the codecora repo standard); a few stale branches (chore/v0.1.1, ci/install-script, docs/hero-landing-page, feat/export-pdf-eps-and-cli-format) could get the same cleanup cosy got when we start active work.