Building Agentic Transform

The problem

Turning a document into a tip-style social graphic is not one model call. You need a background, readable on-image text, and a compose step. Chaining those as separate canvas nodes works, but for “follow my instructions and produce the graphic” we wanted a single automation: agentic_transform.

Early experiments put too much in the prompt loop: critique tools, color heuristics that flipped between black and white, and QA that burned retries on soft “low contrast” false positives. The current design is deliberately narrow.

System shape

Entry point matches every other transform: a Sidekiq-orchestrated automation calls Transform::AgenticTransform, which checks the process file type and hands off to Ai::Agent::Runner.

AutomationChain::GenerateTransformOutput
  → Transform::AgenticTransform
    → Ai::Agent::Runner
         ├─ Patterns::ReactEvaluate → Graph (nodes include quality_gate)
         ├─ Session + ToolRegistry (document toolkit)
         ├─ Tools: text_to_image | text_to_image_overlay | compose
         └─ Evaluate (finish-time QA; called from quality_gate)

Document inputs get a fixed toolkit via ToolRegistry, not a user-editable allowlist. That keeps the agent honest: it can only build the graphic pipeline we support today.

The tool loop (not a free-form chat)

The runner maintains message history and calls OpenAI with tool schemas. Each iteration is a step (capped at 10). When the model emits tool calls, we execute them against an allowlist and append tool results (including artifact labels) back into the conversation.

When the model stops calling tools, we do not trust “I’m done.” The quality_gate node runs only when there is a new deliverable. If evaluate is on and a rubric is set (or the writer used a text overlay), Ai::Agent::Capabilities::Evaluate runs a judge LLM on a private transcript. The judge may call QA tools, then submit_verdict. Only the feedback string re-enters the writer ReAct chat. Reject → retry. Pass (or rejection budget exhausted) → finalize.

• Steps: tool rounds / planning turns (max 10)
• QA rejections: separate budget (max 2), then keep the last deliverable with a warning

Three tools, thin wrappers

The document agent only sees:

1. text_to_image: background/illustration (prompt should not paint the headline into the pixels)
2. text_to_image_overlay: transparent text layer; Tools::OverlayPaint merges QA/args intent, Transform::TextToImageOverlay paints
3. compose: places labeled artifacts into a template layout (almost always stack)

Tools share a tiny Tools::Base concern for call / context plumbing. They return ToolResult with typed Artifacts the runner stores by label so later tools (and compose slots) can reference them.

First-pass style: defaults beat luminance math

Overlay styling used to sample background luminance and flip fill colors. That fought the QA loop and still produced hard-to-read text. We simplified:

• Default to white fill + dark stroke
• Undercolor plate at a fixed opacity
• Font size from copy length + canvas height (agent hints can only shrink, not blow past fit)
• On retry, apply vision recommendations (hex, size, undercolor opacity, shorten)

The agent is told not to invent hex colors. Numeric recommendations from QA are what the runner feeds back into the next overlay call.

Evaluate is finish-time (not a writer tool)

The writer never sees evaluate tools. Overlay memory is recorded when text_to_image_overlay succeeds; the judge is then offered inspect_overlay (which calls Evaluate::Vision) only if that tool ran and the criteria care about readable text. Vision is never selected from artifact type alone.

It sends a thumbnail + expected overlay text to a vision model and asks for JSON:

• score / approved
• issues (too_small, clipped, low_contrast, …)
• recommended style knobs

GPT-4o often flags already-stroked white text as “low contrast.” We soft-override: if the used style is already light fill + dark stroke/undercolor and issues are soft-only, we accept and continue. Hard failures (clipped, missing_text, too_small) still reject.

That cut useless retries without lying about genuinely unreadable overlays.

Compose without inventing a new compositor

compose reuses Layers::TemplateMergeService. Agent runs don’t create ContentTemplate rows: Tools::EphemeralTemplate ducks the template surface in memory, including a full-bleed stack layout for background + overlay.

Artifacts are wrapped in a small duck type so loaders see paths like a project file would. Preset layouts from ContentTemplate::PRESET_TEMPLATES remain available, but the document agent prompt steers hard toward stack.

Observability

ActivityLog records steps, tool args summaries, overlay style used, QA rejections, and whether QA passed or exhausted. Entries publish as transform.agentic_turn activities so the UI can show live progress, not just a final success/fail.

When QA rejects, the reason and recommendations are written into the next user message so the model’s retry is grounded in what failed, not a generic “try again.”

What we kept

• Small fixed tool surface
• Finish-time QA staging (not a model-owned tool)
• Strong defaults + QA overrides
• Reuse of template merge / overlay transform services