Feat(model support): ideogram4 support

[Pfannkuchensack]

Summary

Adds first-class Ideogram 4 (text-to-image) support to InvokeAI — a new open-weight 9.3B single-stream DiT with a Qwen3-VL-8B text encoder and flow-matching sampler.

The defining trait of this model is that it is trained on a structured JSON prompt that describes the scene as a list of regions, each with a bounding box ([y_min, x_min, y_max, x_max], normalized 0–1000, origin top-left) and a text description. Plain text works but is markedly lower quality.

The headline feature here is that this JSON is auto-assembled on the frontend from the existing Canvas Regional Guidance layers: the global prompt becomes the overall description, and each enabled region contributes one element (its drawn rect → bbox, its prompt → description). Users can also paste raw JSON to drive the model directly.

Why this is purely frontend string assembly: Ideogram 4 does not use spatial attention masks for regions (unlike FLUX/SDXL/Z-Image regional guidance) — the region boxes are encoded as text inside the single JSON string fed to Qwen3-VL. So the backend only ever sees one prompt string; no mask-conditioning code is touched.

How

Backend (invokeai/backend/ideogram4/, vendored from the Apache-2.0 reference, copyright headers retained):

• DiT (modeling_ideogram4.py), FLUX2-style KL VAE (autoencoder.py + latent_norm.py), logit-normal flow-match scheduler + presets (scheduler.py, sampler_configs.py), nf4/fp8 quantized loading, and InvokeAI-side denoise.py / text_encoding.py / sampling_utils.py.
• Dual-branch asymmetric CFG: positive runs the conditional transformer over [text]+[image] tokens, negative runs the unconditional transformer over image-only tokens with zeroed LLM features (v = gw·pos + (1−gw)·neg). ⇒ no negative prompt. A transformer_pair.py wrapper keeps both transformers co-resident through the loop so the cache doesn't swap them every step (nf4 ≈ 10 GB resident during denoise; fits 24 GB).
• Model-manager registration: new BaseModelType.Ideogram4, a Qwen3-VL text-encoder type, config detector + diffusers config, and a loader mirroring Z-Image.
• Four invocations: ideogram4_model_loader, ideogram4_text_encoder, ideogram4_denoise, ideogram4_latents_to_image; new Ideogram4ConditioningInfo + field/output; ideogram4_txt2img generation mode.

Frontend:

• buildIdeogram4Prompt.ts — Regions→JSON assembly (raw-JSON passthrough; stable key order; bbox clamped/rounded to 0–1000) with unit tests.
• buildIdeogram4Graph.ts — text2img-only graph builder + enqueue wiring. Uses a decoy string node for the assembled JSON so the linear-UI batch injector doesn't clobber it, while plain-text prompts still flow through the real prompt node (so dynamic prompts / batching keep working).
• Params + UI: a Sampler Preset combobox (Quality 48 / Default 20 / Turbo 12) as the primary control, plus Advanced overrides that actually apply to this model — Steps, Guidance Scale, Schedule Shift (mu) and a Color Palette picker. The irrelevant Advanced controls (VAE, CLIP Skip, CFG Rescale, Seamless, Color Compensation) are hidden for Ideogram 4.
• Metadata recall for all of the above.

Dependencies: bumps transformers to >=5.5,<5.6 (Qwen3-VL landed in 4.57; the encoder needs it) and compel to >=2.4.0,<3, with the necessary adaptations to the FLUX / Z-Image loaders, the safety checker, the HF metadata fetcher and model_util.

Out of scope (v1): img2img / inpaint / outpaint, ControlNet / IP-Adapter / LoRA, and the optional local "Magic Prompt" plain-text→JSON expander (parked — see Merge Plan).

Related Issues / Discussions

• Overlaps the transformers 5.x bump tracked in PR feat - Migrate to Transformers 5.5.4 #9248 — this branch carries the same bump (>=5.5,<5.6) plus the cross-model adaptations it requires.

QA Instructions

Requires the gated weights (ideogram-ai/ideogram-4-nf4 — nf4 is the 24 GB path, CUDA/bitsandbytes only) plus the Qwen3-VL encoder + VAE sub-dependencies.

1. Install & select an Ideogram 4 model; open the Canvas/Generate tab. Confirm the model shows under its own group, dimensions default to 1024×1024 (multiples of 16), and the Generation settings show the Sampler Preset control instead of Scheduler/CFG.
2. Regions → JSON: type an overall description, add 1–2 Regional Guidance layers each with a prompt + a drawn box, and Invoke. In the result's metadata, confirm the assembled JSON has the correct key order and elements[*].bbox (0–1000, [y_min, x_min, y_max, x_max]) matching where you drew the boxes, and that element placement in the image roughly matches.
3. Raw-JSON passthrough: paste a hand-written JSON object into the prompt box → it is sent unchanged (no region wrapping).
4. Plain text: with no regions and a plain prompt, confirm normal text2img still works and that dynamic prompts / batching still expand (decoy-node behavior).
5. Sampler presets: Quality / Default / Turbo produce the expected step counts.
6. Advanced overrides: Steps / Guidance Scale / mu show the active preset's value as the "auto" default and can be overridden + reset; Color Palette swatches inject style_description.color_palette (auto-build mode only — ignored for raw JSON).
7. Metadata recall: load an Ideogram 4 image and recall — preset, overrides and palette restore (and are guarded to only recall onto an Ideogram 4 model).

Frontend gates (from invokeai/frontend/web/): pnpm lint and pnpm test:no-watch (includes buildIdeogram4Prompt.test.ts). tsc clean.

Merge Plan

• Large PR + dependency bump. This raises transformers to >=5.5,<5.6 and compel to >=2.4.0,<3, which touches many models (FLUX, Z-Image, safety checker, HF metadata fetch). Coordinate with / sequence after PR feat - Migrate to Transformers 5.5.4 #9248 (the transformers 5.x bump) to avoid a double-bump conflict, and time it to not collide with a pending release. Broad regression QA across existing model types is warranted, not just Ideogram 4.
• Follow-ups (not in this PR): GGUF loader for the custom DiT (more VRAM headroom), starter_models.py entries, and the optional local Magic Prompt node (blocked on the upstream system-prompts PR feat: add System Prompts library for Expand Prompt button #9152, whose migration_32 collides with main and must be renumbered to 33 first).

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable) — frontend unit tests for the Regions→JSON assembly
• ❗Changes to a redux slice have a corresponding migration — new params fields use zod defaults; confirm if a migration is needed
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

[JPPhoto]

• invokeai/backend/model_manager/load/model_loaders/ideogram4.py:140 accepts Ideogram 4 fp8 models up front, but _load_text_encoder() raises NotImplementedError for fp8 text encoders at invokeai/backend/model_manager/load/model_loaders/ideogram4.py:141-146. The config advertises "nf4 / fp8 quantized" support and identifies any Ideogram4Pipeline folder at invokeai/backend/model_manager/configs/main.py:1292-1313, so an fp8 Ideogram model can be imported, selected, and only fail during prompt encoding. To expose this issue, add a loader/config test that imports an Ideogram folder whose text_encoder/config.json has ideogram_fp8_weight_only: true and asserts it is rejected during identification or fully loads.

• invokeai/frontend/web/src/features/gallery/components/ImageMetadataViewer/ImageMetadataActions.tsx:32-69 does not include the new Ideogram metadata handlers, even though they are registered in invokeai/frontend/web/src/features/metadata/parsing.tsx:1800-1804 and written by the graph at invokeai/frontend/web/src/features/nodes/util/graph/generation/buildIdeogram4Graph.ts:123-129. The Recall Parameters tab will not show individual recall controls for sampler preset, Ideogram steps override, guidance override, mu override, or color palette. To expose this issue, add a metadata action test that passes metadata with ideogram4_sampler_preset, ideogram4_steps, ideogram4_guidance_scale, ideogram4_mu, and ideogram4_color_palette and asserts those rows are rendered.

• invokeai/frontend/web/src/features/nodes/util/graph/generation/buildIdeogram4Graph.ts:134-136 writes ideogram4_caption for structured/regional prompts, but there is no corresponding metadata handler in invokeai/frontend/web/src/features/metadata/parsing.tsx:1754-1808 and no action entry in invokeai/frontend/web/src/features/gallery/components/ImageMetadataViewer/ImageMetadataActions.tsx:32-69. For regional guidance, the actual encoded prompt is the assembled JSON caption, while positive_prompt remains the raw prompt via the decoy node at invokeai/frontend/web/src/features/nodes/util/graph/generation/buildIdeogram4Graph.ts:110-139; users cannot recall or inspect that real prompt from the recall UI. To expose this issue, add a test that builds structured Ideogram metadata with ideogram4_caption and verifies the recall UI can display and recall the encoded caption.

• invokeai/frontend/web/src/features/parameters/components/Core/ParamIdeogram4SamplerPreset.tsx:11-14 hardcodes visible English option labels, and invokeai/frontend/web/src/features/metadata/parsing.tsx:950-1007 hardcodes visible Auto strings for Ideogram metadata values. Only parameters.samplerPreset and parameters.colorPalette were added to en.json, so these new user-visible strings will remain English in localized builds. To expose this issue, add a frontend i18n test or snapshot that renders the Ideogram controls under a non-English locale and asserts option/value labels come from translation keys.

• invokeai/README.md:78 is the only user-facing documentation addition for Ideogram 4, despite the PR adding a new model family, non-commercial weight caveat, model import constraints, sampler presets, advanced overrides, color palette prompting, and regional-guidance-to-JSON behavior. The docs search shows no Ideogram page under docs/. Expected docs should be added under invokeai/docs/src/content/docs/concepts/models.mdx for model/import/license requirements and invokeai/docs/src/content/docs/concepts/prompting-guide.mdx or invokeai/docs/src/content/docs/concepts/image-generation.mdx for Ideogram prompting, sampler presets, color palette, and regional guidance behavior.

• invokeai/backend/model_manager/load/model_loaders/ideogram4.py:170 loads the Qwen3-VL text encoder with strict=False and does not inspect missing or unexpected keys. If the quantized checkpoint layout differs from the constructed AutoModel after quantization_config is cleared at invokeai/backend/model_manager/load/model_loaders/ideogram4.py:153-168, this can silently leave missing parameters instead of failing at load time. I did not find a test fixture proving the real Ideogram nf4 state dict covers this path.