Support reference images for image generation

[paullizer]

Summary

SimpleChat image generation currently generates new images from text-only context. It does not have a way to pass images that were uploaded in chat, referenced from existing conversation context, or found through web search into image generation requests.

User Value

Users expect modern image generation workflows to accept visual references. This would let them upload an image, reuse an image already in the conversation, or ask the app to locate a relevant public image such as a profile photo, product image, logo, or object reference and use it to guide a newly generated image.

Proposed Behavior

Image generation should be able to include one or more reference images gathered from supported sources:

• Images uploaded directly in the chat.
• Images already attached to or referenced by the current chat context.
• Images discovered through web search or other retrieval flows when the user asks to use a public image, such as a LinkedIn profile image, company logo, or object reference.
• Reference-image provenance should be visible enough for users/admins to understand what image source was used.
• Unsupported, inaccessible, unsafe, or ambiguous image sources should fail gracefully with clear user guidance.

Acceptance Criteria

• Users can provide an uploaded image as context for a new image generation request.
• Image generation can reuse eligible images from the active chat context.
• Image generation can use retrieved public image references when the user explicitly asks for them and the source is accessible.
• The backend validates image source, type, size, accessibility, and policy constraints before sending images to the image generation provider.
• The UI makes selected reference images clear before generation and allows users to remove them.
• Tests cover uploaded-image references, chat-context references, retrieved-image references, unsupported images, and provider errors.
• Documentation explains supported reference-image sources, limitations, and privacy/security expectations.

Notes

This likely spans chat attachments/context, web retrieval/search flows, image-generation request construction, provider capability detection, frontend selection/preview UX, and security/privacy review for external image retrieval.

[paullizer]

Implementation update for Phase 1 reference-image generation support on branch feature/image-generation-reference-images.

What changed:

• Added backend reference-image resolution and target workspace selection for chat/workspace images.
• Added save/copy behavior so references and generated outputs land in the correct personal or group workspace, with public workspace references saving generated output to personal.
• Wired reference payloads through /api/chat image generation and /api/chat/image-proposals/generate inline proposal approval.
• Added the chat image-reference tray, “Use as image reference” action for existing chat images, and inline proposal approval support for saved references.
• Added metadata linking conversation, source image, saved reference document, generated image message, and generated workspace document.
• Added feature documentation and release notes entry for v0.250.015.

Validation completed:

• functional_tests/test_image_reference_generation.py: 5/5 passed.
• Route policy tests: 12/12 passed.
• Python compile validation for changed backend and test files passed.
• VS Code diagnostics for changed Python/JS/HTML/docs/test files showed no errors.
• git diff --check passed.

Known local validation gaps:

• node --check could not run because node is not available in the shell.
• The new Playwright UI test file compiles, but pytest is not installed in the current venv, so the UI test was not executed locally.

Follow-up:

• Phase 2 should add web/search-discovered image candidates with explicit user selection before save/use.

[paullizer]

Follow-up update on the Phase 1 implementation after local testing with selected personal workspace images:

Fixes added:

• Selected workspace images are now only auto-collected as generation references when the user explicitly asks to create/generate/draw/design/render a visual.
• Normal Q&A prompts against selected image documents, such as "summarize" or "what is this image about", now inject available image vision metadata directly into chat context instead of depending on search recall.
• Provider 404 Resource not found failures during reference-image generation are now converted into a clear deployment/API/reference-image compatibility message instead of raw provider JSON.
• App/docs/release notes version is now 0.250.017.

Validation completed:

• functional_tests/test_image_reference_generation.py: 8/8 passed.
• Route policy tests: 12/12 passed.
• node --check passed for changed chat JS files.
• Python compile validation passed for changed backend/test files.
• VS Code diagnostics showed no errors for changed Python/JS/HTML/docs/test files.
• git diff --check passed.
• ui_tests/test_chat_image_reference_tray.py ran and skipped cleanly because SIMPLECHAT_PLAYWRIGHT_CHAT_URL is not configured locally.

Local tooling installed for validation:

• Node.js LTS via winget.
• pytest, playwright, azure-mgmt-playwright, and olefile in the repo venv.
• Playwright Chromium browser runtime.

[paullizer]

Follow-up fix for selected-image Q&A streaming error:

• Fixed cannot access local variable 'combined_documents' where it is not associated with a value by initializing the selected-image citation/document list before the selected-image context hook runs in both chat paths.
• This preserves the direct selected-image Q&A context and prevents stream interruption before any content is received.
• Version is now 0.250.018.

Validation:

• functional_tests/test_image_reference_generation.py: 8/8 passed.
• Route policy tests: 12/12 passed.
• Python compile checks passed.
• node --check passed for changed chat JS files.
• VS Code diagnostics showed no errors.
• git diff --check passed.
• ui_tests/test_chat_image_reference_tray.py ran and skipped cleanly because SIMPLECHAT_PLAYWRIGHT_CHAT_URL is not configured locally.

[paullizer]

Follow-up update for provider reference-image failures during approval:

• If the selected image generation deployment rejects true reference-image byte input, SimpleChat now falls back to text-to-image generation using the selected image's vision description.
• The generated proposal card surfaces a warning that the deployment could not process reference-image bytes and that the image was generated from the selected image description instead.
• Reference metadata and workspace save/link behavior are preserved.
• Version is now 0.250.019.

Validation:

• functional_tests/test_image_reference_generation.py: 9/9 passed.
• Route policy tests: 12/12 passed.
• Python compile checks passed.
• node --check passed for changed chat JS files.
• VS Code diagnostics showed no errors.
• git diff --check passed.
• ui_tests/test_chat_image_reference_tray.py ran and skipped cleanly because SIMPLECHAT_PLAYWRIGHT_CHAT_URL is not configured locally.

[paullizer]

Follow-up update for true GPT-image reference editing and saved workspace files:

• Confirmed the OpenAI Python SDK supports multipart images.edit(image=..., prompt=..., model=...) for reference images.
• Updated reference-image edit calls to use Azure OpenAI image API version 2025-04-01-preview or later when older configured image-generation versions are present. This addresses /images/edits returning 404 Resource not found while text-only image generation works.
• Updated the default image generation API version to 2025-04-01-preview for new configurations.
• Saved reference/generated image workspace documents now use readable filenames instead of GUID-heavy names.
• Saved image documents now include the conversations tag plus chat-upload-style metadata: created_from_chat_upload, conversation_id, conversation_url, chat_message_id, link state, and source subtype.
• Version is now 0.250.021.

Validation:

• functional_tests/test_image_reference_generation.py: 11/11 passed.
• functional_tests/test_image_generation_model_compatibility.py: 4/4 passed.
• Route policy tests: 12/12 passed.
• Python compile checks passed.
• node --check passed for changed chat JS files.
• VS Code diagnostics showed no errors.
• git diff --check passed.
• ui_tests/test_chat_image_reference_tray.py ran and skipped cleanly because SIMPLECHAT_PLAYWRIGHT_CHAT_URL is not configured locally.