feat: add secure BYOD database connections

.env.example

@@ -9,3 +9,13 @@ EMBEDDING_PROVIDER=local
 EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
 ENABLE_SCHEMA_RAG=false
 ENABLE_GOLDEN_RECORDS=false
+ENABLE_EXTERNAL_CONNECTIONS=true
+ALLOW_PRIVATE_DATABASE_HOSTS=false
+CONNECTION_ENCRYPTION_KEY=
+AUTH_SIGNING_KEY=
+DATABASE_CONNECTION_TIMEOUT_SECONDS=5
+DATABASE_STATEMENT_TIMEOUT_MS=10000
+DATABASE_MAX_RESULT_ROWS=500
+DATABASE_MAX_SQL_LENGTH=10000
+SCHEMA_FULL_CONTEXT_TABLE_LIMIT=20
+SCHEMA_RETRIEVAL_TABLE_LIMIT=8

README.md

@@ -1,11 +1,11 @@
 # QueryMindAI
 > Ask questions, not SQL.
 
-QueryMindAI is an open-source natural-language analytics platform that converts business questions into safe, explainable SQL queries over structured data.
+QueryMindAI is an open-source AI data workspace that lets users connect a read-only database and query it safely using natural language.
 
 ## Current status
 
-QueryMindAI is an early production-minded release. Implemented today: PostgreSQL schema introspection, optional schema-vector retrieval, provider-based SQL generation, parser validation, read-only execution, verified-example retrieval, query history, demo data, Docker Compose, and Render configuration. Authentication, multi-tenant credential storage, a polished connection manager, and comparative model benchmarks are roadmap items. Some retained dashboard views are explicitly demo UI.
+QueryMindAI is an early production-minded release. Implemented today: encrypted saved PostgreSQL connections, public-host SSRF checks, SSL connections, catalog snapshots, structured SQL generation, explicit approval drafts, parser/table validation, read-only execution, verified examples, history, audit events, Docker Compose, and Render configuration. Signed anonymous workspace sessions provide ownership isolation for the public demo; account login and enterprise identity integration remain roadmap work.
 
 ## Screenshots
 
@@ -24,27 +24,25 @@ QueryMindAI is an early production-minded release. Implemented today: PostgreSQL
 
 ```mermaid
 flowchart LR
-  U[Next.js web] --> A[FastAPI API]
-  A --> P[Query orchestrator]
-  P --> S[Schema introspection / optional RAG]
-  P --> V[Verified examples]
-  P --> L[LLM provider client]
-  P --> G[sqlglot safety gate]
-  G --> D[(Read-only PostgreSQL)]
+  B[Browser] --> W[QueryMindAI Web] --> A[QueryMindAI API]
+  A --> C[Connection Manager] --> S[Schema Catalog]
+  S --> L[LLM Query Planner] --> V[SQL Validator]
+  V --> E[Execution Engine] --> D[(Customer PostgreSQL Database)]
 ```
 
 ### Query lifecycle
 
 ```mermaid
 flowchart LR
-  Q[Validate question] --> S[Retrieve schema] --> E[Optional verified example]
-  E --> L[Generate SQL] --> C[Clean] --> V[Parse + validate]
-  V --> X[Read-only execute] --> R[Structured response]
+  Q[Validate question] --> S[Retrieve bounded schema] --> L[Generate structured SQL]
+  L --> V[Parse, table-check, cap LIMIT] --> D[Store expiring draft]
+  D --> H[Human review and approval] --> R[Revalidate SQL]
+  R --> X[Read-only execute with timeout] --> O[Results and history]
 ```
 
 ## Tech stack
 
-FastAPI, SQLAlchemy, Alembic, PostgreSQL/pgvector, sqlglot, OpenAI-compatible provider API, Next.js 15, React 19, TypeScript, Tailwind CSS, Docker Compose, GitHub Actions, and Render.
+FastAPI, SQLAlchemy, Alembic, PostgreSQL, sqlglot, OpenAI-compatible provider API, Next.js 15, React 19, TypeScript, Tailwind CSS, Docker Compose, GitHub Actions, and Render.
 
 ## Repository structure
 
@@ -59,7 +57,7 @@ scripts        Operator helpers
 
 ## Quickstart
 
-Prerequisites: Python 3.12, Node 20, PostgreSQL 16 with pgvector, and optionally Ollama.
+Prerequisites: Python 3.12, Node 20, PostgreSQL 16+, and optionally Ollama.
 
 ```bash
 cp apps/api/.env.example apps/api/.env
@@ -105,8 +103,10 @@ For the default Compose stack, optional vector features are off so schema intros
 | `LLM_MODEL`, `LLM_FALLBACK_MODEL` | Primary and retry fallback generation models |
 | `EMBEDDING_PROVIDER`, `EMBEDDING_MODEL` | Local semantic-retrieval configuration |
 | `ENABLE_SCHEMA_RAG`, `ENABLE_GOLDEN_RECORDS` | Optional embedding features |
-| `ENABLE_EXTERNAL_CONNECTIONS` | Raw connection endpoint; defaults false |
-| `MAX_QUERY_ROWS`, `STATEMENT_TIMEOUT_MS` | Execution safeguards |
+| `ENABLE_EXTERNAL_CONNECTIONS`, `ALLOW_PRIVATE_DATABASE_HOSTS` | BYOD and SSRF policy flags |
+| `CONNECTION_ENCRYPTION_KEY`, `AUTH_SIGNING_KEY` | Backend-only Fernet and session signing secrets |
+| `DATABASE_*` | Connect timeout, statement timeout, SQL length, and result caps |
+| `SCHEMA_*_TABLE_LIMIT` | Full-context and retrieval bounds |
 | `NEXT_PUBLIC_API_URL` | Browser-visible API `/api/v1` URL |
 
 See the component `.env.example` files for all defaults.
@@ -120,6 +120,10 @@ Semantic retrieval uses lazy, process-cached `sentence-transformers/all-MiniLM-L
 ## API overview
 
 - `GET /health`, `GET /ready`
+- `POST /api/v1/auth/session`
+- `POST/GET/DELETE /api/v1/connections[...]`
+- `POST /api/v1/queries/generate`, `POST /api/v1/queries/execute`
+- `GET /api/v1/queries/history`, `GET /api/v1/queries/{id}`
 - `GET /api/v1/schema`, `GET /api/v1/query-history`
 - `POST /api/v1/query`
 - `POST /api/v1/verified-examples`
@@ -141,9 +145,9 @@ The API validates question size, limits generated SQL length, rejects comments/m
 
 ## Limitations and roadmap
 
-Current limitations include PostgreSQL-only execution, provider-dependent SQL quality, unavailable verified-example saving when embeddings are disabled, no auth/multi-tenancy, no encrypted external credential vault, in-memory external schema cache, and demo-only management screens.
+Current limitations include PostgreSQL-only execution, publicly reachable hosts only, provider-dependent SQL quality, application-level encryption rather than a managed vault, anonymous browser-bound sessions rather than account authentication, no rate limiter, and no guaranteed detection of inherited write privileges.
 
-Roadmap: authenticated workspaces, secure connection management, API-backed connection pages, richer history review, execution-backed evaluations, additional SQL dialects, observability exports, and reviewed embedding lifecycle tools.
+Roadmap: OIDC accounts and organizations, MySQL, richer history review, execution-backed evaluations, observability exports, and a customer-side connector for private databases: `Customer Database → QueryMind Connector inside customer network → outbound TLS → QueryMindAI Cloud`. The connector is documented only and is not implemented.
 
 ## Contributing and license
 

apps/api/.env.example

@@ -13,6 +13,15 @@ EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2
 ENABLE_SCHEMA_RAG=false
 ENABLE_GOLDEN_RECORDS=false
 ENABLE_EXTERNAL_CONNECTIONS=false
+ALLOW_PRIVATE_DATABASE_HOSTS=false
+CONNECTION_ENCRYPTION_KEY=
+AUTH_SIGNING_KEY=
+DATABASE_CONNECTION_TIMEOUT_SECONDS=5
+DATABASE_STATEMENT_TIMEOUT_MS=10000
+DATABASE_MAX_RESULT_ROWS=500
+DATABASE_MAX_SQL_LENGTH=10000
+SCHEMA_FULL_CONTEXT_TABLE_LIMIT=20
+SCHEMA_RETRIEVAL_TABLE_LIMIT=8
 MAX_QUERY_ROWS=100
 STATEMENT_TIMEOUT_MS=10000
 MAX_SQL_LENGTH=20000

apps/api/README.md

@@ -8,7 +8,7 @@ The modular FastAPI service turns validated questions into schema-aware PostgreS
 
 ## Endpoints
 
-`GET /health` is liveness. `GET /ready` checks PostgreSQL. Under `/api/v1`: `GET /schema`, `GET /query-history`, `POST /query`, `POST /verified-examples`; `/golden-record` is a deprecated compatibility alias. `/connect-and-query` returns 403 unless explicitly enabled.
+`GET /health` is liveness and `GET /ready` checks the application PostgreSQL database. BYOD endpoints under `/api/v1` include signed session creation; connection create/list/get/test/refresh/schema/delete; query generate/execute/history/get; and saving an executed query as verified. All BYOD resources are scoped to the verified session subject. Legacy demo endpoints remain for compatibility.
 
 ## Configuration and providers
 
@@ -28,6 +28,8 @@ uvicorn app.main:app --reload --port 8000
 
 Production startup never creates tables. Alembic creates application tables and pgvector. The demo seed is idempotent. Use `../../scripts/create_readonly_role.sql` as a reviewed template and run query traffic with that role.
 
+Saved customer credentials require `CONNECTION_ENCRYPTION_KEY` (a Fernet key). Signed workspace sessions require a separate `AUTH_SIGNING_KEY`. Generate them with the commands in `../../docs/deployment.md`. BYOD accepts URL or structured PostgreSQL input, requires SSL, blocks unsafe networks by default, and stores normalized schema snapshots without row data.
+
 ## Safety model
 
 `sqlglot` parses PostgreSQL and enforces one read-only query, accepts SELECT/CTE SELECT, rejects comments, multiple/write/admin statements, caps LIMIT, and limits SQL size. Execution uses `SET LOCAL TRANSACTION READ ONLY` plus statement timeout. Database permissions remain the primary boundary.

apps/api/app/api/v1/endpoints/byod.py

@@ -0,0 +1,150 @@
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, status
+from sqlalchemy import select, text
+from sqlalchemy.orm import Session
+
+from app.audit.service import record_audit
+from app.auth.dependencies import get_current_user_id, issue_session
+from app.auth.schemas import SessionResponse
+from app.connections.schemas import (ConnectionCreate, ConnectionResponse, ConnectionTestResponse,
+                                     SchemaResponse)
+from app.connections.security import decrypt_config, normalize_connection_input
+from app.connections.service import (create_saved_connection, inspect_and_check, latest_snapshot, masked,
+                                     owned_connection, refresh_owned_schema)
+from app.core.config import settings
+from app.core.exceptions import DependencyError, FeatureDisabledError, NotFoundError, QueryMindError
+from app.db.session import get_db
+from app.models.byod import DatabaseConnection
+from app.query_planner.schemas import (QueryDraftResponse, QueryExecuteRequest, QueryExecutionResponse,
+                                       QueryGenerateRequest)
+from app.query_planner.service import generate_draft
+from app.execution.service import execute_draft
+from app.schemas.query_schema import VerifiedExampleRequest
+from app.services.golden_record_service import save_golden_record
+
+router = APIRouter()
+UserId = Annotated[str, Depends(get_current_user_id)]
+Db = Annotated[Session, Depends(get_db)]
+
+
+def enabled():
+    if not settings.ENABLE_EXTERNAL_CONNECTIONS:
+        raise FeatureDisabledError("Saved external database connections are disabled")
+
+
+@router.post("/auth/session", response_model=SessionResponse)
+def create_workspace_session():
+    enabled(); token, _, expires_at = issue_session()
+    return {"access_token": token, "expires_at": expires_at}
+
+
+@router.post("/connections", response_model=ConnectionResponse, status_code=status.HTTP_201_CREATED)
+def create_connection(request: ConnectionCreate, user_id: UserId, db: Db):
+    enabled()
+    config = normalize_connection_input(request)
+    try:
+        connection, warnings = create_saved_connection(db, user_id, request.name, config)
+        return masked(connection, warnings)
+    except QueryMindError:
+        db.rollback(); raise
+    except Exception as exc:
+        db.rollback()
+        raise DependencyError("The database connection could not be established") from exc
+
+
+@router.get("/connections", response_model=list[ConnectionResponse])
+def list_connections(user_id: UserId, db: Db):
+    enabled()
+    items = db.scalars(select(DatabaseConnection).where(DatabaseConnection.user_id == user_id)
+                       .order_by(DatabaseConnection.created_at.desc())).all()
+    return [masked(item) for item in items]
+
+
+@router.get("/connections/{connection_id}", response_model=ConnectionResponse)
+def get_connection(connection_id: str, user_id: UserId, db: Db):
+    enabled(); return masked(owned_connection(db, connection_id, user_id))
+
+
+@router.post("/connections/{connection_id}/test", response_model=ConnectionTestResponse)
+def test_connection(connection_id: str, user_id: UserId, db: Db):
+    enabled(); saved = owned_connection(db, connection_id, user_id)
+    try:
+        _, read_only, warnings = inspect_and_check(decrypt_config(saved.encrypted_connection_data))
+    except Exception as exc:
+        saved.status = "failed"; record_audit(db, user_id, "connection_tested", saved.id, {"status": "failed"}); db.commit()
+        raise DependencyError("The database connection test failed") from exc
+    saved.status = "connected"; record_audit(db, user_id, "connection_tested", saved.id, {"status": "connected"}); db.commit()
+    return {"status": "connected", "read_only": read_only, "warnings": warnings}
+
+
+@router.post("/connections/{connection_id}/refresh-schema", response_model=SchemaResponse)
+def refresh_schema(connection_id: str, user_id: UserId, db: Db):
+    enabled(); saved = owned_connection(db, connection_id, user_id)
+    snapshot = refresh_owned_schema(db, saved, user_id)
+    return {"connection_id": saved.id, "schema_hash": snapshot.schema_hash,
+            "metadata": snapshot.normalized_metadata, "updated_at": snapshot.updated_at}
+
+
+@router.get("/connections/{connection_id}/schema", response_model=SchemaResponse)
+def get_schema(connection_id: str, user_id: UserId, db: Db):
+    enabled(); saved = owned_connection(db, connection_id, user_id); snapshot = latest_snapshot(db, saved.id)
+    return {"connection_id": saved.id, "schema_hash": snapshot.schema_hash,
+            "metadata": snapshot.normalized_metadata, "updated_at": snapshot.updated_at}
+
+
+@router.delete("/connections/{connection_id}", status_code=status.HTTP_204_NO_CONTENT)
+def delete_connection(connection_id: str, user_id: UserId, db: Db):
+    enabled(); saved = owned_connection(db, connection_id, user_id)
+    record_audit(db, user_id, "connection_deleted", saved.id, {"name": saved.name})
+    db.execute(text("DELETE FROM golden_records WHERE connection_key = :key"), {"key": saved.id})
+    db.execute(text("DELETE FROM schema_embeddings WHERE connection_key = :key"), {"key": saved.id})
+    db.delete(saved); db.commit()
+
+
+@router.post("/queries/generate", response_model=QueryDraftResponse)
+async def generate_query(request: QueryGenerateRequest, user_id: UserId, db: Db):
+    enabled(); draft = await generate_draft(db, user_id, request.connection_id, request.question)
+    return {"draft_id": draft.id, "connection_id": draft.connection_id, "sql": draft.sql,
+            "explanation": draft.explanation, "assumptions": draft.assumptions, "warnings": draft.warnings,
+            "confidence": float(draft.confidence), "expires_at": draft.expires_at}
+
+
+@router.post("/queries/execute", response_model=QueryExecutionResponse)
+def run_query(request: QueryExecuteRequest, user_id: UserId, db: Db):
+    enabled(); return execute_draft(db, user_id, request.draft_id)
+
+
+@router.get("/queries/history")
+def byod_history(user_id: UserId, db: Db, limit: int = 50):
+    enabled()
+    rows = db.execute(text("""
+      SELECT id, connection_id, question, generated_sql AS sql, execution_status, row_count,
+             duration_ms, warnings, created_at FROM query_history
+      WHERE user_id = :user_id ORDER BY created_at DESC LIMIT :limit
+    """), {"user_id": user_id, "limit": min(max(limit, 1), 100)}).mappings().all()
+    return {"items": [dict(row) for row in rows]}
+
+
+@router.post("/queries/{query_id}/verified-example", status_code=status.HTTP_201_CREATED)
+async def save_query_as_verified(query_id: int, user_id: UserId, db: Db):
+    enabled()
+    row = db.execute(text("""SELECT connection_id, question, generated_sql FROM query_history
+      WHERE id = :id AND user_id = :user_id AND execution_status = 'success'"""),
+      {"id": query_id, "user_id": user_id}).mappings().first()
+    if not row: raise NotFoundError("Successful query history item was not found")
+    owned_connection(db, row["connection_id"], user_id)
+    saved = await save_golden_record(VerifiedExampleRequest(question=row["question"], sql=row["generated_sql"],
+                                      connection_key=row["connection_id"], source="approved_execution"), db)
+    return {"id": saved.id, "status": "saved", "message": "Verified example saved."}
+
+
+@router.get("/queries/{query_id}")
+def get_query(query_id: int, user_id: UserId, db: Db):
+    enabled()
+    row = db.execute(text("""
+      SELECT id, connection_id, question, generated_sql AS sql, execution_status, row_count,
+             duration_ms, warnings, created_at FROM query_history WHERE id = :id AND user_id = :user_id
+    """), {"id": query_id, "user_id": user_id}).mappings().first()
+    if not row: raise NotFoundError("Query history item was not found")
+    return dict(row)

apps/api/app/api/v1/endpoints/query.py

@@ -4,20 +4,17 @@
 
 from app.core.config import settings
 from app.core.exceptions import FeatureDisabledError
-from app.db.dynamic_session import get_dynamic_session
 from app.db.session import get_db
-from app.schemas.query_schema import (ConnectQueryRequest, QueryRequest, QueryResponse,
+from app.schemas.query_schema import (QueryRequest, QueryResponse,
                                       VerifiedExampleRequest, VerifiedExampleResponse)
 from app.services.golden_record_service import save_golden_record
-from app.services.query_service import execute_query
-from app.services.schema_introspect_service import get_dynamic_schema
 
 router = APIRouter()
 
 
 @router.post("/query", response_model=QueryResponse)
 async def query_db(request: QueryRequest, db: Session = Depends(get_db)):
-    return await execute_query(db, request.question)
+    raise FeatureDisabledError("Combined generation and execution is disabled; use /queries/generate then /queries/execute")
 
 
 @router.get("/schema")
@@ -43,17 +40,6 @@ def query_history(limit: int = 20, db: Session = Depends(get_db)):
     return {"items": [dict(row) for row in rows], "source": "real_api_data"}
 
 
-@router.post("/connect-and-query", response_model=QueryResponse)
-async def connect_and_query(request: ConnectQueryRequest):
-    if not settings.ENABLE_EXTERNAL_CONNECTIONS:
-        raise FeatureDisabledError("External database connections are disabled")
-    db = get_dynamic_session(request.db_config.model_dump())
-    try:
-        return await execute_query(db, request.question, schema_override=get_dynamic_schema(db))
-    finally:
-        db.close()
-
-
 @router.post("/verified-examples", response_model=VerifiedExampleResponse, status_code=status.HTTP_201_CREATED)
 async def add_verified_example(request: VerifiedExampleRequest, db: Session = Depends(get_db)):
     row = await save_golden_record(request, db)

apps/api/app/audit/service.py

@@ -0,0 +1,8 @@
+from sqlalchemy.orm import Session
+
+from app.models.byod import AuditLog
+
+
+def record_audit(db: Session, user_id: str, action: str, connection_id: str | None = None, metadata: dict | None = None):
+    # Metadata must contain identifiers/status only—never credentials, URLs, SQL results, or decrypted config.
+    db.add(AuditLog(user_id=user_id, connection_id=connection_id, action=action, event_metadata=metadata or {}))