Support toml and alchemy

.changeset/audit-fixes-and-tests.md

@@ -0,0 +1,17 @@
+---
+"api": patch
+"everything-dev": patch
+---
+
+Fix bugs and add test coverage for Phase 3+4 infra/alchemy features
+
+- **B1**: Fix typo in `alchemy.ts` generated comment (`[d eploy]` → `[deploy]`)
+- **B2**: Fix fire-and-forget schema creation in `db/index.ts` — `pool.on("connect")` handlers now `await` both `CREATE SCHEMA` and `SET search_path` queries in sequence (both Neon and pg paths)
+- **B3**: Fix Railway redeploy in `plugin.ts` to read `deploy.service` first, falling back to `ci.railway.service` for backward compatibility
+- **S3+S6**: Extract shared pool helpers (`buildPoolConfig`, `attachPoolSchemaHandlers`, `createCloseHandler`) in `db/index.ts` to eliminate ~70% code duplication between Neon WebSocket and pg connection paths
+- **S5**: Replace non-idiomatic `throw` with `yield* Effect.fail(...)` in `db/layer.ts` drift handlers
+
+**New tests (15 total)**:
+- `merge.test.ts`: 4 tests for `ci.railway` → `deploy` backward compat mapping, plus updated `BOS_CONFIG_ORDER` field assertions for `ci`/`infra`/`deploy`
+- `infra.test.ts`: 3 tests for `buildDatabaseConfigs` with `infraConfig` (shared DATABASE_URL, convention fallback, port preservation)
+- `tests/e2e/toml-config-pipeline.test.ts`: 8 e2e tests covering the full TOML config pipeline — read full config with `[infra]`/`[deploy]`, path resolution, extends merge, `ci.railway` mapping, shared database config, `generateAlchemyRun` output correctness, and TOML round-trip

.changeset/infra-alchemy-phases.md

@@ -0,0 +1,23 @@
+---
+"api": minor
+"everything-dev": minor
+---
+
+Phase 3: Add `[infra]` and `[deploy]` config sections for explicit infrastructure declarations
+
+- **`InfraConfigSchema`** / **`DeployConfigSchema`** in `types.ts` — new Zod schemas for `[infra.database]` (type, schemaMode, dedicated, secret), `[infra.redis]` (enabled, slug), and `[deploy]` (provider, service, redeploy). Added to both `BosConfigInputSchema` and `BosConfigSchema`.
+- **Backward compat**: `ci.railway` → `deploy` mapping during extends resolution in `mergeBosConfigWithExtends`. When `[deploy]` is absent but `ci.railway.service` is present, deplo y is derived automatically.
+- **Config ordering**: `"infra"` and `"deploy"` added to `BOS_CONFIG_ORDER` (before `"app"`).
+- **Infra planner**: `InfraInput` gains optional `infraConfig` field. `allocateDatabases` and `buildDatabaseConfigs` check `infraConfig` first, falling back to convention-based `*_DATABASE_URL` scanning when absent.
+- **Shared database**: When `[infra.database]` is declared, a single shared `DATABASE_URL` is provisioned (per-plugin schema isolation via search_path).
+
+Phase 4: Split DatabaseLive into DriverLive + MigrationLive, add Neon WebSocket Pool
+
+- **Layer split**: `DatabaseLive` decomposed into:
+  - `DriverLive` — manages driver acquire/release only
+  - `MigrationLive` — handles migration apply + drift detection (depends on `DatabaseTag`)
+  - `DatabaseLive = Layer.provideMerge(DriverLive(url), MigrationLive)` — composed, backward-compatible
+  - Enables skipping migrations when Alchemy manages prod migrations.
+- **Neon WebSocket Pool**: `createDatabaseDriver` now checks for `neon.tech` in the URL and uses `@neondatabase/serverless` Pool + `drizzle-orm/neon-serverless` with same `search_path` + `CREATE SCHEMA` support. Gracefully falls back to `pg` Pool if `@neondatabase/serverless` is not installed.
+- **Deploy script generation**: `alchemy.ts` with `generateAlchemyRun()` that writes `alchemy.run.ts` from `[deploy]` config, supporting both Railway and Alchemy providers.
+- **Dependency**: `@neondatabase/serverless: "^1.0.0"` added to api/package.json.

.changeset/schema-cleanup-and-secret.md

@@ -0,0 +1,13 @@
+---
+"api": patch
+"everything-dev": patch
+---
+
+Clean up InfraDatabaseSchema, implement `secret` field, deduplicate `isPlainObject`
+
+- **Remove `schemaMode` from `InfraDatabaseSchema`** — schema isolation (`search_path`) is handled at the driver layer in `api/src/db/layer.ts`, not in the Docker compose infra layer. Having it in both places was misleading
+- **Implement `secret` field** — when a per-plugin record specifies `secret = "CUSTOM_DB_URL"`, that env var name is used instead of the conventional `{PLUGIN}_DATABASE_URL`. Falls back to conventional naming when absent
+- **Import `isPlainObject` from `../merge`** in `cli/infra.ts` instead of redefining it locally
+- **Update `alchemy.ts`** to no longer reference `schemaMode` — replaces the broken inline ternary with a driver-layer isolation comment
+- **Update all test fixtures** to remove `schemaMode` from test data across `infra.test.ts` and `toml-config-pipeline.test.ts`
+- **Add 2 tests** for `secret` field: custom secret name resolution and conventional fallback

.changeset/toml-config-format.md

@@ -0,0 +1,14 @@
+---
+"everything-dev": minor
+---
+
+Add TOML config format support (`bos.config.toml`) as a dual-format alternative to `bos.config.json`
+
+- **`bos.config.toml`**: `findBosConfigPath` probes `bos.config.toml` first, then `bos.config.json` in directory tree walks. Both can't coexist — throws on duplicate.
+- **Format-preserving writes**: `saveBosConfig` and `reportDeployResult` detect the original config format (TOML or JSON) and write back in the same format.
+- **`disabled = true` marker**: Replaces the `null` sentinel pattern for TOML (which has no null). Schema field `disabled` on `BosPluginRefSchema` ensures Zod doesn't strip it. `cleanNullSentinels` strips `disabled: true` entries during merge.
+- **Effect-free sync helpers**: `readBosConfigSource`, `findBosConfigPath`, and `findBosConfigPathInDir` are plain sync functions that throw native errors (not `FiberFailure`), preserving readable error messages.
+- **Directory-only lookup**: Added `findBosConfigPathInDir` for cases where walking up the tree is incorrect (plugin local path resolution in `resolveComposableReference`).
+- **Published form stays JSON**: FastKV registry keys remain `apps/{account}/{gateway}/bos.config.json`; TOML is local authoring only.
+- **Bootstrapping**: `bos init` now copies `bos.config.toml` alongside `bos.config.json` in init patterns, and reads/writes configs via format-aware helpers.
+- **Migration**: 12 raw `JSON.parse(readFileSync(...))` sites migrated to `readBosConfigSource` across build, plugin, upgrade, status, and init modules.

.changeset/toml-gap-fixes.md

@@ -0,0 +1,12 @@
+---
+"api": patch
+"everything-dev": patch
+---
+
+Fix remaining TOML config implementation gaps
+
+- **G2**: Add missing `title` and `description` fields to `BosConfigInputSchema` so they are not stripped during Zod validation (previously existed on the TypeScript interface but not the Zod schema)
+- **G3**: Handle per-plugin record variant of `[infra.database]` in `buildDatabaseConfigs`. When `database` is a `Record<string, InfraDatabase>` (e.g. `[infra.database.auth]`, `[infra.database.api]`), the function now iterates entries and matches them to `*_DATABASE_URL` secrets instead of treating all truthy database values as a single shared config. Extracted `resolveDatabasePort` and `buildDatabaseConfigFromSecret` helpers to share logic with the convention-based scanning path.
+- **G1**: Wire `generateAlchemyRun` into `publish.ts` — after a successful publish, if `bosConfig.deploy` is present, `generateAlchemyRun` is called to write `alchemy.run.ts` with the deploy configuration
+- **G4**: Make 22 user-facing error messages format-agnostic (no longer hardcode `"bos.config.json"`). Affected files: `config.ts`, `plugin.ts`, `cli.ts`, `sync.ts`, `shared-deps.ts`, `upgrade.ts`, `dag.ts`
+- **G5**: Fix `upgrade.ts` plugin config scanning glob/regex to match both `bos.config.json` and `bos.config.toml`

Dockerfile

@@ -30,7 +30,7 @@ COPY --from=builder --chown=appuser:appgroup /app/node_modules ./node_modules
 COPY --from=builder --chown=appuser:appgroup /app/package.json .
 COPY --from=builder --chown=appuser:appgroup /app/bun.lock .
 COPY --from=builder --chown=appuser:appgroup /app/bunfig.toml .
-COPY --from=builder --chown=appuser:appgroup /app/bos.config.json ./
+COPY --from=builder --chown=appuser:appgroup /app/bos.config.* ./
 COPY --from=builder --chown=appuser:appgroup /app/packages/everything-dev ./packages/everything-dev
 COPY --from=builder --chown=appuser:appgroup /app/packages/every-plugin ./packages/every-plugin
 

api/package.json

@@ -26,6 +26,7 @@
     "effect": "catalog:",
     "every-plugin": "catalog:",
     "pg": "catalog:",
+    "@neondatabase/serverless": "^1.0.0",
     "@electric-sql/pglite": "catalog:",
     "zod": "catalog:"
   },

api/src/db/index.ts

@@ -2,6 +2,7 @@ import { mkdirSync } from "node:fs";
 import { dirname } from "node:path";
 import type { PgDatabase, PgQueryResultHKT } from "drizzle-orm/pg-core";
 import { Data } from "every-plugin/effect";
+import type { PoolConfig } from "pg";
 import * as schema from "./schema";
 
 export type Database = PgDatabase<PgQueryResultHKT, typeof schema>;
@@ -11,6 +12,16 @@ export interface DatabaseDriver {
   close(): Promise<void>;
 }
 
+interface PoolLike {
+  on(event: "error", listener: (err: Error) => void): this;
+  on(
+    event: "connect",
+    listener: (client: { query: (sql: string) => Promise<unknown> }) => void,
+  ): this;
+  removeAllListeners(event?: string | symbol): this;
+  end(): Promise<void>;
+}
+
 export class DatabaseError extends Data.TaggedError("DatabaseError")<{
   stage: "driver" | "migration" | "load" | "close";
   migrationTag?: string;
@@ -37,7 +48,46 @@ export function unwrapDatabaseError(error: unknown): string {
   return parts.join(": ");
 }
 
-export async function createDatabaseDriver(url: string): Promise<DatabaseDriver> {
+function buildPoolConfig(url: string): PoolConfig {
+  const isLocal = url.includes("localhost") || url.includes("127.0.0.1");
+  return {
+    connectionString: url,
+    ssl: isLocal
+      ? false
+      : { rejectUnauthorized: process.env.DB_SSL_REJECT_UNAUTHORIZED === "true" },
+    max: Number(process.env.DB_POOL_MAX) || 10,
+    connectionTimeoutMillis: Number(process.env.DB_CONNECTION_TIMEOUT_MS) || 30_000,
+    idleTimeoutMillis: Number(process.env.DB_IDLE_TIMEOUT_MS) || 30_000,
+  };
+}
+
+function attachPoolSchemaHandlers(pool: PoolLike, schemaName: string | undefined): void {
+  pool.on("error", (err: Error) => {
+    console.error("[Database] Unexpected pool error:", err.message);
+  });
+  if (schemaName) {
+    pool.on("connect", async (client) => {
+      await client.query(`CREATE SCHEMA IF NOT EXISTS "${schemaName}"`);
+      await client.query(`SET search_path TO "${schemaName}", public`);
+    });
+  }
+}
+
+function createCloseHandler(pool: PoolLike): () => Promise<void> {
+  let closed = false;
+  return async () => {
+    if (closed) return;
+    closed = true;
+    pool.removeAllListeners("error");
+    pool.removeAllListeners("connect");
+    await pool.end();
+  };
+}
+
+export async function createDatabaseDriver(
+  url: string,
+  schemaName?: string,
+): Promise<DatabaseDriver> {
   if (url.startsWith("pglite:") || url === ":memory:") {
     const { drizzle } = await import("drizzle-orm/pglite");
     const { PGlite } = await import("@electric-sql/pglite");
@@ -47,6 +97,10 @@ export async function createDatabaseDriver(url: string): Promise<DatabaseDriver>
       mkdirSync(dirname(dataDir), { recursive: true });
     }
     const pglite = new PGlite(dataDir);
+    if (schemaName) {
+      await pglite.exec(`CREATE SCHEMA IF NOT EXISTS "${schemaName}"`);
+      await pglite.exec(`SET search_path TO "${schemaName}", public`);
+    }
     const db = drizzle(pglite, { schema });
     return {
       db,
@@ -56,33 +110,28 @@ export async function createDatabaseDriver(url: string): Promise<DatabaseDriver>
     };
   }
 
+  // Neon WebSocket Pool (pooled connection, works with PgBouncer)
+  if (url.includes("neon.tech")) {
+    try {
+      const { Pool } = await import("@neondatabase/serverless");
+      const { drizzle } = await import("drizzle-orm/neon-serverless");
+      const pool = new Pool(buildPoolConfig(url));
+      attachPoolSchemaHandlers(pool, schemaName);
+      return {
+        db: drizzle(pool, { schema }),
+        close: createCloseHandler(pool),
+      };
+    } catch {
+      // @neondatabase/serverless not installed, fall through to pg
+    }
+  }
+
   const { Pool } = await import("pg");
   const { drizzle } = await import("drizzle-orm/node-postgres");
-  const isLocal = url.includes("localhost") || url.includes("127.0.0.1");
-  const pool = new Pool({
-    connectionString: url,
-    ssl: isLocal
-      ? false
-      : { rejectUnauthorized: process.env.DB_SSL_REJECT_UNAUTHORIZED === "true" },
-    max: Number(process.env.DB_POOL_MAX) || 10,
-    connectionTimeoutMillis: Number(process.env.DB_CONNECTION_TIMEOUT_MS) || 30_000,
-    idleTimeoutMillis: Number(process.env.DB_IDLE_TIMEOUT_MS) || 30_000,
-  });
-  pool.on("error", (err: Error) => {
-    console.error("[Database] Unexpected pool error:", err.message);
-  });
-  let closed = false;
+  const pool = new Pool(buildPoolConfig(url));
+  attachPoolSchemaHandlers(pool, schemaName);
   return {
     db: drizzle(pool, { schema }),
-    close: async () => {
-      if (closed) return;
-      closed = true;
-      pool.removeAllListeners("error");
-      console.error(
-        "[Database] pool.end() called from:",
-        new Error("pool.end() stack trace").stack,
-      );
-      await pool.end();
-    },
+    close: createCloseHandler(pool),
   };
 }

api/src/db/layer.ts

@@ -6,13 +6,17 @@ import { detectDrift, loadMigrations, migrate } from "./migrate";
 
 export class DatabaseTag extends Context.Tag("Database")<Database, Database>() {}
 
-export const DatabaseLive = (url: string) =>
+export const DriverLive = (url: string) =>
   Layer.scoped(
     DatabaseTag,
     Effect.gen(function* () {
+      const pluginId = yield* PluginIdTag;
+      const slug = pluginMigrationSlug(pluginId);
+      const schemaName = `plugin_${slug}`;
+
       const driver = yield* Effect.acquireRelease(
         Effect.tryPromise({
-          try: () => createDatabaseDriver(url),
+          try: () => createDatabaseDriver(url, schemaName),
           catch: (cause) => new DatabaseError({ stage: "driver", cause }),
         }),
         (driver) =>
@@ -22,63 +26,80 @@ export const DatabaseLive = (url: string) =>
           }).pipe(Effect.ignore),
       );
 
-      const pluginId = yield* PluginIdTag;
-      const storage = getMigrationStorage(pluginMigrationSlug(pluginId));
-      const { migrations, source } = yield* loadMigrations();
+      return driver.db;
+    }),
+  );
 
-      if (migrations.length === 0) {
-        yield* Effect.logWarning(
-          `[Database] No migrations found (source: ${source}) — schema may be missing`,
+export const MigrationLive = Layer.effect(
+  DatabaseTag,
+  Effect.gen(function* () {
+    const db = yield* DatabaseTag;
+    const pluginId = yield* PluginIdTag;
+    const slug = pluginMigrationSlug(pluginId);
+    const schemaName = `plugin_${slug}`;
+    const storage = getMigrationStorage(slug);
+
+    const { migrations, source } = yield* loadMigrations();
+
+    if (migrations.length === 0) {
+      yield* Effect.logWarning(
+        `[Database] No migrations found (source: ${source}) — schema may be missing`,
+      );
+    } else {
+      const applied = yield* migrate(db, migrations, storage, schemaName);
+
+      if (applied === 0) {
+        yield* Effect.logInfo(
+          `[Database] Schema up to date (0 migrations needed, source: ${source})`,
         );
       } else {
-        const applied = yield* migrate(driver.db, migrations, storage);
-
-        if (applied === 0) {
-          yield* Effect.logInfo(
-            `[Database] Schema up to date (0 migrations needed, source: ${source})`,
-          );
-        } else {
-          yield* Effect.logInfo(
-            `[Database] Applied ${applied}/${migrations.length} migration(s) (source: ${source}, journal: ${storage.schema}.${storage.table})`,
-          );
-        }
+        yield* Effect.logInfo(
+          `[Database] Applied ${applied}/${migrations.length} migration(s) (source: ${source}, journal: ${storage.schema}.${storage.table}, schema: ${schemaName})`,
+        );
+      }
 
-        const drift = yield* detectDrift(driver.db, migrations, storage);
-        if (drift.status === "healthy" || drift.status === "untracked-existing-schema") {
-          yield* Effect.logInfo(`[Database] Ready`);
-        } else if (drift.status === "drift-safe-repair") {
-          yield* Effect.logWarning(
-            `[Database] ⚠️ Migration drift detected: ${drift.missingTables.length} expected table(s) missing: ${drift.missingTables.join(", ")}`,
-          );
-          yield* Effect.logWarning(
-            `[Database] Run \`bos db doctor ${storage.slug}\` to diagnose and \`bos db repair ${storage.slug}\` to fix.`,
-          );
-          throw new DatabaseError({
+      const drift = yield* detectDrift(db, migrations, storage, schemaName);
+      if (drift.status === "healthy" || drift.status === "untracked-existing-schema") {
+        yield* Effect.logInfo(`[Database] Ready`);
+      } else if (drift.status === "drift-safe-repair") {
+        yield* Effect.logWarning(
+          `[Database] ⚠️ Migration drift detected: ${drift.missingTables.length} expected table(s) missing: ${drift.missingTables.join(", ")}`,
+        );
+        yield* Effect.logWarning(
+          `[Database] Run \`bos db doctor ${storage.slug}\` to diagnose and \`bos db repair ${storage.slug}\` to fix.`,
+        );
+        yield* Effect.fail(
+          new DatabaseError({
             stage: "migration",
             migrationTag: "drift-safe-repair",
             cause: new Error(
               `Migration journal has ${drift.appliedHashes} applied hashes but all ${drift.expectedTables.length} expected table(s) are missing. ` +
                 `Run \`bos db repair ${storage.slug}\` to reset the migration history and reapply migrations. ` +
                 `Missing tables: ${drift.missingTables.join(", ")}`,
             ),
-          });
-        }
-        if (drift.status === "drift-manual") {
-          yield* Effect.logWarning(
-            `[Database] ⚠️ Partial migration drift detected: ${drift.missingTables.length}/${drift.expectedTables.length} expected table(s) missing.`,
-          );
-          throw new DatabaseError({
+          }),
+        );
+      }
+      if (drift.status === "drift-manual") {
+        yield* Effect.logWarning(
+          `[Database] ⚠️ Partial migration drift detected: ${drift.missingTables.length}/${drift.expectedTables.length} expected table(s) missing.`,
+        );
+        yield* Effect.fail(
+          new DatabaseError({
             stage: "migration",
             migrationTag: "drift-manual",
             cause: new Error(
               `Partial schema drift — ${drift.missingTables.length}/${drift.expectedTables.length} expected table(s) are missing. ` +
                 `Run \`bos db doctor ${storage.slug}\` for details. Manual intervention required. ` +
                 `Missing tables: ${drift.missingTables.join(", ")}`,
             ),
-          });
-        }
+          }),
+        );
       }
+    }
 
-      return driver.db;
-    }),
-  );
+    return db;
+  }),
+);
+
+export const DatabaseLive = (url: string) => Layer.provideMerge(DriverLive(url), MigrationLive);