Export collections to FTP for depositing and archiving by gabestein · Pull Request #3654 · knowledgefutures/pubpub

gabestein · 2026-07-01T01:29:19Z

FTP Targets & Collection Export

Overview

This branch adds two related features: a superadmin UI for managing SFTP/FTPS deposit targets (with credential testing and AES-256 encrypted storage), and a collection export feature that lets community admins bundle all published pubs in a collection into a ZIP of PDFs and JATS XML files — with optional upload to one or more configured FTP targets.

Changed files

File	Change
tools/migrations/2026_06_30_createFtpTargets.js	New FtpTargets table
tools/migrations/2026_07_01_addNameToFtpTargets.js	Adds name column to FtpTargets
server/ftpTarget/model.ts	New Sequelize model for FtpTarget
server/utils/sftp.ts	New SFTP utility: testSftpConnection, uploadFileViaSftp
server/models.ts	Registers FtpTarget; re-exports it
server/community/model.ts	Adds HasMany(FtpTarget) association
server/routes/superAdminDashboard.tsx	FTP target CRUD routes (create, update, delete, copy) + community search endpoint
utils/superAdmin.ts	Adds ftpTargets to superadmin tab list
client/containers/SuperAdminDashboard/FtpTargets/	New tab UI: create form, editable table, copy/delete dialogs
server/collection/api.ts	New export endpoint on the collection router
server/collection/permissions.ts	Adds collectionExport permission
server/collection/queries.ts	Adds getCollectionExports query
utils/api/contracts/collection.ts	Adds export route to the collection contract
server/routes/dashboardSettings.tsx	Fetches collection exports and FTP targets for collection settings page
workers/tasks/collectionExport.ts	New background task: generates ZIP, uploads to S3, optionally deposits to multiple FTP targets
workers/worker.ts	Registers collectionExport task type
server/utils/workers.ts	Adds collectionExport to allowed worker task types
client/containers/DashboardSettings/CollectionSettings/CollectionSettings.tsx	Adds Export tab
client/containers/DashboardSettings/CollectionSettings/ExportCollectionButton.tsx	New export UI: multi-FTP checkbox selection, polling, export history table with warnings
client/containers/DashboardSettings/CollectionSettings/exportCollectionButton.scss	Styles for export tab

Testing plan

FTP Targets (superadmin)

Navigate to /superadmin?tab=ftpTargets. Confirm the tab renders with an empty state.
Create a target with valid SFTP credentials — confirm connection is tested and the row appears in the table with its name, host, community, and credential badge.
Create a target with bad credentials — confirm a legible error is shown (not a JSON parse error).
Edit an existing target: change the name, host, and file path. Confirm changes persist.
Edit a target and clear credentials by setting username to empty — confirm hasCredentials shows as false.
Copy a target to a different community, with and without Copy credentials. Confirm the new row appears for the correct community.
Delete a target. Confirm it disappears from the list.
Use the filter bar to search by name, host, and community.

Collection Export (collection settings)

Navigate to a collection's dashboard settings → Export tab. Confirm the tab appears only for collection admins.
If no FTP targets are configured for the community, confirm no FTP selector appears.
If FTP targets exist, confirm checkboxes show target names (falling back to host (path) for unnamed targets).
Trigger an export with no FTP target selected. Confirm:
- A "processing" badge appears in export history immediately.
- Polling resolves to "Ready" with a working download link.
- The ZIP contains a top-level folder {subdomain}-{collectionSlug}-{timestamp}/ with per-pub subdirectories each containing a .pdf and .xml.
Trigger an export with one or more FTP targets selected. Confirm the ZIP appears on the remote server and the FTP column in history shows the per-target result.
Trigger an export for a collection that has a pub with no PDF or JATS export. Confirm:
- The export completes (doesn't fail entirely).
- A warning callout lists the pub and which format was missing.
- The history table "Files" column shows the warning tag.
Trigger an export with a target whose credentials are broken. Confirm the FTP column shows the failure and a warning callout names the host and error.
Trigger a second export while one is already in progress. Confirm the API returns the existing task ID without creating a duplicate.
Wait 7+ days (or manually expire the S3 URL) — confirm the history row shows "Expired" and the download link is hidden.

Todo

Hoist FTP targets to KF-Auth/org level and move from 1:1 with communities to 1:many.
Allow for customization of file types, file and bundle naming, upload path, etc. at community and collection scopes
Implement FTPS and other upload mechanisms as needed

Copilot

Pull request overview

This PR introduces configurable FTP deposit targets (managed by superadmins) and a new collection export workflow that packages released pubs into a ZIP (PDF + JATS) and can optionally deposit that ZIP to one or more configured targets.

Changes:

Adds an FtpTargets data model + superadmin CRUD UI/API with SFTP credential testing and encrypted credential storage.
Adds a collection export API/permission, dashboard “Export” tab UI, and a worker task that generates ZIPs, uploads them to S3, and optionally deposits via SFTP.
Updates local dev ergonomics (auto-decrypt env, dev seeding, docker-compose adjustments, README refresh).

Reviewed changes

Copilot reviewed 33 out of 34 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
workers/worker.ts	Registers `collectionExport` in the worker task map.
workers/tasks/collectionExport.ts	Implements ZIP generation, S3 upload, and optional SFTP deposit for collection exports.
utils/superAdmin.ts	Adds `ftpTargets` to the superadmin tab kind list.
utils/api/contracts/collection.ts	Adds `POST /api/collections/export` contract definition.
tools/migrations/2026_06_30_createFtpTargets.js	Creates `FtpTargets` table (host/path/credentials fields, etc.).
tools/migrations/2026_07_01_addNameToFtpTargets.js	Adds `name` column to `FtpTargets`.
server/utils/workers.ts	Allows `collectionExport` as a worker task type.
server/utils/sftp.ts	Adds SFTP helper functions (test connection + upload).
server/sequelize.ts	Seeds dev data in non-production during sequelize sync.
server/seed.ts	Adds dev seed routine for a demo community + related records.
server/routes/superAdminDashboard.tsx	Adds FTP target CRUD + extends community search exclusion logic for FTP targets.
server/routes/dashboardSettings.tsx	Fetches `collectionExports` + `ftpTargets` for the collection settings export tab.
server/models.ts	Registers and exports the `FtpTarget` Sequelize model.
server/kf/provisionLocalUser.ts	Syncs `User.isSuperAdmin` from an OIDC role claim during provisioning.
server/kf/oidc.server.ts	Adds `OIDC_ROLE_CLAIM` configuration export.
server/ftpTarget/model.ts	Defines the `FtpTarget` Sequelize model.
server/community/model.ts	Adds `Community.hasMany(FtpTarget)` association.
server/collection/queries.ts	Adds `getCollectionExports` query (WorkerTask history).
server/collection/permissions.ts	Adds `collectionExport` permission flag.
server/collection/api.ts	Adds the `export` endpoint for triggering collection exports.
README.md	Updates local dev setup instructions and troubleshooting notes.
pnpm-lock.yaml	Locks new dependencies (notably `ssh2-sftp-client` and types).
package.json	Adds `ssh2-sftp-client` and its types.
infra/docker-compose.dev.yml	Updates dev compose configuration (env + volume changes).
infra/dev.sh	Auto-decrypts `infra/.env.local.enc` into `infra/.env.local` if needed.
infra/.env.local.enc	Updates encrypted local env contents.
infra/.env.dev.enc	Updates encrypted dev env contents.
client/containers/SuperAdminDashboard/tabs.tsx	Registers the new “FTP Targets” superadmin tab.
client/containers/SuperAdminDashboard/FtpTargets/index.ts	Exports the FTP Targets tab component.
client/containers/SuperAdminDashboard/FtpTargets/FtpTargets.tsx	Implements the superadmin UI for managing FTP targets.
client/containers/SuperAdminDashboard/FtpTargets/ftpTargets.scss	Styles for the FTP targets tab.
client/containers/DashboardSettings/CollectionSettings/ExportCollectionButton.tsx	Adds collection export UI, polling, history, and FTP target selection.
client/containers/DashboardSettings/CollectionSettings/exportCollectionButton.scss	Styles for the collection export UI.
client/containers/DashboardSettings/CollectionSettings/CollectionSettings.tsx	Adds the “Export” tab to collection settings and wires initial data into the export UI.

Files not reviewed (1)

pnpm-lock.yaml: Generated file

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+type PastExport = {
+	id: string;
+	createdAt: string;
+	isProcessing: boolean;
+	output: { downloadUrl: string; ftpUploaded: boolean } | null;
+	error: string | null;
+};


+		const workerTask = await addWorkerTask({
+			type: 'collectionExport',
+			input: {
+				collectionId,
+				communityId: collection.communityId,
+				ftpTargetIds: ftpTargetIds ?? [],
+			},
+		});


+		if (!ftpType || !['sftp', 'ftps'].includes(ftpType)) {
+			throw new BadRequestError(new Error('ftpType must be "sftp" or "ftps"'));
+		}


+		if (ftpType !== undefined) {
+			if (!['sftp', 'ftps'].includes(ftpType)) {
+				throw new BadRequestError(new Error('ftpType must be "sftp" or "ftps"'));
+			}
+			updates.ftpType = ftpType;
+		}


+					updates.password = encryptedText;
+					updates.passwordInitVec = initVec;
+				}
+				updates.username = username;


+		} else if (password) {
+			const { encryptedText, initVec } = aes256Encrypt(password, env.AES_ENCRYPTION_KEY!);
+			updates.password = encryptedText;
+			updates.passwordInitVec = initVec;
+		}


+		const createData: Record<string, any> = {
+			communityId: destCommunity.id,
+			ftpType: source.ftpType,
+			port: source.port,
+			host: source.host,
+			filePath: source.filePath,
+		};


-	>,
+	userInfo: UserInfoFields,
 ): Promise<InstanceType<typeof User>> {
+	const isSuperAdmin = userInfo[OIDC_ROLE_CLAIM] === 'admin';


+		await client.connect({
+			host: params.host,
+			port: params.port ?? 22,
+			username: params.username,
+			password: params.password,
+			readyTimeout: 10000,
+		});


+	const archiveStream = archiver('zip', { zlib: { level: 6 } });
+	const chunks: Buffer[] = [];
+
+	archiveStream.on('data', (chunk: Buffer) => chunks.push(chunk));
+


tefkah and others added 14 commits June 30, 2026 16:38

fix: seed script

bb77643

fix: remove pgdata3

6e5e2ad

new: Adds ftp targets to superadmin

6d93a60

feat: Test sftp connection on creation or save

552d64c

feat: Zip and deposit collection bundles

263747c

fix lint errors

cca4d6a

fix: init sftp client before mock hoist

3ca6a17

fix lint errors and remove unneeded superadmin test

f8b10e3

use s3 to download and add missing file warnings

b6a7438

fix: warn if PDF can't be downloaded

a0ca74e

fix: always have containing folder

ab6a61c

fix and new: FTP connect errors and multi-upload

eb91dc7

add name to ftp targets

b70ee6e

lint fixes

e95436f

gabestein marked this pull request as ready for review July 1, 2026 17:33

gabestein requested a review from tefkah July 1, 2026 17:33

gabestein changed the title ~~Gs/ftp export~~ Export collections to FTP for depositing and archiving Jul 1, 2026

tefkah requested a review from Copilot July 2, 2026 12:55

Copilot started reviewing on behalf of tefkah July 2, 2026 12:55 View session

Copilot AI reviewed Jul 2, 2026

View reviewed changes

tefkah and others added 9 commits July 2, 2026 15:15

Merge branch 'tfk/seed-script' into gs/ftp-export

985c3ae

fix: check if targetids are valid

674a3e9

fix: stream export

278c0ff

fix: let pubstash run locally, add sftp container

82b5945

fix: stream in upload correctly

c77ea03

fix: actually show ftptargets

ba673d7

fix: check sftp creds on edits

4bdad88

fix: copy ftp target name

eff8d37

allow exports of tag collections

b031ada

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Export collections to FTP for depositing and archiving#3654

Export collections to FTP for depositing and archiving#3654
gabestein wants to merge 23 commits into
mainfrom
gs/ftp-export

gabestein commented Jul 1, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

gabestein commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

FTP Targets & Collection Export

Overview

Changed files

Testing plan

Todo

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gabestein commented Jul 1, 2026 •

edited

Loading