Skip to content

Experimental support for dynamic container engines and task runtimes per process#1

Draft
abhi18av wants to merge 1 commit into
base-upstream-masterfrom
experiments/v25.10.4-base
Draft

Experimental support for dynamic container engines and task runtimes per process#1
abhi18av wants to merge 1 commit into
base-upstream-masterfrom
experiments/v25.10.4-base

Conversation

@abhi18av

Copy link
Copy Markdown
Collaborator

This internal PR highlights the tweaks we need to make sure that nf-nomad is able to utilize multiple task drivers within the same pipeline.

This is very experimental therefore, I'd appreciate if @jagedn can take a look and take this forward whenever he is available next month.

@abhi18av abhi18av force-pushed the experiments/v25.10.4-base branch from 4516542 to e1d7e7f Compare June 24, 2026 13:44
@abhi18av abhi18av changed the base branch from master to base-upstream-master June 24, 2026 13:44
@abhi18av abhi18av changed the title Experiments/v25.10.4 base Experimental support for multiple-executors per execution Jun 24, 2026
@abhi18av

abhi18av commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator Author

Meeting notes (07-July-2026)

  1. We should have a more generic approach to see which other executors benefit from the hybrid driver approach

  2. Kubernetes also benefits from this since it allows containers + virtual-machines (KubeVirt) + WASM

References

========

Webassembly and Bioinformatics

=========

We can also create a wasm driver for Nomad

@jagedn

jagedn commented Jul 9, 2026

Copy link
Copy Markdown

I was playing a little with wasm and I think it will require a lot of changes

as nextflow use bash and files we need to create (or improve current) k8sexecutor with lot of changes so I'll not continue with the wasm idea at least by the moment

@jagedn jagedn changed the title Experimental support for multiple-executors per execution Experimental support for dynamic container engines and task runtimes per process Jul 10, 2026
@jagedn

jagedn commented Jul 10, 2026

Copy link
Copy Markdown

I've changed the title of the PR as the intent is not to have a multi-executor but allow to the current executor be aware of every task to be executed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants