CodeDuel is a MERN online judge with a React frontend, Express API, MongoDB, Redis-backed submission queue, a separate execution worker, and ephemeral Docker sandboxes for C, C++, Java, JavaScript, and Python.
The API never executes submitted code. It validates and stores a submission, then enqueues an identifier. The worker loads server-controlled testcases and creates a new locked-down container for each job. Code execution is disabled by default and fails closed when Docker or a configured image is unavailable.
flowchart LR
Browser["Browser"] --> Nginx["Nginx / React"]
Nginx --> API["Express API"]
API --> Mongo[(MongoDB)]
API --> Redis[(Redis / BullMQ)]
Worker["Execution worker"] --> Redis
Worker --> Mongo
Worker --> Docker["Docker Engine"]
Docker --> Sandbox["Ephemeral untrusted-code container"]
- Node.js 22 LTS and npm for native development.
- Docker Engine or Docker Desktop with Compose v2 for the recommended setup.
- At least 4 GB of free memory when all five language images are installed.
- PowerShell 7 commands are shown below. In Bash, replace
Copy-Itemwithcpand$env:NAME='value'withexport NAME=value.
Never put a real secret in Git. .env files, dependencies, builds, logs, and
local sandbox data are ignored.
The root commands are the supported local workflow on Windows, macOS, and
Linux. They preserve existing .env files and never print generated secrets.
npm run setup
npm run seed
npm run devsetup performs clean installs for server and client, starts the local
MongoDB and Redis containers, waits for their health checks, and verifies that
all five server-controlled sandbox images are installed. If .env or
server/.env is absent, setup creates ignored local files from the placeholder
template with random development secrets. It never overwrites an existing
environment.
dev keeps MongoDB and Redis in Compose and starts three separate native
processes with reload support: Vite, the Express API, and the BullMQ worker.
Open http://localhost:3000; the API is available at
http://localhost:5000. Stop the foreground command
with Ctrl+C.
The common verification commands are:
npm run lint
npm test
npm run build
npm run test:e2e
npm run verify:sandboxRun test:e2e while npm run dev is healthy. It serializes its five browser
scenarios to avoid colliding with the intentional registration rate limit.
verify:sandbox performs real, containerized execution for C, C++, Java,
JavaScript, and Python, then checks wrong-answer, compilation, runtime,
timeout, output-limit, fail-closed, and cleanup behavior. No submitted program
is executed on the host.
From the repository root:
Copy-Item .env.example .envEdit .env and replace JWT_SECRET and ADMIN_PASSWORD. Keep
EXECUTION_ENABLED=false for the initial boot.
docker compose config --quiet
docker compose build
docker compose up -d mongo redis
docker compose --profile tools run --rm maintenance
docker compose up -d
docker compose psOpen http://localhost:3000. The development API is
also bound to 127.0.0.1:5000; MongoDB and Redis are bound to localhost only.
The worker has an execution Compose profile and is intentionally absent while
EXECUTION_ENABLED=false, so the safe default stack does not crash-loop or
claim that execution is available.
The seed is idempotent and creates or updates the configured development
administrator, sample problems, sample cases, hidden cases, and indexes.
Verify the stack:
Invoke-RestMethod http://localhost:5000/health/live
Invoke-RestMethod http://localhost:5000/health/ready
Invoke-WebRequest http://localhost:3000/healthz -UseBasicParsing
docker compose logs --tail 100 api/health/live proves that the API process is alive. /health/ready checks its
required MongoDB and Redis dependencies and, when execution is enabled, worker
and sandbox readiness. Health responses do not include connection strings or
credentials.
The worker is the only service with the Docker client library and the Docker socket. The API and frontend have neither. Before opting in:
docker info
docker pull gcc:14.2-bookworm
docker pull eclipse-temurin:21-jdk-noble
docker pull node:22-bookworm-slim
docker pull python:3.13-slim-bookworm
docker compose --profile execution run --rm --entrypoint node worker -e "new (require('dockerode'))().ping().then(() => console.log('Docker reachable')).catch(() => process.exit(1))"On Linux, set the socket group before recreating the worker:
printf 'DOCKER_GID=%s\n' "$(stat -c '%g' /var/run/docker.sock)" >> .envSet EXECUTION_ENABLED=true in .env, then recreate the API and worker:
docker compose --profile execution up -d --build --force-recreate api worker
Invoke-RestMethod http://localhost:5000/health/readyIf readiness fails, execution remains unavailable; CodeDuel never falls back to
host execution. Check docker compose --profile execution logs worker and disable execution again
until the cause is understood.
The recommended hybrid workflow runs MongoDB and Redis in Compose while the API, worker, and frontend run under Node for fast reloads.
docker compose up -d mongo redis
Copy-Item .env.example server/.envKeep the server/.env MongoDB and Redis URLs at 127.0.0.1. In terminal 1:
Set-Location server
npm ci
npm run seed
npm run devIn terminal 2, only after setting EXECUTION_ENABLED=true in server/.env and
verifying Docker as described above:
Set-Location server
npm run dev:workerIn terminal 3:
Set-Location client
npm ci
npm startFor native development, leave execution disabled unless the local worker can reach Docker and all sandbox images have been pulled. Never enable a no-Docker or host-execution fallback; none is supported.
| Variable | Purpose | Safe development default |
|---|---|---|
NODE_ENV |
development, test, or production |
development |
PORT |
API port | 5000 |
MONGODB_URI |
MongoDB connection URI | local codeduel database |
REDIS_URL |
Redis/BullMQ connection URI | local Redis |
JWT_SECRET |
HMAC signing key, at least 32 characters | replace before start |
JWT_ISSUER, JWT_AUDIENCE |
Required JWT validation claims | codeduel, codeduel-users |
JWT_EXPIRES_IN |
Access-token lifetime | 15m |
CLIENT_ORIGINS |
Comma-separated exact browser origins | local frontend origins |
COOKIE_SECURE |
Restrict auth cookies to HTTPS | false; production forces true |
COOKIE_SAME_SITE |
lax, strict, or none |
lax |
TRUST_PROXY_HOPS |
Trusted production reverse-proxy count, 0-5 | 1 |
EXECUTION_ENABLED |
Explicit sandbox opt-in | false |
SANDBOX_*_IMAGE |
Server-controlled language images | pinned tags in .env.example |
EXECUTION_TIMEOUT_MS |
Per-test wall-clock runtime limit | 5000 |
COMPILATION_TIMEOUT_MS |
Compile wall-clock limit | 10000 |
EXECUTION_MEMORY_MB |
Sandbox memory and swap ceiling | 256 |
EXECUTION_CPUS |
Sandbox CPU quota | 0.5 |
EXECUTION_PIDS_LIMIT |
Sandbox process limit | 64 |
SOURCE_SIZE_LIMIT_BYTES |
Maximum submitted source size | 131072 |
INPUT_SIZE_LIMIT_BYTES |
Maximum custom input size | 262144 |
OUTPUT_SIZE_LIMIT_BYTES |
Combined bounded output | 1048576 |
WORKER_CONCURRENCY |
Concurrent worker jobs | 2 |
LOG_LEVEL |
Structured-log threshold | info |
ADMIN_EMAIL, ADMIN_USERNAME, ADMIN_PASSWORD |
Seeded administrator | replace password |
All required values are validated at startup. Invalid configuration causes an immediate error listing variable names only; secret values are never printed.
The browser API client always sends credentials: 'include'. Before a
state-changing request it fetches GET /api/user/csrf-token, then sends the
returned token in X-CSRF-Token on POST, PUT, PATCH, and DELETE
requests. A CSRF rejection clears the cached token and is retried once with a
new token. Non-browser clients must implement the same cookie-jar and header
flow. CORS accepts credentials only from the exact CLIENT_ORIGINS allowlist.
Access JWTs remain in HTTP-only cookies and are not stored in browser storage.
The API exposes Prometheus text metrics at /metrics, including Node process
metrics, HTTP duration/errors, authentication failures, verdict totals, and
sandbox failures. The included Nginx intentionally does not proxy this endpoint;
scrape it from the private API network or a tightly controlled localhost tunnel.
Application logs are structured JSON on stdout and may also be retained in
MongoDB according to the application's retention policy. Use container log
rotation from the production Compose file and avoid logging source, cookies,
tokens, passwords, or hidden testcase data.
Run the idempotent seed at any time:
docker compose --profile tools run --rm maintenanceFor native development, run npm run seed from server. The password in
ADMIN_PASSWORD must satisfy the application password policy. Roles come from
the database; usernames do not grant administrator access.
The reset command is deliberately development-only and requires an explicit guard:
docker compose --profile tools run --rm -e ALLOW_DB_RESET=true maintenance npm run db:reset
docker compose --profile tools run --rm maintenanceTo remove the entire local Compose database and queue volumes instead:
docker compose down --volumes --remove-orphansBoth reset operations permanently delete local development data. Never run them against a shared or production database.
Server checks:
Set-Location server
npm ci
npm run lint
npm test
npm run test:unit
npm run test:integration
npm run test:security
npm run test:workerTests use mocks or controlled adapters for dangerous execution paths; the automated security suite does not run arbitrary code on the host.
Client checks:
Set-Location client
npm ci
npm run lint
npm test
npm run buildFrom the repository root, the equivalent combined checks are npm run lint,
npm test, and npm run build. Browser and real-sandbox checks remain
explicit because they require the live local stack and Docker respectively:
npm run test:e2e
npm run verify:sandboxContainer and Compose checks:
Set-Location ..
docker compose config --quiet
docker compose --profile execution build api worker frontendRun dependency scans without forced rewrites:
Set-Location server
npm audit
Set-Location ../client
npm auditDo not use npm audit fix --force; review and test dependency migrations in
small groups.
Submission clients provide only a problem ID, language ID, and source. The worker loads enabled testcases from MongoDB in stable order. Hidden inputs and expected outputs are never included in public problem or submission responses.
Output comparison normalizes CRLF/CR line endings to LF, removes trailing spaces and tabs from each line, and ignores newline characters at the end of the complete output. Internal whitespace, leading whitespace, text case, and line order remain significant.
Each job uses an ephemeral container with no network, a read-only root
filesystem, temporary tmpfs workspaces, a non-root user, all Linux
capabilities dropped, no-new-privileges, CPU/memory/PID limits, bounded output,
and compilation/runtime deadlines. The worker copies source through the Docker
API; it does not expose MongoDB, Redis, server files, testcases, or the Docker
socket inside the untrusted-code container. Cleanup forcibly removes the
container after completion and failures.
docker-compose.prod.yml is a standalone production example, not an override
for docker-compose.yml. It publishes only the Nginx frontend. MongoDB, Redis,
the API, and the worker remain on private Compose networks.
Create a dedicated file that is not committed:
Copy-Item .env.example .env.productionGenerate independent random values for JWT, MongoDB root, MongoDB application, Redis, and administrator passwords. URL-encode the application passwords in the connection URIs, then configure at least:
NODE_ENV=production
CLIENT_ORIGINS=https://codeduel.example.com
COOKIE_SECURE=true
COOKIE_SAME_SITE=lax
TRUST_PROXY_HOPS=2
MONGO_ROOT_USERNAME=codeduel_root
MONGO_ROOT_PASSWORD=<random-root-password>
MONGO_APP_USERNAME=codeduel_app
MONGO_APP_PASSWORD=<different-random-app-password>
MONGODB_URI=mongodb://codeduel_app:<url-encoded-app-password>@mongo:27017/codeduel?authSource=codeduel
REDIS_PASSWORD=<random-redis-password>
REDIS_URL=redis://:<url-encoded-redis-password>@redis:6379
HTTP_PORT=8080
EXECUTION_ENABLED=falseValidate, build, initialize, and start:
docker compose --env-file .env.production -f docker-compose.prod.yml config --quiet
docker compose --env-file .env.production -f docker-compose.prod.yml build
docker compose --env-file .env.production -f docker-compose.prod.yml up -d mongo redis
docker compose --env-file .env.production -f docker-compose.prod.yml --profile tools run --rm maintenance
docker compose --env-file .env.production -f docker-compose.prod.yml up -d
docker compose --env-file .env.production -f docker-compose.prod.yml psWhen a production execution host has passed the sandbox checks, set
EXECUTION_ENABLED=true and add --profile execution to the final up, ps,
and worker-log commands. Merely setting the variable without starting that
profile leaves API readiness false, which is the intended fail-closed behavior.
The MongoDB application account is created only when its named volume is first
initialized. Rotating database credentials on an existing volume requires an
intentional MongoDB user update; changing only .env.production is not enough.
Terminate TLS in a load balancer or host reverse proxy in front of port 8080.
Forward the original Host, X-Forwarded-For, and X-Forwarded-Proto headers.
Do not expose the example directly to the internet over plain HTTP. Set
CLIENT_ORIGINS to exact HTTPS origins and keep secure cookies enabled. The
application's proxy-trust setting must match the number of trusted proxies:
use TRUST_PROXY_HOPS=1 for direct Nginx-to-API traffic, or 2 for the
documented external TLS proxy-to-Nginx-to-API path. Never trust arbitrary
forwarding headers from the public internet.
Back up the mongo_data volume regularly and test restores. Redis append-only
data improves queue durability, but it is not a substitute for MongoDB backups.
Deploy updated images with docker compose ... up -d --build; do not delete
production volumes during routine releases.
Start with status and bounded logs:
docker compose ps
docker compose --profile execution logs --tail 200 mongo redis api worker frontend
docker compose configOn Windows/macOS, start Docker Desktop and wait for docker info to succeed. On
Linux, confirm the daemon is running and set DOCKER_GID to the socket's numeric
group. Recreate the worker after changing it:
docker compose --profile execution up -d --force-recreate worker
docker compose --profile execution exec worker docker versionIf the check fails, set EXECUTION_ENABLED=false. Do not run the worker as root
or expose the Docker TCP API as a workaround.
docker compose exec mongo mongosh --quiet --eval "db.adminCommand('ping')"
docker compose logs --tail 100 mongoFor local disposable data only, stop Compose and use the documented volume reset. Never remove a production database volume to repair a startup problem.
docker compose exec redis redis-cli ping
docker compose --profile execution logs --tail 100 redis workerProduction Redis requires authentication; use REDISCLI_AUTH inside a trusted
shell rather than putting the password directly in command history.
The startup error names every missing or invalid variable without revealing
values. Compare .env with .env.example, then validate the resolved Compose
model with docker compose config. Remember that native server commands read
server/.env, while Compose interpolation reads the root .env or the file
passed with --env-file.
Nginx intentionally waits for API readiness. Inspect /health/ready and API
logs first. A liveness success with readiness failure means a required
dependency is unavailable, not that the frontend image is broken.
Change PORT or FRONTEND_PORT in the development .env. For production,
change HTTP_PORT. Database and Redis development ports intentionally bind only
to 127.0.0.1.
- Mounting
/var/run/docker.sockgives the worker effective root-equivalent control of the Docker host. The:romount flag does not make Docker API operations read-only. This risk is constrained to the worker, but it is not eliminated by running the worker process as a non-root user. - For public multi-tenant use, run the worker and Docker daemon on a dedicated, disposable execution host or replace the socket with a purpose-built sandbox service. Do not colocate valuable workloads or secrets on that host.
- Linux containers, resource limits, and disabled networking materially reduce risk but are not a formally hardened isolation boundary. Keep the kernel, Docker Engine, and language images patched and perform live adversarial sandbox testing before a public beta.
- The production example expects external HTTPS termination and a deliberate backup, monitoring, credential-rotation, and image-update process.
- Compose environment variables can be inspected by users who control Docker. A production orchestrator or secret manager should provide secrets with tighter operational access controls.
- Execution remains disabled unless explicitly enabled, and readiness fails closed when the sandbox is unavailable. This is intentional.
CodeDuel is suitable for local development and controlled private testing after the automated suite and live sandbox smoke tests pass. Treat the Compose production file as an auditable deployment starting point, not evidence by itself that an internet-facing judge is production-ready.
See LICENSE.