feat: implement crawlee v4 RequestQueueClient over apify-client by B4nan · Pull Request #643 · apify/apify-sdk-js

B4nan · 2026-06-16T12:34:14Z

Crawlee v4 redesigned the request queue storage layer into a stateful, pull-based RequestQueueBackend interface, modeled on the Python SDK. The SDK's old name-remapping proxy can't satisfy it, so this PR implements it for the Apify platform, with two access modes. The Python SDK's request queue clients were the starting point, but the design follows the JS API: the crawlee JS frontend already keeps large dedup caches, so the backends only manage what the frontend can't.

Request queue access modes

single (default, ApifyRequestQueueSingleBackend) assumes the run is the only consumer of the queue. There is no request locking; the queue head is estimated locally, and requests added by this client are served from a local cache, so a typical request costs no read API calls. On the first add, up to 10k existing requests are prefetched, letting resurrected runs deduplicate re-added requests locally instead of paying a platform write per request. Multiple producers may still add requests concurrently.

shared (ApifyRequestQueueSharedBackend) is safe for any number of concurrent consumers. Fetched requests are locked server-side (listAndLockHead), and the lock duration follows the crawler's expected request processing time (setExpectedRequestProcessingTimeSecs, i.e. handler timeout plus padding). Reclaims release the lock so other consumers can pick the request up immediately, and isFinished consults queueHasLockedRequests so no consumer exits while another still holds work.

The mode is selected per storage backend, as in the Python SDK: Actor.init({ requestQueueAccess: 'shared' }) or new ApifyStorageBackend(client, { requestQueueAccess: 'shared' }). Opening the same queue in both modes at once is not supported; whichever backend opens it first wins (documented on getStorageBackendCacheKey).

Request queue clients now send a stable per-run clientKey (the run id, same as the Python SDK), so hadMultipleClients stays meaningful and a migrated or resurrected run re-acquires the locks of its previous incarnation.

Both backends honor the beta.105 contract: markRequestAsHandled/reclaimRequest of a request that does not exist return undefined without upserting it, and purge() throws, since the platform has no truncate endpoint (Python behaves the same).

Review feedback addressed

ApifyStorageClient renamed to ApifyStorageBackend (class and file), OpenStorageContext.client renamed to backend, stray comment removed.
The single/shared toggle from the review is implemented here rather than tracked in an issue.

Crawlee beta.105 adaptation (was beta.71)

create*Backend now receives crawlee's StorageIdentifier ({id} | {name} | {alias}). The backend resolves aliases itself: __default__ maps to the run's default storage, other aliases resolve via ACTOR_STORAGES_JSON, undeclared aliases on the platform get a clear error, and anywhere else an unnamed storage is created per alias and process. A KVS-persisted alias mapping like Python's AliasResolver, which would let undeclared aliases survive migrations, is left as a follow-up.
The key-value store backend is now a byte transport: getValue reads buffers and leaves parsing to crawlee's frontend (previously the value got parsed twice), listKeys returns the new paginated shape, and recordExists passes through.
getStorageBackendCacheKey partitions crawlee's storage cache by API base URL and token.
Mechanical drift: getGlobalConfig renamed to getGlobalConfiguration; QueueOperationInfo/Constructor/Dictionary moved to @crawlee/types; snakeCaseToCamelCase inlined (removed upstream); ProxyConfiguration internals were privatized upstream (own log, no more base-class field reads); KeyValueStore.config/client fields replaced by serviceLocator/backend.

Validation

Unit tests cover both backends against a mocked apify-client (head estimation, dedup, prefetch, in-progress tracking, forefront ordering, lock duration raising, queueHasLockedRequests, the no-upsert contract), plus the storage backend's alias resolution and access mode wiring. Build, lint, and the full suite (148 tests) pass. The single mode is the successor of the flow validated end-to-end on the platform earlier in this PR; re-validating both modes on the platform against beta.105 is the remaining step.

Crawlee v4 (apify/crawlee#3729) redesigned `RequestQueueClient` into a stateful, pull-based interface (`addBatchOfRequests`, `fetchNextRequest`, `markRequestAsHandled`, `reclaimRequest`, `isEmpty`, `isFinished`) modeled on the Python SDK — replacing the old thin REST wrapper. The old name-remapping proxy can't satisfy it, so this adds a real `ApifyRequestQueueClient` implementing the interface on apify-client's REST API: - `fetchNextRequest` locks the head server-side (`listAndLockHead`) so the queue is safe to share across consumers; `markRequestAsHandled` / `reclaimRequest` update the request and release the lock. - request ids are derived from the unique key the same way as the platform. - `getMetadata`/`drop`, no-op `purge`. Also adds `ApifyStorageClient.setStatusMessage` (Crawlee v4 calls it on the storage client to surface crawl progress) and bumps `@crawlee/*` to `^4.0.0-beta.66`. Validated end-to-end on the Apify platform: a CheerioCrawler run through Apify Proxy completed 29/29 requests driven entirely by the new request-queue client. Note: published `@crawlee/types@beta.66` still ships the pre-redesign interface in its `.d.ts` even though `@crawlee/core`'s runtime calls the new methods, so the client implements a local copy of the new interface (`RequestQueueClientV4`) until crawlee republishes regenerated types.

@71

Crawlee v4 (apify/crawlee#3740) renamed the storage abstraction — `StorageClient` → `StorageBackend`, `RequestQueueClient` → `RequestQueueBackend`, `createXClient` → `createXBackend` — moved the in-memory client into `@crawlee/core` (`MemoryStorageBackend`, dropping the `@crawlee/memory-storage` package), and further refined the request-queue signatures (`fetchNextRequest`/`getRequest` now return `UpdateRequestSchema | undefined`, `null` → `undefined`, async `setExpectedRequestProcessingTimeSecs`). - `ApifyStorageClient` now implements `StorageBackend` (`createDatasetBackend` / `createKeyValueStoreBackend` / `createRequestQueueBackend`). - `ApifyRequestQueueClient` implements the real `RequestQueueBackend` directly — the temporary local interface copy is gone. - `StorageBackend` also dropped `setStatusMessage`, so `Actor.setStatusMessage` now sets the run status directly via apify-client. - Bumps `@crawlee/*` to `^4.0.0-beta.71`. Compile, 123 unit tests, and a local (memory-backend) smoke run on crawlee@71 all pass. The request-queue logic itself was validated end-to-end on the Apify platform against crawlee@66; on-platform validation on @71 is currently blocked by a crawlee packaging gap (its `@crawlee/fs-storage` native binding is unpublished for musl).

janbuchar

The description seems outdated, e.g.:

ApifyStorageClient.setStatusMessage — Crawlee v4 calls this on the storage client to surface crawl progress (it was warning setStatusMessage is not a function).

is no longer true since apify/crawlee#3818

janbuchar · 2026-07-21T09:36:37Z

 export interface OpenStorageContext {
    config: Configuration;
-    client?: StorageClient;
+    client?: StorageBackend;


I guess the field deserves a rename too?

janbuchar · 2026-07-21T09:39:44Z

+        // Crawlee v4's RequestQueueBackend is a stateful, pull-based interface that
+        // can't be satisfied by name-remapping; it's implemented on apify-client.


Suggested change

// Crawlee v4's RequestQueueBackend is a stateful, pull-based interface that

// can't be satisfied by name-remapping; it's implemented on apify-client.

ok thx bye

janbuchar · 2026-07-21T09:42:24Z

 * ```
 */
-export class ApifyStorageClient implements StorageClient {
+export class ApifyStorageClient implements StorageBackend {


Rename the class and file please

janbuchar · 2026-07-21T09:44:02Z

    }

-    async createRequestQueueClient(options?: CreateRequestQueueClientOptions): Promise<RequestQueueClient> {
+    async createRequestQueueBackend(options?: CreateRequestQueueBackendOptions): Promise<RequestQueueBackend> {


The options don't make it possible to toggle between single and shared modes (no locking vs locking) — is this tracked in an issue? Or shall we fix it right here right now?

Implements crawlee v4's RequestQueueBackend as two real platform-backed implementations, modeled on the Apify Python SDK's request queue clients: - single (default): assumes one consumer; no request locking, local head estimation and full-request caching, so most requests are processed without per-request read API calls. A one-time prefetch of existing queue contents lets resurrected runs deduplicate re-added requests locally instead of paying for platform writes. - shared (requestQueueAccess: 'shared'): safe for any number of concurrent consumers via server-side locking (listAndLockHead); the lock duration follows the crawler's expected request processing time. The mode is selected via Actor.init({ requestQueueAccess }) or the ApifyStorageBackend constructor option. Request queue clients now send a stable per-run clientKey (the run id), so a migrated or resurrected run re-acquires the locks of its previous incarnation. Also renames ApifyStorageClient to ApifyStorageBackend (following the crawlee v4 interface rename), resolves crawlee-native alias storage identifiers ('__default__' -> run default, schema storages via ACTOR_STORAGES_JSON), implements getStorageBackendCacheKey, fixes the key-value store backend to behave as a byte transport (buffered reads, paginated listKeys, recordExists), and adapts to crawlee 4.0.0-beta.105 API changes (getGlobalConfiguration, StorageIdentifier factory options, moved types, privatized ProxyConfiguration internals).

Renames the static getter and the instance property, aligning with the Configuration-related renames in crawlee v4 (getGlobalConfiguration).

…config to configuration Same rename as Actor.configuration, keeping the Configuration property naming consistent across the SDK's public classes.

janbuchar

No sense in blocking this, let's try it out in the wild

janbuchar · 2026-08-07T08:13:48Z

+    async createKeyValueStoreBackend(options?: StorageIdentifier): Promise<KeyValueStoreBackend> {
        const id = await this.resolveId(options, 'KeyValueStore');
        const client = this.client.keyValueStore(id);
        return adapt(


I don't know, it would be easier to understand if we just made two dummy classes, DatasetBackend and KeyValueStoreBackend, like for the request queue

B4nan added the adhoc Ad-hoc unplanned task added during the sprint. label Jun 16, 2026

B4nan requested a review from janbuchar June 16, 2026 13:21

janbuchar requested changes Jul 21, 2026

View reviewed changes

B4nan requested a review from szaganek as a code owner August 6, 2026 15:00

B4nan requested a review from janbuchar August 6, 2026 15:10

B4nan added 2 commits August 6, 2026 17:14

refactor!: rename Actor.config to Actor.configuration

3417b12

Renames the static getter and the instance property, aligning with the Configuration-related renames in crawlee v4 (getGlobalConfiguration).

refactor!: rename ProxyConfiguration.config and PlatformEventManager.…

7bafc7f

…config to configuration Same rename as Actor.configuration, keeping the Configuration property naming consistent across the SDK's public classes.

janbuchar approved these changes Aug 7, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat: implement crawlee v4 RequestQueueClient over apify-client - #643

feat: implement crawlee v4 RequestQueueClient over apify-client#643
B4nan wants to merge 5 commits into
v4from
feat/crawlee-v4-request-queue

B4nan commented Jun 16, 2026 •

edited

Loading

Uh oh!

janbuchar left a comment

Uh oh!

janbuchar Jul 21, 2026

Uh oh!

janbuchar Jul 21, 2026

Uh oh!

janbuchar Jul 21, 2026

Uh oh!

janbuchar Jul 21, 2026

Uh oh!

janbuchar left a comment

Uh oh!

janbuchar Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		// Crawlee v4's RequestQueueBackend is a stateful, pull-based interface that
		// can't be satisfied by name-remapping; it's implemented on apify-client.

Uh oh!

Conversation

B4nan commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Request queue access modes

Review feedback addressed

Crawlee beta.105 adaptation (was beta.71)

Validation

Uh oh!

janbuchar left a comment

Choose a reason for hiding this comment

Uh oh!

janbuchar Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

janbuchar Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

janbuchar Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

janbuchar Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

janbuchar left a comment

Choose a reason for hiding this comment

Uh oh!

janbuchar Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

B4nan commented Jun 16, 2026 •

edited

Loading