Skip to content

Implement bulk key fetching and deleting for large queue#44

Open
shioshiota wants to merge 4 commits into
masterfrom
feature/bulk-redis-operation
Open

Implement bulk key fetching and deleting for large queue#44
shioshiota wants to merge 4 commits into
masterfrom
feature/bulk-redis-operation

Conversation

@shioshiota

@shioshiota shioshiota commented Sep 7, 2021

Copy link
Copy Markdown
Contributor

We want to reduce redis load by delete-queue command when the queue has a large number of tasks.
*a large number is about 10^6

I implemented following improvements:

fixes #28

@shioshiota shioshiota force-pushed the feature/bulk-redis-operation branch 2 times, most recently from 7b3b989 to d3e5936 Compare September 7, 2021 07:45
@shioshiota shioshiota force-pushed the feature/bulk-redis-operation branch from d3e5936 to 21b33e1 Compare September 7, 2021 07:52
@shioshiota shioshiota marked this pull request as ready for review September 7, 2021 10:48
@shioshiota shioshiota changed the title [WIP] update: use SScan instead of SMembers Command update: use SScan instead of SMembers Command Sep 7, 2021
@shioshiota shioshiota changed the title update: use SScan instead of SMembers Command Implement bulk key fetching and deleting to handle a large quantity tasks in a queue Sep 7, 2021
@shioshiota shioshiota changed the title Implement bulk key fetching and deleting to handle a large quantity tasks in a queue Implement bulk key fetching and deleting to handle a large quantity of tasks in a queue Sep 7, 2021
@shioshiota shioshiota changed the title Implement bulk key fetching and deleting to handle a large quantity of tasks in a queue Implement bulk key fetching and deleting to handle a number quantity of tasks in a queue Sep 7, 2021
@shioshiota shioshiota requested a review from everpeace September 7, 2021 11:04

@everpeace everpeace left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left several comments. PTAL.

Comment thread pkg/backend/config/config.go Outdated
Comment thread pkg/backend/redis/queue.go Outdated
Comment thread pkg/backend/redis/queue.go Outdated
Client: client,
Backoff: backoffConfig,
ChunkSizeInGet: 1000,
ChunkSizeInDelete: 1000,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind adding tests for chunked deletion (large queue)??

@shioshiota shioshiota Sep 8, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a test for chunked deletion.
f2b8503

@everpeace everpeace changed the title Implement bulk key fetching and deleting to handle a number quantity of tasks in a queue Implement bulk key fetching and deleting for large queue Sep 7, 2021
@everpeace everpeace added the release-note/feature Feature category in release note label Sep 7, 2021
shioshiota and others added 2 commits September 8, 2021 17:19
Co-authored-by: Shingo Omura <everpeace@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-note/feature Feature category in release note

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The same problem as #25 occurs with delete-queue for large queues

2 participants