Update Index sizing guidelines in docs after Autosharding

[ankikuma]

Update the docs here.
Indices that are create with index version greater than or equal to SHARD_OBLIVIOUS_SLICING can grow upto 48 shards (a dynamic setting AUTO_RESHARD_MAX_SHARDS_SETTING), via Autosharding. An index is resharded (to 2x shards) when its' average shard size crosses 100GB.

[github-actions]

Elastic Docs AI PR menu

Check the box to run an AI review for this pull request.

• Review docs changes (docs-review). Status: not started.

Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team.

[github-actions]

🔍 Preview links for changed docs

• deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md
• deploy-manage/deploy/elastic-cloud/serverless.md
• manage-data/data-store/index-basics.md

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[yuvielastic]

Is 4.8TB the max index size that has been tested across all use-cases w/o any performance impact?

[ankikuma]

Thanks for reviewing Yuvi.
I am actually not sure if we have specific tests with large number of shards. Let me check and get back.

The original guidelines calculate the maximum index size based on 100 GB shards and default shard count per project. This is because performance is expected to degrade when shards grow larger than 100 GB.
With Autosharding, we are able to increase the number of shards in the index once the shards get larger than 100 GB. That means we can support a larger index while still keeping the shard size to 100 GB. We currently have an upper limit on the number of shards an index can auto shard upto (=48). This is just because autosharding is a new feature and we wanted to prevent against runaway autosharding. So the 4.8TB number is just 48 x 100 GB.
We believe that autoscaling will take care of distributing the shards across nodes as needed, as the index grows.

[yuvielastic]

Thanks Ankita for the comment. Let's confirm if we have tests that can align with the statement (without any performance impact) so that we can say it with confidence.

[lkts]

I wonder if we want to be this specific. 48 shards is just a limit of autosharding feature and i can imagine us manually resharding to 96 (and beyond?) on a case by case basis. AFAIK there is no technical limitation here?

[ankikuma]

I agree @lkts. We can round that up to ~5 TB. I do mention that the limit can be raised further via overrides. We could reword that as well to say something like "This is a soft limit and can be raised on a case-by-case basis".
I wanted to see if Yuvi agrees with this doc change and how much information we want to give the customer.
I talked to Jason about performance tests and he agrees that it will be good to set a baseline for 6 shards x 100 GB indices to compare against larger 48 x 100 GB indices. I am working with him to run some tests.

[lkts]

I would maybe frame this as "There is no limit and we have confirmed internally that 5 TB definitely works. Work with support if your use case exceeds that.". What do you think (obviously wording can be way nicer)?

[yuvielastic]

Is there a knob that we control internally that we can use to increase the index size above 4.8TB without any performance impact? If yes, do we know the max size?

[ankikuma]

The knob is AUTO_RESHARD_MAX_SHARDS_SETTING, but again, I am not sure about tests.

[yuvielastic]

This seems to be an internal knob that we don't expose to customers, right?

[ankikuma]

That's correct, my understanding is that we don't want to expose shard level details to serverless customers.

[yuvielastic]

Are there any caveats to June 1 2026 or we can saw explicitly that all indices after June 1 2026 across all project types (O11y, Security, ES3) benefit from this.

[ankikuma]

The June 1, 2026 date comes from the fact that the minimum required index version SHARD_OBLIVIOUS_SLICING was committed on May 22 and was picked up in production by June 1.

I can't think of any other caveats.

[lkts]

I wonder if we want to be this specific. 48 shards is just a limit of autosharding feature and i can imagine us manually resharding to 96 (and beyond?) on a case by case basis. AFAIK there is no technical limitation here?

[lkts]

I am not sure we should advertise overrides in public documentation. Is there precedence for this?

[yuvielastic]

So "overrides" is not a public facing word that we use; so we cannot use that.

We have precedence on the index limit (as an example) where we mention that there are limits that can be increased.

Verbiage from the public doc:

The index limit is adjustable and can be increased by request, while others are fixed. To request a limit increase, open a support case, and include your preferred new value and a brief description of your use case. Providing meaningful details around your use case and desired outcome ensures that Elastic can make recommendations that best suit your workload.

So similar to that, if we are mentioning a limit (that can be increased), we can use similar verbiage (or just combine within existing verbiage that it also applies to this limit).

[ankikuma]

Agreed. Perhaps we can say "This is a soft limit and can be raised on a case-by-case basis" ?

[shainaraskas]

can we tell them to reindex if the size becomes an issue? will they automatically get the benefit for models where the index is abstracted, like data streams?

[shainaraskas]

the heading and therefore body might be wrong here. I assume that this index sizing guideline applies to all projects equally, not just ES ones?

[yetanothertw]

Suggesting a small edit for minimalism and to avoid repetition:

Suggested change

	Elasticsearch uses sharding to distribute the data in your index across the cluster - this helps to support growing data volumes and load. In {{serverless-full}}, the sharding is fully managed for you. You don’t need to think about choosing the optimal shard count for an index because the sharding will automatically adjust to suit your needs. That said, there are still some guidelines to share: these are best practices that tend to give the best results over the long term. You should design your indexing and data lifecycle strategy with the size and growth of your data in mind.
	Elasticsearch uses sharding to distribute the data in your index across the cluster - this helps support growing data volumes and load. In {{serverless-full}}, the sharding is fully managed for you and will automatically adjust to suit your needs.
	The following best practices will help you build an indexing and data lifecycle strategy that scales with your data over time:

[yetanothertw]

Removing the colon here as it's included in the preceding paragraph.

Suggested change

:

[yetanothertw]

Suggested change

	* Large numbers of very small (or empty) indices should be avoided whenever possible. Each index has a resource cost within your project and the service must scale your project capacity to accommodate. If you recognize this pattern in your design, consider options for organizing your data into fewer indices.
	* Avoid large numbers of very small or empty indices where possible. Each index has a resource cost within your project and the service must scale its capacity to accommodate. If your design follows this pattern, consider organizing your data into fewer indices.

This looks very much like the indexing strategies we describe in the billing docs. See the index count and size list item.

Should these be aligned more?

[yetanothertw]

Maybe we can say something like:

Suggested change

	* At the other end of the spectrum, a design where each index can grow to many terabytes might also ultimately result in performance trade-offs - consider creating multiple smaller indices that you can query using an [alias](/manage-data/data-store/aliases.md) or naming pattern.
	* Indices that grow too large can introduce performance trade-offs. Consider creating multiple smaller indices you can query using an [alias](/manage-data/data-store/aliases.md) or naming pattern.

[yetanothertw]

Can we define when is appropriate? is it when we deal with time-series data?

Suggested change

	* Use a data stream instead of a single index when appropriate, and take advantage of data stream lifecycle to manage data retention. Using date-based naming patterns with data streams is generally undesirable. It’s better to let the [data stream lifecycle](/manage-data/lifecycle/data-stream.md) take care of aging out data that is no longer relevant.
	* When your data is time-series or append-only, use a data stream instead of a single index and take advantage of the [data stream lifecycle](/manage-data/lifecycle/data-stream.md) to manage data retention. Avoid using date-based naming patterns with data streams and let the lifecycle age out data automatically.

[yetanothertw]

For clarity, I'd keep the mention of index as it's the index limits that are adjustable.

Suggested change

	Adjustable limits can be increased by request, while others are fixed. To request a limit increase, open a support case, and include your preferred new value and a brief description of your use case. Providing meaningful details around your use case and desired outcome ensures that Elastic can make recommendations that best suit your workload.
	Index limits can be increased by request, while others are fixed. To request a limit increase, open a support case, and include your preferred new value and a brief description of your use case. Providing meaningful details around your use case and desired outcome ensures that Elastic can make recommendations that best suit your workload.

[yetanothertw]

Suggested change

**Autoscaled:** To meet your performance requirements, the system automatically adjusts to your workloads. For example, when you have a short term spike on the data you ingest, more resources are allocated for that period of time. When the spike is over, the system uses less resources, without any action on your end. Some project-level limits apply to ensure performance and stability, including a [limit on the number of indices per project](/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md#elasticsearch-differences-serverless-index-size) that can be adjusted by request.

**Autoscaled:** To meet your performance requirements, the system automatically adjusts to your workloads. For example, when you have a short-term spike on the data you ingest, more resources are allocated for that period of time. When the spike is over, the system uses less resources, without any action on your end. Some project-level limits apply to ensure performance and stability, including a [limit on the number of indices per project](/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md#elasticsearch-differences-serverless-index-size) that can be adjusted by request.