Redesign Security LLM performance matrix (#7146)#7193
Open
dhru42 wants to merge 6 commits into
Open
Conversation
Replace the six flat AI Assistant capability columns with an Agent Builder capability breakdown: seven sub-capabilities rolling up into an Overall Agent Builder Score, plus Attack Discovery, Automatic Migration, and a rolled-up Overall Score (default sort). Refresh all model scores from the latest evaluation and show real per-capability scores instead of "Not recommended" labels, with a clear rule that any score of 5 or below means the model is not recommended for that task. Add plain-language definitions for each Agent Builder sub-capability. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Elastic Docs AI PR menuCheck the box to run an AI review for this pull request.
Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team. |
Contributor
🔍 Preview links for changed docs |
Contributor
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
Prefix the seven sub-capability column headers with "Agent Builder:" so it's clear those columns belong to the Agent Builder capability, since Markdown tables can't span a single grouping header across multiple columns. Co-authored-by: Cursor <cursoragent@cursor.com>
Make the seven Agent Builder sub-capability column headers plain (non-bold) and keep the roll-up columns (Overall Agent Builder Score, Attack Discovery, Automatic Migration) and Overall Score bold, so the top-line scores stand out against the sub-capability detail. Co-authored-by: Cursor <cursoragent@cursor.com>
Remove the explanatory sentence about showing real scores instead of a "Not recommended" label, keeping only the concise threshold statement. Co-authored-by: Cursor <cursoragent@cursor.com>
Remove the "applies equally to AI Assistant or Agent Builder" sentence and state that the matrix tests each model across Agent Builder, Attack Discovery, and Automatic Migration. Co-authored-by: Cursor <cursoragent@cursor.com>
Bold the Overall Agent Builder Score data cells (Overall Score cells were already bold) so both roll-up columns are visually partitioned from the individual sub-capability scores. Co-authored-by: Cursor <cursoragent@cursor.com>
natasha-moore-elastic
left a comment
natasha-moore-elastic
left a comment
Contributor
There was a problem hiding this comment.
Thanks for opening the PR, @dhru42! Left a few questions and minor suggestions, but looks good overall.
| # Large language model performance matrix for {{elastic-sec}} [llm-performance-matrix] | ||
|
|
||
| This page summarizes internal test results comparing large language models (LLMs) across {{elastic-sec}} [AI chat](/explore-analyze/ai-features/ai-chat-experiences.md) and AI-powered feature use cases. These ratings apply equally whether you're using [AI Assistant](/solutions/security/ai/ai-assistant.md) or [Agent Builder](/solutions/security/ai/agent-builder/agent-builder.md). To learn more about these use cases, refer to [AI-powered features](/explore-analyze/ai-features.md#security-features). | ||
| This page summarizes internal test results comparing large language models (LLMs) across {{elastic-sec}} [AI chat](/explore-analyze/ai-features/ai-chat-experiences.md) and AI-powered feature use cases. The matrix tests each model across [Agent Builder](/solutions/security/ai/agent-builder/agent-builder.md), Attack Discovery, and Automatic Migration. To learn more about these use cases, refer to [AI-powered features](/explore-analyze/ai-features.md#security-features). |
Contributor
There was a problem hiding this comment.
Suggested change
| This page summarizes internal test results comparing large language models (LLMs) across {{elastic-sec}} [AI chat](/explore-analyze/ai-features/ai-chat-experiences.md) and AI-powered feature use cases. The matrix tests each model across [Agent Builder](/solutions/security/ai/agent-builder/agent-builder.md), Attack Discovery, and Automatic Migration. To learn more about these use cases, refer to [AI-powered features](/explore-analyze/ai-features.md#security-features). | |
| This page summarizes internal test results comparing large language models (LLMs) across {{elastic-sec}} [AI chat](/explore-analyze/ai-features/ai-chat-experiences.md) and AI-powered feature use cases. The matrix tests each model across [Agent Builder](/solutions/security/ai/agent-builder/agent-builder.md), [Attack Discovery](/solutions/security/ai/attack-discovery.md), and [Automatic Migration](/solutions/security/get-started/automatic-migration.md). To learn more about these use cases, refer to [AI-powered features](/explore-analyze/ai-features.md#security-features). |
|
|
||
| ## How the scores are calculated [_how_scores_are_calculated] | ||
|
|
||
| The matrix uses three top-line capability scores — **Agent Builder**, **Attack Discovery**, and **Automatic Migration** — that roll up into a single **Overall Score**. You can read the table top-down, from "how does this model perform across our AI features?" to "how good is it at the specific job I care about?" |
Contributor
There was a problem hiding this comment.
Suggested change
| The matrix uses three top-line capability scores — **Agent Builder**, **Attack Discovery**, and **Automatic Migration** — that roll up into a single **Overall Score**. You can read the table top-down, from "how does this model perform across our AI features?" to "how good is it at the specific job I care about?" | |
| The matrix uses three top-line capability scores — Agent Builder, Attack Discovery, and Automatic Migration — that roll up into a single *Overall Score*. You can use the table to answer questions such as "How does this model perform across Elastic's AI features?" and "How good is it at the specific job I care about?" |
|
|
||
| The matrix uses three top-line capability scores — **Agent Builder**, **Attack Discovery**, and **Automatic Migration** — that roll up into a single **Overall Score**. You can read the table top-down, from "how does this model perform across our AI features?" to "how good is it at the specific job I care about?" | ||
|
|
||
| * **Overall Agent Builder Score** is the average of the seven Agent Builder sub-capabilities (Alert Analysis, Entity Analytics, Threat Hunting, Detection Rules, Workflow Authoring, Triggering Workflows, and Multi-Step Executions). It summarizes how well a model handles agentic Security work end to end. |
Contributor
There was a problem hiding this comment.
Suggested change
| * **Overall Agent Builder Score** is the average of the seven Agent Builder sub-capabilities (Alert Analysis, Entity Analytics, Threat Hunting, Detection Rules, Workflow Authoring, Triggering Workflows, and Multi-Step Executions). It summarizes how well a model handles agentic Security work end to end. | |
| * **Overall Agent Builder Score** is the average of the seven Agent Builder [sub-capabilities](#_agent_builder_sub_capabilities). It summarizes how well a model handles agentic Security work end to end. |
| The matrix uses three top-line capability scores — **Agent Builder**, **Attack Discovery**, and **Automatic Migration** — that roll up into a single **Overall Score**. You can read the table top-down, from "how does this model perform across our AI features?" to "how good is it at the specific job I care about?" | ||
|
|
||
| * **Overall Agent Builder Score** is the average of the seven Agent Builder sub-capabilities (Alert Analysis, Entity Analytics, Threat Hunting, Detection Rules, Workflow Authoring, Triggering Workflows, and Multi-Step Executions). It summarizes how well a model handles agentic Security work end to end. | ||
| * **Overall Score** is the average of the Agent Builder, Attack Discovery, and Automatic Migration scores. It reflects how a model performs across the breadth of our AI features rather than any single workflow, and is the default sort for the tables below. |
Contributor
There was a problem hiding this comment.
Suggested change
| * **Overall Score** is the average of the Agent Builder, Attack Discovery, and Automatic Migration scores. It reflects how a model performs across the breadth of our AI features rather than any single workflow, and is the default sort for the tables below. | |
| * **Overall Score** is the average of the Agent Builder, Attack Discovery, and Automatic Migration scores. It reflects how a model performs across the breadth of Elastic's AI features rather than any single workflow, and is the default sort for the tables below. |
| ### What each Agent Builder sub-capability measures [_agent_builder_sub_capabilities] | ||
|
|
||
| * **Alert Analysis** — Triage an alert, reach the correct disposition, pull related alerts, and enrich with threat intel. | ||
| * **Entity Analytics** — Investigate hosts and users using purpose-built entity lookups and risk context. |
Contributor
There was a problem hiding this comment.
Just checking this is just for hosts and users and doesn't include services?
| * **Overall Agent Builder Score** is the average of the seven Agent Builder sub-capabilities (Alert Analysis, Entity Analytics, Threat Hunting, Detection Rules, Workflow Authoring, Triggering Workflows, and Multi-Step Executions). It summarizes how well a model handles agentic Security work end to end. | ||
| * **Overall Score** is the average of the Agent Builder, Attack Discovery, and Automatic Migration scores. It reflects how a model performs across the breadth of our AI features rather than any single workflow, and is the default sort for the tables below. | ||
|
|
||
| ### What each Agent Builder sub-capability measures [_agent_builder_sub_capabilities] |
Contributor
There was a problem hiding this comment.
Do these sub-capabilities correlate to the built-in Security skills? It seems like some of them match the skills, and some don't.
| * **Detection Rules** — Author a working detection rule, grounded in research where requested. | ||
| * **Workflow Authoring** — Produce a valid, executable automation workflow (verified by actually creating, enabling, and running it). | ||
| * **Triggering Workflows** — Call the correct backed action for the task (for example, a hash lookup, an on-call schedule, or case creation). | ||
| * **Multi-Step Executions** — Chain several steps in the right order, carrying findings forward, without skipping or fabricating steps. |
Contributor
There was a problem hiding this comment.
Does this refer to Workflows steps?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Redesigns and refreshes the Large language model performance matrix for Elastic Security page per #7146.
The previous page listed six flat capability columns evaluated on the legacy AI Assistant and a single average score. This change turns the page into workflow-specific guidance:
Closes #7146
Generative AI disclosure
Tool(s) and model(s) used: Cursor (Claude)