Add MySQL DBI config translator and pipeline orchestrator#2196
Open
Klarsen04 wants to merge 11 commits into
Open
Add MySQL DBI config translator and pipeline orchestrator#2196Klarsen04 wants to merge 11 commits into
Klarsen04 wants to merge 11 commits into
Conversation
Create the core MySQL receiver translator with functional options pattern: - translator.go with endpoint, username, passfile, TLS, localhost detection - translator_test.go with unit tests for all option functions - Register mysqlreceiver.NewFactory() in defaultcomponents - Add MySQLKey and MySQL DBI component name constants to common.go - Add go.mod dependency on mysqlreceiver
Add YAML configs and translator wiring for MySQL DBI pipelines: - dbi_dbload_mysql.yaml: count connector for active sessions by wait/user/db/sql/host - dbi_topsql_mysql.yaml: signaltometrics connector for top query metrics - transform_dbi_fix_start_time_mysql.yaml: transform processor for MySQL timestamps - Update count, signaltometrics, and transform translators to handle MySQL component names - Add unit tests for MySQL connector/processor creation
Add mysql[] array schema to database_insights in schema.json: - endpoint, instance_name, username, password_file - tls.ca_file for TLS configuration - logs.file_path for server log ingestion
Extend the DBI pipeline orchestrator to support MySQL instances: - Add translator_mysql.go with dbiMysqlTranslator implementing 4 pipeline types: - metrics/dbi_mysql_N: counter metrics + DBLoad/TopSQL from connectors - logs/dbi_mysql_N: log-to-metrics via count + signaltometrics connectors - logs/dbi_mysql_rawevents_N: raw query sample/top query events - logs/dbi_mysql_serverlogs_N: MySQL server log file ingestion - Add parseDbiMysqlInstances() to parse mysql[] config array - MySQL-specific resource attributes (db.system.name = "mysql") - MySQL-specific log group paths and monitor user exclusion filter - Unit tests for pipeline wiring with single/multiple instances
Add end-to-end config translation test for MySQL DBI: - dbi_mysql_config_linux.conf: input agent config fixture - dbi_mysql_config_linux.json: parsed JSON intermediate fixture - dbi_mysql_config_linux.yaml: golden YAML output (full OTel pipeline) - .mysql_credentials: test passfile for credential loading - Register TestMySQLDbiConfig in tocwconfig_unix_test.go
- Unify the DBI pipeline translator: merge translator_mysql.go into a single dbiTranslator that branches by engine (carried on the instance config), instead of maintaining a separate MySQL translator. - Make component naming consistent across engines using an engine suffix: count/dbi_dbload_<engine>, signaltometrics/dbi_topsql_<engine>, transform/dbi_fix_start_time_<engine>, and per-instance dbi_resource_<engine>_N, dbi_scope_<engine>_N, dbi_logs_<engine>_<dest>_N, dbi_exclude_monitor_<engine>_N. PostgreSQL components now carry the _postgresql suffix for consistency. - Rename engine-specific connector/processor YAMLs for symmetry (dbi_dbload_postgresql.yaml, dbi_topsql_postgresql.yaml, transform_dbi_fix_start_time_postgresql.yaml); the connector and transform translators select the config by engine. - Add the dbi_scope transform processor (cloudwatch.source/solution) to all MySQL pipelines, matching the PostgreSQL implementation. - Use the shared named resourcedetection/opentelemetry processor. - Add the validateOttlSafe check to the unified Translate() so username and instance_name are validated for both engines. - Fix the host metrics process scraper to match the mysqld process (mysqld.*) when MySQL DBI is configured. - Remove TLS/ca_file from the MySQL schema and receiver; MySQL DBI is localhost-only and always connects with TLS disabled. - Fix the MySQL passfile test fixture to the pgpass-style format (host:port:database:username:password) and assert all seven DB Load metrics. - Regenerate golden YAMLs (mysql, postgres, combined configs).
…upport The CloudWatch Console (Cinch) creates separate SSM parameters for each workload configuration (e.g. one for database-insights-postgres and one for database-insights-mysql). When the agent loads multiple config files from the .d/ directory, the config merger needs to deep-merge the database_insights section so that both postgresql and mysql arrays coexist. Without this rule, the agent treats database_insights from two separate config sources as conflicting values and drops one, causing missing log groups in CloudWatch. Register database_insights as a SectionMergeRule under the collect rule, matching the existing pattern for opentelemetry and collect.
- Move the complete 14-metric Top SQL connector config (dbi_topsql_mysql.yaml) into this PR. The full metric set was previously committed in the MySQL workload-detection PR by mistake, which left this PR's embedded config with only 2 metrics while the golden YAML had 14 (mismatch that failed the translation unit tests). All 14 metrics match what the mysqlreceiver publishes on db.server.top_query events. - Remove the user.name attribute from all Top SQL metrics: top queries come from events_statements_summary_by_digest, aggregated by digest and schema, so the receiver does not publish user.name on top_query events (validated against the forked mysqlreceiver RecordDbServerTopQueryEvent). - Remove the duplicate dbiMysqlKey variable in favor of the shared common.DatabaseInsightsMysqlKey. - Regenerate the MySQL golden YAML.
Update the test credential file to use the MySQL-native INI format (Option C) with required host, port, user, and password fields.
Klarsen04
added
the
ready for testing
Klarsen04
force-pushed
the
larkirs/mysql-pipeline-translator
branch
from
0e04bbc to
28af3da
Compare
Points to amazon-contributing/opentelemetry-collector-contrib PR #577 which replaces pgpass format with .my.cnf INI format for MySQL DBI credential files.
The DB Load connector condition only matched processlist_command ==
"Query", missing prepared statement executions ("Execute") used by
sysbench, ORMs, JDBC, and most real-world applications. This caused
DB Load metrics to undercount active sessions.
Replace the command-based filter with session.status != "idle" to
match all active sessions regardless of command type.
Klarsen04
force-pushed
the
larkirs/mysql-pipeline-translator
branch
from
28af3da to
be7ef57
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add component translators and pipeline orchestrator for MySQL database monitoring via the OpenTelemetry pipeline, enabling the
opentelemetry.collect.database_insights.mysql[]config section.Continuation of #2176
Changes
Component translators:
Pipeline orchestrator:
Bug fix (new in this PR):
processlist_command == "Query"tosession.status != "idle"to correctly count prepared statement executions (Execute command) used by sysbench, ORMs, and JDBC applicationsTests
make buildandmake lintpass