[da-vinci] Add OTel metrics to AggVersionedDaVinciRecordTransformerStats

[m-nagarajan]

Problem Statement

DaVinciRecordTransformerStats has 4 Tehuti sensors for record transformer operations (put/delete latency and error counts) but no OTel counterparts, making these metrics unavailable in OTel-based monitoring dashboards.

Solution

Add 2 OTel metrics to AggVersionedDaVinciRecordTransformerStats with a VENICE_RECORD_TRANSFORMER_OPERATION dimension (PUT/DELETE):

• record_transformer.latency (HISTOGRAM) — DaVinci record transformer operation latency
• record_transformer.error_count (COUNTER) — DaVinci record transformer operation error count

Uses the separate Tehuti+OTel API: Tehuti metrics are managed by the Reporter layer (DaVinciRecordTransformerStatsReporter) with AsyncGauge polling, while OTel records directly at the call point in AggVersionedDaVinciRecordTransformerStats. This follows the established pattern for versioned stats where Tehuti and OTel have fundamentally different recording paths.

Key design decisions:

• Per-store VeniceConcurrentHashMap maps for latency and error count metrics, cleaned up in handleStoreDeleted.
• private final boolean emitOtelMetrics cached at construction; recordOtelLatency and recordOtelErrorCount early-return when OTel is disabled, so the per-store maps are never populated in non-OTel deployments. Mirrors the established AggVersionedDIVStats pattern.
• @VisibleForTesting accessors (hasLatencyMetricFor, latencyStoreCount, etc.) let tests pin the cleanup contract on handleStoreDeleted without exposing the internal maps.

Code changes

• Added new code behind a config.
• Introduced new log lines.
  • Confirmed if logs need to be rate limited to avoid excessive logging.

Concurrency-Specific Checks

Both reviewer and PR author to verify

• Code has no race conditions or thread safety issues.
• Proper synchronization mechanisms (e.g., synchronized, RWLock) are used where needed.
• No blocking calls inside critical sections that could lead to deadlocks or performance degradation.
• Verified thread-safe collections are used (e.g., ConcurrentHashMap, CopyOnWriteArrayList).
• Validated proper exception handling in multi-threaded code to avoid silent thread termination.

How was this PR tested?

• New unit tests added.
• New integration tests added.
• Modified or extended existing tests.
• Verified backward compatibility (if applicable).

New tests:

• AggVersionedDaVinciRecordTransformerStatsOtelTest (9 tests): OTel histogram for put/delete latency, OTel counter for put/delete errors, operation dimension isolation, multi-store isolation (validates per-store entries stay distinct via the doAnswer Store mock so AbstractVeniceAggVersionedStats doesn't collapse them), handleStoreDeleted cleanup contract (asserts per-store maps go from 1 entry → 0 → 1 across record/delete/re-record), NPE prevention with OTel disabled and plain MetricsRepository.
• DaVinciRecordTransformerOtelMetricEntityTest: metric entity validation for both metrics.
• VeniceRecordTransformerOperationTest: dimension value validation (PUT/DELETE).

Does this PR introduce any user-facing or breaking changes?

• No. You can skip the rest of this section.
• Yes. Clearly explain the behavior change and its impact.

[Copilot]

Pull request overview

Adds OpenTelemetry (OTel) coverage for DaVinci record transformer operation stats so they’re available in OTel-based dashboards, aligning this component with the existing metric-entity + dimension patterns used across DaVinci/server stats.

Changes:

• Introduces a new OTel dimension VENICE_RECORD_TRANSFORMER_OPERATION (PUT/DELETE) and its enum value type.
• Adds two new OTel metric entities: record_transformer.latency (histogram) and record_transformer.error_count (counter), and wires them into AggVersionedDaVinciRecordTransformerStats.
• Registers the new metric entity enum in server metric aggregation and adds unit tests validating dimension naming and metric entity definitions/recording.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
internal/venice-client-common/src/main/java/com/linkedin/venice/stats/dimensions/VeniceMetricsDimensions.java	Adds the new VENICE_RECORD_TRANSFORMER_OPERATION dimension definition.
internal/venice-client-common/src/main/java/com/linkedin/venice/stats/dimensions/VeniceRecordTransformerOperation.java	Introduces PUT/DELETE dimension enum implementing VeniceDimensionInterface.
internal/venice-client-common/src/test/java/com/linkedin/venice/stats/dimensions/VeniceMetricsDimensionsTest.java	Extends dimension-name format tests to cover the new dimension.
internal/venice-client-common/src/test/java/com/linkedin/venice/stats/dimensions/VeniceRecordTransformerOperationTest.java	Validates dimension value mappings for PUT/DELETE.
clients/da-vinci-client/src/main/java/com/linkedin/davinci/stats/DaVinciRecordTransformerOtelMetricEntity.java	Defines OTel metric entities for record transformer latency and error count.
clients/da-vinci-client/src/main/java/com/linkedin/davinci/stats/ServerMetricEntity.java	Registers the new metric entity enum so it’s part of server metric aggregation.
clients/da-vinci-client/src/main/java/com/linkedin/davinci/stats/AggVersionedDaVinciRecordTransformerStats.java	Records OTel histogram/counter at the call site with per-store cached metric state.
clients/da-vinci-client/src/test/java/com/linkedin/davinci/stats/ServerMetricEntityTest.java	Updates expected aggregated metric entity count after adding new entities.
clients/da-vinci-client/src/test/java/com/linkedin/davinci/stats/DaVinciRecordTransformerOtelMetricEntityTest.java	Verifies metric entity definitions (name/type/unit/dimensions) for the new entities.
clients/da-vinci-client/src/test/java/com/linkedin/davinci/stats/AggVersionedDaVinciRecordTransformerStatsOtelTest.java	Adds OTel recording tests for latency/errors and validates operation-dimension isolation + NPE prevention.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sushantmane]

Reviewed with help from a few specialized review agents. Four actionable items below — two important (matching the established AggVersionedDIVStats short-circuit pattern, and a multi-store mock that silently collapses), two suggestions. The HISTOGRAM+COUNTER split, separate Tehuti+OTel API choice, and handleStoreDeleted cleanup all look correct.

Separate from the inline comments: the PR is currently CONFLICTING with main and will need a rebase before merge.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

[sushantmane]

lgtm