[controller] Register PARTITION_STATE and STORE_VERSION_STATE schemas at startup by pthirun · Pull Request #2745 · linkedin/venice

pthirun · 2026-04-15T21:09:56Z

Problem Statement

During rolling deployments, servers deployed before the system schema cluster controller
crash-loop when attempting to fetch new PartitionState or StoreVersionState schema versions.
The schemas are only registered by SystemSchemaInitializationRoutine, which runs exclusively
on the leader of the designated system schema cluster.
If that controller hasn't been deployed yet, the schema is missing and servers fail.

KAFKA_MESSAGE_ENVELOPE already has a ControllerClientBackedSystemSchemaInitializer that runs
on any child controller at startup, avoiding this deployment ordering issue. PARTITION_STATE
and STORE_VERSION_STATE do not.

Solution

Add ControllerClientBackedSystemSchemaInitializer for PARTITION_STATE and STORE_VERSION_STATE
alongside the existing KME initializer in VeniceController.java. Any child controller that
starts up will now register missing schema versions via the controller REST API.

Code changes

No new code behind a config. Uses the existing systemSchemaInitializationAtStartTimeEnabled gate.
No new log lines (ControllerClientBackedSystemSchemaInitializer already logs).

Concurrency-Specific Checks

N/A — runs at startup before serving traffic. Schema registration is idempotent.

How was this PR tested?

Compilation verified locally.
The ControllerClientBackedSystemSchemaInitializer is already tested for KME — same code path.
Verified the issue in EI: venice_system_store_PARTITION_STATE only had v1-v20, missing v21.

Does this PR introduce any user-facing or breaking changes?

No. Schema registration is additive and idempotent.

startup Add ControllerClientBackedSystemSchemaInitializer for PARTITION_STATE and STORE_VERSION_STATE, matching the existing pattern for KAFKA_MESSAGE_ENVELOPE. This allows any child controller to register new schema versions at startup, removing the dependency on deploying the system schema cluster controller first. Gated by new config controller.storage.protocol.schema.startup.registration.enabled (default: false) for intentional rollout. Requires the existing system.schema.initialization.at.start.time.enabled to also be true. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

kvargha

Please add some tests to validate the behavior

kvargha · 2026-05-04T21:18:18Z

+        ControllerClientBackedSystemSchemaInitializer partitionStateSchemaInitializer =
+            new ControllerClientBackedSystemSchemaInitializer(
+                AvroProtocolDefinition.PARTITION_STATE,
+                systemStoreCluster,
+                null,
+                null,
+                false,
+                ((VeniceHelixAdmin) admin).getSslFactory(),
+                childControllerUrl,
+                d2ServiceName,
+                regionD2Client,
+                d2ZkHost,
+                sslOnly);
+        partitionStateSchemaInitializer.execute();


Can we extract this out to a helper?

Address review feedback: - Extract createAndExecuteSchemaInitializer() helper to reduce duplication across KME, PARTITION_STATE, and STORE_VERSION_STATE initializers - Add config parsing tests for stateProtocolSchemaStartupRegistrationEnabled (default false + explicit true) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR aims to prevent child-region servers from crash-looping during rolling deployments by ensuring missing system-store schema versions for PARTITION_STATE and STORE_VERSION_STATE can be registered at controller startup (similar to the existing KAFKA_MESSAGE_ENVELOPE startup registration path).

Changes:

Add a new controller config flag to gate startup registration of PARTITION_STATE and STORE_VERSION_STATE schemas.
Refactor controller startup schema initialization to reuse a helper for creating/executing ControllerClientBackedSystemSchemaInitializer.
Add unit tests validating the new config flag default (false) and enabled behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceController.java	Adds helper-based startup schema registration; conditionally registers PARTITION_STATE and STORE_VERSION_STATE schemas.
services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerClusterConfig.java	Introduces a new boolean config field + getter for the startup registration gate.
internal/venice-common/src/main/java/com/linkedin/venice/ConfigKeys.java	Adds the new config key and its Javadoc describing intended behavior/dependency.
services/venice-controller/src/test/java/com/linkedin/venice/controller/TestVeniceControllerClusterConfig.java	Adds tests for the new config flag default and enabled cases.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    new ControllerClientBackedSystemSchemaInitializer(
+        protocolDefinition,
+        systemStoreCluster,
+        null,
+        null,
+        false,
+        ((VeniceHelixAdmin) admin).getSslFactory(),
+        childControllerUrl,
+        d2ServiceName,
+        regionD2Client,
+        d2ZkHost,
+        sslOnly).execute();


+  private static void createAndExecuteSchemaInitializer(
+      AvroProtocolDefinition protocolDefinition,
+      String systemStoreCluster,
+      Admin admin,
+      String childControllerUrl,
+      String d2ServiceName,
+      Optional<D2Client> regionD2Client,
+      String d2ZkHost,
+      boolean sslOnly) {
+    new ControllerClientBackedSystemSchemaInitializer(
+        protocolDefinition,
+        systemStoreCluster,
+        null,
+        null,
+        false,
+        ((VeniceHelixAdmin) admin).getSslFactory(),
+        childControllerUrl,


+  /**
+   * Whether to register PARTITION_STATE and STORE_VERSION_STATE schemas via
+   * ControllerClientBackedSystemSchemaInitializer at controller startup.
+   * Requires {@link #SYSTEM_SCHEMA_INITIALIZATION_AT_START_TIME_ENABLED} to also be true.
+   * Default: false.
+   */
+  public static final String CONTROLLER_STATE_PROTOCOL_SCHEMA_STARTUP_REGISTRATION_ENABLED =
+      "controller.state.protocol.schema.startup.registration.enabled";


  }

+  public boolean isStateProtocolSchemaStartupRegistrationEnabled() {
+    return stateProtocolSchemaStartupRegistrationEnabled;


kvargha · 2026-05-06T03:39:37Z

        ControllerClientBackedSystemSchemaInitializer metaSystemStoreSchemaInitializer =
            new ControllerClientBackedSystemSchemaInitializer(
                AvroProtocolDefinition.METADATA_SYSTEM_SCHEMA_STORE,
                systemStoreCluster,
                AvroProtocolDefinition.METADATA_SYSTEM_SCHEMA_STORE_KEY.getCurrentProtocolVersionSchema(),
                VeniceSystemStoreUtils.DEFAULT_USER_SYSTEM_STORE_UPDATE_QUERY_PARAMS,
                true,
                ((VeniceHelixAdmin) admin).getSslFactory(),
                childControllerUrl,
                d2ServiceName,
                regionD2Client,
                d2ZkHost,
                sslOnly);
        metaSystemStoreSchemaInitializer.execute();


Can this also use the helper?

kvargha · 2026-05-06T03:43:53Z

+  public void testStateProtocolSchemaStartupRegistrationDefaultsToFalse() {
+    Properties baseProps = getBaseSingleRegionProperties(false);
+    VeniceControllerClusterConfig config = new VeniceControllerClusterConfig(new VeniceProperties(baseProps));
+    assertFalse(config.isStateProtocolSchemaStartupRegistrationEnabled());
+  }


Can we add an integration test too where this is enabled?

pthirun force-pushed the pthiruna/register-partition-state-schema-at-startup branch from 2087150 to 3c9b2f2 Compare April 15, 2026 21:34

pthirun force-pushed the pthiruna/register-partition-state-schema-at-startup branch from 3c9b2f2 to 6ccf2de Compare April 15, 2026 22:43

kvargha reviewed May 4, 2026

View reviewed changes

kvargha requested a review from Copilot May 6, 2026 03:38

Copilot started reviewing on behalf of kvargha May 6, 2026 03:39 View session

Copilot AI reviewed May 6, 2026

View reviewed changes

kvargha reviewed May 6, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[controller] Register PARTITION_STATE and STORE_VERSION_STATE schemas at startup#2745

[controller] Register PARTITION_STATE and STORE_VERSION_STATE schemas at startup#2745
pthirun wants to merge 2 commits intolinkedin:mainfrom
pthirun:pthiruna/register-partition-state-schema-at-startup

pthirun commented Apr 15, 2026 •

edited

Loading

Uh oh!

kvargha left a comment

Uh oh!

kvargha May 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

kvargha May 6, 2026

Uh oh!

kvargha May 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pthirun commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem Statement

Solution

Code changes

Concurrency-Specific Checks

How was this PR tested?

Does this PR introduce any user-facing or breaking changes?

Uh oh!

kvargha left a comment

Choose a reason for hiding this comment

Uh oh!

kvargha May 4, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

kvargha May 6, 2026

Choose a reason for hiding this comment

Uh oh!

kvargha May 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pthirun commented Apr 15, 2026 •

edited

Loading