Skip to content

core: VPC endpoint logical ids change across synths, which breaks in-place upgrades #1465

Open

Description

@soberm

Which package(s) are affected?

@aws-blocks/core

Describe the bug

packages/core/src/cdk/vpc.ts builds the construct ids of the Blocks-provisioned VPC endpoints from service.name:

  • L269–273, gateway endpoints: const key = service.name; → VpcGw${key.replace(/[^a-zA-Z0-9]/g, '')}
  • L306–309, interface endpoints: const key = service.name; → VpcIf${key.replace(/[^a-zA-Z0-9]/g, '')}

service.name is a string like com.amazonaws.${Token[AWS.Region.13]}.s3, and that string embeds a CDK token sequence number (13). The number depends on how many tokens were created earlier in the synth, so a change anywhere else in the app can shift it, for example from Region13 to Region8. The construct id changes with it, so the CloudFormation logical id of the endpoint changes too.

On an existing stack, CloudFormation then replaces the S3 gateway endpoint and the CloudWatch Logs interface endpoint. It creates the new endpoints before it deletes the old ones, and the old and new can't exist together:

  • S3 gateway: route table … already has a route for the S3 prefix list.
  • CloudWatch Logs interface (private DNS on): conflicting DNS domain for logs….

The stack update fails and rolls back. A fresh deploy is not affected.

This was hit in an in-place upgrade test for #1447 (deploy main, then update the same stack to the branch): the deployed stack had …Region13… / TOKEN240, and the branch synthesized …Region8…. See #1447 (comment). Any app with Blocks-provisioned VPC endpoints (any in-VPC compute) can hit it on an upgrade that changes the number of tokens created before vpc.ts runs.

Expected behavior

VPC endpoint logical ids stay the same across synths unless the endpoint set itself changes. An upgrade that doesn't change which endpoints are provisioned leaves the endpoints untouched.

Reproduction steps

  1. Deploy an app that provisions Blocks VPC endpoints (for example a VPC-backed Database) from main.
  2. Make a change that creates more or fewer CDK tokens before the VPC endpoints are synthesized (for example, add tokenized conditions to an IAM role created earlier in the app).
  3. Run cdk diff: VpcGwcomamazonawsTokenAWSRegion…s3 and VpcIfcomamazonawsTokenAWSRegion…logs show as replaced.
  4. Deploy: the update fails with the errors above and rolls back.

Code snippet

for (const service of gatewayEndpoints) {
	const key = service.name; // "com.amazonaws.${Token[AWS.Region.13]}.s3"
	if (provisionedGateway.has(key)) continue;
	provisionedGateway.add(key);
	const constructId = `VpcGw${key.replace(/[^a-zA-Z0-9]/g, '')}`; // "VpcGwcomamazonawsTokenAWSRegion13s3"
	new ec2.GatewayVpcEndpoint(scope, constructId, { vpc, service });
}

Additional context

Fix direction: build the construct id (and the dedup key) from a stable key, such as the service short name (s3, logs, ssm, …), instead of service.name.

Changing the id changes each endpoint's logical id once, and that change hits the same create-before-delete conflict on existing stacks. So the fix also needs a decision on the upgrade path, for example:

  • keep the current ids for stacks that already have them, by reading them from the deployed template or a context value; or
  • add a one-time migration step that removes the old endpoints before the new ones are created.

This should ship in its own PR, separate from #1447.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

pending-maintainer-responseWaiting on a response from the maintainerspending-triageIssue is pending triage by the maintainers

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions