SDKQE-3975: carry Capella v4 API key credentials through fit-cli

README.md

@@ -144,6 +144,7 @@ than templated). When running a preset group, an override only has to apply to a
 ## Capella
 When running locally, we use Capella creds from your fit-cli config.  Generally you just need to provide your email address.  We default to using Capella's production environment.
 When running on CI, the user chooses what Capella environment to use (stage, dev, etc.) and we use previously-setup accounts for those. 
+cbdinocluster's cloud deployer authenticates with a Capella v4 organization API key.  By default the shared per-environment key is read from AWS Secrets Manager; override it with CAPELLA_API_KEY / CAPELLA_API_SECRET (or `config edit`).  The v2 username/password are still used for custom image deploys, server version changes, and columnar operations.
 
 ## Capabilities
 Each performer declares what it supports — a set of "caps" — over the `performerCapsFetch` gRPC call.

environments.json5

@@ -7,21 +7,24 @@
   //   capella  → instances[*].setup.capellaEnvironment   (default 'prod')
   //   results  → runs[*].situational.database.resultsEnvironment   (default 'prod'; used to be 'dev')
   //
-  // Only the passwords live in AWS Secrets Manager now (us-west-2); everything else
-  // (endpoint, oid, the shared Capella username, results host) is non-secret and lives here.
+  // Only the passwords and API keys live in AWS Secrets Manager now (us-west-2); everything else
+  // (endpoints, oid, the shared Capella username, results host) is non-secret and lives here.
   // Secrets are fetched at run time using the same AWS credentials fit-cli already needs to
   // create instances — so CI and laptops resolve identically. Access is controlled purely by
   // IAM on the secrets (CI's fit-cli-role + developers' SSO role need secretsmanager:GetSecretValue).
-  //   capella secret JSON → { "password": "..." }
+  //   capella secret JSON → { "password": "...", "apiKey": "...", "apiSecret": "..." }
   //   results secret JSON → { "password": "..." }   (the user is the non-secret `username` on the
   //   results block below; a "username" in the secret overrides it, for emergency role changes)
   //
-  // Capella password: a developer's own (from `config edit` / CAPELLA_PASS env) takes precedence
-  // locally; with none configured we fall back to the shared account password in the secret (and
-  // warn). The username is the shared account below unless overridden. Results creds always from the secret.
+  // Capella credentials: a developer's own (from `config edit` / the CAPELLA_PASS, CAPELLA_API_KEY,
+  // CAPELLA_API_SECRET env vars) take precedence locally; with none configured we fall back to the
+  // shared values in the secret. The apiKey/apiSecret pair is a v4 organization API key;
+  // cbdinocluster's cloud deployer authenticates with it (against v4Endpoint), while the v2
+  // username/password stay needed for custom image deploys, server version changes, and columnar.
+  // The username is the shared account below unless overridden. Results creds always from the secret.
   //
   // A null non-secret value means "not yet provided" — selecting that block fails fast.
-  // bun run secrets set capella/dev password='<password>'
+  // bun run secrets set capella/dev password='<password>' apiKey='<key>' apiSecret='<secret>'
   // bun run secrets set results/dev password='<password>'
 
   // Global defaults — not per-environment. Version strings passed to cbdinocluster
@@ -146,18 +149,21 @@
   capella: {
     prod: {
       endpoint: "https://api.cloud.couchbase.com",
+      v4Endpoint: "https://cloudapi.cloud.couchbase.com",
       oid: "62488bdd-d416-467e-84f7-fc7c1583a083",
       username: "sdk_qe@couchbase.com",
       secretId: "fit-cli/capella/prod",
     },
     dev: {
       endpoint: "https://api.dev.nonprod-project-avengers.com",
+      v4Endpoint: "https://cloudapi.dev.nonprod-project-avengers.com",
       oid: "6af08c0a-8cab-4c1c-b257-b521575c16d0",
       username: "sdk_qe@couchbase.com",
       secretId: "fit-cli/capella/dev",
     },
     stage: {
       endpoint: "https://api.stage.nonprod-project-avengers.com",
+      v4Endpoint: "https://cloudapi.stage.nonprod-project-avengers.com",
       oid: "5ae82d34-5da8-4313-af5f-90ea4142a4c6",
       username: "sdk_qe@couchbase.com",
       secretId: "fit-cli/capella/stage",

specs/credentials-and-secrets.md

@@ -21,6 +21,7 @@ So nothing __too__ secret should be stored - we are talking GHA PATs, Gerrit cre
 We use the known sdkqe@couchbase.com accounts, which are setup in all Capella envs, for all Capella testing by default.
 These are stored in environments.json5 and AWS Secrets [SECRETS1].
 [CAPELLA1] The user can provide a different acount in their fit-cli.  This is used both for localhost testing and clean cloud instance testing, an exception to the [CONFIG1] rule.  
+[CAPELLA2] cbdinocluster's cloud deployer authenticates with a v4 organization API key and secret.  These live per environment in the same AWS secret (apiKey/apiSecret keys) and can be overridden personally, like the password [CAPELLA1].  The v2 username/password are kept alongside: custom or unreleased image deploys, server version changes, and columnar operations still need them.
 
 ## AWS
 [SECRETS2] After encountering various problems when using user's localhost credentials in the clean EC2 testing, have decided to settle on EC2 testing exclusively using info from AWS secrets (Github PAT, Gerrit creds, etc).

src/cluster/cluster-create/default-cbdinocluster-init-config.ts

@@ -46,9 +46,9 @@ export function defaultCbdinoclusterInitArgs(
  * Like {@link defaultCbdinoclusterInitArgs} but leaves Capella *enabled* so that
  * `cbdinocluster init --auto` populates the `capella` block from the `CAPELLA_*`
  * environment variables fit-cli forwards to the box (see
- * `uploadRemoteCapellaConfig` / cbdinocluster's `cmd/init.go`, which reads
- * `CAPELLA_ENDPOINT/USER/PASS/OID`). With a `CAPELLA_USER` present, `--auto`
- * enables and fills in Capella; without one it leaves Capella disabled.
+ * `uploadRemoteCapellaConfig` / cbdinocluster's `cmd/init.go`). With a
+ * `CAPELLA_API_SECRET` present, `--auto` enables and fills in Capella; without
+ * one it leaves Capella disabled.
  *
  * `cloudProvider` picks which direct cloud-infra block cbdinocluster also needs:
  * situational PE (`private-endpoints setup-link`) calls the CSP's API directly, so

src/fit/config/config.ts

@@ -58,6 +58,9 @@ Edit options:
                          (AWS secret "fit-cli/gerrit/ssh-key" used as fallback on EC2).
   --capella-username <u> Your Capella username for situational/SIT runs (env: CAPELLA_USER / CAP_USER).
   --capella-password <p> Your Capella password (env: CAPELLA_PASS / CAP_PASS).
+  --capella-api-key <k>  Your Capella v4 organization API key (env: CAPELLA_API_KEY).
+                         Optional: falls back to the shared per-environment key in AWS Secrets Manager.
+  --capella-api-secret <s> Your Capella v4 organization API secret (env: CAPELLA_API_SECRET).
   --fit-performer-dir <path>  Path to your local transactions-fit-performer checkout, for localhost runs
                          (env: FIT_PERFORMER_DIR). Stored under localhost.repos.
   --cbdinocluster-path <path> Absolute path to the cbdinocluster binary, for non-PATH installs (localhost runs)
@@ -85,6 +88,8 @@ export interface AutoInitCliArgs {
   gerritUser?: string;
   capellaUsername?: string;
   capellaPassword?: string;
+  capellaApiKey?: string;
+  capellaApiSecret?: string;
   /** Local transactions-fit-performer checkout dir (localhost runs). */
   fitPerformerDir?: string;
   cbdinoclusterPath?: string;
@@ -135,6 +140,8 @@ export function parseEditArgs(argv: string[]): AutoInitCliArgs {
   const gerritUser = consumeValue(args, "--gerrit-user");
   const capellaUsername = consumeValue(args, "--capella-username");
   const capellaPassword = consumeValue(args, "--capella-password");
+  const capellaApiKey = consumeValue(args, "--capella-api-key");
+  const capellaApiSecret = consumeValue(args, "--capella-api-secret");
   const fitPerformerDir = consumeValue(args, "--fit-performer-dir");
   const cbdinoclusterPath = consumeValue(args, "--cbdinocluster-path");
   const configPath = consumeValue(args, "--config-path") ?? defaultFitCliConfigPath();
@@ -163,6 +170,8 @@ export function parseEditArgs(argv: string[]): AutoInitCliArgs {
     gerritUser,
     capellaUsername,
     capellaPassword,
+    capellaApiKey,
+    capellaApiSecret,
     fitPerformerDir,
     cbdinoclusterPath,
     configPath,

src/fit/config/edit.ts

@@ -120,12 +120,14 @@ function awsAnswersToConfig(answers: AwsInitAnswers, existingInstanceTypes?: Fit
   return instanceTypes ? { instanceTypes } : undefined;
 }
 
-/** Capella prompt defaults from a saved config: username from config, the rest defaulting to the hardcoded values. */
+/** Capella prompt defaults from a saved config; blank means "no personal override". */
 function capellaDefaultsFromConfig(config?: FitCliConfig): CapellaInitAnswers {
   const c = config?.capella;
   return {
     username: c?.username ?? "",
     password: c?.password ?? "",
+    apiKey: c?.apiKey ?? "",
+    apiSecret: c?.apiSecret ?? "",
   };
 }
 
@@ -393,7 +395,7 @@ async function promptForConfig(existing?: FitCliConfig, configPath?: string): Pr
 
 /**
  * Ask whether to configure Capella (situational/SIT only), and if so, the
- * username plus the five fields that default to the hardcoded values.
+ * personal username/password and v4 organization API key/secret overrides.
  */
 async function promptForCapella(
   existing?: FitCliConfig,
@@ -423,8 +425,8 @@ async function promptForCapella(
   });
 
   console.warn(
-    `\nWarning: Capella password will be saved in plaintext in ${configPath ?? "~/.fit-cli/config.json5"}.\n` +
-    `Set CAPELLA_PASS in your environment to avoid storing it on disk.\n`,
+    `\nWarning: Capella secrets will be saved in plaintext in ${configPath ?? "~/.fit-cli/config.json5"}.\n` +
+    `Set CAPELLA_PASS / CAPELLA_API_SECRET in your environment to avoid storing them on disk.\n`,
   );
   const capellaPassword = await password({
     promptId: "init.capella.password",
@@ -434,11 +436,28 @@ async function promptForCapella(
     mask: "*",
   });
 
+  const apiKey = await password({
+    promptId: "init.capella.api-key",
+    message: defaults.apiKey
+      ? "Capella v4 organization API key (leave blank to keep the current one):"
+      : "Capella v4 organization API key (leave blank to use the shared one from AWS Secrets Manager):",
+    mask: "*",
+  });
+  const apiSecret = await password({
+    promptId: "init.capella.api-secret",
+    message: defaults.apiSecret
+      ? "Capella v4 organization API secret (leave blank to keep the current one):"
+      : "Capella v4 organization API secret (leave blank to skip; set CAPELLA_API_SECRET env var instead):",
+    mask: "*",
+  });
+
   return {
     configureCapella: true,
     capella: {
       username,
       password: trimOptional(capellaPassword) ?? defaults.password,
+      apiKey: trimOptional(apiKey) ?? defaults.apiKey,
+      apiSecret: trimOptional(apiSecret) ?? defaults.apiSecret,
     },
   };
 }
@@ -483,6 +502,7 @@ export function formatConfigForDisplay(config: FitCliConfig): string {
           capella: {
             ...config.capella,
             ...(config.capella.password ? { password: ELIDED } : {}),
+            ...(config.capella.apiSecret ? { apiSecret: ELIDED } : {}),
           },
         }
       : {}),
@@ -567,6 +587,8 @@ function mask(value: string | undefined): string {
 const SECRET_FIELDS = new Set([
   "github.token",
   "capella.password",
+  "capella.apiKey",
+  "capella.apiSecret",
 ]);
 
 /** Check whether a diagnostic-only env var is present, appending an entry to the log. */
@@ -692,9 +714,10 @@ export function buildAutoConfig(
       : undefined;
   const output: FitCliOutputConfig | undefined = format ? { format } : undefined;
 
-  // Capella section: PERSONAL credentials only (username/password). The endpoint and
-  // org id are per-environment and come from environments.json5 at run time. Anchored
-  // on the username: with none there's nothing to log in as, so we skip the section.
+  // Capella section: PERSONAL credentials only (username/password and the v4
+  // organization API key/secret). The endpoints and org id are per-environment and
+  // come from environments.json5 at run time. With neither a username nor an API
+  // key there is nothing personal to store, so we skip the section.
   let capella: FitCliConfig["capella"] | undefined;
   if (args.disableCapella) {
     log.push({ field: "capella.*", source: "--disable-capella", found: false });
@@ -703,17 +726,25 @@ export function buildAutoConfig(
       { name: "CAPELLA_USER", value: env.CAPELLA_USER },
       { name: "CAP_USER", value: env.CAP_USER },
     ]);
-    if (!username) {
-      log.push({ field: "capella.*", source: "(no username)", found: false });
+    const apiKey = resolveField(log, "capella.apiKey", args.capellaApiKey, "--capella-api-key", [
+      { name: "CAPELLA_API_KEY", value: env.CAPELLA_API_KEY },
+    ]);
+    if (!username && !apiKey) {
+      log.push({ field: "capella.*", source: "(no username or API key)", found: false });
     } else {
       const capellaPassword = resolveField(log, "capella.password", args.capellaPassword, "--capella-password", [
         { name: "CAPELLA_PASS", value: env.CAPELLA_PASS },
         { name: "CAP_PASS", value: env.CAP_PASS },
       ]);
+      const apiSecret = resolveField(log, "capella.apiSecret", args.capellaApiSecret, "--capella-api-secret", [
+        { name: "CAPELLA_API_SECRET", value: env.CAPELLA_API_SECRET },
+      ]);
 
       capella = {
-        username,
+        ...(username ? { username } : {}),
         ...(capellaPassword ? { password: capellaPassword } : {}),
+        ...(apiKey ? { apiKey } : {}),
+        ...(apiSecret ? { apiSecret } : {}),
       };
     }
   }

src/fit/config/tests/auto-edit.test.ts

@@ -115,6 +115,21 @@ test("buildAutoConfig: CAP_* env (fit-app-deployment names) populates the person
   });
 });
 
+test("buildAutoConfig: CAPELLA_API_KEY/SECRET populate the personal v4 credentials", () => {
+  const env = {
+    CAPELLA_API_KEY: "org-key",
+    CAPELLA_API_SECRET: "org-secret",
+  };
+
+  const { config } = buildAutoConfig({ args: baseArgs(), env });
+
+  // A v4 API key alone is a valid personal credential; no username is needed.
+  assert.deepEqual(config.capella, {
+    apiKey: "org-key",
+    apiSecret: "org-secret",
+  });
+});
+
 test("buildAutoConfig: CAPELLA_* takes precedence over CAP_* aliases", () => {
   const env = { CAPELLA_USER: "primary@cb.com", CAP_USER: "alias@cb.com" };
 

src/fit/functional/run-from-definition/run-from-definition.ts

@@ -2261,7 +2261,7 @@ export async function runFromDefinition(
             }
             // Forward Capella and AWS settings before init so `cbdinocluster init --auto`
             // (run via a login shell sourcing ~/.profile) picks them up and writes the
-            // capella and aws blocks. Without a username it can't enable Capella, so fail
+            // capella and aws blocks. Without an API secret it can't enable Capella, so fail
             // clearly rather than letting `cbdinocluster allocate` later fail with "no deployers".
             capellaEndpoint = await uploadCapellaCredsForCloudDeployer(
               execution,

src/fit/shared/util/remote-fit-run.ts

@@ -229,13 +229,18 @@ function remoteCapellaConfigPath(rootDir: string): string {
 
 function capellaConfigScript(capella: ResolvedCapellaConfig): string {
   // The env var names cbdinocluster's `init` reads (see its cmd/init.go). With
-  // CAPELLA_USER present, `init --auto` enables Capella and fills the block from
-  // these; the situational init args leave Capella enabled for exactly this.
+  // CAPELLA_API_SECRET present, `init --auto` enables Capella and fills the block
+  // from these; without it the cloud deployer is silently left disabled. The v2
+  // user/pass stay forwarded: custom image deploys, server version changes, and
+  // columnar operations still authenticate with them.
   const lines = [
     `export CAPELLA_USER=${posixQuote(capella.username ?? "")}`,
     `export CAPELLA_ENDPOINT=${posixQuote(capella.endpoint)}`,
     `export CAPELLA_OID=${posixQuote(capella.organizationId)}`,
     `export CAPELLA_PASS=${posixQuote(capella.password)}`,
+    `export CAPELLA_V4_ENDPOINT=${posixQuote(capella.v4Endpoint)}`,
+    `export CAPELLA_API_KEY=${posixQuote(capella.apiKey)}`,
+    `export CAPELLA_API_SECRET=${posixQuote(capella.apiSecret)}`,
   ];
   // Optional: only present for environments the Capella team has issued them for
   // (currently just "dev"). Both have env-var fallbacks in cbdinocluster's `init`,