Change MacOS jobs to use sonoma runners #42926

alopezz · 2025-11-11T08:35:20Z

What does this PR do?

We've added a new group of macOS runners which use a newer version of macOS (see https://github.com/DataDog/ci-platform-machine-images/pull/503 and https://github.com/DataDog/cloud-inventory/pull/46351) with the intention of using that for datadog-agent CI moving forward. This makes it so that we start to use those runners in our CI.

Motivation

ABLD-293

Describe how you validated your changes

CI passing.

Additional Notes

agent-platform-auto-pr · 2025-11-11T08:39:55Z

Gitlab CI Configuration Changes

Modified Jobs

.bazel:runner:macos-amd64

  .bazel:runner:macos-amd64:
    tags:
-   - macos:ventura-amd64
?           ^^ ^^^
+   - macos:sonoma-amd64
?           ^^ ^^
    - specific:true

.bazel:runner:macos-arm64

  .bazel:runner:macos-arm64:
    tags:
-   - macos:ventura-arm64
?           ^^ ^^^
+   - macos:sonoma-arm64
?           ^^ ^^
    - specific:true

agent_dmg-arm64-a7

  agent_dmg-arm64-a7:
    after_script:
    - "if [ \"$SIGN\" = true ]; then\n  security delete-keychain \"build.keychain\"\
      \ || true\nfi\n"
    - sudo umount /Volumes/Agent || true
    artifacts:
      expire_in: 2 weeks
      paths:
      - omnibus/pkg/*.dmg
      - omnibus/pkg/version-manifest.json
    before_script:
    - sudo umount /Volumes/Agent || true
    - rm -rf "$OMNIBUS_GIT_CACHE_DIR" || true
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_IMAGE-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION
      paths:
      - omnibus/vendor/bundle
    needs:
    - go_mod_tidy_check
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - comp/core/gui/guiimpl/systray/**/*
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - omnibus/**/*
        - .gitlab-ci.yml
        - release.json
        - .gitlab/package_build/**/*
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
    - if: $CI_COMMIT_BRANCH == "main" || $DEPLOY_AGENT == "true" || $RUN_ALL_BUILDS
        == "true" || $DDR_WORKFLOW_ID != null
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - allow_failure: true
      when: manual
    script:
    - set -eo pipefail
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  # Get the commit from the build image variable in the format `vPIPELINE_ID-COMMIT`\n\
      \  export BUILDIMAGES_COMMIT=\"${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"\
      $(curl -s https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  curl -Lo dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - echo Setting up Go
    - mkdir -p ~/go
    - export GO_VERSION="$(cat .go-version)"
    - eval "$(gimme $GO_VERSION)"
    - export PATH="$PATH:$GOROOT/bin"
    - echo Go version should be $GO_VERSION
    - go version
    - dda inv check-go-version
    - DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DD_API_KEY
    - DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
      token)" || exit $?; export DD_APP_KEY
    - 'AWS_TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds:
      21600")"
  
      RUNNER_ID="$(curl -s http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token:
      $AWS_TOKEN" || hostname)"
  
      datadog-ci tag --level job --tags macos_runner:"$RUNNER_ID"
  
      echo "Reported runner ID to Datadog: $RUNNER_ID"
  
      '
    - "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
      .[0-9]+\\.(x|[0-9]+)$ ]]; then\n  dda inv -- -e macos.report-versions -l all ||\
      \ true\nfi\n"
    - "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n  echo Trying to remove inactive versions\n\
      \  dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
      \ || true\n  dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
      \ || true\nfi\n"
    - 'export TMPDIR=/tmp/gitlabci
  
      NEWTMPDIR="$RUNNER_TEMP_PROJECT_DIR/gitlabci"
  
      rm -fr "$(realpath $TMPDIR)" "$NEWTMPDIR"
  
      mkdir "$NEWTMPDIR"
  
      sudo ln -fs "$NEWTMPDIR" $TMPDIR
  
      echo "Temporary folder created, TMPDIR=$TMPDIR -> $NEWTMPDIR"
  
      '
    - sudo bash -c "rm -rf /var/cache/omnibus/src/*" || true
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - 'export GOMODCACHE=~/gomodcache
  
      mkdir -p $GOMODCACHE
  
      '
    - bash .gitlab/package_build/build_agent_dmg.sh
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
-   - macos:ventura-arm64
?           ^^ ^^^
+   - macos:sonoma-arm64
?           ^^ ^^
    - specific:true
    timeout: 2h
    variables:
      INTEGRATION_WHEELS_CACHE_BUCKET: dd-agent-omnibus
      INTEGRATION_WHEELS_SKIP_CACHE_UPLOAD: true
      KEYCHAIN_NAME: build.keychain
      NOTARIZATION_ATTEMPTS: 3
      NOTARIZATION_TIMEOUT: 15m
      NOTARIZATION_WAIT_TIME: 15s
      S3_OMNIBUS_CACHE_BUCKET: dd-ci-datadog-agent-omnibus-cache-build-stable
      SIGN: true

agent_dmg-x64-a7

  agent_dmg-x64-a7:
    after_script:
    - "if [ \"$SIGN\" = true ]; then\n  security delete-keychain \"build.keychain\"\
      \ || true\nfi\n"
    - sudo umount /Volumes/Agent || true
    artifacts:
      expire_in: 2 weeks
      paths:
      - omnibus/pkg/*.dmg
      - omnibus/pkg/version-manifest.json
    before_script:
    - sudo umount /Volumes/Agent || true
    - rm -rf "$OMNIBUS_GIT_CACHE_DIR" || true
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_IMAGE-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION
      paths:
      - omnibus/vendor/bundle
    needs:
    - go_mod_tidy_check
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - comp/core/gui/guiimpl/systray/**/*
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - omnibus/**/*
        - .gitlab-ci.yml
        - release.json
        - .gitlab/package_build/**/*
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
    - if: $CI_COMMIT_BRANCH == "main" || $DEPLOY_AGENT == "true" || $RUN_ALL_BUILDS
        == "true" || $DDR_WORKFLOW_ID != null
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - allow_failure: true
      when: manual
    script:
    - set -eo pipefail
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  # Get the commit from the build image variable in the format `vPIPELINE_ID-COMMIT`\n\
      \  export BUILDIMAGES_COMMIT=\"${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"\
      $(curl -s https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  curl -Lo dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - echo Setting up Go
    - mkdir -p ~/go
    - export GO_VERSION="$(cat .go-version)"
    - eval "$(gimme $GO_VERSION)"
    - export PATH="$PATH:$GOROOT/bin"
    - echo Go version should be $GO_VERSION
    - go version
    - dda inv check-go-version
    - DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DD_API_KEY
    - DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
      token)" || exit $?; export DD_APP_KEY
    - 'AWS_TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds:
      21600")"
  
      RUNNER_ID="$(curl -s http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token:
      $AWS_TOKEN" || hostname)"
  
      datadog-ci tag --level job --tags macos_runner:"$RUNNER_ID"
  
      echo "Reported runner ID to Datadog: $RUNNER_ID"
  
      '
    - "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
      .[0-9]+\\.(x|[0-9]+)$ ]]; then\n  dda inv -- -e macos.report-versions -l all ||\
      \ true\nfi\n"
    - "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n  echo Trying to remove inactive versions\n\
      \  dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
      \ || true\n  dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
      \ || true\nfi\n"
    - 'export TMPDIR=/tmp/gitlabci
  
      NEWTMPDIR="$RUNNER_TEMP_PROJECT_DIR/gitlabci"
  
      rm -fr "$(realpath $TMPDIR)" "$NEWTMPDIR"
  
      mkdir "$NEWTMPDIR"
  
      sudo ln -fs "$NEWTMPDIR" $TMPDIR
  
      echo "Temporary folder created, TMPDIR=$TMPDIR -> $NEWTMPDIR"
  
      '
    - sudo bash -c "rm -rf /var/cache/omnibus/src/*" || true
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - 'export GOMODCACHE=~/gomodcache
  
      mkdir -p $GOMODCACHE
  
      '
    - bash .gitlab/package_build/build_agent_dmg.sh
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
-   - macos:ventura-amd64
?           ^^ ^^^
+   - macos:sonoma-amd64
?           ^^ ^^
    - specific:true
    timeout: 2h
    variables:
      INTEGRATION_WHEELS_CACHE_BUCKET: dd-agent-omnibus
      INTEGRATION_WHEELS_SKIP_CACHE_UPLOAD: true
      KEYCHAIN_NAME: build.keychain
      NOTARIZATION_ATTEMPTS: 3
      NOTARIZATION_TIMEOUT: 15m
      NOTARIZATION_WAIT_TIME: 15s
      S3_OMNIBUS_CACHE_BUCKET: dd-ci-datadog-agent-omnibus-cache-build-stable
      SIGN: true

bazel:build-deps:macos-amd64

  bazel:build-deps:macos-amd64:
    cache:
    - key:
        files:
        - .bazelversion
        prefix: bazelversion-$CI_RUNNER_DESCRIPTION
      paths:
      - .cache/bazelisk
      - .cache/bazel/install
      policy: pull-push
      when: on_success
    - key: bazel-$CI_RUNNER_DESCRIPTION
      paths:
      - .cache/bazel/cache
      - .cache/bazel/disk
      - .cache/bazel/repo-contents
      policy: pull-push
      when: on_success
    needs: []
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - bazel build //deps/...
    stage: deps_build
    tags:
-   - macos:ventura-amd64
?           ^^ ^^^
+   - macos:sonoma-amd64
?           ^^ ^^
    - specific:true
    variables:
      BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      BAZEL_DISK_CACHE: $BAZEL_OUTPUT_USER_ROOT/disk
      BAZEL_OUTPUT_USER_ROOT: $XDG_CACHE_HOME/bazel
      BAZEL_REPO_CONTENTS_CACHE: $BAZEL_OUTPUT_USER_ROOT/repo-contents
      XDG_CACHE_HOME: $CI_PROJECT_DIR/.cache

bazel:build-deps:macos-arm64

  bazel:build-deps:macos-arm64:
    cache:
    - key:
        files:
        - .bazelversion
        prefix: bazelversion-$CI_RUNNER_DESCRIPTION
      paths:
      - .cache/bazelisk
      - .cache/bazel/install
      policy: pull-push
      when: on_success
    - key: bazel-$CI_RUNNER_DESCRIPTION
      paths:
      - .cache/bazel/cache
      - .cache/bazel/disk
      - .cache/bazel/repo-contents
      policy: pull-push
      when: on_success
    needs: []
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - bazel build //deps/...
    stage: deps_build
    tags:
-   - macos:ventura-arm64
?           ^^ ^^^
+   - macos:sonoma-arm64
?           ^^ ^^
    - specific:true
    variables:
      BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      BAZEL_DISK_CACHE: $BAZEL_OUTPUT_USER_ROOT/disk
      BAZEL_OUTPUT_USER_ROOT: $XDG_CACHE_HOME/bazel
      BAZEL_REPO_CONTENTS_CACHE: $BAZEL_OUTPUT_USER_ROOT/repo-contents
      XDG_CACHE_HOME: $CI_PROJECT_DIR/.cache

bazel:tests:macos-amd64

  bazel:tests:macos-amd64:
    cache:
    - key:
        files:
        - .bazelversion
        prefix: bazelversion-$CI_RUNNER_DESCRIPTION
      paths:
      - .cache/bazelisk
      - .cache/bazel/install
      policy: pull-push
      when: on_success
    - key: bazel-$CI_RUNNER_DESCRIPTION
      paths:
      - .cache/bazel/cache
      - .cache/bazel/disk
      - .cache/bazel/repo-contents
      policy: pull-push
      when: on_success
    needs: []
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - bazel test //bazel/tests/...
    stage: source_test
    tags:
-   - macos:ventura-amd64
?           ^^ ^^^
+   - macos:sonoma-amd64
?           ^^ ^^
    - specific:true
    variables:
      BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      BAZEL_DISK_CACHE: $BAZEL_OUTPUT_USER_ROOT/disk
      BAZEL_OUTPUT_USER_ROOT: $XDG_CACHE_HOME/bazel
      BAZEL_REPO_CONTENTS_CACHE: $BAZEL_OUTPUT_USER_ROOT/repo-contents
      XDG_CACHE_HOME: $CI_PROJECT_DIR/.cache

bazel:tests:macos-arm64

  bazel:tests:macos-arm64:
    cache:
    - key:
        files:
        - .bazelversion
        prefix: bazelversion-$CI_RUNNER_DESCRIPTION
      paths:
      - .cache/bazelisk
      - .cache/bazel/install
      policy: pull-push
      when: on_success
    - key: bazel-$CI_RUNNER_DESCRIPTION
      paths:
      - .cache/bazel/cache
      - .cache/bazel/disk
      - .cache/bazel/repo-contents
      policy: pull-push
      when: on_success
    needs: []
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - bazel test //bazel/tests/...
    stage: source_test
    tags:
-   - macos:ventura-arm64
?           ^^ ^^^
+   - macos:sonoma-arm64
?           ^^ ^^
    - specific:true
    variables:
      BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      BAZEL_DISK_CACHE: $BAZEL_OUTPUT_USER_ROOT/disk
      BAZEL_OUTPUT_USER_ROOT: $XDG_CACHE_HOME/bazel
      BAZEL_REPO_CONTENTS_CACHE: $BAZEL_OUTPUT_USER_ROOT/repo-contents
      XDG_CACHE_HOME: $CI_PROJECT_DIR/.cache

lint_macos_gitlab_amd64

  lint_macos_gitlab_amd64:
    before_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - export AWS_RETRY_MODE=standard
    - export AWS_RETRY_MAX_ATTEMPTS=5
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      '
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  # Get the commit from the build image variable in the format `vPIPELINE_ID-COMMIT`\n\
      \  export BUILDIMAGES_COMMIT=\"${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"\
      $(curl -s https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  curl -Lo dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DD_API_KEY
    - DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
      token)" || exit $?; export DD_APP_KEY
    - 'AWS_TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds:
      21600")"
  
      RUNNER_ID="$(curl -s http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token:
      $AWS_TOKEN" || hostname)"
  
      datadog-ci tag --level job --tags macos_runner:"$RUNNER_ID"
  
      echo "Reported runner ID to Datadog: $RUNNER_ID"
  
      '
    - "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
      .[0-9]+\\.(x|[0-9]+)$ ]]; then\n  dda inv -- -e macos.report-versions -l all ||\
      \ true\nfi\n"
    - "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n  echo Trying to remove inactive versions\n\
      \  dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
      \ || true\n  dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
      \ || true\nfi\n"
    - 'export TMPDIR=/tmp/gitlabci
  
      NEWTMPDIR="$RUNNER_TEMP_PROJECT_DIR/gitlabci"
  
      rm -fr "$(realpath $TMPDIR)" "$NEWTMPDIR"
  
      mkdir "$NEWTMPDIR"
  
      sudo ln -fs "$NEWTMPDIR" $TMPDIR
  
      echo "Temporary folder created, TMPDIR=$TMPDIR -> $NEWTMPDIR"
  
      '
    - dda inv -- -e rtloader.make
    - dda inv -- -e rtloader.install
    - dda inv -- -e install-tools
    needs:
    - go_deps
    - go_tools_deps
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - dda inv -- -e linter.go --cpus 12 --debug --timeout 60
    stage: lint
    tags:
-   - macos:ventura-amd64
?           ^^ ^^^
+   - macos:sonoma-amd64
?           ^^ ^^
    - specific:true

lint_macos_gitlab_arm64

  lint_macos_gitlab_arm64:
    before_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - export AWS_RETRY_MODE=standard
    - export AWS_RETRY_MAX_ATTEMPTS=5
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      '
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  # Get the commit from the build image variable in the format `vPIPELINE_ID-COMMIT`\n\
      \  export BUILDIMAGES_COMMIT=\"${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"\
      $(curl -s https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  curl -Lo dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DD_API_KEY
    - DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
      token)" || exit $?; export DD_APP_KEY
    - 'AWS_TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds:
      21600")"
  
      RUNNER_ID="$(curl -s http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token:
      $AWS_TOKEN" || hostname)"
  
      datadog-ci tag --level job --tags macos_runner:"$RUNNER_ID"
  
      echo "Reported runner ID to Datadog: $RUNNER_ID"
  
      '
    - "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
      .[0-9]+\\.(x|[0-9]+)$ ]]; then\n  dda inv -- -e macos.report-versions -l all ||\
      \ true\nfi\n"
    - "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n  echo Trying to remove inactive versions\n\
      \  dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
      \ || true\n  dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
      \ || true\nfi\n"
    - 'export TMPDIR=/tmp/gitlabci
  
      NEWTMPDIR="$RUNNER_TEMP_PROJECT_DIR/gitlabci"
  
      rm -fr "$(realpath $TMPDIR)" "$NEWTMPDIR"
  
      mkdir "$NEWTMPDIR"
  
      sudo ln -fs "$NEWTMPDIR" $TMPDIR
  
      echo "Temporary folder created, TMPDIR=$TMPDIR -> $NEWTMPDIR"
  
      '
    - dda inv -- -e rtloader.make
    - dda inv -- -e rtloader.install
    - dda inv -- -e install-tools
    needs:
    - go_deps
    - go_tools_deps
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - dda inv -- -e linter.go --cpus 12 --debug --timeout 60
    stage: lint
    tags:
-   - macos:ventura-arm64
?           ^^ ^^^
+   - macos:sonoma-arm64
?           ^^ ^^
    - specific:true

test_kmt_local_setup_macos

  test_kmt_local_setup_macos:
    before_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - export AWS_RETRY_MODE=standard
    - export AWS_RETRY_MAX_ATTEMPTS=5
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      '
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  # Get the commit from the build image variable in the format `vPIPELINE_ID-COMMIT`\n\
      \  export BUILDIMAGES_COMMIT=\"${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"\
      $(curl -s https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  curl -Lo dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DD_API_KEY
    - DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
      token)" || exit $?; export DD_APP_KEY
    - 'AWS_TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds:
      21600")"
  
      RUNNER_ID="$(curl -s http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token:
      $AWS_TOKEN" || hostname)"
  
      datadog-ci tag --level job --tags macos_runner:"$RUNNER_ID"
  
      echo "Reported runner ID to Datadog: $RUNNER_ID"
  
      '
    - "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
      .[0-9]+\\.(x|[0-9]+)$ ]]; then\n  dda inv -- -e macos.report-versions -l all ||\
      \ true\nfi\n"
    - "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n  echo Trying to remove inactive versions\n\
      \  dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
      \ || true\n  dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
      \ || true\nfi\n"
    - 'export TMPDIR=/tmp/gitlabci
  
      NEWTMPDIR="$RUNNER_TEMP_PROJECT_DIR/gitlabci"
  
      rm -fr "$(realpath $TMPDIR)" "$NEWTMPDIR"
  
      mkdir "$NEWTMPDIR"
  
      sudo ln -fs "$NEWTMPDIR" $TMPDIR
  
      echo "Temporary folder created, TMPDIR=$TMPDIR -> $NEWTMPDIR"
  
      '
    - dda inv -- -e rtloader.make
    - dda inv -- -e rtloader.install
    - dda inv -- -e install-tools
    needs: []
    rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - changes:
        compare_to: $COMPARE_TO_BRANCH
        paths:
        - tasks/**/*
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    script:
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ddbuild.io/DataDog/".insteadOf
      "[email protected]:"
    - export GITHUB_TOKEN=$(dda inv github.get-token-from-app)
    - export KMT_TEST_INFRA_DEFINITIONS_PATH="$(go env GOPATH)"/src/github.com/DataDog/test-infra-definitions
    - dda inv -- kmt.init --remote-setup-only --skip-ssh-setup --exclude-requirements
      $KMT_EXCLUDE_REQUIREMENTS --images ""
    - export PATH=$PATH:$HOME/.pulumi/bin
    - dda inv -- kmt.selfcheck --remote-setup-only --exclude-requirements $KMT_EXCLUDE_REQUIREMENTS
      --show-flare-for-failures
    stage: source_test
    tags:
-   - macos:ventura-amd64
?           ^^ ^^^
+   - macos:sonoma-amd64
?           ^^ ^^
    - specific:true
    variables:
      KMT_EXCLUDE_REQUIREMENTS: Docker,Compiler,UserInDockerGroup,AWSConfig,PulumiPlugins,MacBasePackages

tests_macos_gitlab_amd64

  tests_macos_gitlab_amd64:
    after_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  # Get the commit from the build image variable in the format `vPIPELINE_ID-COMMIT`\n\
      \  export BUILDIMAGES_COMMIT=\"${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"\
      $(curl -s https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  curl -Lo dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    - CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV token) || exit
      $?; export CODECOV_TOKEN
    - dda inv -- -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - export AWS_RETRY_MODE=standard
    - export AWS_RETRY_MAX_ATTEMPTS=5
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      '
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  # Get the commit from the build image variable in the format `vPIPELINE_ID-COMMIT`\n\
      \  export BUILDIMAGES_COMMIT=\"${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"\
      $(curl -s https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  curl -Lo dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DD_API_KEY
    - DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
      token)" || exit $?; export DD_APP_KEY
    - 'AWS_TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds:
      21600")"
  
      RUNNER_ID="$(curl -s http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token:
      $AWS_TOKEN" || hostname)"
  
      datadog-ci tag --level job --tags macos_runner:"$RUNNER_ID"
  
      echo "Reported runner ID to Datadog: $RUNNER_ID"
  
      '
    - "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
      .[0-9]+\\.(x|[0-9]+)$ ]]; then\n  dda inv -- -e macos.report-versions -l all ||\
      \ true\nfi\n"
    - "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n  echo Trying to remove inactive versions\n\
      \  dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
      \ || true\n  dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
      \ || true\nfi\n"
    - 'export TMPDIR=/tmp/gitlabci
  
      NEWTMPDIR="$RUNNER_TEMP_PROJECT_DIR/gitlabci"
  
      rm -fr "$(realpath $TMPDIR)" "$NEWTMPDIR"
  
      mkdir "$NEWTMPDIR"
  
      sudo ln -fs "$NEWTMPDIR" $TMPDIR
  
      echo "Temporary folder created, TMPDIR=$TMPDIR -> $NEWTMPDIR"
  
      '
    - dda inv -- -e rtloader.make
    - dda inv -- -e rtloader.install
    - dda inv -- -e install-tools
    needs:
    - go_deps
    - go_tools_deps
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - dda inv -- -e agent.build
    - dda inv -- -e test --rerun-fails=2 --race --profile --cpus 12 --result-json $TEST_OUTPUT_FILE
      --junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer --coverage
    stage: source_test
    tags:
-   - macos:ventura-amd64
?           ^^ ^^^
+   - macos:sonoma-amd64
?           ^^ ^^
    - specific:true
    variables:
      FLAKY_PATTERNS_CONFIG: $CI_PROJECT_DIR/flaky-patterns-runtime.yaml
      TEST_OUTPUT_FILE: test_output.json

tests_macos_gitlab_arm64

  tests_macos_gitlab_arm64:
    after_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  # Get the commit from the build image variable in the format `vPIPELINE_ID-COMMIT`\n\
      \  export BUILDIMAGES_COMMIT=\"${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"\
      $(curl -s https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  curl -Lo dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - $CI_PROJECT_DIR/tools/ci/junit_upload.sh
    - CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV token) || exit
      $?; export CODECOV_TOKEN
    - dda inv -- -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
    artifacts:
      expire_in: 2 weeks
      paths:
      - $TEST_OUTPUT_FILE
      - junit-*.tgz
      reports:
        annotations:
        - $EXTERNAL_LINKS_PATH
        junit:
        - '**/junit-out-*.xml'
      when: always
    before_script:
    - export VAULT_ADDR=https://vault.us1.ddbuild.io
    - vault login -method=aws -no-print
    - export AWS_RETRY_MODE=standard
    - export AWS_RETRY_MAX_ATTEMPTS=5
    - 'eval $(gimme $(cat .go-version))
  
      export GOPATH=$GOROOT
  
      '
    - "if [ -z \"$TMPDIR\" ]; then\n  echo \"TMPDIR must be set\" >& 2\n  exit 1\nfi\n"
    - export DDA_DIR="$TMPDIR/dda-${CI_JOB_ID}"
    - export PATH="$DDA_DIR:$PATH"
    - export DDA_NO_DYNAMIC_DEPS=1
    - "# Perform installation only if the directory does not exist\nif [ ! -d \"$DDA_DIR\"\
      \ ]; then\n  # Get the commit from the build image variable in the format `vPIPELINE_ID-COMMIT`\n\
      \  export BUILDIMAGES_COMMIT=\"${CI_IMAGE_LINUX#*-}\"\n  export DDA_VERSION=\"\
      $(curl -s https://raw.githubusercontent.com/DataDog/datadog-agent-buildimages/${BUILDIMAGES_COMMIT}/dda.env\
      \ | awk -F= '/^DDA_VERSION=/ {print $2}')\"\n  # Detect architecture and download\
      \ appropriate binary\n  if [ \"$(uname -m)\" = \"arm64\" ]; then\n    dda_target_triple=\"\
      aarch64-apple-darwin\"\n  else\n    dda_target_triple=\"x86_64-apple-darwin\"\n\
      \  fi\n  curl -Lo dda.tar.gz https://github.com/DataDog/datadog-agent-dev/releases/download/${DDA_VERSION}/dda-${dda_target_triple}.tar.gz\n\
      \  tar -xzf dda.tar.gz\n  mkdir -p \"$DDA_DIR\"\n  sudo mv dda $DDA_DIR\n  rm\
      \ -f dda.tar.gz\n  dda self dep sync -f legacy-tasks\n  dda self pip install awscli==1.29.45\n\
      fi\n"
    - DD_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DD_API_KEY
    - DD_APP_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_APP_KEY_ORG2"
      token)" || exit $?; export DD_APP_KEY
    - 'AWS_TOKEN="$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds:
      21600")"
  
      RUNNER_ID="$(curl -s http://169.254.169.254/latest/meta-data/instance-id -H "X-aws-ec2-metadata-token:
      $AWS_TOKEN" || hostname)"
  
      datadog-ci tag --level job --tags macos_runner:"$RUNNER_ID"
  
      echo "Reported runner ID to Datadog: $RUNNER_ID"
  
      '
    - "if [ \"$CI_COMMIT_BRANCH\" = \"main\" ] || [[ \"$CI_COMMIT_BRANCH\" =~ ^[0-9]+\\\
      .[0-9]+\\.(x|[0-9]+)$ ]]; then\n  dda inv -- -e macos.report-versions -l all ||\
      \ true\nfi\n"
    - "if [ \"$((RANDOM%20))\" -eq 0 ]; then\n  echo Trying to remove inactive versions\n\
      \  dda inv -- -e macos.remove-inactive-versions -l python -t \"$PYTHON_VERSION\"\
      \ || true\n  dda inv -- -e macos.remove-inactive-versions -l go -t \"$(cat .go-version)\"\
      \ || true\nfi\n"
    - 'export TMPDIR=/tmp/gitlabci
  
      NEWTMPDIR="$RUNNER_TEMP_PROJECT_DIR/gitlabci"
  
      rm -fr "$(realpath $TMPDIR)" "$NEWTMPDIR"
  
      mkdir "$NEWTMPDIR"
  
      sudo ln -fs "$NEWTMPDIR" $TMPDIR
  
      echo "Temporary folder created, TMPDIR=$TMPDIR -> $NEWTMPDIR"
  
      '
    - dda inv -- -e rtloader.make
    - dda inv -- -e rtloader.install
    - dda inv -- -e install-tools
    needs:
    - go_deps
    - go_tools_deps
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    rules:
    - if: $RUN_UNIT_TESTS == "off"
      when: never
    - if: $CI_COMMIT_BRANCH == "main"
      variables:
        COVERAGE_CACHE_FLAG: --push-coverage-cache
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_COMMIT_TAG != null
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - if: $RUN_UNIT_TESTS == "on"
      variables:
        COVERAGE_CACHE_FLAG: ''
        FAST_TESTS: 'false'
    - variables:
        COVERAGE_CACHE_FLAG: --pull-coverage-cache
        FAST_TESTS: 'true'
    script:
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache.tar.xz
    - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
    - rm -f modcache_tools.tar.xz
    - dda inv -- -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
    - FAST_TESTS_FLAG=""
    - if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
      fi
    - dda inv -- -e agent.build
    - dda inv -- -e test --rerun-fails=2 --race --profile --cpus 12 --result-json $TEST_OUTPUT_FILE
      --junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer --coverage
    stage: source_test
    tags:
-   - macos:ventura-arm64
?           ^^ ^^^
+   - macos:sonoma-arm64
?           ^^ ^^
    - specific:true
    variables:
      FLAKY_PATTERNS_CONFIG: $CI_PROJECT_DIR/flaky-patterns-runtime.yaml
      TEST_OUTPUT_FILE: test_output.json

Changes Summary

Removed	Modified	Added	Renamed
0	13	0	0

ℹ️ Diff available in the job log.

agent-platform-auto-pr · 2025-11-11T09:18:06Z

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor 604ccaa

Successful checks

Info

	Quality gate	Delta	On disk size (MiB)	Delta	On wire size (MiB)
✅	agent_deb_amd64	$${+0}$$	$${684.36}$$ < $${695.26}$$	$${+0.01}$$	$${166.76}$$ < $${171.42}$$
✅	agent_deb_amd64_fips	$${+0}$$	$${678.7}$$ < $${689.56}$$	$${+0.03}$$	$${165.33}$$ < $${168.75}$$
✅	agent_heroku_amd64	$${0}$$	$${337.01}$$ < $${337.56}$$	$${-0.01}$$	$${89.76}$$ < $${90.78}$$
✅	agent_msi	$${0}$$	$${998.46}$$ < $${1015.38}$$	$${+0.02}$$	$${144.24}$$ < $${149.42}$$
✅	agent_rpm_amd64	$${+0}$$	$${684.35}$$ < $${695.24}$$	$${-0.01}$$	$${169.95}$$ < $${172.66}$$
✅	agent_rpm_amd64_fips	$${+0}$$	$${678.69}$$ < $${689.55}$$	$${+0.05}$$	$${167.62}$$ < $${171.31}$$
✅	agent_rpm_arm64	$${+0}$$	$${673.87}$$ < $${684.17}$$	$${+0.04}$$	$${154.83}$$ < $${158.47}$$
✅	agent_rpm_arm64_fips	$${+0}$$	$${669.3}$$ < $${679.54}$$	$${+0}$$	$${153.54}$$ < $${156.53}$$
✅	agent_suse_amd64	$${+0}$$	$${684.35}$$ < $${695.24}$$	$${-0.01}$$	$${169.95}$$ < $${172.66}$$
✅	agent_suse_amd64_fips	$${+0}$$	$${678.69}$$ < $${689.55}$$	$${+0.05}$$	$${167.62}$$ < $${171.31}$$
✅	agent_suse_arm64	$${+0}$$	$${673.87}$$ < $${684.17}$$	$${+0.04}$$	$${154.83}$$ < $${158.47}$$
✅	agent_suse_arm64_fips	$${+0}$$	$${669.3}$$ < $${679.54}$$	$${+0}$$	$${153.54}$$ < $${156.53}$$
✅	docker_agent_amd64	$${+0}$$	$${747.44}$$ < $${747.75}$$	$${+0}$$	$${251.91}$$ < $${252.67}$$
✅	docker_agent_arm64	$${+0}$$	$${761.29}$$ < $${761.44}$$	$${-0}$$	$${242.36}$$ < $${243.19}$$
✅	docker_agent_jmx_amd64	$${+0}$$	$${938.31}$$ < $${938.63}$$	$${+0}$$	$${320.54}$$ < $${321.29}$$
✅	docker_agent_jmx_arm64	$${+0}$$	$${940.76}$$ < $${940.91}$$	$${-0}$$	$${306.96}$$ < $${307.81}$$
✅	docker_cluster_agent_amd64	$${-0}$$	$${206.56}$$ < $${207.49}$$	$${+0}$$	$${69.26}$$ < $${70.2}$$
✅	docker_cluster_agent_arm64	$${-0}$$	$${222.43}$$ < $${223.38}$$	$${-0}$$	$${65.46}$$ < $${66.42}$$
✅	docker_cws_instrumentation_amd64	$${0}$$	$${7.07}$$ < $${7.12}$$	$${+0}$$	$${2.95}$$ < $${3.29}$$
✅	docker_cws_instrumentation_arm64	$${0}$$	$${6.69}$$ < $${6.92}$$	$${+0}$$	$${2.7}$$ < $${3.07}$$
✅	docker_dogstatsd_amd64	$${-0}$$	$${38.47}$$ < $${39.3}$$	$${-0}$$	$${14.85}$$ < $${15.76}$$
✅	docker_dogstatsd_arm64	$${+0}$$	$${37.12}$$ < $${37.94}$$	$${-0}$$	$${14.29}$$ < $${14.83}$$
✅	dogstatsd_deb_amd64	$${0}$$	$${29.69}$$ < $${30.53}$$	$${-0}$$	$${7.82}$$ < $${8.75}$$
✅	dogstatsd_deb_arm64	$${0}$$	$${28.21}$$ < $${29.11}$$	$${+0}$$	$${6.76}$$ < $${7.71}$$
✅	dogstatsd_rpm_amd64	$${0}$$	$${29.69}$$ < $${30.53}$$	$${-0}$$	$${7.83}$$ < $${8.76}$$
✅	dogstatsd_suse_amd64	$${0}$$	$${29.69}$$ < $${30.53}$$	$${-0}$$	$${7.83}$$ < $${8.76}$$
✅	iot_agent_deb_amd64	$${0}$$	$${42.26}$$ < $${43.19}$$	$${+0}$$	$${11.03}$$ < $${11.98}$$
✅	iot_agent_deb_arm64	$${0}$$	$${39.99}$$ < $${40.92}$$	$${+0}$$	$${9.5}$$ < $${10.45}$$
✅	iot_agent_deb_armhf	$${0}$$	$${39.93}$$ < $${40.86}$$	$${-0}$$	$${9.61}$$ < $${10.56}$$
✅	iot_agent_rpm_amd64	$${0}$$	$${42.26}$$ < $${43.19}$$	$${-0}$$	$${11.05}$$ < $${12.0}$$
✅	iot_agent_suse_amd64	$${0}$$	$${42.26}$$ < $${43.19}$$	$${-0}$$	$${11.05}$$ < $${12.0}$$

cit-pr-commenter · 2025-11-11T09:32:09Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 8c903ccf-97cb-495f-bb3f-576c7b06e155

Baseline: 604ccaa
Comparison: 7ce8a11
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+0.23	[+0.06, +0.41]	1	Logs

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	quality_gate_metrics_logs	memory utilization	+0.73	[+0.51, +0.95]	1	Logs bounds checks dashboard
➖	docker_containers_memory	memory utilization	+0.56	[+0.26, +0.86]	1	Logs
➖	ddot_metrics_sum_cumulative	memory utilization	+0.52	[+0.35, +0.69]	1	Logs
➖	file_tree	memory utilization	+0.45	[+0.41, +0.50]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	+0.30	[+0.26, +0.34]	1	Logs bounds checks dashboard
➖	docker_containers_cpu	% cpu utilization	+0.23	[+0.06, +0.41]	1	Logs
➖	ddot_metrics	memory utilization	+0.17	[-0.07, +0.40]	1	Logs
➖	quality_gate_idle	memory utilization	+0.07	[+0.03, +0.11]	1	Logs bounds checks dashboard
➖	file_to_blackhole_500ms_latency	egress throughput	+0.03	[-0.58, +0.63]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	+0.02	[-0.57, +0.62]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.01, +0.01]	1	Logs
➖	quality_gate_logs	% cpu utilization	-0.01	[-2.75, +2.73]	1	Logs bounds checks dashboard
➖	uds_dogstatsd_to_api	ingress throughput	-0.01	[-0.25, +0.22]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	-0.03	[-0.63, +0.57]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	-0.03	[-0.64, +0.57]	1	Logs
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	-0.07	[-0.12, -0.01]	1	Logs
➖	ddot_logs	memory utilization	-0.11	[-0.17, -0.05]	1	Logs
➖	ddot_metrics_sum_cumulativetodelta_exporter	memory utilization	-0.20	[-0.43, +0.03]	1	Logs
➖	otlp_ingest_logs	memory utilization	-0.26	[-0.35, -0.17]	1	Logs
➖	otlp_ingest_metrics	memory utilization	-0.48	[-0.63, -0.33]	1	Logs
➖	ddot_metrics_sum_delta	memory utilization	-0.54	[-0.75, -0.33]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-0.83	[-0.89, -0.77]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	docker_containers_cpu	simple_check_run	10/10
✅	docker_containers_memory	memory_usage	10/10
✅	docker_containers_memory	simple_check_run	10/10
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	cpu_usage	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	memory_usage	10/10	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.

Change MacOS jobs to use sonoma runners

7ce8a11

github-actions bot added medium review PR review might take time team/agent-build team/agent-devx labels Nov 11, 2025

alopezz added the changelog/no-changelog label Nov 11, 2025

alopezz added the qa/done QA done before merge and regressions are covered by tests label Nov 11, 2025

alopezz marked this pull request as ready for review November 11, 2025 14:12

alopezz requested review from a team as code owners November 11, 2025 14:12

rdesgroppes approved these changes Nov 11, 2025

View reviewed changes

alopezz added the ask-review Ask required teams to review this PR label Nov 11, 2025

ofek approved these changes Nov 11, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Change MacOS jobs to use sonoma runners #42926

Change MacOS jobs to use sonoma runners #42926

alopezz commented Nov 11, 2025 •

edited

Loading

Uh oh!

agent-platform-auto-pr bot commented Nov 11, 2025

Modified Jobs

Uh oh!

agent-platform-auto-pr bot commented Nov 11, 2025

Info

Uh oh!

cit-pr-commenter bot commented Nov 11, 2025

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Change MacOS jobs to use sonoma runners #42926

Are you sure you want to change the base?

Change MacOS jobs to use sonoma runners #42926

Conversation

alopezz commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

Uh oh!

agent-platform-auto-pr bot commented Nov 11, 2025

Gitlab CI Configuration Changes

Modified Jobs

Changes Summary

Uh oh!

agent-platform-auto-pr bot commented Nov 11, 2025

Static quality checks

Info

Uh oh!

cit-pr-commenter bot commented Nov 11, 2025

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alopezz commented Nov 11, 2025 •

edited

Loading