crowdstrike: provide alternate endpoint to query host data for GovCloud CIDs (elastic#16007)

This change adds a toggle in the host data stream manifest
to switch between two different device endpoints to query host data.

The "/devices/combined/devices/v1" endpoint is not supported in GovCloud CIDs,
which was introduced in this PR[1]. This impacted GovCloud users to encounter errors
when enabling the host data stream.

This change provides an option for GovCloud users to query an alternate endpoint,
"/devices/entities/devices/v2".

If the GovCloud flag is enabled, the CEL program uses "/devices/queries/devices/v1"
endpoint else it uses "/devices/combined/devices/v1" endpoint.

The implementation for querying "/devices/queries/devices/v1" endpoint is taken
from the previous codebase before combined devices endpoint was introduced
with a few fixes.

[1] elastic#15419

Author: navnit-elastic

packages/crowdstrike/_dev/build/docs/README.md

@@ -20,7 +20,9 @@ The [CrowdStrike](https://www.crowdstrike.com/) integration allows you to easily
 
 - `alert` dataset: It is typically used to retrieve detailed information about unified alerts generated by the CrowdStrike Falcon platform, via Falcon Intelligence Alert API - `/alerts/combined/alerts/v1`.
 
-- `host` dataset: It retrieves all the hosts/devices in your environment providing information such as device metadata, configuration, and status generated by the CrowdStrike Falcon platform, via Falcon Intelligence Host/Device API - `/devices/combined/devices/v1`. It is more focused to provide the management and monitoring information of devices such as login details, status, policies, configuration etc.
+- `host` dataset: It retrieves all the hosts/devices in your environment providing information such as device metadata, configuration, and status generated by the CrowdStrike Falcon platform, via Falcon Intelligence Host/Device API - `/devices/combined/devices/v1`. For GovCloud CIDs it uses `/devices/queries/devices/v1` and `/devices/entities/devices/v2` endpoints. It is more focused to provide the management and monitoring information of devices such as login details, status, policies, configuration etc.
+
+    >  NOTE: GovCloud CID users must enable the GovCloud option in the integration configuration to query the `/devices/queries/devices/v1` endpoint instead of the unsupported `/devices/combined/devices/v1` endpoint.
 
 - `vulnerability` dataset: It retrieves all the vulnerabilities in your environment, providing information such as severity, status, confidence levels, remediation guidance, and affected hosts, as detected by the CrowdStrike Falcon platform, via the Falcon Spotlight Vulnerability API - `/spotlight/combined/vulnerabilities/v1`.
 

packages/crowdstrike/_dev/deploy/docker/files/config-host.yml

@@ -8,6 +8,7 @@ rules:
             - 'application/json'
         body: |
           {"access_token":"xxxx","expires_in":3600,"token_type":"Bearer","refresh_token":"yyyy"}
+  # test cases for /devices/combined/devices/v1 endpoint 
   - path: /devices/combined/devices/v1
     methods: ['GET']
     query_params:
@@ -423,3 +424,346 @@ rules:
             ]
           }
           `}}
+
+  # test cases for /devices/queries/devices/v1 endpoint
+  - path: /devices/queries/devices/v1
+    methods: ['GET']
+    query_params:
+      offset: 0
+      limit: 1
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - application/json
+        body: |
+          {{ minify_json `
+          {
+            "meta": {
+              "query_time": 0.017724698,
+              "pagination": {
+                "offset": 0,
+                "limit": 1,
+                "total": 3
+              },
+              "writes": {
+                "resources_affected": 0
+              },
+              "powered_by": "detectsapi",
+              "trace_id": "a21557a2-abd0-4363-9293-727c38084b3b"
+            },
+            "resources": [
+              "abc"
+            ]
+          }
+          `}}
+  - path: /devices/queries/devices/v1
+    methods: ['GET']
+    query_params:
+      offset: 1
+      limit: 1
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - application/json
+        body: |
+          {{ minify_json `
+          {
+            "meta": {
+              "query_time": 0.017724698,
+              "pagination": {
+                "offset": 1,
+                "limit": 1,
+                "total": 3
+              },
+              "writes": {
+                "resources_affected": 0
+              },
+              "powered_by": "detectsapi",
+              "trace_id": "b21557a2-abd0-4363-9293-727c384b3b"
+            },
+            "resources": [
+              "def"
+            ]
+          }
+          `}}
+  - path: /devices/queries/devices/v1
+    methods: ['GET']
+    query_params:
+      offset: 2
+      limit: 1
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - application/json
+        body: |
+          {{ minify_json `
+          {
+            "meta": {
+              "query_time": 0.017725698,
+              "pagination": {
+                "offset": 2,
+                "limit": 1,
+                "total": 2
+              },
+              "writes": {
+                "resources_affected": 0
+              },
+              "powered_by": "detectsapi",
+              "trace_id": "a31557a2-abd0-4363-9293-727c384b3b"
+            },
+            "resources": []
+          }
+          `}}
+  - path: /devices/entities/devices/v2
+    methods: ['POST']
+    request_body: /.*"abc"*/
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - application/json
+        body: |-
+          {
+           "resources":[
+              {
+                 "agent_load_flags":"0",
+                 "agent_local_time":"2023-11-07T04:51:16.678Z",
+                 "agent_version":"7.05.17603.0",
+                 "bios_manufacturer":"ABCInc.",
+                 "bios_version":"2020.0.1.0.0(iBridge:22.11.000.0.0,0)",
+                 "chassis_type":"9",
+                 "chassis_type_desc":"Laptop",
+                 "cid":"92012896127c4948236ba7601b886b0",
+                 "config_id_base":"6594763",
+                 "config_id_build":"1703",
+                 "config_id_platform":"4",
+                 "connection_ip":"81.2.69.192",
+                 "cpu_signature":"460517",
+                 "device_id":"3114433dbce478ca48d9a828b9b34be",
+                 "device_policies":{
+                    "device_control":{
+                       "applied":true,
+                       "applied_date":"2023-06-20T08:45:26.341093915Z",
+                       "assigned_date":"2023-06-20T08:43:47.736146738Z",
+                       "policy_id":"2f88daf0177f467dae69262a5ce71775",
+                       "policy_type":"device-control"
+                    },
+                    "firewall":{
+                       "applied":true,
+                       "applied_date":"2023-09-11T10:33:44.174488832Z",
+                       "assigned_date":"2023-09-11T10:32:47.853976945Z",
+                       "policy_id":"1ee301f7e3e24e96ad6a23c73aaac1e3",
+                       "policy_type":"firewall",
+                       "rule_set_id":"1ee301f7e3e24e96ad6a23c73aaac1e3"
+                    },
+                    "global_config":{
+                       "applied":true,
+                       "applied_date":"2023-11-07T04:52:59.515775409Z",
+                       "assigned_date":"2023-11-07T04:51:18.94671252Z",
+                       "policy_id":"7e3078b60976486cac5dc998808d9135",
+                       "policy_type":"globalconfig",
+                       "settings_hash":"f01def74"
+                    },
+                    "prevention":{
+                       "applied":true,
+                       "applied_date":"2023-06-08T10:04:47.643357971Z",
+                       "assigned_date":"2023-06-08T10:03:49.505180252Z",
+                       "policy_id":"1024fac1b279424fa7300b8ac2d56be5",
+                       "policy_type":"prevention",
+                       "rule_groups":[],
+                       "settings_hash":"f7a54ca1"
+                    },
+                    "remote_response":{
+                       "applied":true,
+                       "applied_date":"2023-06-08T10:04:47.01735027Z",
+                       "assigned_date":"2023-06-08T10:03:49.505163572Z",
+                       "policy_id":"dabb4def99034f11b9b3d52271584c9f",
+                       "policy_type":"remote-response",
+                       "settings_hash":"8a548e5e"
+                    },
+                    "sensor_update":{
+                       "applied":true,
+                       "applied_date":"2023-11-07T04:52:59.659583066Z",
+                       "assigned_date":"2023-11-07T04:47:43.342175341Z",
+                       "policy_id":"64bfa2bbcd4e46da92a66b107933da11",
+                       "policy_type":"sensor-update",
+                       "settings_hash":"tagged|18;101",
+                       "uninstall_protection":"ENABLED"
+                    }
+                 },
+                 "external_ip":"81.2.69.192",
+                 "first_seen":"2023-06-08T10:00:19Z",
+                 "group_hash":"b607fe25348a46d421ff46e19741b0caf5bbc70bb6da1637f56e97b4e1454d77",
+                 "groups":[
+                    "182388a8dbea4c44b5e019cfd32c2695"
+                 ],
+                 "hostname":"CLM101-131.local",
+                 "kernel_version":"22.6.0",
+                 "last_seen":"2023-11-07T10:25:24Z",
+                 "local_ip":"81.2.69.142",
+                 "mac_address":"14-7d-da-ad-ac-71",
+                 "machine_domain":"SYS",
+                 "major_version":"22",
+                 "meta":{
+                    "version":"6002",
+                    "version_string":"7:43570272778"
+                 },
+                 "minor_version":"6",
+                 "modified_timestamp":"2023-11-07T10:26:53Z",
+                 "os_build":"22G120",
+                 "os_version":"Ventura(13)",
+                 "platform_id":"1",
+                 "platform_name":"Mac",
+                 "policies":[
+                    {
+                       "applied":true,
+                       "applied_date":"2023-06-08T10:04:47.643357971Z",
+                       "assigned_date":"2023-06-08T10:03:49.505180252Z",
+                       "policy_id":"1024fac1b279424fa7300b8ac2d56be5",
+                       "policy_type":"prevention",
+                       "rule_groups":[],
+                       "settings_hash":"f7a54ca1"
+                    }
+                 ],
+                 "product_type_desc":"Workstation",
+                 "provision_status":"Provisioned",
+                 "reduced_functionality_mode":"no",
+                 "serial_number":"FVFDH73HMNHX",
+                 "site_name":"Default-First-Site-Name",
+                 "status":"normal",
+                 "system_manufacturer":"ABCInc.",
+                 "system_product_name":"Air,1",
+                 "tags":[
+                    "tags"
+                 ]
+              }
+           ]
+          }
+  - path: /devices/entities/devices/v2
+    methods: ['POST']
+    request_body: /.*"def"*/
+    responses:
+      - status_code: 200
+        headers:
+          Content-Type:
+            - application/json
+        body: |-
+          {
+           "resources":[
+              {
+                 "agent_load_flags":"0",
+                 "agent_local_time":"2023-11-07T04:51:16.678Z",
+                 "agent_version":"7.05.17603.0",
+                 "bios_manufacturer":"ABCInc.",
+                 "bios_version":"2020.0.1.0.0(iBridge:22.11.000.0.0,0)",
+                 "chassis_type":"9",
+                 "chassis_type_desc":"Laptop",
+                 "cid":"92012896127c4948236ba7601b886b0",
+                 "config_id_base":"6594763",
+                 "config_id_build":"1703",
+                 "config_id_platform":"4",
+                 "connection_ip":"81.2.69.192",
+                 "cpu_signature":"460517",
+                 "device_id":"3114433dbce478ca48d9a828b9b34be",
+                 "device_policies":{
+                    "device_control":{
+                       "applied":true,
+                       "applied_date":"2023-06-20T08:45:26.341093915Z",
+                       "assigned_date":"2023-06-20T08:43:47.736146738Z",
+                       "policy_id":"3f88daf0177f467dae69262a5ce71775",
+                       "policy_type":"device-control"
+                    },
+                    "firewall":{
+                       "applied":true,
+                       "applied_date":"2023-09-11T10:33:44.174488832Z",
+                       "assigned_date":"2023-09-11T10:32:47.853976945Z",
+                       "policy_id":"1ee301f7e3e24e96ad6a23c73aaac1e3",
+                       "policy_type":"firewall",
+                       "rule_set_id":"1ee301f7e3e24e96ad6a23c73aaac1e3"
+                    },
+                    "global_config":{
+                       "applied":true,
+                       "applied_date":"2023-11-07T04:52:59.515775409Z",
+                       "assigned_date":"2023-11-07T04:51:18.94671252Z",
+                       "policy_id":"7e3078b60976486cac5dc998808d9135",
+                       "policy_type":"globalconfig",
+                       "settings_hash":"f01def74"
+                    },
+                    "prevention":{
+                       "applied":true,
+                       "applied_date":"2023-06-08T10:04:47.643357971Z",
+                       "assigned_date":"2023-06-08T10:03:49.505180252Z",
+                       "policy_id":"1024fac1b279424fa7300b8ac2d56be5",
+                       "policy_type":"prevention",
+                       "rule_groups":[],
+                       "settings_hash":"f7a54ca1"
+                    },
+                    "remote_response":{
+                       "applied":true,
+                       "applied_date":"2023-06-08T10:04:47.01735027Z",
+                       "assigned_date":"2023-06-08T10:03:49.505163572Z",
+                       "policy_id":"dabb4def99034f11b9b3d52271584c9f",
+                       "policy_type":"remote-response",
+                       "settings_hash":"8a548e5e"
+                    },
+                    "sensor_update":{
+                       "applied":true,
+                       "applied_date":"2023-11-09T04:52:59.659583066Z",
+                       "assigned_date":"2023-11-09T04:47:43.342175341Z",
+                       "policy_id":"74bfa2bbcd4e46da92a66b107933da11",
+                       "policy_type":"sensor-update",
+                       "settings_hash":"tagged|18;101",
+                       "uninstall_protection":"ENABLED"
+                    }
+                 },
+                 "external_ip":"81.2.69.192",
+                 "first_seen":"2023-06-09T10:00:19Z",
+                 "group_hash":"c607fe25348a46d421ff46e19741b0caf5bbc70bb6da1637f56e97b4e1454d77",
+                 "groups":[
+                    "882388a8dbea4c44b5e019cfd32c2695"
+                 ],
+                 "hostname":"CLM101-141.local",
+                 "kernel_version":"22.6.0",
+                 "last_seen":"2023-11-09T10:25:24Z",
+                 "local_ip":"81.2.69.142",
+                 "mac_address":"14-7d-da-ad-ac-71",
+                 "machine_domain":"SYS",
+                 "major_version":"22",
+                 "meta":{
+                    "version":"6002",
+                    "version_string":"7:43570272778"
+                 },
+                 "minor_version":"6",
+                 "modified_timestamp":"2023-11-09T10:26:53Z",
+                 "os_build":"22G120",
+                 "os_version":"Ventura(13)",
+                 "platform_id":"1",
+                 "platform_name":"Mac",
+                 "policies":[
+                    {
+                       "applied":true,
+                       "applied_date":"2023-06-09T10:04:47.643357971Z",
+                       "assigned_date":"2023-06-09T10:03:49.505180252Z",
+                       "policy_id":"2024fac1b279424fa7300b8ac2d56be5",
+                       "policy_type":"prevention",
+                       "rule_groups":[],
+                       "settings_hash":"m7a54ca1"
+                    }
+                 ],
+                 "product_type_desc":"Workstation",
+                 "provision_status":"Provisioned",
+                 "reduced_functionality_mode":"no",
+                 "serial_number":"FVVDH73HMNHX",
+                 "site_name":"Default-First-Site-Name",
+                 "status":"normal",
+                 "system_manufacturer":"ABCInc.",
+                 "system_product_name":"Air,1",
+                 "tags":[
+                    "tags"
+                 ]
+              }
+           ]
+          }

packages/crowdstrike/changelog.yml

@@ -1,4 +1,11 @@
 # newer versions go on top
+- version: "2.10.0"
+  changes:
+    - description: >-
+        Provide an alternate endpoint to query host data for GovCloud CIDs.
+        The GovCloud CIDs must enable the `GovCloud` flag in the integration configuration to ensure the correct endpoint is used.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/16007
 - version: "2.9.0"
   changes:
     - description: Support handling FDR documents that encode numbers as strings.