Doc: Add elastic-integration filter tutorial

docs/docset.yml

@@ -24,6 +24,7 @@ subs:
   es-serverless:   "Elasticsearch Serverless"
   agent:   "Elastic Agent"
   fleet:   "Fleet"
+  fleet-server:   "Fleet Server"
   integrations:   "Integrations"
   stack:   "Elastic Stack"
   xpack:   "X-Pack"

docs/reference/connecting-to-serverless.md

@@ -22,7 +22,7 @@ Set the value to port :443 instead.
 
 ::::
 
-## Communication between {{ls}} {{es-serverless}} [connecting-to-elasticsearch-serverless]
+## Communication between {{ls}} and {{es-serverless}} [connecting-to-elasticsearch-serverless]
 
 [{{es-serverless}}](docs-content://solutions/search/serverless-elasticsearch-get-started.md) simplifies safe, secure communication between {{ls}} and {{es}}.
 When you configure the Elasticsearch output plugin to use [`cloud_id`](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-cloud_id) and an [`api_key`](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-api_key), no additional SSL configuration is needed.

docs/reference/ea-integration-tutorial.md

@@ -0,0 +1,169 @@
+---
+navigation_title: Tutorial to extend Elastic Integrations
+applies_to:
+  stack: ga 
+  serverless: ga
+products:
+  - id: elasticsearch
+  - id: cloud-hosted
+  - id: cloud-enterprise
+  - id: cloud-kubernetes
+---
+
+# Tutorial: Extend Elastic {{integrations}} with the {{ls}} elastic_integration filter[ea-integrations-tutorial]
+
+You can use {{ls}} to transform events collected by {{agent}}s and paired with an [Elastic integration](integration-docs://reference/index.md).
+
+You get the benefits of Elastic integrations--such as the simplicity of ingesting data from a wide variety of data sources and ensuring compliance with the [Elastic Common Schema (ECS)](ecs://reference/index.md)--combined with the extra processing power of {{ls}}. 
+
+This new functionality is made possible by the [elastic_integration filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md) plugin.
+When you include the `elastic_integration` filter in your configuration, {{ls}} reads certain field values generated by the {{agent}}, 
+and uses them to apply the transformations from Elastic integrations. 
+This ability allows you to further process events in the Logstash pipeline before sending them to their configured destinations.
+
+This tutorial walks you through adding the [Crowdstrike integration](integration-docs://reference/crowdstrike.md), and sending the data to {{ech}}, {{serverless-full}}, or self-managed {{es}}.
+
+
+## Prerequisites [ea-integrations-prereqs]
+
+You need:
+
+* A working {{ech}} or {{serverless-full}} deployment, or self-managed {{es}} cluster
+* A {{ls}} instance
+* {{fleet-server}}
+* An [{{agent}} installed](docs-content://reference/fleet/install-elastic-agents.md) on the hosts you want to collect data from, and configured to [send output to {{ls}}](docs-content://reference/fleet/logstash-output.md)
+* An active Elastic [subscription](https://www.elastic.co/subscriptions).
+* A user configured with the [minimum required privileges](logstash-docs-md://lsr/plugins-filters-elastic_integration.md#plugins-filters-elastic_integration-minimum_required_privileges).
+
+:::{note}
+Even though the focus of this tutorial is {{fleet}}-managed agents, you can use the `elastic_integration` filter and this 
+general approach with [self-managed agents](docs-content://reference/fleet/install-standalone-elastic-agent.md). 
+:::
+
+## Process overview [ea-integrations-process-overview]
+
+* [Configure {{fleet}} to send data from {{agent}} to {{ls}}](#ea-integrations-fleet)
+* [Create an {{agent}} policy with the necessary integration](#ea-integrations-create-policy)
+* [Configure {{ls}} to use the `elastic_integration` filter plugin](#ea-integrations-pipeline)
+
+
+## Configure {{fleet}} to send data from {{agent}} to {{ls}} [ea-integrations-fleet]
+
+1. For {{fleet}}-managed agents, go to {{kib}} and navigate to **Fleet > Settings**.
+2. Create a new output and specify {{ls}} as the output type.
+3. Add the {{ls}} hosts (domain or IP address/s) that the {{agent}} should send data to.
+4. Add the client SSL certificate and the Client SSL certificate key to the configuration.
+5. Click **Save and apply settings** in the bottom right-hand corner of the page.
+
+
+## Create an {{agent}} policy with the necessary integrations [ea-integrations-create-policy]
+
+1. In {{kib}} navigate to **Fleet > Agent** policies, and select **Create agent policy**.
+2. Give this policy a name, and then select **Advanced options**.
+3. Change the **Output for integrations** setting to the {{ls}} output you created.
+4. Click **Create agent policy**. 
+5. Select the policy name, and click **Add integration**.
+
+    This step takes you to the {{integrations}} browser, where you can select an integration that has everything necessary to _integrate_ the data source with your other data in the {{stack}}.
+
+     We'll use Crowdstrike as our example in this tutorial.
+
+6. On the **Crowdstrike** integration overview page, click **Add Crowdstrike** to configure the integration.
+7. Configure the integration to collect the data you need.
+On step 2 at the bottom of the page (**Where to add this integration?**), make sure that the “Existing hosts” option
+is selected and the Agent policy selected is the {{ls}} policy that we created for our {{ls}} output. 
+This policy should be selected by default.
+8. Click **Save and continue**.
+
+    You have the option to add the {{agent}} to your hosts. If you haven't already, [install the {{agent}}](docs-content://reference/fleet/install-elastic-agents.md) on the host where you want to collect data.
+
+
+## Configure {{ls}} to use the `elastic_integration` filter plugin [ea-integrations-pipeline]
+
+1. Create a new [{{ls}} pipeline](/reference/creating-logstash-pipeline.md).
+2. Be sure to include these plugins:
+    * [elastic_agent input](logstash-docs-md://lsr/plugins-inputs-elastic_agent.md) 
+    * [`elastic_integration` filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md) 
+    * [`elasticsearch` output](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md) 
+
+Note that every event sent from the {{agent}} to {{ls}} contains specific meta-fields. 
+{{ls}} expects events to contain a top-level `data_stream` field with `type`, `dataset`, and `namespace` sub-fields.
+
+{{ls}} uses this information and its connection to {{es}} to determine which integrations to apply to the event before sending the event to its destination output.
+{{ls}} frequently synchronizes with {{es}} to ensure that it has the most recent versions of the enabled integrations.
+
+
+### Sample configuration: output to Elastic Cloud [ea-integrations-ech-sample]
+
+This sample illustrates using the `elastic_agent` input and the `elastic_integration` filter for processing in {{ls}}, and then sending the output to {{ech}} or {{serverless-full}}.
+
+Check out the [`elastic_integration` filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md) for the full list of configuration options.
+
+```txt
+input {
+  elastic_agent { port => 5055 }
+}
+
+filter {
+  elastic_integration {
+    cloud_id => "your-cloud:id"
+    api_key => "your-api-key"
+  }
+}
+
+output {
+  stdout {}
+  elasticsearch {
+    cloud_id => "your-cloud:id"
+    api_key => "your-api-key"
+  }
+}
+```
+
+All processing occurs in {{ls}} before events are forwarded to {{ech}} or {{serverless-full}}.
+
+
+### Sample configuration: output to self-managed {{es}} [ea-integrations-es-sample]
+
+This sample illustrates using the `elastic_agent` input and the `elastic_integration` filter for processing in {{ls}}, and then sending the output to {{es}}.
+
+Check out the [`elastic_integration` filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md) for the full list of configuration options.
+
+Check out [minimum required privileges](logstash-docs-md://lsr/plugins-filters-elastic_integration.md#plugins-filters-elastic_integration-minimum_required_privileges) for more info. 
+
+```txt
+input {
+  elastic_agent { port => 5055 }
+}
+
+filter {
+  elastic_integration {
+    hosts => ["{es-host}:9200"]
+    ssl_enabled => true
+    ssl_certificate_authorities => "/usr/share/logstash/config/certs/ca-cert.pem"
+    username => "elastic" 
+    password => "changeme"
+  }
+}
+
+output {
+  stdout {
+    codec => rubydebug # debugs datastream inputs
+  }
+  ## add elasticsearch
+  elasticsearch {
+    hosts => ["{es-host}:9200"]
+    user => "elastic"
+    password => "changeme"
+    ssl_certificate_authorities => "/usr/share/logstash/config/certs/ca-cert.pem"
+  }
+}
+```
+
+:::{note}
+The user credentials that you specify in the `elastic_integration` filter must have sufficient privileges to get information about {{es}} and the integrations that you are using.
+:::
+
+If your {{agent}} and {{ls}} pipelines are configured correctly, events go to {{ls}} for processing before {{ls}} forwards them on to {{es}}.
+
+If you encounter problems, check out the [Troubleshooting](logstash-docs-md://lsr/plugins-filters-elastic_integration.md#plugins-filters-elastic_integration-troubleshooting) section in the `elastic_integration` filter docs.

docs/reference/toc.yml

@@ -59,6 +59,8 @@ toc:
       - file: logstash-centralized-pipeline-management.md
       - file: configuring-centralized-pipelines.md
   - file: using-logstash-with-elastic-integrations.md
+    children:
+      - file: ea-integration-tutorial.md
   # TO DO: Not migrated
   # - file: working-with-logstash-modules.md
   #   children:

docs/reference/using-logstash-with-elastic-integrations.md

@@ -1,6 +1,16 @@
 ---
 mapped_pages:
   - https://www.elastic.co/guide/en/logstash/current/ea-integrations.html
+
+navigation_title: Using Logstash with Elastic integrations
+applies_to:
+  stack: ga 
+  serverless: ga
+products:
+  - id: elasticsearch
+  - id: cloud-hosted
+  - id: cloud-enterprise
+  - id: cloud-kubernetes
 ---
 
 # Using Logstash with Elastic integrations [ea-integrations]