diff --git a/docs/docset.yml b/docs/docset.yml index 5d701c7660f..ae74102ddcb 100644 --- a/docs/docset.yml +++ b/docs/docset.yml @@ -24,6 +24,7 @@ subs: es-serverless: "Elasticsearch Serverless" agent: "Elastic Agent" fleet: "Fleet" + fleet-server: "Fleet Server" integrations: "Integrations" stack: "Elastic Stack" xpack: "X-Pack" diff --git a/docs/reference/connecting-to-serverless.md b/docs/reference/connecting-to-serverless.md index 8e381b63b45..aeccb03cccd 100644 --- a/docs/reference/connecting-to-serverless.md +++ b/docs/reference/connecting-to-serverless.md @@ -22,7 +22,7 @@ Set the value to port :443 instead. :::: -## Communication between {{ls}} {{es-serverless}} [connecting-to-elasticsearch-serverless] +## Communication between {{ls}} and {{es-serverless}} [connecting-to-elasticsearch-serverless] [{{es-serverless}}](docs-content://solutions/search/serverless-elasticsearch-get-started.md) simplifies safe, secure communication between {{ls}} and {{es}}. When you configure the Elasticsearch output plugin to use [`cloud_id`](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-cloud_id) and an [`api_key`](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md#plugins-outputs-elasticsearch-api_key), no additional SSL configuration is needed. diff --git a/docs/reference/ea-integration-tutorial.md b/docs/reference/ea-integration-tutorial.md new file mode 100644 index 00000000000..dcd0ef2e858 --- /dev/null +++ b/docs/reference/ea-integration-tutorial.md @@ -0,0 +1,169 @@ +--- +navigation_title: Tutorial to extend Elastic Integrations +applies_to: + stack: ga + serverless: ga +products: + - id: elasticsearch + - id: cloud-hosted + - id: cloud-enterprise + - id: cloud-kubernetes +--- + +# Tutorial: Extend Elastic {{integrations}} with the {{ls}} elastic_integration filter[ea-integrations-tutorial] + +You can use {{ls}} to transform events collected by {{agent}}s and paired with an [Elastic integration](integration-docs://reference/index.md). + +You get the benefits of Elastic integrations--such as the simplicity of ingesting data from a wide variety of data sources and ensuring compliance with the [Elastic Common Schema (ECS)](ecs://reference/index.md)--combined with the extra processing power of {{ls}}. + +This new functionality is made possible by the [elastic_integration filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md) plugin. +When you include the `elastic_integration` filter in your configuration, {{ls}} reads certain field values generated by the {{agent}}, +and uses them to apply the transformations from Elastic integrations. +This ability allows you to further process events in the Logstash pipeline before sending them to their configured destinations. + +This tutorial walks you through adding the [Crowdstrike integration](integration-docs://reference/crowdstrike.md), and sending the data to {{ech}}, {{serverless-full}}, or self-managed {{es}}. + + +## Prerequisites [ea-integrations-prereqs] + +You need: + +* A working {{ech}} or {{serverless-full}} deployment, or self-managed {{es}} cluster +* A {{ls}} instance +* {{fleet-server}} +* An [{{agent}} installed](docs-content://reference/fleet/install-elastic-agents.md) on the hosts you want to collect data from, and configured to [send output to {{ls}}](docs-content://reference/fleet/logstash-output.md) +* An active Elastic [subscription](https://www.elastic.co/subscriptions). +* A user configured with the [minimum required privileges](logstash-docs-md://lsr/plugins-filters-elastic_integration.md#plugins-filters-elastic_integration-minimum_required_privileges). + +:::{note} +Even though the focus of this tutorial is {{fleet}}-managed agents, you can use the `elastic_integration` filter and this +general approach with [self-managed agents](docs-content://reference/fleet/install-standalone-elastic-agent.md). +::: + +## Process overview [ea-integrations-process-overview] + +* [Configure {{fleet}} to send data from {{agent}} to {{ls}}](#ea-integrations-fleet) +* [Create an {{agent}} policy with the necessary integration](#ea-integrations-create-policy) +* [Configure {{ls}} to use the `elastic_integration` filter plugin](#ea-integrations-pipeline) + + +## Configure {{fleet}} to send data from {{agent}} to {{ls}} [ea-integrations-fleet] + +1. For {{fleet}}-managed agents, go to {{kib}} and navigate to **Fleet > Settings**. +2. Create a new output and specify {{ls}} as the output type. +3. Add the {{ls}} hosts (domain or IP address/s) that the {{agent}} should send data to. +4. Add the client SSL certificate and the Client SSL certificate key to the configuration. +5. Click **Save and apply settings** in the bottom right-hand corner of the page. + + +## Create an {{agent}} policy with the necessary integrations [ea-integrations-create-policy] + +1. In {{kib}} navigate to **Fleet > Agent** policies, and select **Create agent policy**. +2. Give this policy a name, and then select **Advanced options**. +3. Change the **Output for integrations** setting to the {{ls}} output you created. +4. Click **Create agent policy**. +5. Select the policy name, and click **Add integration**. + + This step takes you to the {{integrations}} browser, where you can select an integration that has everything necessary to _integrate_ the data source with your other data in the {{stack}}. + + We'll use Crowdstrike as our example in this tutorial. + +6. On the **Crowdstrike** integration overview page, click **Add Crowdstrike** to configure the integration. +7. Configure the integration to collect the data you need. +On step 2 at the bottom of the page (**Where to add this integration?**), make sure that the “Existing hosts” option +is selected and the Agent policy selected is the {{ls}} policy that we created for our {{ls}} output. +This policy should be selected by default. +8. Click **Save and continue**. + + You have the option to add the {{agent}} to your hosts. If you haven't already, [install the {{agent}}](docs-content://reference/fleet/install-elastic-agents.md) on the host where you want to collect data. + + +## Configure {{ls}} to use the `elastic_integration` filter plugin [ea-integrations-pipeline] + +1. Create a new [{{ls}} pipeline](/reference/creating-logstash-pipeline.md). +2. Be sure to include these plugins: + * [elastic_agent input](logstash-docs-md://lsr/plugins-inputs-elastic_agent.md) + * [`elastic_integration` filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md) + * [`elasticsearch` output](logstash-docs-md://lsr/plugins-outputs-elasticsearch.md) + +Note that every event sent from the {{agent}} to {{ls}} contains specific meta-fields. +{{ls}} expects events to contain a top-level `data_stream` field with `type`, `dataset`, and `namespace` sub-fields. + +{{ls}} uses this information and its connection to {{es}} to determine which integrations to apply to the event before sending the event to its destination output. +{{ls}} frequently synchronizes with {{es}} to ensure that it has the most recent versions of the enabled integrations. + + +### Sample configuration: output to Elastic Cloud [ea-integrations-ech-sample] + +This sample illustrates using the `elastic_agent` input and the `elastic_integration` filter for processing in {{ls}}, and then sending the output to {{ech}} or {{serverless-full}}. + +Check out the [`elastic_integration` filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md) for the full list of configuration options. + +```txt +input { + elastic_agent { port => 5055 } +} + +filter { + elastic_integration { + cloud_id => "your-cloud:id" + api_key => "your-api-key" + } +} + +output { + stdout {} + elasticsearch { + cloud_id => "your-cloud:id" + api_key => "your-api-key" + } +} +``` + +All processing occurs in {{ls}} before events are forwarded to {{ech}} or {{serverless-full}}. + + +### Sample configuration: output to self-managed {{es}} [ea-integrations-es-sample] + +This sample illustrates using the `elastic_agent` input and the `elastic_integration` filter for processing in {{ls}}, and then sending the output to {{es}}. + +Check out the [`elastic_integration` filter](logstash-docs-md://lsr/plugins-filters-elastic_integration.md) for the full list of configuration options. + +Check out [minimum required privileges](logstash-docs-md://lsr/plugins-filters-elastic_integration.md#plugins-filters-elastic_integration-minimum_required_privileges) for more info. + +```txt +input { + elastic_agent { port => 5055 } +} + +filter { + elastic_integration { + hosts => ["{es-host}:9200"] + ssl_enabled => true + ssl_certificate_authorities => "/usr/share/logstash/config/certs/ca-cert.pem" + username => "elastic" + password => "changeme" + } +} + +output { + stdout { + codec => rubydebug # debugs datastream inputs + } + ## add elasticsearch + elasticsearch { + hosts => ["{es-host}:9200"] + user => "elastic" + password => "changeme" + ssl_certificate_authorities => "/usr/share/logstash/config/certs/ca-cert.pem" + } +} +``` + +:::{note} +The user credentials that you specify in the `elastic_integration` filter must have sufficient privileges to get information about {{es}} and the integrations that you are using. +::: + +If your {{agent}} and {{ls}} pipelines are configured correctly, events go to {{ls}} for processing before {{ls}} forwards them on to {{es}}. + +If you encounter problems, check out the [Troubleshooting](logstash-docs-md://lsr/plugins-filters-elastic_integration.md#plugins-filters-elastic_integration-troubleshooting) section in the `elastic_integration` filter docs. diff --git a/docs/reference/toc.yml b/docs/reference/toc.yml index 7681c3e4334..e27cf354956 100644 --- a/docs/reference/toc.yml +++ b/docs/reference/toc.yml @@ -59,6 +59,8 @@ toc: - file: logstash-centralized-pipeline-management.md - file: configuring-centralized-pipelines.md - file: using-logstash-with-elastic-integrations.md + children: + - file: ea-integration-tutorial.md # TO DO: Not migrated # - file: working-with-logstash-modules.md # children: diff --git a/docs/reference/using-logstash-with-elastic-integrations.md b/docs/reference/using-logstash-with-elastic-integrations.md index bea5e8731ca..3c67d81a4af 100644 --- a/docs/reference/using-logstash-with-elastic-integrations.md +++ b/docs/reference/using-logstash-with-elastic-integrations.md @@ -1,6 +1,16 @@ --- mapped_pages: - https://www.elastic.co/guide/en/logstash/current/ea-integrations.html + +navigation_title: Using Logstash with Elastic integrations +applies_to: + stack: ga + serverless: ga +products: + - id: elasticsearch + - id: cloud-hosted + - id: cloud-enterprise + - id: cloud-kubernetes --- # Using Logstash with Elastic integrations [ea-integrations]