Merge pull request #30 from shazChaudhry/v7.0.0

Upgraded to V7.x

Author: shazChaudhry

README.md

@@ -43,17 +43,17 @@ For the full list of free features that are included in the basic license, see:
 
 # Get docker compose files
 You will need these files to deploy Eleasticsearch, Logstash, Kibana, and Beats. So, first SSH in to the master node of the Docker Swarm cluster allocated to running Elastic Stack and clone this repo by following these commands:
-  * `alias git='docker run -it --rm --name git -v $PWD:/git -w /git alpine/git'` _(This alias is only required if git is *not* already installed on your machine. This alias will allow you to clone the repo using a git container)_
+  * `alias git='docker run -it --rm --name git -u $(id -u ${USER}):$(id -g ${USER}) -v $PWD:/git -w /git alpine/git'` _(This alias is only required if git is *not* already installed on your machine. This alias will allow you to clone the repo using a git container)_
   * `git version`
   * `git clone https://github.com/shazChaudhry/docker-elastic.git`
-  * `sudo chown -R $USER:$USER docker-elastic`
   * `cd docker-elastic`
 
 # Deploy Elastic Stack
 * SSH in to the master node of the Docker Swarm cluster allocated to running Elastic Stack. Deploy Elastic stack by running the following commands:
-  * `export ELASTIC_VERSION=6.7.0`
+  * `export ELASTIC_VERSION=7.1.1`
   * `export ELASTICSEARCH_USERNAME=elastic`
   * `export ELASTICSEARCH_PASSWORD=changeme`
+  * `export INITIAL_MASTER_NODES=node1` _(See Important discovery and cluster formation settings: https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-settings.html#initial_master_nodes)_
   * `export ELASTICSEARCH_HOST=node1` _(node1 is default value if you are creating VirtualBox with the provided Vagrantfile. Otherwise, change this value to one of your VMs in the swarm cluster)_
   * `docker network create --driver overlay --attachable elastic`
   * `docker stack deploy --compose-file docker-compose.yml elastic` _(Assuming you have only two VMs, this will deploy a reverse proxy, logstash, Kibana and 2x Elasticsearch instances in Master / data nodes configuration. Please note that Elasticsearch is configured to start as a global service which means elasticsearch data nodes will be scalled out automatically as soon as new VMs are added to the Swarm cluster. Here is an explaination on various Elasticsearch cluster nodes: https://discuss.elastic.co/t/node-types-in-an-elasticsearch-cluster/25488)_
@@ -67,7 +67,7 @@ You will need these files to deploy Eleasticsearch, Logstash, Kibana, and Beats.
 SSH in to the master node of the Docker Swarm cluster allocated to running containerized custom applicatins and beats. Clone this repo and change directory as per the instructions above.
 
 Execute the following commands to deploy filebeat and metricbeat:
-  * `export ELASTIC_VERSION=6.7.0`
+  * `export ELASTIC_VERSION=7.1.1`
   * `export ELASTICSEARCH_USERNAME=elastic`
   * `export ELASTICSEARCH_PASSWORD=changeme`
   * `export ELASTICSEARCH_HOST=node1` _(node1 is default value if you are creating VirtualBox with the provided Vagrantfile. Otherwise, change this value to your Elasticsearch host)_

Vagrantfile

@@ -4,12 +4,12 @@
 $docker_swarm_init = <<SCRIPT
 docker swarm init --advertise-addr 192.168.99.101 --listen-addr 192.168.99.101:2377
 docker swarm join-token --quiet worker > /vagrant/worker_token
+docker swarm join-token --quiet manager > /vagrant/manager_token
 SCRIPT
 
 Vagrant.configure("2") do |config|
-	# https://app.vagrantup.com/ralfkrause/boxes/centos7
-	config.vm.box 									= "ralfkrause/centos7"
-	config.vm.box_version 					= "1.0.0"
+	# https://app.vagrantup.com/bento/boxes/ubuntu-18.04
+	config.vm.box 									= "bento/ubuntu-18.04"
 	config.hostmanager.enabled 			= true
 	config.hostmanager.manage_host 	= true
 	config.hostmanager.manage_guest = true
@@ -21,7 +21,7 @@ Vagrant.configure("2") do |config|
 		node1.vm.network :private_network, ip: "192.168.99.101"
 		node1.vm.provider :virtualbox do |v|
 			v.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
-			v.customize ["modifyvm", :id, "--memory", 4000]
+			v.customize ["modifyvm", :id, "--memory", 3000]
 			v.customize ["modifyvm", :id, "--name", "node1"]
 		end
 		node1.vm.provision :shell, inline: $docker_swarm_init

auditbeat-README.md

@@ -1,9 +1,9 @@
 Eagerly waiting for Docker 19.06 release which will bring --privileged flag to Docker Swarm Mode https://github.com/moby/moby/issues/24862#issuecomment-451594187. support for capabilities https://github.com/moby/moby/pull/38380
 
-Until capabilities are availale in docker swarm mode, execute te following instructions on each node where auditbeat is required
+Until capabilities are available in docker swarm mode, execute the following instructions on each node where auditbeat is required
 
 Firstly, set the system variables as needed:
-- export ELASTIC_VERSION=6.7.0
+- export ELASTIC_VERSION=7.1.1
 - export ELASTICSEARCH_USERNAME=elastic
 - export ELASTICSEARCH_PASSWORD=changeme
 - export ELASTICSEARCH_HOST=node1
@@ -26,7 +26,7 @@ And than run the command below:
     --env ELASTICSEARCH_PASSWORD=${ELASTICSEARCH_PASSWORD:-changeme} \
     --env ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST:-node1} \
     --env KIBANA_HOST=${KIBANA_HOST:-node1} \
-    docker.elastic.co/beats/auditbeat:${ELASTIC_VERSION:-6.7.0} \
+    docker.elastic.co/beats/auditbeat:${ELASTIC_VERSION:-7.1.1} \
     --strict.perms=false
 ```
 

auditbeat-docker-compose.yml

@@ -6,7 +6,7 @@ version: "3.7"
 services:
 
   auditbeat:
-    image: docker.elastic.co/beats/auditbeat:${ELASTIC_VERSION:-6.7.0}
+    image: docker.elastic.co/beats/auditbeat:${ELASTIC_VERSION:-7.1.1}
     # https://github.com/docker/swarmkit/issues/1951
     hostname: "{{.Node.Hostname}}-auditbeat"
     # Need to override user so we can access the log files, and docker.sock

deployStack.sh

@@ -1,9 +1,10 @@
 #!/bin/bash
 
-export ELASTIC_VERSION=6.7.0
+export ELASTIC_VERSION=7.1.1
 export ELASTICSEARCH_USERNAME=elastic
 export ELASTICSEARCH_PASSWORD=changeme
 export ELASTICSEARCH_HOST=node1
+export INITIAL_MASTER_NODES=node1
 
 docker network create --driver overlay --attachable elastic
 docker stack deploy --compose-file docker-compose.yml elastic

docker-compose.yml

@@ -34,18 +34,17 @@ services:
       - BIND_PORTS=9200
 
   elasticsearch:
-    image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION:-6.7.0}
-    # https://github.com/docker/swarmkit/issues/1951
-    hostname: "{{.Node.Hostname}}-elasticsearch"
+    image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION:-7.1.1}
     environment:
-      - discovery.zen.ping.unicast.hosts=elasticsearch
-      - discovery.zen.minimum_master_nodes=1
+      # https://github.com/docker/swarmkit/issues/1951
+      - node.name={{.Node.Hostname}}
+      - discovery.seed_hosts=elasticsearch
+      - cluster.initial_master_nodes=${INITIAL_MASTER_NODES:-node1}
+      - cluster.name=DevOps
       - ELASTIC_PASSWORD=${ELASTICSEARCH_PASSWORD:-changeme}
       - xpack.security.enabled=true
       - xpack.monitoring.collection.enabled=true
       - xpack.license.self_generated.type=trial
-      - cluster.name=DevOps
-      - node.name={{.Node.Hostname}}-elasticsearch
       - network.host=0.0.0.0
     networks:
       - elastic
@@ -62,7 +61,7 @@ services:
         - com.df.srcPort=9200
 
   logstash:
-    image: docker.elastic.co/logstash/logstash:${ELASTIC_VERSION:-6.7.0}
+    image: docker.elastic.co/logstash/logstash:${ELASTIC_VERSION:-7.1.1}
     hostname: "{{.Node.Hostname}}-logstash"
     environment:
       - XPACK_MONITORING_ELASTICSEARCH_URL=http://elasticsearch:9200
@@ -77,7 +76,7 @@ services:
         target: /usr/share/logstash/pipeline/logstash.conf
 
   kibana:
-    image: docker.elastic.co/kibana/kibana:${ELASTIC_VERSION:-6.7.0}
+    image: docker.elastic.co/kibana/kibana:${ELASTIC_VERSION:-7.1.1}
     hostname: "{{.Node.Hostname}}-kibana"
     environment:
       - ELASTICSEARCH_URL=http://elasticsearch:9200

examples/learn_autodiscover.md

@@ -9,10 +9,10 @@ Use the provided Vagrantfile to create 3x VMs:
 
 ### Versions
 Example has been tested in following versions:
-- Elasticsearch 6.7.0
-- Kibana 6.7.0
-- Filebeat 6.7.0
-- Metricbeat 6.7.0
+- Elasticsearch 7.1.1
+- Kibana 7.1.1
+- Filebeat 7.1.1
+- Metricbeat 7.1.1
 
 ### Elastic Stack - Apache2 module
 Ensure there is nothing listening on port 80.

examples/learn_filebeat.md

@@ -2,9 +2,9 @@ The examples here are for learning purpose only and show how to start filebeat a
 
 ### Versions
 Example has been tested in following versions:
-- Elasticsearch 6.7.0
-- Kibana 6.7.0
-- Filebeat 6.7.0
+- Elasticsearch 7.1.1
+- Kibana 7.1.1
+- Filebeat 7.1.1
 
 
 ### Prerequisites
@@ -28,7 +28,7 @@ docker container run --rm \
 --network host \
 --volume filebeat:/usr/share/filebeat/data \
 --volume $PWD:/tmp \
-docker.elastic.co/beats/filebeat:6.7.0 \
+docker.elastic.co/beats/filebeat:7.1.1 \
 -e --modules=nginx --setup  -M "nginx.access.var.paths=[/tmp/nginx_logs]" \
 -E output.elasticsearch.hosts='node1:9200' \
 -E output.elasticsearch.username=elastic \
@@ -65,7 +65,7 @@ chmod go-w ./nyc_collision_filebeat.yml
 ```
 Modify the paths to `/tmp/nyc_collision_data.csv` and replace the word _prospectors_ with `inputs` in nyc_collision_filebeat.yml
 ```
-docker container run --name filebeat --rm --network host --volume filebeat:/usr/share/filebeat/data --volume $PWD:/tmp docker.elastic.co/beats/filebeat:6.7.0 \
+docker container run --name filebeat --rm --network host --volume filebeat:/usr/share/filebeat/data --volume $PWD:/tmp docker.elastic.co/beats/filebeat:7.1.1 \
 -e -c /tmp/nyc_collision_filebeat.yml \
 -E output.elasticsearch.hosts='node1:9200' \
 -E output.elasticsearch.username=elastic \

filebeat-docker-compose.yml

@@ -5,7 +5,7 @@ services:
   # How to Tune Elastic Beats Performance: A Practical Example with Batch Size, Worker Count, and More
   # https://www.elastic.co/blog/how-to-tune-elastic-beats-performance-a-practical-example-with-batch-size-worker-count-and-more?blade=tw&hulk=social
   filebeat:
-    image: docker.elastic.co/beats/filebeat:${ELASTIC_VERSION:-6.7.0}
+    image: docker.elastic.co/beats/filebeat:${ELASTIC_VERSION:-7.1.1}
     # https://github.com/docker/swarmkit/issues/1951
     hostname: "{{.Node.Hostname}}-filebeat"
     # Need to override user so we can access the log files, and docker.sock

metricbeat-docker-compose.yml

@@ -3,7 +3,7 @@ version: "3.7"
 services:
 
   metricbeat:
-    image: docker.elastic.co/beats/metricbeat:${ELASTIC_VERSION:-6.7.0}
+    image: docker.elastic.co/beats/metricbeat:${ELASTIC_VERSION:-7.1.1}
     # https://github.com/docker/swarmkit/issues/1951
     hostname: "{{.Node.Hostname}}-metricbeat"
     user: root