ELK + logstash + rabbitmq 构建数据采集

环境

centos8 + docker

docker 安装rabbitmq

• pull rabbitmq镜像, 注意带-management表示是有后台管理网页功能

docker pull rabbitmq:3.9-management

• 创建并运行容器

sudo docker run -d --name rabbitmq \
--hostname localhost \
-p 4369:4369 \
-p 5671:5671 \
-p 5672:5672 \
-p 15671:15671 \
-p 15672:15672 \
-p 25672:52672 rabbitmq:3.9-management

docker 安装 logstash

docker hub 上复制 pull 命令，我pull的版本是8.1.2

docker pull logstash:8.1.2

本地创建logstash的config文件夹

mkdir ~/docker/logstash/config
mkdir ~/docker/logstash/config/conf.d

在config文件中添加logstash.yml文件

http.host: "0.0.0.0"
path.config: /usr/share/logstash/config/conf.d/*.conf
path.logs: /usr/share/logstash/logsxpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: [ "http://127.0.0.1:9200" ]

在conf.d文件中添加输入输出配置文件my_config.conf

input {rabbitmq {host => "127.0.0.1"port => 5672vhost => "virtual host"user => "user name"password => "password"exchange => "log"queue => "log_info"durable => truecodec => "json"}
}output {stdout { }elasticsearch {hosts => ["127.0.0.1:9200"]index => "rabbitmq_log"}
}

给上面创建的文件夹和文件添加权限，否则后面创建容器时会提示 “/usr/share/logstash/config/logstash.yml permission denied"（权限不足的问题困扰了我一天，git和stack overflow 都没有合适的答案，最后在CSDN上找到了解答）

chmod -R 777 logstash

启动容器

sudo docker run -dit --name logstash \
-p 5044:5044 \
-p 9600:9600  \
--privileged=true \
-e ES_JAVA_OPTS="-Xms512m -Xmx512m" \
-v /home/wbxbuilds/docker/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml \
-v /home/wbxbuilds/docker/logstash/config/conf.d:/usr/share/logstash/config/conf.d \
-v /home/wbxbuilds/docker/logstash/logs:/usr/share/logstash/logs logstash:8.1.2

docker 安装ELK

从dockerhub上拷贝pull命令，我下载的时7.0.0版本，网上说ELK的版本要和logstash一致，但是我目前还没遇到问题

创建文件夹

mkdir ~/docker/elasticsearch/config
mkdir ~/docker/elasticsearch/data
mkdir ~/docker/elasticsearch/plugins

添加配置文件elasticsearch.yml到config目录下

http.host: 0.0.0.0
http.cors.enabled: true
http.cors.allow-origin: "*"

给文件添加权限（被上面整怕了）

chmod -R 777 elasticsearch

启动容器

sudo docker run -d --name elasticsearch \
-p 9200:9200 \
-p 9300:9300 \
-e "discovery.type=single-node" \
-e ES_JAVA_OPTS="-Xms256m -Xmx256m" \
-v /home/wbxbuilds/docker/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \
-v /home/wbxbuilds/docker/elasticsearch/data:/usr/share/elasticsearch/data \
-v /home/wbxbuilds/docker/elasticsearch/plugins:/usr/share/elasticsearch/plugins elasticsearch:7.17.0