filebeat: 10.0.0.41
redis: 10.0.0.42
logstash: 10.0.0.43
elasticsearch: 10.0.0.44
kibana: 10.0.0.45
架构如下:
一、filebeat:10.0.0.41
1、安装filebeat,自己作多个filebeat,我只做一个
cat > /etc/yum.repos.d/artifacts.repo <<EOF [elastic-6.x] name=Elastic repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md EOF
yum -y install filebeat
2、编辑filebeat的配置文件,并下载nginx产生日志
yum -y install epel*
yum -y install nginx
vim /etc/filebeat/filebeat.yml
#其他配置都可以删除
filebeat.inputs: #收集日志
- type: log #类型enabled: true #始终收集paths:- /var/log/nginx/*.log #收集/var/log/nginx/下的以 .log 结尾的日志
output.redis: #输出日志,固定格式hosts: ["10.0.0.42:6379"] #输出在redis里
# password: "123456" #redis认证密码,我没有key: "log_file" #日志redis的key值db: 0 #redis数据库0timeout: 5 #延时5s
3、重启filebeat
systemctl restart filebeat
systemctl start nginx
4、测试
可以在安装filebeat的服务器上测试一下nginx的服务,并产生日志
curl -I 10.0.0.41:80
二、redis:10.0.0.42
1 下载redis5.0
wget http://download.redis.io/releases/redis-5.0.2.tar.gz
2、安装redis编译需要的环境
yum -y install gcc gcc-c++
3、编译安装redis
tar zxf redis-5.0.2.tar.gz
cd redis-5.0.2/
make MALLOC=libc && make install
4、写master拷贝redis配置至/etc/redis.conf、并修改成以守护进程启动
cp redis.conf /etc/redis.conf
vim /etc/redis.conf
5、启动redis主
/usr/local/bin/redis-server /etc/redis.conf
6、写slave拷贝redis配置至/etc/redis_slave.conf、修改端口,设置主从同步并修改成以守护进程启动
cp redis.conf /etc/redis_slave.conf
vim /etc/redis_slave.conf
7、启动redis从
/usr/local/bin/redis-server /etc/redis_slave.conf
8、查看端口
9、测试主从是否成功,并接收到filebeat日志
三、logstash:10.0.0.43
1、首先拉软件包,基于java语言开发首先配置java环境,推荐1G运行内存
2、安装logstash过滤收集工具,和客户端有点相似
rpm -ivh logstash-6.6.0.rpm
3、配置logstash、并启动logstash、自己加过滤规则
添加nginx过滤规则
vim /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/nginx_access
URIPARAM1 [A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]]*
NGINXACCESS %{IPORHOST:client_ip} (%{USER:ident}|- ) (%{USER:auth}|-) \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} (%{NOTSPACE:request}|-)(?: HTTP/%{NUMBER:http_version})?|-)" %{NUMBER:status} (?:%{NUMBER:bytes}|-) "(?:%{URI:referrer}|-)" "%{GREEDYDATA:agent}"
配置logstash
vim /etc/logstash/conf.d/redis_log.conf
input {
redis {
port => "6379"
host => "10.0.0.42" #记住收取了redis里的数据,redis将没有数据
# password => "6lapp" #redis认证密码,我没有
data_type => "list"
type => "log"
key => "log_file"
}
}
filter {grok {match => { "message" => "%{NGINXACCESS}" } #对应nginx,上一题过滤正则}
}
output {
elasticsearch {
hosts => "10.0.0.44:9200"
index => "logstash-%{+YYYY.MM.dd}"
}
}
systemctl start logstash
4、查看端口
四、elasticsearch:10.0.0.44
1、首先拉软件包,基于java语言开发首先配置java环境,推荐1G运行内存
2、首先安装elasticsearch-6.6.0.rpm
rpm -ivh elasticsearch-6.6.0.rpm
3、修改elasticsearch配置,并启动elasticsearch
vim /etc/elasticsearch/elasticsearch.yml
systemctl start elasticsearch
五、kibana:10.0.0.45
1、首先拉软件包,推荐1G运行内存,不需要java环境
2、安装kibana
rpm -ivh kibana-6.6.0-x86_64.rpm
3、修改kibana配置、并启动kibana
vim /etc/kibana/kibana.yml
systemctl start kibana
4、效果出现索引,并且过滤成功,如果过滤报警号,删除索引重启