filebeat+logstash对springcloud项目日志进行解析_综合

项目地址：https://gitee.com/lwydyby/springcloud-adplatform
该项目的日志目前可以分为三种：
(1)项目启动日志
(2)controller层的操作日志（通过aop实现记录）
(3)项目非controller层异常日志
由于分为3种格式，则需要三种macth进行匹配：

grok {       match => ["message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} com.gameley.common.Interceptor.HttpAspect - %{GREEDYDATA:log_json}","message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} %{NOTSPACE:classname} \- (?<error>[\s\S]*)","message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} %{NOTSPACE:classname} \- %{GREEDYDATA:otherinfo}" ]}

使用logstash自带表达式的格式一般为：%{SYNTAX:SEMANTIC}
即： SYNTAX代表匹配值的类型，例如，0.11可以NUMBER类型所匹配，10.222.22.25可以使用IP匹配。 *SEMANTIC表示存储该值的一个变量声明，client_ip_address`这个字段里边，类似数据库的列名
具体可参考：http://blog.csdn.net/liukuan73/article/details/52318243
而自定义匹配需要使用正则表达式，格式为

( ?<列名>正则表达式)

注：[\s\S]*为匹配剩余所有的字符，我这里是用来匹配错误日志的详细信息的
由于controller层日志是以json格式进行的存储，除使用grok之外还需使用 json进行解析

  json {  source => "log_json"  target => "content"  remove_field=>["logjson"]  }

完整配置文件为：

input {beats {port => 5044}
}
filter{grok {       match => ["message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} com.gameley.common.Interceptor.HttpAspect - %{GREEDYDATA:log_json}","message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} %{NOTSPACE:classname} \- (?<error>[\s\S]*)","message","%{TIMESTAMP_ISO8601:timestamp} \[%{NOTSPACE:thread}\] %{WORD:type} %{NOTSPACE:classname} \- %{GREEDYDATA:otherinfo}" ]}   json {  source => "log_json"  target => "content"  remove_field=>["logjson"]  }  
}
output {elasticsearch {hosts => ["localhost"]manage_template => falseindex => "%{[fields][logIndex]}-%{+YYYY.MM.dd}"document_type => "%{[fields][docType]}"}
}

除需要配置logstash之外，还需要对filebeat进行配置，以设置如何发送日志（处理多行日志）：

filebeat:prospectors:-paths:- "/data/logs/*.log"fields:logIndex: userservicedocType: springcloudproject: adplatformmultiline:pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'negate: truematch: aftertimeout: 5s
output:logstash:hosts: ["localhost:5044"]

日志文件样例：

2018-02-22 16:09:59.876 [main] INFO  o.s.integration.channel.PublishSubscribeChannel - Channel 'user-service:dev:8763.errorChannel' has 1 subscriber(s).
2018-02-22 16:09:59.876 [main] INFO  o.s.integration.endpoint.EventDrivenConsumer - started _org.springframework.integration.errorLogger
2018-02-22 16:09:59.913 [main] INFO  o.s.i.endpoint.SourcePollingChannelAdapter - started sleuthStreamSpanReporter.poll.inboundChannelAdapter
2018-02-22 16:09:59.913 [main] INFO  o.s.context.support.DefaultLifecycleProcessor - Starting beans in phase 2147482647
2018-02-22 16:09:59.913 [main] INFO  o.s.context.support.DefaultLifecycleProcessor - Starting beans in phase 2147483647
2018-02-22 16:09:59.966 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8763"]
2018-02-22 16:09:59.991 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8763"]
2018-02-22 16:09:59.992 [main] INFO  org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2018-02-22 16:10:00.081 [main] INFO  o.s.b.c.e.tomcat.TomcatEmbeddedServletContainer - Tomcat started on port(s): 8763 (http)
2018-02-22 16:10:00.082 [main] INFO  o.s.c.n.e.s.EurekaAutoServiceRegistration - Updating port to 8763
2018-02-22 16:10:00.088 [main] INFO  com.gameley.UserServiceApplication - Started UserServiceApplication in 281.136 seconds (JVM running for 285.806)
2018-02-22 16:10:13.234 [http-nio-8763-exec-1] INFO  o.a.c.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring FrameworkServlet 'dispatcherServlet'
2018-02-22 16:10:13.234 [http-nio-8763-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - FrameworkServlet 'dispatcherServlet': initialization started
2018-02-22 16:10:13.301 [http-nio-8763-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - FrameworkServlet 'dispatcherServlet': initialization completed in 67 ms
2018-02-22 16:12:26.850 [http-nio-8763-exec-10] INFO  com.gameley.common.Interceptor.HttpAspect - {
   "address":"http://192.168.0.43:8763/user/validate","args":"[admin, 123456]","classMethod":"com.gameley.controller.UserController.validate","httpMethod":"POST","ip":"192.168.0.43","result":"UserInfo{id='32', username='null', password='null', name='admin', description='null', updTime=null}","time":1041}
2018-02-22 16:12:27.971 [task-scheduler-5] INFO  o.s.integration.codec.kryo.CompositeKryoRegistrar - registering [40, java.io.File] with serializer org.springframework.integration.codec.kryo.FileSerializer
2018-02-22 16:12:28.276 [http-nio-8763-exec-9] INFO  com.gameley.common.Interceptor.HttpAspect - {
   "address":"http://192.168.0.43:8763/user/validate","args":"[admin, 123456]","classMethod":"com.gameley.controller.UserController.validate","httpMethod":"POST","ip":"192.168.0.43","result":"UserInfo{id='32', username='null', password='null', name='admin', description='null', updTime=null}","time":7}