Springboot3整合ELK实现日志可视化
一、环境准备
Elasticsearch、Logstash、Kibana,组合起来可以搭建线上日志系统
ELK中各个服务的作用
-
Elasticsearch:用于存储收集到的日志信息；
-
Logstash:用于收集日志，SpringBoot应用整合了Logstash以后会把日志发送给Logstash,Logstash再把日志转发给Elasticsearch；
-
Kibana:通过Web端的可视化界面来查看日志。
1、安装Elasticsearch
-
下载镜像
如果本地有镜像，则无需下载。是否有镜像，可通过 docker images 命令查看。
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.8.2 -
创建挂载目录
mkdir -p /usr/local/elasticsearch/data && chmod 777 /usr/local/elasticsearch/data mkdir -p /usr/local/elasticsearch/plugins && chmod 777 /usr/local/elasticsearch/plugins mkdir -p /usr/local/elasticsearch/logs && chmod 777 /usr/local/elasticsearch/logs -
docker run 运行容器
docker run -d --name elasticsearch \ --restart=always \ -p 9200:9200 \ -p 9300:9300 \ -v /usr/local/elasticsearch/data:/usr/share/elasticsearch/data \ -v /usr/local/elasticsearch/plugins:/usr/share/elasticsearch/plugins \ -v /usr/local/elasticsearch/logs:/usr/share/elasticsearch/logs \ -e "discovery.type=single-node" \ -e ES_JAVA_OPTS="-Xms512m -Xmx512m" \ -e xpack.security.enabled=false \ docker.elastic.co/elasticsearch/elasticsearch:8.8.2 -
docker 运行 elasticsearch 时，有可能会报如下错误：
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]解决办法是：在宿主机
/etc/sysctl.conf文件最后添加一行：vm.max_map_count=262144，然后执行命令sysctl -p。echo "vm.max_map_count=262144" >> /etc/sysctl.conf sysctl -p #让配置生效 -
验证
通过
docker ps查看容器的运行信息。访问网址
http://主机IP:9200/，会有一点点延迟，会看到类似如下内容:{ "name" : "18263d8efab1", "cluster_name" : "docker-cluster", "cluster_uuid" : "DzaRMEosQ1CLD0b-MkgeQg", "version" : { "number" : "8.8.2", "build_flavor" : "default", "build_type" : "docker", "build_hash" : "98e1271edf932a480e4262a471281f1ee295ce6b", "build_date" : "2023-06-26T05:16:16.196344851Z", "build_snapshot" : false, "lucene_version" : "9.6.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" } -
默认配置【不用改】
ES 默认的配置文件 elasticsearch.yml 的内容只有 2 行：
cluster.name: "docker-cluster" network.host: 0.0.0.0
2、安装IK分词器
复制ik分词器到/usr/local/elasticsearch/plugins目录中即可
3、安装Kibana
-
下载镜像
docker pull docker.elastic.co/kibana/kibana:8.8.2 -
修改配置
创建/usr/local/kibana/config/kibana.yml
mkdir -p /usr/local/kibana/config cat > /usr/local/kibana/config/kibana.yml << EOF server.name: kibana server.host: "0" elasticsearch.hosts: [ "http://192.172.0.6:9200" ] # elasticsearch 改为elasticsearch容器的地址 monitoring.ui.container.elasticsearch.enabled: true i18n.locale: "zh-CN" # 设置kibana为中文 EOF cat > /usr/local/kibana/config/kibana.yml << EOF server.name: kibana server.host: "0.0.0.0" elasticsearch.hosts: [ "http://192.172.0.14:9200" ] monitoring.ui.container.elasticsearch.enabled: true i18n.locale: "zh-CN" EOF -
创建容器
docker run -id -p 5601:5601 \ --name kibana \ --restart=always \ -v /usr/local/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml \ -e "xpack.reporting.roles.enabled=false" \ -e “ELASTICSEARCH_URL=http://192.172.0.14:9200” \ docker.elastic.co/kibana/kibana:8.8.2
4、安装Logstash
-
拉取镜像
docker pull docker.elastic.co/logstash/logstash:8.8.2 -
创建挂载目录
mkdir -p /usr/local/logstash/config -
创建配置文件
-
创建logstash.conf
cat > /usr/local/logstash/config/logstash.conf << EOF input { beats { mode => "server" host => "0.0.0.0" #可以远程访问的机器 port => 5044 #logstash控制台访问端口 codec => json } } output { elasticsearch { #elasticsearch地址，多个以','隔开 hosts => ["192.172.0.14:9200"] #创建的elasticsearch索引名，可以自定义也可以使用下面的默认 index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" } } EOF -
创建logstash.yml
cat > /usr/local/logstash/config/logstash.yml << EOF http.host: "0.0.0.0" xpack.monitoring.elasticsearch.hosts: ["http://192.172.0.14:9200"] #es服务器地址 EOF
-
-
创建容器
docker run -d --name logstash \ -m 1000M --restart=always \ -p 5044:5044 -p 9600:9600 \ --privileged=true \ -e ES_JAVA_OPTS="-Duser.timezone=Asia/Shanghai" \ -v /usr/local/logstash/config/logstash.conf:/usr/share/logstash/pipeline/logstash.conf \ -v /usr/local/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml \ docker.elastic.co/logstash/logstash:8.8.2二、 SpringBoot整合Logstash
1、添加依赖
<!-- logstash -->
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>8.0</version>
</dependency>
<!-- lombok-->
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
2、修改logback配置文件
修改 logback-spring.xml配置 文件 增加 logstash 配置
<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false">
<!--应用名称-->
<springProperty scope="context" name="appName" source="spring.application.name"/>
<!--输出到logstash的appender-->
<appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<!-- logstash服务器地址-->
<destination>192.172.0.14:5044</destination>
<encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder">
<customFields>{"spring.application.name":"${appName}"}</customFields>
</encoder>
</appender>
<!-- 本地日志打印格式-->
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%5p | %-40.40logger{39} : %m%n</pattern>
<charset>utf8</charset>
</encoder>
</appender>
<!-- 配置项目最大的包名-->
<logger name="com.woniu" level="DEBUG" additivity="false">
<appender-ref ref="CONSOLE"/>
<appender-ref ref="LOGSTASH"/>
</logger>
<root level="INFO">
<appender-ref ref="CONSOLE"/>
<appender-ref ref="LOGSTASH"/>
</root>
</configuration>
3、修改yml
spring:
application:
name: logs-demo
#日志配置
logging:
config: classpath:logback-spring.xml
- 启动项目查看是否成功推送日志
