SpringBoot集成ElasticSearch实现支持错别字检索和关键字高亮的模糊查询
文章目录
- 一、背景
- 二、环境准备
- 1.es8集群
- 2.Kibana
- 3.Canal
- 三、集成到SpringBoot
- 1.新增依赖
- 2.es配置类
- 3.建立索引
- 4.修改查询方法
- 四、修改前端
一、背景
我们在开发项目的搜索引擎的时候,如果当数据量庞大、同时又需要支持全文检索模糊查询,甚至你想做到像百度、CSDN等大型网站的搜索效果——支持错别字查询和关键字高亮显示,搜索结果根据相关性排序,如下图所示,那么就需要集成ElasticSearch来实现了。
本文用若依的“通知公告”页面的搜索功能来演示如何集成ElasticSearch到SpringBoot项目中改造优化搜索引擎,最终效果如下图所示
二、环境准备
需要部署好的中间件如下:
- ElasticSearch(8.14.3)【集群】
- Kibana(8.14.3)
- Canal(1.1.8)
本文主要讲解es项目实战使用,关于部署部分就简单说下
1.es8集群
参考:ElasticSearch8集群的安装部署_es8搭建集群-CSDN博客
我这里用了三台虚拟机来部署es集群,这里推荐一个es浏览器GUI插件——Elasticvue,可视化操作界面,很方便,可用来监看节点状态、索引数据等。如下图所示
elasticsearch.yml 如下所示(另外两台要改下node.name)
cluster.name: es-cluster
node.name: node-1
# 节点属性
node.roles: [master,data]
path.data: /data/es/elasticsearch-8.14.3/data
path.logs: /data/es/elasticsearch-8.14.3/logs
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.168.100.182:9300", "192.168.100.183:9300", "192.168.100.184:9300"]
cluster.initial_master_nodes: ["node-1","node-2","node-3"]
# 安全认证
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true # 注意第一个空格
keystore.path: /data/es/elasticsearch-8.14.3/config/certs/http.p12
truststore.path: /data/es/elasticsearch-8.14.3/config/certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: /data/es/elasticsearch-8.14.3/elastic-certificates.p12
truststore.path: /data/es/elasticsearch-8.14.3/elastic-certificates.p12
http.host: [_local_, _site_]
ingest.geoip.downloader.enabled: false
xpack.security.http.ssl.client_authentication: none
2.Kibana
参考:Elasticsearch集群以及kibana安装_elasticsearch集群安装-CSDN博客
kibana.yml
注意这里的账号是es里本身自带的,然后密码是后期随机生成的(kibana不能配置es超管用户)
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts:
- "https://192.168.100.182:9200"
- "https://192.168.100.183:9200"
- "https://192.168.100.184:9200"
elasticsearch.username: "kibana_system"
elasticsearch.password: "z3P+5vZVRh2dh10p=i-s"
elasticsearch.ssl.verificationMode: none
i18n.locale: "zh-CN"
3.Canal
参考:canal实现MySQL和ES同步实践_canal mysql es-CSDN博客
canal.adapter-1.1.8\conf\application.yml
server:
port: 8081
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
canal.conf:
mode: tcp #tcp kafka rocketMQ rabbitMQ
flatMessage: true
zookeeperHosts:
syncBatchSize: 1000
retries: -1
timeout:
accessKey:
secretKey:
consumerProperties:
# canal tcp consumer
canal.tcp.server.host: 127.0.0.1:11111
canal.tcp.zookeeper.hosts:
canal.tcp.batch.size: 500
canal.tcp.username:
canal.tcp.password:
# kafka consumer
kafka.bootstrap.servers: 127.0.0.1:9092
kafka.enable.auto.commit: false
kafka.auto.commit.interval.ms: 1000
kafka.auto.offset.reset: latest
kafka.request.timeout.ms: 40000
kafka.session.timeout.ms: 30000
kafka.isolation.level: read_committed
kafka.max.poll.records: 1000
# rocketMQ consumer
rocketmq.namespace:
rocketmq.namesrv.addr: 127.0.0.1:9876
rocketmq.batch.size: 1000
rocketmq.enable.message.trace: false
rocketmq.customized.trace.topic:
rocketmq.access.channel:
rocketmq.subscribe.filter:
# rabbitMQ consumer
rabbitmq.host:
rabbitmq.virtual.host:
rabbitmq.username:
rabbitmq.password:
rabbitmq.resource.ownerId:
srcDataSources:
defaultDS:
url: jdbc:mysql://127.0.0.1:3306/ry-vue?useUnicode=true
username: canal
password: canal123
canalAdapters:
- instance: example # canal instance Name or mq topic name
groups:
- groupId: g1
outerAdapters:
- name: logger
- name: es8
hosts: https://192.168.100.183:9200 # 127.0.0.1:9200 for rest mode
properties:
mode: rest # transport or rest
security.ca.path: C:\Users\znak\Desktop\elasticsearch.crt
security.ssl.verification_mode: none # ✅ 忽略 SSL 证书验证
security.auth: elastic:rwfejTWwHo666BmrQ2QW # only used for rest mode
cluster.name: es-cluster
注意这里有个elasticsearch.crt
证书要配置,我的Canal是部署在window本地的,es开启了ssl认证,canal adapter(客户端)需要有这个证书才能访问,【这里虽然我配置了忽略ssl验证,但是发现没效果)
cmd执行:
#生成证书
echo -n | openssl s_client -connect https://192.168.100.182:9200 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > elasticsearch.crt
#导入到Java 信任库
keytool -importcert -alias elasticsearch-cert -keystore "E:\xuyue\jre1.8\lib\security\cacerts" -file "C:\Users\znak\Desktop\elasticsearch.crt"
canal.adapter-1.1.8\conf\es8\sys_notice.yml
这个是同步mysql的相关配置,本文以若依的通知表sys_notice
演示,所以这里配置的是这个表的sql,可以参考一下
dataSourceKey: defaultDS
destination: example
esMapping:
_index: sys_notice
_type: _doc
_id: notice_id
upsert: true
sql: "SELECT sn.notice_id as id, sn.notice_id, sn.notice_title, sn.notice_type, CONVERT(sn.notice_content USING utf8mb4) AS notice_content, sn.status, sn.create_time, sn.update_time,sn.remark,sn.create_by,sn.update_by FROM sys_notice sn"
commitBatch: 3000
启动canal后,默认是增量同步,如果原先数据表里已经有数据,想要先同步到es,可以手动调用全量同步接口,如下所示:
curl http://127.0.0.1:8081/etl/es8/sys_notice.yml -X POST
三、集成到SpringBoot
1.新增依赖
ruoyi-system/pom.xml
<dependency>
<groupId>co.elastic.clients</groupId>
<artifactId>elasticsearch-java</artifactId>
<version>8.1.0</version>
<exclusions>
<exclusion>
<artifactId>jakarta.json-api</artifactId>
<groupId>jakarta.json</groupId>
</exclusion>
<exclusion>
<artifactId>elasticsearch-rest-client</artifactId>
<groupId>org.elasticsearch.client</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
<version>8.1.0</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>jakarta.json</groupId>
<artifactId>jakarta.json-api</artifactId>
<version>2.1.2</version>
</dependency>
<dependency>
<groupId>jakarta.json.bind</groupId>
<artifactId>jakarta.json.bind-api</artifactId>
<version>3.0.0</version>
</dependency>
<!--官方文档中如果遇到ClassNotFouind:jakarta.json.spi.JsonProvider,就需要导入这个包-->
<dependency>
<groupId>org.glassfish</groupId>
<artifactId>jakarta.json</artifactId>
<version>2.0.1</version>
</dependency>
2.es配置类
ElasticsearchConfig.java
package com.ruoyi.system.config;
import co.elastic.clients.elasticsearch.ElasticsearchClient;
import co.elastic.clients.transport.rest_client.RestClientTransport;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.ssl.SSLContexts;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import javax.net.ssl.SSLContext;
import java.security.KeyManagementException;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
@Configuration
public class ElasticsearchConfig {
@Bean
public ElasticsearchClient elasticsearchClient() throws NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
// 认证信息
BasicCredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials("elastic", "rwfejTWwHo666BmrQ2QW"));
// 创建不验证证书的 SSLContext
SSLContext sslContext = SSLContexts.custom()
.loadTrustMaterial((chain, authType) -> true) // 信任所有证书
.build();
// 创建 RestClient
RestClientBuilder builder = RestClient.builder(
new HttpHost("192.168.100.182", 9200, "https"),
new HttpHost("192.168.100.183", 9200, "https"),
new HttpHost("192.168.100.184", 9200, "https")
).setHttpClientConfigCallback(httpClientBuilder ->
httpClientBuilder.setSSLContext(sslContext).setDefaultCredentialsProvider(credentialsProvider));
// 创建 Elasticsearch Java API Client
RestClient restClient = builder.build();
RestClientTransport transport = new RestClientTransport(
restClient, new co.elastic.clients.json.jackson.JacksonJsonpMapper(new ObjectMapper())
);
return new ElasticsearchClient(transport);
}
}
3.建立索引
索引创建是一个一次性操作,这里我直接建了一个测试类来创建
注意:
- 这里字段我用的下划线格式,但是实体类里的是驼峰,所以后续需要手动映射下;同时注意数据库同步到es的时候,字段也要注意映射关系
- 需要模糊查询的用text类型,并选择合适的分词器
- 日期这里我加了yyyy-MM-dd’T’HH:mm:ssXXX这种类型,是因为我用canal同步mysql到es的时候,日期格式一直是这种格式(虽然mysql是yyyy-MM-dd HH:mm:ss)
@Test
void testIndexExample() throws IOException {
//1.创建索引
String indexName = "sys_notice";
//检查索引是否已存在
BooleanResponse exists = client.indices().exists(c -> c.index(indexName));
//如果存在,则删除索引
if (exists.value()) {
client.indices().delete(c -> c.index(indexName));
} else {
// 创建索引
// 设置索引配置
CreateIndexResponse createIndexResponse = client.indices().create(c -> c
.index(indexName)
.settings(s -> s
.numberOfShards("1")
.numberOfReplicas("1")
)
.mappings(m -> m
.properties("notice_id", p -> p.keyword(k -> k))
.properties("notice_title", p -> p.text(t -> t.analyzer("ik_max_word")))
.properties("notice_type", p -> p.keyword(k -> k))
.properties("notice_content", p -> p.text(t -> t.analyzer("ik_max_word")))
.properties("status", p -> p.keyword(k -> k))
.properties("create_by", p -> p.keyword(k -> k))
.properties("create_time", p -> p.date(d -> d.format("yyyy-MM-dd HH:mm:ss||yyyy-MM-dd'T'HH:mm:ssXXX||epoch_millis")))
.properties("update_by", p -> p.keyword(k -> k))
.properties("update_time", p -> p.date(d -> d.format("yyyy-MM-dd HH:mm:ss||yyyy-MM-dd'T'HH:mm:ssXXX||epoch_millis")))
.properties("remark", p -> p.text(t -> t.analyzer("ik_smart")))
)
);
if (Boolean.TRUE.equals(createIndexResponse.acknowledged())) {
System.out.println("索引 " + indexName + " 创建成功");
} else {
System.out.println("索引 " + indexName + " 创建失败");
}
}
}
索引建好我们可以用kibana来查看
GET /sys_notice
4.修改查询方法
com.ruoyi.system.service.impl.SysNoticeServiceImpl
@Resource
private ElasticsearchClient client;
/**
* 查询公告列表
*
* @param notice 公告信息
* @return 公告集合
*/
@Override
public List<SysNotice> selectNoticeList(SysNotice notice)
{
Integer pageNum = Convert.toInt(ServletUtils.getParameter(PAGE_NUM), 1);
Integer pageSize = Convert.toInt(ServletUtils.getParameter(PAGE_SIZE), 10);
SysNoticeOfES noticeOfES = new SysNoticeOfES();
BeanUtils.copyBeanProp(noticeOfES, notice);
noticeOfES.setNotice_title(notice.getNoticeTitle());
noticeOfES.setCreate_by(notice.getCreateBy());
noticeOfES.setNotice_type(notice.getNoticeType());
List<SysNoticeOfES> noticesFromEs = searchNoticesFromEs(noticeOfES, pageNum, pageSize);
if (noticesFromEs != null && noticesFromEs.size() > 0) {
//转化为List<SysNotice>
return noticesFromEs.stream()
.map(noticeOfES1 -> {
SysNotice sysNotice = new SysNotice();
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
BeanUtils.copyBeanProp(sysNotice, noticeOfES1);
sysNotice.setNoticeId(noticeOfES1.getId());
sysNotice.setNoticeTitle(noticeOfES1.getNotice_title());
sysNotice.setCreateBy(noticeOfES1.getCreate_by());
sysNotice.setUpdateBy(noticeOfES1.getUpdate_by());
sysNotice.setNoticeContent(noticeOfES1.getNotice_content());
sysNotice.setNoticeType(noticeOfES1.getNotice_type());
try {
sysNotice.setCreateTime(sdf.parse(sdf.format(noticeOfES1.getCreate_time())));
sysNotice.setUpdateTime(noticeOfES1.getUpdate_time() != null ? sdf.parse(sdf.format(noticeOfES1.getUpdate_time())) : null);
} catch (ParseException e) {
e.printStackTrace();
throw new GlobalException("日期转换异常");
}
sysNotice.setUpdateBy(noticeOfES1.getUpdate_by());
return sysNotice;
})
.collect(Collectors.toList());
} else {
System.out.println("从数据库中查询");
return noticeMapper.selectNoticeList(notice);
}
}
private List<SysNoticeOfES> searchNoticesFromEs(SysNoticeOfES notice, int pageNum, int pageSize) {
try {
// 计算 ES 分页参数
int from = (pageNum - 1) * pageSize;
// 构建 ES 查询
SearchRequest request = new SearchRequest.Builder()
.index("sys_notice") // ES 索引名称
.query(q -> q.bool(b -> {
if (notice.getNotice_title() != null) {
b.must(m -> m.match(t -> t.field("notice_title").query(notice.getNotice_title()).fuzziness("AUTO")));
}
if (notice.getCreate_by() != null) {
b.must(m -> m.wildcard(t -> t.field("create_by").value("*" + notice.getCreate_by() + "*")));
}
if (notice.getNotice_type() != null) {
b.must(m -> m.term(t -> t.field("notice_type").value(notice.getNotice_type())));
}
return b;
}))
.highlight(h -> h.fields("notice_title", hf -> hf
.preTags("<span style='color:red;'>") // 高亮前缀
.postTags("</span>")).requireFieldMatch(false))
.from(from)
.size(pageSize)
.build();
// 执行查询
SearchResponse<SysNoticeOfES> response = client.search(request, SysNoticeOfES.class);
// 解析结果
return response.hits().hits().stream()
.map(hit -> {
SysNoticeOfES sysNoticeOfES = hit.source(); // 获取原始的 SysNotice 数据
if (hit.highlight() != null && hit.highlight().containsKey("notice_title")) {
// 从 highlight 中获取高亮的内容,假设只有一个高亮值
assert sysNoticeOfES != null;
sysNoticeOfES.setNotice_title(hit.highlight().get("notice_title").get(0)); // 更新 noticeTitle 为高亮文本
}
return sysNoticeOfES; // 返回更新后的 SysNotice
})
.collect(Collectors.toList());
} catch (Exception e) {
e.printStackTrace();
return new ArrayList<>();
}
}
requireFieldMatch(false)
是指允许模糊查询高亮
fuzziness("AUTO")
表示:
- 如果查询词 ≤ 2 个字符,不允许错别字。
- 如果查询词 3~5 个字符,允许 1 个错别字。
- 如果查询词 ≥ 6 个字符,允许 2 个错别字。
上面的查询用es语法相当于(用下图的参数为例):
GET /sys_notice/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"notice_title": {
"query": "若一集成es",
"fuzziness": "AUTO"
}
}
},
{
"wildcard": {
"create_by": {
"value": "*ad*"
}
}
},
{
"term": {
"notice_type": "1"
}
}
]
}
},
"highlight": {
"fields": {
"notice_title": {
"pre_tags": ["<span style='color:red;'>"],
"post_tags": ["</span>"]
}
},
"require_field_match": false
},
"from": 0,
"size": 10
}
在kibana执行的结果:
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 3.1798086,
"hits": [
{
"_index": "sys_notice",
"_id": "10",
"_score": 3.1798086,
"_source": {
"notice_title": "测试若依集成es",
"notice_type": "1",
"notice_content": "<p>一条测试通知</p>",
"status": "0",
"create_time": "2025-03-13T10:39:09+08:00",
"update_time": null,
"remark": null,
"create_by": "admin",
"update_by": "",
"id": 10
},
"highlight": {
"notice_title": [
"测试<span style='color:red;'>若</span>依<span style='color:red;'>集成</span><span style='color:red;'>es</span>"
]
}
},
{
"_index": "sys_notice",
"_id": "11",
"_score": 2.9974954,
"_source": {
"notice_title": "测试若依与es集成-3",
"notice_type": "1",
"notice_content": "<p>111</p>",
"status": "0",
"create_time": "2025-03-13T10:39:52+08:00",
"update_time": null,
"remark": null,
"create_by": "admin",
"update_by": "",
"id": 11
},
"highlight": {
"notice_title": [
"测试<span style='color:red;'>若</span>依与<span style='color:red;'>es</span><span style='color:red;'>集成</span>-3"
]
}
},
{
"_index": "sys_notice",
"_id": "2",
"_score": 1.4532554,
"_source": {
"notice_title": "维护通知:2018-07-01 若依系统凌晨维护",
"notice_type": "1",
"notice_content": "维护内容",
"status": "0",
"create_time": "2024-04-30T11:53:10+08:00",
"update_time": null,
"remark": "管理员",
"create_by": "admin",
"update_by": "",
"id": 2
},
"highlight": {
"notice_title": [
"维护通知:2018-07-01 <span style='color:red;'>若</span>依系统凌晨维护"
]
}
}
]
}
}
com.ruoyi.system.domain.SysNoticeOfES
package com.ruoyi.system.domain;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.ruoyi.common.core.domain.BaseEntity;
import com.ruoyi.common.core.domain.BaseEntityOfES;
import com.ruoyi.common.xss.Xss;
import org.apache.commons.lang3.builder.ToStringBuilder;
import org.apache.commons.lang3.builder.ToStringStyle;
import javax.validation.constraints.NotBlank;
import javax.validation.constraints.Size;
/**
* 通知公告表 sys_notice
*
* @author ruoyi
*/
public class SysNoticeOfES extends BaseEntityOfES
{
private static final long serialVersionUID = 1L;
/** 公告ID */
private Long id;
/** 公告标题 */
private String notice_title;
/** 公告类型(1通知 2公告) */
private String notice_type;
/** 公告内容 */
private String notice_content;
/** 公告状态(0正常 1关闭) */
private String status;
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public String getNotice_title() {
return notice_title;
}
public void setNotice_title(String notice_title) {
this.notice_title = notice_title;
}
public String getNotice_type() {
return notice_type;
}
public void setNotice_type(String notice_type) {
this.notice_type = notice_type;
}
public String getNotice_content() {
return notice_content;
}
public void setNotice_content(String notice_content) {
this.notice_content = notice_content;
}
public String getStatus() {
return status;
}
public void setStatus(String status) {
this.status = status;
}
}
com.ruoyi.common.core.domain.BaseEntityOfES
package com.ruoyi.common.core.domain;
import com.fasterxml.jackson.annotation.JsonFormat;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonInclude;
import com.fasterxml.jackson.annotation.JsonProperty;
import java.io.Serializable;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;
/**
* Entity基类
*
* @author ruoyi
*/
public class BaseEntityOfES implements Serializable
{
private static final long serialVersionUID = 1L;
/** 搜索值 */
@JsonIgnore
private String searchValue;
/** 创建者 */
private String create_by;
/** 创建时间 */
@JsonFormat(pattern = "yyyy-MM-dd'T'HH:mm:ssXXX")
private Date create_time;
/** 更新者 */
private String update_by;
/** 更新时间 */
@JsonFormat(pattern = "yyyy-MM-dd'T'HH:mm:ssXXX")
private Date update_time;
/** 备注 */
private String remark;
/** 请求参数 */
@JsonInclude(JsonInclude.Include.NON_EMPTY)
private Map<String, Object> params;
public String getSearchValue()
{
return searchValue;
}
public void setSearchValue(String searchValue)
{
this.searchValue = searchValue;
}
public String getCreate_by() {
return create_by;
}
public void setCreate_by(String create_by) {
this.create_by = create_by;
}
public Date getCreate_time() {
return create_time;
}
public void setCreate_time(Date create_time) {
this.create_time = create_time;
}
public String getUpdate_by() {
return update_by;
}
public void setUpdate_by(String update_by) {
this.update_by = update_by;
}
public Date getUpdate_time() {
return update_time;
}
public void setUpdate_time(Date update_time) {
this.update_time = update_time;
}
public String getRemark() {
return remark;
}
public void setRemark(String remark) {
this.remark = remark;
}
public Map<String, Object> getParams()
{
if (params == null)
{
params = new HashMap<>();
}
return params;
}
public void setParams(Map<String, Object> params)
{
this.params = params;
}
}
四、修改前端
最后,前端要修改table的展示,使其能够识别html标签,实现关键字高亮显示的效果
ruoyi-ui/src/views/system/notice/index.vue
<el-table-column
label="公告标题"
align="center"
prop="noticeTitle"
:show-overflow-tooltip="true">
<template slot-scope="scope">
<span v-html="scope.row.noticeTitle"></span> <!-- 允许 HTML 高亮显示 -->
</template>
</el-table-column>