Elasticsearch 之 ElasticsearchRestTemplate 聚合查询
前言:
上一篇我们分享了 ElasticsearchRestTemplate 的常用普通查询,本篇我们使用 ElasticsearchRestTemplate 来完成 Elasticsearch 更为复杂的聚合查询。
Elasticsearch 系列文章传送门
Elasticsearch 基础篇【ES】
Elasticsearch Windows 环境安装
Elasticsearch 之 ElasticsearchRestTemplate 普通查询
聚合查询
聚合查询是指从数据中提取统计信息,聚合又可以分为以下三类:
- 度量聚合:度量聚合其实就是计算平均值、求和、计数、最大值、最小值等的聚合查询。
- 桶聚合:桶聚合其实就是按指定的字段进行分组后聚合,例如先按汽车类型分类后,求汽车平均价格。
- 嵌套聚合:嵌套聚合其实就在嵌套二字中,也就是文档的嵌套,比如汽车除了一些主要信息之外,还有一些次要零部件的信息,在汽车主信息中嵌套一个次要的车载手机支架信息文档对象,当我们需要统计某个汽车类型的车载手机支架信息的时候,就需要使用到嵌套聚合了。
度量聚合
Max(最大值查询)
查询最大值是一个非常常见的业务场景,而查询最大值就需要使用到聚合查询了,聚合查询需要使用到 AggregationBuilders,使用 ElasticsearchRestTemplate 完成最大值查询代码如下:
public double carMaxPrice() {
MaxAggregationBuilder maxAggregationBuilder = AggregationBuilders.max("maxPrice").field("price");
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
NativeSearchQuery build = queryBuilder.withQuery(QueryBuilders.matchAllQuery()).addAggregation(maxAggregationBuilder).build();
SearchHits<CarDO> search = elasticsearchRestTemplate.search(build, CarDO.class);
Aggregations aggregations = search.getAggregations();
double maxPrice = 0;
if (aggregations != null) {
Max max = aggregations.get("maxPrice");
maxPrice = max.getValue();
}
return maxPrice;
}
Min(最小值查询)
查询最小值是一个非常常见的业务场景,而查询最小值就需要使用到聚合查询了,聚合查询需要使用到 AggregationBuilders,使用 ElasticsearchRestTemplate 完成最小值查询代码如下:
public double carMinPrice() {
MinAggregationBuilder minAggregationBuilder = AggregationBuilders.min("minPrice").field("price");
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
NativeSearchQuery build = queryBuilder.withQuery(QueryBuilders.matchAllQuery()).addAggregation(minAggregationBuilder).build();
SearchHits<CarDO> search = elasticsearchRestTemplate.search(build, CarDO.class);
Aggregations aggregations = search.getAggregations();
double minPrice = 0;
if (aggregations != null) {
Min min = aggregations.get("minPrice");
minPrice = min.getValue();
}
return minPrice;
}
Avg(平均值查询)
计算平均值是一个非常常见的业务场景,如果没有聚合查询我们就需要把目标数据全部查询出来之后,在进行求平均操作,而 Elasticsearch 给我们提供了平均值的聚合查询方法,查询平均值的代码如下:
public double carAvgPrice() {
AvgAggregationBuilder avgAggregationBuilder = AggregationBuilders.avg("avgPrice").field("price");
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
NativeSearchQuery build = queryBuilder.withQuery(QueryBuilders.matchAllQuery()).addAggregation(avgAggregationBuilder).build();
SearchHits<CarDO> search = elasticsearchRestTemplate.search(build, CarDO.class);
Aggregations aggregations = search.getAggregations();
double avgPrice = 0;
if (aggregations != null) {
Avg avg = aggregations.get("avgPrice");
avgPrice = avg.getValue();
}
return avgPrice;
}
Sum(求和查询)
计算求和是一个非常常见的业务场景,如果没有聚合查询我们就需要把目标数据全部查询出来之后,在进行求和操作,而 Elasticsearch 给我们提供了求和的聚合查询方法,代码如下:
public double carSumPrice() {
SumAggregationBuilder sumAggregationBuilder = AggregationBuilders.sum("sumPrice").field("price");
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
NativeSearchQuery build = queryBuilder.withQuery(QueryBuilders.matchAllQuery()).addAggregation(sumAggregationBuilder).build();
SearchHits<CarDO> search = elasticsearchRestTemplate.search(build, CarDO.class);
Aggregations aggregations = search.getAggregations();
double sumPrice = 0;
if (aggregations != null) {
Sum sum = aggregations.get("sumPrice");
sumPrice = sum.getValue();
}
return sumPrice;
}
Max、Min(同时查询最大、最小值)
前面我们分别演示了最大值、最小值、平均值、求和的场景,其实如果有同时要求最大值、最小值、平均值、求和的场景的时候,我们只可以在同一次查询得到结果的,代码如下:
public void aggregationCarPrice() {
MaxAggregationBuilder maxAggregationBuilder = AggregationBuilders.max("maxPrice").field("price");
MinAggregationBuilder minAggregationBuilder = AggregationBuilders.min("minPrice").field("price");
AvgAggregationBuilder avgAggregationBuilder = AggregationBuilders.avg("avgPrice").field("price");
SumAggregationBuilder sumAggregationBuilder = AggregationBuilders.sum("sumPrice").field("price");
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
NativeSearchQuery build = queryBuilder.withQuery(QueryBuilders.matchAllQuery())
.addAggregation(maxAggregationBuilder)
.addAggregation(minAggregationBuilder)
.addAggregation(avgAggregationBuilder)
.addAggregation(sumAggregationBuilder)
.build();
SearchHits<CarDO> search = elasticsearchRestTemplate.search(build, CarDO.class);
Aggregations aggregations = search.getAggregations();
double maxPrice = 0;
if (aggregations != null) {
Max max = aggregations.get("maxPrice");
maxPrice = max.getValue();
Min min = aggregations.get("minPrice");
double minPrice = min.getValue();
Avg avg = aggregations.get("avgPrice");
double avgPrice = avg.getValue();
Sum sum = aggregations.get("sumPrice");
double sumPrice = sum.getValue();
log.info("汽车最贵价格:{},汽车最便宜价格:{},汽车平均价格:{},汽车价格之和:{}", maxPrice, minPrice, avgPrice, sumPrice);
}
}
可以看到我们使用一次查询就可以得到最大值、最小值、平均值、求和。
桶聚合
数值类字段桶聚合
我们统计一下不同价格的汽车各有多少个,实现这个业务也需要使用聚合查询,代码如下:
public void categoryPriceCountCar() {
TermsAggregationBuilder termsAggregationBuilder = AggregationBuilders.terms("priceCount").field("price");
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
NativeSearchQuery build = queryBuilder.withQuery(QueryBuilders.matchAllQuery())
.addAggregation(termsAggregationBuilder)
.build();
SearchHits<CarDO> search = elasticsearchRestTemplate.search(build, CarDO.class);
Aggregations aggregations = search.getAggregations();
if (aggregations != null) {
Terms terms = (Terms) aggregations.asMap().get("priceCount");
for (Terms.Bucket bucket : terms.getBuckets()) {
String price = bucket.getKeyAsString();
long priceCount = bucket.getDocCount();
log.info("汽车价格:{},有:{} 辆", price, priceCount);
}
}
}
字符串类型字段桶聚合
我们统计一下不同颜色的汽车各有多少辆,实现这个业务也需要使用聚合查询,使用按价格统计的方式来实现,代码如下:
public void categoryColorCountCar() {
TermsAggregationBuilder termsAggregationBuilder = AggregationBuilders.terms("colorCount").field("color");
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
NativeSearchQuery build = queryBuilder.withQuery(QueryBuilders.matchAllQuery())
.addAggregation(termsAggregationBuilder)
.build();
SearchHits<CarDO> search = elasticsearchRestTemplate.search(build, CarDO.class);
Aggregations aggregations = search.getAggregations();
if (aggregations != null) {
Terms terms = (Terms) aggregations.asMap().get("colorCount");
for (Terms.Bucket bucket : terms.getBuckets()) {
String color = bucket.getKeyAsString();
long colorCount = bucket.getDocCount();
log.info("汽车颜色:{},有:{} 辆", color, colorCount);
}
}
}
执行上述代码我们发现控制台报错如下:
org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=illegal_argument_exception, reason=Fielddata is disabled on text fields by default. Set fielddata=true on [color] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.]
at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:603) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:179) ~[elasticsearch-7.9.3.jar:7.9.3]
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892) ~[elasticsearch-rest-high-level-client-7.9.3.jar:7.9.3]
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869) ~[elasticsearch-rest-high-level-client-7.9.3.jar:7.9.3]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626) ~[elasticsearch-rest-high-level-client-7.9.3.jar:7.9.3]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583) ~[elasticsearch-rest-high-level-client-7.9.3.jar:7.9.3]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553) ~[elasticsearch-rest-high-level-client-7.9.3.jar:7.9.3]
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069) ~[elasticsearch-rest-high-level-client-7.9.3.jar:7.9.3]
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.lambda$search$10(ElasticsearchRestTemplate.java:256) ~[spring-data-elasticsearch-4.1.8.jar:4.1.8]
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.execute(ElasticsearchRestTemplate.java:343) ~[spring-data-elasticsearch-4.1.8.jar:4.1.8]
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.search(ElasticsearchRestTemplate.java:256) ~[spring-data-elasticsearch-4.1.8.jar:4.1.8]
at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.search(AbstractElasticsearchTemplate.java:446) ~[spring-data-elasticsearch-4.1.8.jar:4.1.8]
at com.order.service.service.impl.CarEsServiceImpl.categoryColorCountCar(CarEsServiceImpl.java:288) ~[classes/:na]
at com.order.service.controller.ElasticsearchController.categoryColorCountCar(ElasticsearchController.java:164) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) ~[spring-web-5.3.6.jar:5.3.6]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) ~[spring-web-5.3.6.jar:5.3.6]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) ~[spring-webmvc-5.3.6.jar:5.3.6]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) ~[spring-webmvc-5.3.6.jar:5.3.6]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.6.jar:5.3.6]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.6.jar:5.3.6]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1060) ~[spring-webmvc-5.3.6.jar:5.3.6]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:962) ~[spring-webmvc-5.3.6.jar:5.3.6]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.6.jar:5.3.6]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.3.6.jar:5.3.6]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:626) ~[tomcat-embed-core-9.0.45.jar:4.0.FR]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.6.jar:5.3.6]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733) ~[tomcat-embed-core-9.0.45.jar:4.0.FR]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.45.jar:9.0.45]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.6.jar:5.3.6]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.6.jar:5.3.6]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.6.jar:5.3.6]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.6.jar:5.3.6]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93) ~[spring-boot-actuator-2.4.5.jar:2.4.5]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.6.jar:5.3.6]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.6.jar:5.3.6]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.3.6.jar:5.3.6]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) ~[tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) [tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) [tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) [tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374) [tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) [tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707) [tomcat-embed-core-9.0.45.jar:9.0.45]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.45.jar:9.0.45]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.45.jar:9.0.45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
错误的核心信息如下:
[type=illegal_argument_exception, reason=Fielddata is disabled on text fields by default. Set fielddata=true on [color] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.]
大概得意思就是 color 这个字段有问题,上面我们使用 price 字段就可以,为啥 color 字段就有问题呢?
原因是因为 price 是数值类型的字段,而 color 是 text 类型的字段,在 Elasticsearch 中 text 类型的字段不能直接进行分组聚合,我们需要将其标记为 keyword 类型的字符串可以分组聚合。
在字段上增加如下如下注解:
//颜色
@Field(store = true, type = FieldType.Keyword)
private String color;
同时修改查询代码如下:
public void categoryColorCountCar() {
TermsAggregationBuilder termsAggregationBuilder = AggregationBuilders.terms("colorCount").field("color.keyword");
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
NativeSearchQuery build = queryBuilder.withQuery(QueryBuilders.matchAllQuery())
.addAggregation(termsAggregationBuilder)
.build();
SearchHits<CarDO> search = elasticsearchRestTemplate.search(build, CarDO.class);
Aggregations aggregations = search.getAggregations();
if (aggregations != null) {
Terms terms = (Terms) aggregations.asMap().get("colorCount");
for (Terms.Bucket bucket : terms.getBuckets()) {
String color = bucket.getKeyAsString();
long colorCount = bucket.getDocCount();
log.info("汽车颜色:{},有:{} 辆", color, colorCount);
}
}
}
执行结果如下:
2024-12-30 20:43:58.621 INFO 42108 --- [nio-8086-exec-2] c.o.s.service.impl.CarEsServiceImpl : 汽车颜色:钻石黑,有:2 辆
2024-12-30 20:43:58.621 INFO 42108 --- [nio-8086-exec-2] c.o.s.service.impl.CarEsServiceImpl : 汽车颜色:雅灰,有:2 辆
执行结果如何预期,上述编码的重点在 color.keyword,我们在查询的字段后面要带上 keyword。
嵌套聚合查询本篇暂不讲,在实现嵌套聚合查询的时候踩了很多坑,找了多种解决方法才实现,下一篇会对嵌套查询场景单独分享。
总结:本篇简单分享了使用 ElasticsearchRestTemplate 实现 Elasticsearch 的各种聚合查询,希望可以帮助到对 ElasticsearchRestTemplate API 使用不熟悉的朋友们。
如有不正确的地方欢迎各位指出纠正。