当前位置：首页 > article >正文

java查询es超过10000条数据

article 2025/3/25 21:58:29

java查询es超过10000条数据

背景:需要每天零点导出es中日志数据到数据库中给数据分析人员做清洗，然后展示给业务人员。但在es中默认一次最多只能查询10000条数据。
在这里我就只贴一下关键代码

SearchRequest searchRequest = new SearchRequest("索引名");
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        //时间字段
RangeQueryBuilder rangeQuery = QueryBuilders.rangeQuery("logdate")
                .from("2025-03-20 00:00") // 设置开始时间
                .to("2025-03-20 23:00")         // 设置结束时间
                .format("yyyy-MM-dd HH:mm")    //设置时间格式
                .includeLower(true)            // 包含起始时间点
                .includeUpper(false);          // 不包含结束时间点
searchSourceBuilder.query(rangeQuery);
searchSourceBuilder.size(1000); //每批次大小
SortBuilder sortBuilder = SortBuilders.fieldSort("logDate").order(SortOrder.ASC);
searchSourceBuilder.sort(sortBuilder);        searchSourceBuilder.sort(SortBuilders.fieldSort("_id").order(SortOrder.ASC));
Object[] searchAfter = null;
do {
       if (searchAfter != null){
             searchSourceBuilder.searchAfter(searchAfter);
        }
       searchRequest.source(searchSourceBuilder);
       SearchResponse search = null;
       try {
            search = client.search(searchRequest, RequestOptions.DEFAULT);
       } catch (IOException e) {
            throw new RuntimeException(e);
       }
        SearchHit[] hits = search.getHits().getHits();
         if (hits.length==0){
              break;
          }
        for (SearchHit hit : hits) {
                System.out.println(hit.getSourceAsString());
         }
            //获取最后一个文档的排序值
            searchAfter = hits[hits.length-1].getSortValues();
        }while (searchAfter != null);
        System.out.println("总共查询到 "+i+" 条数据");
    }

上面这段代码的本质就是用的分页查询
所用es相关依赖为

<dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-elasticsearch</artifactId>
            <version>2.5.14</version>
        </dependency>