当前位置：首页 > article >正文

ElasticSearch备考 -- Manage the index lifecycle (ILM)

article 2024/11/6 12:04:39

一、题目

在集群中，数据首先分布在data_hot节点，rollover 设置max_age:3d, max_docs:5,max_size:50gb, 优先级为100。

max_age:15s, forcemarge 段合并，数据迁移到data_warm节点，副本数为0，优先级为50

max_age:30s, 数据从data_warm迁移到data_cold节点

max_age:60s, 数据删除

运用ilm，创建index template名字为task_ilm_template,匹配模式task_ilm-*，并指定ilm 策略，创建索引task_ilm_00001注意分布情况

二、思考

索引生命周期管理ILM是对索引数据管理的一种自动化方式，主要是通过policy+索引模版结合使用。

通过创建ILM，在policy中可以指定多个阶段，生命周期阶段包括：

Hot: The index is actively being updated and queried.
Warm: The index is no longer being updated but is still being queried.
Cold: The index is no longer being updated and is queried infrequently. The information still needs to be searchable, but it’s okay if those queries are slower.
Frozen: The index is no longer being updated and is queried rarely. The information still needs to be searchable, but it’s okay if those queries are extremely slow.
Delete: The index is no longer needed and can safely be removed.

每个阶段可以制定不同的动作action，action主要有以下：

Rollover: Creates a new write index when the current one reaches a certain size, number of docs, or age.
Shrink: Reduces the number of primary shards in an index.
Force merge: Triggers a force merge to reduce the number of segments in an index’s shards.
Delete: Permanently remove an index, including all of its data and metadata.

通过创建索引模版并配置ILM策略，可以在使用模版创建索引后，对数据起到一个自动化生命周期管理的目的。

三、解题

Step 1、前置工作修改集群

1）集群节点角色配置

node-1：data_hot, data_content
node-2：data_warm, data_content
node-3：data_cold, data_content

2）索引生命周期刷新间隔

对于生产环境非必要配置，主要是为了索引生命周期刷新轮询间隔为1s，在验证时候能快速看到效果。

PUT _cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval": "1s"
  }
}

Step 2、创建ILM策略

可以通过命令行和图形化配置两种方式

【命令行方式】

PUT _ilm/policy/task_test_policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          },
          "rollover": {
            "max_size": "50gb",
            "max_primary_shard_size": "50gb",
            "max_age": "3d",
            "max_docs": 5
          }
        }
      },
      "warm": {
        "min_age": "15s",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1,
            "index_codec": "best_compression"
          },
          "set_priority": {
            "priority": 50
          },
          "allocate": {
            "number_of_replicas": 0
          }
        }
      },
      "cold": {
        "min_age": "30s",
        "actions": {
          "readonly": {}
        }
      },
      "delete": {
        "min_age": "60s",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

【图形化方式】

打开kibana在左侧菜单中找到management展开点击stack management

在stack management中找到index lifecycle polices菜单，点击右侧的create ploicy的按钮

配置hot phase

配置 warm phase

配置 cold phase

配置 delete phase，需要在 cold phase中点击删除按钮，会增加一个delete phase

Step 3、创建索引模版配置ILM策略

注意：在创建索引模版的时候要指定lifecycle，也就是刚刚创建的ILM策略。此外，还要指定rollover滚动操作别名，缺少别名创建的index也不能正常执行ILM策略。

【命令行方式】

PUT _index_template/task_ilm_template
{
  "settings": {
    "index": {
      "lifecycle": {
        "name": "task_test_policy",
        "rollover_alias": "task_ilm"
      },
      "routing": {
        "allocation": {
          "include": {
            "_tier_preference": null
          }
        }
      },
      "number_of_shards": "5",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "properties": {
      "age": {
        "type": "integer"
      },
      "name": {
        "type": "text"
      }
    }
  },
  "aliases": {}
}

【图像界面方式】

在左侧菜单中点击data，在右侧主页面中选择index template 标签页，并点击create template按钮

创建索引模版操作如下

此处需要注意：

1）匹配模式最后通配符要使用“-”中杠

2）需要指定配置，重点需要配置数据节点

创建模版中所需的字段及类型

最后点击Create template按钮

Step 4、创建索引验证

注意这里有两个点

1）需要指定索引的别名，并且可以基于别名进行写入操作

2）数据的批量写入操作是基于别名的

# DELETE task_ilm-000001  
PUT task_ilm-000001 
{
  "aliases": {
    "task_ilm": {
       "is_write_index":true
    }
  }
}

POST task_ilm/_bulk
{"create":{"_id":1}}
{"name":"key 1","age":1}
{"create":{"_id":2}}
{"name":"key 2","age":2}
{"create":{"_id":3}}
{"name":"key 3","age":3}
{"create":{"_id":4}}
{"name":"key 4","age":4}
{"create":{"_id":5}}
{"name":"key 5","age":5}

POST task_ilm/_bulk
{"create":{"_id":6}}
{"name":"key 6","age":6}

创建后索引获取索引配置