当前位置：首页 > article >正文

Easysearch 使用 AWS S3 进行快照备份与还原：完整指南及常见错误排查

article 2025/3/11 6:43:30

Easysearch 可以使用 AWS S3 作为远程存储库，进行索引的快照（Snapshot）备份和恢复。同时，Easysearch 内置了 S3 插件，无需额外安装。以下是完整的配置和操作步骤。

1. 在 AWS S3 上创建存储桶

登录 AWS 控制台，进入 S3 服务。
创建一个新存储桶（例如 easysearch-backups）。
启用版本控制（可选，但推荐）。
权限配置：确保 IAM 角色具有访问 S3 的权限。

{
"Version": "2012-10-17",
"Statement": [{
    "Action": [
      "s3:ListBucket"
    ],
    "Effect": "Allow",
    "Resource": [
      "arn:aws:s3:::s3-bucket-name"
    ]
  },
  {
    "Action": [
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject"
    ],
    "Effect": "Allow",
    "Resource": [
      "arn:aws:s3:::s3-bucket-name/*"
    ]
  }
]
}

2. 在 Console 上注册 S3 作为快照存储库

使用 Console DevTools 或 API

在 Easysearch 的 DevTools 执行：

PUT _snapshot/my_s3_repository
{
  "type": "s3",
  "settings": {
    "bucket": "easysearch-backups",
    "base_path": ""
  }
}

注意：

bucket 需要填写你的 S3 存储桶名称。
region 需要替换成你的 AWS S3 所在区域，SDK 默认美东区。
如果 Bucket 在中国区，还需添加 endpoint: https://s3.<region>.amazonaws.com.cn 参数。

3. 创建快照

一旦 my_s3_repository 注册完成，就可以创建快照：

PUT _snapshot/my_s3_repository/my_snapshot_001
{
  "indices": "my_index",
  "include_global_state": false
}

查看当前存储的快照：

GET _snapshot/my_s3_repository/_all

4. 从 AWS S3 还原快照

当你需要恢复索引时：

POST _snapshot/my_s3_repository/my_snapshot_001/_restore
{
  "indices": "my_index",
  "rename_pattern": "my_index",
  "rename_replacement": "restored_my_index"
}

说明：这会从 my_snapshot_001 还原 my_index，但以 restored_my_index 命名，避免与现有索引冲突。

如果要直接覆盖原索引（确保 my_index 为空或已删除）：

POST _snapshot/my_s3_repository/my_snapshot_001/_restore
{
  "indices": "my_index",
  "ignore_unavailable": true,
  "include_global_state": false
}

5. 可能的错误与解决方案

错误信息	可能原因	解决方案
`repository_s3 plugin not installed`	没有安装 `repository-s3` 插件	运行 `bin/elasticsearch-plugin install repository-s3` 并重启
`NoSuchBucket`	S3 存储桶不存在	确保 S3 存储桶名称正确
`AccessDenied`	权限不足	确保 S3 存储桶策略正确，检查 IAM 角色
`index_closed_exception`	目标索引已关闭	先 `POST my_index/_open` 再恢复
`index_already_exists_exception`	目标索引已存在	先 `DELETE my_index` 再恢复

6. 快照恢复常见错误排查

报错 1：无法连接到 S3


{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[my_s3_repository] path [/] is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[my_s3_repository] path [/] is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [//tests-sXkmh3q5ThCCIX2VJp609g/master.dat] using a single upload",
      "caused_by": {
        "type": "sdk_client_exception",
        "reason": "Failed to connect to service endpoint: ",
        "caused_by": {
          "type": "socket_timeout_exception",
          "reason": "Connect timed out"
        }
      }
    }
  },
  "status": 500
}

解决方案：

在 keystore 中添加 AWS 凭证：

sudo ./bin/easysearch-keystore add s3.client.default.access_key
sudo ./bin/easysearch-keystore add s3.client.default.secret_key

如果运行在 EC2 上，确保实例挂载了 IAM Role。

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[my_s3_repositor1] path  is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[my_s3_repositor1] path  is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [tests-sUUzs-mTSZeYw1qk372DkQ/master.dat] using a single upload",
      "caused_by": {
        "type": "sdk_client_exception",
        "reason": "The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/"
      }
    }
  },
  "status": 500
}

报错 2：索引已存在，无法恢复

{
  "error": {
    "root_cause": [
      {
        "type": "snapshot_restore_exception",
        "reason": "[my_s3_repository:1/9gIDCgSySwKzQqEYvaGM_w] cannot restore index [my_index] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name"
      }
    ],
    "type": "snapshot_restore_exception",
    "reason": "[my_s3_repository:1/9gIDCgSySwKzQqEYvaGM_w] cannot restore index [my_index] because an open index with same name already exists in the cluster. Either close or delete the existing index or restore the index under a different name by providing a rename pattern and replacement name"
  },
  "status": 500
}

解决方案：

删除现有索引后恢复：
```
DELETE /my_index
```
关闭索引后恢复：
```
POST /my_index/_close
```

恢复为新的索引名称：

POST _snapshot/my_s3_repository/1/_restore
{
  "indices": "my_index",
  "rename_pattern": "my_index",
  "rename_replacement": "restored_my_index"
}

报错 3：权限错误

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "no permissions for [] and User [name=admin, external_roles=[admin]]"
      }
    ],
    "type": "security_exception",
    "reason": "no permissions for [] and User [name=admin, external_roles=[admin]]"
  },
  "status": 403
}

解决方案：

确保用户有 manage_snapshots 角色权限。
排除 .security 索引或全局状态，否则无法恢复。

在这里插入图片描述

POST _snapshot/my_s3_repositor1/snapshot_002/_restore
{
  "indices": "-.security",
  "ignore_unavailable": true,
  "include_global_state": false
}

📌 存储库（Repository）管理 API

存储库用于存储快照，Elasticsearch 支持 AWS S3、GCS、本地等存储。

1️⃣ 查看所有已注册的存储库

GET _snapshot/_all

示例返回：

{
  "my_s3_repository": {
    "type": "s3",
    "settings": {
      "bucket": "es-snapshots-bucket",
      "region": "us-east-1"
    }
  }
}

2️⃣ 查看特定存储库信息

GET _snapshot/my_s3_repository

3️⃣ 创建存储库（AWS S3 示例）

PUT _snapshot/my_s3_repository
{
  "type": "s3",
  "settings": {
    "bucket": "es-snapshots-bucket",
  }
}

4️⃣ 删除存储库

DELETE _snapshot/my_s3_repository

⚠ 删除存储库不会删除快照，需要手动删除快照！

📌 快照（Snapshot）管理 API

快照用于备份和恢复索引数据。

1️⃣ 创建快照

备份特定索引

PUT _snapshot/my_s3_repository/snapshot_001
{
  "indices": "my_index",
  "include_global_state": false
}

备份所有索引

PUT _snapshot/my_s3_repository/snapshot_002
{
  "include_global_state": true
}

2️⃣ 查看所有快照

GET _snapshot/my_s3_repository/_all

3️⃣ 查看特定快照信息

GET _snapshot/my_s3_repository/snapshot_001

4️⃣ 删除快照

DELETE _snapshot/my_s3_repository/snapshot_001

📌 快照恢复（Restore）API

恢复已备份的索引。

1️⃣ 还原单个索引

POST _snapshot/my_s3_repository/snapshot_001/_restore
{
  "indices": "my_index",
  "ignore_unavailable": true,
  "include_global_state": false
}

2️⃣ 还原索引并重命名

POST _snapshot/my_s3_repository/snapshot_001/_restore
{
  "indices": "my_index",
  "rename_pattern": "my_index",
  "rename_replacement": "restored_my_index"
}

3️⃣ 还原所有索引

POST _snapshot/my_s3_repository/snapshot_002/_restore

📌 快照状态 API

查询快照的执行状态。

1️⃣ 查看当前快照任务

GET _snapshot/_status

2️⃣ 查看特定快照状态

GET _snapshot/my_s3_repository/snapshot_001/_status

API	用途
`GET _snapshot/_all`	查看所有存储库
`GET _snapshot/my_s3_repository`	查看特定存储库
`PUT _snapshot/my_s3_repository`	创建存储库
`DELETE _snapshot/my_s3_repository`	删除存储库
`PUT _snapshot/my_s3_repository/snapshot_001`	创建快照
`GET _snapshot/my_s3_repository/_all`	查看所有快照
`GET _snapshot/my_s3_repository/snapshot_001`	查看快照详情
`DELETE _snapshot/my_s3_repository/snapshot_001`	删除快照
`POST _snapshot/my_s3_repository/snapshot_001/_restore`	还原快照
`GET _snapshot/_status`	查看快照状态