当前位置：首页 > article >正文

在Openshift（K8S）上通过EMQX Operator部署Emqx集群

article 2025/1/18 7:24:53

EMQX Operator 简介

EMQX Broker/Enterprise 是一个云原生的 MQTT 消息中间件。我们提供了 EMQX Kubernetes Operator 来帮助您在 Kubernetes 的环境上快速创建和管理 EMQX Broker/Enterprise 集群。它可以大大简化部署和管理 EMQX 集群的流程，对于管理和配置的知识要求也更低。它把部署和管理的工作变成一种低成本的、标准化的、可重复性的能力。

EMQX Operator 包括但不限于以下功能：

简化 EMQX 部署：通过 EMQX 自定义资源声明 EMQX 集群，并快速的部署，更多的内容，请查看快速开始。

管理 EMQX 集群：对 EMQX 进行自动化运维操作，包括集群升级、运行时数据持久化、根据 EMQX 的状态更新 Kubernetes 的资源等，更多的内容，请查看管理 EMQX 集群。

EMQX 与 EMQX Operator 的兼容性列表

EMQX 企业版

EMQX 企业版	EMQX Operator Version	APIVersion	Kind
4.3.x （包含）～ 4.4	1.2.1, 1.2.2, 1.2.3	apps.emqx.io/v1beta3	EmqxEnterprise
4.4.6 （包含）～ 4.4.8	1.2.5	apps.emqx.io/v1beta3	EmqxEnterprise
4.4.8 （包含）～ 4.4.14	1.2.6, 1.2.7, 1.2.8, 2.0.0, 2.0.1, 2.0.2, 2.0.3	apps.emqx.io/v1beta3	EmqxEnterprise
4.4.14 （包含）或更高 4.4.x	2.1.0, 2.1.1	apps.emqx.io/v1beta4	EmqxEnterprise
5.0.0 （包含）～ 5.0.23	2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.1.0, 2.1.1	apps.emqx.io/v2alpha1	EMQX
5.1.1 或更高	2.2.0	apps.emqx.io/v2beta1	EMQX

EMQX 开源版

EMQX 开源版	EMQX Operator Version	APIVersion	Kind
4.3.x （包含）～ 4.4	1.2.1, 1.2.2, 1.2.3	apps.emqx.io/v1beta3	EmqxBroker
4.4.6 （包含）～ 4.4.8	1.2.5	apps.emqx.io/v1beta3	EmqxBroker
4.4.8 （包含）～ 4.4.14	1.2.6, 1.2.7, 1.2.8, 2.0.0, 2.0.1, 2.0.2, 2.0.3	apps.emqx.io/v1beta3	EmqxBroker
4.4.14 或更高 4.4.x	2.1.0, 2.1.1	apps.emqx.io/v1beta4	EmqxBroker
5.0.6 （包含）～ 5.0.8	2.0.0, 2.0.1, 2.0.3	apps.emqx.io/v2alpha1	EMQX
5.0.8 （包含）～ 5.0.14	2.0.2	apps.emqx.io/v2alpha1	EMQX
5.0.14 （包含）～ 5.0.23	2.1.0, 2.1.1	apps.emqx.io/v2alpha1	EMQX
5.1.1 或更高	2.2.0	apps.emqx.io/v2beta1	EMQX

如何选择 Kubernetes 版本

EMQX Operator 要求 Kubernetes 集群的版本号 >=1.24。

Kubernetes 版本	EMQX Operator 兼容性	注释
1.24 更高	支持所有功能
1.22 ( 包含) ～ 1.23	支持，但是不包含 MixedProtocolLBService	EMQX 集群只能在 LoadBalancer 类型的 Service 中使用一个协议，例如 TCP 或 UDP。
1.21 ( 包含) ～ 1.22	支持，但是不包含 Pod 删除开销	EMQX Core + Replicant 模式集群时，更新 EMQX 集群无法准确的删除 Pod。
1.20 ( 包含) ～ 1.21	支持，但是如果使用 `NodePort` 类型的 Service，需要手动管理 `.spec.ports[].nodePort`	更多的详情，请查看 Kubernetes changelog.
1.16 ( 包含) ～ 1.20	支持，但是不推荐，因为缺乏足够的测试
低于 1.16	不支持	低于 1.16 版本的 Kubernetes 不支持 `apiextensions/v1` APIVersion。

快速开始

在本文中，我们将指导您完成高效设置 EMQX Operator 环境、安装 EMQX Operator，然后使用它部署 EMQX 所需的步骤。通过遵循本节中概述的指南，您将能够使用 EMQX Operator 有效地安装和管理 EMQX。

准备环境

在部署 EMQX Operator 之前，请确认以下组件已经准备就绪：

一个正在运行的 Kubernetes 集群，关于 Kubernetes 的版本，请查看如何选择 Kubernetes 版本
一个可以访问 Kubernetes 集群的 kubectl 工具。您可以使用 kubectl cluster-info 命令检查 Kubernetes 集群的状态。
Helm 3 或更高

安装 EMQX Operator

安装 cert-manger。

TIP

需要 cert-manager 版本 1.1.6 或更高。如果 cert-manager 已经安装并启动，请跳过此步骤。

你可以使用 Helm 来安装 cert-manager。

$ helm repo add jetstack https://charts.jetstack.io
$ helm repo update
$ helm upgrade --install cert-manager jetstack/cert-manager \
 --namespace cert-manager \
 --create-namespace \
 --set installCRDs=true

或者按照 cert-manager 安装指南来安装它。

WARNING

如果您在 Google Kubernetes Engine（GKE）上安装它。那么通过默认配置安装可能会导致 bootstraping 问题。所以通过增加 --set global.leaderElection.namespace=cert-manager 这个配置为 leader 选举使用不同的命名空间。查看 cert-manager 兼容性

运行以下命令来安装 EMQX Operator。

bash

$ helm repo add emqx https://repos.emqx.io/charts
$ helm repo update
$ helm upgrade --install emqx-operator emqx/emqx-operator \
  --namespace emqx-operator-system \
  --create-namespace

等待 EMQX Operator 就绪。

$ kubectl wait --for=condition=Ready pods -l "control-plane=controller-manager" -n emqx-operator-system

pod/emqx-operator-controller-manager-57bd7b8bd4-h2mcr condition met

现在你已经成功的安装 EMQX Operator，你可以继续下一步了。在部署 EMQX 部分中，您将学习如何使用 EMQX Operator 来部署 EMQX。

部署 EMQX

EMQX Enterprise 5

apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
   name: emqx-ee
spec:
   image: emqx/emqx-enterprise:5.6

EMQX Open Source 5

apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
   name: emqx
spec:
   image: emqx:5

EMQX Enterprise 4

apiVersion: apps.emqx.io/v1beta4
kind: EmqxEnterprise
metadata:
   name: emqx-ee
spec:
   template:
     spec:
       emqxContainer:
         image:
           repository: emqx/emqx-ee
           version: 4.4.19

EMQX Open Source 4

apiVersion: apps.emqx.io/v1beta4
kind: EmqxBroker
metadata:
   name: emqx
spec:
   template:
     spec:
       emqxContainer:
         image:
           repository: emqx
           version: 4.4.19

将下面的 YAML 配置文件保存为 emqx.yaml。

apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
   name: emqx-ee
spec:
   image: emqx/emqx-enterprise:5.6

并使用 kubectl apply 命令来部署 EMQX。

$ kubectl apply -f emqx.yaml

关于 EMQX 自定义资源的更多信息，请查看 API 参考

检查 EMQX 集群状态，请确保 STATUS 为 Running，这可能需要一些时间等待 EMQX 集群准备就绪。

$ kubectl get emqx

NAME      IMAGE                        STATUS    AGE
emqx-ee   emqx/emqx-enterprise:5.1.0   Running   2m55s

在OCP上部署EMQX

1.安装 `cert-manger`。

cert-manager Operator for Red Hat OpenShift

安装完

namespace : cert-manager-operator下效果如下
在这里插入图片描述

namespace : cert-manager下效果如下

在这里插入图片描述

2.安装EMQX Operator

版本选择：

EMQX 开源版：4.4.14

EMQX Operator Version ： 2.2.22

$ helm search repo emqx/emqx-operator --versions
$ helm pull emqx/emqx-operator --version 2.2.22
# 解压后安装
$ helm install emqx-operator -f values.yaml .

$ oc wait --for=condition=Ready pods -l "control-plane=controller-manager" -n emqx-operator-system
pod/emqx-operator-controller-manager condition met

3.安装EMQX

storageClassName 填写为自己的。

apiVersion: apps.emqx.io/v1beta4
kind: EmqxBroker
metadata:
   name: emqx
spec:
  persistent:
    metadata:
      name: emqx
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 200Mi
      storageClassName: xxxx
  template:
    spec:
      emqxContainer:
        image:
          repository: emqx
          version: 4.4.14
        volumeMounts:
          - mountPath: /opt/emqx/etc/emqx.conf
            name: emqx-config-volume
            subPath: emqx.conf
      volumes:
        - configMap:
            name: emqx-config
          name: emqx-config-volume

声明emqx-config,可以根据需求自己修改。

kind: ConfigMap
apiVersion: v1
metadata:
  name: emqx-config
data:
  emqx.conf: |-
    cluster.name = emqxcl
    cluster.proto_dist = inet_tcp
    cluster.discovery = manual
    cluster.autoheal = on
    cluster.autoclean = 5m
    node.name = emqx@127.0.0.1
    node.cookie = emqxsecretcookie
    node.data_dir = data
    node.global_gc_interval = 15m
    node.crash_dump = log/crash.dump
    node.dist_use_interface = 0.0.0.0
    node.dist_listen_min = 6369
    node.dist_listen_max = 6369
    node.backtrace_depth = 16
    rpc.mode = async
    rpc.async_batch_size = 256
    rpc.port_discovery = stateless
    rpc.connect_timeout = 5s
    rpc.send_timeout = 5s
    rpc.authentication_timeout = 5s
    rpc.call_receive_timeout = 15s
    rpc.socket_keepalive_idle = 900s
    rpc.socket_keepalive_interval = 75s
    rpc.socket_keepalive_count = 9
    rpc.socket_sndbuf = 1MB
    rpc.socket_recbuf = 1MB
    rpc.socket_buffer = 1MB
    log.to = file
    log.level = warning
    log.dir = log
    log.file = emqx.log
    log.formatter = text
    log.rotation = on
    log.rotation.size = 10MB
    log.rotation.count = 5
    listener.tcp.external.proxy_protocol = on
    allow_anonymous = true
    acl_nomatch = allow
    acl_file = etc/acl.conf
    enable_acl_cache = on
    acl_cache_max_size = 32
    acl_cache_ttl = 1m
    acl_deny_action = ignore
    flapping_detect_policy = 30, 1m, 5m
    auth_order = none
    acl_order = none
    mqtt.max_packet_size = 1MB
    mqtt.max_clientid_len = 65535
    mqtt.max_topic_levels = 128
    mqtt.max_qos_allowed = 2
    mqtt.max_topic_alias = 65535
    mqtt.retain_available = true
    mqtt.wildcard_subscription = true
    mqtt.shared_subscription = true
    mqtt.exclusive_subscription = false
    mqtt.ignore_loop_deliver = false
    mqtt.strict_mode = false
    zone.external.idle_timeout = 15s
    zone.external.enable_acl = on
    zone.external.enable_ban = on
    zone.external.enable_stats = on
    zone.external.acl_deny_action = ignore
    zone.external.force_gc_policy = 16000|16MB
    zone.external.keepalive_backoff = 0.75
    zone.external.max_subscriptions = 0
    zone.external.upgrade_qos = off
    zone.external.max_inflight = 32
    zone.external.retry_interval = 30s
    zone.external.max_awaiting_rel = 100
    zone.external.await_rel_timeout = 300s
    zone.external.session_expiry_interval = 2h
    zone.external.max_mqueue_len = 1000
    zone.external.mqueue_priorities = none
    zone.external.mqueue_default_priority = highest
    zone.external.mqueue_store_qos0 = true
    zone.external.enable_flapping_detect = off
    zone.external.use_username_as_clientid = false
    zone.external.ignore_loop_deliver = false
    zone.external.strict_mode = false
    zone.internal.allow_anonymous = true
    zone.internal.enable_stats = on
    zone.internal.enable_acl = off
    zone.internal.acl_deny_action = ignore
    zone.internal.max_subscriptions = 0
    zone.internal.max_inflight = 128
    zone.internal.max_awaiting_rel = 1000
    zone.internal.max_mqueue_len = 10000
    zone.internal.mqueue_store_qos0 = true
    zone.internal.enable_flapping_detect = off
    zone.internal.ignore_loop_deliver = false
    zone.internal.strict_mode = false
    zone.internal.bypass_auth_plugins = true
    listener.tcp.external = 0.0.0.0:1883
    listener.tcp.external.acceptors = 8
    listener.tcp.external.max_connections = 1024000
    listener.tcp.external.max_conn_rate = 1000
    listener.tcp.external.active_n = 100
    listener.tcp.external.zone = external
    listener.tcp.external.access.1 = allow all
    listener.tcp.external.backlog = 1024
    listener.tcp.external.send_timeout = 15s
    listener.tcp.external.send_timeout_close = on
    listener.tcp.external.nodelay = true
    listener.tcp.external.reuseaddr = true
    listener.tcp.internal = 127.0.0.1:11883
    listener.tcp.internal.acceptors = 4
    listener.tcp.internal.max_connections = 1024000
    listener.tcp.internal.max_conn_rate = 1000
    listener.tcp.internal.active_n = 1000
    listener.tcp.internal.zone = internal
    listener.tcp.internal.backlog = 512
    listener.tcp.internal.send_timeout = 5s
    listener.tcp.internal.send_timeout_close = on
    listener.tcp.internal.recbuf = 64KB
    listener.tcp.internal.sndbuf = 64KB
    listener.tcp.internal.nodelay = false
    listener.tcp.internal.reuseaddr = true
    listener.ssl.external = 8883
    listener.ssl.external.acceptors = 16
    listener.ssl.external.max_connections = 102400
    listener.ssl.external.max_conn_rate = 500
    listener.ssl.external.active_n = 100
    listener.ssl.external.zone = external
    listener.ssl.external.access.1 = allow all
    listener.ssl.external.handshake_timeout = 15s
    listener.ssl.external.keyfile = etc/certs/key.pem
    listener.ssl.external.certfile = etc/certs/cert.pem
    listener.ssl.external.cacertfile = etc/certs/cacert.pem
    crl_cache_http_timeout = 15s
    crl_cache_refresh_interval = 15m
    listener.ssl.external.ciphers = TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_CHACHA20_POLY1305_SHA256,TLS_AES_128_CCM_SHA256,TLS_AES_128_CCM_8_SHA256,ECDHE-ECDSA-AES256-GCM-SHA384,ECDHE-RSA-AES256-GCM-SHA384,ECDHE-ECDSA-AES256-SHA384,ECDHE-RSA-AES256-SHA384,ECDHE-ECDSA-DES-CBC3-SHA,ECDH-ECDSA-AES256-GCM-SHA384,ECDH-RSA-AES256-GCM-SHA384,ECDH-ECDSA-AES256-SHA384,ECDH-RSA-AES256-SHA384,DHE-DSS-AES256-GCM-SHA384,DHE-DSS-AES256-SHA256,AES256-GCM-SHA384,AES256-SHA256,ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-RSA-AES128-GCM-SHA256,ECDHE-ECDSA-AES128-SHA256,ECDHE-RSA-AES128-SHA256,ECDH-ECDSA-AES128-GCM-SHA256,ECDH-RSA-AES128-GCM-SHA256,ECDH-ECDSA-AES128-SHA256,ECDH-RSA-AES128-SHA256,DHE-DSS-AES128-GCM-SHA256,DHE-DSS-AES128-SHA256,AES128-GCM-SHA256,AES128-SHA256,ECDHE-ECDSA-AES256-SHA,ECDHE-RSA-AES256-SHA,DHE-DSS-AES256-SHA,ECDH-ECDSA-AES256-SHA,ECDH-RSA-AES256-SHA,AES256-SHA,ECDHE-ECDSA-AES128-SHA,ECDHE-RSA-AES128-SHA,DHE-DSS-AES128-SHA,ECDH-ECDSA-AES128-SHA,ECDH-RSA-AES128-SHA,AES128-SHA
    listener.ssl.external.recbuf = 4KB
    listener.ssl.external.sndbuf = 4KB
    listener.ssl.external.reuseaddr = true
    listener.ws.external = 8083
    listener.ws.external.mqtt_path = /ws
    listener.ws.external.acceptors = 4
    listener.ws.external.max_connections = 102400
    listener.ws.external.max_conn_rate = 1000
    listener.ws.external.active_n = 100
    listener.ws.external.zone = external
    listener.ws.external.access.1 = allow all
    listener.ws.external.backlog = 1024
    listener.ws.external.send_timeout = 15s
    listener.ws.external.send_timeout_close = on
    listener.ws.external.nodelay = true
    listener.ws.external.mqtt_piggyback = multiple
    listener.ws.external.check_origin_enable = false
    listener.ws.external.allow_origin_absence = true
    listener.ws.external.check_origins = http://localhost:18083, http://127.0.0.1:18083
    listener.wss.external = 8084
    listener.wss.external.mqtt_path = /ws
    listener.wss.external.acceptors = 4
    listener.wss.external.max_connections = 102400
    listener.wss.external.max_conn_rate = 1000
    listener.wss.external.active_n = 100
    listener.wss.external.zone = external
    listener.wss.external.access.1 = allow all
    listener.wss.external.keyfile = etc/certs/key.pem
    listener.wss.external.certfile = etc/certs/cert.pem
    listener.wss.external.ciphers = TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256,TLS_CHACHA20_POLY1305_SHA256,TLS_AES_128_CCM_SHA256,TLS_AES_128_CCM_8_SHA256,ECDHE-ECDSA-AES256-GCM-SHA384,ECDHE-RSA-AES256-GCM-SHA384,ECDHE-ECDSA-AES256-SHA384,ECDHE-RSA-AES256-SHA384,ECDHE-ECDSA-DES-CBC3-SHA,ECDH-ECDSA-AES256-GCM-SHA384,ECDH-RSA-AES256-GCM-SHA384,ECDH-ECDSA-AES256-SHA384,ECDH-RSA-AES256-SHA384,DHE-DSS-AES256-GCM-SHA384,DHE-DSS-AES256-SHA256,AES256-GCM-SHA384,AES256-SHA256,ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-RSA-AES128-GCM-SHA256,ECDHE-ECDSA-AES128-SHA256,ECDHE-RSA-AES128-SHA256,ECDH-ECDSA-AES128-GCM-SHA256,ECDH-RSA-AES128-GCM-SHA256,ECDH-ECDSA-AES128-SHA256,ECDH-RSA-AES128-SHA256,DHE-DSS-AES128-GCM-SHA256,DHE-DSS-AES128-SHA256,AES128-GCM-SHA256,AES128-SHA256,ECDHE-ECDSA-AES256-SHA,ECDHE-RSA-AES256-SHA,DHE-DSS-AES256-SHA,ECDH-ECDSA-AES256-SHA,ECDH-RSA-AES256-SHA,AES256-SHA,ECDHE-ECDSA-AES128-SHA,ECDHE-RSA-AES128-SHA,DHE-DSS-AES128-SHA,ECDH-ECDSA-AES128-SHA,ECDH-RSA-AES128-SHA,AES128-SHA
    listener.wss.external.backlog = 1024
    listener.wss.external.send_timeout = 15s
    listener.wss.external.send_timeout_close = on
    listener.wss.external.mqtt_piggyback = multiple
    listener.wss.external.check_origin_enable = false
    listener.wss.external.allow_origin_absence = true
    listener.wss.external.check_origins = https://localhost:8084, https://127.0.0.1:8084
    modules.loaded_file = data/loaded_modules
    module.presence.qos = 1
    plugins.etc_dir = etc/plugins/
    plugins.loaded_file = data/loaded_plugins
    plugins.expand_plugins_dir = etc/plugins/
    broker.sys_interval = 1m
    broker.sys_heartbeat = 30s
    broker.session_locking_strategy = quorum
    broker.shared_subscription_strategy = random
    broker.shared_dispatch_ack_enabled = false
    broker.route_batch_clean = off
    sysmon.long_gc = 0
    sysmon.long_schedule = 240ms
    sysmon.large_heap = 8MB
    sysmon.busy_port = false
    sysmon.busy_dist_port = true
    os_mon.cpu_check_interval = 60s
    os_mon.cpu_high_watermark = 80%
    os_mon.cpu_low_watermark = 60%
    os_mon.mem_check_interval = 60s
    os_mon.sysmem_high_watermark = 70%
    os_mon.procmem_high_watermark = 5%
    vm_mon.check_interval = 30s
    vm_mon.process_high_watermark = 80%
    vm_mon.process_low_watermark = 60%
    alarm.actions = log,publish
    alarm.size_limit = 1000
    alarm.validity_period = 24h

默认的Operator不会安装dashborad，所以下一步我们需要安装

service.yaml

kind: Service
apiVersion: v1
metadata:
  name: emqx
spec:
  ports:
    - name: http-management-8081
      protocol: TCP
      port: 8081
      targetPort: 8081
    - name: http-dashboard-18083
      protocol: TCP
      port: 18083
      targetPort: 18083
  internalTrafficPolicy: Cluster
  type: ClusterIP
  selector:
    apps.emqx.io/instance: emqx
    apps.emqx.io/managed-by: emqx-operator

route.yaml
host字段需要修改

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: emqx-dashborad
spec:
  host: xxxxxx
  to:
    kind: Service
    name: emqx
    weight: 100
  port:
    targetPort: http-dashboard-18083
  wildcardPolicy: None
  tls: null