k8s部署juicefs

本文k8s较老采用老版本的juicefs，中间件也都是部署在k8s上。测试是否能成功创建动态pvc挂在到测试pod当中并查看到数据信息。一些偏理论知识就不多说了，直接开始操作

官方演示视频：​​​​​​JuiceFS CSI Driver 简介 - Office Hours 20220629_哔哩哔哩_bilibili 

官方文档：安装 | JuiceFS Document Center

 k8s部署mysql-CSDN博客

k8s部署minio集群-CSDN博客

参考以上两篇文章将中间件依赖环境先搭建起来

准备juicefs部署文件

k8s.yaml（juicefs）

# DO NOT EDIT: generated by 'kustomize build'
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-controller-sa
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-node-sa
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-external-node-service-role
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - create
  - update
  - delete
  - patch
  - watch
- apiGroups:
  - ""
  resources:
  - pods/log
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - create
  - update
  - delete
  - patch
- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - get
  - create
  - update
  - delete
  - patch
- apiGroups:
  - ""
  resources:
  - nodes/proxy
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - persistentvolumes
  verbs:
  - get
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-external-provisioner-role
rules:
- apiGroups:
  - ""
  resources:
  - persistentvolumes
  verbs:
  - get
  - list
  - watch
  - create
  - delete
- apiGroups:
  - ""
  resources:
  - persistentvolumeclaims
  verbs:
  - get
  - list
  - watch
  - update
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - list
  - watch
  - create
  - update
  - patch
- apiGroups:
  - storage.k8s.io
  resources:
  - csinodes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
  - create
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - pods
  - pods/log
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
  - delete
- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - get
  - create
  - update
  - patch
  - delete
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - get
  - list
  - watch
  - create
  - update
  - patch
- apiGroups:
  - apps
  resources:
  - daemonsets
  verbs:
  - get
  - list
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - get
  - watch
  - list
  - delete
  - update
  - create
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - watch
  - list
  - delete
  - update
  - create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-node-service-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: juicefs-csi-external-node-service-role
subjects:
- kind: ServiceAccount
  name: juicefs-csi-node-sa
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-provisioner-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: juicefs-external-provisioner-role
subjects:
- kind: ServiceAccount
  name: juicefs-csi-controller-sa
  namespace: kube-system
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-controller
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: juicefs-csi-controller
      app.kubernetes.io/instance: juicefs-csi-driver
      app.kubernetes.io/name: juicefs-csi-driver
      app.kubernetes.io/version: master
  serviceName: juicefs-csi-controller
  template:
    metadata:
      labels:
        app: juicefs-csi-controller
        app.kubernetes.io/instance: juicefs-csi-driver
        app.kubernetes.io/name: juicefs-csi-driver
        app.kubernetes.io/version: master
    spec:
      containers:
      - args:
        - --endpoint=$(CSI_ENDPOINT)
        - --logtostderr
        - --nodeid=$(NODE_NAME)
        - --leader-election
        - --v=5
        env:
        - name: CSI_ENDPOINT
          value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: JUICEFS_MOUNT_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: JUICEFS_MOUNT_PATH
          value: /var/lib/juicefs/volume
        - name: JUICEFS_CONFIG_PATH
          value: /var/lib/juicefs/config
        image: juicedata/juicefs-csi-driver:v0.19.0
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: healthz
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 3
        name: juicefs-plugin
        ports:
        - containerPort: 9909
          name: healthz
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 512Mi
        securityContext:
          capabilities:
            add:
            - SYS_ADMIN
          privileged: true
        volumeMounts:
        - mountPath: /var/lib/csi/sockets/pluginproxy/
          name: socket-dir
        - mountPath: /jfs
          mountPropagation: Bidirectional
          name: jfs-dir
        - mountPath: /root/.juicefs
          mountPropagation: Bidirectional
          name: jfs-root-dir
      - args:
        - --csi-address=$(ADDRESS)
        - --timeout=60s
        - --enable-leader-election
        - --v=5
        env:
        - name: ADDRESS
          value: /var/lib/csi/sockets/pluginproxy/csi.sock
        image: quay.io/k8scsi/csi-provisioner:v1.6.0
        name: csi-provisioner
        volumeMounts:
        - mountPath: /var/lib/csi/sockets/pluginproxy/
          name: socket-dir
      - args:
        - --csi-address=$(ADDRESS)
        - --health-port=$(HEALTH_PORT)
        env:
        - name: ADDRESS
          value: /csi/csi.sock
        - name: HEALTH_PORT
          value: "9909"
        image: quay.io/k8scsi/livenessprobe:v1.1.0
        name: liveness-probe
        volumeMounts:
        - mountPath: /csi
          name: socket-dir
      priorityClassName: system-cluster-critical
      serviceAccount: juicefs-csi-controller-sa
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      volumes:
      - emptyDir: {}
        name: socket-dir
      - hostPath:
          path: /var/lib/juicefs/volume
          type: DirectoryOrCreate
        name: jfs-dir
      - hostPath:
          path: /var/lib/juicefs/config
          type: DirectoryOrCreate
        name: jfs-root-dir
  volumeClaimTemplates: []
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/component: node
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: juicefs-csi-node
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: juicefs-csi-node
      app.kubernetes.io/instance: juicefs-csi-driver
      app.kubernetes.io/name: juicefs-csi-driver
      app.kubernetes.io/version: master
  template:
    metadata:
      labels:
        app: juicefs-csi-node
        app.kubernetes.io/instance: juicefs-csi-driver
        app.kubernetes.io/name: juicefs-csi-driver
        app.kubernetes.io/version: master
    spec:
      containers:
      - args:
        - --endpoint=$(CSI_ENDPOINT)
        - --logtostderr
        - --nodeid=$(NODE_NAME)
        - --v=5
        - --enable-manager=true
        env:
        - name: CSI_ENDPOINT
          value: unix:/csi/csi.sock
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: JUICEFS_MOUNT_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: KUBELET_PORT
          value: "10250"
        - name: JUICEFS_MOUNT_PATH
          value: /var/lib/juicefs/volume
        - name: JUICEFS_CONFIG_PATH
          value: /var/lib/juicefs/config
        image: juicedata/juicefs-csi-driver:v0.19.0
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - rm /csi/csi.sock
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: healthz
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 3
        name: juicefs-plugin
        ports:
        - containerPort: 9909
          name: healthz
          protocol: TCP
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 512Mi
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /var/lib/kubelet
          mountPropagation: Bidirectional
          name: kubelet-dir
        - mountPath: /csi
          name: plugin-dir
        - mountPath: /dev
          name: device-dir
        - mountPath: /jfs
          mountPropagation: Bidirectional
          name: jfs-dir
        - mountPath: /root/.juicefs
          mountPropagation: Bidirectional
          name: jfs-root-dir
      - args:
        - --csi-address=$(ADDRESS)
        - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
        - --v=5
        env:
        - name: ADDRESS
          value: /csi/csi.sock
        - name: DRIVER_REG_SOCK_PATH
          value: /var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock
        image: quay.io/k8scsi/csi-node-driver-registrar:v2.1.0
        name: node-driver-registrar
        volumeMounts:
        - mountPath: /csi
          name: plugin-dir
        - mountPath: /registration
          name: registration-dir
      - args:
        - --csi-address=$(ADDRESS)
        - --health-port=$(HEALTH_PORT)
        env:
        - name: ADDRESS
          value: /csi/csi.sock
        - name: HEALTH_PORT
          value: "9909"
        image: quay.io/k8scsi/livenessprobe:v1.1.0
        name: liveness-probe
        volumeMounts:
        - mountPath: /csi
          name: plugin-dir
      dnsPolicy: ClusterFirstWithHostNet
      priorityClassName: system-node-critical
      serviceAccount: juicefs-csi-node-sa
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      volumes:
      - hostPath:
          path: /var/lib/kubelet
          type: Directory
        name: kubelet-dir
      - hostPath:
          path: /var/lib/kubelet/csi-plugins/csi.juicefs.com/
          type: DirectoryOrCreate
        name: plugin-dir
      - hostPath:
          path: /var/lib/kubelet/plugins_registry/
          type: Directory
        name: registration-dir
      - hostPath:
          path: /dev
          type: Directory
        name: device-dir
      - hostPath:
          path: /var/lib/juicefs/volume
          type: DirectoryOrCreate
        name: jfs-dir
      - hostPath:
          path: /var/lib/juicefs/config
          type: DirectoryOrCreate
        name: jfs-root-dir
---
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  labels:
    app.kubernetes.io/instance: juicefs-csi-driver
    app.kubernetes.io/name: juicefs-csi-driver
    app.kubernetes.io/version: master
  name: csi.juicefs.com
spec:
  attachRequired: false
  podInfoOnMount: false

secret.yaml（存储数据秘钥信息）

apiVersion: v1
kind: Secret
metadata:
  name: juicefs-secret
  namespace: default
  labels:
    # 增加该标签以启用认证信息校验
    juicefs.com/validate-secret: "true"
type: Opaque
stringData:
  name: data 
  metaurl: mysql://root:PR38Ke2sP@(mysql.juicefs.svc.cluster.local:3306)/juicefs 
  storage: minio 
  bucket: http://minio-service.juicefs.svc.cluster.local:9000/juicefs
  access-key: admin
  secret-key: 0oO1iIlLq9g
  #MINIO_ACCESS_KEY: admin 
  #MINIO_SECRET_KEY: 0oO1iIlLq9g 

#这里minio用来做数据存储，需要注意的是9000/juicefs这里是你的桶实现创建好，name: data会自行拼接到juicefs后面。也需要提前创建data出来

 storageclasses.yaml（根据秘钥信息创建的sc）

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: juicefs-sc
provisioner: csi.juicefs.com
parameters:
  csi.storage.k8s.io/provisioner-secret-name: juicefs-secret
  csi.storage.k8s.io/provisioner-secret-namespace: default
  csi.storage.k8s.io/node-publish-secret-name: juicefs-secret
  csi.storage.k8s.io/node-publish-secret-namespace: default
reclaimPolicy: Retain

pvc-test.yaml（测试pvc挂载pod）

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: juicefs-pvc
  #namespace: juicefs 
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      # 从 StorageClass 中申请 10GiB 存储容量
      storage: 3Gi
  storageClassName: juicefs-sc
---
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  #namespace: juicefs
spec:
  containers:
  - name: busybox
    image: gcr.io/google-containers/busybox:latest
    imagePullPolicy: IfNotPresent
    command: ["sh", "-c", "tail -f /dev/null"]
    tty: true  # 启用TTY以便可以连接到容器
    volumeMounts:  # 确保此行与 'containers' 下的其他字段对齐
    - mountPath: /opt
      name: juicefs-pv
      mountPropagation: HostToContainer
  restartPolicy: Never  # 设置为 Never 以便测试完成后不会自动重启
  volumes:  # 确保此行与 'containers' 对齐
  - name: juicefs-pv
    persistentVolumeClaim:
      claimName: juicefs-pvc

测试步骤 

apply k8s.yaml 后会得到以下pod

当juicefs-csi相关pod成功拉起以后需要生成基于mysql与minio的secret。这个secret是用来给storageclasses使用的，让它能够调用csi去mount文件创建挂载点。

以上配置创建好以后apply pvc-test来测试动态创建pvc效果，发现会启用一个这种pod。是一个挂载点pod。

发现测试pod也成功创建了pvc也创建了并且绑定了busyboxpod

感兴趣可以看一下minio跟mysql数据存储的状态。这里就不截图演示了。

官方文档给的样例模板：创建和使用 PV | JuiceFS Document Center

#juicefs相关镜像
juicedata/juicefs-csi-driver:v0.19.0
quay.io/k8scsi/csi-node-driver-registrar:v2.1.0
quay.io/k8scsi/csi-provisioner:v1.6.0
quay.io/k8scsi/livenessprobe:v1.1.0
juicedata/mount:ce-v1.0.4