DevOps实战：用Kubernetes和Argo打造自动化CI/CD流程（2）

DevOps实战：用Kubernetes和Argo打造自动化CI/CD流程（2）

背景

架构图

正片开始之前，请一定先熟悉上面的架构图，跟着我的步骤，一步一步执行成功，相信后续根据自己特定的需求定制CI/CD。

需求

用户更新代码，提交commit 到 master branch，DevOps Argo自动进行服务的测试，构建，更新服务。

正片开始

文件目录

GitHub - jackwillsmith/go-gin

.
|-- Dockerfile
|-- Dockerfile-bk
|-- Makefile
|-- ab_test.md
|-- docker-compose.yaml
|-- go.mod
|-- go.sum
|-- install_argo.sh
|-- main.go                                   # 程序入口
|-- main_test.go                              # 单元测试文件
|-- manifest
|   |-- argo-events-clusterrolebinding.yaml   # argo-events sa default
|   |-- argo-workflow-clusterrole.yaml        # argo clusterrole
|   |-- argo-workflow-clusterrolebinding.yaml # argo clusterrolebinding
|   |-- github-eventsources.yaml              # github eventsource
|   |-- github-sensor.yaml                    # github webhook
|   |-- go-gin-deployment-workflow.yaml       # go-gin workflow
|   |-- mani.yaml                             # go-gin deployments,service
|-- readme.md

go-gin manifest都创建在 argo-events namespace下

ArgoCD

1. 登录argocd UI

root@master:/home/eilinge/argo-cd# kubectl -n argocd get svc
NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
argocd-server                             NodePort    10.43.238.233   <none>        80:30878/TCP,443:32063/ TCP   11d # ClusterIP -> NodePort


# 获取argocd admin 密码
root@master:/home/eilinge/argo-cd# kubectl -n argocd get secret argocd-initial-admin-secret --output=jsonpath={.data.password} |base64 -d

2. 创建go-gin的deployment,service

创建成功

等待同步。点击进入详情

root@master:/home/eilinge# kubectl -n argo-events get all|grep go-gin
pod/go-gin-577b868bd6-79cf7                      1/1     Running     0               24h

service/go-gin                      ClusterIP   10.43.245.228   <none>        8080/TCP                     46h

deployment.apps/go-gin                      1/1     1            1           46h

replicaset.apps/go-gin-577b868bd6                      1         1         1       24h

Argo Workflow

部署Argo Workflow

DevOps实战：用Kubernetes和Argo打造自动化CI/CD流程（1）-CSDN博客

root@master:/home/eilinge/argo-cd# kubectl -n argo get all
NAME                                      READY   STATUS    RESTARTS        AGE
pod/argo-server-67bfcbc559-bxqwd          1/1     Running   3 (2d4h ago)    9d
pod/workflow-controller-b84cc4f5b-fg5ss   1/1     Running   9 (3h43m ago)   30h

NAME                  TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/argo-server   NodePort   10.43.242.65   <none>        2746:30865/TCP   9d

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argo-server           1/1     1            1           9d
deployment.apps/workflow-controller   1/1     1            1           9d

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/argo-server-58f9864f85          0         0         0       9d
replicaset.apps/argo-server-67bfcbc559          1         1         1       9d
replicaset.apps/argo-server-b99696f87           0         0         0       9d
replicaset.apps/workflow-controller-b84cc4f5b   1         1         1       9d

登录argo workflow UI

第一次登录时，需要进行token认证。 Access Token - Argo Workflows - The workflow engine for Kubernetes

go-gin-workflow.yaml

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: buildkit
spec:
  arguments:
    parameters:
      - name: repo
        value: https://github.com/jackwillsmith/go-gin.git
      - name: branch
        value: master
      - name: path
        value: .
      - name: image
        value: eilinge/go-gin:v1.2
      - name: servername
        value: go-gin
      - name: namespace
        value: argo-events
      - name: port
        value: 8080

  entrypoint: main
  # We use a volume claim template so that we can have a shared workspace.
  volumeClaimTemplates:
    - metadata:
        name: work
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 64Mi
  templates:
    - name: main
      dag:
        tasks:  # 部署的流程
          - name: clone     # 1. clone 从远程仓库下载到本地
            template: clone
            arguments:
              parameters:
                - name: repo
                  value: "{{workflow.parameters.repo}}"
                - name: branch
                  value: "{{workflow.parameters.branch}}"
          - name: gotest     # 2. gotest 执行go test,进行单元测试
            template: gotest
            arguments:
              parameters:
                - name: path
                  value: "{{workflow.parameters.path}}"
            depends: "clone"
          - name: build       # 3. 在pod中构建go build -o 可执行文件
            template: build
            arguments:
              parameters:
                - name: path
                  value: "{{workflow.parameters.path}}"
            depends: "gotest"
          - name: image       # 4. 在pod中构建 image
            template: image
            arguments:
              parameters:
                - name: path
                  value: "{{workflow.parameters.path}}"
                - name: image
                  value: "{{workflow.parameters.image}}"
            depends: "build"
          - name: workload    # 5. 更新go-gin deployment服务
            template: go-gin-server
            arguments:
              parameters:
                - name: servername
                  value: "{{workflow.parameters.servername}}"
                - name: namespace
                  value: "{{workflow.parameters.namespace}}"
                - name: image
                  value: "{{workflow.parameters.image}}"
            depends: "image"

    - name: clone
      inputs:
        parameters:
          - name: repo
          - name: branch
      container:
        volumeMounts:
          - mountPath: /work
            name: work
        image: docker.m.daocloud.io/alpine/git:v2.26.2
        workingDir: /work         # 不同task 之间通过/work 目录进行传递文件
        # Do a shallow clone, which is the fastest way to clone, by using the
        # --depth, --branch, and --single-branch options
        args:
          - clone
          - --depth               # 根据具体项目进行调整
          - "1"
          - --branch
          - "{{inputs.parameters.branch}}"
          - --single-branch
          - "{{inputs.parameters.repo}}"
          - .

    - name: gotest
      inputs:
        parameters:
          - name: path
      container:
        image: golang:1.22.5
        volumeMounts:
          - mountPath: /work
            name: work
        workingDir: /work/{{inputs.parameters.path}}
        env:                       # golang容器中执行 go test -v ./...
          # Because this is not a Gomodule, we must turn modules off.
          - name: GO111MODULE
            value: "on"
          - name: CGO_ENABLED
            value: "0"
          - name: GOPROXY
            value: "https://goproxy.cn,direct"
        command:
          - go
        args:
          - test
          - -v
          - ./...

    - name: build
      inputs:
        parameters:
          - name: path
      container:
        image: golang:1.22.5
        volumeMounts:
          - mountPath: /work
            name: work
        workingDir: /work/{{inputs.parameters.path}}
        env:                      # golang容器中执行 go build -o
          # Because this is not a Gomodule, we must turn modules off.
          - name: GO111MODULE
            value: "on"
          - name: CGO_ENABLED
            value: "0"
          - name: GOPROXY
            value: "https://goproxy.cn,direct"
        command:
          - go
        args:
          - build
          - -v
          - -o
          - /work/out/app       # golang main.go可执行文件

    - name: image
      inputs:
        parameters:
          - name: path
          - name: image
      # Mount the configuration so we can push the image.
      # This should create the /.docker/config.json file.
      volumes:
        - name: buildkitd-socket
          hostPath:
            path: /run/buildkit/buildkitd.sock # 需要将k3s节点的builkitd.sock 挂载到容器中
            type: Socket
      container:
        readinessProbe:
          exec:
            command: [ sh, -c, "buildctl debug workers" ]
        image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/moby/buildkit:latest
        volumeMounts:
          - name: work
            mountPath: /work
          - name: buildkitd-socket
            mountPath: /run/buildkit/buildkitd.sock   # 构建image的buildkitd.sock
        workingDir: /work/{{inputs.parameters.path}}
        env:
          - name: BUILDKITD_FLAGS
            value: --oci-worker-no-process-sandbox
        command:
          - buildctl-daemonless.sh  # 可进入容器，查看详情 相当于执行 docker build
        args:
          - build
          - --frontend
          - dockerfile.v0
          - --local
          - context=.
          - --local
          - dockerfile=.
          - --output
          - type=image,name=docker.io/{{inputs.parameters.image}},push=false

    - name: go-gin-server
      daemon: true
      inputs:
        parameters:
          - name: servername
          - name: namespace
          - name: image
      resource:
        action: patch            # 这里通过patch 修改argocd创建的deployment资源，而不是create
        manifest: |
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: {{inputs.parameters.servername}}
            namespace: {{inputs.parameters.namespace}}
          spec:
            template:
              metadata:
                creationTimestamp: "{{workflow.creationTimestamp}}"  # 只修改创建时间即可，可以使最新image生效
              spec:
                containers:
                - image: "{{inputs.parameters.image}}"
                  name: "{{inputs.parameters.servername}}" # 执行pod中具体container

创建workflow

argo-events

部署 github-eventsource

kubectl -n argo-events apply -f github-eventsources.yaml

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: github
spec:
  service:                            # 创建service :12000
    ports:
      - name: example
        port: 12000
        targetPort: 12000
  github:
    example:
      repositories:                   # 关联github 仓库
        - owner: jackwillsmith
          names:
            - go-gin
      webhook:                              # 监听 :12000/push 路由
        # endpoint to listen to events on
        endpoint: /push
        # port to run internal HTTP server on
        port: "12000"
        # HTTP request method to allow. In this case, only POST requests are accepted
        method: POST
      events:                               # 监听 events:push
        - "push"

      # type of the connection between event-source and Github.
      # You should set it to false to avoid man-in-the-middle and other attacks.
      insecure: true
      # Determines if notifications are sent when the webhook is triggered
      active: true
      # The media type used to serialize the payloads
      contentType: json

root@master:/home/eilinge/argo-cd# kubectl -n argo-events get all |grep github-eventsource
pod/github-eventsource-d6zmx-665c64c5c8-59svh    1/1     Running     0                30h

service/github-eventsource-svc      NodePort    10.43.229.201   <none>        12000:31906/TCP              5d1h

deployment.apps/github-eventsource-d6zmx    1/1     1            1           5d1h

replicaset.apps/github-eventsource-d6zmx-665c64c5c8    1         1         1       5d1h
ch

创建Sensor

部署github-sensor

kubectl -n argo-events apply -f github-sensor.yaml

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
    name: github
spec:
    template:
        serviceAccountName: operate-workflow-sa
    dependencies:
        - name: test-dep
          eventSourceName: github
          eventName: example
          filters:
              data:
                  # Type of Github event that triggered the delivery: [pull, push, issues, label, test,...]
                  # https://docs.github.com/en/developers/webhooks-and-events/webhook-events-and-payloads
                  - path: headers.X-Github-Event   # 定义监听 webhook event push
                    type: string
                    value:
                        - push
                  - path: body.ref                 # 定义github go-gin master branch
                    type: string
                    value:
                        - master
                        - "refs/heads/master"
    triggers:
        - template:
              name: github-workflow-trigger
              argoWorkflow:
                  operation: resubmit  # resubmit argo workflow
                  source:
                      resource:
                          apiVersion: argoproj.io/v1alpha1
                          kind: Workflow
                          metadata:
                              name: buildkit # workflow name exists in argo workflow
          retryStrategy:
              steps: 3

root@master:/home/eilinge/argo-cd# kubectl -n argo-events get all |grep github-sensor
pod/github-sensor-jwwvn-654f5d584-p9cvz          1/1     Running     0                25h
deployment.apps/github-sensor-jwwvn         1/1     1            1           28h
replicaset.apps/github-sensor-jwwvn-654f5d584          1         1         1       25h

github go-gin项目创建webhook

由于作者是在自己电脑的虚拟机中，部署的k3s节点，github无法直接进行访问，需要内网穿透才能在公网进行访问。可以通过Frp服务实现。

开发个人Ollama-Chat–9 Frp穿透_ollama api frps-CSDN博客

测试

经常上述的部署流程，已经将架构图中所需的资源都创建成功了，现在进行测试。

ISSUE

1. Argo Rollouts 实现蓝绿发布未写明？

蓝绿发布属于网关层，后续会更新通过专业网关服务Higress进行发布

2. Argo Workflow资源创建后，会有用户权限不足，无法操作kubernetes 资源。

解决方法放置在go-gin项目的manifest文件夹下的clusterrole.yaml, clusterrolebinding.ayml