Kubernetes(k8s)安装详细过程

• 1个主节点和两个工作节点
  172.19.6.5 k8smaster.lab.com
  172.19.6.6 k8sworker1.lab.com
  172.19.6.9 k8sworker2.lab.com
• 节点需求：
  OS: Ubuntu 22.04
  RAM:Minimum 2GB RAM or more
  CPU:Minimum 2 CPU cores / or 2 vCPU
  硬盘空间：20 GB free disk space on /var or more
  网路：每个节点间能够互联

1 每个节点设置hostname

主节点

sudo hostnamectl set-hostname "k8smaster.lab.com"

工作节点：

sudo hostnamectl set-hostname "k8sworker1.lab.com"   // 1st worker node
sudo hostnamectl set-hostname "k8sworker2.lab.com"   // 2nd worker node

在每个节点上的 /etc/hosts 文件中添加以下条目

172.19.6.5 k8smaster.lab.com
172.19.6.6 k8sworker1.lab.com
172.19.6.9 k8worker2.lab.com

2禁用swap并添加内核参数

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

在所有节点上加载以下内核模块

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

为 Kubernetes 设置以下内核参数，在 tee 命令下运行

sudo tee /etc/sysctl.d/kubernetes.conf <<EOT
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOT

重新加载以上更改，运行

sudo sysctl --system

3 安装Containerd Runtime

在本次安装中，我们为 Kubernetes 集群使用 containerd 运行时。 因此，要安装containerd，首先安装其依赖项。

sudo apt install -y curl gnupg2 software-properties-common apt-transport-https ca-certificates

启用docker存储库

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmour -o /etc/apt/trusted.gpg.d/docker.gpg
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

现在，运行以下 apt 命令来安装 containerd

sudo apt update
sudo apt install -y containerd.io

配置containerd，以便开始使用systemd作为cgroup。

containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1
sudo sed -i 's/SystemdCgroup \= false/SystemdCgroup \= true/g' /etc/containerd/config.toml

restart并enable containerd服务

sudo systemctl restart containerd
sudo systemctl enable containerd

4 添加 Apt Kubernetes Repository

默认 Ubuntu 22.04 软件包存储库中不提供 Kubernetes 软件包。 所以我们需要添加 Kubernetes 存储库。 运行以下命令来下载公共签名密钥

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

接下来，运行以下 echo 命令来添加 Kubernetes apt 存储库。

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

5) 安装Kubectl, Kubeadm and Kubelet

添加存储库后，在所有节点上安装 kubectl、kubelet 和 Kubeadm 等 Kubernetes 组件。 执行以下命令集

sudo apt update
sudo apt install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

6安装Kubernetes Cluster

现在已经准备好初始化Kubernetes 集群了。 仅在主节点上运行以下 Kubeadm 命令。

sudo kubeadm init --control-plane-endpoint=k8smaster.lab.com

Output(出现错误）:

root@k8smaster:~# kubeadm init --control-plane-endpoint=k8smaster.lab.com
I0406 12:36:17.596979    1531 version.go:256] remote version is much newer: v1.29.3; falling back to: stable-1.31
[init] Using Kubernetes version: v1.31.8
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR Port-6443]: Port 6443 is in use
        [ERROR Port-10259]: Port 10259 is in use
        [ERROR Port-10257]: Port 10257 is in use
        [ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
        [ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
        [ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
        [ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
        [ERROR Port-10250]: Port 10250 is in use
        [ERROR Port-2379]: Port 2379 is in use
        [ERROR Port-2380]: Port 2380 is in use
        [ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

如果遇到以上错误，运行一下命令，充值kubenetes cluster

kubeadm reset

再次运行init(还是出现错误）

root@k8smaster:~# kubeadm init --control-plane-endpoint=k8smaster.lab.com
I0406 15:42:41.829195    3206 version.go:256] remote version is much newer: v1.29.3; falling back to: stable-1.31
[init] Using Kubernetes version: v1.31.8
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0406 15:42:42.741170    3206 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8smaster.lab.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.19.6.6]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8smaster.lab.com localhost] and IPs [172.19.6.6 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8smaster.lab.com localhost] and IPs [172.19.6.6 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.

Unfortunately, an error has occurred:
        timed out waiting for the condition

This error is likely caused by:
        - The kubelet is not running
        - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
        - 'systemctl status kubelet'
        - 'journalctl -xeu kubelet'

Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
        - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
        Once you have found the failing container, you can inspect its logs with:
        - 'crictl --runtime-endpoint unix:///var/run/containerd/containerd.sock logs CONTAINERID'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

根据提示，stop kubelet service

systemctl stop kubelet.service

再次执行成功：

root@k8smaster:~#  kubeadm init --control-plane-endpoint=k8smaster.lab.com
I0406 16:54:34.831559    1168 version.go:256] remote version is much newer: v1.29.3; falling back to: stable-1.31
[init] Using Kubernetes version: v1.31.8
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W0406 16:54:37.285001    1168 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8smaster.lab.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.19.6.5]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8smaster.lab.com localhost] and IPs [172.19.6.5 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8smaster.lab.com localhost] and IPs [172.19.6.5 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 120.010037 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8smaster.lab.com as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8smaster.lab.com as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: jdd15r.513m0kwnj23nqag7
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:
 mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join k8smaster.lab.com:6443 --token jdd15r.513m0kwnj23nqag7 \
        --discovery-token-ca-cert-hash sha256:8a80bc0195730ff1a01164d8ba6a2eed6d232dbdd2559de27b03f3a266471019 \
        --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join k8smaster.lab.com:6443 --token jdd15r.513m0kwnj23nqag7 \
        --discovery-token-ca-cert-hash sha256:8a80bc0195730ff1a01164d8ba6a2eed6d232dbdd2559de27b03f3a266471019

初始化完成后，将看到一条消息，其中包含有关如何将工作节点加入集群的说明。 记下 kubeadm join 命令以供将来参考。
因此，要开始与集群交互，请在主节点上运行以下命令

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

接下来，尝试运行以下 kubectl 命令来查看集群和节点状态

 kubectl cluster-info

Output:

Kubernetes control plane is running at https://k8smaster.lab.com:6443
CoreDNS is running at https://k8smaster.lab.com:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
root@k8smaster:~#

root@k8smaster:~# kubectl get nodes

Output:

NAME                  STATUS     ROLES           AGE   VERSION
k8smaster.lab.com   NotReady   control-plane   14m   v1.31.8

7 Join工作节点到Cluster

在每个工作节点上，使用在第 6 步初始化主节点后记下的 kubeadm join 命令。
```bash
kubeadm join k8smaster.lab.com:6443 --token jdd15r.513m0kwnj23nqag7 \
        --discovery-token-ca-cert-hash sha256:8a80bc0195730ff1a01164d8ba6a2eed6d232dbdd2559de27b03f3a266471019

結果輸出如下(connection refused)：

root@k8sworker1:~# kubeadm join k8smaster.lab.com:6443 --token jdd15r.513m0kwnj23nqag7 \
        --discovery-token-ca-cert-hash sha256:8a80bc0195730ff1a01164d8ba6a2eed6d232dbdd2559de27b03f3a266471019
[preflight] Running pre-flight checks
error execution phase preflight: couldn't validate the identity of the API Server: Get "https://k8smaster.lab.com:6443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 172.19.6.6:6443: connect: connection refused
To see the stack trace of this error execute with --v=5 or higher

尝试telent k8smaster

root@k8sworker1:~# telnet k8smaster.lab.com 6443
Trying 172.19.6.6...
telnet: Unable to connect to remote host: Connection refused

最后，发现/etc/hosts文件内的记录，将k8smaster的ip写错了（正确的是172.19.6.5才是master)

172.19.6.6 k8smaster.lab.com
172.19.6.5 k8sworker1.lab.com
172.19.6.9 k8sworker2.lab.com

修正后，继续执行成功，输出如下：

root@k8sworker1:~# kubeadm join k8smaster.lab.com:6443 --token jdd15r.513m0kwnj23nqag7 \
        --discovery-token-ca-cert-hash sha256:8a80bc0195730ff1a01164d8ba6a2eed6d232dbdd2559de27b03f3a266471019
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

在k8sworker2上执行同样的操作：

[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

工作节点的上述输出确认两个节点已加入集群。使用 kubectl 命令从主节点检查节点状态

kubectl get nodes

Output如下：

E0407 09:25:19.041797   42547 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0407 09:25:19.044456   42547 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0407 09:25:19.048820   42547 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0407 09:25:19.050554   42547 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E0407 09:25:19.055329   42547 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?

原来worker节点也要设置管理权限

• 非root账户

  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

• root账户，则直接配置环境变量：

export KUBECONFIG=/etc/kubernetes/kubelet.conf

继续get nodes:

root@k8sworker2:~# kubectl get nodes
NAME                   STATUS     ROLES           AGE     VERSION
k8smaster.lab.com    NotReady   control-plane   16h     v1.31.2
k8sworker1.lab.com   NotReady   <none>          13m     v1.31.2
k8sworker2.lab.com   NotReady   <none>          7m15s   v1.31.2

我们可以看到节点状态为“NotReady”，因此要使其处于活动状态。 我们必须安装 CNI（Container Network Interface）或网络附加插件，例如 Calico、Flannel 和 Weave-net。

8 安装 Cilium 网络插件

需要一个网络插件来启用集群中 Pod 之间的通信。 运行以下 kubectl 命令从主节点安装 Calico 网络插件

export KUBECONFIG=~/.kube

安裝Cilium

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

sudo cilium install --version 1.16.4

验证 kube-system 命名空间中 pod 的状态

kubectl get pods -n kube-system

Output:

root@k8smaster:~/.kube# kubectl get pods -n kube-system -o wide
NAME                                          READY   STATUS    RESTARTS         AGE   IP           NODE                   NOMINATED NODE   READINESS GATES
cilium-dpkzb                                  1/1     Running   1 (10d ago)      12d   172.19.6.8   k8sworker1.lab.com   <none>           <none>
cilium-envoy-5zhvb                            1/1     Running   20 (10d ago)     26d   172.19.6.5   k8smaster.lab.com    <none>           <none>
cilium-envoy-bwxsc                            1/1     Running   14 (10d ago)     26d   172.19.6.8   k8sworker1.lab.com   <none>           <none>
cilium-operator-54c7465577-v8tk5              1/1     Running   473 (2d1h ago)   26d   172.19.6.8   k8sworker1.lab.com   <none>           <none>
cilium-operator-54c7465577-ztn6h              1/1     Running   71 (2d1h ago)    26d   172.19.6.5   k8smaster.lab.com    <none>           <none>
cilium-zg8vs                                  1/1     Running   1 (10d ago)      12d   172.19.6.5   k8smaster.lab.com    <none>           <none>
coredns-7748f8cdfb-4tbxm                      1/1     Running   1 (10d ago)      14d   10.0.1.25    k8sworker1.lab.com   <none>           <none>
coredns-7748f8cdfb-8rv2f                      1/1     Running   0                10d   10.0.1.186   k8sworker1.lab.com   <none>           <none>
etcd-k8smaster.lab.com                      1/1     Running   84 (10d ago)     31d   172.19.6.5   k8smaster.lab.com    <none>           <none>
kube-apiserver-k8smaster.lab.com            1/1     Running   89 (2d1h ago)    31d   172.19.6.5   k8smaster.lab.com    <none>           <none>
kube-controller-manager-k8smaster.lab.com   1/1     Running   178 (2d1h ago)   31d   172.19.6.5   k8smaster.lab.com    <none>           <none>
kube-proxy-9fndr                              1/1     Running   18 (10d ago)     30d   172.19.6.8   k8sworker1.lab.com   <none>           <none>
kube-proxy-xjsg9                              1/1     Running   26 (10d ago)     31d   172.19.6.5   k8smaster.lab.com    <none>           <none>
kube-scheduler-k8smaster.lab.com            1/1     Running   181 (2d1h ago)   31d   172.19.6.5   k8smaster.lab.com    <none>           <none>

以上输出，所有pod已经处于running状态