当前位置： 首页 > article >正文

Istio 自动注入 sidecar 失败导致无法访问webhook服务

article 2024/11/20 17:32:22

        最近工作中在部署Istio环境的过程中发现官方示例启动的pod不能访问不到Istio的webhook，这个问题也是困扰了我一天，特此记录，便于日后查阅。

        我把他归类到sidecar注入失败的情况，报错如下：

        1、第一种可能（我遇到的情况）

        如果自动注入时，报如下错误信息：

2023-10-26T02:15:22.051580Z error installer Internal error occurred: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded
2023-10-26T02:15:32.149109Z error installer failed to create "EnvoyFilter/istio-system/stats-filter-1.14": Internal error occurred: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded

        造成上述问题的原因是 kube-apiserver 的 --enable-admission-plugins 没有配置 MutatingAdmissionWebhook,ValidatingAdmissionWebhook参数，所以解决问题的方法就是找到 kube-apiserver.yaml 配置文件，我是通过kubeadm安装的，所以路径在 /etc/kubernetes/manifests 下，把 --enable-admission-plugins 修改为 NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook 后保存文件，删掉 kube-apiserver 的pod，让他自己根据新的配置文件重新启动（注意：如果是用kubeadm安装的修改内容如果错误，可能会导致k8s集群中的kube-apiserver全部挂掉，导致无法访问集群，这时就需要从官网下载kube-apiserver二进制文件，重新拉起一个进程，再执行删除pod的操作）

        注：这里我多加了一个配置 --feature-gates=ServerSideApply=false，这个配置是关闭特性门控服务端自动更新，因为我还报了另外一个错误：failed to update resource with server-side apply for obj

        修改后文件内容如下：

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.1.61:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.1.61
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --feature-gates=RemoveSelfLink=false,ServerSideApply=false
    image: registry.aliyuncs.com/google_containers/kube-apiserver:v1.23.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 192.168.1.61
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 192.168.1.61
        path: /readyz
        port: 6443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 192.168.1.61
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: etc-pki
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: etc-pki
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
status: {}

        如果上述配置你已经配过了，还是出现错误： failed to create "EnvoyFilter/istio-system/stats-filter-1.16": Internal error occurred: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded，按照下述操作排除Istio网络问题（懒得翻译咯~）：

        Injection works by the API server connecting to the webhook deployment (Istiod). This may cause issues if there are connectivity issues, such as firewalls, blocking this call. Depending on the Kubernetes configuration, this may required a firewall rule on port 443 or port 15017; instructions for doing so on GKE can be found here.

        In order to check if the API server can access the pod, we can send a request proxied through the api server:

        An example of a request that succeeds (no body found is returned from the service and indicates we do have connectivity):

$ kubectl get --raw /api/v1/namespaces/istio-system/services/https:istiod:https-webhook/proxy/inject -v4
I0618 07:39:46.663871   36880 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server rejected our request for an unknown reason",
  "reason": "BadRequest",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "no body found"
      }
    ]
  },
  "code": 400
}]
F0618 07:39:46.663940   36880 helpers.go:115] Error from server (BadRequest): the server rejected our request for an unknown reason

        Similarly, we can send a request from another pod:

$ curl https://istiod.istio-system:443/inject -k
no body found

         And from the istiod pod directly (note: the port here is 15017, as this is the targetPort for the Service):

$ curl https://localhost:15017/inject -k
no body found

        With this information you should be able to isolate where the breakage occurs.

        如果Istio网络没有问题，还是出现错误： failed to create "EnvoyFilter/istio-system/stats-filter-1.16": Internal error occurred: failed calling webhook "rev.validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded，可以不用怀疑，这一定是你的服务器资源不够用导致出现的问题，因为Docker/Containerd守护进程正在处理大量的请求或者运行多个容器，那么它的响应时间可能会延迟，导致"Context Deadline Exceeded"错误，所以请扩展你的Kubernetes集群服务器的内存/硬盘/CPU，这个问题也就解决了。

        我的情况排查到最后发现是Kubernetes集群内存/硬盘/CPU资源不够导致的，多执行几遍Istio安装命令就安装成功了：

        2、第二种可能

        安装 Istio 时，配置了 enableNamespacesByDefault: false

sidecarInjectorWebhook:
  enabled: true
  # 变量为true，就会为所有命名空间开启自动注入功能。如果赋值为false，则只有标签为istio-injection的命名空间才会开启自动注入功能
  enableNamespacesByDefault: false
  rewriteAppHTTPProbe: false

        解决方法：

# 设置标签
$ kubectl label namespace default istio-injection=enabled --overwrite

# 查看
$ kubectl get namespace -L istio-injection
NAME                   STATUS   AGE    ISTIO-INJECTION
default                Active   374d   enabled

        如果要重新禁用注入istio sidecar，执行下面命令：

$ kubectl label namespace default istio-injection=disabled --overwrite

        3、第三种可能

        安装 Istio 时，设置 autoInject: disabled

proxy:
  includeIPRanges: 192.168.16.0/20,192.168.32.0/20
  # 是否开启自动注入功能，取值enabled则该pods只要没有被注解为sidecar.istio.io/inject: "false",就会自动注入。如果取值为disabled，则需要为pod设置注解sidecar.istio.io/inject: "true"才会进行注入
  autoInject: disabled

        解决方法：

  • 第一个方法：设置 autoInject: enabled
  • 第二个方法：在 Pod 或者 Deployment 声明 sidecar.istio.io/inject: "true"

http://www.kler.cn/a/105747.html

相关文章：

  • Gin路由深入
  • 蓝桥杯第22场小白入门赛2~5题
  • Vue 3 中 ref 属性详解：操作 DOM 元素的利器
  • 如何在C#中处理必盈接口返回的股票数据？
  • 在ubuntu上安装ubuntu22.04并ros2 humble版本的docker容器记录
  • 【含开题报告+文档+PPT+源码】基于springboot的教师评价系统的设计与实现
  • DevOps持续集成-Jenkins（3）
  • 20231025 技能点
  • 【微信小程序】实现投票功能（附源码）
  • DevOps 笔记
  • react中使用jquery 语法
  • sql server 生成连续日期和数字
  • Java实现人脸识别和指纹认证
  • KVM虚拟化常见问题汇总
  • springcloud gateway转发后getServerName被更改的问题
  • 掌握CSS Flexbox，打造完美响应式布局，适配各种设备！
  • 【git命令】删除分支
  • docker服务CPU飙高排查
  • 基于STM32+OneNet设计的物联网智能鱼缸(2023升级版)
  • stable-diffusion-webui环境部署
  • uwb nlos(非视)研究-由一篇论文结合gpt深挖创新方法-拯救苦苦挣扎的研究生。
  • 浏览器是怎么执行JS的？——消息队列与事件循环
  • Visual Studio Professional 2019 软件安装教程（附安装包下载）
  • JVM——一些零散的概念（后续学习深入了再补充）
  • 【Linux】Centos yum源替换
  • 面试必考精华版Leetcode215. 数组中的第K个最大元素