Kubernetes学习（七）补充：基于自定义指标进行扩缩容
资源指标只包含CPU、内存，一般来说也够了。但如果想根据自定义指标:如请求qps/5xx错误数来实现HPA，就需要使用自定义指标了，目前比较成熟的实现是 Prometheus Custom Metrics。自定义指标由Prometheus来提供，再利用k8s-prometheus-adpater聚合到apiserver，实现和核心指标（metric-server)同样的效果。
1、部署Prometheus
自行部署Prometheus，需要采集Pod指标
2、部署 Custom Metrics Adapter
prometheus采集到的metrics并不能直接给k8s用，因为两者数据格式不兼容，还需要另外一个组件(k8s-prometheus-adpater)，将prometheus的metrics 数据格式转换成k8s API接口能识别的格式，转换以后，因为是自定义API，所以还需要用Kubernetes aggregator在主APIServer中注册，以便直接通过/apis/来访问。
GitHub - kubernetes-sigs/prometheus-adapter: An implementation of the custom.metrics.k8s.io API using PrometheusGitHub - kubernetes-sigs/prometheus-adapter: An implementation of the custom.metrics.k8s.io API using PrometheusGitHub - kubernetes-sigs/prometheus-adapter: An implementation of the custom.metrics.k8s.io API using Prometheus
该 PrometheusAdapter 有一个稳定的Helm Charts，我们直接使用。镜像可以替换成国内镜像。
验证适配器是否注册到APIServer：
[root@master chart]# kubectl get apiservices |grep custom
v1beta1.custom.metrics.k8s.io kube-system/prometheus-adapter True 3m56s
[root@master chart]#3、部署测试应用
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: metrics-app
name: metrics-app
spec:
replicas: 2
selector:
matchLabels:
app: metrics-app
template:
metadata:
labels:
app: metrics-app
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "80"
prometheus.io/path: "/metrics"
spec:
containers:
- image: art.local:8081/docker-local/lizhenliang/metrics-app
name: metrics-app
ports:
- name: web
containerPort: 80
resources:
requests:
cpu: 200m
memory: 256Mi
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 3
periodSeconds: 5
livenessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 3
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: metrics-app
labels:
app: metrics-app
spec:
type: NodePort
ports:
- name: web
port: 80
targetPort: 80
nodePort: 30099
selector:
app: metrics-app
该metrics-app暴露了一个Prometheus指标接口，提供2个指标：
http_requests_total
http_requests_per_second
4、创建自定义HPA策略
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: metrics-app-hpa-custom
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: metrics-app
minReplicas: 1
maxReplicas: 10
metrics:
- type: Pods
pods:
metric:
name: http_requests_per_second
target:
type: AverageValue
averageValue: 800m
[root@master k8s]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
metrics-app-hpa-custom Deployment/metrics-app <unknown>/800m 1 10 2 27s
[root@master k8s]#此时适配器还不知道要什么指标（http_requests_per_second），HPA也就获取不到Pod提供指标。
5、配置适配器收集特定的指标
编辑PrometheusAdapter创建的prometheus-adapter Config Map：
新增查询规则
- seriesQuery: 'http_requests_total{kubernetes_namespace!="",kubernetes_pod_name!=""}'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
name:
matches: "^(.*)_total"
as: "${1}_per_second"
metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'重新部署PrometheusAdapter后，再次查看hpa，此时能够获取到自定义指标
[root@master log]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
metrics-app-hpa-custom Deployment/metrics-app 499m/800m 1 10 2 76m
[root@master log]#6、压测，观察自定义指标扩缩容
[root@localhost ~]# ab -n 100000 -c 100 http://179.220.56.232:30099/metrics[root@master ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
metrics-app-hpa-custom Deployment/metrics-app 163846m/800m 1 10 10 5h46m
[root@master ~]# kubectl describe hpa metrics-app-hpa-custom
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name: metrics-app-hpa-custom
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Mon, 20 Mar 2023 10:01:59 +0800
Reference: Deployment/metrics-app
Metrics: ( current / target )
"http_requests_per_second" on pods: 499m / 800m
Min replicas: 1
Max replicas: 10
Deployment pods: 3 current / 3 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from pods metric http_requests_per_second
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 24m horizontal-pod-autoscaler New size: 4; reason: pods metric http_requests_per_second above target
Normal SuccessfulRescale 24m horizontal-pod-autoscaler New size: 8; reason: pods metric http_requests_per_second above target
Normal SuccessfulRescale 24m horizontal-pod-autoscaler New size: 10; reason: pods metric http_requests_per_second above target
Normal SuccessfulRescale 17m horizontal-pod-autoscaler New size: 7; reason: All metrics below target
Normal SuccessfulRescale 12m horizontal-pod-autoscaler New size: 5; reason: All metrics below target
Normal SuccessfulRescale 7m1s horizontal-pod-autoscaler New size: 4; reason: All metrics below target
Normal SuccessfulRescale 2m1s horizontal-pod-autoscaler New size: 3; reason: All metrics below targe