93、k8s之hpa+helm
一、HPA
HPA:
pod的数量进行扩缩容
针对对控制器创建的pod
deployment：
replicas:
静态：edit
yaml：apply -f
HPA：基于cpu的利用率来实现pod的数量大的自动伸缩。
Horizontal Pod Autoscaling
yaml 文件------主流------>必须要有资源控制的字段才能生效。
命令行 生成HPA的配置。
yaml必要条件：
前置必要条件：控制器创建，而且必须能设置副本数
配置的必要条件：必须要声明pod的资源控制
1、metrices-server hpa使用的依赖环境。k8s使用集群资源的集中查询器。收集资源使用数据，给hpa，scheduller使用
echo 1 > /proc/sys/vm/drop_caches # 释放页面缓存
echo 2 > /proc/sys/vm/drop_caches # 释放目录项和inode缓存
echo 3 > /proc/sys/vm/drop_caches # 释放所有缓存
2、扩容和缩容的速度：
扩容一旦达到阈值，会立即扩容，缩容速度相对较慢。
为了保证pod的正常工作，扩容必须要快。
缩容的时候为了保证pod的资源突然又变大了，可以继续维持pod的数量，在一定时间之内，pod占用的资源维持在较低的比率，然后开始慢慢缩容。
[root@master01 opt]# mkdir hap
[root@master01 opt]# cd /opt/
[root@node01 opt]# rz -E ##所有节点
rz waiting to receive.
[root@node01 opt]# ls
metrics-server.tar
[root@master01 opt]# docker load -i metrics-server.tar ##所有节点
[root@master01 opt]# cd hap/
[root@master01 hap]# rz -E
rz waiting to receive.
[root@master01 hap]# ls
components.yaml
[root@master01 hap]# vim components.yaml
[root@master01 hap]# kubectl apply -f components.yaml
[root@master01 hap]# kubectl get pod -n kube-system
[root@master01 hap]# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
master01 536m 13% 3405Mi 44%
node01 64m 1% 2204Mi 28%
node02 81m 2% 2559Mi 33%
[root@master01 hap]# kubectl explain hpa
KIND: HorizontalPodAutoscaler
VERSION: autoscaling/v1
[root@master01 hap]# vim hpa.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hpa-test1
labels:
hpa: test1
spec:
replicas: 1
selector:
matchLabels:
hpa: test1
template:
metadata:
labels:
hpa: test1
spec:
containers:
- name: centos
image: centos:7
command: ["/bin/bash", "-c", "yum install -y epel-release --nogpgcheck && yum install -y stress --nogpgcheck && sleep 3600"]
volumeMounts:
- name: yum1
mountPath: /etc/yum.repos.d/
resources:
limits:
cpu: "1"
memory: 512Mi
volumes:
- name: yum1
hostPath:
path: /etc/yum.repos.d/
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: hpa-centos1
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: hpa-test1
#就是用来匹配需要监控器类型和名称
#资源占用率高，要扩容，扩容的数量
#资源占用率低，要缩容，最小的保持数量
#定义扩缩容的指标，阈值。
minReplicas: 1
maxReplicas: 6
targetCPUUtilizationPercentage: 50
#占用50%的cpu
[root@master01 hap]# kubectl logs -f hpa-test1-7558dfb467-bv85l
Loaded plugins: fastestmirror, ovl
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Determining fastest mirrors
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error"
[root@master01 hap]# cd /etc/yum.repos.d/
[root@master01 yum.repos.d]# ls
backup CentOS-Debuginfo.repo CentOS-Vault.repo kubernetes.repo
Centos-7.repo CentOS-fasttrack.repo docker-ce.repo local.repo
CentOS-Base.repo CentOS-Media.repo epel.repo
CentOS-CR.repo CentOS-Sources.repo epel-testing.repo
[root@master01 yum.repos.d]# rm -rf local.repo
------------------------报错，yum问题------------
[root@master01 hap]# kubectl logs -f hpa-test1-7558dfb467-r8t9t
Loaded plugins: fastestmirror, ovl
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Determining fastest mirrors
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error"
One of the configured repositories failed (Unknown),
and yum doesn't have enough cached data to continue. At this point the only
safe thing yum can do is fail. There are a few ways to work "fix" this:
1. Contact the upstream for the repository and get them to fix the problem.
2. Reconfigure the baseurl/etc. for the repository, to point to a working
upstream. This is most often useful if you are using a newer
distribution release than is supported by the repository (and the
packages for the previous distribution release still work).
3. Run the command with the repository temporarily disabled
yum --disablerepo=<repoid> ...
4. Disable the repository permanently, so yum won't use it by default. Yum
will then just ignore the repository until you permanently enable it
again or use --enablerepo for temporary usage:
yum-config-manager --disable <repoid>
or
subscription-manager repos --disable=<repoid>
5. Configure the failing repository to be skipped, if it is unavailable.
Note that yum will try to contact the repo. when it runs most commands,
so will have to try and fail each time (and thus. yum will be be much
slower). If it is a very temporary problem though, this is often a nice
compromise:
yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true
Cannot find a valid baseurl for repo: base/7/x86_64
[root@master01 hap]# kubectl delete -f hpa.yaml
deployment.apps "hpa-test1" deleted
horizontalpodautoscaler.autoscaling "hpa-centos1" deleted
[root@master01 hap]# kubectl apply -f hpa.yaml
deployment.apps/hpa-test1 created
horizontalpodautoscaler.autoscaling/hpa-centos1 created
[root@master01 hap]# kubectl apply -f hpa.yaml
deployment.apps/hpa-test1 created
horizontalpodautoscaler.autoscaling/hpa-centos1 created
[root@master01 hap]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hpa-test1-7558dfb467-r8t9t 0/1 Error 0 2s 10.244.1.3 node01 <none> <none>
---------------------------------------------
查看日志发现yum源找不到
--------------------所有节点都需要配置yum源------------
[root@master01 yum.repos.d]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
--2024-09-12 10:58:54-- http://mirrors.aliyun.com/repo/Centos-7.repo
正在解析主机 mirrors.aliyun.com (mirrors.aliyun.com)... 58.222.47.242, 58.222.47.240, 121.228.130.155, ...
正在连接 mirrors.aliyun.com (mirrors.aliyun.com)|58.222.47.242|:80... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度：2523 (2.5K) [application/octet-stream]
正在保存至: “/etc/yum.repos.d/CentOS-Base.repo”
100%[========================================>] 2,523 --.-K/s 用时 0s
2024-09-12 10:58:54 (250 MB/s) - 已保存 “/etc/yum.repos.d/CentOS-Base.repo” [2523/2523])
[root@node01 yum.repos.d]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
[root@node02 yum.repos.d]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
---------------------重新配置yum源------------------
-----------------------或者文本不配置epel-------------
[root@master01 hap]# vim hpa.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hpa-test1
labels:
hpa: test1
spec:
replicas: 1
selector:
matchLabels:
hpa: test1
template:
metadata:
labels:
hpa: test1
spec:
containers:
- name: centos
image: centos:7
command: ["/bin/bash", "-c", "yum install -y stress --nogpgcheck && sleep 3600"]
volumeMounts:
- name: yum1
mountPath: /etc/yum.repos.d/
resources:
limits:
cpu: "1"
memory: 512Mi
volumes:
- name: yum1
hostPath:
path: /etc/yum.repos.d/
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: hpa-centos1
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: hpa-test1
#就是用来匹配需要监控器类型和名称
#资源占用率高，要扩容，扩容的数量
#资源占用率低，要缩容，最小的保持数量
#定义扩缩容的指标，阈值。
minReplicas: 1
maxReplicas: 6
targetCPUUtilizationPercentage: 50
#占用50%的cpu
-----------------------------------
[root@master01 hap]# kubectl delete -f hpa.yaml
deployment.apps "hpa-test1" deleted
horizontalpodautoscaler.autoscaling "hpa-centos1" deleted
[root@master01 hap]# kubectl apply -f hpa.yaml
deployment.apps/hpa-test1 created
horizontalpodautoscaler.autoscaling/hpa-centos1 created
[root@master01 hap]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hpa-test1-7558dfb467-zwhh7 1/1 Running 0 4s 10.244.2.16 node02 <none> <none>
[root@master01 opt]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 0%/50% 1 6 1 29m
-------------------------------------
[root@master01 hap]# kubectl logs -f hpa-test1-7558dfb467-zwhh7
Loaded plugins: fastestmirror, ovl
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Repository contrib is listed more than once in the configuration
Determining fastest mirrors
* base: mirrors.aliyun.com
* epel: repo.jing.rocks
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
http://mirrors.aliyuncs.com/centos/7/os/x86_64/repodata/6d0c3a488c282fe537794b5946b01e28c7f44db79097bb06826e1c0c88bad5ef-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed connect to mirrors.aliyuncs.com:80; Connection refused"
Trying other mirror.
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-14 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
epel-release noarch 7-14 epel 15 k
Transaction Summary
================================================================================
Install 1 Package
Total download size: 15 k
Installed size: 25 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : epel-release-7-14.noarch 1/1
warning: /etc/yum.repos.d/epel-testing.repo created as /etc/yum.repos.d/epel-testing.repo.rpmnew
warning: /etc/yum.repos.d/epel.repo created as /etc/yum.repos.d/epel.repo.rpmnew
Verifying : epel-release-7-14.noarch 1/1
Installed:
epel-release.noarch 0:7-14
Complete!
Loaded plugins: fastestmirror, ovl
Repository base is listed more than once in the configuration
Repository updates is listed more than once in the configuration
Repository extras is listed more than once in the configuration
Repository centosplus is listed more than once in the configuration
Repository contrib is listed more than once in the configuration
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* epel: repo.jing.rocks
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package stress.x86_64 0:1.0.4-16.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
stress x86_64 1.0.4-16.el7 epel 39 k
Transaction Summary
================================================================================
Install 1 Package
Total download size: 39 k
Installed size: 94 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : stress-1.0.4-16.el7.x86_64 1/1
install-info: No such file or directory for /usr/share/info/stress.info
Verifying : stress-1.0.4-16.el7.x86_64 1/1
Installed:
stress.x86_64 0:1.0.4-16.el7
Complete!
----------------------------------------------
[root@master01 hap]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hpa-test1-7558dfb467-zwhh7 1/1 Running 0 35m 10.244.2.16 node02 <none> <none>
nfs1-76f66b958-68wpl 1/1 Running 0 6d1h 10.244.2.173 node02 <none> <none>
[root@master01 hap]# kubectl top node node02
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
node02 75m （k8s使用） 1% （所有） 2845Mi 36%
[root@master01 hap]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 （目前使用） 0%/50%（设定的阈值） 1（最小） 6（最多） 1（目前副本数） 36m
top
到node02
按1
查看cpu使用情况
[root@master01 hap]# kubectl exec -it hpa-test1-7558dfb467-zwhh7 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@hpa-test1-7558dfb467-zwhh7 /]# stress -c 4
stress: info: [67] dispatching hogs: 4 cpu, 0 io, 0 vm, 0 hdd
[root@master01 opt]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 88%/50% 1 6 2 43m
[root@master01 yum.repos.d]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hpa-test1-7558dfb467-rj7h5 1/1 Running 0 44m 10.244.1.4 node01 <none> <none>
[root@hpa-test1-7558dfb467-zwhh7 /]# stress -c 4
stress: info: [67] dispatching hogs: 4 cpu, 0 io, 0 vm, 0 hdd
[root@master01 opt]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 88%/50% 1 6 2 43m
[root@master01 opt]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 49%/50% 1 6 4 43m
[root@master01 opt]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 49%/50% 1 6 4 43m
[root@master01 opt]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 25%/50% 1 6 4 44m
[root@master01 opt]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 25%/50% 1 6 4 44m
[root@master01 opt]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 25%/50% 1 6 4 44m
[root@master01 opt]# kubectl get hpa
[root@master01 opt]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hpa-test1-7558dfb467-cfn8l 1/1 Running 0 2m33s 10.244.2.17 node02 <none> <none>
hpa-test1-7558dfb467-rj7h5 1/1 Running 0 3m34s 10.244.1.4 node01 <none> <none>
hpa-test1-7558dfb467-swv7m 1/1 Running 0 2m33s 10.244.0.42 master01 <none> <none>
hpa-test1-7558dfb467-zwhh7 1/1 Running 0 45m 10.244.2.16 node02 <none> <none>
nfs1-76f66b958-68wpl 1/1 Running 0 6d1h 10.244.2.173 node02 <none> <none>
查看扩容的pod的日志
[root@master01 opt]# kubectl logs -f hpa-test1-7558dfb467-cfn8l
[root@master01 opt]# kubectl exec -it hpa-test1-7558dfb467-rj7h5 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@hpa-test1-7558dfb467-rj7h5 /]# stress -c 4
stress: info: [65] dispatching hogs: 4 cpu, 0 io, 0 vm, 0 hdd
top
到node01
按1
查看cpu使用情况
[root@master01 yum.repos.d]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 65%/50% 1 6 5 54m
[root@master01 yum.repos.d]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 65%/50% 1 6 5 54m
[root@master01 yum.repos.d]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 65%/50% 1 6 5 54m
[root@master01 yum.repos.d]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hpa-test1-7558dfb467-bq64d 1/1 Running 0 77s 10.244.2.19 node02 <none> <none>
hpa-test1-7558dfb467-ns6hs 1/1 Running 0 31s 10.244.1.5 node01 <none> <none>
hpa-test1-7558dfb467-rj7h5 1/1 Running 0 12m 10.244.1.4 node01 <none> <none>
hpa-test1-7558dfb467-swv7m 1/1 Running 0 11m 10.244.0.42 master01 <none> <none>
hpa-test1-7558dfb467-zl7hf 1/1 Running 0 77s 10.244.2.18 node02 <none> <none>
nfs1-76f66b958-68wpl 1/1 Running 0 6d1h 10.244.2.173 node02 <none> <none>
停止运行
stress: info: [66] dispatching hogs: 4 cpu, 0 io, 0 vm, 0 hdd
^C
对应节点查看top
[root@hpa-test1-7558dfb467-swv7m /]# stress -c 4
stress: info: [66] dispatching hogs: 4 cpu, 0 io, 0 vm, 0 hdd
^C
[root@master01 yum.repos.d]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 0%/50% 1 6 5 60m
[root@hpa-test1-7558dfb467-rj7h5 /]# stress -c 4
stress: info: [65] dispatching hogs: 4 cpu, 0 io, 0 vm, 0 hdd
^C
[root@master01 yum.repos.d]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 0%/50% 1 6 5 60m
[root@master01 yum.repos.d]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 0%/50% 1 6 4 61m
[root@master01 yum.repos.d]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos1 Deployment/hpa-test1 0%/50% 1 6 1 85m
[root@master01 yum.repos.d]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hpa-test1-7558dfb467-rj7h5 1/1 Running 0 44m 10.244.1.4 node01 <none> <none>
nfs1-76f66b958-68wpl 1/1 Running 0 6d2h 10.244.2.173 node02 <none> <none>
创建命名空间并做资源限制
[root@master01 etc]# cd /opt/
[root@master01 opt]# kubectl create ns xy102
namespace/xy102 created
[root@master01 opt]# vim xy102.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: ns-xy102
namespace: xy102
#对指定的命令空间进行资源限制
spec:
hard:
pods: "20"
#设置该命名空间可以创建pod的最大数量
requests.cpu: "2"
#只能使用2个cpu
requests.memory: 1Gi
#只能使用1Gi的内存
limits.cpu: "3"
limits.memory: 2Gi
#limits最多能使用多少
configmaps: "10"
#创建configmap的数量限制
persistentvolumeclaims: "4"
#创建pvc请求的限制
secrets: "10"
#创建加密配置文件的限制
services: "10"
#创建services的限制
~
[root@master01 opt]# kubectl apply -f xy102.yaml
resourcequota/ns-xy102 created
[root@master01 opt]# kubectl describe namespaces xy102
Name: xy102
Labels: <none>
Annotations: <none>
Status: Active
Resource Quotas
Name: ns-xy102
Resource Used Hard
-------- --- ---
configmaps 1 10
limits.cpu 0 3
limits.memory 0 2Gi
persistentvolumeclaims 0 4
pods 0 20
requests.cpu 0 2
requests.memory 0 1Gi
secrets 1 10
services 0 10
No LimitRange resource.
limitRange:
只要是创建在这个命令空间的pod，都会根据limitRange的配置来对所有的pod进行统一的资源限制
[root@master01 opt]# vim xy102.te.yaml
apiVersion: v1
kind: LimitRange
metadata:
name: xy102-limit
namespace: xy102
#指定命名空间
spec:
limits:
- default:
#直接加default就相当于是上限
memory: 1Gi
cpu: "4"
defaultRequest:
#这个就是软限制：
memory: 1Gi
cpu: "4"
type: Container
#指定类型:Container pod pvc
~
[root@master01 opt]# kubectl apply -f xy102.te.yaml
limitrange/xy102-limit created
[root@master01 opt]# kubectl describe namespaces xy102
Name: xy102
Labels: <none>
Annotations: <none>
Status: Active
Resource Quotas
Name: ns-xy102
Resource Used Hard
-------- --- ---
configmaps 1 10
limits.cpu 0 3
limits.memory 0 2Gi
persistentvolumeclaims 0 4
pods 0 20
requests.cpu 0 2
requests.memory 0 1Gi
secrets 1 10
services 0 10
Resource Limits
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container cpu - - 4 4 -
Container memory - - 1Gi 1Gi -
作业：
stateful 能够实现扩缩容
最小的pod数是1,但是副本的数量定义是3，那么缩容会缩到3还是1.
[root@master01 hap]# vim hpa1.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: hpa-test2
labels:
hpa: test2
spec:
serviceName: "my-headless-service"
replicas: 3
selector:
matchLabels:
hpa: test2
template:
metadata:
labels:
hpa: test2
spec:
containers:
- name: centos
image: centos:7
command: ["/bin/bash", "-c", "yum install -y epel-release --nogpgcheck && yum install -y stress --nogpgcheck && sleep 3600"]
volumeMounts:
- name: yum2
mountPath: /etc/yum.repos.d/
resources:
limits:
cpu: "1"
memory: 512Mi
volumes:
- name: yum2
hostPath:
path: /etc/yum.repos.d/
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: hpa-centos2
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: StatefulSet
name: hpa-test2
#就是用来匹配需要监控器类型和名称
#资源占用率高，要扩容，扩容的数量
#资源占用率低，要缩容，最小的保持数量
#定义扩缩容的指标，阈值。
minReplicas: 1
maxReplicas: 6
targetCPUUtilizationPercentage: 50
#占用50%的cpu
---
apiVersion: v1
kind: Service
metadata:
name: my-headless-service
spec:
ports:
- port: 80
name: http
clusterIP: None
selector:
hpa: test2
[root@master01 hap]# kubectl apply -f hpa1.yaml
statefulset.apps/hpa-test2 created
horizontalpodautoscaler.autoscaling/hpa-centos2 configured
service/my-headless-service created
[root@master01 hap]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hpa-test2-0 1/1 Running 0 105s 10.244.2.20 node02 <none> <none>
hpa-test2-1 1/1 Running 0 104s 10.244.1.6 node01 <none> <none>
hpa-test2-2 1/1 Running 0 102s 10.244.0.43 master01 <none> <none>
准备进入node02
准备进入node02
[root@master01 hap]# kubectl exec -it hpa-test2-0 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@hpa-test2-0 /]# stress -c 4
stress: info: [66] dispatching hogs: 4 cpu, 0 io, 0 vm, 0 hdd
[root@master01 hap]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos2 StatefulSet/hpa-test2 33%/50% 1 6 3 11m
等了一会，坐等缩容
[root@master01 hap]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos2 StatefulSet/hpa-test2 49%/50% 1 6 2 12m
[root@master01 hap]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hpa-test2-0 1/1 Running 0 7m41s 10.244.2.20 node02 <none> <none>
hpa-test2-1 1/1 Running 0 7m40s 10.244.1.6 node01 <none> <none>
nfs1-76f66b958-68wpl 1/1 Running 0 6d2h 10.244.2.173 node02 <none> <none>
停止cpu高负荷
[1]+ Stopped stress -c 4
[root@master01 hap]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos2 StatefulSet/hpa-test2 0%/50% 1 6 2 16m
[root@master01 hap]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-centos2 StatefulSet/hpa-test2 0%/50% 1 6 1 21m
[root@master01 hap]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hpa-test2-0 1/1 Running 0 16m 10.244.2.20 node02 <none> <none>
nfs1-76f66b958-68wpl 1/1 Running 0 6d3h 10.244.2.173 node02 <none> <none>
# 作业：
Statefulset 能够实现扩缩容
最小的pod数是1,但是副本的数量定义是3，那么缩容会缩到3还是1
[root@master01 hap]# vim hpa1.yaml
[root@master01 hap]# kubectl apply -f hpa1.yaml
2、[root@master01 hap]# kubectl get pod -o wide
3、[root@master01 hap]# kubectl get hpa
4、开始缩容
5、最终副本数变为1
报错
作业中存在报错
[root@master01 hap]# kubectl apply -f hpa1.yaml
error: error validating "hpa1.yaml": error validating data: ValidationError(StatefulSet.spec): missing required field "serviceName" in io.k8s.api.apps.v1.StatefulSetSpec; if you choose to ignore these errors, turn validation off with --validate=false
求助ai
这个错误信息指出在你的 StatefulSet 配置（hpa1.yaml 文件）中缺少了必需的字段 serviceName。在 StatefulSet 的规格（spec）中，serviceName 是一个必需的字段，它用于指定与 StatefulSet 中的 Pods 相关联的 headless service 的名称。这个 headless service 负责在 DNS 中为 StatefulSet 的 Pods 提供稳定的网络身份。
要解决这个问题，你需要按照以下步骤操作：
-
检查并更新你的 YAML 文件：
在你的hpa1.yaml文件中，找到StatefulSet的定义部分，并确保在spec下有一个serviceName字段，其值应该是一个已经存在的 headless service 的名称。例如：apiVersion: apps/v1 kind: StatefulSet metadata: name: my-statefulset spec: serviceName: "my-headless-service" # 确保这个 service 已经存在 replicas: 3 selector: matchLabels: app: myapp template: metadata: labels: app: myapp spec: containers: - name: mycontainer image: myimage # ... 其他容器配置 # ... 其他 StatefulSet 配置 -
创建或验证 headless service：
确保名为my-headless-service的 headless service 已经存在。Headless service 的定义与普通 service 类似，但不需要指定type字段或clusterIP（它会自动设置为None）。如果你还没有创建这个 service，你需要先创建一个，如下所示：apiVersion: v1 kind: Service metadata: name: my-headless-service spec: ports: - port: 80 name: http clusterIP: None selector: app: myapp -
重新应用 YAML 文件：
在更新了 YAML 文件并确保了 headless service 的存在后，使用以下命令重新应用你的 YAML 文件：bash复制代码 kubectl apply -f hpa1.yaml -
检查状态：
使用以下命令检查StatefulSet和 headless service 的状态：kubectl get statefulsets kubectl get svc my-headless-service
如果你遵循了这些步骤但仍然遇到问题，可能需要检查你的 Kubernetes 集群的状态或查看更详细的错误日志。另外，请注意文件名 hpa1.yaml 通常用于 HorizontalPodAutoscaler（HPA）的配置，但在这个上下文中，你似乎是在配置一个 StatefulSet。如果这只是一个文件名的混淆，建议将其更改为更合适的名称，如 statefulset.yaml。
二、helm
helm：
nginx ingress
deployment
svc
ingress
helm提供了一个模版，可以一键化的部署微服务。
通过打包的方式，把所有需要的yaml文件集合在一起，然后一键部署。
可以支持回滚。
helm
k8s的部署可以配置，可以集成，可以动态的修改。
三个概念：
Chart：部署安装k8s微服务的模板，类似于linux里面的rpm包。
Repository：仓库，仓库用来保存Chart
Release：当我们使用Chart部署微服务时。每部署一次就会有一个release。理解为版本号。
[root@master01 opt]# tar -xf helm-v3.10.0-linux-amd64.tar.gz
[root@master01 opt]# cd linux-amd64/
[root@master01 linux-amd64]# mv helm /usr/local/bin/
##做一下helm自动补全
[root@master01 linux-amd64]# vim /etc/profile
source <(helm completion bash)
[root@master01 linux-amd64]# source /etc/profile
#helm查看版本号
[root@master01 linux-amd64]# helm version
version.BuildInfo{Version:"v3.10.0", GitCommit:"ce66412a723e4d89555dc67217607c6579ffcb21", GitTreeState:"clean", GoVersion:"go1.18.6"}
##安装仓库和模板
----------------
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo add stable http://mirror.azure.cn/kubernetes/charts
helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
helm repo add incubator https://charts.helm.sh/incubator
-----------------------------
[root@master01 linux-amd64]# helm repo add bitnami https://charts.bitnami.com/bitnami
[root@master01 linux-amd64]#helm repo add stable http://mirror.azure.cn/kubernetes/charts
[root@master01 linux-amd64]#helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/ charts
[root@master01 linux-amd64]#helm repo add incubator https://charts.helm.sh/incubator
##仓库里面有模版-----chart
# helm repo update #更新当前所有仓库的chart
[root@master01 helm]# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "aliyun" chart repository
...Successfully got an update from the "stable" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
查看仓库
[root@master01 linux-amd64]# helm repo list
NAME URL
bitnami https://charts.bitnami.com/bitnami
aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
stable http://mirror.azure.cn/kubernetes/charts
查看nginx
[root@master01 linux-amd64]# helm search repo aliyun | grep nginx
aliyun/nginx-ingress 0.9.5 0.10.2 An nginx Ingress controller that uses ConfigMap...
aliyun/nginx-lego 0.3.1 Chart for nginx-ingress-controller and kube-lego
[root@master01 linux-amd64]# helm search repo stable | grep nginx
stable/nginx-ingress 1.41.3 v0.34.1 DEPRECATED! An nginx Ingress controller that us...
stable/nginx-ldapauth-proxy 0.1.6 1.13.5 DEPRECATED - nginx proxy with ldapauth
stable/nginx-lego 0.3.1 Chart for nginx-ingress-controller and kube-lego
查看chart模板详情
[root@master01 linux-amd64]# helm show chart aliyun/nginx-lego
apiVersion: v1
deprecated: true
description: Chart for nginx-ingress-controller and kube-lego
keywords:
- kube-lego
- nginx-ingress-controller
- nginx
- letsencrypt
maintainers:
- email: jack.zampolin@gmail.com
name: Jack Zampolin
name: nginx-lego
sources:
- https://github.com/kubernetes/contrib/tree/master/ingress/controllers/nginx
- https://github.com/jetstack/kube-lego/tree/master/examples/nginx
version: 0.3.1
##删除仓库
[root@master01 linux-amd64]# helm repo remove aliyun
[root@master01 linux-amd64]# helm install redis1 stable/redis -n default
[root@master01 linux-amd64]# kubectl get pod
NAME READY STATUS RESTARTS AGE
hpa-test2-0 1/1 Running 1 66m
hpa-test2-1 1/1 Running 0 106s
nfs1-76f66b958-68wpl 1/1 Running 0 6d3h
redis1-master-0 0/1 Pending 0 9s
redis1-slave-0 0/1 Pending 0 9s
[root@master01 linux-amd64]# kubectl describe pod redis1-master-0
[root@master01 linux-amd64]# helm ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
redis1 default 1 2024-09-12 13:52:43.74847638 +0800 CST deployed redis-10.5.7 5.0.7
[root@master01 linux-amd64]# helm uninstall redis1
release "redis1" uninstalled
[root@master01 linux-amd64]# kubectl get pod
NAME READY STATUS RESTARTS AGE
hpa-test2-0 1/1 Running 1 69m
hpa-test2-1 1/1 Running 0 4m13s
nfs1-76f66b958-68wpl 1/1 Running 0 6d3h
[root@master01 linux-amd64]# cd /opt/
[root@master01 opt]# mkdir helm
[root@master01 opt]# cd helm/
[root@master01 helm]# helm create nginx1
[root@master01 helm]# yum -y install tree
[root@master01 helm]# tree nginx1
nginx1/
├── charts #依赖环境，一般为空，也不需要
├── Chart.yaml #包含chart的元信息，chart的版本，名称等等
├── templates #包含了部署k8s的应用pod的模版文件
│ ├── deployment.yaml ##基于控制器
│ ├── _helpers.tpl
│ ├── hpa.yaml ##做hpa的监控，自动伸缩
│ ├── ingress.yaml ##对外访问
│ ├── NOTES.txt
│ ├── serviceaccount.yaml ##创建服务账号
│ ├── service.yaml ##创建service的清单
│ └── tests
│ └── test-connection.yaml
└── values.yaml ##我们的配置在values里面完成，集合在这个配置里面，当配置完成之后，可以通过values配置把参数传给templates里面的模板文件，进行覆盖。
ingress-nginx
[root@master01 helm]# ls
nginx1 nginx1-0.1.0.tgz
[root@master01 helm]# cd nginx1/
[root@master01 nginx1]# ls
charts Chart.yaml templates values.yaml
[root@master01 nginx1]# vim values.yaml
replicaCount: 3 ##更改副本数
tag: "1.22"
#tag:使用镜像的版本号，不改默认最新
ingress:
enabled: true
hosts:
- host: www.xy102.com
paths:
- path: /
pathType: Prefix
resources:
limits:
cpu: 100m
memory: 128Mi
autoscaling:
enabled: true
minReplicas: 1
maxReplicas: 6
targetCPUUtilizationPercentage: 80
[root@master01 helm]# helm lint nginx1
==> Linting nginx1
[INFO] Chart.yaml: icon is recommended
[WARNING] templates/hpa.yaml: autoscaling/v2beta1 HorizontalPodAutoscaler is deprecated in v1.22+, unavailable in v1.25+; use autoscaling/v2 HorizontalPodAutoscaler
1 chart(s) linted, 0 chart(s) failed
[root@master01 helm]# helm package nginx1/
Successfully packaged chart and saved it to: /opt/helm/nginx1-0.1.0.tgz
#第一次部署，release就是版本号。版本号1
[root@master01 helm]# helm install nginx1 /opt/helm/nginx1-0.1.0.tgz
NAME: nginx1
LAST DEPLOYED: Thu Sep 12 14:09:44 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
http://www.xy102.com/
##删除pod
[root@master01 nginx1]# helm uninstall nginx1
release "nginx1" uninstalled
[root@master01 helm]# kubectl edit deployments.apps nginx1
[root@master01 helm]# kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 17d <none>
my-headless-service ClusterIP None <none> 80/TCP 170m hpa=test2
nginx1 ClusterIP 10.96.180.35 <none> 80/TCP 2m15s app.kubernetes.io/instance=nginx1,app.kubernetes.io/name=nginx1
[root@master01 helm]# curl 10.96.180.35
##发现没有开放ingress-nginx
[root@master01 ingress]# kubectl apply -f mandatory.yaml
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
Warning: rbac.authorization.k8s.io/v1beta1 Role is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 Role
role.rbac.authorization.k8s.io/nginx-ingress-role created
Warning: rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created
limitrange/ingress-nginx created
[root@master01 ingress]# ls
auth https ingress-nginx1.yaml mandatory.yaml service-nodeport.yaml traefik
[root@master01 ingress]# kubectl apply -f service-nodeport.yaml
service/ingress-nginx created
[root@master01 ingress]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 17d
my-headless-service ClusterIP None <none> 80/TCP 3h45m
nginx1 ClusterIP 10.96.167.13 <none> 80/TCP 14m
[root@master01 ingress]# kubectl get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx NodePort 10.96.26.61 <none> 80:31895/TCP,443:30785/TCP 9s
[root@master01 ingress]# vim /etc/hosts
192.168.168.81 master01 www.xy102.com www.test1.com
[root@master01 ingress]# curl www.test1.com
404 page not found
[root@master01 ingress]# curl www.test1.com:31895
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
NodePort
[root@master01 nginx1]# vim values.yaml
service:
type: NodePort
port: 80
#如果自定义nodeport的端口，需要改配置文件
nodePort: 30001
ingress:
enabled: false
className: ""
annotations: {}
[root@master01 templates]# vim service.yaml
apiVersion: v1
kind: Service
metadata:
name: {{ include "nginx1.fullname" . }}
labels:
{{- include "nginx1.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
nodePort: {{ .Values.service.nodePort}}
[root@master01 helm]# helm upgrade nginx1 nginx1
Release "nginx1" has been upgraded. Happy Helming!
NAME: nginx1
LAST DEPLOYED: Thu Sep 12 15:48:21 2024
NAMESPACE: default
STATUS: deployed
REVISION: 2
NOTES:
1. Get the application URL by running these commands:
export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services nginx1)
export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
[root@master01 helm]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 17d
my-headless-service ClusterIP None <none> 80/TCP 3h3m
nginx1 NodePort 10.96.180.35 <none> 80:30001/TCP 15m
[root@master01 helm]# curl 192.168.168.81:30001
[root@master01 helm]# curl www.xy102.com:30001
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@master01 helm]# helm history nginx1
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Thu Sep 12 15:34:11 2024 superseded nginx1-0.1.0 1.16.0 Install complete
2 Thu Sep 12 15:48:21 2024 deployed nginx1-0.1.0 1.16.0 Upgrade complete
nginx1回滚到1
[root@master01 helm]# helm rollback nginx1 1
Rollback was a success! Happy Helming!
[root@master01 helm]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
nginx1 <none> www.xy102.com 80 58s
ne> 80:30001/TCP 15m
[root@master01 helm]# curl 192.168.168.81:30001
[root@master01 helm]# curl www.xy102.com:30001
Welcome to nginx!
If you see this page, the nginx web server is successfully installed and working. Further configuration is required.
For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.
Thank you for using nginx.
[root@master01 helm]# helm history nginx1
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Thu Sep 12 15:34:11 2024 superseded nginx1-0.1.0 1.16.0 Install complete
2 Thu Sep 12 15:48:21 2024 deployed nginx1-0.1.0 1.16.0 Upgrade complete
nginx1回滚到1
[root@master01 helm]# helm rollback nginx1 1
Rollback was a success! Happy Helming!
[root@master01 helm]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
nginx1 www.xy102.com 80 58s