麒麟v10 arm64 部署 kubesphere 3.4 修改记录
arm64环境，默认安装 kubesphere 3.4 ，需要修改几个地方的镜像，并且会出现日志无法显示
1 fluentbit:v1.9.4 报错
<jemalloc>: Unsupported system page size
Error in GnuTLS initialization: ASN1 parser: Element was not found.
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
<jemalloc>: Unsupported system page size
malloc: Cannot allocate memory
level=error msg="Fluent bit exited" error="exit status 1"
level=info msg=backoff delay=8s
解决
更换为 fluentbit:v2.0.6
kubectl patch fluentbit -n kubesphere-logging-system fluent-bit --type='json' -p='[{"op": "replace", "path": "/spec/image", "value":"10.1.7.220/aicore/arm64/kubesphere/fluent-bit:v2.0.6"}]'`
2 defaultbackend 架构错误
里面写死了 defaultbackend-amd64:1.4
解决
更换为 defaultbackend-arm64:1.4
kubectl set image -n kubesphere-controls-system deployment/default-http-backend default-http-backend=defaultbackend-arm64:1.4
3 elasticsearch-curator 架构错误
kubesphere/elasticsearch-curator:v5.7.6 只有amd-x64
解决
更换为 kubesphere/elasticsearch-curator:v5.7.6-arm64
kubectl patch job/elasticsearch-logging-curator-elasticsearch-curator-28826940 -n kubesphere-logging-system \
--type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/elasticsearch-curator/image", "value":"kubesphere/elasticsearch-curator:v5.7.6-arm64"}]'
4 buysbox:laster 加载不到
目前 docker-io 访问不了，但是 opensearch 的 initContainers 使用 buysbox:laster，但 imagePullPolicy:always， 需要修改
解决
kubectl patch StatefulSet/opensearch-cluster-data -n kubesphere-logging-system --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/initContainers/0/imagePullPolicy", "value":"IfNotPresent"}]'
kubectl patch StatefulSet/opensearch-cluster-master -n kubesphere-logging-system --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/initContainers/0/imagePullPolicy", "value":"IfNotPresent"}]'
5 日志无法显示
根据这个帖子解决了，=
https://ask.kubesphere.io/forum/d/23443-kubesphere341-jie-ru-wai-bu-elasticsearchhou-eszhong-ke-yi-shou-ji-dao-ri-zhi-ye-mian-cha-bu-dao-eszhong-ke-yi-jian-suo/9
原因：
经过检查发现是配置字典 kubesphere-config 中的 indexPrefix 和 output 中的 logstashPrefix 配置不一致导致的
解决方案：
如果在kubesphere 页面进行容器日志查询、资源事件查询、审计日志查询需要配置kubesphere-config 中的 indexPrefix 和 output 中的 logstashPrefix一致
kubesphere-config中的配置如下：
output 中的 logstashPrefix 配置如下：
fluent-bit 未启动
在反复安装卸载后，有时 fluent-bit 会没出现
解决
创建 fluent-bit.yaml
apiVersion: logging.kubesphere.io/v1alpha2
kind: FluentBit
metadata:
name: fluent-bit
namespace: kubesphere-logging-system
labels:
app.kubernetes.io/name: fluent-bit
spec:
image: kubesphere/fluent-bit:v2.0.6
positionDB:
emptyDir: {}
resources:
limits:
cpu: 500m
memory: 200Mi
requests:
cpu: 10m
memory: 25Mi
fluentBitConfigName: fluent-bit-config
tolerations:
- operator: Exists
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/edge
operator: DoesNotExist
执行
kubectl create -f fluent-bit.yaml
整理安装脚本
下载 源码
https://github.com/kubesphere/ks-installer
修改文件
roles\download\defaults\main.yml
#ks-logger
docker_elasticsearch_curator_repo: "{{ base_repo }}{{ namespace_override | default('kubesphere') }}/elasticsearch-curator"
- docker_elasticsearch_curator_tag: v5.7.6-amd64
+ docker_elasticsearch_curator_tag: v5.7.6-arm64
#ingress:
- defaultbackend_repo: "{{ base_repo }}{{ namespace_override | default('mirrorgooglecontainers') }}/defaultbackend-arm64"
+ defaultbackend_repo: "{{ base_repo }}{{ namespace_override | default('mirrorgooglecontainers') }}/defaultbackend-arm64"
defaultbackend_tag: 1.4
roles\common\templates\custom-values-opensearch-data.yaml.j2
global:
- dockerRegistry: ""
+ dockerRegistry: "{{ base_repo }}"
roles\common\templates\custom-values-opensearch-master.yaml.j2
global:
- dockerRegistry: ""
+ dockerRegistry: "{{ base_repo }}"
roles\ks-core\config\tasks\main.yaml
- block:
- name: KubeSphere | Getting es index prefix
shell: >
- {{ bin_dir }}/kubectl get cm -n kubesphere-system kubesphere-config -o jsonpath='{.data.kubesphere\.yaml}' | grep "logging:" -A 2 | grep "indexPrefix" | awk '{print $2}'
+ {{ bin_dir }}/kubectl get cm -n kubesphere-system kubesphere-config -o jsonpath='{.data.kubesphere\.yaml}' | grep "logging:" -A 6 | grep "indexPrefix" | awk '{print $2}'
register: es_indexPrefix
roles\ks-core\config\templates\kubesphere-config.yaml.j2
{% if esIndexPrefix is defined %}
indexPrefix: {{ esIndexPrefix }}
- {% elif common.opensearch.opensearchPrefix is defined %}
+ {% elif common.opensearch.opensearchPrefix is defined and common.opensearch.enabled == true %}
indexPrefix: ks-{{ common.opensearch.opensearchPrefix }}-logging
{% else %}
- indexPrefix: ks-{{ common.es.elkPrefix }}-log
+ indexPrefix: ks-{{ common.es.elkPrefix }}-logging
{% endif %}
- {% if common.opensearch.opensearchPrefix is defined %}
+ {% if common.opensearch.opensearchPrefix is defined and common.opensearch.enabled == true %}
indexPrefix: ks-{{ common.opensearch.opensearchPrefix }}-events
{% else %}
indexPrefix: ks-{{ common.es.elkPrefix }}-events
{% endif %}
roles\ks-core\ks-core\files\ks-core\values.yaml
- defaultbackend_repo: "mirrorgooglecontainers/defaultbackend-amd64"
+ defaultbackend_repo: "mirrorgooglecontainers/defaultbackend-arm64"
roles\ks-logging\templates\custom-output-elasticsearch-logging.yaml.j2
- logstashPrefix: "ks-{{ common.es.elkPrefix }}-log"
+ logstashPrefix: "ks-{{ common.es.elkPrefix }}-logging"
Dockerfile-arm64
FROM kubesphere/ks-installer:v3.4.1
COPY roles/download/defaults/main.yml /kubesphere/installer/roles/download/defaults/main.yml
COPY roles/common/templates/custom-values-opensearch-data.yaml.j2 /kubesphere/installer/roles/common/templates/
COPY roles/common/templates/custom-values-opensearch-master.yaml.j2 /kubesphere/installer/roles/common/templates/
COPY roles/ks-core/ks-core/files/ks-core/values.yaml /kubesphere/installer/roles/ks-core/ks-core/files/ks-core/values.yaml
COPY roles/ks-core/config/templates/kubesphere-config.yaml.j2 /kubesphere/installer/roles/ks-core/config/templates/kubesphere-config.yaml.j2
COPY roles/ks-core/config/tasks/main.yaml /kubesphere/installer/roles/ks-core/config/tasks/main.yaml
COPY roles/ks-logging/templates/custom-output-elasticsearch-logging.yaml.j2 /kubesphere/installer/roles/ks-logging/templates/custom-output-elasticsearch-logging.yaml.j2
构建
docker build -t kubesphere/ks-installer:v3.4.1-arm64 -f Dockerfile-arm64 .
这样下次安装就可以全部正常了