loki failed to flush

loki 报错

level=error ts=2024-12-27T08:13:10.450140686Z caller=flush.go:143 org_id=fake msg="failed to flush" err="failed to flush chunks: store put chunk: open /data/loki/chunks/ZmFrZS85ODBmM2U3NzliODg2MjY1OjE5M2VhNDVkYTc4OjE5M2VhNDVlNDdkOjVmMjA1MzNj: no space left on device, num_chunks: 957, labels: {app=\"ceph-osd\", component=\"cephclusters.ceph.rook.io\", container=\"osd\", filename=\"/var/log/pods/rook-ceph_rook-ceph-osd-71-784c6ddd99-5gdt5_356ef22e-c54a-4b78-a774-c950e2292450/osd/5.log\", instance=\"71\", job=\"rook-ceph/ceph-osd\", namespace=\"rook-ceph\", node_name=\"stor5\", pod=\"rook-ceph-osd-71-784c6ddd99-5gdt5\"}"

排查

登录 loki-0 pod
查看磁盘 inode

df -hi |grep -w /data
/dev/rbd2               160.0M      3.9M    156.1M   2% /data

查看磁盘使用

df -h |grep -w /data
/dev/rbd2                 2.5T      1.2T      1.2T  51% /data

查看 pod 所在主机的内核日志

dmesg -T|grep ext4|head -10
[Fri Dec 27 11:48:33 2024] EXT4-fs warning (device rbd2): ext4_dx_add_entry:2548: Large directory feature is not enabled on this filesystem
[Fri Dec 27 11:48:33 2024] EXT4-fs warning (device rbd2): ext4_dx_add_entry:2544: Directory (ino: 4194306) index full, reach max htree level :2
[Fri Dec 27 11:48:33 2024] EXT4-fs warning (device rbd2): ext4_dx_add_entry:2548: Large directory feature is not enabled on this filesystem
[Fri Dec 27 11:48:33 2024] EXT4-fs warning (device rbd2): ext4_dx_add_entry:2544: Directory (ino: 4194306) index full, reach max htree level :2
[Fri Dec 27 11:48:33 2024] EXT4-fs warning (device rbd2): ext4_dx_add_entry:2548: Large directory feature is not enabled on this filesystem
[Fri Dec 27 11:48:33 2024] EXT4-fs warning (device rbd2): ext4_dx_add_entry:2544: Directory (ino: 4194306) index full, reach max htree level :2
[Fri Dec 27 11:48:33 2024] EXT4-fs warning (device rbd2): ext4_dx_add_entry:2548: Large directory feature is not enabled on this filesystem
[Fri Dec 27 11:48:33 2024] EXT4-fs warning (device rbd2): ext4_dx_add_entry:2544: Directory (ino: 4194306) index full, reach max htree level :2
[Fri Dec 27 11:48:33 2024] EXT4-fs warning (device rbd2): ext4_dx_add_entry:2548: Large directory feature is not enabled on this filesystem
[Fri Dec 27 11:48:44 2024] EXT4-fs warning (device rbd2): ext4_dx_add_entry:2544: Directory (ino: 4194306) index full, reach max htree level :2

问题出在

EXT4-fs warning (device rbd2): ext4_dx_add_entry:2544: Directory (ino: 4194306) index full, reach max htree level :2

参考issue
Loki (in Docker) reports “no space left on device” but there’s plenty of space/inodes #1502

解决

本文是用的ceph sc，将csi.storage.k8s.io/fstype: 的值改为 xfs。

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-ceph-block-xfs
parameters:
  clusterID: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/fstype: xfs
provisioner: rook-ceph.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate