ceph 18.2.4二次开发，docker镜像制作

编译环境要求

#需要ubuntu 22.04版本 参考https://docs.ceph.com/en/reef/start/os-recommendations/
#磁盘空间最好大于200GB
#内存如果小于100GB 会有OOM的情况发生,需要重跑 目前遇到内存占用最高为92GB

替换阿里云ubuntu 22.04源

将下面内容写入/etc/apt/sources.list 文件

mv /etc/apt/sources.list /etc/apt/sources.list.bak
#内容见下方
vim /etc/apt/sources.list
#更新apt源
apt update

deb https://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiverse
#deb-src https://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiverse

deb https://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiverse
#deb-src https://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiverse

deb https://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse
#deb-src https://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse

# deb https://mirrors.aliyun.com/ubuntu/ jammy-proposed main restricted universe multiverse
# deb-src https://mirrors.aliyun.com/ubuntu/ jammy-proposed main restricted universe multiverse

deb https://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse
#deb-src https://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse

拉取代码

git clone https://github.com/ceph/ceph.git
#切换tag
git checkout v18.2.4 -b v18.2.4
#下载子模块
git submodule update --init --recursive

安装依赖

sudo apt install curl

export PIP_INDEX_URL=https://mirrors.aliyun.com/pypi/simple/

#安装依赖包
./install-deps.sh

#检查当前的 GCC 版本 是否为c++ 11
#gcc --version
#g++ --version

sudo apt install python3-routes reprepro npm docker.io

sudo apt install `cat doc_deps.deb.txt`
#npm阿里源
npm config set registry https://registry.npmmirror.com
#npm腾讯源
#npm config set registry https://mirrors.cloud.tencent.com/npm/

编译deb包

#脚本内容见下方,注意脚本中的注释
#test.sh的内容根据官方提供的make-debs.sh进行了微调整
#也可以直接运行make-debs.sh 替代这一步
touch test.sh

vim test.sh
#执行脚本 先打出来一个ceph的tar包, 手动将make-dict与exit这两行注释掉以后执行第二遍
sh test.sh

#大概需要运行 2个小时左右
#运行成功后 默认会在/tmp/release/Ubuntu/WORKDIR生成对应的deb安装包
#运行过程中可能会有错误，大概率是因为从github.com上下载包没有下载成功导致
sh test.sh

#!/usr/bin/env bash

set -xe

. /etc/os-release
base=${1:-/tmp/release}
releasedir=$base/$NAME/WORKDIR

mkdir -p $releasedir
#可以不打开
#清理工作目录中未被 Git 跟踪的文件和目录
#git clean -dxf

vers=$(git describe --match "v*" | sed s/^v//)
#打包代码，依赖 会生成ceph-$vers.tar.bz2
#建议第一次执行在下面 添加exit 0 
./make-dist $vers
exit 0


cp ceph-$vers.tar.bz2 $releasedir/ceph_$vers.orig.tar.bz2
tar -C $releasedir -jxf $releasedir/ceph_$vers.orig.tar.bz2

cp -a debian $releasedir/ceph-$vers/debian
cd $releasedir
perl -ni -e 'print if(!(/^Package: .*-dbg$/../^$/))' ceph-$vers/debian/control
perl -pi -e 's/--dbg-package.*//' ceph-$vers/debian/rules


dvers="$vers-1"

cd ceph-$vers

# ceph (18.2.4-1) stable; urgency=medium 取版本 18.2.4-1
chvers=$(head -1 debian/changelog | perl -ne 's/.*\(//; s/\).*//; print')
# 如果两个版本号获取不同，更改版本号
if [ "$chvers" != "$dvers" ]; then
   DEBEMAIL="contact@ceph.com" dch -D $VERSION_CODENAME --force-distribution -b -v "$dvers" "new version"
fi

#获取一半的cpu核数 计算后续的job的数量 -jxx
: ${NPROC:=$(($(nproc) / 2))}
#嫌慢可以全部cpu跑起来
#: ${NPROC:=$(nproc)}

if test $NPROC -gt 1 ; then
    j=-j${NPROC}
fi

#dpkg-buildpackage -j24 -uc -us -b
PATH=/usr/lib/ccache:$PATH dpkg-buildpackage $j -uc -us

cd ../..

mkdir -p $VERSION_CODENAME/conf
cat > $VERSION_CODENAME/conf/distributions <<EOF
Codename: $VERSION_CODENAME
Suite: stable
Components: main
Architectures: $(dpkg --print-architecture) source
EOF

if [ ! -e conf ]; then
    ln -s $VERSION_CODENAME/conf conf
fi

#reprepro --basedir /tmp/release/Ubuntu include focal WORKDIR/ceph_18.2.4-1_amd64.changes

reprepro --basedir $(pwd) include $VERSION_CODENAME WORKDIR/*.changes

echo $dvers > $VERSION_CODENAME/version

制作本机ubuntu源

#安装nginx
sudo apt install nginx

#将制作好的安装包路径 加个软连到nginx root_path下面
ln -s /tmp/release/Ubuntu /var/www/html/ubuntu

#修改nginx配置 添加autoindex on; 配置项 注意带分号
vi /etc/nginx/nginx.conf

#重启nginx服务
sudo systemctl restart nginx

#没有安装firewalld可以忽略这个步骤
#查看firewalld状态 如果是running 需要开放端口
sudo firewall-cmd --state
#开放80 443端口
sudo firewall-cmd --permanent --add-port=80/tcp
sudo firewall-cmd --permanent --add-port=443/tcp
#重新加载firewalld配置
sudo firewall-cmd --reload
#浏览器访问 http://主机IP/ubuntu 看看能不能正常访问



#主机IP 需要根据自己的实际ip修改
echo "deb [arch=amd64 trusted=yes] http://主机IP/ubuntu jammy main" >> /etc/apt/sources.list
#更新源
sudo apt update

#安装cephadm软件 注意看deb的路径是不是自己的本机
sudo apt install cephadm ceph-common

打包Docker镜像

cd ~

mkdir docker && cd docker

touch Dockerfile
#内容见下方，记得将"主机IP"修改自己的刚刚配置的主机地址
vi Dockerfile

#编译镜像
docker build -t 'my/ceph' .

#推送远端hub
docker push my/ceph:latest

FROM ubuntu:jammy


ENV DEBIAN_FRONTEND=noninteractive

#安装需要的软件
RUN apt-get update && apt-get install -y ca-certificates


#调整apt源
RUN echo "deb https://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiverse" > /etc/apt/sources.list && \
	echo "deb https://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiverse" >> /etc/apt/sources.list && \
	echo "deb https://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
	echo "deb https://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
	echo "deb [arch=amd64 trusted=yes] http://主机IP/ubuntu jammy main" >> /etc/apt/sources.list && \
	apt update

	
 

#安装ceph
RUN apt install -y ceph ceph-volume ceph-exporter  python3-cffi udev

#开放端口
EXPOSE 6789 6800 6801 6802 6803 6804 6805 80 5000

# 设置内存锁限制
RUN echo '@ceph - memlock 204800' >> /etc/security/limits.conf && \
    echo '@root - memlock 204800' >> /etc/security/limits.conf

# 记录初始磁盘使用量
RUN INITIAL_SIZE="$(bash -c 'sz="$(du -sm --exclude=/proc /)" ; echo "${sz%*/}"')"

# Postinstall 清理
RUN ( echo "apt clean" && apt-get clean && \
      echo "apt autoclean" && apt-get autoclean ) || \
      ( retval=$? && cat /var/log/apt/history.log && exit $retval )

# 清理不必要的 apt、deb、dpkg 数据
RUN rm -rf \
    /var/lib/apt/lists/* \
    /var/cache/debconf/* \
    /var/log/apt/ \
    /var/log/dpkg.log

# 编辑 lvm.conf 文件
RUN echo "About to edit lvm.conf" && \
	sed -i -e 's/^\([[:space:]#]*udev_rules =\) 1$/\1 0/' -e 's/^\([[:space:]#]*udev_sync =\) 1$/\1 0/' -e 's/^\([[:space:]#]*obtain_device_list_from_udev =\) 1$/\1 0/' /etc/lvm/lvm.conf && \
	echo "About to validate lvm.conf edits" && \
	# validate the sed command worked as expected
	grep -sqo "udev_sync = 0" /etc/lvm/lvm.conf && \
	grep -sqo "udev_rules = 0" /etc/lvm/lvm.conf && \
	grep -sqo "obtain_device_list_from_udev = 0" /etc/lvm/lvm.conf && \
	echo "Edits validated OK" && \
	mkdir -p /var/run/ganesha


# 清理不必要的文件和目录
RUN rm -rf \
    /etc/{selinux,systemd,udev} \
    /lib/{lsb,udev} \
    /tmp/* \
    /usr/lib{,64}/{locale,systemd,udev,dracut} \
    /usr/share/{doc,info,locale,man} \
    /usr/share/{bash-completion,pkgconfig/bash-completion.pc} \
    /var/log/* \
    /var/tmp/*

# 清理 Python 编译文件
RUN find / -xdev -name "*.pyc" -o -name "*.pyo" -exec rm -f {} \;

# 压缩 ceph-dencoder
RUN if [ -f /usr/bin/ceph-dencoder ]; then gzip -9 /usr/bin/ceph-dencoder; fi

# 清理 radosgw 相关文件
RUN rm -f /usr/lib/ceph/mgr/dashboard/static/AdminLTE-*/plugins/datatables/extensions/TableTools/images/psd/*

# 清空日志文件
RUN find /var/log/ -type f -exec truncate -s 0 {} \;

# 记录最终磁盘使用量
RUN FINAL_SIZE="$(bash -c 'sz="$(du -sm --exclude=/proc /)" ; echo "${sz%*/}"')"
RUN REMOVED_SIZE=$((INITIAL_SIZE - FINAL_SIZE))
RUN echo "Cleaning process removed ${REMOVED_SIZE}MB"
RUN echo "Dropped container size from ${INITIAL_SIZE}MB to ${FINAL_SIZE}MB"

# 验证安装的软件包
RUN apt-cache show \
    ca-certificates \
    e2fsprogs \
    ceph-common \
    ceph-mon \
    ceph-osd \
    ceph-mds \
    cephfs-mirror \
    cephfs-top \
    rbd-mirror \
    ceph-mgr \
    ceph-mgr-cephadm \
    ceph-mgr-dashboard \
    ceph-mgr-k8sevents \
    ceph-mgr-rook \
    ceph-grafana-dashboards \
    kmod \
    lvm2 \
    gdisk \
    smartmontools \
    nvme-cli \
    udev \
    sg3-utils \
    procps \
    hostname \
    radosgw \
    nfs-ganesha \
    nfs-ganesha-ceph \
    attr \
    ceph-fuse \
    rbd-nbd \
    ${CRIMSON_PACKAGES} \
    ceph-immutable-object-cache \
    ceph-volume \
    ceph-exporter \
    ceph-node-proxy

RUN echo 'Packages verified successfully'

遇到的报错

CMake Error at xsimd_ep-stamp/xsimd_ep-download-RELEASE.cmake:37 (message):

CMake Error at xsimd_ep-stamp/xsimd_ep-download-RELEASE.cmake:37 (message):
  Command failed: 1

   '/usr/bin/cmake' '-Dmake=' '-Dconfig=' '-P' '/tmp/release/Ubuntu/WORKDIR/ceph-18.2.4-35-gac3a54a4db5/obj-x86_64-linux-gnu/src/arrow/cpp/src/xsimd_ep-stamp/xsimd_ep-download-RELEASE-impl.cmake'

  See also

    /tmp/release/Ubuntu/WORKDIR/ceph-18.2.4-35-gac3a54a4db5/obj-x86_64-linux-gnu/src/arrow/cpp/src/xsimd_ep-stamp/xsimd_ep-download-*.log
  
#网络问题 最好加个代理 重新执行

Could not find platform independent libraries Consider setting $PYTHONHOME to [:<exec_prefix>]

mv /var/lib/dpkg/info/ /var/lib/dpkg/info_old/
mkdir /var/lib/dpkg/info/
apt-get update
apt-get -f install
mv /var/lib/dpkg/info/* /var/lib/dpkg/info_old/
rm -rf /var/lib/dpkg/info
mv /var/lib/dpkg/info_old/ /var/lib/dpkg/info

make-dict执行的太慢

由于make-dict 中需要下载外网资源 可以先下载完毕后,放到执行脚本目录，修改makde-dict注释download_from相关函数调用

cephadm启动报错 /usr/bin/ceph: stderr Error EIO: Module ‘cephadm’ has experienced an error and cannot handle commands: ContainerInspectInfo(image_id=‘438bbc41efbb76533bdb38441b1b804159559d6e170ac7bb2404dd8f3d6b1e58’, ceph_version=‘ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)’, repo_digests=[‘’]) Unable to set up “admin” label; assuming older version of Ceph

#查看RepoDigests为空，这个需要推送一个hub仓库填充一下即可
docker inspect imagesid

2024-11-18 09:52:18,352 7f21e3e421c0 INFO /usr/bin/docker: stderr FileNotFoundError: [Errno 2] No such file or directory: ‘udevadm’

#缺少udevadm软件
udevadm --version 
#上面不能正常运行，进行安装操作
apt install udev