当前位置：首页 > article >正文

搭建大数据技能竞赛比赛环境容器docker模块A-容器绑定物理网卡

article 2025/3/12 21:22:30

搭建大数据平台docker(master slave1 slave2)

Docker指定版本安装，docker 20.10
搭建大数据技能竞赛比赛环境容器docker模块A

环境规划

\	宿主机（centos7.9 min）	master(容器)	slave1(容器)	slave2(容器)
IP	192.168.100.100	192.168.100.101	192.168.100.102	192.168.100.103
镜像	CentOS-7.2009	bigdata:1.2	bigdata:1.0	bigdata:1.0
网卡	eth0	eth1	eth2	eth3
资源	2C_2G_50G	\	\	\

给VM宿主机配置4张网卡，一张网卡绑定一个容器。

系统配置

配置宿主机的IP为静态IP 192.168.100.100/24（自行配置）

更换国内阿里云yum源

curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

建立缓存

yum makecache

安装需要的依赖软件包

yum install -y yum-utils device-mapper-persistent-data lvm2 wget perl

下载阿里云docker-ce仓库

yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

安装docker

安装指定版本的docker-ce-20.10.20

yum install -y docker-ce-20.10.20 docker-ce-cli-20.10.20 containerd.io

查看版本

# docker -v
Docker version 20.10.20, build 9fdeb9c

配置镜像加速

mkdir /etc/docker

配置国内加速镜像

加速地址来源 docker镜像加速

cat >/etc/docker/daemon.json <<EOF
{
    "registry-mirrors": [
        "https://docker.1ms.run",
        "https://docker.xuanyuan.me"
    ]
}
EOF

重载配置&重启docker&设置开机自启

systemctl daemon-reload
systemctl restart docker
systemctl enable docker

构建镜像bigdata

拉取镜像 centos:7.9.2009

docker pull centos:7.9.2009

查看镜像是否正常拉取到

docker image list

构建bigdata:1.0镜像

编写Dockerfile

vi Dockerfile

FROM centos:7.9.2009

RUN mkdir -p /opt/software && mkdir -p /opt/module

#更换国内阿里云yum源
RUN curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
RUN sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo
RUN yum makecache

#安装语言包
RUN yum -y install kde-l10n-Chinese glibc-common vim
RUN localedef -c -f UTF-8 -i zh_CN zh_CN.utf8
RUN echo "LANG=zh_CN.UTF-8" >> /etc/locale.conf

#安装sshd
RUN yum install -y openssh-server openssh-clients vim net-tools lrzsz
RUN sed -i '/^HostKey/'d /etc/ssh/sshd_config
RUN echo 'HostKey /etc/ssh/ssh_host_rsa_key' >> /etc/ssh/sshd_config

#生成ssh-key
RUN ssh-keygen -t rsa -b 2048 -f /etc/ssh/ssh_host_rsa_key

#更改root用户登录密码
RUN echo 'root:123456' | chpasswd

#声明22端口 
EXPOSE 22

CMD [ "/usr/sbin/init"]

构建镜像

docker build -t bigdata:1.0 ./

查看镜像是否构建成功

# docker image list
REPOSITORY   TAG        IMAGE ID       CREATED         SIZE
bigdata      1.0        0f9f5e7a3278   3 minutes ago   1.34GB
centos       7.9.2009   eeb6ee3f44bd   3 years ago     204MB

构建bigdata:1.2镜像

再创建一个带mysql的容器来作为master节点

根据上面创建的bigdata:1.0镜像再添加一个数据库MYSQL5.7进去变为bigdata:1.2镜像

# 新建一个目录
mkdir master
cd master
vi Dockerfile

FROM bigdata:1.0

# 安装 MySQL 5.7 及依赖
RUN yum install -y wget && \
    wget https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm && \
    rpm -ivh mysql57-community-release-el7-11.noarch.rpm && \
    rm -r mysql57-community-release-el7-11.noarch.rpm && \
    yum install -y --nogpgcheck \
        mysql-community-server \
        mysql-community-client \
        mysql-community-libs \
        perl && \
    yum clean all && \
    rm -rf /var/cache/yum

# 创建数据目录并设置权限
RUN mkdir -p /var/lib/mysql && \
    chown -R mysql:mysql /var/lib/mysql

RUN echo 'bind-address = 0.0.0.0' >> /etc/my.cnf && cat /etc/my.cnf

# 添加初始化脚本
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

# 声明数据卷
VOLUME ["/var/lib/mysql"]

EXPOSE 3306 22
ENTRYPOINT ["/entrypoint.sh"]
CMD [ "/usr/sbin/init"]

在同一个目录下创建数据库初始化改密码脚本entrypoint.sh

#!/bin/bash
#entrypoint.sh
# 初始化数据库
if [ ! -d "/var/lib/mysql/mysql" ]; then
    mysqld --initialize --user=mysql --datadir=/var/lib/mysql
    /usr/sbin/mysqld --user=mysql --datadir=/var/lib/mysql &
    sleep 15
    temp_password=$(grep 'temporary password' /var/log/mysqld.log | awk '{print $NF}')
    mysql -uroot -p"$temp_password" --connect-expired-password -e \
      "ALTER USER 'root'@'localhost' IDENTIFIED BY '123456'; FLUSH PRIVILEGES;"
    mysql -uroot -p123456 -e \
      "GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION; FLUSH PRIVILEGES;"
    mysqladmin -uroot -p123456 shutdown
fi

exec "$@"

在同一目录下运行下面命令构建

# 构建镜像
docker build -t bigdata:1.2 .

查看docker所有镜像，可以看到已经完成构建（1.0版本的没有数据库适用于slaves节点，1.1版本有数据库适用于master节点）

# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
bigdata 1.1 27b8485f2320 12 minutes ago 2.34GB
bigdata 1.0 0f9f5e7a3278 28 hours ago 1.34GB
centos 7.9.2009 eeb6ee3f44bd 3 years ago 204MB

创建容器master、slave1、slave2

容器第一个网卡使用docker的默认网络，后面再挂载一个物理网卡

创建master容器

由于在安装某些服务时需要用到数据库服务，所以master要使用带数据库的bigdata:1.1镜像

运行这个创建master

docker create -it --name master --privileged=true -h master bigdata:1.1 /usr/sbin/init

进阶操作-数据持久化（这两个选一个）

为了数据库的数据持久化可以挂载一目录到容器（创建时要提前建好宿主机目录/opt/mysql）

docker create -it --name master --privileged=true -h master -v /opt/mysql:/var/lib/mysql bigdata:1.1 /usr/sbin/init

创建slave1、lsave2容器

使用bigdata:1.0镜像

docker create -it --name slave1 --privileged=true -h slave1 bigdata:1.0 /usr/sbin/init
docker create -it --name slave2 --privileged=true -h slave2 bigdata:1.0 /usr/sbin/init

查看容器

[root@Bigdata-A ~]# docker ps -a
CONTAINER ID   IMAGE         COMMAND            CREATED          STATUS    PORTS     NAMES
ecf92ac8a10c   bigdata:1.1   "/usr/sbin/init"   8 seconds ago    Created             slave2
742f93c9c67a   bigdata:1.0   "/usr/sbin/init"   10 seconds ago   Created             slave1
c93a42a85908   bigdata:1.0   "/usr/sbin/init"   14 seconds ago   Created             master

下载绑定物理网卡的脚本

wget https://raw.gitcode.com/xfcloud/BIgdataA/raw/main/docker/bigdata-A

赋予执行权限：

chmod +x bigdata-A

执行脚本：

./bigdata-A

脚本的使用方式：项目文件预览 - BIgdataA - GitCode

启动容器master slave1 slave2

由docker start master slave1 slave2变为脚本启动./bigdata-A

注意：每次重启时容器都会解绑物理网卡，所以每次启动都需要用脚本启动./bigdata-A

启动所有容器

./bigdata-A --start

检查网络状态

# docker exec -it master ifconfig eth1
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.100.101 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:8f:a8:4a txqueuelen 1000 (Ethernet)
RX packets 260229 bytes 384800593 (366.9 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 16055 bytes 1485007 (1.4 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

# docker exec -it slave1 ifconfig eth2
eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.100.102 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:8f:a8:54 txqueuelen 1000 (Ethernet)
RX packets 295 bytes 40306 (39.3 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 134 bytes 19836 (19.3 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

# docker exec -it slave2 ifconfig eth3
eth3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.100.103 netmask 255.255.255.0 broadcast 0.0.0.0
ether 00:0c:29:8f:a8:5e txqueuelen 1000 (Ethernet)
RX packets 300 bytes 41170 (40.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 137 bytes 20546 (20.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

测试与宿主机的连通性

# docker exec -it master ping 192.168.100.100 -c 4

PING 192.168.100.100 (192.168.100.100) 56(84) bytes of data.
64 bytes from 192.168.100.100: icmp_seq=1 ttl=64 time=0.468 ms
64 bytes from 192.168.100.100: icmp_seq=2 ttl=64 time=0.934 ms
64 bytes from 192.168.100.100: icmp_seq=3 ttl=64 time=1.21 ms
64 bytes from 192.168.100.100: icmp_seq=4 ttl=64 time=0.917 ms

--- 192.168.100.100 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 0.468/0.883/1.213/0.266 ms

测试master的数据库

# mysql -uroot -p123456 -h 192.168.100.101
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.44 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]>

测试前要自己先安装数据库客户端，可以直接yum install mysql -y安装