centos 7 静态编译docker

happy_boy

目的
　　1）去dockerfile，处理网络请求问题，缩短编译耗时
2）方便内部统一版本维护
3）方便接入内部流水线作业
4）编译static版本，避免系统库动态依赖问题

一、准备工作编译环境

1）系统环境

[root@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
[root@localhost ~]# uname  -a
Linux localhost.localdomain 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
docker version ： docker-ce-18.09
2）docker相关

　　docker-ce变化
dokcer从17.06 项目由moby变更为docker-ce,docker客户端与服务端项目分离开
docker客户端代码块在cli目录
docker服务端代码块在engine中
　　containerd runc proxy init未变化
相关项目文件见：
版本不同略有变化，最新版本（18.09），相关关联的commit id也在installer文件中
docker-ce/components/engine/hack/dockerfile/install/
containerd.installer    gometalinter.installer  proxy.installer         tini.installer          vndr.installer
dockercli.installer     install.sh              runc.installer          tomlv.installer
golang版本见：docker-ce/components/engine/Dockerfile.e2e

　　相关项目代码库：

　　https://github.com/opencontainers/runc.git  docker-runc
https://github.com/krallin/tini.git   docker-init
https://github.com/containerd/containerd.git  docker-containerd || docker-containerd-shim || docker-containerd-ctr
https://github.com/docker/docker-ce.git   docker || dockerd
https://github.com/docker/libnetwork.git docker-proxy

3）找到对应golang版本

cat docker-ce/components/engine/Dockerfile.e2e  
docker-ce/components/engine/Dockerfile.e2e:FROM golang:1.10.6-alpine3.7 as builder  
从docker与golang对应dockerfile找到原始对应关系
https://github.com/docker-library/golang/tree/366fe83ed839938cd04b2d546a06e2aee25a39a2
这边选择直接下载不用编译的go版本
https://dl.google.com/go/go1.10.6.linux-amd64.tar.gz
4）配置基础编译环境

　　根据containerd的docker镜像编译方式可知redhat系列的gcc版本过低且不支持enable-default-pie选项，需要安装编译gcc 6.3.x版本;
编译runc时报/usr/bin/ld: cannot find -lseccomp，默认的redhat系列无libseccomp-static支持

a)将原来的libseccomp软件包删掉

　　rpm -ivh https://cbs.centos.org/kojifiles/packages/libseccomp/2.3.2/3.el7/x86_64/libseccomp-2.3.2-3.el7.x86_64.rpm https://cbs.centos.org/kojifiles/packages/libseccomp/2.3.2/3.el7/x86_64/libseccomp-devel-2.3.2-3.el7.x86_64.rpm  https://cbs.centos.org/kojifiles/packages/libseccomp/2.3.2/3.el7/x86_64/libseccomp-static-2.3.2-3.el7.x86_64.rpm
[root@localhost src]# rpm -qa|grep  libseccomp
libseccomp-2.3.2-3.el7.x86_64
libseccomp-static-2.3.2-3.el7.x86_64
libseccomp-devel-2.3.2-3.el7.x86_64

b)编译使用gcc 6.3.0环境
　　yum group install "Development Tools"
yum install redhat-lsb rpm-build rpm-sign check dejagnu expect zlib-devel
[root@localhost github.com]# git clone https://github.com/BobSteagall/gcc-builder.git
Cloning into 'gcc-builder'...
remote: Enumerating objects: 215, done.
remote: Total 215 (delta 0), reused 0 (delta 0), pack-reused 215
Receiving objects: 100% (215/215), 35.75 KiB | 0 bytes/s, done.
Resolving deltas: 100% (149/149), done.
　　[root@localhost github.com]# cd gcc-builder/
　　[root@localhost gcc-builder]# git checkout gcc6
Branch gcc6 set up to track remote branch gcc6 from origin.
Switched to a new branch 'gcc6'
　　[root@localhost gcc-builder]# vi gcc-build-vars.sh
export GCC_VERSION=6.3.0
　　root@localhost gcc-builder]# vi configure-gcc.sh
elif [ "$GCC_PLATFORM" == "Linux" ]
then
$GCC_SRC_DIR/configure -v               \
--with-pkgversion="$GCC_PKG_NAME"   \
--enable-default-pie                \  ---增加enable-default-pie
--enable-languages=c,c++            \
　　[root@localhost gcc-builder]# ./build-gcc.sh | tee build.log
　　[root@localhost gcc-builder]# ./stage-gcc.sh
　　[root@localhost gcc-builder]# ./pack-gcc.sh
　　[root@localhost gcc-builder]# cd dist/usr/local
[root@localhost local]# cp -r  bin/ gcc/ /usr/local/
[root@localhost local]# chown -R root:root /usr/local/gcc/6.3.0/
[root@localhost local]# chown root:root /usr/local/bin/gcc630
[root@localhost local]# source /usr/local/bin/setenv-for-gcc630.sh
[root@localhost local]# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/local/gcc/6.3.0/libexec/gcc/x86_64-kewb-linux-gnu/6.3.0/lto-wrapper
Target: x86_64-kewb-linux-gnu
Configured with: /usr/local/docker/src/github.com/gcc-builder/gcc-6.3.0/configure -v --with-pkgversion='KEWB Computing Build' --prefix=/usr/local/gcc/6.3.0 --program-suffix= --enable-tls --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-default-pie --enable-languages=c,c++ --enable-lto --enable-bootstrap --disable-nls --disable-multilib --disable-install-libiberty --disable-werror --with-system-zlib
Thread model: posix
gcc version 6.3.0 (KEWB Computing Build)

二、下载软件

编译环境配置

go:  /usr/local/go1.10.6
dockerd:         /usr/local/docker/src/github.com/docker/docker  engine目录
docker:         /usr/local/docker/src/github.com/docker/cli
proxy:         /usr/local/docker/src/github.com/docker/libnetwork
init:         /usr/local/docker/src/github.com/tini
runc:        /usr/local/docker/src/github.com/opencontainers/runc
containerd: /usr/local/docker/src/github.com/containerd/containerd/

下载go设置环境变量

　　cd /usr/local
wget https://dl.google.com/go/go1.10.6.linux-amd64.tar.gz
mkdir -p /usr/local/go1.10.6
tar   -C /usr/local/go1.10.6 -zxvf  go1.10.6.linux-amd64.tar.gz
go_version=/usr/local/go1.10.6/go
export  PATH=${go_version}/bin/:$PATH
export GOROOT=${go_version}/
export GOPATH=/usr/local/docker
export DOCKER_GITCOMMIT=4c52b90/18.09

下载docker dockerd docker-proxy相关代码

　　mkdir -p /usr/local/docker/src/github.com/docker
cd /usr/local/docker/src/github.com/docker
git clone https://github.com/docker/docker-ce.git
git clone https://github.com/docker/libnetwork.git
cp -r docker-ce/components/engine docker
cp -r docker-ce/components/cli cli

下载docker-init相关代码

　　cd /usr/local/docker/src/github.com
git clone https://github.com/krallin/tini.git

下载docker-runc相关代码

　　mkdir -p /usr/local/docker/src/github.com/opencontainers
cd /usr/local/docker/src/github.com/opencontainers
git clone https://github.com/opencontainers/runc.git

下载docker-containerd...相关代码

　　mkdir -p /usr/local/docker/src/github.com/containerd
cd /usr/local/docker/src/github.com/containerd
git clone  https://github.com/containerd/containerd.git

三、编译二制文件

　　切换docker-ce至18.09版本
cd /usr/local/docker/src/github.com/docker/docker-ce
git checkout 18.09
Branch 18.09 set up to track remote branch 18.09 from origin.
Switched to a new branch '18.09'

1）据docker 编译命令安装基础软件包

　　docker-ce/components/packaging/image/Dockerfile.engine-dm
yum group install -y 'Development Tools'
yum install -y     bash     ca-certificates     cmake     gcc     git     glibc-static     libtool     make
yum install -y     btrfs-progs-devel     device-mapper-devel     libseccomp-devel     selinux-policy-devel     systemd-devel

2）编译runc

　　根据docker-ce/components/engine/hack/dockerfile/install/runc.installer切换至对应commit id(RUNC_COMMIT=96ec2177ae841256168fcf76954f7177af9446eb)
cd /usr/local/docker/src/github.com/opencontainers/runc
[root@localhost runc]#  git checkout -q 96ec2177ae841256168fcf76954f7177af9446eb
　　#If using RHEL7 kernels (3.10.0 el7), disable kmem accounting/limiting
[root@localhost runc]# make BUILDTAGS="seccomp apparmor selinux nokmem"  static
[root@localhost runc]# ldd runc
not a dynamic executable

3）编译containerd
　　据编译命令编译

　　docker-ce/components/engine/hack/dockerfile/install/containerd.installer
CONTAINERD_COMMIT=9754871865f7fe2f4e74d43e2fc7ccd237edcbce # v1.2.2
　　cd /usr/local/docker/src/github.com/containerd/containerd/
[root@localhost containerd]# git checkout -q  9754871865f7fe2f4e74d43e2fc7ccd237edcbce
[root@localhost containerd]# make EXTRA_FLAGS="-buildmode pie" EXTRA_LDFLAGS='-extldflags "-fno-PIC -static"' BUILDTAGS="netgo osusergo static_build"
　　[root@localhost containerd]# ldd bin/ctr
not a dynamic executable
[root@localhost containerd]# ldd bin/containerd*
bin/containerd:
not a dynamic executable
bin/containerd-shim:
not a dynamic executable
bin/containerd-shim-runc-v1:
not a dynamic executable
bin/containerd-stress:
not a dynamic executable

4)编译docker-init

　　cd /usr/local/docker/src/github.com/tini
[root@localhost tini]# git checkout -q fec3683b971d9c3ef73f284f176672c44b44866
[root@localhost tini]# cmake .
[root@localhost tini]# make tini-static
[root@localhost tini]# ldd tini-static
not a dynamic executable
[root@localhost tini]# cp tini-static docker-init

5)编译docker-proxy

　　cd /usr/local/docker/src/github.com/docker/libnetwork
[root@localhost libnetwork]# git checkout -q 2cfbf9b1f98162a55829a21cc603c76072a75382
[root@localhost libnetwork]# CGO_ENABLED=0 go build   -o docker-proxy github.com/docker/libnetwork/cmd/proxy
[root@localhost libnetwork]# ldd docker-proxy
not a dynamic executable

6)编译docker dockerd

　　cd /usr/local/docker/src/github.com/docker/cli
[root@localhost cli]#export VERSION=18.09
[root@localhost cli]#export GITCOMMIT=4c52b90
[root@localhost cli]# make binary
　　WARNING: you are not in a container.
Use "make -f docker.Makefile binary" or set
DISABLE_WARN_OUTSIDE_CONTAINER=1 to disable this warning.
　　Press Ctrl+C now to abort.
　　WARNING: binary creates a Linux executable. Use cross for macOS or Windows.
./scripts/build/binary
Building statically linked build/docker-linux-amd64
[root@localhost cli]# ldd build/docker
not a dynamic executable
　　[root@localhost cli]# build/docker -v
Docker version 18.09, build 4c52b90
　　cd /usr/local/docker/src/github.com/docker/docker
[root@localhost docker]# hack/make.sh binary
#WARNING! I don't seem to be running in a Docker container.
#The result of this command might be an incorrect build, and will not be
#officially supported.
#Try this instead: make all

　　Removing bundles/
　　---> Making bundle: binary (in bundles/binary)
Building: bundles/binary-daemon/dockerd-18.09
github.com/docker/docker/cmd/dockerd
/tmp/go-link-867197439/000008.o: In function mygetgrouplist':/usr/local/go1.10.6/go/src/os/user/getgrouplist_unix.go:15: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking/tmp/go-link-867197439/000007.o: In functionmygetgrgid_r':
/usr/local/go1.10.6/go/src/os/user/cgo_lookup_unix.go:38: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-867197439/000007.o: In function mygetgrnam_r':/usr/local/go1.10.6/go/src/os/user/cgo_lookup_unix.go:43: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking/tmp/go-link-867197439/000007.o: In functionmygetpwnam_r':
/usr/local/go1.10.6/go/src/os/user/cgo_lookup_unix.go:33: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/go-link-867197439/000007.o: In function `mygetpwuid_r':

　　/usr/local/go1.10.6/go/src/os/user/cgo_lookup_unix.go:28: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Created binary: bundles/binary-daemon/dockerd-18.09
　　[root@localhost docker]# bundles/binary-daemon/dockerd -v
Docker version 18.09, build 4c52b90/18.09
[root@localhost docker]# ldd bundles/binary-daemon/dockerd
not a dynamic executable

拷贝编译完成的docker相关二制文件
　　[root@localhost ~]# mkdir /tmp/18.09-docker
[root@localhost ~]#cd /usr/local/docker/src/github.com
[root@localhost github.com]# cp docker/libnetwork/docker-proxy  /tmp/18.09-docker/
[root@localhost github.com]# cp docker/cli/build/docker /tmp/18.09-docker/
[root@localhost github.com]# cp docker/docker/bundles/binary-daemon/dockerd  /tmp/18.09-docker/
[root@localhost github.com]# cp containerd/containerd/bin/ctr  containerd/containerd/bin/containerd containerd/containerd/bin/containerd-shim   /tmp/18.09-docker/
[root@localhost github.com]# cp tini/docker-init /tmp/18.09-docker/
[root@localhost github.com]# cp  opencontainers/runc/runc /tmp/18.09-docker/
[root@localhost github.com]# ls -lrt /tmp/18.09-docker/
total 160688
-rwxr-xr-x. 1 root root  2841376 Jan 25 01:38 docker-proxy
-rwxr-xr-x. 1 root root 50711753 Jan 25 01:38 docker
-rwxr-xr-x. 1 root root 53918880 Jan 25 01:39 dockerd
-rwxr-xr-x. 1 root root 28075792 Jan 25 01:40 containerd
-rwxr-xr-x. 1 root root  4968800 Jan 25 01:40 containerd-shim
-rwxr-xr-x. 1 root root 15816304 Jan 25 01:40 ctr
-rwxr-xr-x. 1 root root   845080 Jan 25 01:41 docker-init
-rwxr-xr-x. 1 root root  7352008 Jan 25 01:42 runc
　　[root@localhost ~]# docker version
Client:
Version:           18.09
API version:       1.39
Go version:        go1.10.6
Git commit:        4c52b90
Built:             Fri Jan 25 10:31:01 2019
OS/Arch:           linux/amd64
Experimental:      false
　　Server:
Engine:
Version:          18.09
API version:      1.39 (minimum version 1.12)
Go version:       go1.10.6
Git commit:       4c52b90/18.09
Built:            Fri Jan 25 10:33:42 2019
OS/Arch:          linux/amd64
Experimental:     false

参考资料：
　　https://bobsteagall.com/2017/12/30/gcc-builder/
https://askubuntu.com/questions/1078516/disable-enable-default-pie-for-gcc
https://github.com/rust-lang/rust/issues/47037
http://gnu.mirror.constant.com/gcc/
https://github.com/containerd/containerd/blob/master/BUILDING.md
https://cbs.centos.org/koji/buildinfo?buildID=17550
https://download.docker.com/linux/static/stable/x86_64/