python插件 docker build离线安装手记
由于公司线上安全要求，不能直接使用pip安装，也没有自行部署pypi源，于是有了此篇文章。本文主要以etcd3插件为示例。要求：
python插件需要附属在自己的软件包程序中，以os.path.append的方式加载
实现步骤：
1、搜索找到etcd3的pthon插件
A）通过找pypi插件界面：https://pypi.org/project/etcd3/并下载最新软件包
B)在pypi插件界面进入项目的homepage地址：https://github.com/kragniz/python-etcd3
C) 在homepage中找到关联插件包说明：https://github.com/kragniz/python-etcd3/blob/master/requirements.txt
关联插件：grpcio>=1.2.0tenacity==4.10.0 protobuf==3.5.2.post1
2.1、搜索etcd3关联插件grpcio
A)下载相关包 https://pypi.org/project/grpcio/
B) 进入homepage地址：https://grpc.io ，找到python相关：https://grpc.io/docs/quickstart/python.html，找到源码地址：https://github.com/grpc/grpc
C) 在软件版本中找到关联插件包说明：https://github.com/grpc/grpc/blob/master/requirements.txt
关联插件：coverage>=4.0 cython>=0.27 enum34>=1.0.4 protobuf>=3.5.0.post1 six>=1.10 wheel>=0.29
2.2、搜索etcd3关联插件tenacity
A) 下载相关包 https://pypi.org/project/tenacity/
B) 进入homepage地址：https://github.com/jd/tenacity/
C）关联插件包说明：https://github.com/jd/tenacity/blob/master/requirements.txt
关联插件：six>=1.9.0 futures>=3.0;python_version=='2.7' monotonic>=0.6;python_version=='2.7'
2.3、搜索etcd3关联插件protobuf
A） 下载相关包 https://pypi.org/project/protobuf/
B) 进入homepage地址https://developers.google.com/protocol-buffers/ 找到源码地址：https://github.com/google/protobuf/tree/master/python
C) 确认无关联插件
3、 搜索grpcio、tenacity关联插件....至确认无相关关联
4、安装时，软件安装时诸如grpcio插件，作者在写pypi插件时，有些必要的软件包会及时提示，直接下载过来安装即可；
但有些依赖性错误找不到方向，跟自己写代码一样，总有些写好了依赖，在交付时运行代码不一定检查，所以有必要结合源码的关联插件信息，诸如：
gcc: error: src/python/grpcio/grpc/_cython/cygrpc.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
creating tmp/tmpe125oS
gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c /tmp/tmpe125oS/a.c -o tmp/tmpe125oS/a.o
Traceback (most recent call last):
File "setup.py", line 310, in <module>
cmdclass=COMMAND_CLASS,
File "/usr/lib64/python2.7/distutils/core.py", line 152, in setup
dist.run_commands()
File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands
self.run_command(cmd)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/site-packages/setuptools/command/install.py", line 73, in run
self.do_egg_install()
File "/usr/lib/python2.7/site-packages/setuptools/command/install.py", line 93, in do_egg_install
self.run_command('bdist_egg')
File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/site-packages/setuptools/command/bdist_egg.py", line 185, in run
cmd = self.call_command('install_lib', warn_dir=0)
File "/usr/lib/python2.7/site-packages/setuptools/command/bdist_egg.py", line 171, in call_command
self.run_command(cmdname)
File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/site-packages/setuptools/command/install_lib.py", line 20, in run
self.build()
File "/usr/lib64/python2.7/distutils/command/install_lib.py", line 111, in build
self.run_command('build_ext')
File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command
self.distribution.run_command(command)
File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command
cmd_obj.run()
File "/usr/lib/python2.7/site-packages/setuptools/command/build_ext.py", line 52, in run
_build_ext.run(self)
File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/tmp/grpcio-1.12.0/src/python/grpcio/commands.py", line 297, in build_extensions
"Failed `build_ext` step:\n{}".format(formatted_exception))
commands.CommandError: Failed `build_ext` step:
Traceback (most recent call last):
File "/tmp/grpcio-1.12.0/src/python/grpcio/commands.py", line 292, in build_extensions
build_ext.build_ext.build_extensions(self)
File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 448, in build_extensions
self.build_extension(ext)
File "/usr/lib/python2.7/site-packages/setuptools/command/build_ext.py", line 186, in build_extension
_build_ext.build_extension(self,ext)
File "/usr/lib64/python2.7/distutils/command/build_ext.py", line 498, in build_extension
depends=ext.depends)
File "/usr/lib64/python2.7/distutils/ccompiler.py", line 574, in compile
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
File "/usr/lib64/python2.7/distutils/unixccompiler.py", line 132, in _compile
raise CompileError, msg
CompileError: command 'gcc' failed with exit status 4
5、由于是将python插件直接copy至软件目录再import的方式使用，期间遇到类似这样的问题：
无grpc类等模块问题
ImportError: No module named grpc
将site-packages的grpcio-1.12.0-py2.7-linux-x86_64.egg文件拷贝至存放第三方包目录下后用unzip解压出来即可
unzipgrpcio-1.12.0-py2.7-linux-x86_64.egg 会解压出grpc目录，sys.path.append('第三方目录')，import grpc -->help() -->grpc查看对应类方法等
有类，方法不存在问题
Traceback (most recent call last):
File "test.py", line 25, in <module>
import tuplesync
File "/root/test/src/lcp/tuplesync.py", line 11, in <module>
import link_master as lm
File "/root/test/src/lcp/link_master.py", line 26, in <module>
import etcd3
File "/root/test/py_third/etcd3/__init__.py", line 3, in <module>
import etcd3.etcdrpc as etcdrpc
File "/root/test/py_third/etcd3/etcdrpc/__init__.py", line 1, in <module>
from .rpc_pb2 import *
File "/root/test/py_third/etcd3/etcdrpc/rpc_pb2.py", line 18, in <module>
from etcd3.etcdrpc import auth_pb2 as auth__pb2
File "/root/test/py_third/etcd3/etcdrpc/auth_pb2.py", line 217, in <module>
import grpc
File "/root/test/py_third/grpc/__init__.py", line 192, in <module>
> File "/root/test/py_third/grpc/__init__.py", line 203, in ChannelConnectivity
> AttributeError: 'module' object has no attribute 'ConnectivityState'
这个问题比较困惑，毕竟软件包等都是安装成功的而且也能看到相关方法，后经同事提醒他之前安装需要拷贝一个pkg_resources的目录，然后去查了查，这个目录是用来管理ptyon插件信息的，于是将编译机的pkg_resources.py也拷贝至第三方包目录运行软件后一切正常，这个pkg_resources.py文件是setuptools-39.1.0生成的。
5、在试验中，都通过了验证后，将流程串联起来。
平台发起打包命令docker buildbuild-arg用于传递版本参数（区分打包版本），ftp密码（上传到对应内部yum源），ftp目录等（区分测试包、正式包）
大概流程：
角色：打包平台；jenkins任务；git代码仓库；特定打包机；yum源
a、流水线平台界面操作，触发jenkins任务->拉取代码，执行特定脚本传递参数等
b、jenkins任务在特定一台机上执行docker build，安装相关依赖，拉取内部代码，build成rpm包，上传到内部yum源
在docker build的时候发现一个问题，纯手工操作，或者docker run个基础镜像后手动执行python插件安装一步步操作毫无问题；可是让自动build的grpcio总是提示CompileError: command 'gcc' failed with exit status 4，毫无头绪，后来先将grpcio相关依赖包很打了一个镜像B后，再执行安装grpcio插件也是如此；在镜像B的基础上验证测试无果后，随机重新安装了一次无相关的插件，再次执行安装grpcio就通过了；于是在docker build自动安装的过程中执行grpcio安装前，再执行一次其他插件的安装，这样编译就通过了，怀疑是插件未及时更新，但验证镜像B时所有的相关包都可以正常用python引用，神奇的问题。
后记：
本次自动安装python插件(先后顺序，dnspython重复）
six-1.11.0
setuptools-39.1.0
dnspython-1.15.0
ullib3-1.22
pyDatalog-0.17.1
pbr-4.0.3
tenacity-4.12.0
enum34-1.1.6
monotonic-1.5
futures-3.2.0
protobuf-3.5.2
dnspython-1.15.0
grpcio-1.12.0
etcd3-0.8.0
页:
[1]