CentOS7.3.1611部署k8s1.5.2集群

ikyujyhtg

CentOS7.3.1611部署k8s1.5.2集群

参考资料
Kubernetes权威指南（第二版）
http://jevic.blog.iyunv.com/2183736/1881455
https://my.oschina.net/u/1791060/blog/830023
http://blog.csdn.net/lic95/article/details/55015284
https://coreos.com/etcd/docs/latest/clustering.html


架构
k8s-master  安装etcd，kubernetes-server/client
k8s-node1   安装docker,kubernetes-node/client，flannel
k8s-node2   安装docker,kubernetes-node/client，flannel


一，YUM安装的版本如下
CentOS7.3.1611 Yum安装
kubernetes-1.4.0-0.1.git87d9d8d.el7
会安装kubernets-master,node,client及其相关依赖项
kubernetes-master-1.4.0-0.1.git87d9d8d.el7
会产生三个二进制程序kube-apiserver  kube-controller-manager  kube-scheduler
kubernetes-node-1.4.0-0.1.git87d9d8d.el7
会安装很多依赖包包括docker-1.12.5-14.el7.centos，会安装kubelet  kube-proxy
kubernetes-client-1.4.0-0.1.git87d9d8d.el7
会产生一个二进制程序kubectl
kubernetes-unit-test-1.4.0-0.1.git87d9d8d.el7
会安装很多依赖包包括etcd-3.0.15-1.el7，golang，gcc，glibc，rsync等                 
flannel-0.5.5-2.el7
会产生一个二进制程序flannel


二，本文选择二进制包版本安装最新版测试
github地址：
etct: https://github.com/coreos/etcd/releases
flannel: https://github.com/coreos/flannel/releases
kubernetes: https://github.com/kubernetes/kubernetes/releases
docker: https://docs.docker.com/engine/installation/linux/centos/

k8s 1.5.2
https://dl.k8s.io/v1.5.2/kubernetes-server-linux-amd64.tar.gz
会产生11个二进制程序hyperkube  kubectl   kubelet  kube-scheduler  kubeadm  kube-controller-manager  kube-discovery  kube-proxy  kube-apiserver kube-dns  kubefed         
https://dl.k8s.io/v1.5.2/kubernetes-client-linux-amd64.tar.gz
会产生两个二进制程序kube-proxy  kubefed
etcd 3.1.10
https://github.com/coreos/etcd/r ... -linux-amd64.tar.gz
docker 1.13.1
https://get.docker.com/builds/Linux/x86_64/docker-1.13.1.tgz
flannel
https://github.com/coreos/flanne ... -linux-amd64.tar.gz


三，部署环境
1，准备工作
1），系统最小化安装,然后yum update,升级到最新版本CentOS7.3.1611
2），设置hostname及hosts
[iyunv@k8s-master ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.17.3.20  k8s-master
172.17.3.7   k8s-node1
172.17.3.8   k8s-node2
3），校对时间
[iyunv@k8s-master ~]# ntpdate ntp1.aliyun.com &&hwclock -w
4），关闭selinux及防火墙
[iyunv@k8s-master ~]# sed -i s'/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
[iyunv@k8s-master ~]# systemctl disable firewalld; systemctl stop firewalld
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
5），重启服务器

2，Master节点部署
1），部署etcd服务（目前单点）
[iyunv@k8s-master ~]# tar zxvf etcd-v3.1.0-linux-amd64.tar.gz -C /usr/local/
[iyunv@k8s-master ~]# mv /usr/local/etcd-v3.1.0-linux-amd64/ /usr/local/etcd
[iyunv@k8s-master ~]# ln -s /usr/local/etcd/etcd /usr/local/bin/etcd
[iyunv@k8s-master ~]# ln -s /usr/local/etcd/etcdctl /usr/local/bin/etcdctl

设置systemd服务文件/usr/lib/systemd/system/etcd.service
[Unit]
Description=Eted Server
After=network.target

[Service]
WorkingDirectory=/data/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd
Type=notify
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

其中WorkingDirector表示etcd数据保存的目录，需要在启动etcd服务之前进行创建

etcd单点默认配置
[iyunv@k8s-master ~]# cat /etc/etcd/etcd.conf
ETCD_NAME=k8s1
ETCD_DATA_DIR="/data/etcd"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

etcd服务启动
[iyunv@k8s-master ~]# systemctl daemon-reload
[iyunv@k8s-master ~]# systemctl enable etcd.service
[iyunv@k8s-master ~]# systemctl start etcd.service

etcd服务检查
[iyunv@k8s-master ~]# etcdctl cluster-health
member 869f0c691c5458a3 is healthy: got healthy result from http://0.0.0.0:2379
cluster is healthy
[iyunv@k8s-master ~]# etcdctl member list
869f0c691c5458a3: name=k8s1 peerURLs=http://172.17.3.20:2380 clientURLs=http://0.0.0.0:2379 isLeader=true


2）部署kube-apiserver服务
安装kube-apiserver
[iyunv@k8s-master ~]# tar zxvf kubernetes-server-linux-amd64.tar.gz  -C /usr/local/
[iyunv@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-apiserver /usr/local/bin/kube-apiserver
其他服务顺便做下软链接
[iyunv@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/hyperkube /usr/local/bin/hyperkube
[iyunv@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kubeadm /usr/local/bin/kubeadm
[iyunv@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-controller-manager /usr/local/bin/kube-controller-manager
[iyunv@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kubectl  /usr/local/bin/kubectl
[iyunv@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-discovery /usr/local/bin/kube-discovery
[iyunv@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-dns  /usr/local/bin/kube-dns
[iyunv@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kubefed /usr/local/bin/kubefed
[iyunv@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kubelet /usr/local/bin/kubelet      
[iyunv@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-proxy /usr/local/bin/kube-proxy
[iyunv@k8s-master ~]# ln -s /usr/local/kubernetes/server/bin/kube-scheduler /usr/local/bin/kube-scheduler

配置kubernetes system config
[iyunv@k8s-master ~]# cat /etc/kubernetes/config
KUBE_LOGTOSTDERR="--logtostderr=false"
KUBE_LOG_DIR="--log-dir=/data/logs/kubernetes"
KUBE_LOG_LEVEL="--v=2"
KUBE_ALLOW_PRIV="--allow-privileged=false"
KUBE_MASTER="--master=http://172.17.3.20:8080"

设置systemd服务文件/usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target
After=etcd.service

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/apiserver
User=kube
ExecStart=/usr/bin/kube-apiserver \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBE_ETCD_SERVERS \
            $KUBE_API_ADDRESS \
            $KUBE_API_PORT \
            $KUBELET_PORT \
            $KUBE_ALLOW_PRIV \
            $KUBE_SERVICE_ADDRESSES \
            $KUBE_ADMISSION_CONTROL \
            $KUBE_API_ARGS
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

配置kuber-apiserver启动参数
[iyunv@k8s-master ~]# cat /etc/kubernetes/apiserver
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"
KUBE_API_ARGS=" "

启动kube-api-servers服务
[iyunv@k8s-master ~]# systemctl daemon-reload
[iyunv@k8s-master ~]# systemctl enable kube-apiserver.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.
[iyunv@k8s-master ~]# systemctl start kube-apiserver.service

验证服务
http://172.17.3.20:8080/


3）部署kube-controller-manager服务
设置systemd服务文件/usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=kube-apiserver.service
Requires=kube-apiserver.service

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/controller-manager
ExecStart=/usr/local/bin/kube-controller-manager \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBE_LOG_DIR \
            $KUBE_MASTER \
            $KUBE_CONTROLLER_MANAGER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target


配置kube-controller-manager启动参数
[iyunv@k8s-master ~]# cat /etc/kubernetes/controller-manager
KUBE_CONTROLLER_MANAGER_ARGS=""

启动kube-controller-manager服务
[iyunv@k8s-master ~]# systemctl daemon-reload
[iyunv@k8s-master ~]# systemctl enable kube-controller-manager
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.
[iyunv@k8s-master ~]# systemctl start kube-controller-manager


4）部署kube-scheduler服务
设置systemd服务文件/usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler Plugin
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=kube-apiserver.service
Requires=kube-apiserver.service

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/scheduler
ExecStart=/usr/local/bin/kube-scheduler \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBE_LOG_DIR \
            $KUBE_MASTER \
            $KUBE_SCHEDULER_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

配置kube-schedulerr启动参数
[iyunv@k8s-master ~]# cat /etc/kubernetes/schedulerr
KUBE_SCHEDULER_ARGS=""

启动kube-scheduler服务
[iyunv@k8s-master ~]# systemctl daemon-reload
[iyunv@k8s-master ~]# systemctl enable kube-scheduler
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.
[iyunv@k8s-master ~]# systemctl start kube-scheduler


2，Node节点部署
1）安装docker(或者yum instll docker)
[iyunv@k8s-node1 ~]# tar zxvf docker-1.13.1.tgz -C /usr/local
这里默认docker安装并启动，方便后面测试
[iyunv@k8s-node1 ~]# systemctl start docker.service


2）安装kubernetes客户端
安装kubelet，kube-proxy
[iyunv@k8s-master ~]# tar zxvf kubernetes-client-linux-amd64.tar.gz  -C /usr/local/
[iyunv@k8s-node1 ~]# ln -s /usr/local/kubernetes/client/bin/kubectl /usr/local/bin/kubectl
[iyunv@k8s-node1 ~]# ln -s /usr/local/kubernetes/client/bin/kubefed /usr/local/bin/kubefed
kube-proxy包默认client没有可以从server拷贝过来
[iyunv@k8s-node1 ~]# ln -s /usr/local/kubernetes/client/bin/kube-proxy /usr/local/bin/kube-proxy
[iyunv@k8s-node1 ~]# ln -s /usr/local/kubernetes/client/bin/kubelet /usr/local/bin/kubelet  


3）部署kubelet服务
配置kubernetes system config
[iyunv@k8s-node1 ~]# cat /etc/kubernetes/config
KUBE_LOGTOSTDERR="--logtostderr=false"
KUBE_LOG_DIR="--log-dir=/data/logs/kubernetes"
KUBE_LOG_LEVEL="--v=2"
KUBE_ALLOW_PRIV="--allow-privileged=false"
KUBE_MASTER="--master=http://172.17.3.20:8080"

设置systemd服务文件/usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=docker.service
Requires=docker.service

[Service]
WorkingDirectory=/data/kubelet
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBE_LOG_DIR \
            $KUBELET_API_SERVER \
            $KUBELET_ADDRESS \
            $KUBELET_PORT \
            $KUBELET_HOSTNAME \
            $KUBE_ALLOW_PRIV \
            $KUBELET_POD_INFRA_CONTAINER \
            $KUBELET_ARGS
Restart=on-failure

[Install]
WantedBy=multi-user.target

配置kubelet启动参数
[iyunv@k8s-node1 ~]# cat /etc/kubernetes/kubelet
KUBELET_ADDRESS="--address=0.0.0.0"
KUBELET_PORT="--port=10250"
KUBELET_HOSTNAME="--hostname-override=k8s-node1"
KUBELET_API_SERVER="--api-servers=http://172.17.3.20:8080"
KUBELET_POD_INFRA_CONTAINER="--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest"
KUBELET_ARGS=""

启动kubelet服务
[iyunv@k8s-node1 ~]# systemctl daemon-reload
[iyunv@k8s-node1 ~]# systemctl enable kubelet.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[iyunv@k8s-node1 ~]# systemctl start kubelet.service


4），部署kube-proxy服务

设置systemd服务文件/usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=network.target

[Service]
EnvironmentFile=-/etc/kubernetes/config
EnvironmentFile=-/etc/kubernetes/proxy
ExecStart=/usr/local/bin/kube-proxy \
            $KUBE_LOGTOSTDERR \
            $KUBE_LOG_LEVEL \
            $KUBE_LOG_DIR \
            $KUBE_MASTER \
            $KUBE_PROXY_ARGS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

配置kubelet启动参数
[iyunv@k8s-node1 ~]# cat /etc/kubernetes/proxy
KUBE_PROXY_ARGS=""

启动kubelet服务
[iyunv@k8s-node1 ~]# systemctl daemon-reload
[iyunv@k8s-node1 ~]# systemctl enable kube-proxy.service
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.
[iyunv@k8s-node1 ~]# systemctl start kube-proxy.service

验证节点是否启动
[iyunv@k8s-node1 ~]# kubectl get nodes
NAME        STATUS    AGE
k8s-node1   Ready     9m



3，配置网络
1），配置etcd
[iyunv@k8s-master ~]# etcdctl set /k8s/network/config '{ "Network": "10.1.0.0/16" }'
{ "Network": "10.1.0.0/16" }
[iyunv@k8s-master ~]# etcdctl get /k8s/network/config   
{ "Network": "10.1.0.0/16" }



2)，安装flannel
[iyunv@k8s-master ~]# tar -zxvf flannel-v0.7.0-linux-amd64.tar.gz  -C /usr/local/bin/
flanneld
mk-docker-opts.sh
README.md

[iyunv@k8s-node1 ~]# tar zxvf flannel-v0.7.0-linux-amd64.tar.gz -C /usr/local/flannel
[iyunv@k8s-node1 ~]# ln -s /usr/local/flannel/flannel /usr/local/bin/flanneld
[iyunv@k8s-node1 ~]# ln -s /usr/local/flannel/mk-docker-opts.sh   /usr/local/bin/mk-docker-opts.sh


3），配置flannel（配置比较麻烦，start脚本和启动脚本参考yum安装时生成的配置）
设置systemd服务文件[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
EnvironmentFile=/etc/sysconfig/flanneld
EnvironmentFile=-/etc/sysconfig/docker-network
ExecStart=/usr/local/bin/flanneld-start $FLANNEL_OPTIONS
ExecStartPost=/usr/local/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service

其中flanneld-start为
[iyunv@k8s-node1 ~]# cat /usr/local/bin/flanneld-start
#!/bin/sh

exec /usr/local/bin/flanneld \
        -etcd-endpoints=${FLANNEL_ETCD_ENDPOINTS:-${FLANNEL_ETCD}} \
        -etcd-prefix=${FLANNEL_ETCD_PREFIX:-${FLANNEL_ETCD_KEY}} \
        "$@"


编辑flannel，设置etcd相关信息
[iyunv@k8s-node1 ~]# cat /etc/sysconfig/flanneld
FLANNEL_ETCD_ENDPOINTS="http://172.17.3.20:2379"
FLANNEL_ETCD_PREFIX="/k8s/network"


4），启动flannel
[iyunv@k8s-node1 ~]# systemctl daemon-reload
[iyunv@k8s-node1 ~]# systemctl enable flanneld.service
[iyunv@k8s-node1 ~]# systemctl start flanneld.service

完成后确认网络接口docker0的IP地址属于flannel0的子网：
[iyunv@k8s-node1 ~]# ip addr
6: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN qlen 500
    link/none
    inet 10.1.89.0/16 scope global flannel0
       valid_lft forever preferred_lft forever
7: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN
    link/ether 02:42:f1:e4:7c:a3 brd ff:ff:ff:ff:ff:ff
    inet 10.1.89.1/24 scope global docker0
       valid_lft forever preferred_lft forever