|
|
OpenShift简介
微服务架构应用日渐广泛，Docker和Kubernetes技术是不可或缺的。Red Hat OpenShift 3是建立在Docker和Kubernetes基础之上的容器应用平台，用于开发和部署企业应用程序。
OpenShift版本
OpenShift Dedicated(Enterprise)
- Private, high-availability OpenShift clusters hosted on Amazon Web Services or Google Cloud Platform
- Delivered as a hosted service and supported by Red Hat
OpenShift Container Platform(Enterprise)
- Across cloud and on-premise infrastructure
- Customizable, with full administrative control
OKD
OpenShift开源社区版(Origin Community Distribution of Kubernetes)
OpenShift架构
- Master Node提供的组件：API Server (负责处理客户端请求, 包括node、user、 administrator和其他的infrastructure系统)；Controller Manager Server (包括scheduler和replication controller)；OpenShift客户端工具 (oc)
- Compute Node(Application Node) 部署application
- Infra Node 运行router、image registry和其他的infrastructure服务
- etcd 可以部署在Master Node，也可以单独部署， 用来存储共享数据：master state、image、 build、deployment metadata等
- Pod 最小的Kubernetes object，可以部署一个或多个container
安装计划
软件环境
- AWS RHEL 7.5/CentOS 7.6
- OKD 3.10/3.11
- Ansible 2.6.3/2.7.5
- Docker 1.13.1
- Kubernetes 1.10/1.11
使用Ansible安装openshift，仅需配置一些Node信息和参数即可完成集群安装，大大提高了安装速度。
本文档也适用于CentOS 7：
CentOS 7不需订阅这一步骤，需安装NetworkManager：
# yum -y install NetworkManager
# systemctl start NetworkManager
CentOS 7需编辑/etc/sysconfig/network-scripts/ifcfg-eth0，增加NM_CONTROLLED=yes，否则不能成功安装ServiceMonitor。
安装openshift后各节点会自动增加yum仓库CentOS-OpenShift-Origin310/311.repo，其内容如下：
[centos-openshift-origin311]
name=CentOS OpenShift Origin
baseurl=http://mirror.centos.org/centos/7/paas/x86_64/openshift-origin311/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-PaaS
[centos-openshift-origin311-testing]
name=CentOS OpenShift Origin Testing
baseurl=http://buildlogs.centos.org/centos/7/paas/x86_64/openshift-origin311/
enabled=0
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-PaaS
[centos-openshift-origin311-debuginfo]
name=CentOS OpenShift Origin DebugInfo
baseurl=http://debuginfo.centos.org/centos/7/paas/x86_64/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-PaaS
[centos-openshift-origin311-source]
name=CentOS OpenShift Origin Source
baseurl=http://vault.centos.org/centos/7/paas/Source/openshift-origin311/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-PaaS
为提高安装速度，CentOS 7可配置自己的yum仓库，需要使用的有centos-openshift-origin310/311、base、updates、extras。
AWS Linux目前不支持OpenShift。
硬件需求
Masters
- 最小4 vCPU
- 最小16 GB RAM
- /var/最小40 GB硬盘空间
- /usr/local/bin/最小1 GB硬盘空间
- 临时目录最小1 GB硬盘空间
Nodes
- 1 vCPU
- 最小8 GB RAM
- /var/最小15 GB硬盘空间
- /usr/local/bin/最小1 GB硬盘空间
- 临时目录最小1 GB硬盘空间
安装类型
RPM-based Installations
System Container Installations
Delivery Mechanism
RPM packages using yum
System container images using docker
Service Management
systemd
docker and systemd units
Operating System
Red Hat Enterprise Linux (RHEL)
RHEL Atomic Host
RPM安装通过包管理器来安装和配置服务，system container安装使用系统容器镜像来安装服务， 服务运行在独立的容器内。
从OKD 3.10开始， 如果使用Red Hat Enterprise Linux (RHEL)操作系统，将使用RPM方法安装OKD组件。如果使用RHEL Atomic，将使用system container方法。不同安装类型提供相同的功能， 安装类型的选择依赖于操作系统、你想使用的服务管理和系统升级方法。
本文使用RPM安装方法。
Node ConfigMaps
Configmaps定义Node配置， OKD 3.10忽略openshift_node_labels值。默认创建了下面的ConfigMaps：
- node-config-master
- node-config-infra
- node-config-compute
- node-config-all-in-one
- node-config-master-infra
集群安装时选择node-config-master、node-config-infra、node-config-compute。
环境场景
- Master、Compute、Infra Node各一，etcd部署在master上
- Master、Compute、Infra Node各三，etcd部署在master上
为快速了解OpenShift安装，我们先使用第一种环境，成功后再安装第二种环境。Ansible一般使用单独的机器，两种情况分别需要创建4和10台EC2。
前期准备
更新系统
# yum update
Red Hat订阅
安装OpenShift需要Red Hat账号并订阅了RHEL，依次执行以下命令启用必须的repo:
# subscription-manager register
# subscription-manager list --available
# subscription-manager attach --pool=8a85f98b62dd96fc0162f04efb0e6350
# subscription-manager repos --list
# subscription-manager repos --enable rhel-7-server-ansible-2.6-debug-rpms
# subscription-manager repos --enable rhel-7-server-rpms
# subscription-manager repos --enable rhel-7-server-extras-rpms
检查SELinux
检查/etc/selinux/config，确保内容如下：
SELINUX=enforcing
SELINUXTYPE=targeted
配置DNS
为了使用更清晰的名字，需要创建额外的DNS服务器，为EC2配置合适的域名，如下：
master1.itrunner.org A 10.64.33.100
master2.itrunner.org A 10.64.33.103
node1.itrunner.org A 10.64.33.101
node2.itrunner.org A 10.64.33.102
EC2需要配置DNS服务器，创建dhclient.conf文件
# vi /etc/dhcp/dhclient.conf
添加如下内容：
supersede domain-name-servers 10.164.18.18;
配置完毕后需要重启才能生效，重启后/etc/resolv.conf内容如下：
# Generated by NetworkManager
search cn-north-1.compute.internal
nameserver 10.164.18.18
OKD使用了dnsmasq，安装成功后会自动配置所有Node，/etc/resolv.conf会被修改，nameserver变为本机IP。Pod将使用Node作为DNS，Node转发请求。
# nameserver updated by /etc/NetworkManager/dispatcher.d/99-origin-dns.sh
# Generated by NetworkManager
search cluster.local cn-north-1.compute.internal itrunner.org
nameserver 10.64.33.100
配置hostname
hostnamectl set-hostname --static master1.itrunner.org
编辑/etc/cloud/cloud.cfg文件，在底部添加以下内容：
preserve_hostname: true
安装基础包
所有Node需安装。
# yum install wget git net-tools bind-utils yum-utils iptables-services bridge-utils bash-completion kexec-tools sos psacct
安装Docker
所有Node需安装。
# yum install docker
# systemctl enable docker
# systemctl start docker
检查docker安装：
# docker info
注意：Docker必须为1.13.1，不能使用Docker官方版本。
安装Ansible
RHEL安装ansible：
# yum install ansible
CentOS使用EPEL Repository安装ansible：
# yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
# sed -i -e "s/^enabled=1/enabled=0/" /etc/yum.repos.d/epel.repo
# yum -y --enablerepo=epel install ansible pyOpenSSL
Ansible需要能访问其他所有机器才能完成安装，因此需要配置免密登录。可使用ssh-keygen重新生成密钥对，若使用ec2-user密钥，可使用PuTTYgen工具Export OpenSSH key，然后将私钥拷贝到ec2-user/.ssh目录下，私钥修改为默认名称id_rsa，然后授权：
$ cd .ssh/
$ chmod 600 *
配置成功后逐一测试连接：
ssh master1.itrunner.org
如使用密码或需要密码的密钥登录，请使用keychain。
配置Security Group
Security Group
Port
All OKD Hosts
tcp/22 from host running the installer/Ansible
etcd Security Group
tcp/2379 from masters, tcp/2380 from etcd hosts
Master Security Group
tcp/8443 from 0.0.0.0/0, tcp/53 from all OKD hosts, udp/53 from all OKD hosts, tcp/8053 from all OKD hosts, udp/8053 from all OKD hosts
Node Security Group
tcp/10250 from masters, udp/4789 from nodes
Infrastructure Nodes
tcp/443 from 0.0.0.0/0, tcp/80 from 0.0.0.0/0
配置ELB
第二种场景下需要配置ELB。
使用外部ELB时，Inventory文件不需定义lb，需要指定openshift_master_cluster_hostname、openshift_master_cluster_public_hostname、openshift_master_default_subdomain三个参数（请参见后面章节）。
openshift_master_cluster_hostname和openshift_master_cluster_public_hostname负责master的load balance，ELB定义时指向Master Node，其中openshift_master_cluster_hostname供内部使用，openshift_master_cluster_public_hostname供外部访问（Web Console），两者可以设置为同一域名，但openshift_master_cluster_hostname所使用的ELB必须配置为Passthrough。
为了安全，生产环境openshift_master_cluster_hostname和openshift_master_cluster_public_hostname应设置为两个不同域名。
openshift_master_default_subdomain定义OpenShift部署应用的域名，ELB指向Infra Node。
因此，共需创建三个ELB：
- openshift_master_cluster_hostname 必须创建网络负载均衡器，协议为TCP，默认端口8443，Target要使用IP方式。
- openshift_master_cluster_public_hostname ALB，协议HTTPS，默认端口8443；协议HTTP，端口80。
- openshift_master_default_subdomain ALB，协议HTTPS，默认端口443；协议HTTP，端口80和8080。
为了方便使用，openshift_master_cluster_public_hostname、openshift_master_default_subdomain一般配置为企业的域名，不直接使用AWS ELB的DNS名称。
注意：要使用ALB，Classic Load Balancer不支持wss协议，web console中不能查看log，不能使用terminal。
安装OpenShift
下载openshift-ansible
$ cd ~
$ git clone https://github.com/openshift/openshift-ansible
$ cd openshift-ansible
$ git checkout release-3.11
若要使用自定义的CentOS-OpenShift-Origin仓库，编辑文件~/openshift-ansible/roles/openshift_repos/templates/CentOS-OpenShift-Origin311.repo.j2，替换centos-openshift-origin311的baseurl，如下：
[centos-openshift-origin311]
name=CentOS OpenShift Origin
baseurl=http://10.188.12.119/centos/7/paas/x86_64/openshift-origin311/
创建初始用户
我们使用密码验证登录OpenShift，创建两个初始用户admin和developer：
# yum install -y httpd-tools
# htpasswd -c /home/ec2-user/htpasswd admin
# htpasswd /home/ec2-user/htpasswd developer
在下节的Inventory文件中，可以使用openshift_master_htpasswd_users、openshift_master_htpasswd_file两种方式配置初始用户，如下：
# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
# Defining htpasswd users
#openshift_master_htpasswd_users={'admin': '$apr1$qriH3ihA$LLxkL.EAH5Ntv3a4036nl/', 'developer': '$apr1$SkmCPrCP$Yn1JMxDwHzPOdYl9iPax80'}
# or
#openshift_master_htpasswd_file=/home/ec2-user/htpasswd
OpenShift安装成功后密码保存在master的/etc/origin/master/htpasswd文件内。
配置Inventory文件
Inventory文件定义了host和配置信息，默认文件为/etc/ansible/hosts。
场景一
master、compute、infra各一个结点，etcd部署在master上。
# Create an OSEv3 group that contains the masters, nodes, and etcd groups
[OSEv3:children]
masters
nodes
etcd
# Set variables common for all OSEv3 hosts
[OSEv3:vars]
# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=ec2-user
# If ansible_ssh_user is not root, ansible_become must be set to true
ansible_become=true
openshift_deployment_type=origin
openshift_disable_check=disk_availability,docker_storage,memory_availability,docker_image_availability
# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
# Defining htpasswd users
#openshift_master_htpasswd_users={'user1': '', 'user2': ''}
# or
#openshift_master_htpasswd_file=
# host group for masters
[masters]
master1.itrunner.org
# host group for etcd
[etcd]
master1.itrunner.org
# host group for nodes, includes region info
[nodes]
master1.itrunner.org openshift_node_group_name='node-config-master'
compute1.itrunner.org openshift_node_group_name='node-config-compute'
infra1.itrunner.org openshift_node_group_name='node-config-infra'
场景二
master、compute、infra各三个结点，在非生产环境下，load balance可以不使用外部ELB，使用HAProxy，etcd可以单独部署，也可以与master部署在一起。
- Multiple Masters Using Native HA with External Clustered etcd
# Create an OSEv3 group that contains the master, nodes, etcd, and lb groups.
# The lb group lets Ansible configure HAProxy as the load balancing solution.
# Comment lb out if your load balancer is pre-configured.
[OSEv3:children]
masters
nodes
etcd
lb
# Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=root
openshift_deployment_type=origin
# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
# Defining htpasswd users
#openshift_master_htpasswd_users={'user1': '', 'user2': ''}
# or
#openshift_master_htpasswd_file=
# Native high availbility cluster method with optional load balancer.
# If no lb group is defined installer assumes that a load balancer has
# been preconfigured. For installation the value of
# openshift_master_cluster_hostname must resolve to the load balancer
# or to one or all of the masters defined in the inventory if no load
# balancer is present.
openshift_master_cluster_method=native
openshift_master_cluster_hostname=openshift-internal.example.com
openshift_master_cluster_public_hostname=openshift-cluster.example.com
# apply updated node defaults
openshift_node_kubelet_args={'pods-per-core': ['10'], 'max-pods': ['250'], 'image-gc-high-threshold': ['90'], 'image-gc-low-threshold': ['80']}
# enable ntp on masters to ensure proper failover
openshift_clock_enabled=true
# host group for masters
[masters]
master[1:3].example.com
# host group for etcd
[etcd]
etcd[1:3].example.com
# Specify load balancer host
[lb]
lb.example.com
# host group for nodes, includes region info
[nodes]
master[1:3].example.com openshift_node_group_name='node-config-master'
node[1:3].example.com openshift_node_group_name='node-config-compute'
infra-node[1:3].example.com openshift_node_group_name='node-config-infra'
- Multiple Masters Using Native HA with Co-located Clustered etcd
# Create an OSEv3 group that contains the master, nodes, etcd, and lb groups.
# The lb group lets Ansible configure HAProxy as the load balancing solution.
# Comment lb out if your load balancer is pre-configured.
[OSEv3:children]
masters
nodes
etcd
lb
# Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=root
openshift_deployment_type=origin
# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
# Defining htpasswd users
#openshift_master_htpasswd_users={'user1': '', 'user2': ''}
# or
#openshift_master_htpasswd_file=
# Native high availability cluster method with optional load balancer.
# If no lb group is defined installer assumes that a load balancer has
# been preconfigured. For installation the value of
# openshift_master_cluster_hostname must resolve to the load balancer
# or to one or all of the masters defined in the inventory if no load
# balancer is present.
openshift_master_cluster_method=native
openshift_master_cluster_hostname=openshift-internal.example.com
openshift_master_cluster_public_hostname=openshift-cluster.example.com
# host group for masters
[masters]
master[1:3].example.com
# host group for etcd
[etcd]
master[1:3].example.com
# Specify load balancer host
[lb]
lb.example.com
# host group for nodes, includes region info
[nodes]
master[1:3].example.com openshift_node_group_name='node-config-master'
node[1:3].example.com openshift_node_group_name='node-config-compute'
infra-node[1:3].example.com openshift_node_group_name='node-config-infra'
使用外部ELB，需要指定openshift_master_cluster_hostname、openshift_master_cluster_public_hostname、openshift_master_default_subdomain，不需定义lb。
# Create an OSEv3 group that contains the master, nodes, etcd, and lb groups.
# The lb group lets Ansible configure HAProxy as the load balancing solution.
# Comment lb out if your load balancer is pre-configured.
[OSEv3:children]
masters
nodes
etcd
# Since we are providing a pre-configured LB VIP, no need for this group
#lb
# Set variables common for all OSEv3 hosts
[OSEv3:vars]
# SSH user, this user should allow ssh based auth without requiring a password
ansible_ssh_user=ec2-user
# If ansible_ssh_user is not root, ansible_become must be set to true
ansible_become=true
openshift_deployment_type=origin
openshift_disable_check=disk_availability,docker_storage,memory_availability,docker_image_availability
# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
# Defining htpasswd users
#openshift_master_htpasswd_users={'user1': '', 'user2': ''}
# or
#openshift_master_htpasswd_file=
# Native high availability cluster method with optional load balancer.
# If no lb group is defined installer assumes that a load balancer has
# been preconfigured. For installation the value of
# openshift_master_cluster_hostname must resolve to the load balancer
# or to one or all of the masters defined in the inventory if no load
# balancer is present.
openshift_master_cluster_method=native
openshift_master_cluster_hostname=openshift-master-internal-123456b57ac7be6c.elb.cn-north-1.amazonaws.com.cn
openshift_master_cluster_public_hostname=openshift.itrunner.org
openshift_master_default_subdomain=apps.itrunner.org
#openshift_master_api_port=443
#openshift_master_console_port=443
# host group for masters
[masters]
master[1:3].itrunner.org
# host group for etcd
[etcd]
master[1:3].itrunner.org
# Since we are providing a pre-configured LB VIP, no need for this group
#[lb]
#lb.itrunner.org
# host group for nodes, includes region info
[nodes]
master[1:3].itrunner.org openshift_node_group_name='node-config-master'
app[1:3].itrunner.org openshift_node_group_name='node-config-compute'
infra[1:3].itrunner.org openshift_node_group_name='node-config-infra'
安装OpenShift
一切准备就绪，使用ansible安装OpenShift非常简单，仅需运行prerequisites.yml和deploy_cluster.yml两个playbook。
$ ansible-playbook ~/openshift-ansible/playbooks/prerequisites.yml
$ ansible-playbook ~/openshift-ansible/playbooks/deploy_cluster.yml
如没有使用默认的inventory文件，可以使用-i指定文件位置：
$ ansible-playbook [-i /path/to/inventory] ~/openshift-ansible/playbooks/prerequisites.yml
$ ansible-playbook [-i /path/to/inventory] ~/openshift-ansible/playbooks/deploy_cluster.yml
以上两步均可重复运行。
deploy过程中如出现错误，可尝试清空节点.ansible文件夹的内容，修改或删除.kube/config、/etc/ansible/facts.d/openshift.fact文件，修正后先运行错误提示中的playbook，然后再运行deploy_cluster.yml。
deploy过程中如出现长时间等待的情况，可在节点运行journalctl -f查看日志查找原因。
安装成功后输出如下：
prerequisites.yml
PLAY RECAP *************************************************************************************************************************************************************
localhost : ok=11 changed=0 unreachable=0 failed=0
app1.itrunner.org : ok=59 changed=12 unreachable=0 failed=0
app2.itrunner.org : ok=59 changed=12 unreachable=0 failed=0
app3.itrunner.org : ok=59 changed=12 unreachable=0 failed=0
infra1.itrunner.org : ok=59 changed=12 unreachable=0 failed=0
infra2.itrunner.org : ok=59 changed=12 unreachable=0 failed=0
infra3.itrunner.org : ok=59 changed=12 unreachable=0 failed=0
master1.itrunner.org : ok=79 changed=12 unreachable=0 failed=0
master2.itrunner.org : ok=64 changed=12 unreachable=0 failed=0
master3.itrunner.org : ok=64 changed=12 unreachable=0 failed=0
INSTALLER STATUS *******************************************************************************************************************************************************
Initialization : Complete (0:01:07)
deploy_cluster.yml
PLAY RECAP *************************************************************************************************************************************************************
localhost : ok=11 changed=0 unreachable=0 failed=0
app1.itrunner.org : ok=114 changed=16 unreachable=0 failed=0
app2.itrunner.org : ok=114 changed=16 unreachable=0 failed=0
app3.itrunner.org : ok=114 changed=16 unreachable=0 failed=0
infra1.itrunner.org : ok=114 changed=16 unreachable=0 failed=0
infra2.itrunner.org : ok=114 changed=16 unreachable=0 failed=0
infra3.itrunner.org : ok=114 changed=16 unreachable=0 failed=0
master1.itrunner.org : ok=685 changed=162 unreachable=0 failed=0
master2.itrunner.org : ok=267 changed=45 unreachable=0 failed=0
master3.itrunner.org : ok=267 changed=45 unreachable=0 failed=0
INSTALLER STATUS *******************************************************************************************************************************************************
Initialization : Complete (0:01:06)
Health Check : Complete (0:00:30)
Node Bootstrap Preparation : Complete (0:03:23)
etcd Install : Complete (0:00:42)
Master Install : Complete (0:03:28)
Master Additional Install : Complete (0:00:34)
Node Join : Complete (0:00:47)
Hosted Install : Complete (0:00:43)
Cluster Monitoring Operator : Complete (0:00:12)
Web Console Install : Complete (0:00:40)
Console Install : Complete (0:00:35)
metrics-server Install : Complete (0:00:00)
Service Catalog Install : Complete (0:03:20)
验证安装
# oc get nodes
场景一，使用master hostname访问web console: https://master1.itrunner.org:8443/console
场景二，使用域名访问web console: https://openshift.itrunner.org:8443/console
用户权限
使用system:admin登录：
# oc login -u system:admin
用户授权：
# oc adm policy add-cluster-role-to-user cluster-admin admin
# oc adm policy add-role-to-user admin admin
CLI配置文件
oc login命令自动创建和管理CLI配置文件~/.kube/config。
卸载OpenShift
使用安装时的inventory文件
$ ansible-playbook ~/openshift-ansible/playbooks/adhoc/uninstall.yml
新建一个inventory文件，配置要卸载的node:
[OSEv3:children]
nodes
[OSEv3:vars]
ansible_ssh_user=ec2-user
openshift_deployment_type=origin
[nodes]
node3.example.com openshift_node_group_name='node-config-infra'
指定新的inventory文件，运行uninstall.yml playbook：
$ ansible-playbook -i /path/to/new/file ~/openshift-ansible/playbooks/adhoc/uninstall.yml
参考资料
OpenShift
OpenShift Github
OpenShift Documentation
OKD
OKD Latest Documentation
Ansible Documentation
External Load Balancer Integrations with OpenShift Enterprise 3
Red Hat OpenShift on AWS
Docker Documentation
Kubernetes Documentation
Kubernetes中文社区
Kubernetes-基于EFK进行统一的日志管理
SSL For Free
|
|
QQ群⑧：
窥视卡
雷达卡
发表于 2019-2-16 07:07:42
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡