OpenStack-FlatDHCP(multi_host)实施方案

常青树

服务器规划：
控制节点：eth0 : 10.10.128.11  ,  eth2: 10.10.64.11
计算节点：eth2 : 10.10.128.12  ,  eth4:10.10.64.12
计算节点：eth2 : 10.10.128.13  ,  eth4:10.10.64.13
网段规划:
公网：163.167.28.70
内网：10.10.128.0/24
私网：10.10.64.0/24
虚拟机网段：10.11.11.0/24
操作系统：
Linux：ubuntu-12.04.1-server-amd64
服务：

root@ubuntu-11:~# nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-scheduler   ubuntu-11                            nova             enabled    :-)   2012-11-20 03:54:48
nova-consoleauth ubuntu-11                            nova             enabled    :-)   2012-11-20 03:54:47
nova-compute     ubuntu-13                            nova             enabled    :-)   2012-11-20 03:54:50
nova-network     ubuntu-13                            nova             enabled    :-)   2012-11-20 03:54:43
nova-cert        ubuntu-11                            nova             enabled    :-)   2012-11-20 03:54:46
nova-compute     ubuntu-12                            nova             enabled    :-)   2012-11-20 03:54:43
nova-network     ubuntu-12                            nova             enabled    :-)   2012-11-20 03:54:50



一、安装OpenStack的控制节点
1、开启root权限

sudo passwd root
设置新密码

2、网卡设置

# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet static
address 10.10.128.11
netmask 255.255.128.0
network 10.10.128.0
broadcast 10.10.255.255
gateway 10.10.128.254
# dns-* options are implemented by the resolvconf package, if installed
dns-nameservers 10.10.128.254
auto eth2
iface eth2 inet static
address 10.10.64.11
netmask 255.255.255.0
network 10.10.64.0

重启网络：

sudo /etc/init.d/networking restart

3、Install Base OS & bridge-utils

sudo apt-get update  
sudo apt-get upgrade  
sudo apt-get install bridge-utils  

4、设置NTP Server
sudo apt-get install ntp
修改/etc/ntp.conf

server ntp.ubuntu.com  
server 127.127.1.0  
fudge 127.127.1.0 stratum 10  

重启NTP服务

sudo service ntp restart

5、Install mysql-server and python-mysqldb package

sudo apt-get install mysql-server python-mysqldb
密码为: mygreatsecret

修改/etc/mysql/my.cnf
bind-address = 0.0.0.0  
重启mysql

sudo restart mysql

Create MySQL databases to be used with nova, glance and keystone.
以下脚本粘贴到doMysql.sh,修改权限chmod +x doMsql.sh后执行，按提示输入主机名字。

#!/bin/bash
read -p "Enter HostName(For example: computer-node1):" HOSTNAME
echo "hostname is :$HOSTNAME"
sudo mysql -uroot -pmygreatsecret -e 'CREATE DATABASE nova;'  
sudo mysql -uroot -pmygreatsecret -e 'CREATE USER novadbadmin;'  
sudo mysql -uroot -pmygreatsecret -e "GRANT ALL PRIVILEGES ON nova.* TO 'novadbadmin'@'%';"  
sudo mysql -uroot -pmygreatsecret -e "SET PASSWORD FOR 'novadbadmin'@'%' = PASSWORD('novasecret');"  
sudo mysql -uroot -pmygreatsecret -e 'CREATE DATABASE glance;'  
sudo mysql -uroot -pmygreatsecret -e 'CREATE USER glancedbadmin;'  
sudo mysql -uroot -pmygreatsecret -e "GRANT ALL PRIVILEGES ON glance.* TO 'glancedbadmin'@'%';"  
sudo mysql -uroot -pmygreatsecret -e "SET PASSWORD FOR 'glancedbadmin'@'%' = PASSWORD('glancesecret');"  
sudo mysql -uroot -pmygreatsecret -e 'CREATE DATABASE keystone;'  
sudo mysql -uroot -pmygreatsecret -e 'CREATE USER keystonedbadmin;'  
sudo mysql -uroot -pmygreatsecret -e "GRANT ALL PRIVILEGES ON keystone.* TO 'keystonedbadmin'@'%';"  
sudo mysql -uroot -pmygreatsecret -e "SET PASSWORD FOR 'keystonedbadmin'@'%' = PASSWORD('keystonesecret');"  
sudo mysql -uroot -pmygreatsecret -e "GRANT ALL PRIVILEGES ON nova.* TO 'novadbadmin'@'localhost';"  
sudo mysql -uroot -pmygreatsecret -e "SET PASSWORD FOR 'novadbadmin'@'localhost' = PASSWORD('novasecret');"  
sudo mysql -uroot -pmygreatsecret -e "GRANT ALL PRIVILEGES ON glance.* TO 'glancedbadmin'@'localhost';"  
sudo mysql -uroot -pmygreatsecret -e "SET PASSWORD FOR 'glancedbadmin'@'localhost' = PASSWORD('glancesecret');"  
sudo mysql -uroot -pmygreatsecret -e "GRANT ALL PRIVILEGES ON keystone.* TO 'keystonedbadmin'@'localhost';"  
sudo mysql -uroot -pmygreatsecret -e "SET PASSWORD FOR 'keystonedbadmin'@'localhost' = PASSWORD('keystonesecret');"  
sudo mysql -uroot -pmygreatsecret -e "GRANT ALL PRIVILEGES ON nova.* TO 'novadbadmin'@'$HOSTNAME';"  
sudo mysql -uroot -pmygreatsecret -e "SET PASSWORD FOR 'novadbadmin'@'$HOSTNAME' = PASSWORD('novasecret');"  
sudo mysql -uroot -pmygreatsecret -e "GRANT ALL PRIVILEGES ON glance.* TO 'glancedbadmin'@'$HOSTNAME';"  
sudo mysql -uroot -pmygreatsecret -e "SET PASSWORD FOR 'glancedbadmin'@'$HOSTNAME' = PASSWORD('glancesecret');"  
sudo mysql -uroot -pmygreatsecret -e "GRANT ALL PRIVILEGES ON keystone.* TO 'keystonedbadmin'@'$HOSTNAME';"  
sudo mysql -uroot -pmygreatsecret -e "SET PASSWORD FOR 'keystonedbadmin'@'$HOSTNAME' = PASSWORD('keystonesecret');"  

6、Install Keystone
sudo apt-get install keystone python-keystone python-keystoneclient  
修改文件/etc/keystone/keystone.conf
admin_token = admin
修改/etc/keystone/keystone.conf

connection = sqlite:////var/lib/keystone/keystone.db  
改为  
connection = mysql://keystonedbadmin:keystonesecret@10.10.64.11/keystone

重启keystone
sudo service keystone restart  
同步数据库
sudo keystone-manage db_sync  
添加环境变量到/root/.bashrc

export SERVICE_ENDPOINT="http://localhost:35357/v2.0"  
export SERVICE_TOKEN=admin

使环境变量生效
source /root/.bashrc
Creating Tenants,Creating Users,Creating Roles,Listing Tenants, Users and Roles,Adding Roles to Users in Tenants,Creating Services,Creating Endpoints
把以下内容保存到create_keystone_data.sh，然后执行./ create_keystone_data.sh

#!/bin/bash
#easy to run Keystone
#Creating Tenants
#Creating Users
#Creating Roles
#Listing Tenants, Users and Roles
#Adding Roles to Users in Tenants
#Creating Services
#Creating Endpoints
#
function get_id () {
echo `$@ | awk '/ id / { print $4 }'`
}
# Tenants
echo "-----------------------------Tenants-----------------------------------------"
ADMIN_TENANT=$(get_id keystone tenant-create --name=admin)
SERVICE_TENANT=$(get_id keystone tenant-create --name=service)
echo "ADMIN_TENANT=$ADMIN_TENANT"
echo "SERVICE_TENANT=$SERVICE_TENANT"
echo "-----------------------------Create Tenants Ending-----------------------------"
read -p "Enter your host(For example: xxx@.qq.com):" MAIL
echo "send mail to:$MAIL"
# Users
echo "-----------------------------Users-----------------------------------------"
ADMIN_USER=$(get_id keystone user-create --name=admin --pass=admin --email=$MAIL)
NOVA_USER=$(get_id keystone user-create --name=nova --pass=nova   --email=$MAIL)
GLANCE_USER=$(get_id keystone user-create --name=glance --pass=glance   --email=$MAIL)
SWIFT_USER=$(get_id keystone user-create --name=swift --pass=swift   --email=$MAIL)
echo "ADMIN_USER=$ADMIN_USER"
echo "NOVA_USER=$NOVA_USER"
echo "GLANCE_USER=$GLANCE_USER"
echo "SWIFT_USER=$SWIFT_USER"
echo "-----------------------------Create User Ending-----------------------------"
# Roles
echo "-----------------------------Roles-----------------------------------------"
ADMIN_ROLE=$(get_id keystone role-create --name=admin)
MEMBER_ROLE=$(get_id keystone role-create --name=Member)
echo "ADMIN_ROLE=$ADMIN_ROLE"
echo "MEMBER_ROLE=$MEMBER_ROLE"
echo "-----------------------------Create Roles Ending-----------------------------"
# Add Roles to Users in Tenants
echo "-----------------------------Add Roles to Users in Tenants-----------------------------------------"
echo "keystone user-role-add --user=$ADMIN_USER --role=$ADMIN_ROLE --tenant_id=$ADMIN_TENANT"
keystone user-role-add --user=$ADMIN_USER --role=$ADMIN_ROLE --tenant_id=$ADMIN_TENANT
echo "-----------------------------Add Roles to Users in Tenants Ending-----------------------------"

# TODO(termie): these two might be dubious
echo "-----------------------------TODO(termie): these two might be dubious-----------------------------------------"
echo "keystone user-role-add --user=$NOVA_USER --role=$ADMIN_ROLE --tenant_id=$SERVICE_TENANT"
echo "keystone user-role-add --user=$GLANCE_USER --role=$ADMIN_ROLE --tenant_id=$SERVICE_TENANT"
echo "keystone user-role-add --user=$SWIFT_USER --role=$ADMIN_ROLE --tenant_id=$SERVICE_TENANT"
keystone user-role-add --user=$NOVA_USER --role=$ADMIN_ROLE --tenant_id=$SERVICE_TENANT
keystone user-role-add --user=$GLANCE_USER --role=$ADMIN_ROLE --tenant_id=$SERVICE_TENANT
keystone user-role-add --user=$SWIFT_USER --role=$ADMIN_ROLE --tenant_id=$SERVICE_TENANT
echo "-----------------------------these two might be dubious Ending-----------------------------"
# The Member role is used by Horizon and Swift so we need to keep it:
echo "------------------------The Member role is used by Horizon and Swift so we need to keep it---------------------------------"
keystone user-role-add --user=$ADMIN_USER --role=$MEMBER_ROLE --tenant_id=$ADMIN_TENANT
echo "keystone user-role-add --user=$ADMIN_USER --role=$MEMBER_ROLE --tenant_id=$ADMIN_TENANT"
echo "-----------------------------The Member role is used by Horizon and Swift so we need to keep it Ending-----------------------------"
read -p "Enter your host(For example: 10.10.128.11):" HOSTADDR
echo "Your host is:$HOSTADDR"

#Creating Services
echo "------------------------Creating Services---------------------------------"
COMPUTESERVICE=$(get_id keystone service-create --name=nova --type=compute --description='OpenStack_Compute_Service')
VOLUMESERVICE=$(get_id keystone service-create --name=volume --type=volume --description='OpenStack_Volume_Service')
IMAGESERVICE=$(get_id keystone service-create --name=glance --type=image --description='OpenStack_Image_Service')
STORAGESERVICE=$(get_id keystone service-create --name=swift --type=object-store --description='OpenStack_Storage_Service')
IDENTITYSERVICE=$(get_id keystone service-create --name=keystone --type=identity --description='OpenStack_Identity_Service')
EC2SERVICE=$(get_id keystone service-create --name=ec2 --type=ec2 --description='EC2_Service')
echo "COMPUTESERVICE=$COMPUTESERVICE"
echo "VOLUMESERVICE=$VOLUMESERVICE"
echo "IMAGESERVICE=$IMAGESERVICE"
echo "STORAGESERVICE=$STORAGESERVICE"
echo "IDENTITYSERVICE=$IDENTITYSERVICE"
echo "EC2SERVICE=$EC2SERVICE"
echo "-----------------------------Creating Services Ending-----------------------------"
echo "-----------------------------Creating Endpoints-----------------------------"
keystone endpoint-create --region myregion --service_id=$COMPUTESERVICE --publicurl "http://$HOSTADDR:8774/v2/%(tenant_id)s" --adminurl "http://$HOSTADDR:8774/v2/%(tenant_id)s" --internalurl "http://$HOSTADDR:8774/v2/%(tenant_id)s"
keystone endpoint-create --region myregion --service_id=$VOLUMESERVICE --publicurl "http://$HOSTADDR:8776/v1/%(tenant_id)s" --adminurl "http://$HOSTADDR:8776/v1/%(tenant_id)s" --internalurl "http://$HOSTADDR:8776/v1/%(tenant_id)s"
keystone endpoint-create --region myregion --service_id=$IMAGESERVICE --publicurl "http://$HOSTADDR:9292/v1" --adminurl "http://$HOSTADDR:9292/v1" --internalurl "http://$HOSTADDR:9292/v1"
keystone endpoint-create --region myregion --service_id=$STORAGESERVICE --publicurl "http://$HOSTADDR:8080/v1/AUTH_%(tenant_id)s" --adminurl "http://$HOSTADDR:8080/v1" --internalurl "http://$HOSTADDR:8080/v1/AUTH_%(tenant_id)s"
keystone endpoint-create --region myregion --service_id=$IDENTITYSERVICE --publicurl http://$HOSTADDR:5000/v2.0 --adminurl http://$HOSTADDR:35357/v2.0 --internalurl http://$HOSTADDR:5000/v2.0
keystone endpoint-create --region myregion --service_id=$EC2SERVICE --publicurl http://$HOSTADDR:8773/services/Cloud --adminurl http://$HOSTADDR:8773/services/Admin --internalurl http://$HOSTADDR:8773/services/Cloud
echo "-----------------------------Creating Endpoints Ending-----------------------------"

执行脚本

chmod +x create_keystone_data.sh
./ create_keystone_data.sh

7、Install glance
sudo apt-get install glance glance-api glance-client glance-common glance-registry python-glance
修改/etc/glance/glance-api-paste.ini 和 /etc/glance/glance-registry-paste.ini

admin_tenant_name = %SERVICE_TENANT_NAME%  
admin_user = %SERVICE_USER%  
admin_password = %SERVICE_PASSWORD%  
改为  
admin_tenant_name = service  
admin_user = glance  
admin_password = glance

修改/etc/glance/glance-registry.conf

sql_connection = mysql://glancedbadmin:glancesecret@10.10.64.11/glance
#末尾追加  
[paste_deploy]  
flavor = keystone

末尾追加以下内容到/etc/glance/glance-api.conf

[paste_deploy]  
flavor = keystone

修改/etc/glance/glance-api.conf
rabbit_host = 10.10.64.11
Create glance schema in the MySQL database.:

sudo glance-manage version_control 0  
sudo glance-manage db_sync  

重启glance-api 和glance-registry

sudo restart glance-api  
sudo restart glance-registry

添加环境变量到/root/.bashrc

export OS_TENANT_NAME=admin  
export OS_USERNAME=admin  
export OS_PASSWORD=admin  
export OS_AUTH_URL="http://localhost:5000/v2.0/"

使环境变量生效
source /root/.bashrc  
打印镜像列表
glance index

8、Install nova 除了nova-compute和nova-network

sudo apt-get install nova-api nova-cert nova-doc nova-objectstore nova-scheduler rabbitmq-server novnc nova-consoleauth

修改/etc/nova/nova.conf

#common
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--s3_host=10.10.64.11
--ec2_host=10.10.64.11
--rabbit_host=10.10.64.11
--glance_api_servers=10.10.64.11:9292
--image_service=nova.image.glance.GlanceImageService
--sql_connection=mysql://novadbadmin:novasecret@10.10.64.11/nova
--ec2_path=http://10.10.128.11:8773/services/Cloud
--api_paste_config=/etc/nova/api-paste.ini
--start_guests_on_host_boot=true
--resume_guests_state_on_host_boot=true
--network_manager=nova.network.manager.FlatDHCPManager
--connection_type=libvirt
--root_helper=sudo nova-rootwrap
--verbose=false
--use-syslog=false
--enabled_apis=ec2,osapi_compute,osapi_volume,metadata
--metadata_host=10.10.64.11
#hypervisors
--libvirt_type=kvm
--libvirt_use_virtio_for_bridges=true
#--libvirt_vif_driver =nova.virt.libvirt.vif.LibvirtBridgeDriver
--use_usb_tablet=true
--libvirt_ovs_bridge=br-int
#keystone
--auth_strategy=keystone
--keystone_ec2_url=http://10.10.128.11:5000/v2.0/ec2tokens
#Scheduler
--scheduler_driver=nova.scheduler.multi.MultiScheduler
#network
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--multi_host=true
--public_interface=eth0
--flat_interface=eth2
--flat_network_bridge=br100
--fixed_range=10.11.11.0/24
--flat_injected=false
--force_dhcp_release=true
--network_size=256
--allow_same_net_traffic=true
#vnc
--vnc_enabled=true
--novncproxy_base_url= http://163.167.28.70:6080/vnc_auto.html
#--vncserver_proxyclient_address=
#--vncserver_listen=
--vnc_keymap=en-us
#log
#--logging_exception_prefix="%(asctime)s TRACE %(name)s %(instance)s"
#--publish_errors=true
#compute
#--compute_driver=nova.virt.connection.get_connection
#--instances_path= /mnt/nova/instances/

修改权限

sudo chown -R nova:nova /etc/nova  
sudo chmod 644 /etc/nova/nova.conf  

修改/etc/nova/api-paste.ini

admin_tenant_name = %SERVICE_TENANT_NAME%  
admin_user = %SERVICE_USER%  
admin_password = %SERVICE_PASSWORD%  
改为  
admin_tenant_name = service  
admin_user = nova  
admin_password = nova

数据同步

sudo nova-manage db sync

重启nova服务

sudo restart nova-api; sudo restart nova-objectstore; sudo restart nova-scheduler; sudo restart nova-consoleauth;

查看服务

root@ubuntu-50:~# sudo nova-manage service list
Binary           Host                                 Zone             Status     State Updated_At
nova-scheduler   ubuntu-50                            nova             enabled    :-)   2012-10-28 04:46:52
nova-consoleauth ubuntu-50                            nova             enabled    :-)   2012-10-28 04:46:52

9、Install OpenStack Dashboard

sudo apt-get install openstack-dashboard

遇到
Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
添加内容到/etc/apache2/httpd.conf
ServerName 127.0.1.1
重启apache
sudo service apache2 restart
打开浏览器，输入http://10.10.128.11,输入admin@admin登录。
二、安装计算节点
1、安装网桥

sudo apt-get install qemu-kvm libvirt-bin virt-manager bridge-utils

2、修改/etc/network/interfaces

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth2
iface eth2 inet static
address 10.10.128.12
netmask 255.255.128.0
network 10.10.128.0
broadcast 10.10.255.255
gateway 10.10.128.254
auto eth4
iface eth4 inet manual
auto br12
iface br12 inet static
address 10.10.64.12
netmask 255.255.255.0
#network 10.10.64.0
#broadcast 10.10.64.255
bridge_ports eth4

重启网络：
sudo /etc/init.d/networking restart

3、安装NTP Client
sudo apt-get install ntp
修改/etc/ntp.conf
server 10.10.64.11
重启NTP
sudo service ntp restart
4、设置ipv4转发
sysctl -w net.ipv4.ip_forward=1

5、安装nova-network,nova-compute
sudo apt-get install nova-compute nova-compute-kvm nova-network
修改/etc/nova/nova.conf ，与控制节点一致

#common
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--s3_host=10.10.64.11
--ec2_host=10.10.64.11
--rabbit_host=10.10.64.11
--glance_api_servers=10.10.64.11:9292
--image_service=nova.image.glance.GlanceImageService
--sql_connection=mysql://novadbadmin:novasecret@10.10.64.11/nova
--ec2_path=http://10.10.128.11:8773/services/Cloud
--api_paste_config=/etc/nova/api-paste.ini
--start_guests_on_host_boot=true
--resume_guests_state_on_host_boot=true
--network_manager=nova.network.manager.FlatDHCPManager
--connection_type=libvirt
--root_helper=sudo nova-rootwrap
--verbose=false
--use-syslog=false
--enabled_apis=ec2,osapi_compute,osapi_volume,metadata
--metadata_host=10.10.64.11
#hypervisors
--libvirt_type=kvm
--libvirt_use_virtio_for_bridges=true
#--libvirt_vif_driver =nova.virt.libvirt.vif.LibvirtBridgeDriver
--use_usb_tablet=true
--libvirt_ovs_bridge=br-int
#keystone
--auth_strategy=keystone
--keystone_ec2_url=http://10.10.128.11:5000/v2.0/ec2tokens
#Scheduler
--scheduler_driver=nova.scheduler.multi.MultiScheduler
#network
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--multi_host=true
--public_interface=eth2
--flat_interface=eth4
--flat_network_bridge=br100
--fixed_range=10.11.11.0/24
--flat_injected=false
--force_dhcp_release=true
--network_size=256
--allow_same_net_traffic=true
#vnc
--vnc_enabled=true
--novncproxy_base_url= http://163.167.28.70:6080/vnc_auto.html
--vncserver_proxyclient_address=10.10.64.12
--vncserver_listen=10.10.64.12
--vnc_keymap=en-us
#log
#--logging_exception_prefix="%(asctime)s TRACE %(name)s %(instance)s"
#--publish_errors=true
#compute
#--compute_driver=nova.virt.connection.get_connection
--instances_path=/mnt/nova/instances/

修改权限

sudo chown -R nova:nova /mnt/nova/
sudo chown -R nova:nova /mnt/nova/instances/

重启服务
sudo restart nova-compute; sudo restart nova-network; sudo restart libvirt-bin;

三、初始化系统
1、上传镜像

glance add name="qcow2_win08_35G" is_public=true container_format=ovf disk_format=qcow2 < qcow2_win08_35G.img
glance add name="qcow2_ubuntu64" is_public=true container_format=ovf disk_format=qcow2 < qcow2_ubuntu64.img
## nova image-list

2、创建网络

nova-manage network create --label=br100 --fixed_range_v4=10.11.11.0/24 --num_network=1  --multi_host=T --bridge=br100
#ip addr
#brctl show

3、安全组(防火墙规则)

nova secgroup-add-rule default tcp 1 65535 0.0.0.0/0
nova secgroup-add-rule default udp 1 65535 0.0.0.0/0
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0

Create the win-server security group to allow ping (icmp), ssh (tcp/22), and http (tcp/80) from everywhere (0.0.0.0/0):

nova secgroup-create win-server "win server running on default port"
nova secgroup-add-rule win-server icmp -1 -1 0.0.0.0/0
nova secgroup-add-rule win-server tcp 22 22 0.0.0.0/0
nova secgroup-add-rule win-server tcp 3389 3389 0.0.0.0/0
nova secgroup-add-rule win-server tcp 80 80 0.0.0.0/0
nova secgroup-add-rule win-server tcp 20 20 0.0.0.0/0
nova secgroup-add-rule win-server tcp 21 21 0.0.0.0/0

4、创建浮动IP
nova-manage floating create --pool=pool1 --ip_range=10.10.128.192/26
5、使用镜像创建实例(虚拟机)



6、绑定浮动IP



7、登录到虚拟机

8、VNC控制台