OpenStack使用Bosh部署CloudFoundry(四)—使用Micro Bosh部署Bosh
目录(?)[-][*]上传Bosh Stemcell到Micro Bosh中
[*]下载Bosh Release代码并打包
[*]上传Bosh Release包到Micro Bosh中
[*]确认已经上传的stemcell和release
[*]创建部署Bosh的Manifest文件
[*]部署Bosh
[*]验证部署是否成功
版本说明：
CloudFoundry：V2版本-v149+（本文后续章节中的Manifest适用于149+版本的CF Release）
OpenStack：Grizzly版本+nova-network
CloudFoundry部署交流QQ群：176302388
本文介绍如何使用Micro Bosh来部署Bosh，部署需要创建8个虚拟机来部署8个Bosh组件，所以，至少得准备好8个Fixed IP，后边使用Bosh的时候需要访问Director组件及配置DNS，所以准备至少2个Floating IP，本文中为（10.68.19.131和10.68.19.132），Bosh组件的详细介绍可以参看下边连接。
BOSH组件介绍：http://docs.cloudfoundry.com/docs/running/bosh/components/
下面是部署步骤：
1、上传Bosh Stemcell到Micro Bosh中
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]root@bosh-cli:~/bosh-workspace/stemcells# bosh upload stemcell bosh-stemcell-latest-openstack-kvm-ubuntu.tgz
[*]
[*]Verifying stemcell...
[*]File exists and readable OK
[*]Using cached manifest...
[*]Stemcell properties OK
[*]
[*]Stemcell info
[*]-------------
[*]Name: bosh-openstack-kvm-ubuntu
[*]Version: 1478
[*]
[*]Checking if stemcell already exists...
[*]No
[*]
[*]Uploading stemcell...
[*]
[*]latest-bosh-s: 100% |ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo| 251.3MB11.3MB/s Time: 00:00:22
[*]
[*]Director task 1
[*]
[*]Update stemcell
[*]extracting stemcell archive (00:00:06)
[*]verifying stemcell manifest (00:00:00)
[*]checking if this stemcell already exists (00:00:00)
[*]uploading stemcell bosh-stemcell/776 to the cloud (00:01:53)
[*]save stemcell bosh-stemcell/776 (f207bfcd-f0cb-4ea2-b753-022e6234ebfa) (00:00:00)
[*]Done 5/5 00:01:59
[*]
[*]Task 1 done
[*]Started 2013-12-16 09:51:59 UTC
[*]Finished 2013-12-16 09:53:58 UTC
[*]Duration 00:01:59
[*]
[*]Stemcell uploaded and created
2、下载Bosh Release代码并打包
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]root@bosh-cli:~# cd ~/bosh-workspace
[*]root@bosh-cli:~# git clone git://github.com/cloudfoundry/bosh.git
耗费时间较长，请耐心等待。
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]root@bosh-cli:~# cd ~/bosh-workspace/bosh
[*]root@bosh-cli:~# apt-get install libxslt-dev libxml2-dev libmysql-ruby libmysqlclient-dev libpq-dev
[*]root@bosh-cli:~# bundle install --local
[*]root@bosh-cli:~# bundle exec rake release:create_dev_release
可以先修改~/bosh-workspace/bosh/Gemfile的source为淘宝源：
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]source 'http://ruby.taobao.org'
打包结束得到以下输出：
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]bosh create release --force
[*]Syncing blobs...
[*]
[*]Building DEV release
[*]---------------------------------
[*]
[*]....
[*]
[*]Release version: 13.1-dev
[*]Release manifest: /root/bosh-workspace/bosh/release/dev_releases/bosh-13.1-dev.yml
3、上传Bosh Release包到Micro Bosh中
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]root@bosh-cli:~/bosh-workspace# bosh upload release ~/bosh-workspace/bosh/release/dev_releases/bosh-13.1-dev.yml
成功后得到以下输出：
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]....
[*]
[*]Creating new jobs
[*]blobstore/3.1-dev (00:00:00)
[*]powerdns/3.1-dev (00:00:00)
[*]redis/3 (00:00:00)
[*]nats/5 (00:00:00)
[*]postgres/4.1-dev (00:00:00)
[*]director/11.2-dev (00:00:00)
[*]health_monitor/5.1-dev (00:00:00)
[*]registry/0.1-dev (00:00:00)
[*]Done 8/8 00:00:00
[*]
[*]Release has been created
[*]bosh/13.1-dev (00:00:00)
[*]Done 1/1 00:00:00
[*]
[*]Task 2 done
[*]Started 2013-12-16 10:06:26 UTC
[*]Finished 2013-12-16 10:06:46 UTC
[*]Duration 00:00:20
[*]
[*]Release uploaded
4、确认已经上传的stemcell和release
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]root@bosh-cli:~# bosh stemcells
[*]
[*]+---------------------------+---------+--------------------------------------+
[*]| Name | Version | CID |
[*]+---------------------------+---------+--------------------------------------+
[*]| bosh-openstack-kvm-ubuntu | 1478 | e64c6da5-f2ab-48f2-b03d-79c29d1acc43 |
[*]+---------------------------+---------+--------------------------------------+
[*]
[*]Stemcells total: 1
[*]root@bosh-cli:~# bosh releases
[*]
[*]+------+-----------+-------------+
[*]| Name | Versions| Commit Hash |
[*]+------+-----------+-------------+
[*]| bosh | 13.1-dev* | a8e43f41+ |
[*]+------+-----------+-------------+
[*](*) Currently deployed
[*](+) Uncommitted changes
[*]
[*]Releases total: 1
5、创建部署Bosh的Manifest文件
创建目录
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]root@bosh-cli:~# mkdir -p ~/bosh-workspace/deployments/bosh-openstack
[*]root@bosh-cli:~# cd ~/bosh-workspace/deployments/bosh-openstack
复制模板文件
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]root@bosh-cli:~# cp ~/bosh-workspace/bosh/release/examples/bosh-openstack-dynamic.yml bosh-openstack.yml
如果OpenStack中采用的是quantum网络组件，则使用下边的模板文件
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]root@bosh-cli:~# cp ~/bosh-workspace/bosh/release/examples/bosh-openstack-manual.yml bosh-openstack.yml
本次安装中使用nova-network网络组件，所以采用bosh-openstack-dynamic.yml，修改其中标记有# CHANGE 的配置项，下边是本环境中的配置，大家可参考：
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]---
[*]name: bosh-openstack
[*]director_uuid: 5d8c682c-290a-4194-bfc6-60248567ceab # CHANGE
[*]
[*]release:
[*]name: bosh
[*]version: latest
[*]
[*]compilation:
[*]workers: 3
[*]network: default
[*]reuse_compilation_vms: true
[*]cloud_properties:
[*] instance_type: v1.compile # CHANGE
[*]
[*]update:
[*]canaries: 1
[*]canary_watch_time: 3000-120000
[*]update_watch_time: 3000-120000
[*]max_in_flight: 4
[*]max_errors: 1
[*]
[*]networks:
[*]- name: floating
[*] type: vip
[*] cloud_properties: {}
[*]- name: default
[*] type: dynamic
[*] cloud_properties: {}
[*]
[*]resource_pools:
[*]- name: common
[*] network: default
[*] size: 8
[*] stemcell:
[*] name: bosh-openstack-kvm-ubuntu
[*] version: latest
[*] cloud_properties:
[*] instance_type: v1.micro # CHANGE
[*]
[*]jobs:
[*]- name: nats
[*] template: nats
[*] instances: 1
[*] resource_pool: common
[*] networks:
[*] - name: default
[*] default:
[*]
[*]- name: redis
[*] template: redis
[*] instances: 1
[*] resource_pool: common
[*] networks:
[*] - name: default
[*] default:
[*]
[*]- name: postgres
[*] template: postgres
[*] instances: 1
[*] resource_pool: common
[*] persistent_disk: 16384
[*] networks:
[*] - name: default
[*] default:
[*]
[*]- name: powerdns
[*] template: powerdns
[*] instances: 1
[*] resource_pool: common
[*] networks:
[*] - name: default
[*] default:
[*] - name: floating
[*] static_ips:
[*] - 10.68.19.131 # CHANGE
[*]
[*]- name: blobstore
[*] template: blobstore
[*] instances: 1
[*] resource_pool: common
[*] networks:
[*] - name: default
[*] default:
[*]
[*]- name: director
[*] template: director
[*] instances: 1
[*] resource_pool: common
[*] persistent_disk: 16384
[*] networks:
[*] - name: default
[*] default:
[*] - name: floating
[*] static_ips:
[*] - 10.68.19.132 # CHANGE
[*]
[*]- name: registry
[*] template: registry
[*] instances: 1
[*] resource_pool: common
[*] networks:
[*] - name: default
[*] default:
[*]
[*]- name: health_monitor
[*] template: health_monitor
[*] instances: 1
[*] resource_pool: common
[*] networks:
[*] - name: default
[*] default:
[*]
[*]properties:
[*]nats:
[*] address: 0.nats.default.bosh-openstack.microbosh
[*] user: nats
[*] password: nats
[*]
[*]redis:
[*] address: 0.redis.default.bosh-openstack.microbosh
[*] password: redis
[*]
[*]postgres: &bosh_db
[*] host: 0.postgres.default.bosh-openstack.microbosh
[*] user: postgres
[*] password: postgres
[*] database: bosh
[*]
[*]dns:
[*] address: 10.68.19.131 # CHANGE
[*] db: *bosh_db
[*] recursor: 10.68.19.130 # CHANGE
[*]
[*]blobstore:
[*] address: 0.blobstore.default.bosh-openstack.microbosh
[*] agent:
[*] user: agent
[*] password: agent
[*] director:
[*] user: director
[*] password: director
[*]
[*]director:
[*] name: bosh
[*] address: 0.director.default.bosh-openstack.microbosh
[*] db: *bosh_db
[*]
[*]registry:
[*] address: 0.registry.default.bosh-openstack.microbosh
[*] db: *bosh_db
[*] http:
[*] user: registry
[*] password: registry
[*]
[*]hm:
[*] http:
[*] user: hm
[*] password: hm
[*] director_account:
[*] user: admin
[*] password: admin
[*] resurrector_enabled: true
[*]
[*]ntp:
[*] - 0.north-america.pool.ntp.org
[*] - 1.north-america.pool.ntp.org
[*]
[*]openstack:
[*] auth_url: http://10.68.19.61:5000/v2.0 # CHANGE
[*] username: VF # CHANGE
[*] api_key: 1 # CHANGE
[*] tenant: VF_CF # CHANGE
[*] region: myregion # CHANGE
[*] default_security_groups: ["default"] # CHANGE
[*] default_key_name: vkey # CHANGE
注意：director_uuid是Micro Bosh的Director的UUID，可以使用bosh status查看并复制到Manifest文件中。
6、部署Bosh
设置Bosh的部署文件
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]root@bosh-cli:~/bosh-workspace/deployments# bosh deployment ~/bosh-workspace/deployments/bosh-openstack/bosh-openstack.yml
[*]Deployment set to `/root/bosh-workspace/deployments/bosh-openstack/bosh-openstack.yml'
执行部署命令
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]root@bosh-cli:~/bosh-workspace/deployments# bosh deploy
部署时可能会遇到错误RateLimit报错，报错内容如下：
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]E, ERROR -- : OpenStack API Request Entity Too Large error. Check task debug log for details.
[*]/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/helpers.rb:20:in `cloud_error'
[*]/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/helpers.rb:39:in `rescue in with_openstack'
[*]/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/helpers.rb:25:in `with_openstack'
[*]/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/cloud.rb:512:in `block in set_vm_metadata'
[*]/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_formatter.rb:46:in `with_thread_name'
[*]/var/vcap/packages/director/gem_home/gems/bosh_openstack_cpi-1.5.0.pre.1478/lib/cloud/openstack/cloud.rb:511:in `set_vm_metadata'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/metadata_helper.rb:14:in `update_vm_metadata'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:225:in `block (2 levels) in compile_package'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:290:in `prepare_vm'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:224:in `block in compile_package'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/lock_helper.rb:89:in `block in with_compile_lock'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/lock.rb:58:in `lock'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/lock_helper.rb:88:in `with_compile_lock'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:217:in `compile_package'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:205:in `block (2 levels) in process_task'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/event_log.rb:58:in `track'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:203:in `block in process_task'
[*]/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_formatter.rb:46:in `with_thread_name'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:199:in `process_task'
[*]/var/vcap/packages/director/gem_home/gems/director-1.5.0.pre.1478/lib/director/package_compiler.rb:174:in `block (4 levels) in compile_packages'
[*]/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_pool.rb:83:in `call'
[*]/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_pool.rb:83:in `block (2 levels) in create_thread'
[*]/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_pool.rb:67:in `loop'
[*]/var/vcap/packages/director/gem_home/gems/bosh_common-1.5.0.pre.1478/lib/common/thread_pool.rb:67:in `block in create_thread'
解决办法：请查看本人博客：http://blog.iyunv.com/tiger435/article/details/8808623#t5，部署成功反馈：
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]root@bosh-cli:~/bosh-workspace/deployments# bosh deploy
[*]Getting deployment properties from director...
[*]Unable to get properties list from director, trying without it...
[*]Compiling deployment manifest...
[*]Cannot get current deployment information from director, possibly a new deployment
[*]Please review all changes carefully
[*]Deploying `bosh-openstack.yml' to `microbosh-openstack' (type 'yes' to continue): yes
[*]
[*]Director task 89
[*]
[*]Preparing deployment
[*]binding deployment (00:00:00)
[*]binding releases (00:00:00)
[*]
[*]....
[*]
[*]Updating job nats
[*]nats/0 (canary) (00:00:29)
[*]Done 1/1 00:00:29
[*]
[*]Updating job redis
[*]redis/0 (canary) (00:00:24)
[*]Done 1/1 00:00:24
[*]
[*]Updating job postgres
[*]postgres/0 (canary) (00:00:44)
[*]Done 1/1 00:00:44
[*]
[*]Updating job powerdns
[*]powerdns/0 (canary) (00:00:25)
[*]Done 1/1 00:00:25
[*]
[*]Updating job blobstore
[*]blobstore/0 (canary) (00:00:25)
[*]Done 1/1 00:00:25
[*]
[*]Updating job director
[*]director/0 (canary) (00:00:58)
[*]Done 1/1 00:00:58
[*]
[*]Updating job registry
[*]registry/0 (canary) (00:00:33)
[*]Done 1/1 00:00:33
[*]
[*]Updating job health_monitor
[*]health_monitor/0 (canary) (00:00:28)
[*]Done 1/1 00:00:28
[*]
[*]Task 3 done
[*]Started 2013-12-17 02:41:15 UTC
[*]Finished 2013-12-17 02:57:08 UTC
[*]Duration 00:15:53
[*]
[*]Deployed `bosh-openstack.yml' to `microbosh-openstack'
8、验证部署是否成功
viewplaincopyhttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/CODE_ico.pnghttp://onexin.iyunv.com/source/plugin/onexin_bigdata/https://code.iyunv.com/assets/ico_fork.svg
[*]root@bosh-cli:~/bosh-workspace/deployments/bosh-openstack# bosh vms
[*]Deployment `bosh-openstack'
[*]
[*]Director task 90
[*]
[*]Task 90 done
[*]
[*]+------------------+---------+---------------+--------------------------+
[*]| Job/index | State | Resource Pool | IPs |
[*]+------------------+---------+---------------+--------------------------+
[*]| blobstore/0 | running | common | 50.50.0.11 |
[*]| director/0 | running | common | 50.50.0.10, 10.68.19.132 |
[*]| health_monitor/0 | running | common | 50.50.0.13 |
[*]| nats/0 | running | common | 50.50.0.5 |
[*]| postgres/0 | running | common | 50.50.0.7 |
[*]| powerdns/0 | running | common | 50.50.0.9, 10.68.19.131|
[*]| redis/0 | running | common | 50.50.0.6 |
[*]| registry/0 | running | common | 50.50.0.12 |
[*]+------------------+---------+---------------+--------------------------+
[*]
[*]VMs total: 8
如上，所有Job的状态都是running，则部署成功。
页:
[1]