Puppet整合Foreman(五)：整合Mcollective

爱运维网

一、功能说明

•   Marionette Collective（MCollective）是一个与Puppet关系密切的服务运行框架。
  
•   MC依赖Ruby1.9.3或1.8.7，通过Stomp协议通信，因此依赖rubygem 1.2.2+
  
•   在puppet客户端上安装Mcollective服务端
  
•   在puppet服务端上安装Mcollective客户端
  
•   在puppet服务端或单独安装MQ中间件
  
  

参考官方文档：https://docs.puppetlabs.com/mcollective/deploy/standard.html

二、安装ActiveMQ
1.  说明
    RabbitMQ是一个实现了高级消息排队协议（AMQP）的消息队列服务。 RabbitMQ基于OTP(Open Telecom Platform，开发电信平台)进行构建， 并使用Erlang语言和运行时环境来实现。
    ActiveMQ 是Apache出品，最流行的，能力强劲的开源消息总线。 ActiveMQ 是一个完全支持JMS1.1和J2EE 1.4规范的 JMS Provider实现
    如果是单独安装需要设置主机名、HOSTS、puppet认证，这里是安装在puppet服务端。

2.  安装

1 2	[iyunv@puppet ~]# yum install -y tanukiwrapper activemq-info-provider [iyunv@puppet ~]# yum install -y activemq

    下载慢，失败了几次，多试试
3.  配置
    修改配置前最好备份下，并启动服务查看下进程是否运行

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

[iyunv@puppet ~]# cp /etc/activemq/activemq.xml /etc/activemq/activemq.xml.bak [iyunv@puppet ~]# vi /etc/activemq/activemq.xml #<>标签要按字母顺序排列，在标签<persistenceAdapter>后添加 <plugins>      <simpleAuthenticationPlugin>         <users>             <authenticationUser username="admin" password="admin@123" groups="mcollective,admins,everyone"/>             #账号"mcollective"将用于mc客户端配置             <authenticationUser username="mcollective" password="mcopwd123" groups="mcollective,everyone"/>               </users>     </simpleAuthenticationPlugin>     <authorizationPlugin>         <map>             <authorizationMap>                 <authorizationEntries>                     <authorizationEntry queue=">" write="admins" read="admins" admin="admins" />                     <authorizationEntry topic=">" write="admins" read="admins" admin="admins" />                     <authorizationEntry topic="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />                     <authorizationEntry queue="mcollective.>" write="mcollective" read="mcollective" admin="mcollective" />                     <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>                  </authorizationEntries>              </authorizationMap>          </map>      </authorizationPlugin> </plugins> #修改连接端口标签，只保留需要的即可（另外要注意的是注释附号内部不能包含注释符号） <transportConnectors>     <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/> </transportConnectors>

     Reloading httpd: not reloading due to configuration syntax error
    自动生成的httpd配置会导致httpd启动报错，proxy配置不对，可以不用这个配置也能访问

1	[iyunv@puppet ~]# mv /etc/httpd/conf.d/activemq-httpd.conf /etc/httpd/conf.d/activemq-httpd.conf.bak

4.  启动

1 2 3 4 5 6 7 8 9 10

[iyunv@puppet ~]# service activemq restart [iyunv@puppet ~]# chkconfig activemq on [iyunv@puppet ~]# netstat -tnlp |grep java tcp        0      0 :::61613                    :::*                        LISTEN      5644/java            tcp        0      0 :::47248                    :::*                        LISTEN      5644/java            tcp        0      0 :::8161                     :::*                        LISTEN      5644/java [iyunv@puppet ~]# tailf /var/log/activemq/activemq.log [iyunv@puppet ~]# iptables -I INPUT -p tcp --dport 8161 -j ACCEPT [iyunv@puppet ~]# iptables -I INPUT -p tcp --dport 61613 -j ACCEPT [iyunv@puppet ~]# service iptables save

5.  访问
    http://IP:8161/admin/


三、安装MC客户端
1.  安装
    安装在puppetmaster服务器上，使用root账号运行MC相关命令，采用puppetmaster的证书
    MC客户端不需要启动进程

1	[iyunv@puppet ~]# yum -y install mcollective-*

2.  配置
   修改配置前备份下配置文件

1	[iyunv@puppet ~]# mv /etc/mcollective/client.cfg /etc/mcollective/client.cfg.bak

    这里不使用ssl连接activemq，因为生成keystory.jks证书时失败，待解决

1	[iyunv@puppet ~]# vi /etc/mcollective/client.cfg

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

main_collective = mcollective collectives = mcollective libdir = /usr/libexec/mcollective logfile = /var/log/mcollective.log loglevel = info daemonize = 1 #Active MQ connector settings: connector = activemq direct_addressing = 1 plugin.activemq.pool.size = 1 #activemq服务器地址、端口、用户账号 plugin.activemq.pool.1.host = puppetmaster.ewin.com plugin.activemq.pool.1.port = 61613 plugin.activemq.pool.1.user = mcollective plugin.activemq.pool.1.password = mcopwd123 #启用ssl连接activemq，使用MC服务器的puppet证书 #plugin.activemq.pool.1.ssl = 1 #plugin.activemq.pool.1.ssl.ca = /var/lib/puppet/ssl/certs/ca.pem #plugin.activemq.pool.1.ssl.cert = /var/lib/puppet/ssl/certs/puppetmaster.ewin.com.pem #plugin.activemq.pool.1.ssl.key = /var/lib/puppet/ssl/private_keys/puppetmaster.ewin.com.pem #plugin.activemq.pool.1.ssl.fallback = 0 #SSL seurity plugin settings 安全插件默认为PSK: securityprovider = ssl plugin.ssl_server_public = /var/lib/puppet/ssl/certs/mcollective-servers.pem plugin.ssl_client_public = /var/lib/puppet/ssl/certs/puppetmaster.ewin.com.pem plugin.ssl_client_private = /var/lib/puppet/ssl/private_keys/puppetmaster.ewin.com.pem # Facts factsource = yaml plugin.yaml = /etc/mcollective/facts.yaml

    mcollective-servers.pem证书在下一步中生成。
    将文件中的puppet服务器主机名修改成你自己的:

1	:%s/puppetmaster.ewin.com/puppet.yourdomain.cn/g

四、安装MC服务端
说明：既然有puppet了，就不必一台一台来安装MC服务端了，这里写个mcollective模块推送到puppet agent。
I、Linux系统
1、    创建目录
      在puppet master服务器中创建mcollective模块需要的目录

1 2	[iyunv@puppet ~]# mkdir -p /etc/puppet/modules/mcollective/{manifests,templates,files} [iyunv@puppet ~]# mkdir -p /etc/puppet/modules/mcollective/files/pem/clients

2、    创建类

1	[iyunv@puppet ~]# vi /etc/puppet/modules/mcollective/manifests/init.pp

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

class mcollective{     #/etc/activemq/activemq.xml中配置的mcollective用户的密码，变量会传递到erb模块中     $activemq_server = "puppetmaster.ewin.com"     $mcollective_password = "mcopwd123"     package { ['mcollective','mcollective-puppet-agent','mcollective-service-agent']:         ensure => installed,     }     service { 'mcollective':         ensure  => running,         enable  => true,         require => Package['mcollective'],     }     #通过SVN提交的，需要过滤.svn目录     file { '/etc/mcollective':         ensure  => directory,         source  => 'puppet:///modules/mcollective/pem',         ignore  => '.svn',         owner   => root,         group   => root,         mode    => '0640',         recurse => remote,         notify  => Service['mcollective'],     }     file { '/etc/mcollective/server.cfg':         ensure  => file,         owner   => root,         group   => root,         mode    => 400,         content => template("mcollective/server.cfg.erb"),         notify  => Service['mcollective'],     } }

3、   创建配置文件模板

1	[iyunv@puppet ~]# vi /etc/puppet/modules/mcollective/templates/server.cfg.erb

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

main_collective = mcollective collectives = mcollective libdir = /usr/libexec/mcollective logfile = /var/log/mcollective.log loglevel = info daemonize = 1 connector = activemq direct_addressing = 1 plugin.activemq.pool.size = 1 plugin.activemq.pool.1.host = <%= @activemq_server %> plugin.activemq.pool.1.port = 61613 plugin.activemq.pool.1.user = mcollective plugin.activemq.pool.1.password = <%= @mcollective_password %> securityprovider = ssl plugin.ssl_client_cert_dir = /etc/mcollective/clients plugin.ssl_server_private = /etc/mcollective/server_private.pem plugin.ssl_server_public = /etc/mcollective/server_public.pem #plugin.puppet.resource_allow_managed_resources = true #plugin.puppet.resource_type_whitelist = exec,file factsource = yaml plugin.yaml = /etc/mcollective/facts.yaml registerinterval = 600

4、    生成证书
      生成共享证书，将会同步给节点

1 2 3 4 5 6 7 8

[iyunv@puppet ~]# puppet cert generate mcollective-servers Notice: mcollective-servers has a waiting certificate request Notice: Signed certificate request for mcollective-servers Notice: Removing file Puppet::SSL::CertificateRequest mcollective-servers at '/var/lib/puppet/ssl/ca/requests/mcollective-servers.pem' Notice: Removing file Puppet::SSL::CertificateRequest mcollective-servers at '/var/lib/puppet/ssl/certificate_requests/mcollective-servers.pem' [iyunv@puppet ~]# cp /var/lib/puppet/ssl/private_keys/mcollective-servers.pem  /etc/puppet/modules/mcollective/files/pem/server_private.pem [iyunv@puppet ~]# cp /var/lib/puppet/ssl/public_keys/mcollective-servers.pem  /etc/puppet/modules/mcollective/files/pem/server_public.pem [iyunv@puppet ~]# chmod 644 /etc/puppet/modules/mcollective/files/pem/server_*

      将MC客户端（puppetmaster）证书同步给节点

1 2	[iyunv@puppet ~]# cp /var/lib/puppet/ssl/certs/puppetmaster.ewin.com.pem  /etc/puppet/modules/mcollective/files/pem/clients/ [iyunv@puppet ~]# chmod 644 /etc/puppet/modules/mcollective/files/pem/clients/*

5、  节点更新
（1）、未安装foreman

1 2 3 4 5 6 7 8 9 10

[iyunv@puppet ~]# vi /etc/puppet/manifests/puppetagent.pp node "puppetagent.ewin.com" {     include mcollective } [iyunv@puppet ~]# vi /etc/puppet/manifests/site.pp import "puppetagent.pp" [iyunv@puppet ~]# vi /etc/puppet/puppet.conf [agent] report = true server = puppetmaster.ewin.com

    节点上Puppet agent进程需要关闭， 会导致agent不主动同步master， 需要在master设置crontab定期执行mco puppet runonce

1 2 3

[iyunv@puppet ~]# service puppet stop [iyunv@puppet ~]# chkconfig puppet off [iyunv@puppet ~]# puppet agent -t

（2）、已安装foreman
    如果事先部署了foreman，需要在foreman页面->配置->Puppet类-从Puppet-Proxy导入->选择联接的环境->主机中编辑puppetagent->类标签中添加mcollective模块->在节点上执行puppet agent -t

1 2 3

[iyunv@puppet ~]# service puppet stop [iyunv@puppet ~]# chkconfig puppet off [iyunv@puppet ~]# puppet agent -t

II、Windows系统
1、    软件说明
（1）、      ruby
        windows上需要安装ruby来运行mcollective
        http://rubyinstaller.org/
        下载rubyinstaller-1.9.3-p551.exe
（2）、     mcollective
        mcollective本身并没有相应的windows服务端安装包
        http://repo.mornati.net/mcollective/
        下载2.3.2下的mcollective_2_3_2_Setup.exe
        先安装ruby才能运行bat脚本安装gem包和安装服务
（3）、      puppet插件
        https://github.com/puppetlabs/mcollective-puppet-agent/releases
        1.7.1版本无法正常加载puppet agent，插件 1.6.1就没问题
        https://codeload.github.com/pupp ... pet-agent-1.6.1.zip
        解压后将里面的文件复制粘贴到c:\mcollective\plugins\mcollective，提示替换时同意替换
（4）、      ssl密钥
        密钥文件将被上传到客户端的以下路径
        c:\mcollective\etc\ssl\clients\puppetmaster.ewin.com.pem
        c:\mcollective\etc\ssl\server_private.pem
        c:\mcollective\etc\ssl\server_public.pem

2、    win-mcollective模块
（1）、模块目录结构
    在puppet的module下创建win-mcollective模块目录：

   

    在files目录下还有两个程序没有在上图显示：

    mcollective_2_3_2_Setup.exe

    rubyinstaller-1.9.3-p551.exe

（2）、      files
        下载ruby、mcollective程序，以及解压后的mcollective-puppet-agent插件。
        将puppet服务端证书文件存放在files下的相应位置。

（3）、    manifests
        init.pp

1 2 3 4 5 6

class win-mcollective{      $activemq_server = "puppetmaster.ewin.com"      $mcollective_password = "mcopwd123"      $mcollective_path      = "C:\Program Files\mcollective"      include win-mcollective::install,win-mcollective::config,win-mcollective::service }

        install.pp

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71

class win-mcollective::install {     if $operatingsystem == "windows" {         $filepath   = "C:/Users/Administrator/Downloads"         $installdir   = "C:/Program Files"         Exec{ path => ["${installdir}/Ruby193/bin","C:/Windows/system32","${installdir}/Puppet Labs/Puppet/bin" ] }         #安装ruby         file { 'ruby':             path    => "${filepath}/rubyinstaller-1.9.3-p551.exe",             ensure  => file,             owner   => 'Administrator',             mode    => '0755',             group   => 'Administrators',             source  => 'puppet:///modules/win-mcollective/rubyinstaller-1.9.3-p551.exe',         }         exec { 'ruby':             command => 'cmd.exe /c Start "puppet-install" /w "C:/Users/Administrator/Downloads/rubyinstaller-1.9.3-p551.exe" /SILENT /DIR="C:/Program Files/Ruby193"',             provider => 'windows',             creates     => "${installdir}/Ruby193/bin/ruby.exe",             require=> File['ruby'],         }         #安装mcollective         file { 'mcollective':             path    => "${filepath}/mcollective_2_3_2_Setup.exe",             ensure  => file,             owner   => 'Administrator',             mode    => '0755',             group   => 'Administrators',             source  => 'puppet:///modules/win-mcollective/mcollective_2_3_2_Setup.exe',         }         exec { 'mcollective':             command  => 'cmd.exe /c Start "puppet-install" /w "C:/Users/Administrator/Downloads/mcollective_2_3_2_Setup.exe" /SILENT /DIR="C:/Program Files/mcollective"',             provider => 'windows',             creates  => "${installdir}/mcollective/bin/mco.bat",             require  => File['mcollective'],         }         # 安装gem包         exec { 'install_gems':             command  => 'cmd.exe /c gem install -l -f "C:/Program Files/mcollective/gems/*.gem"',             provider => 'windows',             unless   => 'cmd.exe /c gem list --local | findstr win32-dir',             require  => [Exec['ruby'],Exec['mcollective']],         }         # 安装服务         exec { 'install_service':             command  => 'cmd.exe /c C:/Progra~1/mcollective/bin/register_service.bat',             provider => 'windows',             unless   => 'cmd.exe /c net start | find "The Marionette Collective"',             require  => Exec['install_gems'],         }         #puppet插件和aq连接认证文件         file { "${installdir}/mcollective/plugins/mcollective":             ensure  => directory,             ignore  => '.svn',             source_permissions => ignore,             source  => 'puppet:///modules/win-mcollective/mcollective-puppet-agent-1.6.1',             recurse => true,             require => Exec['mcollective'],         }         file { "${installdir}/mcollective/etc/ssl":         ensure  => directory,         ignore  => '.svn',         source_permissions => ignore,                 source  => 'puppet:///modules/win-mcollective/pem',                 recurse => true,         require => Exec['mcollective'],         }     } }

        config.pp   

1 2 3 4 5 6 7 8

class win-mcollective::config {     file { 'C:\\Progra~1\\mcollective\\etc\\server.cfg':          ensure  => present,          content => template("win-mcollective/server.cfg.erb"),          notify  => Class['win-mcollective::service'],          require => Class['win-mcollective::install'],     } }

        service.pp

1 2 3 4 5 6 7

class win-mcollective::service {     service { 'mcollectived':      ensure  => running,      enable  => true,      require => Class['win-mcollective::config'],     } }

（3）、    templates

        server.cfg.erb

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

main_collective = mcollective collectives = mcollective libdir = <%= @mcollective_path %>\plugins;C:\Program Files\Puppet Labs\Puppet\puppet\lib;C:\Program Files\Puppet Labs\Puppet\facter\lib logfile = <%= @mcollective_path %>\mcollective.log loglevel = info daemonize = 1 securityprovider = ssl plugin.ssl_client_cert_dir = <%= @mcollective_path %>\etc\ssl\clients plugin.ssl_server_private = <%= @mcollective_path %>\etc\ssl\server_private.pem plugin.ssl_server_public = <%= @mcollective_path %>\etc\ssl\server_public.pem connector = activemq plugin.activemq.pool.size = 1 plugin.activemq.pool.1.host = <%= @activemq_server %> plugin.activemq.pool.1.port = 61613 plugin.activemq.pool.1.user = mcollective plugin.activemq.pool.1.password = <%= @mcollective_password %> plugin.puppet.command = "C:\Program Files\Puppet Labs\Puppet\bin\puppet.bat" agent plugin.puppet.config = C:\ProgramData\PuppetLabs\puppet\etc\puppet.conf factsource = yaml plugin.yaml = <%= @mcollective_path %>\etc\facts.yaml

小结：以上完成后就可以使用MCO命令了，详见下一编文章。

五、整合Foreman
1、  说明
    要求：Mcollective+ActiveMQ部署完成，且mco puppet -v runonce命令运行成功
    实现：在foreman上通过mcollective触发puppet agent命令
2、  设置
    在foreman页面上设置puppetrun为true

3、  配置

1	[iyunv@puppet ~]# vi /etc/foreman-proxy/settings.d/puppet.yml

1 2 3

:enabled: true :puppet_conf: /etc/puppet/puppet.conf :puppet_provider: mcollective

1 2	[iyunv@puppet ~]# chmod u+w /etc/sudoers.d/foreman-proxy [iyunv@puppet ~]# vi /etc/sudoers.d/foreman-proxy

1 2	foreman-proxy ALL = NOPASSWD : /usr/bin/puppet cert *, /usr/bin/mco puppet runonce * Defaults:foreman-proxy !requiretty

      重启服务

1	[iyunv@puppet ~]# /etc/init.d/foreman-proxy restart

4、  计划
    前文提到要关闭agent的puppet服务，需要使用计划来执行mco命令，执行同步

1	[iyunv@puppet ~]# crontab -e

1	SHELL=/bin/bash */1 * * * * /usr/bin/mco puppet runonce *

      测试下每分钟执行命令； 正常后改为*/30每半小时

1	[iyunv@puppet ~]# service crond reload

     查看计划日志

1	[iyunv@puppet ~]# tailf -n 3  /var/log/cron

1 2 3

Apr 25 16:22:01 puppetmaster CROND[11613]: (root) CMD (/usr/bin/mco puppet runonce *) Apr 25 16:23:01 puppetmaster CROND[11713]: (root) CMD (/usr/bin/mco puppet runonce *) Apr 25 16:24:01 puppetmaster CROND[11823]: (root) CMD (/usr/bin/mco puppet runonce *)

    查看foreman报告
   

5、  测试
    foreman页面上－主机－选择执行－运行puppet：（不选择主机时，“选择执行”下拉菜单不会显示）
    查看日志

1 2	[iyunv@puppet ~]# tailf  /var/log/foreman-proxy/proxy.log 10.188.1.53 - - [23/Apr/2015 16:19:19] "POST /run HTTP/1.1" 200 - 0.5008

    开启debug模式，显示更详细的信息，排错的时候使用

1 2	[iyunv@puppet ~]# vi /etc/foreman-proxy/settings.yml :log_level: DEBUG

六、遇到过的错误
1、  activemq启动不了
    activemq not runnin

1	[iyunv@puppet ~]#

    没有装java工具tanukiwrapper
2、节点连接activemq失败

1 2	[iyunv@puppetagent ~]# tailf /var/log/mcollective.log I, [2015-04-23T14:21:44.337723 #11451]  INFO -- : activemq.rb:129:in `on_connectfail' TCP Connection to stomp://mcollective@puppetmaster.ewin.com:61613 failed on attempt 9

    puppet模块mcollective中类定义里的activemq_server值写错了
3、mco命令执行失败
   
    /etc/activemq/activemq.xml和 /etc/mcollective/client.cfg中的用户密码必须一致
4、运行PUPPET失败排错
（1)、查看日志

1	[iyunv@puppet ~]# t

      这是由于在puppetmaster上应用了puppet模块，把puppet.conf配置冲掉了

1	WARN -- : Non-null exit code when executing '/usr/bin/sudo/usr/bin/mcopuppetrunonce-It-search01-55.ewin.com'

      http://www.pythian.com/blog/some ... etrun-with-foreman/
（2）、检查puppetmaster
      mcollective  mcollective-common    是否安装
      foreman-proxy    是否正常启动
      activemq    是否连接正常
      mco ping     是否检测到客户端
      mco puppet runonce -I agent.ewin.com    是否成功执行
      mco inventory puppetagent.ewin.com    是否成功安装puppet插件
（3）、检查puppetagent
      mcollective-puppet mcollective-client    是否安装
      /var/log/mcollective.log    查看日志是否有错误信息
      系统时间是否正常