Cloud in Action: Learn and Practice Chef on Ubuntu

qmya00

　　xueguofeng2011@gmail.com

　　
　　I heard Chef many times but never took it seriously. Recently I’m learning AWS OpsWorks -  the hosted services to use Chef cookbooks and solutions for configration management and automation, and fully stuck by Chef this time. I did  some research on Internet and was confused by many terminologies used by Chef, such as kitchen, cookbook, recipe and knife etc, which makes me a little bit distracted. For most of blogs and articles on Internet, they are helpful to review Chef instead of learning it at the first time, I was anxious with learning the new stuff and facing a steep learning curve. After investing the whole two days on learning Ruby – another OOP and explanation language like Python, installing and playing with Chef on my virtual lab environment, finally I was able to understand the core concepts of Chef and move forward with AWS OpsWorks. Learning unknowns is always a painful and sweet journey !

　　
　　Let’s review the core concepts regarding to Chef before going to the lab and configuration:
　　
　　Chef, Puppet, Ansible and SaltStack present different paths to achieve a common goal of managing vast numbers of computers efficiently, with minimal input from developers and sysadmins. Chef is one of the most popular confiugration management tools, and it uses Ruby as its reference language to define patterns in recipes and cookbooks to configure, deploy and manage nodes across the network. Chef provides several management mode:
　　
　　1)      Client-Server. The main management mode which contains three components: A Chef Client is an agent that runs locally on every node managed by Chef, it performs the following steps to bring the node into the expected state: registering and authenticating the node with Chef Server, synchronizing cookbooks and configuring the node. The Chef Server acts as a hub for configuration data to store the cookbooks and policies that are applied to nodes, and metadata that describes each registered node that is being managed by Chef Client. Chef Development Kit on Chef Workstation is a package that contains everything needed to start using Chef, such as Chef Client, Chef and Knife CLI and test tools, to build and test cookbooks and interact with Chef Server.

　　2)      Chef-Solo. Doesn’t require the Chef Server and doesn’t support the following functionality by the Client-Server mode either: centralized distribution of cookbooks, a centralized API Gateway that interacts with and integrates infrasture components, and authentication or authorization. All the managed network nodes are only installed with the Chef Client software and use the local mode of Chef-Client to run the recipes.

　　
　　3)      Hosted Chef - the cloud-based SaaS. For example, AWS OpsWorks is a configuration management service that helps you configure and operate applications in a hybrid cloud environment by using Puppet or Chef. AWS OpsWorks for Chef Automate lets you create AWS-managed Chef servers that include Chef Automate premium features, and use the Chef DK and other Chef tooling to manage them. AWS OpsWorks Stacks provides a simple and flexible way to create and manage stacks and applications, and run s the Chef recipes using Chef-Solo to handle tasks such as installing packages on instances, deploying apps, and running scripts. It supports both AWS EC2 instances and on-premises servers that are installed with the OpsWorks Stack Agent and have connection to AWS public endpoints.
　　
　　Cookbooks are the configuration units allowing us to configure and perform specific tasks within Chef on the remote nodes and bring a node into a specific state. Cookbooks are created on the workstation and then uploaded to a Chef Server. From there, recipes and policies described within the cookbook can be assigned to nodes as part of the node's "run-list". A run-list is a sequential list of recipes and roles that are run on a node by chef-client in order to bring the node into compliance with the policy you set for it.

　　A recipe is the main workhorse of the cookbook. A cookbook can contain more than one recipe, or depend on outside recipes. Recipes are used to declare the state of different resources. A recipe is a list related resources that tell Chef how the system should look if it implements the recipe. When Chef runs the recipe, it checks each resource for compliance to the declared state. If the system matches, it moves on to the next resource; otherwise, it attempts to move the resource into the given state.

　　The chef-client will run a recipe only when asked. When the chef-client runs the same recipe more than once, the results will be the same system state each time. When a recipe is run against a system, but nothing has changed on either the system or in the recipe, the chef-client won’t change anything.

　　
　　For the Client-Server management mode, 4 VMs have been created by me to practice the soltuion. Below are the configurations:
　　

　　

Chef Server	Chef Workstation	Node1	Node2
// Upgrade the system – Ubuntu 16.04 LTS sudo apt-get update sudo apt-get upgrade 　　sudo apt-get dist-upgrade 　　 　　// Change the interace name to eth0 sudo gedit /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" GRUB_CMDLINE_LINUX="net.ifnames=0   biosdevname=0" sudo update-grub 　　sudo grub-mkconfig -o   /boot/grub/grub.cfg 　　 // Install openssh & git sudo apt-get install openssh-server sudo apt-get install openssh-client sudo apt-get install sysv-rc-conf sudo sysv-rc-conf ssh on sudo sysv-rc-conf --list | grep ssh netstat -ta | grep ssh 　　 　　sudo apt-get install   git-core
// Configue interface, IP addr, hostname and DNS
sudo gedit   /etc/network/interfaces auto eth0 iface eth0 inet static address 10.0.0.51 netmask 255.255.255.0 gateway 10.0.0.2 dns-nameserver 10.0.1.1	sudo gedit   /etc/network/interfaces auto eth0 iface eth0 inet static address 10.0.0.52 netmask 255.255.255.0 gateway 10.0.0.2 　　dns-nameserver 10.0.1.1	sudo gedit   /etc/network/interfaces auto eth0 iface eth0 inet static address 10.0.0.61 netmask 255.255.255.0 gateway 10.0.0.2 　　dns-nameserver 10.0.1.1	sudo gedit   /etc/network/interfaces auto eth0 iface eth0 inet static address 10.0.0.62 netmask 255.255.255.0 gateway 10.0.0.2 dns-nameserver   10.0.1.1
// Configure hostname
sudo gedit   /etc/hostname server	sudo gedit   /etc/hostname workstation	sudo gedit   /etc/hostname node1	sudo gedit /etc/hostname node2
// Configure the hosts file
sudo   gedt /etc/hosts 　　127.0.0.1              localhost 　　127.0.1.1              server.xgf.com server	sudo   gedt /etc/hosts 　　127.0.0.1              localhost 127.0.1.1              workstation.xgf.com   workstation	sudo   gedt /etc/hosts 　　127.0.0.1                localhost 127.0.1.1                node1.xgf.com node1	sudo   gedt /etc/hosts 　　127.0.0.1                localhost 127.0.1.1                node2.xgf.com node2
10.0.0.51       server.xgf.com server 　　10.0.0.52       workstation.xgf.com workstation 　　10.0.0.61       node1.xgf.com node1 10.0.0.62         node2.xgf.com node2
// Install the   Chef Server 　　 　　wget https://packages.chef.io/files/stable/chef-server/12.17.33/ubuntu/16.04/chef-server-core_12.17.33-1_amd64.deb 　　 　　sudo dpkg -i   chef-server-core_12.17.33-1_amd64.deb 　　 　　sudo chef-server-ctl   reconfigure 　　 　　// Create a   Chef User and Organization 　　 　　mkdir ~/.chef 　　 　　// Create an user 　　sudo chef-server-ctl   user-create gset guofeng xue xueguofeng2011@gmail.com 'ipcc2014' --filename ~/.chef/gset.pem 　　 　　// Create an organization and   add the user to the admins 　　sudo chef-server-ctl   org-create california "California, USA" --association_user gset   --filename ~/.chef/california.pem 　　 　　gset@server:~/.chef$ ls -l 　　total 8 　　-rw-r--r-- 1 root root 1674   Aug 20 17:42 california.pem 　　-rw-r--r-- 1 root root 1674   Aug 20 17:40 gset.pem	// Install   ChefDK on Chef Workstation 　　 　　wget   https://packages.chef.io/files/stable/chefdk/3.1.0/ubuntu/16.04/chefdk_3.1.0-1_amd64.deb 　　 　　sudo dpkg -i   chefdk_3.1.0-1_amd64.deb
// Install the   GUI - Chef Manage 　　 　　wget   https://packages.chef.io/files/stable/chef-manage/2.5.16/ubuntu/16.04/chef-manage_2.5.16-1_amd64.deb    　　 　　sudo   dpkg -i chef-manage_2.5.16-1_amd64.deb 　　 　　sudo   chef-manage-ctl reconfigure 　　 　　//   Visit the GUI 　　 　　https://server.xgf.com/organizations/california/nodes 　　https://server.xgf.com/organizations/california/cookbooks

Chef Workstation	Node1	Node2
// Generate the   chef-repo and move into it 　　gset@workstation:~$ chef generate   app chef-repo 　　gset@workstation:~$ cd chef-repo 　　 　　// Copy the RSA   private keys generated when setting up the Chef Server to the workstation 　　gset@workstation:~/chef-repo$ mkdir .chef 　　gset@workstation:~/chef-repo$ 　　scp   gset@10.0.0.51:~/.chef/*.pem ~/chef-repo/.chef/ 　　 　　gset@workstation:~/chef-repo/.chef$   ls -l 　　total   16 　　-rw-r--r--   1 gset gset 1674 Aug 20 17:47 california.pem 　　-rw-r--r--   1 gset gset 1674 Aug 20 17:47 gset.pem 　　-rw-rw-r--   1 gset gset  496 Aug 20 18:02 knife.rb 　　drwxrwxr-x   2 gset gset 4096 Aug 20 18:05 trusted_certs
// Add the   version control with Github 　　gset@workstation:~/chef-repo$ git config   --global user.name xueguofeng 　　gset@workstation:~/chef-repo$ git config   --global user.email xueguofeng2011@gmail.com 　　gset@workstation:~/chef-repo$ echo   ".chef" > .gitignore 　　gset@workstation:~/chef-repo$ git add . 　　gset@workstation:~/chef-repo$ git commit -m   "initial commit" 　　gset@workstation:~/chef-repo$git status 　　On   branch master 　　nothing   to commit, working directory clean
// Generate   knife.rb – the knife configuration file 　　gset@workstation:~/chef-repo$ cd .chef 　　gset@workstation:~/chef-repo/.chef$ gedit knife.rb 　　current_dir =   File.dirname(__FILE__) 　　log_level                :info 　　log_location             STDOUT 　　node_name                'gset' 　　client_key               "gset.pem" 　　validation_client_name   'california-validator' 　　validation_key           "california-validator.pem" 　　chef_server_url          'https://server.xgf.com/organizations/california' 　　cache_type               'BasicFile' 　　cache_options(   :path => "#{ENV['HOME']}/.chef/checksums" ) 　　cookbook_path              ["#{current_dir}/../cookbooks"] 　　 　　// Copy the   needed SSL certificates from the Chef Server 　　gset@workstation:~/chef-repo/.chef$ cd .. 　　gset@workstation:~/chef-repo$ knife ssl fetch 　　gset@workstation:~/chef-repo$ knife client   list
// Bootstrap a   node - install the Chef Client and validate the node 　　gset@workstation:~/chef-repo/.chef$ knife bootstrap 10.0.0.61 -x gset -P ipcc2014 --sudo   --node-name node1 　　gset@workstation:~/chef-repo/.chef$ knife   bootstrap 10.0.0.62 -x gset -P ipcc2014 --sudo --node-name node2 　　gset@workstation:~/chef-repo$ knife client   list 　　gset@workstation:~/chef-repo$ knife node   list 　　 　　gset@workstation:~/chef-repo$   knife client list 　　california-validator 　　node1 　　node2 　　 　　gset@workstation:~/chef-repo$   knife node list 　　node1 　　node2
// Create a   cookbook – “nginx” 　　gset@workstation:~/chef-repo/cookbooks$ chef generate   cookbook nginx 　　 　　gset@workstation:~/chef-repo/cookbooks/nginx$   ls -l 　　total   40 　　-rw-rw-r--   1 gset gset   77 Aug 20 18:29 Berksfile 　　-rw-rw-r--   1 gset gset  148 Aug 20 18:29   CHANGELOG.md 　　-rw-rw-r--   1 gset gset 1090 Aug 20 18:29 chefignore 　　drwxrwxr-x   3 gset gset 4096 Aug 20 18:34 files 　　-rw-rw-r--   1 gset gset   70 Aug 20 18:29 LICENSE 　　-rw-rw-r--   1 gset gset  748 Aug 20 18:29   metadata.rb 　　-rw-rw-r--   1 gset gset   53 Aug 20 18:29 README.md 　　drwxrwxr-x   2 gset gset 4096 Aug 20 21:37 recipes 　　drwxrwxr-x   3 gset gset 4096 Aug 20 18:29 spec 　　drwxrwxr-x   3 gset gset 4096 Aug 20 18:29 test 　　 　　// Create the   default recipe – “default.rb”, which will install and run Nginx, and transfer   the html file into Nginx’s root   　　gset@workstation:~/chef-repo/cookbooks/nginx/recipes$ gedit   default.rb 　　package 'nginx'   do 　　  action :install 　　end 　　service 'nginx'   do 　　  action [ :enable, :start ] 　　end 　　cookbook_file   "/var/www/html/index.html" do 　　  source "index.html" 　　  mode "0644" 　　end 　　 　　// Make the   index.html file   　　gset@workstation:~/chef-repo/cookbooks/nginx/files/default$ gedit   index.html 　　 　　   　　    Hello World 　　   　　   　　    Welcome to Chef ! 　　       --- Guofeng Xue 　　   　　 　　 　　// Upload the   cookbook – “nginx” to the Chef Server 　　gset@workstation:~/chef-repo$   knife   cookbook upload nginx 　　gset@workstation:~/chef-repo$   knife   cookbook upload –a 　　 　　gset@workstation:~/chef-repo/cookbooks/nginx$   knife cookbook upload -a 　　Uploading   chef-repo      [0.1.0] 　　Uploading   learn_chef_httpd [0.2.0] 　　Uploading   nginx          [0.1.0] 　　Uploading   xgf            [0.1.0] 　　Uploaded   all cookbooks. 　　 　　// Modify the   run-list of nodes 　　gset@workstation:~/chef-repo$   export   EDITOR=vi 　　gset@workstation:~/chef-repo$ knife node   edit node1 　　{ 　　  "name": "node1", 　　  "chef_environment":   "_default", 　　  "normal": { 　　    "tags": [    ] 　　  }, 　　  "run_list": [    “recipe[nginx]”  ] 　　} 　　gset@workstation:~/chef-repo$ knife node   edit node2 　　{ 　　  "name": "node2", 　　  "chef_environment":   "_default", 　　  "normal": { 　　    "tags": [    ] 　　  }, 　　  "run_list": [    “recipe[nginx]”,” recipe[xgf]”  ] 　　} 　　 　　gset@workstation:~/chef-repo$   knife node show node1 　　Node   Name:   node1 　　Environment:   _default 　　FQDN:        node1.xgf.com 　　IP:          10.0.0.61 　　Run   List:    recipe[nginx], recipe[xgf] 　　Roles:       　　Recipes:     nginx, nginx::default, xgf, xgf::default 　　Platform:    ubuntu 16.04 　　Tags:      　　 　　gset@workstation:~/chef-repo$   knife node show node2 　　Node   Name:   node2 　　Environment:   _default 　　FQDN:        node2.xgf.com 　　IP:          10.0.0.62 　　Run   List:    recipe[nginx] 　　Roles:       　　Recipes:     nginx, nginx::default 　　Platform:    ubuntu 16.04 　　Tags:   　　 　　// SSH into the   nodes and run the Chef Client 　　gset@workstation:~/chef-repo$   ssh   gset@10.0.0.61 　　gset@node1:~$   sudo   chef-client 　　gset@workstation:~/chef-repo$   ssh   gset@10.0.0.62 　　gset@node2:~$   sudo   chef-client	chef-client –v 　　 　　chef-solo -v	chef-client -v 　　 　　chef-solo –v
gset@workstation:~/chef-repo$   chef -v 　　Chef   Development Kit Version: 3.1.0 　　chef-client   version: 14.2.0 　　delivery   version: master (6862f27aba89109a9630f0b6c6798efec56b4efe) 　　berks   version: 7.0.4 　　kitchen   version: 1.22.0 　　inspec   version: 2.1.72 　　gset@workstation:~/chef-repo$   chef-client -v 　　Chef:   14.2.0 　　gset@workstation:~/chef-repo$   chef-solo -v 　　Chef:   14.2.0	sudo   chef-client	sudo   chef-client

　　With the Client-Server management mode, we can also use the local mode of Chef Client on the managed nodes:
　　

// Create a local recipe 　　gset@node2:~$   cat hello.rb 　　directory   "/home/gset/xgf111" do 　　    action :create 　　end 　　 　　// Run the recipe locally 　　gset@node2:~$   sudo chef-client --local-mode hello.rb

　　For the Chef-Solo management mode, below are the configurations:
　　

//   Install the Chef Client with the Chef install script 　　curl -L https://omnitruck.chef.io/install.sh | sudo bash 　　 　　// Create the recipe 　　gset@ubuntu16:~/chef-solo/cookbooks/helloworld/recipes$   cat default.rb 　　directory   "/home/gset/xgf3" do 　　    action :create 　　end 　　file   "/home/gset/xgf3/abc.txt" do 　　    content "Hello World!" 　　end 　　 　　// Build the configure file 　　gset@ubuntu16:~$   cat solo.rb 　　file_cache_path   "/home/gset/chef-solo" 　　cookbook_path   ["/home/gset/chef-solo/cookbooks"] 　　 　　// Define the run_list object 　　gset@ubuntu16:~$   cat node.json 　　{ 　　   "run_list": [   "recipe[helloworld]" ] 　　} 　　 　　// Run the recipe with Chef-Sole 　　gset@ubuntu16:~$   chef-solo -c ~/solo.rb -j ~/node.json

An Overview of Chef

https://docs.chef.io/chef_overview.html

　　Install a Chef Server Workstation on Ubuntu 18.04
　　https://www.linode.com/docs/applications/configuration-management/install-a-chef-server-workstation-on-ubuntu-18-04/
　　How To Create Simple Chef Cookbooks to Manage Infrastructure on Ubuntu
　　https://www.digitalocean.com/community/tutorials/how-to-create-simple-chef-cookbooks-to-manage-infrastructure-on-ubuntu
　　Nodes - Install via URL
　　https://docs.chef.io/install_omnibus.html
　　Executing Cookbooks with Chef-Solo
　　https://aggarwalarpit.wordpress.com/2015/12/01/executing-cookbooks-with-chef-solo/comment-page-1/