I realized late last night the current vSphere Big Data Extensions fling does not have HAProxy built into it for the Mesos cluster deployments. After a bit of reading and testing new pieces inside the Chef recipes, I have added support so that HAProxy is running on all of the Mesos nodes. The first thing is to add the HAProxy package to the /opt/serengeti/chef/cookbooks/mesos/recipes/install.rb file:
72 %w( unzip libcurl haproxy ).each do |pkg| 73 yum_package pkg do 74 action :install 75 end 76 end
There is also a script that Mesosphere provides to modify the HAProxy configuration file and reload the rules when changes occur. You can find instructions on the file and how to incorporate it on the Mesosphere page.
Note: I had to edit ‘sudo’ out of the lines inside the script in order for Chef to execute it properly.
After copying the file haproxy-marathon-bridge into my Chef server, I added the following code to the same install.rb file to get things all setup and configured properly:
82 directory "/etc/haproxy-marathon-bridge" do 83 owner 'root' 84 group 'root' 85 mode '0755' 86 action :create 87 end 88 89 template '/usr/local/bin/haproxy-marathon-bridge' do 90 source 'haproxy-marathon-bridge.erb' 91 action :create 92 end 93 94 master_ips = mesos_masters_ip 95 slave_ips = mesos_slaves_ip 96 97 all_ips = master_ips 98 all_ips += slave_ips 99 100 template '/etc/haproxy-marathon-bridge/marathons' do 101 source 'marathons.erb' 102 variables( 103 haproxy_server_list: all_ips 104 ) 105 action :create 106 end 107 108 execute 'configure haproxy' do 109 command 'chkconfig haproxy on; service haproxy start' 110 end 111 112 execute 'setup haproxy-marathon-bridge' do 113 command 'chmod 755 /usr/local/bin/haproxy-marathon-bridge; /usr/local/bin/haproxy-marathon-bridge install_cronjob' 114 end
There is also a bit of supporting code needed for lines 94-98 above that were added to /opt/serengeti/chef/cookbooks/mesos/libraries/default.rb:
1 module Mesosphere
2
3 def mesos_masters_ip
4 servers = all_providers_fqdn_for_role("mesos_master")
5 Chef::Log.info("Mesos master nodes in cluster #{node[:cluster_name]} are: #{servers.inspect}")
6 servers
7 end
8
9 def mesos_slaves_ip
10 servers = all_providers_fqdn_for_role("mesos_slave")
11 Chef::Log.info("Mesos slave nodes in cluster #{node[:cluster_name]} are: #{servers.inspect}")
12 servers
13 end
14
15 end
16
17 class Chef::Recipe; include Mesosphere; end
The last thing needed is a new template file for the /etc/haproxy-marathon-bridge/marathons file that is needed by the script provided by Mesosphere. I created the file /opt/serengeti/chef/cookbooks/mesos/templates/default/marathons.erb:
1 # Configuration file for haproxy-marathon-bridge script
2 <%
3 ha_url_list = []
4 @haproxy_server_list.each do |ha_server|
5 ha_url_list << "#{ha_server}"
6 end
7 %>
8 <%= ha_url_list.join(":8080\n") + ":8080" %>
At this point, all of the modifications can be uploaded to the Chef server with the command knife cookbook upload -a and a new cluster can be deployed with HAProxy support.
After deploying a nginx workload, you scale it out and check the /etc/haproxy/haproxy.cfg file on a master node and see entries like:
[root@hadoopvm388 haproxy]# cat haproxy.cfg global daemon log 127.0.0.1 local0 log 127.0.0.1 local1 notice maxconn 4096 defaults log global retries 3 maxconn 2000 timeout connect 5000 timeout client 50000 timeout server 50000 listen stats bind 127.0.0.1:9090 balance mode http stats enable stats auth admin:admin listen nginx-80 bind 0.0.0.0:80 mode tcp option tcplog balance leastconn server nginx-10 hadoopvm382.localdomain:31000 check server nginx-9 hadoopvm390.localdomain:31000 check server nginx-8 hadoopvm387.localdomain:31000 check server nginx-7 hadoopvm389.localdomain:31000 check server nginx-6 hadoopvm386.localdomain:31000 check server nginx-5 hadoopvm383.localdomain:31000 check server nginx-4 hadoopvm378.localdomain:31001 check server nginx-3 hadoopvm381.localdomain:31000 check server nginx-2 hadoopvm385.localdomain:31000 check server nginx-1 hadoopvm378.localdomain:31000 check
Enjoy!
