JBoss EAP 6 monitoring using remoting-jmx and Zabbix-IT猪猪

8516830

　　Setting up Zabbix monitoring for JBoss EAP 6 turns out te be quite different as for JBoss EAP 5. This is because the method of fetching JMX data in EAP 5 is no longer available in EAP 6.
　　In this post I’ll describe a method on how to enable JMX monitoring for JBoss EAP 5 and JBoss EAP 6 within the same Zabbix installation.
　　Update 22-01-2014: You can download a custom zabbix-java-gateway RPM file for Zabbix 2.2.1 on RHEL/CentOS6 64-bit from my site here. The process of creating the RPM can be found here.
　　Please note: You still need to add the remoting-libraries from your EAP installation to /usr/sbin/zabbix_java/lib if you use my RPM.
　　Problem:
　　JBoss EAP 5 supports JMX monitoring using RMI, where JBoss EAP 6 does not. EAP 6 uses “remoting-jmx” instead of “rmi”. This is a problem because the Zabbix Java Gateway only supports RMI for the JMX interface, preventing us from monitoring JBoss EAP 6 using JMX monitoring.
　　Goal:
　　My goal in this blogpost is to enable JMX monitoring for JBoss EAP 5 and JBoss EAP 6 on the same Zabbix Server.
　　Steps:
　　1.Modify source code of the Zabbix Java Gateway to enable both RMI and remoting-jmx
　　2.Add the required libraries to the Zabbix Java Gateway
　　3.Enable remote management in JBoss EAP 6
　　4.Configure a JBoss EAP 5 and JBoss EAP 6 host to test the new configuration

　　5.Using JConsole to>　　Let’s get started.
　　1. Modify source code of the Zabbix Java Gateway to enable both RMI and remoting-jmx
　　Like stated before, the Zabbix Java Gateway component only supports JMX over RMI. This is because the service URL it is using to connect to the remote server is hardcoded in JMXItemChecker.java:
　　url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + conn + ":" + port + "/jmxrmi");
　　At the time of writing the most current version of Zabbix is 2.2.1, and there is no way to customise this JMX url. JBoss EAP 6 doesn’t support RMI and expects remoting-jmx.
　　Replacing the url with this one will enable remoting-jmx, at the cost of losing RMI monitoring:
　　url = new JMXServiceURL("service:jmx:remoting-jmx://" + conn + ":" + port);

　　In my environment I want to monitor both JBoss EAP 5 and EAP 6, and I don’t want to add a second zabbix server to do so. So while searching for a way to do this with minimal changes to the zabbix source code I had the>
　　It’s far from>　　Here are my notes of modifying the source code and building new binaries on a clean CentOS 6.5 machine.
　　Preparing the build server:
　　# yum -y update
　　# reboot
　　# yum groupinstall "Development Tools"
　　# yum -y install mysql-server httpd php mysql-devel php-mysql libxml2-devel libcurl-devel php-mbstring php-bcmath php-gd php-xml
　　Getting the software:
　　# cd /opt
　　# wget http://sourceforge.net/projects/zabbix/files/ZABBIX%20Latest%20Stable/2.2.1/zabbix-2.2.1.tar.gz/download
　　# tar -xvzf zabbix-2.2.1.tar.gz
　　Modify the JMXItemChecker.java:
　　# vi zabbix-2.2.1/src/zabbix_java/src/com/zabbix/gateway/JMXItemChecker.java
　　Change:
　　1 String conn = request.getString(JSON_TAG_CONN);
　　2 int port = request.getInt(JSON_TAG_PORT);
　　3
　　4 url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + conn + ":" + port + "/jmxrmi");
　　5 jmxc = null;
　　6 mbsc = null;
　　view raw JMXItemChecker.java hosted with  by GitHub
　　To:
　　1 String conn = request.getString(JSON_TAG_CONN);
　　2 int port = request.getInt(JSON_TAG_PORT);
　　3
　　4 //Dirty solution for ZBXNEXT-1274
　　5 Integer remoting = new Integer("7777");
　　6 int retval = remoting.compareTo(port);
　　7
　　8 if (retval == 0)
　　9 {
　　10         url = new JMXServiceURL("service:jmx:remoting-jmx://" + conn + ":" + port);
　　11 }
　　12 else
　　13 {
　　14         url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + conn + ":" + port + "/jmxrmi");
　　15 }
　　16 jmxc = null;
　　17 mbsc = null;
　　view raw JMXItemChecker.java hosted with  by GitHub
　　This enables remoting-jmx for hosts with a JMX Interface on port 7777 only, while keeping RMI for any other port. Of course you can change this port to be whatever you need, just be sure to match the port you configure in JBoss.
　　That is all there is to it, let’s compile  the binaries and install our modified version of Zabbix:
　　# cd zabbix-2.2.1
　　# ./configure --enable-server --enable-agent --with-mysql --with-libcurl --with-libxml2 --enable-java
　　# make install
　　Configure the Java Pollers:
　　# vi /usr/local/etc/zabbix_server.conf
　　Add these values and save:
　　JavaGateway=10.37.129.2
　　StartJavaPollers=5
　　JavaGatewayPort=10052
　　3. Add the required libraries to the Zabbix Java Gateway
　　Before we can start the java gateway we need to tell it how to use remoting-jmx. We can use the jar files from our JBoss EAP 6 installation to do this.
　　Find the following jar files in your /modules/system/layers/base/:
　　$ cat needed_modules.txt
　　jboss-as-remoting
　　jboss-logging
　　jboss-logmanager
　　jboss-marshalling
　　jboss-marshalling-river
　　jboss-remoting
　　jboss-sasl
　　jcl-over-slf4j
　　jul-to-slf4j-stub
　　log4j-jboss-logmanager
　　remoting-jmx
　　slf4j-api
　　xnio-api
　　xnio-nio
　　Place these in the java gateway lib directory:
　　$ for i in $(cat needed_modules.txt); do find /opt/jboss-eap-6.0 -iname ${i}*.jar -exec cp {} /usr/local/sbin/zabbix_java/lib/ \; ; done
　　If you’re reading this post I assume setting up the Zabbix database and web frontend is trivial so I won’t cover it here. Let’s start the java gateway and zabbix server and add a JBoss EAP 5 and JBoss EAP 6 host to test our modifications.
　　3. Enable remote management in JBoss EAP 6
　　I’m using a .zip deployment of JBoss EAP 6, unzipped in /opt/jboss-eap-6.0 and running on a host with an external IP number of 10.37.129.5.
　　In this version of JBoss a user is required to do remote monitoring. Fortunately JBoss comes with the add-user.sh script to do this for you:
　　[root@jboss6 bin]# cd /opt/jboss-eap-6.0/bin/
　　[root@jboss6 bin]# ./add-user.sh
　　What type of user do you wish to add?
　　a) Management User (mgmt-users.properties)
　　b) Application User (application-users.properties)
　　(a):
　　Enter the details of the new user to add.
　　Realm (ManagementRealm) :
　　Username : zabbix
　　Password : zabbix_1
　　Re-enter Password : zabbix_1
　　About to add user 'zabbix' for realm 'ManagementRealm'
　　Is this correct yes/no? yes
　　Added user 'zabbix' to file '/opt/jboss-eap-6.0/standalone/configuration/mgmt-users.properties'
　　Added user 'zabbix' to file '/opt/jboss-eap-6.0/domain/configuration/mgmt-users.properties'
　　Is this new user going to be used for one AS process to connect to another AS process?
　　e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls.
　　yes/no? no
　　We can now use the user “zabbix” with a password of “zabbix_1” for remote management. We also need to configure the management interface of JBoss to listen on port 7777.
　　I’m going to run my JBoss instance in standalone mode with the standalone-full profile. To configure the management port, open:
　　/opt/jboss-eap-6.0/standalone/configuration/standalone-full.xml
　　Search for this line:
　　
　　Change the port number to 7777:
　　
　　Save the file and start the JBoss instance:
　　/opt/jboss-eap-6.0/bin/standalone.sh --server-config=standalone-full.xml -b 10.37.129.5 -Djboss.bind.address.management=10.37.129.5
　　4. Configure a JBoss EAP 5 and JBoss EAP 6 host to test the new configuration
　　In the Zabbix web interface go to: Configuration -> Hosts -> Create host.
　　The first host I will add is a JBoss EAP 5 host, so it will require JMX RMI monitoring:
　　Hostname: jboss.kanbier.lan
　　Groups: Linux servers
　　JMX Interfaces: jboss.kanbier.lan DNS 9999
　　5. Using JConsole to>
　　There are at least 2 ways to get the JMX keys fron JBoss so you can add them  to Zabbix. In my example for JBoss EAP 6 I use the key:
"jboss.as:subsystem=datasources,data-source=ExampleDS","jndiName"　　This fetches the “jndiName” from the ExampleDS data-source. How to construct  this key? You can easily browse the items that are available for monitoring  using either JConsole or jboss-cli.sh.  In this example I’ll be using  JConsole.
　　JConsole is a Java GUI application that connects to the management interface  of JBoss in very much the same way as the Zabbix Java Gateway. It is shipped  with JBoss and located in /bin.
　　Before you start it, make sure your JBoss EAP 6 instance is running using the  configuration described in the instructions above.
　　These are the steps I took to get it started on jboss6.kanbier.lan with  external ip 10.37.129.5.
$ export PATH=/usr/java/latest/bin:$PATH　　
$ export JAVA_HOME=/usr/java/latest
　　
$ /opt/jboss-eap-6.0/bin/jconsole.sh
　　This should start a GUI if you have an X session available. Fill in:
Check: Remote Process　　
service:jmx:remoting-jmx://10.37.129.5:7777
　　
Username: zabbix
　　
Password: zabbix_1
　　But there is one downside, the data available is only realtime. So no history  unless you keep JConsole running forever. This is where Zabbix comes in. You can  pull the data from the management interface just like JConsole does and store it  in Zabbix so you can build up a history, analyse trends and all things Zabbix  does best.
　　Let’s say you want to monitor the HeapMemoryUsage in Zabbix. This requires  some knowledge of JBoss, but here is how to find it in JConsole.
　　Click on the MBeans tab -> java.lang -> Memory -> Attributes ->  HeapMemoryUsage: