Groovy JMX Bean Monitoring

Alexandru Ersenie
Categories: Tech

Head Note

It’s been a while since i have last written an article, i guess life got a little bit more complex ever since it was filled by our two little kids 🙂 Time is now such an expensive resource, haha.

Anyway, back to the topic.

I have finally had some time at work to put some work into refactoring our Load Testing Framework. One of the key topics i had set my eyes on was a reliable way of monitoring the application servers, a way with very low overhead, stable and most of all adaptable. First i played a lot with the REST Monitoring Interface that Glassfish offers, check my other postings relating to that. REST Monitoring was quite cool, but it only allowed monitoring what Glassfish allowed by setting the respective monitoring levels ( JDBC connection pool, EJB Container, etc.) While that may suffice to some, it had following disadvantages:

  • Monitoring could be performed only on the levels exposed by Glassfish ( through module monitoring levels), meaning i could not get any System Load information for example
  • Each resource had to be queried separately
  • Each query result had to be parsed in order to be imported into the monitoring database
  • Relative overhead
  • Sometimes, under heavy load, resources were not available for querying

Besides that, i had to use a combination of CURL for performing the Request, XML processing and unix editors ( like sed or awk ) to get all this results straight. Not to mention the effort i had to put whenever i needed to set up a new monitoring item.

Further on, i had the problem that i could only monitor one application server at a time. Since we run our tests in a distributed environment, i needed something that i could use to easily monitor one to many application servers, at the same time! Since i had played with Groovy before, doing some integration with our Jenkins, i set once more my eyes on it.

JMX MBean Monitoring with Groovy

For the sake of keeping this article simple, we’ll use the best free Java APM  Tool there is on the market: VisualVM.

As defined in the JMX Specification, client applications (as ours) obtain MBeans through an MBean Server Connection. Once we have obtained an MBean server connection, we can use it to query the underlying beans, retrieving their attributes (of course, operating on the beans is possible as well)

Let’s have a look at two software components exposing a lot of debugging information through MBeans, Glassfish and OpenMQ

Glassfish MBean Monitoring

Glassfish by default exposes its’ JMX Server on port 8686. A default connection will also require username and password, which by default are: admin / adminadmin. Once you have connected to the JMX Server, you’ll notice a new tab called “MBeans”

Glassfish MBeans Tab

Glassfish MBeans Tab

Module Monitoring Levels in Glassfish

As you can see, the exposed MBeans are on the left. The most interesting for us will be the ones giving “runtime” information on performance metrics like: used database connections, commited transactions, queue lenghts, number of open connections and so on. In Glassfish you get all this for free, by enabling the so called “Module Monitoring” ( Just expand the MBean called “amx”, and navigate to the child called “Module Monitoring Levels”)

Module Monitoring Levels in Glassfish

Module Monitoring Levels in Glassfish

As you can see, we have enabled the monitoring for some of the modules, among them being:

  • JDBC Connection Pool: information regarding the number of acquired logical connections, physical connections, connection timeouts, etc.
  • Thread Pool: information regarding the number of active threads, total threads, etc
  • JVM: information regarding the current but also peak usage of the memory spaces

Let’s use the JDBC Connection Pool MBean to check on the current pool usage statistics. Expand the node called “amx:jdbc-connection-pool-mon”

JDBC Connection Pool Monitoring - Attribute value view

JDBC Connection Pool Monitoring – Attribute value view

We can see now metrics like:

  • number of free connections in the pool
  • number of logical connections acquired from the pool
  • number of currently used jdbc connections in the pool
  • etc.

So if we’d wanted to take a live peek at the system, check on it’s resource monitoring, we could do that easily following the steps above. But that is not all to it. How about things that Glassfish does not expose via its main Mbean “amx”, things like : Garbage Collection statistics, Memory Usage statistics for all Memory Spaces, Operating System statistics like CPU Usage and so on? Let’s take a look at the “java.lang” Mbean

Other MBeans

Other MBeans

We can retrieve CPU usage, Compilation statistics, Memory Space statistics, Garbage Collection Statistics and so on… The one thing missing is regularly checking this information, and aggregating the results. Which brings me to the todays’ topic. Before that, let’s summarize a little what we have acchieved up to now:

  1. Connect to a Glassfish Server using JMX connection (service:jmx:rmi:///jndi/rmi://server:8686/jmxrmi)
  2. Connect to an MBean Server using VisualVM and the MBeans plugin
  3. Configure monitoring levels in Glassfish (Module Monitoring Levels MBean)
  4. Retrieve some performance metrics from the JDBC Connection Pool Monitor
  5. Retrieve other performance metrics independent of Glassfish (JVM, Operating System, etc.)

Dynamically querying MBeans with Groovy

Let’s say we would like to keep a constant eye on the JDBC Connection Pool, and would like to retrieve it’s metrics every couple of seconds. Let’s take a look at the Groovy MBean specification:

Its constructor looks like this:

GroovyMBean(MBeanServerConnection server, ObjectName name)

We need a MBean Server Connection, and an object name. The object names of all exposed MBeans can be viewed in the metadata Tab of the MBeans Browser:

JDBC Connection Pool - Mbean Metadata

JDBC Connection Pool – Mbean Metadata

So let us first create a Connector class that connects us to the JMX Server. We will use it to pass a list of servers that we want to connect to later.

public class Connector {
static server
def serverUrl, user, password
Connector (serverUrl, user, password) {
this.serverUrl = serverUrl
this.user = user
this.password = password
}

def connect () {
HashMap environment = new HashMap();
String[] credentials = [user, password];
environment.put (JMXConnector.CREDENTIALS, credentials);

// Connect to remote MBean Server
def jmxUrl = 'service:jmx:rmi:///jndi/rmi://'+serverUrl+'/jmxrmi'
try {
println jmxUrl
server = JmxFactory.connect(new JmxUrl(jmxUrl),environment).MBeanServerConnection
return server
}
catch (Exception e) {
println("Could not connect to mbean")
//System.exit(0)
}
}
}

Since we now have the connection, we need the MBean’s connection name in order to work with it. I will use the expression “entry point” instead of the object name. All we have to do now is to create a new MBean Object:

def entryPoint='amx:pp=/mon/server-mon[server],type=jdbc-connection-pool-mon,name=resources/EocPool'</code>
def monitoringBean = new GroovyMBean(connection,entryPoint)</code>

We could now just go after the beans attributes by retrieving them using the full path. Let’ say we want to monitor the number of connections acquired:

def connAcquired=monitoringBean.numconnacquired.count

That would be all to it, this would give us back the value of the attribute. We could of course do a map of attributes, and retrieve the attributes one by one:

attributeMap = [ numconnacquired:count, numconnfree:current, numconnused:current]
attributeMap.each { key,value -> println (key,value)}

This would then look something like this:

numconnacquired 2117663
numconnfree 41
numconnused 54

Of course, if we wanted to monitor other Beans as well, we’ll have to create an attribute map for each of the beans as well. Why not go the other way around? Get the MBean, get all its attributes, retrieve all values for all attributes, and use only whatever we’d need. Let’s create a map of MBeans, containing a label (that we will later use for logging) and the entry point (object name) We will first do this for the JDBC Connection Pool and Thread Pool Monitoring Beans:

beanMap=[
ThreadPoolMonitor: [label:'Monitor - Thread Pool', entryPoint:'amx:pp=/mon/server-mon[server],type=thread-pool-mon,name=network/jk-main-listener-1/thread-pool'],
JdbcMonitor: [label:'Monitor - JDBC Pool', entryPoint:'amx:pp=/mon/server-mon[server],type=jdbc-connection-pool-mon,name=resources/EocPool'],
]

for (e in connectorMap ) {
beanMap.each { key,value -> setMonitoringBean(e.key,e.value,beanMap."$key".entryPoint,beanMap."$key".label)}
}

Let’s take a look at the setMonitoringBean method, which takes following arguments: server, connection, object name and label:

public static void setMonitoringBean(server,connection, entryPoint, label) {

// Initialize the value Map. We will hold all attributes and values in this map
def valueMap=[:]
def timestamp = new Date().format("yyyy-MM-dd HH:mm:ss")
try {
// Connect to the MBean using the given server and MBean Connection Information
def monitoringBean = new GroovyMBean(connection,entryPoint)
// Get the MBeans existing attributes and add them to a map so we could traverse it
def attributeList = [ monitoringBean.listAttributeDescriptions()].flatten()
// We need to split the MBean entry point so we can check on the type of the bean. Currently we support two types: composite types (having child attributes) and long types (single values)
// Traverse each attribute and store the values in the value map
//def beanObjectName=monitoringBean.name()
attributeList.each {
def splitter=it.split(' ')
def attributeType=splitter[1], attributeName=splitter[2]
if (attributeType=='javax.management.openmbean.CompositeData' ){
// Only store numeric number; filter out timestamp attributes
monitoringBean."${attributeName}".contents.each { key,value ->
try {
if ("$value".matches("[0-9].*") && !"$key".matches(".*Time")) valueMap.put(label+"|"+attributeName+"-"+key,value)
}
catch (Exception e) {logFile << 'Exception returned when checking attribute: '+attributeName+'\t'+e+'\n'}
}
}
else
{
if (attributeType=='long' || attributeType=='double' || attributeType=='java.lang.Long' || attributeType=='java.lang.Integer'){
// Directly store the value of the attribute, since it is a simple attribute
try {
def valueHolder=monitoringBean."${attributeName}"
valueMap.put(label+"|"+attributeName,valueHolder)

}
catch (Exception e) {logFile << 'Exception returned when checking attribute: '+attributeName+'\t'+e+'\n'} } } } // Flush the valueMap into the ResultFile valueMap.each { entry -> resultFile << server+"|"+timestamp+"|"+"$entry".replaceAll('=','|')+"|"+testId+"\n"}
valueMap.clear()
}
catch (Exception e) {
logFile << 'Something went wrong\n'
println e
valueMap.clear()
}

This would now return the following results:

myserver|2015-10-29 17:23:11|Monitor - Thread Pool|corethreads-count|5
myserver|2015-10-29 17:23:11|Monitor - Thread Pool|currentthreadsbusy-count|0
myserver|2015-10-29 17:23:11|Monitor - Thread Pool|totalexecutedtasks-count|49424
myserver|2015-10-29 17:23:11|Monitor - Thread Pool|maxthreads-count|1024
myserver|2015-10-29 17:23:11|Monitor - Thread Pool|currentthreadcount-count|83
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numpotentialconnleak-count|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconnsuccessfullymatched-count|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconnfailedvalidation-count|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconnreleased-count|343676
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|waitqueuelength-count|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconnfree-current|95
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconnfree-highWaterMark|95
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconnfree-lowWaterMark|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|connrequestwaittime-current|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|connrequestwaittime-highWaterMark|4586
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|connrequestwaittime-lowWaterMark|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconnused-current|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconnused-highWaterMark|67
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconnused-lowWaterMark|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconndestroyed-count|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconnacquired-count|343676
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|averageconnwaittime-count|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconntimedout-count|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconnnotsuccessfullymatched-count|0
myserver|2015-10-29 17:23:11|Monitor - JDBC Pool|numconncreated-count|95

Some words on the attributes and their types: We can have attributes of type:

  • composite (with subattributes)
  • long
  • integer
  • string
  • bollean

That is the reason we need to check on each of the attributes to see if we have to retrieve it’s subattributes or its value. Doing this we can dynamically retrieve all attributes of an MBean, deciding afterwards which we should use and which not.

If we add a map of servers as well, we’ll only need to connect once, and then retrieve the results by polling the servers periodically

def serverList = [
server1:”server1:”+monitoringPort,
server2:”server2:”+monitoringPort,
server3:”server3:”+monitoringPort,
]

All we have to do now is to add a loop and poll the MBeans in a loop. My implementation relies on the existence of a control file. As long as the file exists, the beans will be polled in 5 seconds interval.

Aggregation of JMX Monitoring collected with Groovy

All we need to do now is to import the data into a database of our choice, and draw the charts accordingly. This would look something like:

Groovy JMX JDBC Connection Pool Monitoring

Groovy JMX JDBC Connection Pool Monitoring

Adding monitoring for a new MBean

All you have to do is to extend the BeanMap with the new (one or more) Beans. Let’s say we would like some statistics on the memory spaces:


beanMap=[
ThreadPoolMonitor: [label:'Monitor - Thread Pool', entryPoint:'amx:pp=/mon/server-mon[server],type=thread-pool-mon,name=network/jk-main-listener-1/thread-pool'],
JdbcMonitor: [label:'Monitor - JDBC Pool', entryPoint:'amx:pp=/mon/server-mon[server],type=jdbc-connection-pool-mon,name=resources/EocPool'],
MemoryEden: [label:'Monitor - MemoryEden', entryPoint:'java.lang:type=MemoryPool,name=Par Eden Space'],
MemoryPerm: [label:'Monitor - MemoryPerm', entryPoint:'java.lang:type=MemoryPool,name=CMS Perm Gen'],
MemoryOld: [label:'Monitor - MemoryOld', entryPoint:'java.lang:type=MemoryPool,name=CMS Old Gen'],
MemorySurvivor: [label:'Monitor - MemorySurvivor', entryPoint:'java.lang:type=MemoryPool,name=Par Survivor Space']
]

That’s it, No other hussle, no nothing. Just add a new server, or a new bean, and there you go. And the best part of it, it connects only once to each of the servers, and then acts as an aggregator…Groovy isn’t it?

An aggregated report could then look like this:

Groovy JMX Monitoring Report

Groovy JMX Monitoring Report

Feel free to use the groovy script i have created, adapt it, extend it and so on. Critic opinions are welcomed as much as improvement ideas 😉 Let’s keep the open source going…APM tools can be so expensive noawadays.

I can go home to my kids now. Now this is groovy!

/*
Created: Alexandru Ersenie
Groovy script for monitoring Application Server over JMX Protocol
Usage: groovy $script_name $test_id $workspace $jmxPort
Usage example: groovy jmx_mon.groovy 21322 /home/testing 22086

Define the list of servers you want to monitor : class Main -> serverList
Define the list of mbeans you want to monitor : class Main -> beanMap
Define the user and password for the JMX Connection: class Connector (default admin:adminadmin)
Define the polling period in milliseconds: class Main -> pollTimer (default: 5000)
*/
import javax.management.ObjectName
import javax.management.remote.JMXConnectorFactory as JmxFactory
import javax.management.remote.JMXServiceURL as JmxUrl
import java.util.HashMap
import javax.management.remote.*
import java.text.DateFormat
import java.util.regex.*

public class Monitoring {
String connection, entryPoint
static logFile,resultFile,monitoringFile,testId
//Constructor
Monitoring () {
this.connection = connection
this.entryPoint = entryPoint
}
public static void setMonitoringBean(server,connection, entryPoint, label) {

// Initialize the value Map. We will hold all attributes and values in this map
def valueMap=[:]
def timestamp = new Date().format("yyyy-MM-dd HH:mm:ss")
try {
// Connect to the MBean using the given server and MBean Connection Information
def monitoringBean = new GroovyMBean(connection,entryPoint)
// Get the MBeans existing attributes and add them to a map so we could traverse it
def attributeList = [ monitoringBean.listAttributeDescriptions()].flatten()
// We need to split the MBean entry point so we can check on the type of the bean. Currently we support two types: composite types (having child attributes) and long types (single values)
// Traverse each attribute and store the values in the value map
//def beanObjectName=monitoringBean.name()
attributeList.each {
def splitter=it.split(' ')
def attributeType=splitter[1], attributeName=splitter[2]
if (attributeType=='javax.management.openmbean.CompositeData' ){
// Only store numeric number; filter out timestamp attributes
monitoringBean."${attributeName}".contents.each { key,value ->
try {
if ("$value".matches("[0-9].*") && !"$key".matches(".*Time")) valueMap.put(label+"|"+attributeName+"-"+key,value)
}
catch (Exception e) {logFile << 'Exception returned when checking attribute: '+attributeName+'\t'+e+'\n'}
}
}
else
{
if (attributeType=='long' || attributeType=='double' || attributeType=='java.lang.Long' || attributeType=='java.lang.Integer'){
// Directly store the value of the attribute, since it is a simple attribute
try {
def valueHolder=monitoringBean."${attributeName}"
valueMap.put(label+"|"+attributeName,valueHolder)

}
catch (Exception e) {logFile << 'Exception returned when checking attribute: '+attributeName+'\t'+e+'\n'} } } } // Flush the valueMap into the ResultFile valueMap.each { entry -> resultFile << server+"|"+timestamp+"|"+"$entry".replaceAll('=','|')+"|"+testId+"\n"}
valueMap.clear()
}
catch (Exception e) {
logFile << 'Something went wrong\n' println e valueMap.clear() } } public static void main(String[] args) { testId=args[0] def runtimeFolder=args[1] def monitoringPort=args[2] def beanMap=[:] def pollTimer=5000 resultFile = new File(runtimeFolder+"/monitoring/glassfish_stats.log") monitoringFile = new File(runtimeFolder+"/monitoring/control_file") logFile = new File (runtimeFolder+"/run.log") resultFile.write '' def serverList = [ def serverList = [ server1:"server1:"+monitoringPort, server2:"server2:"+monitoringPort, server3:"server3:"+monitoringPort, ] ] def connectorMap = [:] if (monitoringFile.exists()) {de serverList.each { key, value ->
if (key!='tstjms201c') {
println key
connectorMap.put(key,new Connector(value,'admin','adminadmin').connect())}
else {
println key
connectorMap.put(key,new Connector(value,'admin','admin').connect())
}
}
}
else
{
println("Monitoring file not found, monitoring will now exit")
logFile << 'Monitoring file not found, monitoring will not be performed' System.exit(0) } beanMap=[ ThreadPoolMonitor: [label:'Monitor - Thread Pool', entryPoint:'amx:pp=/mon/server-mon[server],type=thread-pool-mon,name=network/jk-main-listener-1/thread-pool'], JdbcMonitor: [label:'Monitor - JDBC Pool', entryPoint:'amx:pp=/mon/server-mon[server],type=jdbc-connection-pool-mon,name=resources/EocPool'], MemoryEden: [label:'Monitor - MemoryEden', entryPoint:'java.lang:type=MemoryPool,name=Par Eden Space'], MemoryPerm: [label:'Monitor - MemoryPerm', entryPoint:'java.lang:type=MemoryPool,name=CMS Perm Gen'], MemoryOld: [label:'Monitor - MemoryOld', entryPoint:'java.lang:type=MemoryPool,name=CMS Old Gen'], MemorySurvivor: [label:'Monitor - MemorySurvivor', entryPoint:'java.lang:type=MemoryPool,name=Par Survivor Space'] ] while (monitoringFile.exists()) { { for (e in connectorMap ) { beanMap.each { key,value -> setMonitoringBean(e.key,e.value,beanMap."$key".entryPoint,beanMap."$key".label)}
}
sleep(pollTimer)
}
}
}
public class Connector {
static server
def serverUrl, user, password
Connector (serverUrl, user, password) {
this.serverUrl = serverUrl
this.user = user
this.password = password
}

def connect () {
HashMap environment = new HashMap();
String[] credentials = [user, password];
environment.put (JMXConnector.CREDENTIALS, credentials);

// Connect to remote MBean Server
def jmxUrl = 'service:jmx:rmi:///jndi/rmi://'+serverUrl+'/jmxrmi'
try {
println jmxUrl
server = JmxFactory.connect(new JmxUrl(jmxUrl),environment).MBeanServerConnection
return server
}
catch (Exception e) {
println("Could not connect to mbean")
//System.exit(0)
}
}
}

Print Friendly, PDF & Email