Sunday 10 August 2014

memory leak Java java.lang.OutOfMemoryError: Java heap space alarm on zabbix

What to do when your java virtual machine is out if memory, when you java virtual machine has a memory leak? How to get alarm that java virtual machine has this issue? When user  calls you up that java application do not work, you will easily find java heap: out of memory message in java log. But how to get alarm when memory leak starts and out of memory java message starts to show in you log? I you use zabbix as you monitoring system then this is easy.Write a simples script that will loo for "out of memory" pattern in last 10 or t 20 lines of java virtual machine log file. When this pattern starts to show in last 10 or 20 lines,this will trigger alarm on zabbix server and that will send mail or sms to you!

Ok, script should look like this.
server#cat java_out.sh
#!/bin/bash
LOG='java_log_file'
OUT='/tmp/java_out_of_memory.txt'
a=`tail -10 $LOG|grep 'java.lang.OutOfMemoryError: Java heap space'|wc -l`
if [ a = = 0 ]
then
echo 0>$OUT
else
echo 1>$OUT
fi

In line where LOG is defined put location of your java application log.
Put this script in cron so that it triggers every minute.


* * * * * java_out.sh


In you zabbix_agent.conf insert line where you will define this new data that will be send to zabbix server. It should look something like this

UserParameter java_out_of_memory, cat /tmp/java_out_of_memory.txt|awk '{print $1}'

Restart zabbix agent.
server#/etc/init.d/zabbix_agent stop
server#/etc/init.d/zabbix_agent start

How to check if zabbix server gets java out of memory item that you created? On zabbix server execute this

zabbix_server#zabbix_get -s IP_of_server -k java_out_of_memory

If exit is 0, there is no problem with java memory leak. If exit is 1, you have memory leak because somewhere in last 10 lines of java log files you have java heap: out of memory  message.

Now through web interface insert in zabbix server java_out_of_memory item! With this you will have time diagram of how often memory leak has happen.


No comments: