Recently I had issue considering file system and glassfish!
Users were complaining that glassfish application stoped working! I logged on server and check glassfish server.log!
server#cd /opt/glassfish3/glassfish/domains/domain1/logs
server# tail server.log
server.log_2013-03-05T11-03-08:[#|2013-01-03T12:16:01.092+0100|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=23;_ThreadName=Thread-2;|WARNING: 8 bytes remaining but no space left|#]
server.log_2013-03-05T11-03-08:[#|2013-01-03T12:16:01.092+0100|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=23;_ThreadName=Thread-2;|WARNING: 300 bytes remaining but no space left|#]
server.log_2013-03-05T11-03-08:[#|2013-01-03T12:16:01.092+0100|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=23;_ThreadName=Thread-2;|WARNING: 300 bytes remaining but no space left|#]
server.log_2013-03-05T11-03-08:[#|2013-01-03T12:16:01.094+0100|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=23;_ThreadName=Thread-2;|WARNING: 8 bytes remaining but no space left|#]
server.log_2013-03-05T11-03-08:[#|2013-01-03T12:16:01.094+0100|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=23;_ThreadName=Thread-2;|WARNING: 8 bytes remaining but no space left|#]
Ok, no space left on that partition! But when I checked disk free space, there was space left on all partitions!
server# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-rootlv
9.2G 1.8G 7.0G 21% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 97M 32M 60M 35% /boot
/dev/mapper/rootvg-optlv
14G 1.6G 12G 12% /opt
/dev/mapper/rootvg-varlv
4.9G 282M 4.4G 7% /var
So there was free space left, but from some reason glassfish could not write on file system. I checked if I can write anything on file system. It turns out that I can write on all file system except on file system where glasshfish was mounted!
Hm....
After some thinking I desided to check status of inodes on partitions!
server# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/rootvg-rootlv
612000 55749 556251 10% /
tmpfs 490591 1 490590 1% /dev/shm
/dev/sda1 25688 39 25649 1% /boot
/dev/mapper/rootvg-optlv
897600 0 897600 100% /opt
/dev/mapper/rootvg-varlv
321280 2267 319013 1% /var
Ok, so I found the problem! There were no free inodes left. How many inodes you have on partition depends on size of that partition and what file system you use!
What is inode?
" Inode is a data structure used to represent a filesystem object, which can be one of various things including a file or a directory. "
Bassicly, number of inodes on files system defines max number of files on directories together!
So, I had close to 900 000 files or folders on my /opt partition! Because file system considering space was not full, there was to be some folder with many small size files!
I found my problem in this folder /opt/glassfish3/glassfish/domains/domain1/generated/jsp/!
I restart my glassfish server once a day. During that restart for every glasshfish application that is running on that server, loader_directories are created! I once knew why this folders are created(but now I dont)1 Anyway, this folder are not big but they do contain many files! It is save to delete them with no hesitation! Once I delete them, I check number of inodes!
server# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/rootvg-rootlv
612000 55750 556250 10% /
tmpfs 490591 1 490590 1% /dev/shm
/dev/sda1 25688 39 25649 1% /boot
/dev/mapper/rootvg-optlv
897600 18998 878602 3% /opt
/dev/mapper/rootvg-varlv
321280 2267 319013 1% /var
I restart glassfish server
server]# /etc/init.d/GlassFish_domain1 restart
and there were no more complaines about glassfish application!
Now, every few months I deleted there folders!
Users were complaining that glassfish application stoped working! I logged on server and check glassfish server.log!
server#cd /opt/glassfish3/glassfish/domains/domain1/logs
server# tail server.log
server.log_2013-03-05T11-03-08:[#|2013-01-03T12:16:01.092+0100|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=23;_ThreadName=Thread-2;|WARNING: 8 bytes remaining but no space left|#]
server.log_2013-03-05T11-03-08:[#|2013-01-03T12:16:01.092+0100|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=23;_ThreadName=Thread-2;|WARNING: 300 bytes remaining but no space left|#]
server.log_2013-03-05T11-03-08:[#|2013-01-03T12:16:01.092+0100|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=23;_ThreadName=Thread-2;|WARNING: 300 bytes remaining but no space left|#]
server.log_2013-03-05T11-03-08:[#|2013-01-03T12:16:01.094+0100|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=23;_ThreadName=Thread-2;|WARNING: 8 bytes remaining but no space left|#]
server.log_2013-03-05T11-03-08:[#|2013-01-03T12:16:01.094+0100|INFO|glassfish3.1.2|javax.enterprise.system.std.com.sun.enterprise.server.logging|_ThreadID=23;_ThreadName=Thread-2;|WARNING: 8 bytes remaining but no space left|#]
Ok, no space left on that partition! But when I checked disk free space, there was space left on all partitions!
server# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-rootlv
9.2G 1.8G 7.0G 21% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 97M 32M 60M 35% /boot
/dev/mapper/rootvg-optlv
14G 1.6G 12G 12% /opt
/dev/mapper/rootvg-varlv
4.9G 282M 4.4G 7% /var
So there was free space left, but from some reason glassfish could not write on file system. I checked if I can write anything on file system. It turns out that I can write on all file system except on file system where glasshfish was mounted!
Hm....
After some thinking I desided to check status of inodes on partitions!
server# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/rootvg-rootlv
612000 55749 556251 10% /
tmpfs 490591 1 490590 1% /dev/shm
/dev/sda1 25688 39 25649 1% /boot
/dev/mapper/rootvg-optlv
897600 0 897600 100% /opt
/dev/mapper/rootvg-varlv
321280 2267 319013 1% /var
Ok, so I found the problem! There were no free inodes left. How many inodes you have on partition depends on size of that partition and what file system you use!
What is inode?
" Inode is a data structure used to represent a filesystem object, which can be one of various things including a file or a directory. "
Bassicly, number of inodes on files system defines max number of files on directories together!
So, I had close to 900 000 files or folders on my /opt partition! Because file system considering space was not full, there was to be some folder with many small size files!
I found my problem in this folder /opt/glassfish3/glassfish/domains/domain1/generated/jsp/!
I restart my glassfish server once a day. During that restart for every glasshfish application that is running on that server, loader_directories are created! I once knew why this folders are created(but now I dont)1 Anyway, this folder are not big but they do contain many files! It is save to delete them with no hesitation! Once I delete them, I check number of inodes!
server# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/rootvg-rootlv
612000 55750 556250 10% /
tmpfs 490591 1 490590 1% /dev/shm
/dev/sda1 25688 39 25649 1% /boot
/dev/mapper/rootvg-optlv
897600 18998 878602 3% /opt
/dev/mapper/rootvg-varlv
321280 2267 319013 1% /var
I restart glassfish server
server]# /etc/init.d/GlassFish_domain1 restart
and there were no more complaines about glassfish application!
Now, every few months I deleted there folders!
No comments:
Post a Comment