7 unix commands for trouble-shooting memory related issue

Your application has been running fine for months until a recent linux patch are applied and you want to figure out what's going wrong.

These commands will be handy during your adventure of figuring out the root cause.

top

top command will give a general idea of how busy your system is and what resources are depleting. It will also list the top consumers of your computer cpu and memory. 

free -ht

free will list all the RAM usage statistics. -h is human readable, -t will print out total. The Total line is where you should find out the amount of free memory for your program. 

df -h

df -h let you check if you have disk full. -h is human readable.

uname -a

this command allow you to check the OS related information. From the output, you will find the kernel version, which may have bug related to performance issue.

lsof -nP | grep '(deleted)'

sometimes, you deleted a large file but your running application still have a reference to the file descriptor. The disk space won't be freed until the application exits. You can find these invisible files with command lsof -nP | grep '(deleted)'.

sar -r

when you want to find the memory usage in the past, sar command will do the job. The command sar have many flags for different resources like cpu, memory or IO, -r flag list the memory usage for today. 

ps -ef | grep java

find out if your java application is still running. To your surprise, it might just exit without any runtime exception in the log. The reason might be memory related.

When java program is running in VM that is configured to overcommit memory, the JVM will suffer from performance issue. A garbage collection which normally completes in hundreds of milliseconds might take minutes to complete in such an environment and the applications within that JVM are frozen during this time. 

“Java provides a special challenge in virtual environments due to the JVM’s introduction of a third level of memory management. The balloon driver draws memory from the virtual machine without impacting throughput because the guest OS efficiently claims pages that its processes are not using. But in the case of Java, the guest OS is unaware of how the JVM is using memory and is forced to select memory pages as arbitrarily and inefficiently as ESX’s swap routine. Neither ESX nor the guest OS can efficiently take memory from the JVM without significantly degrading performance. Memory in Java is managed inside the JVM and efforts by the host or guest to remove pages will both degrade Java applications’ performance. In these environments it is wise to manually set the JVM’s heap size and specify memory reservations for the virtual machine in ESX to account for the JVM, OS, and heap.“