Best Way to Troubleshooting Memory Leak in RedHat Linux Server

JockVSJock

I've got a RedHat Linux box that is a VM Guest (at this time, I don't know what the host is, I'm working on getting that info), and we are having alot of issues with running out of RAM at this time, and I think a memory leak. We have Oracle running on this box along with performing backups too. Sometimes a simple backup of 40 GB that should only take 1 to 2 hours can take 10 to 12 hours to complete. The other day we dropped a bunch of tables, indexes and views in the database and then rebuilt them for some maintenance and it took more time when what it should have.

My co-workers and I have setup a cron job to run every 3 hours that frees pagecache, dentries and inodes in cache memory


# sync; echo 3 > /proc/sys/vm/drop_caches

Got the info for this from this website: http://tecadmin.net/flush-memory-cache-on-linux-server/

This works, however it is a temporary band-aid, and we want a solution.

Along with monitoring free -m and top, I haven't had a chance to dig into any of the logs under /var/log to figure out where the culprit may lie.

We won't be able to install 3rd party software such as Valgrind to try and troubleshoot, because a request like that has to go thru multiple layers of management, which will take some time.

Is is a good idea to look thru various logs to figure out where the issue or any other recommendations on how to approach to fix the issue?

thanks

darkerosxx

If you can't use anything like valgrind, I don't see how you can isolate the issue to one process without limiting the system to run one process at a time while monitoring for memory leak in other ways. You could try running top, shift-f, and choose to sort by memory, and see what shows up there, but how would you know if that's actual memory or not? You can't. Go ahead and start the process to get valgrind. You should have something like that available on every system that is running any version of software that hasn't already been tested.

JockVSJock

darkerosxx wrote: »

If you can't use anything like valgrind, I don't see how you can isolate the issue to one process without limiting the system to run one process at a time while monitoring for memory leak in other ways. You could try running top, shift-f, and choose to sort by memory, and see what shows up there, but how would you know if that's actual memory or not? You can't. Go ahead and start the process to get valgrind. You should have something like that available on every system that is running any version of software that hasn't already been tested.

Thanks for the feedback.

Now that I think about it, Valgrind comes installed with RedHat Linux and we'll have to check it out.

Do you think it would even be worth checking out the VM host to eliminate all possibilities of issues?

ehnde

This might help: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/chap-Oracle_9i_and_10g_Tuning_Guide-Setting_Shared_Memory.html

Expect

The way I track memory leaks is very simple and requires no special programs.

run a periodic ps program/script, every few mins / hours depending on the memory leakage behavior, compare the outputs using a diff tool (like WinMerge) and that way you can identify the problematic process and perhaps it may be tweak-able.

Providing us with some more details on what is running on your system may help us help you, for example if it is a web server then apache can be tweaked to help in cases of over memory consumption.