I'm working on one of amazon's linux distros (4.4.11-23.53.amzn1.x86_64). (1gb ram)on that instance
I am running node.js with the forever module. (It makes sure to restart the node.js process if it crashes).
My experience is:
The web server works for about 2-3 hours then the process just disappears. Before running the webserver about 70% of the memory is taken and about 80% after.
I looked into every file in /var/log and found nothing relevant to the killed processes. And also logged all outputs from my node server and got nothing from there either.
I'm assuming linux killed my process because it ran out of memory but I'm not sure how to double check. And not sure why the memory use would increase when my web application memory usage does not increase over time.
1 Answer
To get an idea of why your process dies, make its parent process print the exit status. The exit status indicates whether the program due to a signal and includes an 8-bit exit code from the process. The 8-bit exit code is conventionally 0 for success and 1..125 to indicate an error.
In a C/Perl/… program, you can query the whole exit status. Consult the documentation of your language's library. From a shell script, the exit status including the signal information is packed into a single 8-bit value, with signals being reported as values 128+n where n is the signal number.
#!/bin/sh myprogram ret=$? if [ $ret -eq 0 ]; then echo "The program exited normally";; elif [ $ret -gt 128 ]; then echo "The program died of signal $((ret-128)): $(kill -l $ret)" else echo "The program failed with status $ret" fi If Linux kills your program because the system has run out of memory, this is recorded in the system logs, specifically in the kernel logs which can be displayed with the command dmesg and which are normally recorded in a file under /var/log (I don't know which file name Amazon's distribution uses).
Don't AssUMe that your process ran out of memory and was killed because of that. Check what happened. The most plausible explanation is that the program has a bug and crashed.
dmesgoutput and should be logged in one of the syslog files. stackoverflow.com/questions/624857/… If you don't see this then it's not an OOM issue.