2

I'm developing a compute cluster and I'm trying to determine the maximum amount of RAM I can give to a single process. On a machine with 16GB of ram, the answer is that I can allocate (and touch) 15680MB before the out of memory killer is invoked (overcommit_memory=2, overcommit_ratio=0).

That might seem halfway reasonable on a regular linux distro, but I've compiled my own minimal kernel (v 3.16), and my test app is running as sbin/init - IE there is absolutely nothing else in memory.

If I simulate a machine with just 512MB of ram for comparison, I can allocate 468MB, an overhead of just 44MB, which seems very reasonable.

So why is there so much more kernel memory on the 16gb machine?

Reading the kernel docs it would seem that some percent of memory is reserved for the kernel for "safety", which can be tuned by /proc/sys/vm/. But I've found that changing admin_reserve_kbytes and user_reserve_kbytes makes absolutely no difference in when the OOM killer is invoked.

Is there something else that grows proportionally to the amount of RAM Installed?

Improve this question
3
  • 1
    The kernel has many internal data structures which grow depending on the amount of available RAM. Commented Apr 28, 2015 at 4:11
  • Can you share /proc/meminfo taken just before the OOM killer is invoked? Commented Apr 29, 2015 at 17:13
  • Here it is on the machine with 3.68GB. Seems stackexchange doesn't allow image sharing, so here' s a link: postimg.org/image/s0ri6pphz Commented May 1, 2015 at 16:23

3 Answers 3

Reset to default
4

A fairly old (2006) paper discusses some of the kernel overhead that scales with memory size: http://halobates.de/memorywaste.pdf

From it:

page struct (keeps track of page allocation): 1.37%
page tables: 0.5%

Meanwhile, my 16GB test machine shows a total overhead of 4.3%

so around 2% is still unaccounted for. Thus, the question is still not completely answered, and the answer is based on a very old kernel. But this is the closest by far that I've come to answering the question.

Improve this answer
1
  • That was an interesting question, that's too bad. I don't see why anything else than memory paging would have to scale with how much physical memory the OS has... Commented Mar 16, 2018 at 15:54
3

The short answer to your question is as much as it possibly can, once it gives processes what they can use. The alternative is to leave the memory free which is wasteful. A machine with 16GB can't use 12GB today so it can use 20GB tomorrow. Any memory not used right this second is potential to save I/O and other effort that's forever lost -- you can't save memory for later.

If you're thinking, "but I need my memory free now so I can use it later", purge such nonsense from your mind. You can use it now and use it later. There's no tradeoff to make here. In fact, if you're using it now, you can use it later just by not doing anything. If you're not using it now, you have to do work to use it later. So using it now makes it more likely you'll use it later.

You seem to have a misconception that memory can only be used by a process or it's kernel overhead. That's not true. On a typical system, the vast majority of physical memory is neither free, used by processes, nor consumed by kernel overhead.

For example, suppose your system has plenty of free memory, and you run top, and then quit out. Your system has two choices. It can keep the top program in RAM or it can discard the pages that contained it. Let's look at both choices:

If it makes the RAM free, that takes effort. And worse, it's effort that has to be undone in order to use the RAM. And if you run top again, the program will have to be loaded from disk, which is very slow.

If it keeps the pages in RAM, and you run top again, it doesn't have to load the problem from disk. That's a huge win. And if the memory is needed for some other purpose, it can transition directly to that purpose without having to go to the effort of making the memory used and removing it from the free pool.

So the pages of RAM will stay in use holding the executable. This use is not associated with any process, since top isn't running. But it's not free, since it contains data that may be useful. And it's not overhead -- should the memory be more useful doing something else, the contents can just be discarded since the executable can be paged from disk if needed.

And, in fact, on a typical system under typical load, the vast majority of memory contains information that might be useful in the future but can be discarded if memory is needed for some other purpose.

If you're looking at /proc/meminfo, the number you should probably be looking at is MemAvailable. That combines both free memory and memory that contains information that can be trivially discarded should more free memory be needed.

Improve this answer
7
  • 2
    That's true, but not relevant here: the question is about the total memory available for userland. Your deleted answer is actually more relevant. Commented Apr 28, 2015 at 21:37
  • Agreed, David new post totally misses the point of the question. The question is how much total RAM can single process use (remember, this is sbin/init here, so nothing else but the kernel is around). The ram is not held around for the future, but used immediately in our machine learning task. Commented Apr 28, 2015 at 23:43
  • If the question is about how much RAM is is available for userland, why did you respond with information about how much RAM is free? I'm totally baffled now. Commented Apr 29, 2015 at 0:03
  • I included the "free" line because it seems to suggest that the kernel is keeping back some amount of memory for unspecified purposes, which it won't let user-land have. A reasonable alternative conclusion is that the free calculation is a guesstimate, and is just plain wrong. But, since I've touched all my pages, it seems to suggest the estimate is based on untouched pages in the kernel. So far, nobody has really addressed the question what's taking up all that space, if it indeed is really used: it's not the page tables, see deleted comment. Commented Apr 29, 2015 at 5:04
  • To be clear David, your current "answer" is very clear and reasonable, except is has nothing to do with my question. I'm sorry if my question was unclear. Commented Apr 29, 2015 at 5:09
0

The answer depends on how much memory your server has available to it when the kernel starts. When using memory ballooning, this is based on the maximum it can balloon to.

For linux Kernel 4.15:

  • With 4GB of RAM, it will reserve 4.8%
  • 8GB of RAM: 3.2%
  • 16GB of RAM: 2.3%
  • 32GB of RAM: 1.9%
  • 64GB of RAM: 1.7%

So why is there so much more kernel memory on the 16gb machine?

Relatively there isn't. Assuming your number was right, you're looking at 8.6% reserved for 512MB, and just 2.3% for 16GB.

Why these proportions are chosen in the kernel isn't something I know off-hand, but you can see the break-down in dmesg:

[ 0.000000] Memory: 65920612K/67108324K available (12300K kernel code, 2473K rwdata, 4272K rodata, 2408K init, 2416K bss, 1187712K reserved, 0K cma-reserved)
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.