1

We are debugging a situation where the cached/shared memory increase and increase until the system reach OOM-killer.

We have set shmax and shmall in sysctl.conf but without any visible effect. Do we need to enable something more for shmax/shmall to work? Or can some part of the system go beyond this limit, how hard is it enforced? Can buggy user space application or only bugs in kernel/drivers cause it? The application that we debug use graphics and video decoding. Can drivers go beyond the max limits?

kernel.shmmax = 2147483648 kernel.shmall = 524288

Linux kernel is 5.15.71(from Yocto meta-intel). Our system has 4GB ram and no swap (we tried to enable swap but it did not help with the stability of the system). We use Wayland/weston but not systemd. We set the value in sysctl.conf and reboot for it to take effect. We also confirmed the values with ipcs. We tried to set the shared memory to max 2 GB.

ipcs -l ------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 2097152 max total shared memory (kbytes) = 2097152 min seg size (bytes) = 1

Here is some example output from free, meminfo, smem, etc a few minutes before it reaches OOM.

free -w total used free shared buffers cache available Mem: 3844036 479428 263444 2711864 11324 3089840 585716 Swap: 0 0 0 #### cat /proc/meminfo MemTotal: 3844036 kB MemFree: 262680 kB MemAvailable: 584940 kB Buffers: 11324 kB Cached: 3055620 kB SwapCached: 0 kB Active: 98764 kB Inactive: 645792 kB Active(anon): 732 kB Inactive(anon): 394288 kB Active(file): 98032 kB Inactive(file): 251504 kB Unevictable: 2708620 kB Mlocked: 100 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 12 kB Writeback: 0 kB AnonPages: 386388 kB Mapped: 162732 kB Shmem: 2711864 kB KReclaimable: 34208 kB Slab: 68656 kB SReclaimable: 34208 kB SUnreclaim: 34448 kB KernelStack: 4640 kB PageTables: 5904 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 1922016 kB Committed_AS: 4068728 kB VmallocTotal: 34359738367 kB VmallocUsed: 15104 kB VmallocChunk: 0 kB Percpu: 1040 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 72236 kB DirectMap2M: 3938304 kB DirectMap1G: 2097152 kB #### smem PID User Command Swap USS PSS RSS ….. 1306 weston /usr/libexec/wpe-webkit-1.1 0 27192 51419 98928 1379 weston /usr/libexec/wpe-webkit-1.1 0 190268 214958 266040 Area Used Cache Noncache firmware/hardware 0 0 0 kernel image 0 0 0 kernel dynamic memory 3030848 2938432 92416 userspace memory 555656 162732 392924 free memory 257532 257532 0 Map PIDs AVGPSS PSS …… /usr/lib/libcrypto.so.3 20 527 10544 /usr/lib/dri/iris_dri.so 5 2196 10982 /usr/lib/dri/iHD_drv_video.so 1 20356 20356 /usr/lib/libWPEWebKit-1.1.so.0.2.6 5 14539 72697 [heap] 45 2060 92700 <anonymous> 45 5970 268688

Edit: Added df info for tmpfs. The tmpfs mounts showed with df does not show any extra ordinary size increase.

/# df -h Filesystem Size Used Avail Use% Mounted on /dev/root 9.8G 1.9G 7.4G 21% / devtmpfs 1.9G 2.1M 1.9G 1% /dev tmpfs 1.9G 636K 1.9G 1% /run tmpfs 751M 5.8M 745M 1% /var/volatile tmpfs 40K 0 40K 0% /mnt/.psplash
Improve this question
2
  • SwapTotal: 0 kB? If you need reliability, disable memory overcommit, disable the out-of-fuel, err, OOM killer, and provide your system with enough swap so it can run reliably. Commented Feb 20, 2023 at 16:29
  • I tested to add 16GB of swap, it delayed the crash with a few hours but did not make the system stable. But I have trouble see what is using now close to 20GB of memory...Things seems to be cleared/swapped. Here are some values when it was 2-3GB left of swap: MemTotal: 3844024 kB MemFree: 108624 kB MemAvailable: 173040 kB Cached: 3356196 kB SwapCached: 30340 kB Active: 339312 kB Inactive: 3124336 kB Active(anon): 294284 kB Inactive(anon): 3121236 kB SwapTotal: 16777212 kB SwapFree: 2978812 kB Shmem: 3303908 kB Commented Feb 21, 2023 at 15:20

1 Answer 1

Reset to default
0

SHMAX and SHMALL won't constraint the size of your miscellaneous tmpfs.

Since tmpfs lives completely in the page cache and on swap, all tmpfs pages will be shown as “Shmem” in /proc/meminfo and “Shared” in free(1).

Check with the df utility how many of these filesystem are actually mounted on your system and eventually limit their maximum possible size thanks to the size= parameter of corresponding mount operation.

Of course if your application uses this kind of filesystem and no swap space is made available, the application might well block or stop processing because it won't find any space left on device.

Improve this answer
5
  • We looked into tmpfs and could not see any increase in usage. The size is large but usage is always very small. Is tmpfs the only shared/cached memory not affected by shmax/shmall? Commented Feb 21, 2023 at 10:35
  • Yes indeed : GEM buffers. cf accepted answer as part of unix.stackexchange.com/questions/482795/… Commented Feb 21, 2023 at 11:48
  • Without swap, we see that gem objects increase until OOM. With swap, just before OOM it clear/swap out. i915_gem_objects in debugfs goes down alot at that point. But then something else eat up all swap at it still reach OOM. I don't see anything about limiting gem in that answer thread. Commented Feb 21, 2023 at 15:36
  • Oh come on ! Your question was : "What shared memory is not controlled by SHMAX/SHMALL". Then I answer : tmpfs and GEM buffers. I just gave the link in order to justify my statement regarding GEM buffers. If you are now looking for a way to limit GEM buffers then it is IMHO another question. Commented Feb 21, 2023 at 15:45
  • I think what I wanted to know was a complete (or near complete) list of things that was outside of shamax/shmall control. And if so how to control them. tmpfs seems to be easy to set a max limit on, but gem seems to be able to do what ever it wants... But I will take this as a learning input on writing clear questions and accept this answer. Commented Feb 22, 2023 at 15:13

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.