I have an Ubuntu 20.04.4 server with 32GB RAM. The server is running a bunch of LXD containers and two VMs (libvirt+qemu+kvm).
After startup, with all services running, the RAM utilization is about ~12GB.
After 3-4 weeks the RAM utilization reaches ~90%.
If I stop all containers and VMs the utilization is still ~20GB.
However, I cannot figure out who is claiming this memory.
I have already tried clearing the cache, but that doesn't change much.
I compiled the kernel with support for kmemleak but it did not detect anything useful but shows up in slabtop.
systemd-cgtop:
/ 593 - 23.7G - - machine.slice - - 1.4G - - system.slice 116 - 301.1M - - user.slice 11 - 141.9M - - user.slice/user-1000.slice 11 - 121.6M - - system.slice/systemd-journald.service 1 - 83.8M - - user.slice/user-1000.slice/session-297429.scope 5 - 81.0M - - system.slice/libvirtd.service 22 - 46.2M - - user.slice/user-1000.slice/[email protected] 6 - 39.8M - - system.slice/snapd.service 36 - 19.8M - - system.slice/cron.service 1 - 19.3M - - init.scope 1 - 14.0M - - system.slice/systemd-udevd.service 1 - 13.2M - - system.slice/multipathd.service 7 - 10.8M - - system.slice/NetworkManager.service 3 - 5.8M - - system.slice/networkd-dispatcher.service 1 - 5.4M - - system.slice/ssh.service 1 - 5.0M - - system.slice/ModemManager.service 3 - 4.5M - - system.slice/systemd-networkd.service 1 - 3.5M - - system.slice/accounts-daemon.service 3 - 3.5M - - system.slice/udisks2.service 5 - 3.4M - - system.slice/polkit.service 3 - 3.0M - - system.slice/rsyslog.service 4 - 2.8M - - system.slice/systemd-resolved.service 1 - 2.4M - - system.slice/unattended-upgrades.service 2 - 1.8M - - system.slice/dbus.service 1 - 1.8M - - system.slice/systemd-logind.service 1 - 1.7M - - system.slice/smartmontools.service 1 - 1.5M - - system.slice/systemd-machined.service 1 - 1.5M - - system.slice/systemd-timesyncd.service 2 - 1.4M - - system.slice/virtlogd.service 1 - 1.3M - - system.slice/rtkit-daemon.service 3 - 1.2M - - /proc/meminfo:
MemTotal: 32718604 kB MemFree: 11480728 kB MemAvailable: 11612788 kB Buffers: 28 kB Cached: 144512 kB SwapCached: 855404 kB Active: 520504 kB Inactive: 541588 kB Active(anon): 441708 kB Inactive(anon): 484240 kB Active(file): 78796 kB Inactive(file): 57348 kB Unevictable: 18664 kB Mlocked: 18664 kB SwapTotal: 33043136 kB SwapFree: 32031680 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 94680 kB Mapped: 126592 kB Shmem: 660 kB KReclaimable: 432484 kB Slab: 10784740 kB SReclaimable: 432484 kB SUnreclaim: 10352256 kB KernelStack: 10512 kB PageTables: 5052 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 49402436 kB Committed_AS: 1816364 kB VmallocTotal: 34359738367 kB VmallocUsed: 152512 kB VmallocChunk: 0 kB Percpu: 8868864 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 19383100 kB DirectMap2M: 14053376 kB DirectMap1G: 0 kB slabtop:
Active / Total Objects (% used) : 30513607 / 33423869 (91.3%) Active / Total Slabs (% used) : 1384092 / 1384092 (100.0%) Active / Total Caches (% used) : 123 / 203 (60.6%) Active / Total Size (% used) : 9965969.20K / 10757454.91K (92.6%) Minimum / Average / Maximum Object : 0.01K / 0.32K / 16.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 27156909 26001970 95% 0.30K 1194104 26 9552832K kmemleak_object 754624 742232 98% 0.06K 11791 64 47164K kmalloc-64 654675 278378 42% 0.57K 23382 28 374112K radix_tree_node 593436 348958 58% 0.08K 11636 51 46544K Acpi-State 559744 418325 74% 0.03K 4373 128 17492K kmalloc-32 496320 483104 97% 0.12K 15510 32 62040K kernfs_node_cache 487104 155952 32% 0.06K 7611 64 30444K vmap_area 394240 165965 42% 0.14K 14080 28 56320K btrfs_extent_map 355580 342674 96% 0.09K 7730 46 30920K trace_event_file 339573 338310 99% 4.00K 42465 8 1358880K kmalloc-4k 306348 154794 50% 0.19K 7410 42 59280K dentry 145931 104400 71% 1.13K 11552 28 369664K btrfs_inode 137728 137174 99% 0.02K 538 256 2152K kmalloc-16 112672 74034 65% 0.50K 3671 32 58736K kmalloc-512 102479 62366 60% 0.30K 4093 26 32744K btrfs_delayed_node 68880 66890 97% 2.00K 4305 16 137760K kmalloc-2k 66656 48345 72% 0.25K 2083 32 16664K kmalloc-256 64110 47818 74% 0.59K 2376 27 38016K inode_cache 50176 50176 100% 0.01K 98 512 392K kmalloc-8 44710 43744 97% 0.02K 263 170 1052K lsm_file_cache 43056 11444 26% 0.25K 1418 32 11344K pool_workqueue 36480 29052 79% 0.06K 570 64 2280K kmalloc-rcl-64 33920 25846 76% 0.06K 530 64 2120K anon_vma_chain 24822 14264 57% 0.19K 832 42 6656K kmalloc-192 23552 23552 100% 0.03K 184 128 736K fsnotify_mark_connector 23517 17994 76% 0.20K 603 39 4824K vm_area_struct 19572 14909 76% 0.09K 466 42 1864K kmalloc-rcl-96 18262 15960 87% 0.09K 397 46 1588K anon_vma 14548 12905 88% 1.00K 459 32 14688K kmalloc-1k 14162 14162 100% 0.05K 194 73 776K file_lock_ctx 13104 12141 92% 0.09K 312 42 1248K kmalloc-96 13062 13062 100% 0.19K 311 42 2488K cred_jar 13056 10983 84% 0.12K 408 32 1632K kmalloc-128 12192 8922 73% 0.66K 508 24 8128K proc_inode_cache 11730 11444 97% 0.69K 1444 46 46208K squashfs_inode_cache 11067 11067 100% 0.08K 217 51 868K task_delay_info 10752 10752 100% 0.03K 84 128 336K kmemleak_scan_area 10656 8666 81% 0.25K 333 32 2664K filp 10252 10252 100% 0.18K 235 44 1880K kvm_mmu_page_header 10200 10200 100% 0.05K 120 85 480K ftrace_event_field 10176 10176 100% 0.12K 318 32 1272K pid 9906 9906 100% 0.10K 254 39 1016K Acpi-ParseExt 9600 9213 95% 0.12K 300 32 1200K kmalloc-rcl-128 9520 9520 100% 0.07K 170 56 680K Acpi-Operand 8502 8063 94% 0.81K 218 39 6976K sock_inode_cache 7733 7733 100% 0.70K 169 46 5408K shmem_inode_cache 7392 7231 97% 0.19K 176 42 1408K skbuff_ext_cache 6552 6552 100% 0.19K 163 42 1304K kmalloc-rcl-192 6480 6480 100% 0.11K 180 36 720K khugepaged_mm_slot 6144 6144 100% 0.02K 24 256 96K ep_head 5439 5439 100% 0.42K 147 37 2352K btrfs_ordered_extent 5248 4981 94% 0.25K 164 32 1312K skbuff_head_cache 4792 4117 85% 4.00K 606 8 19392K biovec-max 4326 4326 100% 0.19K 103 42 824K proc_dir_entry 4125 4125 100% 0.24K 125 33 1000K tw_sock_TCPv6 3978 3978 100% 0.10K 102 39 408K buffer_head 3975 3769 94% 0.31K 159 25 1272K mnt_cache 3328 3200 96% 1.00K 104 32 3328K RAW 3136 3136 100% 1.12K 112 28 3584K signal_cache 3072 2560 83% 0.03K 24 128 96K dnotify_struct 2910 2820 96% 1.06K 97 30 3104K UNIX 2522 2396 95% 1.19K 97 26 3104K RAWv6 2448 2448 100% 0.04K 24 102 96K pde_opener 2400 2400 100% 0.50K 75 32 1200K skbuff_fclone_cache 2112 2080 98% 1.00K 66 32 2112K biovec-64 1695 1587 93% 2.06K 113 15 3616K sighand_cache 1518 1518 100% 0.69K 33 46 1056K files_cache 1500 1500 100% 0.31K 60 25 480K nf_conntrack 1260 894 70% 6.06K 252 5 8064K task_struct 1260 1260 100% 1.06K 42 30 1344K mm_struct 1222 1158 94% 2.38K 94 13 3008K TCPv6 1150 1150 100% 0.34K 25 46 400K taskstats 924 924 100% 0.56K 33 28 528K task_group 888 888 100% 0.21K 24 37 192K file_lock_cache 864 864 100% 0.11K 24 36 96K btrfs_trans_handle 855 855 100% 2.19K 62 14 1984K TCP 851 851 100% 0.42K 23 37 368K uts_namespace 816 816 100% 0.12K 24 34 96K seq_file 816 816 100% 0.04K 8 102 32K ext4_extent_status 792 792 100% 0.24K 24 33 192K tw_sock_TCP 782 782 100% 0.94K 23 34 736K mqueue_inode_cache 720 720 100% 0.13K 24 30 96K pid_namespace 704 704 100% 0.06K 11 64 44K kmem_cache_node 648 648 100% 1.16K 24 27 768K perf_event 640 640 100% 0.12K 20 32 80K scsi_sense_cache 624 624 100% 0.30K 24 26 192K request_sock_TCP 624 624 100% 0.15K 24 26 96K fuse_request 596 566 94% 8.00K 149 4 4768K kmalloc-8k 576 576 100% 1.31K 24 24 768K UDPv6 494 494 100% 0.30K 19 26 152K request_sock_TCPv6 480 480 100% 0.53K 16 30 256K user_namespace 432 432 100% 1.15K 16 27 512K ext4_inode_cache 416 416 100% 0.25K 13 32 104K kmem_cache 416 416 100% 0.61K 16 26 256K hugetlbfs_inode_cache 390 390 100% 0.81K 10 39 320K fuse_inode 306 306 100% 0.04K 3 102 12K bio_crypt_ctx 292 292 100% 0.05K 4 73 16K mbcache 260 260 100% 1.56K 13 20 416K bdev_cache 256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s 232 232 100% 4.00K 29 8 928K names_cache 192 192 100% 1.98K 12 16 384K request_queue 170 170 100% 0.02K 1 170 4K mod_hash_entries 168 168 100% 4.12K 24 7 768K net_namespace 155 155 100% 0.26K 5 31 40K numa_policy 132 132 100% 0.72K 3 44 96K fat_inode_cache 128 128 100% 0.25K 4 32 32K dquot 128 128 100% 0.06K 2 64 8K ext4_io_end 108 108 100% 2.61K 9 12 288K x86_emulator 84 84 100% 0.19K 2 42 16K ext4_groupinfo_4k 68 68 100% 0.12K 2 34 8K jbd2_journal_head 68 68 100% 0.12K 2 34 8K abd_t 64 64 100% 8.00K 16 4 512K irq_remap_cache 64 64 100% 2.00K 4 16 128K biovec-128 63 63 100% 4.06K 9 7 288K x86_fpu 56 56 100% 0.07K 1 56 4K fsnotify_mark 56 56 100% 0.14K 2 28 8K ext4_allocation_context 42 42 100% 0.75K 1 42 32K dax_cache 40 40 100% 0.20K 1 40 8K ip4-frags 36 36 100% 7.86K 9 4 288K kvm_vcpu 30 30 100% 1.06K 1 30 32K dmaengine-unmap-128 24 24 100% 0.66K 1 24 16K ovl_inode 15 15 100% 2.06K 1 15 32K dmaengine-unmap-256 6 6 100% 16.00K 3 2 96K zio_buf_comb_16384 0 0 0% 0.01K 0 512 0K kmalloc-rcl-8 0 0 0% 0.02K 0 256 0K kmalloc-rcl-16 0 0 0% 0.03K 0 128 0K kmalloc-rcl-32 0 0 0% 0.25K 0 32 0K kmalloc-rcl-256 0 0 0% 0.50K 0 32 0K kmalloc-rcl-512 0 0 0% 1.00K 0 32 0K kmalloc-rcl-1k 0 0 0% 2.00K 0 16 0K kmalloc-rcl-2k 0 0 0% 4.00K 0 8 0K kmalloc-rcl-4k 0 0 0% 8.00K 0 4 0K kmalloc-rcl-8k 0 0 0% 0.09K 0 42 0K dma-kmalloc-96 0 0 0% 0.19K 0 42 0K dma-kmalloc-192 0 0 0% 0.01K 0 512 0K dma-kmalloc-8 0 0 0% 0.02K 0 256 0K dma-kmalloc-16 0 0 0% 0.03K 0 128 0K dma-kmalloc-32 0 0 0% 0.06K 0 64 0K dma-kmalloc-64 0 0 0% 0.12K 0 32 0K dma-kmalloc-128 0 0 0% 0.25K 0 32 0K dma-kmalloc-256 0 0 0% 0.50K 0 32 0K dma-kmalloc-512 0 0 0% 1.00K 0 32 0K dma-kmalloc-1k 0 0 0% 2.00K 0 16 0K dma-kmalloc-2k 0 0 0% 4.00K 0 8 0K dma-kmalloc-4k 0 0 0% 8.00K 0 4 0K dma-kmalloc-8k 0 0 0% 0.12K 0 34 0K iint_cache 0 0 0% 1.00K 0 32 0K PING 0 0 0% 0.75K 0 42 0K xfrm_state 0 0 0% 0.37K 0 43 0K request_sock_subflow 0 0 0% 1.81K 0 17 0K MPTCP 0 0 0% 0.62K 0 25 0K dio 0 0 0% 0.19K 0 42 0K userfaultfd_ctx_cache 0 0 0% 0.03K 0 128 0K ext4_pending_reservation 0 0 0% 0.08K 0 51 0K ext4_fc_dentry_update 0 0 0% 0.04K 0 102 0K fat_cache 0 0 0% 0.81K 0 39 0K ecryptfs_auth_tok_list_item 0 0 0% 0.02K 0 256 0K ecryptfs_file_cache 0 0 0% 0.94K 0 34 0K ecryptfs_inode_cache 0 0 0% 2.82K 0 11 0K dm_uevent 0 0 0% 3.23K 0 9 0K kcopyd_job 0 0 0% 1.19K 0 26 0K PINGv6 0 0 0% 0.18K 0 44 0K ip6-frags 0 0 0% 2.00K 0 16 0K MPTCPv6 0 0 0% 0.13K 0 30 0K fscrypt_info 0 0 0% 0.25K 0 32 0K fsverity_info 0 0 0% 1.25K 0 25 0K AF_VSOCK 0 0 0% 0.19K 0 42 0K kcf_sreq_cache 0 0 0% 0.50K 0 32 0K kcf_areq_cache 0 0 0% 0.19K 0 42 0K kcf_context_cache 0 0 0% 4.00K 0 8 0K zfs_btree_leaf_cache 0 0 0% 0.44K 0 36 0K ddt_entry_cache 0 0 0% 1.22K 0 26 0K zio_cache 0 0 0% 0.05K 0 85 0K zio_link_cache 0 0 0% 0.50K 0 32 0K zio_buf_comb_512 0 0 0% 1.00K 0 32 0K zio_buf_comb_1024 0 0 0% 1.50K 0 21 0K zio_buf_comb_1536 0 0 0% 2.00K 0 16 0K zio_buf_comb_2048 0 0 0% 2.50K 0 12 0K zio_buf_comb_2560 0 0 0% 3.00K 0 10 0K zio_buf_comb_3072 0 0 0% 3.50K 0 9 0K zio_buf_comb_3584 0 0 0% 4.00K 0 8 0K zio_buf_comb_4096 0 0 0% 8.00K 0 4 0K zio_buf_comb_5120 0 0 0% 8.00K 0 4 0K zio_buf_comb_6144 0 0 0% 8.00K 0 4 0K zio_buf_comb_7168 0 0 0% 8.00K 0 4 0K zio_buf_comb_8192 0 0 0% 12.00K 0 2 0K zio_buf_comb_10240 0 0 0% 12.00K 0 2 0K zio_buf_comb_12288 0 0 0% 16.00K 0 2 0K zio_buf_comb_14336 0 0 0% 16.00K 0 2 0K lz4_cache 0 0 0% 0.24K 0 33 0K sa_cache 0 0 0% 0.96K 0 33 0K dnode_t 0 0 0% 0.32K 0 24 0K arc_buf_hdr_t_full 0 0 0% 0.38K 0 41 0K arc_buf_hdr_t_full_crypt 0 0 0% 0.09K 0 42 0K arc_buf_hdr_t_l2only 0 0 0% 0.08K 0 51 0K arc_buf_t 0 0 0% 0.38K 0 42 0K dmu_buf_impl_t 0 0 0% 0.37K 0 43 0K zil_lwb_cache 0 0 0% 0.15K 0 26 0K zil_zcw_cache 0 0 0% 0.13K 0 30 0K sio_cache_0 0 0 0% 0.15K 0 26 0K sio_cache_1 0 0 0% 0.16K 0 24 0K sio_cache_2 0 0 0% 1.06K 0 30 0K zfs_znode_cache 0 0 0% 0.09K 0 46 0K zfs_znode_hold_cache 
cat /sys/kernel/debug/kmemleak? And before you enabledkmemleakwith your kernel, what showed up as the top slab?/proc/meminfoafter you enabledmemleak, and there we saw that that Slab was taking most of the memory, and the top hitter in slab waskmemleak_object. I would like to see the content of/proc/meminfowithout enablingmemleak.