Increased RAM usage with kernel 6.7

Description:

There's a sudden increase in RAM usage with kernel 6.7, which sounds like a bug to me. Below is the output of the free -h command when starting my Xfce session with the linux 6.7.arch3-1 and linux-lts 6.6.12-1 kernels (which I'm now using because of this problem):

# linux 6.7.arch3-1
               total        used        free      shared  buff/cache   available
Mem:           3,4Gi       1,1Gi       2,1Gi       3,0Mi       541Mi       2,4Gi

# linux-lts 6.6.12-1
               total        used        free      shared  buff/cache   available
Mem:           3,4Gi       762Mi       2,4Gi       3,1Mi       503Mi       2,7Gi

The difference is "only" 300Mi at startup, but then it gets worse over the course of the session, probably reaching 500-600Mi (at least), so that a small machine like this, still usable with the lts kernel and Xfce, starts to become really unusable with the non-lts kernel (basically, it's no longer possible to open even slightly imposing software like Thunderbird in addition to Firefox without exceeding 90% RAM usage).

Additional info:

• package version(s): 6.7.arch3-1
• config and/or log files:
• link to upstream bug report, if any:

Steps to reproduce:

• Upgrade from linux 6.6.10.arch1-1 to 6.7.arch3-1 (or switch between linux and linux-lts) and compare RAM usage

added priority3-normal severity4-low statusunconfirmed labels

assigned to @anthraxx, @morganamilo, @toolybird, and @thurstylark

changed the description

which sounds like a bug to me

Maybe.. but there's no firm indication of an Arch packaging bug i.e. you would need to report it upstream to the kernel folks (see below). But one possibility which seems a bit suss in the Arch config between 6.6 and 6.7 is the enabling of ZRAM_MEMORY_TRACKING. A recompile with that option turned off is an easy test. Anyway, please let us know what you find out.

FWIW, I can replicate your test results. Although, for testing I typically run 2GB VM's and they are still as smooth as butter.. even with tons of apps loaded. BTW, not sure if free is the best way to measure memory usage on Linux.. which is often misunderstood. There always seems to be threads about memory usage in the forum. The latest one is here.

• Standard Arch advice for debugging kernel regressions
• git bisect to identify the offending upstream commit
• How to report kernel bugs

added statuswaiting-input label and removed statusunconfirmed label

assigned to @heftig, @dvzrv, and @tpowa and unassigned @morganamilo, @toolybird, and @thurstylark

Any obvious difference in the output of slabtop -osc?

The only thing jumping out for me...but is it relevant? Dunno

$ grep Acpi-State *slabtop.txt
6.6.13-slabtop.txt:  1581   1329  84%    0.08K     31       51       124K Acpi-State             
6.7.0-slabtop.txt: 93228  90214  96%    0.08K   1828       51      7312K Acpi-State             
6.7.1-slabtop.txt:104346 100356  96%    0.08K   2046       51      8184K Acpi-State

I'll look at it later and report back.

No, I don't think there's anything noteworthy, here are the first ten lines in both cases:

$ cat slabtop-linux 
 Actif / Total Objets (% utilisé)  : 292361 / 293977 (99,5%)
 Actif / Total Slabs (% utilisé)   : 9247 / 9247 (100,0%)
 Actif / Total Caches (% utilisé)  : 124 / 175 (70,9%)
 Actif / Taille Totale (% utilisé) : 61969,51K / 62527,55K (99,1%)
 Minimum / Moyenne / Maximum (Objet): 0,01K / 0,21K / 8,00K

  OBJS ACTIF  UTIL OBJ DIM   SLABS OBJ/SLAB CACHE DIM  NOM                    
  9126   9126 100%    1,16K    702       13     11232K ext4_inode_cache       
 24192  24192 100%    0,19K   1152       21      4608K dentry                 
  6708   6708 100%    0,63K    559       12      4472K inode_cache            
 31200  31200 100%    0,12K    975       32      3900K kernfs_node_cache      
 20055  20055 100%    0,19K    955       21      3820K cred_jar               
  6510   6510 100%    0,57K    465       14      3720K radix_tree_node        
  5824   5824 100%    0,50K    364       16      2912K kmalloc-512            
 26715  26715 100%    0,10K    685       39      2740K buffer_head            
   340    328  96%    7,81K     85        4      2720K task_struct            
   472    472 100%    4,00K     59        8      1888K kmalloc-4k             

$ cat slabtop-linux-lts 
 Actif / Total Objets (% utilisé)  : 304477 / 305962 (99,5%)
 Actif / Total Slabs (% utilisé)   : 9044 / 9044 (100,0%)
 Actif / Total Caches (% utilisé)  : 123 / 174 (70,7%)
 Actif / Taille Totale (% utilisé) : 57447,67K / 57988,37K (99,1%)
 Minimum / Moyenne / Maximum (Objet): 0,01K / 0,19K / 8,00K

  OBJS ACTIF  UTIL OBJ DIM   SLABS OBJ/SLAB CACHE DIM  NOM                    
  4927   4927 100%    1,16K    379       13      6064K ext4_inode_cache       
  6684   6684 100%    0,63K    557       12      4456K inode_cache            
 31008  31008 100%    0,12K    969       32      3876K kernfs_node_cache      
 20160  20160 100%    0,19K    960       21      3840K cred_jar               
 19887  19887 100%    0,19K    947       21      3788K dentry                 
  6496   6496 100%    0,57K    464       14      3712K radix_tree_node        
  5776   5776 100%    0,50K    361       16      2888K kmalloc-512            
   336    332  98%    7,81K     84        4      2688K task_struct            
 25506  25506 100%    0,10K    654       39      2616K buffer_head            
 18080  17724  98%    0,12K    565       32      2260K kmalloc-128            

Assuming it is the same issue https://bbs.archlinux.org/viewtopic.php?pid=2145691#p2145691 found the causal commit to be efa7df3e mm: align larger anonymous mappings on THP boundaries

But one possibility which seems a bit suss in the Arch config between 6.6 and 6.7 is the enabling of ZRAM_MEMORY_TRACKING. A recompile with that option turned off is an easy test. Anyway, please let us know what you find out.

So, since I had started this test, I let it finish, even though the tests already done in the forum thread seemed to make it obsolete. After 36 hours of compilation (!), I was finally able to test the resulting kernel, and I confirm that it doesn't change anything. Needless to say, I won't be able to do any more tests of this nature on my own, i.e. when it comes to building the kernel.

I have also tested the kernels proposed in https://bbs.archlinux.org/viewtopic.php?pid=2145699#p2145699, and confirm that the first kernel (with the offending commit reverted) fixes the problem, while the second (with the patch supposed to fix the problem, I believe) does not.

I'd also like to point out that using kernel option transparent_hugepage=never seems to cancel the effect of the offending commit, i.e. I get essentially the same memory usage at session startup whatever the kernel version tested if this option is enabled (from linux-lts 6.6.13-1 to linux 6.7.arch3-1 through the various intermediate builds):

               total        used        free      shared  buff/cache   available
Mem:           3.4Gi       479Mi       2.5Gi       2.2Mi       510Mi       3.0Gi

NB1: Maybe free isn't the best way to measure memory usage, but since I started with it, I'll continue.

NB2: All this only concerns memory usage at startup, I suppose that in all rigor we should also check that memory usage then increases normally during the session.

Looking more closely at the offending commit message, and reading a bit of the transparent_hugepage doc, I understand better why setting this option to never solves the problem and I'd be tempted to believe that this isn't a bug but, so to speak, expected behavior, or at any rate an expected side effect considered acceptable.

So I'm not going to launch into an upstream bug report, which sounds complicated and impractical, only to be told that it's a feature. I'll leave you to close this issue in case you'd like to keep it open instead.

FWIW I would try transparent_hugepage=madvise instead of never. That will let apps that want to use THP actually use it.

Why not, memory usage at startup seems to be closer to what it was before:

               total        used        free      shared  buff/cache   available
Mem:           3,4Gi       673Mi       2,5Gi       2,2Mi       509Mi       2,8Gi

I'm going to test it a bit over time to see if it stays correct during a session. In any case, I've been testing all day with transparent_hugepage=never and the gain in terms of memory usage over time is very significant without any noticeable impact on performance for me, so I might go back to this setting in the end because I find it's better than before.

closed

added resolutioncompleted label

I believe this should be resolved by [1], now backported to 6.7.6.

[1] https://lore.kernel.org/all/20231221065943.2803551-2-shy828301@gmail.com/T/#m5282ee478f170156bdfe21354b2967c0883e9a3e

Indeed, without specifying anything for transparent_hugepage, my memory usage at startup is close to what it was before, even rather lower:

               total        used        free      shared  buff/cache   available
Mem:           3,4Gi       687Mi       2,4Gi       2,1Mi       610Mi       2,8Gi

That said, now that I've discovered transparent_hugepage=never thanks to this story, I'm keeping it, I'm happy with it :)