Skip to content
  • Vlastimil Babka's avatar
    mm, slub: move disabling/enabling irqs to ___slab_alloc() · 8d7cdc8a
    Vlastimil Babka authored
    
    
    Currently __slab_alloc() disables irqs around the whole ___slab_alloc().  This
    includes cases where this is not needed, such as when the allocation ends up in
    the page allocator and has to awkwardly enable irqs back based on gfp flags.
    Also the whole kmem_cache_alloc_bulk() is executed with irqs disabled even when
    it hits the __slab_alloc() slow path, and long periods with disabled interrupts
    are undesirable.
    
    As a first step towards reducing irq disabled periods, move irq handling into
    ___slab_alloc(). Callers will instead prevent the s->cpu_slab percpu pointer
    from becoming invalid via get_cpu_ptr(), thus preempt_disable(). This does not
    protect against modification by an irq handler, which is still done by disabled
    irq for most of ___slab_alloc(). As a small immediate benefit,
    slab_out_of_memory() from ___slab_alloc() is now called with irqs enabled.
    
    kmem_cache_alloc_bulk() disables irqs for its fastpath and then re-enables them
    before calling ___slab_alloc(), which then disables them at its discretion. The
    whole kmem_cache_alloc_bulk() operation also disables preemption.
    
    When  ___slab_alloc() calls new_slab() to allocate a new page, re-enable
    preemption, because new_slab() will re-enable interrupts in contexts that allow
    blocking (this will be improved by later patches).
    
    The patch itself will thus increase overhead a bit due to disabled preemption
    (on configs where it matters) and increased disabling/enabling irqs in
    kmem_cache_alloc_bulk(), but that will be gradually improved in the following
    patches.
    
    Note in __slab_alloc() we need to change the #ifdef CONFIG_PREEMPT guard to
    CONFIG_PREEMPT_COUNT to make sure preempt disable/enable is properly paired in
    all configurations. On configs without involuntary preemption and debugging
    the re-read of kmem_cache_cpu pointer is still compiled out as it was before.
    
    [ Mike Galbraith <efault@gmx.de>: Fix kmem_cache_alloc_bulk() error path ]
    Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Signed-off-by: default avatarSebastian Andrzej Siewior <bigeasy@linutronix.de>
    8d7cdc8a