mirror of
https://github.com/torvalds/linux
synced 2024-09-06 09:51:23 +00:00
05c5323b2a
Let's track the mapcount of large folios in a single value. The mapcount of a large folio currently corresponds to the sum of the entire mapcount and all page mapcounts. This sum is what we actually want to know in folio_mapcount() and it is also sufficient for implementing folio_mapped(). With PTE-mapped THP becoming more important and more widely used, we want to avoid looping over all pages of a folio just to obtain the mapcount of large folios. The comment "In the common case, avoid the loop when no pages mapped by PTE" in folio_total_mapcount() does no longer hold for mTHP that are always mapped by PTE. Further, we are planning on using folio_mapcount() more frequently, and might even want to remove page mapcounts for large folios in some kernel configs. Therefore, allow for reading the mapcount of large folios efficiently and atomically without looping over any pages. Maintain the mapcount also for hugetlb pages for simplicity. Use the new mapcount to implement folio_mapcount() and folio_mapped(). Make page_mapped() simply call folio_mapped(). We can now get rid of folio_large_is_mapped(). _nr_pages_mapped is now only used in rmap code and for debugging purposes. Keep folio_nr_pages_mapped() around, but document that its use should be limited to rmap internals and debugging purposes. This change implies one additional atomic add/sub whenever mapping/unmapping (parts of) a large folio. As we now batch RMAP operations for PTE-mapped THP during fork(), during unmap/zap, and when PTE-remapping a PMD-mapped THP, and we adjust the large mapcount for a PTE batch only once, the added overhead in the common case is small. Only when unmapping individual pages of a large folio (e.g., during COW), the overhead might be bigger in comparison, but it's essentially one additional atomic operation. Note that before the new mapcount would overflow, already our refcount would overflow: each mapping requires a folio reference. Extend the focumentation of folio_mapcount(). Link: https://lkml.kernel.org/r/20240409192301.907377-5-david@redhat.com Signed-off-by: David Hildenbrand <david@redhat.com> Reviewed-by: Yin Fengwei <fengwei.yin@intel.com> Cc: Chris Zankel <chris@zankel.net> Cc: Hugh Dickins <hughd@google.com> Cc: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Max Filippov <jcmvbkbc@gmail.com> Cc: Miaohe Lin <linmiaohe@huawei.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Peter Xu <peterx@redhat.com> Cc: Richard Chang <richardycc@google.com> Cc: Rich Felker <dalias@libc.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Yang Shi <shy828301@gmail.com> Cc: Yoshinori Sato <ysato@users.sourceforge.jp> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|---|---|---|
| .. | ||
| damon | ||
| active_mm.rst | ||
| allocation-profiling.rst | ||
| arch_pgtable_helpers.rst | ||
| balance.rst | ||
| bootmem.rst | ||
| free_page_reporting.rst | ||
| highmem.rst | ||
| hmm.rst | ||
| hugetlbfs_reserv.rst | ||
| hwpoison.rst | ||
| index.rst | ||
| ksm.rst | ||
| memory-model.rst | ||
| mmu_notifier.rst | ||
| multigen_lru.rst | ||
| numa.rst | ||
| oom.rst | ||
| overcommit-accounting.rst | ||
| page_allocation.rst | ||
| page_cache.rst | ||
| page_frags.rst | ||
| page_migration.rst | ||
| page_owner.rst | ||
| page_reclaim.rst | ||
| page_table_check.rst | ||
| page_tables.rst | ||
| physical_memory.rst | ||
| process_addrs.rst | ||
| remap_file_pages.rst | ||
| shmfs.rst | ||
| slab.rst | ||
| slub.rst | ||
| split_page_table_lock.rst | ||
| swap.rst | ||
| transhuge.rst | ||
| unevictable-lru.rst | ||
| vmalloc.rst | ||
| vmalloced-kernel-stacks.rst | ||
| vmemmap_dedup.rst | ||
| z3fold.rst | ||
| zsmalloc.rst | ||