freebsd-src/sys/vm
Mark Johnston b16b4c22d2 vm_page: Implement lazy page initialization
FreeBSD's boot times have decreased to the point where vm_page array
initialization represents a significant fraction of the total boot time.
For example, when booting FreeBSD in Firecracker (a VMM designed to
support lightweight VMs) with 128MB and 1GB of RAM, vm_page
initialization consumes 9% (3ms) and 37% (21.5ms) of the kernel boot
time, respectively.  This is generally relevant in cloud environments,
where one wants to be able to spin up VMs as quickly as possible.

This patch implements lazy initialization of (most) page structures,
following a suggestion from cperciva@.  The idea is to introduce a new
free pool, VM_FREEPOOL_LAZYINIT, into which all vm_page structures are
initially placed.  For this to work, we need only initialize the first
free page of each chunk placed into the buddy allocator.  Then, early
page allocations draw from the lazy init pool and initialize vm_page
chunks (up to 16MB, 4096 pages) on demand.  Once APs are started, an
idle-priority thread drains the lazy init pool in the background to
avoid introducing extra latency in the allocator.  With this scheme,
almost all of the initialization work is moved out of the critical path.

A couple of vm_phys operations require the pool to be drained before
they can run: vm_phys_find_range() and vm_phys_unfree_page().  However,
these are rare operations.  I believe that
vm_phys_find_freelist_contig() does not require any special treatment,
as it only ever accesses the first page in a power-of-2-sized free page
chunk, which is always initialized.

For now the new pool is only used on amd64 and arm64, since that's where
I can easily test and those platforms would get the most benefit.

Reviewed by:	alc, kib
Differential Revision:	https://reviews.freebsd.org/D40403
2024-06-13 21:19:00 -04:00
..
_vm_phys.h Revert "subr_pctrie: use ilog2(x) instead of fls(x)-1" 2024-06-03 13:07:42 -05:00
_vm_radix.h radix_trie: have vm_radix use pctrie code 2023-09-12 02:42:38 -05:00
device_pager.c cdev_pager_allocate(): ensure that the cdev_pager_ops ctr is called only once 2024-05-12 04:13:00 +03:00
memguard.c sys: Remove $FreeBSD$: one-line .c pattern 2023-08-16 11:54:36 -06:00
memguard.h sys: Remove $FreeBSD$: two-line .h pattern 2023-08-16 11:54:11 -06:00
phys_pager.c sys: Automated cleanup of cdefs and other formatting 2023-11-26 22:24:00 -07:00
pmap.h pmap: move the smp_targeted_tlb_shutdown pointer stuff to amd64 pmap.h 2024-06-06 08:15:08 +03:00
redzone.c sys: Automated cleanup of cdefs and other formatting 2023-11-26 22:24:00 -07:00
redzone.h sys: Remove $FreeBSD$: two-line .h pattern 2023-08-16 11:54:11 -06:00
sg_pager.c sys: Remove $FreeBSD$: one-line .c pattern 2023-08-16 11:54:36 -06:00
swap_pager.c swap_pager: pctrie_reclaim_cb in meta_free_all 2024-06-13 13:52:25 -05:00
swap_pager.h sys: Remove ancient SCCS tags. 2023-11-26 22:23:30 -07:00
uma.h uma: Permit specifying max of cache line and some custom alignment 2023-11-02 09:30:03 -04:00
uma_align_mask.h uma: Make the cache alignment mask unsigned 2023-11-02 09:30:03 -04:00
uma_core.c uma: Fix improper uses of UMA_MD_SMALL_ALLOC 2024-05-26 07:27:37 +02:00
uma_dbg.c Fix build on powerpc after previous commit. 2023-11-09 21:21:47 -05:00
uma_dbg.h sys: Remove $FreeBSD$: two-line .h pattern 2023-08-16 11:54:11 -06:00
uma_int.h sys: Remove $FreeBSD$: two-line .h pattern 2023-08-16 11:54:11 -06:00
vm.h pmap: Skip some superpage promotion attempts that will fail 2024-06-04 00:38:05 -05:00
vm_domainset.c Avoid waiting on physical allocations that can't possibly be satisfied 2023-12-23 23:01:40 -06:00
vm_domainset.h Avoid waiting on physical allocations that can't possibly be satisfied 2023-12-23 23:01:40 -06:00
vm_dumpset.h sys: Remove $FreeBSD$: two-line .h pattern 2023-08-16 11:54:11 -06:00
vm_extern.h vm: Change the return types of kernacc and useracc to bool 2024-05-10 13:43:56 -07:00
vm_fault.c pmap: Skip some superpage promotion attempts that will fail 2024-06-04 00:38:05 -05:00
vm_glue.c vm: Change the return types of kernacc and useracc to bool 2024-05-10 13:43:56 -07:00
vm_init.c Adjust comments referencing vm_mem_init() 2024-05-27 18:37:40 -03:00
vm_kern.c Avoid waiting on physical allocations that can't possibly be satisfied 2023-12-23 23:01:40 -06:00
vm_kern.h vm: improve kstack_object pindex calculation to avoid pindex holes 2024-04-10 17:37:20 +02:00
vm_map.c vm: Reduce address space fragmentation 2024-06-13 15:13:45 -05:00
vm_map.h sys: Remove ancient SCCS tags. 2023-11-26 22:23:30 -07:00
vm_meter.c sys: Automated cleanup of cdefs and other formatting 2023-11-26 22:24:00 -07:00
vm_mmap.c Remove never implemented sbrk and sstk syscalls 2023-12-04 20:36:08 +00:00
vm_object.c vm: Eliminate a redundant call to vm_reserv_break_all() 2024-06-05 12:39:47 -05:00
vm_object.h sys: Remove ancient SCCS tags. 2023-11-26 22:23:30 -07:00
vm_page.c vm_page: Implement lazy page initialization 2024-06-13 21:19:00 -04:00
vm_page.h vm_page: Let vm_page_init_page() take a pool parameter 2024-06-13 21:18:59 -04:00
vm_pageout.c vm_pageout_scan_inactive: take a lock break 2024-05-24 08:52:58 -07:00
vm_pageout.h sys: Remove ancient SCCS tags. 2023-11-26 22:23:30 -07:00
vm_pagequeue.h vm_pageout_scan_inactive: take a lock break 2024-05-24 08:52:58 -07:00
vm_pager.c pbuf_ctor(): Stop using LK_NOWAIT, use LK_NOWITNESS 2024-05-26 10:20:52 -07:00
vm_pager.h sys: Remove ancient SCCS tags. 2023-11-26 22:23:30 -07:00
vm_param.h sys: Remove ancient SCCS tags. 2023-11-26 22:23:30 -07:00
vm_phys.c vm_page: Implement lazy page initialization 2024-06-13 21:19:00 -04:00
vm_phys.h vm_page: Implement lazy page initialization 2024-06-13 21:19:00 -04:00
vm_radix.c uma: Deduplicate uma_small_alloc 2024-05-25 19:24:46 +02:00
vm_radix.h vm_radix: define vm_radix_insert_lookup_lt and use in vm_page_rename 2024-06-06 10:26:50 -07:00
vm_reserv.c Revert "subr_pctrie: use ilog2(x) instead of fls(x)-1" 2024-06-03 13:07:42 -05:00
vm_reserv.h vm_reserv: Add vm_reserv_is_populated 2024-04-07 12:28:52 -05:00
vm_swapout.c vm: improve kstack_object pindex calculation to avoid pindex holes 2024-04-10 17:37:20 +02:00
vm_swapout_dummy.c sys: Automated cleanup of cdefs and other formatting 2023-11-26 22:24:00 -07:00
vm_unix.c sys: Automated cleanup of cdefs and other formatting 2023-11-26 22:24:00 -07:00
vnode_pager.c Stop treating size 0 as unknown size in vnode_create_vobject(). 2024-05-23 06:08:14 +00:00
vnode_pager.h Add vnode_pager_clean_{a,}sync(9) 2024-01-11 18:44:53 +02:00