freebsd-src/sys/kern
Jeff Roberson d4665eaa66 Implement a safe memory reclamation feature that is tightly coupled with UMA.
This is in the same family of algorithms as Epoch/QSBR/RCU/PARSEC but is
a unique algorithm.  This has 3x the performance of epoch in a write heavy
workload with less than half of the read side cost.  The memory overhead
is significantly lessened by limiting the free-to-use latency.  A synthetic
test uses 1/20th of the memory vs Epoch.  There is significant further
discussion in the comments and code review.

This code should be considered experimental.  I will write a man page after
it has settled.  After further validation the VM will begin using this
feature to permit lockless page lookups.

Both markj and cperciva tested on arm64 at large core counts to verify
fences on weaker ordering architectures.  I will commit a stress testing
tool in a follow-up.

Reviewed by:	mmacy, markj, rlibby, hselasky
Discussed with:	sbahara
Differential Revision:	https://reviews.freebsd.org/D22586
2020-01-31 00:49:51 +00:00
..
bus_if.m Add necessary bits for Linux KPI to work correctly on powerpc 2019-08-04 19:28:10 +00:00
capabilities.conf Add a shm_open2 syscall to support upcoming memfd_create 2019-09-25 17:59:15 +00:00
clock_if.m
cpufreq_if.m
device_if.m
genassym.sh
genoffset.c Merge td_epochnest with td_no_sleeping. 2019-10-29 17:28:25 +00:00
genoffset.sh expose thread_lite definition to tied modules 2018-07-03 02:50:07 +00:00
imgact_aout.c vfs: drop the mostly unused flags argument from VOP_UNLOCK 2020-01-03 22:29:58 +00:00
imgact_binmisc.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
imgact_elf.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
imgact_elf32.c
imgact_elf64.c
imgact_shell.c Add helper functions to copy strings into struct image_args. 2018-11-29 21:00:56 +00:00
init_main.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
init_sysent.c Jail and capability mode for shm_rename; add audit support for shm_rename 2019-11-18 13:31:16 +00:00
kern_acct.c vfs: drop the mostly unused flags argument from VOP_UNLOCK 2020-01-03 22:29:58 +00:00
kern_alq.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_clock.c sleep(9), sleepqueue(9): const'ify wchan pointers 2019-12-24 16:19:33 +00:00
kern_clocksource.c Rename hardclock_cnt() to hardclock() and remove the old implementation. 2018-09-06 02:10:59 +00:00
kern_condvar.c
kern_conf.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_cons.c kern_cons: add a stub kbdinit for configs with no keyboard/console drivers 2019-12-26 15:47:19 +00:00
kern_context.c Remove superfluous bzero in getcontext/swapcontext/sendsig 2018-11-26 20:56:05 +00:00
kern_cpu.c hwpstate_intel(4): Silence/fix Coverity reports 2020-01-29 03:15:34 +00:00
kern_cpuset.c bitset: rename confusing macro NAND to ANDNOT 2019-12-13 09:32:16 +00:00
kern_ctf.c vfs: drop the mostly unused flags argument from VOP_UNLOCK 2020-01-03 22:29:58 +00:00
kern_descrip.c vfs: provide F_ISUNIONSTACK as a kludge for libc 2020-01-17 14:42:25 +00:00
kern_dtrace.c systrace: track it like sdt probes 2018-04-27 15:16:34 +00:00
kern_dump.c [PPC64] Initial kernel minidump implementation 2019-10-14 13:04:04 +00:00
kern_environment.c Follow up r352244: kenv: tighten up assertions 2019-09-12 14:34:46 +00:00
kern_et.c
kern_event.c Avoid relying on header pollution from sys/refcount.h. 2019-07-29 20:26:01 +00:00
kern_exec.c vfs: drop the mostly unused flags argument from VOP_UNLOCK 2020-01-03 22:29:58 +00:00
kern_exit.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_fail.c Use callout(9) instead of deprecated timeout(9) for fail points. 2019-12-13 19:26:04 +00:00
kern_ffclock.c
kern_fork.c schedlock 1/4 2019-12-15 21:11:15 +00:00
kern_hhook.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_idle.c
kern_intr.c Enter the network epoch for interrupt handlers of INTR_TYPE_NET. 2020-01-23 01:24:47 +00:00
kern_jail.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_kcov.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_khelp.c
kern_kthread.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_ktr.c Add KERNEL_PANICKED macro for use in place of direct panicstr tests 2020-01-12 06:07:54 +00:00
kern_ktrace.c vfs: drop the mostly unused flags argument from VOP_UNLOCK 2020-01-03 22:29:58 +00:00
kern_linker.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_lock.c lockmgr: don't touch the lock past unlock 2020-01-24 07:42:57 +00:00
kern_lockf.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_lockstat.c Add lockmgr(9) probes to the lockstat DTrace provider. 2019-08-21 23:43:58 +00:00
kern_loginclass.c Replace hand-rolled unrefs if > 1 with refcount_release_if_not_last 2018-12-07 16:11:45 +00:00
kern_malloc.c Change malloc_domain() to return the allocation size to the caller. 2020-01-09 15:02:48 +00:00
kern_mbuf.c Use a separate lock for the zone and keg. This provides concurrency 2020-01-04 03:15:34 +00:00
kern_mib.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_module.c Use NULL for SYSINIT's last arg, which is a pointer type 2018-05-18 17:58:09 +00:00
kern_mtxpool.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_mutex.c Add KERNEL_PANICKED macro for use in place of direct panicstr tests 2020-01-12 06:07:54 +00:00
kern_ntptime.c Clear pad bytes in the struct exported by kern.ntp_pll.gettime. 2018-11-20 20:32:10 +00:00
kern_osd.c
kern_physio.c Allocate pager bufs from UMA instead of 80-ish mutex protected linked list. 2019-01-15 01:02:16 +00:00
kern_pmc.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_poll.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_priv.c Check for probes enabled in priv_check_cred before evaluting the error. 2018-12-19 23:28:29 +00:00
kern_proc.c vfs: drop the mostly unused flags argument from VOP_UNLOCK 2020-01-03 22:29:58 +00:00
kern_procctl.c Add procctl(PROC_STACKGAP_CTL) 2019-09-03 18:56:25 +00:00
kern_prot.c malloc: remove assumptions about MINALLOCSIZE 2020-01-14 02:14:02 +00:00
kern_racct.c proc: eliminate the zombproc list 2019-08-28 16:18:23 +00:00
kern_rangelock.c rangelock: add rangelock_cookie_assert 2019-09-15 02:59:53 +00:00
kern_rctl.c call racct_proc_ucred_changed() under the proc lock 2018-04-20 13:08:04 +00:00
kern_resource.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_rmlock.c Plug a warning in read-mostly spinlocks reported by gcc. 2019-12-27 13:37:19 +00:00
kern_rwlock.c locks: add default delay struct 2020-01-05 12:48:19 +00:00
kern_sdt.c
kern_sema.c
kern_sendfile.c Consistently use busy and vm_page_valid() rather than touching page bits 2020-01-23 04:54:49 +00:00
kern_sharedpage.c Fix the last few cases that grab without busy or valid. The grab functions must 2019-12-02 22:38:25 +00:00
kern_shutdown.c Add "panicked" boolean which can be tested instead of panicstr 2020-01-12 06:09:10 +00:00
kern_sig.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_switch.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_sx.c locks: add default delay struct 2020-01-05 12:48:19 +00:00
kern_synch.c Provide an API for interlocked refcount sleeps. 2020-01-19 18:18:17 +00:00
kern_syscalls.c add a sanity check to the system call registration code 2019-12-11 15:52:29 +00:00
kern_sysctl.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_tc.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_thr.c schedlock 1/4 2019-12-15 21:11:15 +00:00
kern_thread.c vfs: prealloc vnodes in getnewvnode_reserve 2020-01-11 22:58:14 +00:00
kern_time.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_timeout.c Remove the deprecated timeout(9) interface. 2019-12-13 21:03:12 +00:00
kern_tslog.c
kern_ubsan.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_umtx.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
kern_uuid.c Use sbuf_cat() in GEOM confxml generation. 2019-06-19 15:36:02 +00:00
kern_xxx.c Normalize COMPAT_43 syscall declarations. 2018-12-04 16:48:47 +00:00
ksched.c
link_elf.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
link_elf_obj.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
linker_if.m
Make.tags.inc Remove a couple of harmless stray references to nandfs. 2019-06-25 16:39:25 +00:00
Makefile sysent targets: further cleanup and deduplication 2020-01-18 20:37:45 +00:00
makesyscalls.sh Restore the ability to set capenabled directly in syscalls.conf. 2019-09-30 20:58:29 +00:00
md4c.c
md5c.c
msi_if.m
p1003_1b.c
pic_if.m
posix4_mib.c Use NULL for SYSINIT's last arg, which is a pointer type 2018-05-18 17:58:09 +00:00
sched_4bsd.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
sched_ule.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
serdev_if.m
stack_protector.c Revert r346292 (permit_nonrandom_stackcookies) 2019-05-13 23:37:44 +00:00
subr_acl_nfs4.c Remove unused argument to priv_check_cred. 2018-12-11 19:32:16 +00:00
subr_acl_posix1e.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_atomic64.c emulated atomic64: disable interrupts as the lock mechanism on !SMP 2020-01-03 18:29:20 +00:00
subr_autoconf.c
subr_blist.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_boot.c When parsing command line stuff, treat tabs and spaces the same. 2019-04-18 22:52:12 +00:00
subr_bufring.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_bus.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_bus_dma.c Add an external mbuf buffer type that holds multiple unmapped pages. 2019-06-29 00:48:33 +00:00
subr_busdma_bufalloc.c Add malloc_domainset(9) and _domainset variants to other allocator KPIs. 2018-10-30 18:26:34 +00:00
subr_capability.c kern_sendit: use pre-initialized rights 2018-05-23 01:48:09 +00:00
subr_clock.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_compressor.c GZIO: Update to use zlib 1.2.11. 2019-08-25 07:50:44 +00:00
subr_counter.c Fix pre-SI_SUB_CPU initialization of per-CPU counters. 2018-07-10 00:18:12 +00:00
subr_coverage.c Extract the coverage sanitizer KPI to a new file. 2019-01-29 11:04:17 +00:00
subr_csan.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_devmap.c Print upper 32 bits in devmap table entries 2019-12-20 03:40:53 +00:00
subr_devstat.c Use atomic for start_count in devstat_start_transaction(). 2019-12-30 03:13:38 +00:00
subr_disk.c Fix botched merge with 355066 2019-03-12 05:10:41 +00:00
subr_dummy_vdso_tc.c Move most of the contents of opt_compat.h to opt_global.h. 2018-04-06 17:35:35 +00:00
subr_early.c Add a file missed in r339321 2018-10-12 00:32:45 +00:00
subr_epoch.c Change argument order of epoch_call() to more natural, first function, 2020-01-17 06:10:24 +00:00
subr_eventhandler.c Include ktr.h in more compilation units 2019-05-21 20:38:48 +00:00
subr_fattime.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_filter.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_firmware.c Factor out check for mounted root file system. 2019-11-28 08:47:36 +00:00
subr_gtaskqueue.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_hash.c
subr_hints.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_intr.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_kdb.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_kobj.c Prevent some kobj memory allocation failures from panicking the system. 2019-01-31 22:27:39 +00:00
subr_lock.c locks: add default delay struct 2020-01-05 12:48:19 +00:00
subr_log.c
subr_mchain.c
subr_module.c Have preload_delete_name() free pages backing preloaded data. 2018-07-19 20:00:28 +00:00
subr_msgbuf.c msgbuf: Light detailing (const'ify and bool'itize) 2018-08-09 17:42:27 +00:00
subr_param.c riscv: restore default HZ=1000, keep QEMU at HZ=100 2019-09-07 05:13:31 +00:00
subr_pcpu.c schedlock 1/4 2019-12-15 21:11:15 +00:00
subr_pctrie.c kern/subr_pctrie: Fix mismatched signedness in assertion comparison 2019-04-06 21:56:24 +00:00
subr_pidctrl.c When pidctrl_daemon() is called multiple times within an interval, it 2018-06-07 07:48:50 +00:00
subr_power.c Extract eventfilter declarations to sys/_eventfilter.h 2019-05-20 00:38:23 +00:00
subr_prf.c Add KERNEL_PANICKED macro for use in place of direct panicstr tests 2020-01-12 06:07:54 +00:00
subr_prof.c ANSIfy sys/kern 2018-06-01 13:26:45 +00:00
subr_rangeset.c Implement rangesets. 2019-02-20 09:38:19 +00:00
subr_rman.c add support for marking interrupt handlers as suspended 2018-12-17 17:11:00 +00:00
subr_rtc.c Improve error messages from clock_if.m method failures. 2018-09-02 20:17:51 +00:00
subr_sbuf.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_scanf.c Add support for 'j', 't' and 'z' flags to kernel sscanf(). 2019-08-16 19:46:22 +00:00
subr_sfbuf.c
subr_sglist.c Add an external mbuf buffer type that holds multiple unmapped pages. 2019-06-29 00:48:33 +00:00
subr_sleepqueue.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_smp.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_smr.c Implement a safe memory reclamation feature that is tightly coupled with UMA. 2020-01-31 00:49:51 +00:00
subr_stack.c Add debugging facility EPOCH_TRACE that checks that epochs entered are 2019-09-25 18:26:31 +00:00
subr_stats.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_syscall.c Don't pass error from syscallenter() to syscallret(). 2019-07-15 21:25:16 +00:00
subr_taskqueue.c schedlock 1/4 2019-12-15 21:11:15 +00:00
subr_terminal.c kernel terminal should initialize fg and bg variables before calling TUNABLE_INT_FETCH 2019-09-26 07:19:26 +00:00
subr_trap.c vfs: prealloc vnodes in getnewvnode_reserve 2020-01-11 22:58:14 +00:00
subr_turnstile.c schedlock 4/4 2019-12-15 21:26:50 +00:00
subr_uio.c simplify control flow so that gcc knows we never pass save to curthread_pflags_restore 2018-05-19 04:04:44 +00:00
subr_unit.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_vmem.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
subr_witness.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
sys_capability.c Let kern.trap_enotcap be set as a tunable. 2018-12-06 17:29:37 +00:00
sys_generic.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
sys_getrandom.c Fix a typo in r356667 comment 2020-01-12 23:52:16 +00:00
sys_pipe.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
sys_procdesc.c procdesc: allow to collect status through wait(1) if process is traced 2019-11-25 18:33:21 +00:00
sys_process.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
sys_socket.c
syscalls.c sysent: regenerate after r352747. 2019-09-26 15:41:10 +00:00
syscalls.master Jail and capability mode for shm_rename; add audit support for shm_rename 2019-11-18 13:31:16 +00:00
systrace_args.c sysent: regenerate after r354835 2019-11-18 23:31:12 +00:00
sysv_ipc.c sysv: get rid of fork/exit hooks if the code is compiled in 2019-05-04 19:05:30 +00:00
sysv_msg.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
sysv_sem.c Move most of the contents of opt_compat.h to opt_global.h. 2018-04-06 17:35:35 +00:00
sysv_shm.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
tty.c vfs: introduce v_irflag and make v_type smaller 2019-12-08 21:30:04 +00:00
tty_compat.c Move most of the contents of opt_compat.h to opt_global.h. 2018-04-06 17:35:35 +00:00
tty_info.c Add KERNEL_PANICKED macro for use in place of direct panicstr tests 2020-01-12 06:07:54 +00:00
tty_inq.c tty: use __unused annotation instead to silence warnings 2018-05-19 04:48:26 +00:00
tty_outq.c tty: use __unused annotation instead to silence warnings 2018-05-19 04:48:26 +00:00
tty_pts.c tty_pts: don't rely on tty header pollution for sys/mutex.h 2019-11-29 03:56:01 +00:00
tty_tty.c Extract eventfilter declarations to sys/_eventfilter.h 2019-05-20 00:38:23 +00:00
tty_ttydisc.c Replace ttyprintf with sbuf_printf and tty drain routine 2018-10-20 18:31:36 +00:00
uipc_accf.c
uipc_debug.c Load balance sockets with new SO_REUSEPORT_LB option. 2018-06-06 15:45:57 +00:00
uipc_domain.c - Move global network epoch definition to epoch.h, as more different 2020-01-15 03:34:21 +00:00
uipc_ktls.c Make sure the VNET is properly set when calling tcp_drop() from 2020-01-21 11:43:25 +00:00
uipc_mbuf.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
uipc_mbuf2.c m_pulldown(): Change an if () panic() into a KASSERT(). 2019-11-06 22:40:19 +00:00
uipc_mbufhash.c
uipc_mqueue.c Make mqueue objects work across a fork again. 2020-01-27 22:36:54 +00:00
uipc_sem.c Remove unused argument to priv_check_cred. 2018-12-11 19:32:16 +00:00
uipc_shm.c Don't hold the object lock while calling getpages. 2020-01-19 23:47:32 +00:00
uipc_sockbuf.c sysctl: hide 2.x era compat node 2020-01-02 01:23:43 +00:00
uipc_socket.c Cleanup unneeded includes that crept in with r353292. 2019-10-09 16:59:42 +00:00
uipc_syscalls.c Only enable COMPAT_43 changes for syscalls ABI for a.out processes. 2019-08-11 19:16:07 +00:00
uipc_usrreq.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
vfs_acl.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
vfs_aio.c vfs: drop the mostly unused flags argument from VOP_UNLOCK 2020-01-03 22:29:58 +00:00
vfs_bio.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
vfs_cache.c cache: revert r352613 now that vhold does not take locks 2020-01-20 19:52:23 +00:00
vfs_cluster.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
vfs_default.c vfs: remove vop loop from vop_sigdefer 2020-01-26 07:05:06 +00:00
vfs_export.c Ensure that directory entry padding bytes are zeroed. 2018-11-23 22:24:59 +00:00
vfs_extattr.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
vfs_hash.c vfs: incomplete pass at converting more ints to u_long 2020-01-11 22:56:20 +00:00
vfs_init.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
vfs_lookup.c vfs: drop the mostly unused flags argument from VOP_UNLOCK 2020-01-03 22:29:58 +00:00
vfs_mount.c Add kern_unmount() and use in Linuxulator. No functional changes. 2020-01-24 11:57:55 +00:00
vfs_mountroot.c vfs: drop the mostly unused flags argument from VOP_UNLOCK 2020-01-03 22:29:58 +00:00
vfs_subr.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
vfs_syscalls.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
vfs_vnops.c Remove duplicated empty lines from kern/*.c 2020-01-30 20:05:05 +00:00
vnode_if.src vfs: drop the mostly unused flags argument from VOP_UNLOCK 2020-01-03 22:29:58 +00:00