Michael Kelley 7296f2301a swiotlb: reduce swiotlb pool lookups ... With CONFIG_SWIOTLB_DYNAMIC enabled, each round-trip map/unmap pair in the swiotlb results in 6 calls to swiotlb_find_pool(). In multiple places, the pool is found and used in one function, and then must be found again in the next function that is called because only the tlb_addr is passed as an argument. These are the six call sites: dma_direct_map_page: 1. swiotlb_map -> swiotlb_tbl_map_single -> swiotlb_bounce dma_direct_unmap_page: 2. dma_direct_sync_single_for_cpu -> is_swiotlb_buffer 3. dma_direct_sync_single_for_cpu -> swiotlb_sync_single_for_cpu -> swiotlb_bounce 4. is_swiotlb_buffer 5. swiotlb_tbl_unmap_single -> swiotlb_del_transient 6. swiotlb_tbl_unmap_single -> swiotlb_release_slots Reduce the number of calls by finding the pool at a higher level, and passing it as an argument instead of searching again. A key change is for is_swiotlb_buffer() to return a pool pointer instead of a boolean, and then pass this pool pointer to subsequent swiotlb functions. There are 9 occurrences of is_swiotlb_buffer() used to test if a buffer is a swiotlb buffer before calling a swiotlb function. To reduce code duplication in getting the pool pointer and passing it as an argument, introduce inline wrappers for this pattern. The generated code is essentially unchanged. Since is_swiotlb_buffer() no longer returns a boolean, rename some functions to reflect the change: * swiotlb_find_pool() becomes __swiotlb_find_pool() * is_swiotlb_buffer() becomes swiotlb_find_pool() * is_xen_swiotlb_buffer() becomes xen_swiotlb_find_pool() With these changes, a round-trip map/unmap pair requires only 2 pool lookups (listed using the new names and wrappers): dma_direct_unmap_page: 1. dma_direct_sync_single_for_cpu -> swiotlb_find_pool 2. swiotlb_tbl_unmap_single -> swiotlb_find_pool These changes come from noticing the inefficiencies in a code review, not from performance measurements. With CONFIG_SWIOTLB_DYNAMIC, __swiotlb_find_pool() is not trivial, and it uses an RCU read lock, so avoiding the redundant calls helps performance in a hot path. When CONFIG_SWIOTLB_DYNAMIC is *not* set, the code size reduction is minimal and the perf benefits are likely negligible, but no harm is done. No functional change is intended. Signed-off-by: Michael Kelley <mhklinux@outlook.com> Reviewed-by: Petr Tesarik <petr@tesarici.cz> Signed-off-by: Christoph Hellwig <hch@lst.de>		2024-07-10 07:59:03 +02:00
..
bpf	asm-generic fixes for 6.10	2024-06-27 10:53:52 -07:00
cgroup	Misc fixes:	2024-05-19 11:38:15 -07:00
configs	hardening updates for 6.10-rc1	2024-05-13 14:14:05 -07:00
debug	kdb: Simplify management of tmpbuffer in kdb_read()	2024-04-26 17:13:31 +01:00
dma	swiotlb: reduce swiotlb pool lookups	2024-07-10 07:59:03 +02:00
entry	entry: Respect changes to system call number by trace_sys_enter()	2024-03-12 13:23:32 +01:00
events	perf/core: Fix missing wakeup when waiting for context reference	2024-06-05 15:52:33 +02:00
futex	printk: Change type of CONFIG_BASE_SMALL to bool	2024-05-06 17:39:09 +02:00
gcov	gcov: add support for GCC 14	2024-06-15 10:43:06 -07:00
irq	genirq/irqdesc: Prevent use-after-free in irq_find_at_or_after()	2024-05-24 12:49:35 +02:00
kcsan	kcsan, compiler_types: Introduce __data_racy type qualifier	2024-05-07 11:39:50 -07:00
livepatch	livepatch: Rename KLP_* to KLP_TRANSITION_*	2024-05-09 15:48:01 +02:00
locking	- Core Frameworks	2024-05-22 10:49:54 -07:00
module	kallsyms: rework symbol lookup return codes	2024-06-27 17:43:40 +02:00
power	vfs-6.10-rc2.fixes	2024-05-27 08:09:12 -07:00
printk	Revert "printk: Save console options for add_preferred_console_match()"	2024-06-25 07:58:10 +02:00
rcu	Merge branches 'fixes.2024.04.15a', 'misc.2024.04.12a', 'rcu-sync-normal-improve.2024.04.15a', 'rcu-tasks.2024.04.15a' and 'rcutorture.2024.04.15a' into rcu-merge.2024.04.15a	2024-05-01 13:04:02 +02:00
sched	bitmap patches for 6.10	2024-05-21 15:29:01 -07:00
time	hrtimer: Prevent queuing of hrtimer without a function callback	2024-06-25 16:54:27 +02:00
trace	kallsyms: rework symbol lookup return codes	2024-06-27 17:43:40 +02:00
.gitignore
acct.c	kernel misc: Remove the now superfluous sentinel elements from ctl_table array	2024-04-24 09:43:53 +02:00
async.c	async: Use a dedicated unbound workqueue with raised min_active	2024-02-09 11:13:59 -10:00
audit.c	audit: use KMEM_CACHE() instead of kmem_cache_create()	2024-01-25 10:12:22 -05:00
audit.h	audit: correct audit_filter_inodes() definition	2023-07-21 12:17:25 -04:00
audit_fsnotify.c
audit_tree.c	fsnotify: create a wrapper fsnotify_find_inode_mark()	2024-04-04 16:24:16 +02:00
audit_watch.c	fsnotify: create a wrapper fsnotify_find_inode_mark()	2024-04-04 16:24:16 +02:00
auditfilter.c	ima: Avoid blocking in RCU read-side critical section	2024-06-13 14:26:50 -04:00
auditsc.c	audit,io_uring: io_uring openat triggers audit reference count underflow	2023-10-13 18:34:46 +02:00
backtracetest.c	backtracetest: Convert from tasklet to BH workqueue	2024-02-05 13:22:34 -10:00
bounds.c	bounds: Use the right number of bits for power-of-two CONFIG_NR_CPUS	2024-04-29 08:29:29 -07:00
capability.c	lsm: constify the 'target' parameter in security_capget()	2023-08-08 16:48:47 -04:00
cfi.c
compat.c
configs.c
context_tracking.c	context_tracking: Make context_tracking_key __ro_after_init	2024-03-22 11:18:18 +01:00
cpu.c	cpu: Fix broken cmdline "nosmp" and "maxcpus=0"	2024-06-23 20:04:14 +02:00
cpu_pm.c
crash_core.c	Mainly singleton patches, documented in their respective changelogs.	2024-05-19 14:02:03 -07:00
crash_reserve.c	crash: add prefix for crash dumping messages	2024-05-08 08:41:26 -07:00
cred.c	cred: Use KMEM_CACHE() instead of kmem_cache_create()	2024-02-23 17:33:31 -05:00
delayacct.c	delayacct: Remove the now superfluous sentinel elements from ctl_table array	2024-04-24 09:43:54 +02:00
dma.c
elfcorehdr.c	crash: remove dependency of FA_DUMP on CRASH_DUMP	2024-02-23 17:48:22 -08:00
exec_domain.c
exit.c	mm: optimize the redundant loop of mm_update_owner_next()	2024-07-03 12:29:24 -07:00
exit.h	exit: add internal include file with helpers	2023-09-21 12:03:50 -06:00
extable.c
fail_function.c
fork.c	mm: rename mm_put_huge_zero_page to mm_put_huge_zero_folio	2024-04-25 20:56:20 -07:00
freezer.c	Linux 6.7-rc6	2023-12-23 15:52:13 +01:00
gen_kheaders.sh	kheaders: use command -v to test for existence of cpio	2024-05-30 01:13:20 +09:00
groups.c	groups: Convert group_info.usage to refcount_t	2023-09-29 11:28:39 -07:00
hung_task.c	kernel misc: Remove the now superfluous sentinel elements from ctl_table array	2024-04-24 09:43:53 +02:00
iomem.c	kernel/iomem.c: remove __weak ioremap_cache helper	2023-08-21 13:37:28 -07:00
irq_work.c
jump_label.c	jump_label,module: Don't alloc static_key_mod for __ro_after_init keys	2024-03-22 11:18:16 +01:00
kallsyms.c	kallsyms: rework symbol lookup return codes	2024-06-27 17:43:40 +02:00
kallsyms_internal.h	kallsyms: Avoid weak references for kallsyms symbols	2024-05-02 19:48:26 +09:00
kallsyms_selftest.c	mm: vmalloc: enable memory allocation profiling	2024-04-25 20:55:57 -07:00
kallsyms_selftest.h
kcmp.c	file: convert to SLAB_TYPESAFE_BY_RCU	2023-10-19 11:02:48 +02:00
Kconfig.freezer
Kconfig.hz
Kconfig.kexec	crash: clean up kdump related config items	2024-02-23 17:48:22 -08:00
Kconfig.locks
Kconfig.preempt
kcov.c	kcov: don't lose track of remote references during softirqs	2024-06-15 10:43:08 -07:00
kexec.c	crash: add a new kexec flag for hotplug support	2024-04-23 14:59:01 +10:00
kexec_core.c	kernel misc: Remove the now superfluous sentinel elements from ctl_table array	2024-04-24 09:43:53 +02:00
kexec_elf.c
kexec_file.c	crash: add a new kexec flag for hotplug support	2024-04-23 14:59:01 +10:00
kexec_internal.h	crash: remove dependency of FA_DUMP on CRASH_DUMP	2024-02-23 17:48:22 -08:00
kheaders.c
kprobes.c	kprobe/ftrace: fix build error due to bad function definition	2024-05-17 19:17:55 -07:00
ksyms_common.c
ksysfs.c	vmlinux: Avoid weak reference to notes section	2024-05-02 19:48:26 +09:00
kthread.c	kunit: Handle test faults	2024-05-06 14:22:02 -06:00
latencytop.c	kernel misc: Remove the now superfluous sentinel elements from ctl_table array	2024-04-24 09:43:53 +02:00
Makefile	crash: split crash dumping code out from kexec_core.c	2024-02-23 17:48:22 -08:00
module_signature.c
notifier.c
nsproxy.c	pidfd: add pidfs	2024-03-01 12:23:37 +01:00
numa.c	kernel/numa.c: Move logging out of numa.h	2023-12-20 19:26:30 -05:00
padata.c	padata: Disable BH when taking works lock on MT path	2024-04-12 15:07:51 +08:00
panic.c	kernel misc: Remove the now superfluous sentinel elements from ctl_table array	2024-04-24 09:43:53 +02:00
params.c	params: Fix multi-line comment style	2023-12-01 09:51:44 -08:00
pid.c	pidfs: remove config option	2024-03-13 12:53:53 -07:00
pid_namespace.c	zap_pid_ns_processes: clear TIF_NOTIFY_SIGNAL along with TIF_SIGPENDING	2024-06-15 10:43:06 -07:00
pid_sysctl.h	kernel misc: Remove the now superfluous sentinel elements from ctl_table array	2024-04-24 09:43:53 +02:00
profile.c	profiling: Remove create_prof_cpu_mask().	2024-04-27 11:17:48 -07:00
ptrace.c	ptrace_attach: shift send(SIGSTOP) into ptrace_set_stopped()	2024-02-22 15:38:52 -08:00
range.c
reboot.c	kernel misc: Remove the now superfluous sentinel elements from ctl_table array	2024-04-24 09:43:53 +02:00
regset.c	regset: use kvzalloc() for regset_get_alloc()	2024-04-25 21:07:03 -07:00
relay.c	kernel: relay: remove relay_file_splice_read dead code, doesn't work	2023-12-29 12:22:27 -08:00
resource.c	Quite a lot of kexec work this time around. Many singleton patches in	2024-01-09 11:46:20 -08:00
resource_kunit.c
rseq.c
scftorture.c	scftorture: Pause testing after memory-allocation failure	2023-07-14 15:02:57 -07:00
scs.c
seccomp.c	sysctl changes for v6.10-rc1	2024-05-17 17:31:24 -07:00
signal.c	virtio: features, fixes, cleanups	2024-05-23 12:04:36 -07:00
smp.c	CSD lock commits for v6.7	2023-10-30 17:56:53 -10:00
smpboot.c	kthread: add kthread_stop_put	2023-10-04 10:41:57 -07:00
smpboot.h
softirq.c	softirq: Fix suspicious RCU usage in __do_softirq()	2024-04-29 05:03:51 +02:00
stackleak.c	sysctl changes for v6.10-rc1	2024-05-17 17:31:24 -07:00
stacktrace.c	stacktrace: fix kernel-doc typo	2023-12-29 12:22:29 -08:00
static_call.c
static_call_inline.c
stop_machine.c
sys.c	RISC-V Patches for the 6.10 Merge Window, Part 1	2024-05-22 09:56:00 -07:00
sys_ni.c	syscalls: fix compat_sys_io_pgetevents_time64 usage	2024-06-25 15:57:20 +02:00
sysctl-test.c
sysctl.c	kernel misc: Remove the now superfluous sentinel elements from ctl_table array	2024-04-24 09:43:53 +02:00
task_work.c	task_work: add kerneldoc annotation for 'data' argument	2023-09-19 13:21:32 -07:00
taskstats.c	taskstats: fill_stats_for_tgid: use for_each_thread()	2023-10-04 10:41:57 -07:00
torture.c	torture: Print out torture module parameters	2023-09-24 17:24:01 +02:00
tracepoint.c
tsacct.c
ucount.c	sysctl changes for v6.10-rc1	2024-05-17 17:31:24 -07:00
uid16.c
uid16.h
umh.c	umh: Remove the now superfluous sentinel elements from ctl_table array	2024-04-24 09:43:53 +02:00
up.c	smp: Change function signatures to use call_single_data_t	2023-09-13 14:59:24 +02:00
user-return-notifier.c
user.c	printk: Change type of CONFIG_BASE_SMALL to bool	2024-05-06 17:39:09 +02:00
user_namespace.c	user_namespace: remove unnecessary NULL values from kbuf	2024-02-22 15:38:52 -08:00
usermode_driver.c
utsname.c
utsname_sysctl.c	kernel misc: Remove the now superfluous sentinel elements from ctl_table array	2024-04-24 09:43:53 +02:00
vhost_task.c	vhost_task: Handle SIGKILL by flushing work and exiting	2024-05-22 08:31:15 -04:00
vmcore_info.c	mm: free up PG_slab	2024-04-25 20:56:00 -07:00
watch_queue.c	watch_queue: fix kcalloc() arguments order	2023-12-21 13:17:54 +01:00
watchdog.c	Mainly singleton patches, documented in their respective changelogs.	2024-05-19 14:02:03 -07:00
watchdog_buddy.c	watchdog/hardlockup: move SMP barriers from common code to buddy code	2023-06-19 16:25:28 -07:00
watchdog_perf.c	kernel/watchdog_perf.c: tidy up kerneldoc	2024-05-08 08:41:29 -07:00
workqueue.c	workqueue: Refactor worker ID formatting and make wq_worker_comm() use full ID string	2024-05-20 13:28:40 -10:00
workqueue_internal.h	workqueue: Drop the special locking rule for worker->flags and worker_pool->flags	2023-08-07 15:57:22 -10:00