mirror of
https://github.com/torvalds/linux
synced 2024-10-15 15:59:15 +00:00
7c05e7f3e7
The motivation of inlining bpf_kptr_xchg() comes from the performance profiling of bpf memory allocator benchmark. The benchmark uses bpf_kptr_xchg() to stash the allocated objects and to pop the stashed objects for free. After inling bpf_kptr_xchg(), the performance for object free on 8-CPUs VM increases about 2%~10%. The inline also has downside: both the kasan and kcsan checks on the pointer will be unavailable. bpf_kptr_xchg() can be inlined by converting the calling of bpf_kptr_xchg() into an atomic_xchg() instruction. But the conversion depends on two conditions: 1) JIT backend supports atomic_xchg() on pointer-sized word 2) For the specific arch, the implementation of xchg is the same as atomic_xchg() on pointer-sized words. It seems most 64-bit JIT backends satisfies these two conditions. But as a precaution, defining a weak function bpf_jit_supports_ptr_xchg() to state whether such conversion is safe and only supporting inline for 64-bit host. For x86-64, it supports BPF_XCHG atomic operation and both xchg() and atomic_xchg() use arch_xchg() to implement the exchange, so enabling the inline of bpf_kptr_xchg() on x86-64 first. Reviewed-by: Eduard Zingerman <eddyz87@gmail.com> Signed-off-by: Hou Tao <houtao1@huawei.com> Link: https://lore.kernel.org/r/20240105104819.3916743-2-houtao@huaweicloud.com Signed-off-by: Alexei Starovoitov <ast@kernel.org> |
||
---|---|---|
.. | ||
preload | ||
arraymap.c | ||
bloom_filter.c | ||
bpf_cgrp_storage.c | ||
bpf_inode_storage.c | ||
bpf_iter.c | ||
bpf_local_storage.c | ||
bpf_lru_list.c | ||
bpf_lru_list.h | ||
bpf_lsm.c | ||
bpf_struct_ops.c | ||
bpf_struct_ops_types.h | ||
bpf_task_storage.c | ||
btf.c | ||
cgroup.c | ||
cgroup_iter.c | ||
core.c | ||
cpumap.c | ||
cpumask.c | ||
devmap.c | ||
disasm.c | ||
disasm.h | ||
dispatcher.c | ||
hashtab.c | ||
helpers.c | ||
inode.c | ||
Kconfig | ||
link_iter.c | ||
local_storage.c | ||
log.c | ||
lpm_trie.c | ||
Makefile | ||
map_in_map.c | ||
map_in_map.h | ||
map_iter.c | ||
memalloc.c | ||
mmap_unlock_work.h | ||
mprog.c | ||
net_namespace.c | ||
offload.c | ||
percpu_freelist.c | ||
percpu_freelist.h | ||
prog_iter.c | ||
queue_stack_maps.c | ||
reuseport_array.c | ||
ringbuf.c | ||
stackmap.c | ||
syscall.c | ||
sysfs_btf.c | ||
task_iter.c | ||
tcx.c | ||
tnum.c | ||
trampoline.c | ||
verifier.c |