system/linux
system/linux
1
0
Fork 0
mirror of https://github.com/torvalds/linux synced 2024-10-20 10:19:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
linux/kernel/bpf
History
Andrey Ignatov 0640c77c46 bpf: Avoid retpoline for bpf_for_each_map_elem 
Similarly to 09772d92cd ("bpf: avoid retpoline for
lookup/update/delete calls on maps") and 84430d4232 ("bpf, verifier:
avoid retpoline for map push/pop/peek operation") avoid indirect call
while calling bpf_for_each_map_elem.

Before (a program fragment):

  ; if (rules_map) {
   142: (15) if r4 == 0x0 goto pc+8
   143: (bf) r3 = r10
  ; bpf_for_each_map_elem(rules_map, process_each_rule, &ctx, 0);
   144: (07) r3 += -24
   145: (bf) r1 = r4
   146: (18) r2 = subprog[+5]
   148: (b7) r4 = 0
   149: (85) call bpf_for_each_map_elem#143680  <-- indirect call via
                                                    helper

After (same program fragment):

   ; if (rules_map) {
    142: (15) if r4 == 0x0 goto pc+8
    143: (bf) r3 = r10
   ; bpf_for_each_map_elem(rules_map, process_each_rule, &ctx, 0);
    144: (07) r3 += -24
    145: (bf) r1 = r4
    146: (18) r2 = subprog[+5]
    148: (b7) r4 = 0
    149: (85) call bpf_for_each_array_elem#170336  <-- direct call

On a benchmark that calls bpf_for_each_map_elem() once and does many
other things (mostly checking fields in skb) with CONFIG_RETPOLINE=y it
makes program faster.

Before:

  ============================================================================
  Benchmark.cpp                                              time/iter iters/s
  ============================================================================
  IngressMatchByRemoteEndpoint                                80.78ns 12.38M
  IngressMatchByRemoteIP                                      80.66ns 12.40M
  IngressMatchByRemotePort                                    80.87ns 12.37M

After:

  ============================================================================
  Benchmark.cpp                                              time/iter iters/s
  ============================================================================
  IngressMatchByRemoteEndpoint                                73.49ns 13.61M
  IngressMatchByRemoteIP                                      71.48ns 13.99M
  IngressMatchByRemotePort                                    70.39ns 14.21M

Signed-off-by: Andrey Ignatov <rdna@fb.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://lore.kernel.org/bpf/20211006001838.75607-1-rdna@fb.com
2021-10-05 19:22:33 -07:00
..
preload libbpf: Make libbpf_version.h non-auto-generated 2021-09-13 15:36:47 -07:00
arraymap.c bpf: Replace callers of BPF_CAST_CALL with proper function typedef 2021-09-28 16:27:18 -07:00
bpf_inode_storage.c bpf: Fix spelling mistakes 2021-05-24 21:13:05 -07:00
bpf_iter.c bpf: Refactor BPF_PROG_RUN into a function 2021-08-17 00:45:07 +02:00
bpf_local_storage.c bpf: Prevent deadlock from recursive bpf_task_storage_[get|delete] 2021-02-26 11:51:48 -08:00
bpf_lru_list.c bpf_lru_list: Read double-checked variable once without lock 2021-02-10 15:54:26 -08:00
bpf_lru_list.h bpf: Fix a typo "inacitve" -> "inactive" 2020-04-06 21:54:10 +02:00
bpf_lsm.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next 2021-06-17 11:54:56 -07:00
bpf_struct_ops.c bpf: Handle return value of BPF_PROG_TYPE_STRUCT_OPS prog 2021-09-14 11:09:50 -07:00
bpf_struct_ops_types.h bpf: tcp: Support tcp_congestion_ops in bpf 2020-01-09 08:46:18 -08:00
bpf_task_storage.c bpf: Consolidate task_struct BTF_ID declarations 2021-08-25 10:37:05 -07:00
btf.c bpf: selftests: Add selftests for module kfunc support 2021-10-05 17:07:42 -07:00
cgroup.c bpf: Migrate cgroup_bpf to internal cgroup_bpf_attach_type enum 2021-08-23 17:50:24 -07:00
core.c bpf: Introduce BPF support for kernel module function calls 2021-10-05 17:07:41 -07:00
cpumap.c bpf: cpumap: Implement generic cpumap 2021-07-07 20:01:45 -07:00
devmap.c bpf, devmap: Exclude XDP broadcast to master device 2021-08-09 23:25:14 +02:00
disasm.c bpf: Relicense disassembler as GPL-2.0-only OR BSD-2-Clause 2021-09-02 14:49:23 +02:00
disasm.h bpf: Relicense disassembler as GPL-2.0-only OR BSD-2-Clause 2021-09-02 14:49:23 +02:00
dispatcher.c bpf: Remove bpf_image tree 2020-03-13 12:49:52 -07:00
hashtab.c bpf: Replace callers of BPF_CAST_CALL with proper function typedef 2021-09-28 16:27:18 -07:00
helpers.c bpf: Replace callers of BPF_CAST_CALL with proper function typedef 2021-09-28 16:27:18 -07:00
inode.c bpf: Fix regression on BPF_OBJ_GET with non-O_RDWR flags 2021-06-22 14:57:43 +02:00
Kconfig sock_map: Relax config dependency to CONFIG_NET 2021-07-15 18:17:49 -07:00
local_storage.c bpf: Increase supported cgroup storage value size 2021-07-27 15:59:29 -07:00
lpm_trie.c bpf: Allow RCU-protected lookups to happen from bh context 2021-06-24 19:41:15 +02:00
Makefile bpf: Enable task local storage for tracing programs 2021-02-26 11:51:47 -08:00
map_in_map.c bpf: Remember BTF of inner maps. 2021-07-15 22:31:10 +02:00
map_in_map.h bpf: Add map_meta_equal map ops 2020-08-28 15:41:30 +02:00
map_iter.c bpf: Implement link_query callbacks in map element iterators 2020-08-21 14:01:39 -07:00
net_namespace.c bpf: Add support for forced LINK_DETACH command 2020-08-01 20:38:28 -07:00
offload.c bpf, offload: Replace bitwise AND by logical AND in bpf_prog_offload_info_fill 2020-02-17 16:53:49 +01:00
percpu_freelist.c bpf: Use raw_spin_trylock() for pcpu_freelist_push/pop in NMI 2020-10-06 00:04:11 +02:00
percpu_freelist.h bpf: Use raw_spin_trylock() for pcpu_freelist_push/pop in NMI 2020-10-06 00:04:11 +02:00
prog_iter.c bpf: Refactor bpf_iter_reg to have separate seq_info member 2020-07-25 20:16:32 -07:00
queue_stack_maps.c bpf: Eliminate rlimit-based memory accounting for queue_stack_maps maps 2020-12-02 18:32:46 -08:00
reuseport_array.c bpf: Fix spelling mistakes 2021-05-24 21:13:05 -07:00
ringbuf.c bpf: Fix false positive kmemleak report in bpf_ringbuf_area_alloc() 2021-06-28 15:57:46 +02:00
stackmap.c bpf, mm: Fix lockdep warning triggered by stack_map_get_build_id_offset() 2021-09-10 22:24:23 +02:00
syscall.c bpf: Use kvmalloc for map keys in syscalls 2021-08-20 00:09:49 +02:00
sysfs_btf.c bpf: Load and verify kernel module BTFs 2020-11-10 15:25:53 -08:00
task_iter.c bpf: Consolidate task_struct BTF_ID declarations 2021-08-25 10:37:05 -07:00
tnum.c bpf, tnums: Provably sound, faster, and more precise algorithm for tnum_mul 2021-06-01 13:34:15 +02:00
trampoline.c bpf: Introduce helper bpf_get_branch_snapshot 2021-09-13 10:53:50 -07:00
verifier.c bpf: Avoid retpoline for bpf_for_each_map_elem 2021-10-05 19:22:33 -07:00