For invop providers (i.e., fbt and kinst) we can simply reach into the
invop trapframe to fetch argument registers for arguments 0-7; for
argument 8 and beyond we have to read the value off of the stack.
Reviewed by: Domagoj Stolfa, avg
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D45649
SDT calls dtrace_probe() directly, and this can be used to pass up to
five probe arguments directly. To pass the sixth argument (SDT
currently doesn't support more than this), we use a hack: just add
additional parameters to the call and cast dtrace_probe accordingly.
This happens to work on amd64, but doesn't work in general.
Modify SDT to call dtrace_probe() after storing arguments beyond the
first five in thread-local storage. Implement sdt_getargval() to fetch
extra argument values this way. An alternative would be to use invop
handlers instead and make sdt_probe_func point to a breakpoint
instruction, so that one can extract arguments using the breakpoint
exception trapframe, but this makes the providers more expensive when
enabled and doesn't seem justified. This approach works well unless we
want to add more than one or two more parameters to SDT probes, which
seems unlikely at present.
In particular, this fixes fetching the last argument of most ip and tcp
probes on arm64.
Reported by: rwatson
Reviewed by: Domagoj Stolfa
MFC after: 1 month
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D45648
The idea here is to avoid a memory access and conditional branch per
probe site. Instead, the probe is represented by an "unreachable"
unconditional function call. asm goto is used to store the address of
the probe site (represented by a no-op sled) and the address of the
function call into a tracepoint record. Each SDT probe carries a list
of tracepoints.
When the probe is enabled, the no-op sled corresponding to each
tracepoint is overwritten with a jmp to the corresponding label. The
implementation uses smp_rendezvous() to park all other CPUs while the
instruction is being overwritten, as this can't be done atomically in
general. The compiler moves argument marshalling code and the
sdt_probe() function call out-of-line, i.e., to the end of the function.
Per gallatin@ in D43504, this approach has less overhead when probes are
disabled. To make the implementation a bit simpler, I removed support
for probes with 7 arguments; nothing makes use of this except a
regression test case. It could be re-added later if need be.
The approach taken in this patch enables some more improvements:
1. We can now automatically fill out the "function" field of SDT probe
names. The SDT macros let the programmer specify the function and
module names, but this is really a bug and shouldn't have been
allowed. The intent was to be able to have the same probe in
multiple functions and to let the user restrict which probes actually
get enabled by specifying a function name or glob.
2. We can avoid branching on SDT_PROBES_ENABLED() by adding the ability
to include blocks of code in the out-of-line path. For example:
if (SDT_PROBES_ENABLED()) {
int reason = CLD_EXITED;
if (WCOREDUMP(signo))
reason = CLD_DUMPED;
else if (WIFSIGNALED(signo))
reason = CLD_KILLED;
SDT_PROBE1(proc, , , exit, reason);
}
could be written
SDT_PROBE1_EXT(proc, , , exit, reason,
int reason;
reason = CLD_EXITED;
if (WCOREDUMP(signo))
reason = CLD_DUMPED;
else if (WIFSIGNALED(signo))
reason = CLD_KILLED;
);
In the future I would like to use this mechanism more generally, e.g.,
to remove branches and marshalling code used by hwpmc, and generally to
make it easier to add new tracepoint consumers without having to add
more conditional branches to hot code paths.
Reviewed by: Domagoj Stolfa, avg
MFC after: 2 months
Differential Revision: https://reviews.freebsd.org/D44483
LLD has the -zbti-report=error argument to check if the BTI note is
present when linking. To allow for this to be used when linking the
kernel and modules:
- Add the BTI note to the remaining assembly files
- Mark ptrauth.c as protected by BTI
- Disable -zbti-report for vmm hypervisor switching code as it's not
used there.
The linux64 module doesn't build with the flag as it includes vdso code
that doesn't include the note.
Reviewed by: imp, kib, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45466
This reverts commit e92491d95f.
The general idea looked good to me. In particular, it allowed to save
some memory and avoid memory allocation failures when a large buffer
size was requested along with ring and fill policies.
But I didn't take into account that the second, supposedly unused
buffer, was actually used as the scratch buffer. The scratch buffer is
used as a temporary space for DTrace subroutines like copyin, copyinstr,
and alloca.
I think that the change can be fixed by allocating a separate smaller
buffer for the scratch buffer, but that fix would require more work than
I am able to do now. Hence the revert.
Reported by: Domagoj Stolfa
Diagnosed by: Domagoj Stolfa, markj
MFC after: immediately
No functional change, but this reduces diffs with CheriBSD downstream.
Reviewed by: andrew
Sponsored by: University of Cambridge, Google, Inc.
Differential Revision: https://reviews.freebsd.org/D44344
No functional change, but this reduces diffs with CheriBSD downstream.
Reviewed by: andrew
Sponsored by: University of Cambridge, Google, Inc.
Differential Revision: https://reviews.freebsd.org/D44342
Introduce the allocuio() and freeuio() functions to allocate and
deallocate struct uio. This hides the actual allocator interface, so it
is easier to modify the sub-allocation layout of struct uio and the
corresponding iovec array.
Obtained from: CheriBSD
Reviewed by: kib, markj
MFC after: 2 weeks
Sponsored by: CHaOS, EPSRC grant EP/V000292/1
Differential Revision: https://reviews.freebsd.org/D43711
This should disable allocation of the second per-CPU principal buffer
which is never used. This will also enable additional asserts
for buffers that are never switched.
profile provider probes fire in the context of a timer interrupt. Thus,
the "regs" action can make use of the interrupt trap frame to get
register values when the interrupt happened in kernel mode. Make that
trap frame available when possible so that "regs" works more or less as
it already does with the fbt and kinst providers.
MFC after: 1 week
Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.
Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/
Sponsored by: Netflix
Remove ancient SCCS tags from the tree, automated scripting, with two
minor fixup to keep things compiling. All the common forms in the tree
were removed with a perl script.
Sponsored by: Netflix
Most of the first block of pad bytes are now used for space accounting
purposes. No functional change intended.
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
fbt computes incorrect instruction position for AArch64 kernel module symbol.
The issue is with the for loop, it does an extra increment of instr pointer
after the required instruction is found. Hence, a wrong instruction is
targeted for patching.
Signed-off-by: Igor Ostapenko <pm@igoro.pro>
Fixes: 980746e5cb ("fbt: simplify arm64 function-prologue parsing")
Reviewed by: markj
Pull Request: https://github.com/freebsd/freebsd-src/pull/855
MFC after: 1 week
kinst currently only traces functions that start and end with the usual
function prologue and epilogue respectively. Ignoring functions that do
not have an epilogue however, makes the filtering too strict, as this
means that we can not trace functions that never return (e.g
vnlru_proc()). This patch relaxes the filtering and only checks whether
the function pushes the frame pointer.
Reviewed by: markj
Approved by: markj
Differential Revision: https://reviews.freebsd.org/D41876
When using printm(), one should always pass a scratch pointer to it.
This is achieved by calling printm with memref
BEGIN { printm(fixed_len, memref(ptr, var_len)); }
which will return a pointer to the DTrace scratch space of size
sizeof(uintptr_t) * 2. However, one can easily call printm() as follows
BEGIN { printm(10, (void *)NULL); }
and panic the kernel as a result. This commit does two things:
(1) adds a new macro DTRACE_INSCRATCHPTR(mstate, ptr, howmany) which
checks if a certain pointer is in the DTrace scratch space;
(2) uses DTRACE_INSCRATCHPTR() to implement a check on printm()'s DIFO
return value in order to avoid the panic and sets CPU_DTRACE_BADADDR
if the address is not in the scratch space.
Reviewed by: markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D41722
This reverts commit 22508c8b6c.
The t_kinst_curprobe field is no longer needed by kinst.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41031
Using per-CPU and per-thread trampolines is expensive and error-prone,
since we're rewriting the same memory blocks constantly. Per-probe
trampolines solve this problem by giving each probe its own block of
executable memory, which more or less remains the same after the initial
write.
What this patch does, is get rid of the initialization code which
allocates a trampoline for each thread, and instead let each port of
kinst allocate a trampoline for each new probe created. It also sets up
the infrastructure needed to support the new trampoline scheme.
This change is not currently supported on amd64, as the amd64 port needs
further changes to work, so this is a temporary/gradual patch to fix the
riscv and arm64 ports.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40962
Currently kinst checks if only the first instruction is 'push %rbp',
essentially excluding functions that do push RBP, but not in the first
instruction. This patch modifies kinst to check for 'push %rbp', as
well, as a following 'pop %rbp', anywhere in the function. This behavior
also matches that of FBT.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40283
Under COMPAT_32BIT we are compiling 32-bit code and so __LP64__ is not
defined, __ILP32__ is, and thus the check is completely redundant.
Reviewed by: brooks, jhb, imp
Differential Revision: https://reviews.freebsd.org/D40917
HAS_INLINE_FLS and similar macros are defined always.
Removes the redundant tests for these always-true conditions.
Reviewed by: mhorne
Differential Revision: https://reviews.freebsd.org/D40707
Needed by the forthcoming RISC-V and ARM64 ports.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40872
Use unprivileged loads to access user memory. Without this, the
accesses trap and various dtrace actions such as ustack() fail.
Reviewed by: andrew
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40540
It was used in one place and was added specifically to support dtrace
stack unwinding code. Write an equivalent expression using struct
unwind_state instead. No functional change intended.
Reviewed by: andrew
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40538
The forthcoming RISC-V and ARM64 ports of kinst introduce a new field
named "t_kinst_curprobe", so "t_kinst" (which points to a trampoline)
becomes a misleading name.
No functional change intended.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40507
Centralize KINST_TRAMP_FILL_PATTERN and KINST_TRAMP_FILL_SIZE to reduce
redefinitions, and use the architecture-dependent kinst_patchval_t as
their size.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40406
match_opcode() is defined in FBT, kinst, and dtrace_subr.c. The function
prologue-checking functions are defined in FBT and kinst.
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40335
Tracing memcpy() would crash the kernel, because we'd also trace the
memcpy() calls from kinst_invop(). To fix this, introduce kinst_memcpy()
whose arguments are 'volatile', so that we avoid having the compiler
replace it with a regular memcpy().
Reviewed by: markj
Approved by: markj (mentor)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D40284
