Now armv8_crypto is using FPU_KERN_NOCTX, this results in a kernel panic
in armv8_crypto.c:armv8_crypto_cipher_setup:
panic: recursive fpu_kern_enter while in PCB_FP_NOSAVE state
This is because in armv8_crypto.c:armv8_crypto_cipher_process,
directly after calling fpu_kern_enter() a call is made to
armv8_crypto_cipher_setup(), resulting in nested calls to
fpu_kern_enter() without the required fpu_kern_leave() in between.
Move fpu_kern_enter() in armv8_crypto_cipher_process() after the
call to armv8_crypto_cipher_setup() to resolve this.
Reviewed by: markj, andrew
Fixes: 6485286f53 ("armv8_crypto: Switch to using FPU_KERN_NOCTX")
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D41671
After commit 937b4473be aesni_cipher_crypt() and aesni_cipher_mac()
execute in a FPU_KERN_NOCTX section, which means that they must run with
preemption disabled. These functions handle discontiguous I/O buffers
by allocating a contiguous buffer and copying as necessary, but this
allocation cannot happen with preemption disabled. Fix the problem by
pushing the FPU section down into aesni_cipher_crypt() and
aesni_cipher_mac(). In particular, encrypt-then-auth transforms need
not be handled with a single FPU section.
Reported by: syzbot+78258dbb02eb92157357@syzkaller.appspotmail.com
Discussed with: jhb
Fixes: 937b4473be ("aesni: Switch to using FPU_KERN_NOCTX.")
crypto_unregister_all already disables new sessions and waits for
existing sessions to be destroyed before returning.
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D41581
crypto_unregister_all already disables new sessions and waits for
existing sessions to be destroyed before returning.
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D41579
There is a bug in the OpenSSL script which generates this file; the bug
is in the process of being fixed upstream.
Specifically, when generating the output, bsaes-armv7.pl strips some
labels that are used when the output asm is compiled with __KERNEL__
defined, resulting in a build error. As a step towards adding armv7
support to ossl(4), manually patch the generated asm. The upstream fix
will be imported later.
Reviewed by: andrew, jhb, emaste
MFC after: 1 week
Sponsored by: Klara, Inc.
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D41303
Chacha20+Poly1305 doesn't use an ossl_cipher instance the way AES-GCM
does, so ossl_lookup_cipher() failed causing ossl_newsession() to
always fail for Chacha20+Poly1305 sessions.
Reported by: gallatin (ktls_test fails with ossl.ko loaded)
Fixes: 9a3444d91c ossl: Add a VAES-based AES-GCM implementation for amd64
Tested by: gallatin
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D40580
This is required on i386. The patch has no functional change, since
AES-GCM isn't implemented for that platform.
Fixes: 9a3444d91c ("ossl: Add a VAES-based AES-GCM implementation for amd64")
Reported by: Jenkins
This lets one use ossl(4) for AES-GCM operations on contemporary amd64
platforms. A kernel benchmark indicates that this gives roughly
equivalent throughput to aesni(4) for various buffer sizes.
Bulk processing is done in aesni-gcm-x86_64.S, the rest is handled in a
C wrapper ported from OpenSSL's gcm128.c.
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Reviewed by: jhb
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39967
aes-gcm-avx512.S is generated from OpenSSL 3.1 and implements AES-GCM.
ossl_x86.c detects whether the CPU implements the required AVX512
instructions; if not, the ossl(4) module does not provide an AES-GCM
implementation. The VAES implementation increases throughput for all
buffer sizes in both directions, up to 2x for sufficiently large
buffers.
The "process" implementation is in two parts: a generic OCF layer in
ossl_aes.c that calls a set of MD functions to do the heavy lifting.
The intent there is to make it possible to add other implementations for
other platforms, e.g., to reduce the diff required for D37421.
A follow-up commit will add a fallback path to legacy AES-NI, so that
ossl(4) can be used in preference to aesni(4) on all amd64 platforms.
In the long term we would like to replace aesni(4) and armv8crypto(4)
with ossl(4).
Note, currently this implementation will not be selected by default
since aesni(4) and ossl(4) return the same probe priority for crypto
sessions, and the opencrypto framework selects the first registered
implementation to break a tie. Since aesni(4) is compiled into the
kernel, aesni(4) wins. A separate change may modify ossl(4) to have
priority.
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
Reviewed by: jhb
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39783
This is needed to let OpenSSL 3.1 routines detect VAES and VPCLMULQDQ
extensions. The intent is to import ASM routines which implement
AES-GCM using VEX-prefixed AES-NI instructions.
No functional change intended.
Sponsored by: Stormshield
Sponsored by: Klara, Inc.
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D39782
The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.
Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix
All the other functions used pointers for from/to instead of
fixed-size array parameters. More importantly, this function can
accept pointers to buffers of multiple blocks, not just a single
block.
Reported by: GCC -Warray-parameter
Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D37547
OpenZFS changed the define _OPENSOLARIS_SYS_TYPES_H_ to
_SPL_SYS_TYPES_H_ to guard the sys/types.h compatibility file
inclusion. Follow the change here. The only place in the tree
_OPENSOLARIS_SYS_TYPES_H_ is mentioned is in the /*
_OPENSOLARIS_SYS_TYPES_H_ */ at the end of sys/types.h. That needs to be
changed upstream in OpenZFS since we don't like changing things in
FreeBSD's tree.
Sponsored by: Netflix
Reviewed by: tsoome, delphij
Differential Revision: https://reviews.freebsd.org/D35891
All supported compilers (modern versions of GCC and clang) support
this.
Many places didn't have an #else so would just silently do the wrong
thing. Ancient versions of icc (the original motivation for this) are
no longer a compiler FreeBSD supports.
PR: 263102 (exp-run)
Reviewed by: brooks, imp
Differential Revision: https://reviews.freebsd.org/D34797
Currently armv8crypto copies the scheme used in aesni(9), where payload
data and output buffers are allocated on the fly if the crypto buffer is
not virtually contiguous. This scheme is simple but incurs a lot of
overhead: for an encryption request with a separate output buffer we
have to
- allocate a temporary buffer to hold the payload
- copy input data into the buffer
- copy the encrypted payload to the output buffer
- zero the temporary buffer before freeing it
We have a handy crypto buffer cursor abstraction now, so reimplement the
armv8crypto routines using that instead of temporary buffers. This
introduces some extra complexity, but gallatin@ reports a 10% throughput
improvement with a KTLS workload without additional CPU usage. The
driver still allocates an AAD buffer for AES-GCM if necessary.
Reviewed by: jhb
Tested by: gallatin
Sponsored by: Ampere Computing LLC
Submitted by: Klara Inc.
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D28950
This is in preparation for using buffer cursors. No functional change
intended.
Reviewed by: jhb
Sponsored by: Ampere Computing LLC
Submitted by: Klara Inc.
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D28948
This adds a wrapper around libsodium's curve25519 support matching
Linux's curve25519 API. The intended use case for this is WireGuard.
Note that this is not integrated with OCF as it is not related to
symmetric operations on data.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33935
This is a synchronous software API which wraps the existing software
implementation shared with OCF. Note that this will not currently
use optimized backends (such as ossl(4)) but may be appropriate for
operations on small buffers.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33524
These callbacks allow multiple contiguous blocks to be manipulated in
a single call. Note that any trailing partial block for a stream
cipher must still be passed to encrypt/decrypt_last.
While here, document the setkey and reinit hooks and reorder the hooks
in 'struct enc_xform' to better reflect the life cycle.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33529
This is the same as 4285655adb ("aesni: Avoid a potential
out-of-bounds load in AES_GCM_encrypt()") except for the decryption
path.
Reported by: Jenkins (KASAN job)
Reviewed by: cem
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33474
As with sha256 add support for accelerated sha512 support to libmd on
arm64. This depends on clang 13+ to build as this is the first release
with the needed intrinsics. Gcc should also support them, however from
a currently unknown release.
Reviewed by: cem
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33373
Only pre-allocate auth contexts when a session-wide key is provided or
for sessions without keys. For sessions with per-operation keys,
always initialize the on-stack context directly rather than
initializing the session context in swcr_authprepare (now removed) and
then copying that session context into the on-stack context.
This approach permits parallel auth operations without needing a
serializing lock. In addition, the previous code assumed that auth
sessions always provided an initial key unlike cipher sessions which
assume either an initial key or per-op keys.
While here, fix the Blake2 auth transforms to function like other auth
transforms where Setkey is invoked after Init rather than before.
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33316
This centralizes the check for valid nonce lengths for AES-GCM.
While here, remove some duplicate checks for valid AES-GCM tag lengths
from ccp(4) and ccr(4).
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33194
GCC 9 doesn't define a SHA256_Transform symbol when the stub just wraps
SHA256_Transform_c resulting in an undefined symbol for
_libmd_SHA256_Transform in libmd.so.
Discussed with: andrew, jrtc27
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D31945
