There is a documented bug in sendfile.2 which notes that sendfile(2)
does not raise an error if it fails to copy out the number of bytes
written. Explicitly ignore the error from copyout() calls in
preparation for annotating copyout() with __result_use_check.
Reviewed by: glebius, kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43129
This is in preparation for annotating copyin() and related functions
with __result_use_check.
Reviewed by: mav, imp, asomers
Tested by: asomers
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43130
A recent change made it possible for cpu_set_upcall() to return an
error. Do that here instead of ignoring an error from copyin().
Reviewed by: jhibbits
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43105
This is in preparation for annotating copyin() and related functions
with __result_use_check.
Reviewed by: olce, dchagin
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43104
Some implementations copy data to userspace, an operation which can in
principle fail. In preparation for adding a __result_use_check
annotation to copyin() and related functions, let implementations of
cpu_set_upcall() return an error, and check for errors when copying data
to user memory.
Reviewed by: kib, jhb
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43100
This is in preparation for adding a __result_use_check annotation to
copyin() and related functions.
Reviewed by: imp, kib, jhb
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43099
If copyin() fails, the driver will blindly proceed with whatever had
been in the uninitialized DMA buffer. This is not what we want. Check
for copyin failures.
This is in preparation for annotating copyin() and related functions
with __result_use_check.
Reviewed by: ram
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43097
A failed copyin will cause the driver to use the contents of
uninitialized buffers instead, which is unlikely to be the behaviour
that we want. Check for errors.
This is in preparation for annotating copyin() and related functions
with __result_use_check.
Reviewed by: imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43098
This is in preparation for annotating copyin() and related functions
with __result_use_check.
Reviewed by: wulf
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43103
If copyin() fails, the driver will proceed blindly with a zeroed buffer,
which is not what we want. In preparation for annotating copyin() with
__result_use_check, start checking for errors.
Reviewed by: wulf
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D43102
same as the placeholder at the end. This is required to allow GAS to
start at zero, for integration with vmm.
Also, in iommu_gas_remove(), accept placeholders after remove op.
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
to allow to shut down assert in iommu_gas_cmp_entries() when used
against fake entry to search for specific place in the tree.
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
In commit 88640c0e8b the new EINTEGRITY errno value was added, and
this caused us to carry a patch for upstream libc++ since that time.
Because it can cause merge conflicts when importing libc++ code from
upstream, I have submitted an upstream pull request to get most of that
patch integrated.
It turns out that we do not need the errno.h part of it at all, since
all supported FreeBSD versions define EOWNERDEAD and ENOTRECOVERABLE,
and therefore the block that juggles with ELAST values is never used in
FreeBSD. At the moment it only applies to older versions of Linux, or
possibly other platforms.
Therefore the only part that needs to stay is the definition of a enum
errc value for EINTEGRITY, and this is made optional upon EINTEGRITY
being defined, to make it suitable for upstreaming.
No functional change is intended.
MFC after: 1 week
The scalar implementation is fairly simplistic and only performs
slightly better than the generic C implementation. It could be
improved by using the same algorithm as for memchr, but it would
have been a lot more complicated.
The baseline implementation is similar to timingsafe_memcmp. It's
slightly slower than memchr() due to the more complicated main
loop, but I don't think that can be significantly improved.
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42925
The "values" test case is specifically crafted to detect the off-by-one
error previous discovered in the scalar strchrnul implementation.
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42925
POSIX.1-2004 and the upcoming C23 agree that memccpy()'s arguments
are restrict qualified and must not overlap. In 2002, restrict
qualifiers were added to <string.h>'s declaration of the function.
Make things official and document that the arguments must not
overlap.
See also: 61b60edfd3
Approved by: kib
MFC after: 1 month
MFC to: stable/14
Based on the strlcpy code from D42863, this patch adds a SIMD-enhanced
implementation of memccpy for amd64. A scalar implementation calling
into memchr and memcpy to do the job is provided, too.
Please note that this code does not behave exactly the same as the C
implementation of memccpy for overlapping inputs. However, overlapping
inputs are not allowed for this function by ISO/IEC 9899:1999 and neither
has the C implementation any code to deal with the possibility. It just
proceeds byte-by-byte, which may or may not do the expected thing for
some overlaps. We do not document whether overlapping inputs are
supported in memccpy(3).
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42902
I previously forgot to mention these as they are set up through
contrib/arm-optimized/routines/string.
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
This should pick up our optimised memchr(), strlen(), and strlcpy()
when strlcat() is called.
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42863
Somewhat similar to stpncpy, but different in that we need to compute
the full source length even if the buffer is shorter than the source.
strlcat is implemented as a simple wrapper around strlcpy. The scalar
implementation of strlcpy just calls into strlen() and memcpy() to do
the job.
Perf-wise we're very close to stpncpy. The code is slightly slower as
it needs to carry on with finding the source string length even if the
buffer ends before the string.
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42863
A straightforward derivation from the stpncpy unit test.
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42863
strcat has a bespoke scalar assembly implementation we
inherited from NetBSD. While it performs well, it is
better to call into our SIMD implementations if any SIMD
features are available at all. So do that and implement
strcat() by calling into strlen() and strcpy() if these
are available.
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Reviison: https://reviews.freebsd.org/D42600
This was surprisingly annoying to get right, despite being such a simple
function. A scalar implementation is also provided, it just calls into
our optimised memchr(), memcpy(), and memset() routines to carry out its
job.
I'm quite happy with the performance. glibc only beats us for very long
strings, likely due to the use of AVX-512. The scalar implementation
just calls into our optimised memchr(), memcpy(), and memset() routines,
so it has a high overhead to begin with but then performs ok for the
amount of effort that went into it. Still beats the old C code, except
for very short strings.
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42519
This adds additional unit tests validating the function for
All possible alignment offsets of source and destination.
Also extend the test to allow testing of an external stpncpy
implementation, which greatly simplifies the development of
custom implementations.
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42519
The strsep() function is basically strcspn() with extra steps.
On amd64, we now have an optimised implementation of strcspn(),
so instead of implementing the inner loop manually, just call
into the optimised routine.
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42346
Also mention missing rindex() entry, which is provided through
strrchr().
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42217
The baseline implementation is very straightforward, while the scalar
implementation suffers from register pressure and the need to use SWAR
techniques similar to those used for strchr().
Sponsored by: The FreeBSD Foundation
Tested by: developers@, exp-run
Approved by: mjg
MFC after: 1 month
MFC to: stable/14
PR: 275785
Differential Revision: https://reviews.freebsd.org/D42217
