The AVX2 implementation might waste up to a page of stack memory because
of a wrong alignment calculation. This will, in the worst case, increase
the stack usage of sha1_transform_avx2() alone to 5.4 kB -- way to big
for a kernel function. Even worse, it might also allocate *less* bytes
than needed if the stack pointer is already aligned bacause in that case
the 'sub %rbx, %rsp' is effectively moving the stack pointer upwards,
not downwards.
Fix those issues by changing and simplifying the alignment calculation
to use a 32 byte alignment, the alignment really needed.
Cc: Chandramouli Narayanan <mouli@linux.intel.com>
Signed-off-by: Mathias Krause <minipli@googlemail.com>
Reviewed-by: H. Peter Anvin <hpa@linux.intel.com>
Reviewed-by: Marek Vasut <marex@denx.de>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Commit 7c1da8d0d0 "crypto: sha - SHA1 transform x86_64 AVX2"
accidentally disabled the AVX variant by making the avx_usable() test
not only fail in case the CPU doesn't support AVX or OSXSAVE but also
if it doesn't support AVX2.
Fix that regression by splitting up the AVX/AVX2 test into two
functions. Also test for the BMI1 extension in the avx2_usable() test
as the AVX2 implementation not only makes use of BMI2 but also BMI1
instructions.
Cc: Chandramouli Narayanan <mouli@linux.intel.com>
Signed-off-by: Mathias Krause <minipli@googlemail.com>
Reviewed-by: H. Peter Anvin <hpa@linux.intel.com>
Reviewed-by: Marek Vasut <marex@denx.de>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This git patch adds x86_64 AVX2 optimization of SHA1
transform to crypto support. The patch has been tested with 3.14.0-rc1
kernel.
On a Haswell desktop, with turbo disabled and all cpus running
at maximum frequency, tcrypt shows AVX2 performance improvement
from 3% for 256 bytes update to 16% for 1024 bytes update over
AVX implementation.
This patch adds sha1_avx2_transform(), the glue, build and
configuration changes needed for AVX2 optimization of
SHA1 transform to crypto support.
sha1-ssse3 is one module which adds the necessary optimization
support (SSSE3/AVX/AVX2) for the low-level SHA1 transform function.
With better optimization support, transform function is overridden
as the case may be. In the case of AVX2, due to performance reasons
across datablock sizes, the AVX or AVX2 transform function is used
at run-time as it suits best. The Makefile change therefore appends
the necessary objects to the linkage. Due to this, the patch merely
appends AVX2 transform to the existing build mix and Kconfig support
and leaves the configuration build support as is.
Signed-off-by: Chandramouli Narayanan <mouli@linux.intel.com>
Reviewed-by: Marek Vasut <marex@denx.de>
Acked-by: H. Peter Anvin <hpa@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The crypto algorithm modules utilizing the crypto daemon could
be used early when the system start up. Using module_init
does not guarantee that the daemon's work queue is initialized
when the cypto alorithm depending on crypto_wq starts. It is necessary
to initialize the crypto work queue earlier at the subsystem
init time to make sure that it is initialized
when used.
Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
(struct caam_ctx) ctx->key_dma needs to be unmapped
when context is cleaned up.
Signed-off-by: Horia Geanta <horia.geanta@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Add test vectors for aead with null encryption and md5,
respectively sha1 authentication.
Input data is taken from test vectors listed in RFC2410.
Signed-off-by: Horia Geanta <horia.geanta@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
These defines might be needed by crypto drivers.
Signed-off-by: Horia Geanta <horia.geanta@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Commit 61bb86bba1
("crypto: caam - set descriptor sharing type to SERIAL")
changed the descriptor sharing mode from SHARE_WAIT to SHARE_SERIAL.
All descriptor commands that handle the "ok to share" and
"error propagation" settings should also go away, since they have no
meaning for SHARE_SERIAL.
Signed-off-by: Horia Geanta <horia.geanta@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The ahash_def_finup() can make use of the request save/restore functions,
thus make it so. This simplifies the code a little and unifies the code
paths.
Note that the same remark about free()ing the req->priv applies here, the
req->priv can only be free()'d after the original request was restored.
Finally, squash a bug in the invocation of completion in the ASYNC path.
In both ahash_def_finup_done{1,2}, the function areq->base.complete(X, err);
was called with X=areq->base.data . This is incorrect , as X=&areq->base
is the correct value. By analysis of the data structures, we see the areq is
of type 'struct ahash_request' , areq->base is of type 'struct crypto_async_request'
and areq->base.completion is of type crypto_completion_t, which is defined in
include/linux/crypto.h as:
typedef void (*crypto_completion_t)(struct crypto_async_request *req, int err);
This is one lead that the X should be &areq->base . Next up, we can inspect
other code which calls the completion callback to give us kind-of statistical
idea of how this callback is used. We can try:
$ git grep base\.complete\( drivers/crypto/
Finally, by inspecting ahash_request_set_callback() implementation defined
in include/crypto/hash.h , we observe that the .data entry of 'struct
crypto_async_request' is intended for arbitrary data, not for completion
argument.
Signed-off-by: Marek Vasut <marex@denx.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Fabio Estevam <fabio.estevam@freescale.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Shawn Guo <shawn.guo@linaro.org>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The functions to save original request within a newly adjusted request
and it's counterpart to restore the original request can be re-used by
more code in the crypto/ahash.c file. Pull these functions out from the
code so they're available.
Signed-off-by: Marek Vasut <marex@denx.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Fabio Estevam <fabio.estevam@freescale.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Shawn Guo <shawn.guo@linaro.org>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Add documentation for the pointer voodoo that is happening in crypto/ahash.c
in ahash_op_unaligned(). This code is quite confusing, so add a beefy chunk
of documentation.
Moreover, make sure the mangled request is completely restored after finishing
this unaligned operation. This means restoring all of .result, .base.data
and .base.complete .
Also, remove the crypto_completion_t complete = ... line present in the
ahash_op_unaligned_done() function. This type actually declares a function
pointer, which is very confusing.
Finally, yet very important nonetheless, make sure the req->priv is free()'d
only after the original request is restored in ahash_op_unaligned_done().
The req->priv data must not be free()'d before that in ahash_op_unaligned_finish(),
since we would be accessing previously free()'d data in ahash_op_unaligned_done()
and cause corruption.
Signed-off-by: Marek Vasut <marex@denx.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Fabio Estevam <fabio.estevam@freescale.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Shawn Guo <shawn.guo@linaro.org>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Found by the kbuild test robot, the first argument to caam_init_rng
has a spurious ampersand.
Reported-by: kbuild test robot <fengguang.wu@intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
HIGHMEM pages may not be mapped so we must kmap them before accessing.
This resolves a random OOPs error that was showing up during OpenSSL SHA tests.
Signed-off-by: Joel Fernandes <joelf@ti.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch allocates memory from DMAable region to the caam_rng_ctx object,
earlier it had been statically allocated which resulted in errorneous
behaviour on inserting the caamrng module at the runtime.
Signed-off-by: Nitesh Lal <NiteshNarayanLal@freescale.com>
Acked-by: Ruchika Gupta <ruchika.gupta@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This adds the function blkcipher_aead_walk_virt_block, which allows the caller
to use the blkcipher walk API to handle the input and output scatterlists.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
In order to allow other uses of the blkcipher walk API than the blkcipher
algos themselves, this patch copies some of the transform data members to the
walk struct so the transform is only accessed at walk init time.
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
When bringing a new RNG source online, it seems like it would make sense
to use some of its bytes to make the system entropy pool more random,
as done with all sorts of other devices that contain per-device or
per-boot differences.
Signed-off-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Jason Cooper <jason@lakedaemon.net>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Optimize the hashing operation in the MXS-DCP by doing two adjustments:
1) Given that the output buffer for the hash is now always correctly aligned,
we can just use the buffer for the DCP DMA to store the resulting hash.
We thus get rid of one copying of data. Moreover, we remove an entry from
dcp_coherent_block{} and thus lower the memory footprint of the driver.
2) We map the output buffer for the hash for DMA only in case we will output
the hash, not always, as it was now.
Signed-off-by: Marek Vasut <marex@denx.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Fabio Estevam <fabio.estevam@freescale.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Shawn Guo <shawn.guo@linaro.org>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The DCP needs the bounce buffers, DMA descriptors and result buffers aligned
to 64 bytes (yet another hardware limitation). Make sure they are aligned by
properly aligning the structure which contains them during allocation.
Signed-off-by: Marek Vasut <marex@denx.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Fabio Estevam <fabio.estevam@freescale.com>
Cc: Herbert Xu <herbert@gondor.apana.org.au>
Cc: Shawn Guo <shawn.guo@linaro.org>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Make omap_des_copy_needed(), omap_des_copy_sgs(), because these
functions are used only in this file.
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Acked-by: Joel Fernandes <joelf@ti.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use SIMPLE_DEV_PM_OPS macro in order to make the code simpler.
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use SIMPLE_DEV_PM_OPS macro in order to make the code simpler.
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use SIMPLE_DEV_PM_OPS macro in order to make the code simpler.
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Acked-by: Nishanth Menon <nm@ti.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use devm_*() functions to make cleanup paths simpler.
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use devm_*() functions to make cleanup paths simpler.
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use devm_clk_get() to make cleanup paths simpler.
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use devm_clk_get() to make cleanup paths simpler.
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use devm_clk_get() to make cleanup paths simpler.
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Acked-by: Peter Korsgaard <peter@korsgaard.com>
Acked-by: Nicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
1) SSYNC instruction is blackfin specific and takes no effect in this driver.
2) DMA descriptor and SG middle buffer are in DMA coherent memory. No need
to flush.
3) Turn kzalloc, ioremap and request_irq into managed device APIs respectively.
Signed-off-by: Sonic Zhang <sonic.zhang@analog.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
When the crypto layer is able to queue up a command for processing
by the CCP on the initial call to ccp_crypto_enqueue_request and
the CCP returns -EBUSY, then if the backlog flag is not set the
command needs to be freed and not added to the active command list.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Invoke the callback routine associated with the crypto context
if an error is encountered sending the command to the CCP during
backlog processing. This is needed to free any resources used
by the command.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
If a CCP command has been queued for processing at the
crypto layer then, when dequeueing it for processing, the
"can backlog" flag must be set so that the request isn't
lost if the CCP backlog queue limit is reached.
Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
My guess is that this little endian configuration is never found in real
life, but if it were then the writel() arguments are in the wrong order
so the driver would crash immediately.
Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Acked-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This driver has never been hooked up in any board file, and cannot be
instantiated via device tree. I've been told that, at least on Tegra20,
the HW is slower at crypto than the main CPU. I have no test-case for
it. Hence, remove it.
Cc: Varun Wadekar <vwadekar@nvidia.com>
Signed-off-by: Stephen Warren <swarren@nvidia.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Add config and build options for the omap-des driver.
Signed-off-by: Joel Fernandes <joelf@ti.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Add omap-des driver with platform data for OMAP4/AM43xx. Support added for DES
ECB and CBC modes. Also add support for 3DES operation where 3 64-bit keys are
used to perform a DES encrypt-decrypt-encrypt (des3_ede) operation on a buffer.
Tests have been conducted with the CRYPTO test manager, and functionality is
verified at different page length alignments.
Signed-off-by: Joel Fernandes <joelf@ti.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
We should test the error case for each platform_get_irq() assignment and
propagate the error accordingly.
Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
Acked-by: Marek Vasut <marex@denx.de>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
We checked "nbytes < bsize" before so it can't happen here.
Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Acked-by: Jussi Kivilinna <jussi.kivilinna@iki.fi>
Acked-by: Johannes Götzfried <johannes.goetzfried@cs.fau.de>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use devm_ioremap_resource() in order to make the code simpler,
and remove redundant return value check of platform_get_resource()
because the value is checked by devm_ioremap_resource().
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use devm_ioremap_resource() in order to make the code simpler,
and remove redundant return value check of platform_get_resource()
because the value is checked by devm_ioremap_resource().
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Acked-by: Peter Korsgaard <jacmet@sunsite.dk>
Acked-by: Nicolas Ferre <nicolas.ferre@atmel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use devm_ioremap_resource() in order to make the code simpler,
and remove redundant return value check of platform_get_resource()
because the value is checked by devm_ioremap_resource().
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use devm_ioremap_resource() in order to make the code simpler,
and remove redundant return value check of platform_get_resource()
because the value is checked by devm_ioremap_resource().
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use devm_ioremap_resource() in order to make the code simpler,
and remove redundant return value check of platform_get_resource()
because the value is checked by devm_ioremap_resource().
Signed-off-by: Jingoo Han <jg1.han@samsung.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
We added the soft module dependency of crc32c module alias
to generic crc32c module so other hardware accelerated crc32c
modules could get loaded and used before the generic version.
We also renamed the crypto/crc32c.c containing the generic
crc32c crypto computation to crypto/crc32c_generic.c according
to convention.
Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
SEC ERA has to be retrieved by reading the "fsl,sec-era" property
from the device tree. This property is updated/filled in by
u-boot.
Signed-off-by: Alex Porosanu <alexandru.porosanu@freescale.com>
Reviewed-by: Horia Geanta <horia.geanta@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
My static checker complains that:
drivers/char/hw_random/core.c:341 hwrng_register()
warn: we tested 'old_rng' before and it was 'false'
The problem is that sometimes we test "if (!old_rng)" and sometimes we
test "if (must_register_misc)". The static checker knows they are
equivalent but a human being reading the code could easily be confused.
I have simplified the code by removing the "must_register_misc" variable
and I have removed the redundant check on "if (!old_rng)".
Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Reviewed-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
stmp_reset_block() may fail, so check its return value and propagate it in the
case of error.
Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
Acked-by: Marek Vasut <marex@denx.de>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Using devm_kzalloc() can make the code cleaner.
While at it, remove the devm_kzalloc error message as there is standard OOM
message done by the core.
Signed-off-by: Fabio Estevam <fabio.estevam@freescale.com>
Acked-by: Marek Vasut <marex@denx.de>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
