system/freebsd-src
system/freebsd-src
1
0
Fork 0
mirror of https://github.com/freebsd/freebsd-src synced 2024-09-16 06:52:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add kernel-side support for in-kernel TLS.

Browse source 
KTLS adds support for in-kernel framing and encryption of Transport
Layer Security (1.0-1.2) data on TCP sockets.  KTLS only supports
offload of TLS for transmitted data.  Key negotation must still be
performed in userland.  Once completed, transmit session keys for a
connection are provided to the kernel via a new TCP_TXTLS_ENABLE
socket option.  All subsequent data transmitted on the socket is
placed into TLS frames and encrypted using the supplied keys.

Any data written to a KTLS-enabled socket via write(2), aio_write(2),
or sendfile(2) is assumed to be application data and is encoded in TLS
frames with an application data type.  Individual records can be sent
with a custom type (e.g. handshake messages) via sendmsg(2) with a new
control message (TLS_SET_RECORD_TYPE) specifying the record type.

At present, rekeying is not supported though the in-kernel framework
should support rekeying.

KTLS makes use of the recently added unmapped mbufs to store TLS
frames in the socket buffer.  Each TLS frame is described by a single
ext_pgs mbuf.  The ext_pgs structure contains the header of the TLS
record (and trailer for encrypted records) as well as references to
the associated TLS session.

KTLS supports two primary methods of encrypting TLS frames: software
TLS and ifnet TLS.

Software TLS marks mbufs holding socket data as not ready via
M_NOTREADY similar to sendfile(2) when TLS framing information is
added to an unmapped mbuf in ktls_frame().  ktls_enqueue() is then
called to schedule TLS frames for encryption.  In the case of
sendfile_iodone() calls ktls_enqueue() instead of pru_ready() leaving
the mbufs marked M_NOTREADY until encryption is completed.  For other
writes (vn_sendfile when pages are available, write(2), etc.), the
PRUS_NOTREADY is set when invoking pru_send() along with invoking
ktls_enqueue().

A pool of worker threads (the "KTLS" kernel process) encrypts TLS
frames queued via ktls_enqueue().  Each TLS frame is temporarily
mapped using the direct map and passed to a software encryption
backend to perform the actual encryption.

(Note: The use of PHYS_TO_DMAP could be replaced with sf_bufs if
someone wished to make this work on architectures without a direct
map.)

KTLS supports pluggable software encryption backends.  Internally,
Netflix uses proprietary pure-software backends.  This commit includes
a simple backend in a new ktls_ocf.ko module that uses the kernel's
OpenCrypto framework to provide AES-GCM encryption of TLS frames.  As
a result, software TLS is now a bit of a misnomer as it can make use
of hardware crypto accelerators.

Once software encryption has finished, the TLS frame mbufs are marked
ready via pru_ready().  At this point, the encrypted data appears as
regular payload to the TCP stack stored in unmapped mbufs.

ifnet TLS permits a NIC to offload the TLS encryption and TCP
segmentation.  In this mode, a new send tag type (IF_SND_TAG_TYPE_TLS)
is allocated on the interface a socket is routed over and associated
with a TLS session.  TLS records for a TLS session using ifnet TLS are
not marked M_NOTREADY but are passed down the stack unencrypted.  The
ip_output_send() and ip6_output_send() helper functions that apply
send tags to outbound IP packets verify that the send tag of the TLS
record matches the outbound interface.  If so, the packet is tagged
with the TLS send tag and sent to the interface.  The NIC device
driver must recognize packets with the TLS send tag and schedule them
for TLS encryption and TCP segmentation.  If the the outbound
interface does not match the interface in the TLS send tag, the packet
is dropped.  In addition, a task is scheduled to refresh the TLS send
tag for the TLS session.  If a new TLS send tag cannot be allocated,
the connection is dropped.  If a new TLS send tag is allocated,
however, subsequent packets will be tagged with the correct TLS send
tag.  (This latter case has been tested by configuring both ports of a
Chelsio T6 in a lagg and failing over from one port to another.  As
the connections migrated to the new port, new TLS send tags were
allocated for the new port and connections resumed without being
dropped.)

ifnet TLS can be enabled and disabled on supported network interfaces
via new '[-]txtls[46]' options to ifconfig(8).  ifnet TLS is supported
across both vlan devices and lagg interfaces using failover, lacp with
flowid enabled, or lacp with flowid enabled.

Applications may request the current KTLS mode of a connection via a
new TCP_TXTLS_MODE socket option.  They can also use this socket
option to toggle between software and ifnet TLS modes.

In addition, a testing tool is available in tools/tools/switch_tls.
This is modeled on tcpdrop and uses similar syntax.  However, instead
of dropping connections, -s is used to force KTLS connections to
switch to software TLS and -i is used to switch to ifnet TLS.

Various sysctls and counters are available under the kern.ipc.tls
sysctl node.  The kern.ipc.tls.enable node must be set to true to
enable KTLS (it is off by default).  The use of unmapped mbufs must
also be enabled via kern.ipc.mb_use_ext_pgs to enable KTLS.

KTLS is enabled via the KERN_TLS kernel option.

This patch is the culmination of years of work by several folks
including Scott Long and Randall Stewart for the original design and
implementation; Drew Gallatin for several optimizations including the
use of ext_pgs mbufs, the M_NOTREADY mechanism for TLS records
awaiting software encryption, and pluggable software crypto backends;
and John Baldwin for modifications to support hardware TLS offload.

Reviewed by:	gallatin, hselasky, rrs
Obtained from:	Netflix
Sponsored by:	Netflix, Chelsio Communications
Differential Revision:	https://reviews.freebsd.org/D21277
This commit is contained in:
John Baldwin 2019-08-27 00:01:56 +00:00
parent a70e17eeca
commit b2e60773c6
Notes: svn2git 2020-12-20 02:59:44 +00:00 
svn path=/head/; revision=351522
34 changed files with 3014 additions and 47 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

24
sbin/ifconfig/ifconfig.8
View file

@ -28,7 +28,7 @@
.\" From: @(#)ifconfig.8 8.3 (Berkeley) 1/5/94
.\" $FreeBSD$
.\"
.Dd August 15, 2019
.Dd August 26, 2019
.Dt IFCONFIG 8
.Os
.Sh NAME
@ -538,6 +538,28 @@ large receive offloading, enable LRO on the interface.
If the driver supports
.Xr tcp 4
large receive offloading, disable LRO on the interface.
.It Cm txtls
Transmit TLS offload encrypts Transport Layer Security (TLS) records and
segments the encrypted record into one or more
.Xr tcp 4
segments over either
.Xr ip 4
or
.Xr ip6 4 .
If the driver supports transmit TLS offload,
enable transmit TLS offload on the interface.
Some drivers may not be able to support transmit TLS offload for
.Xr ip 4
and
.Xr ip6 4
packets, so they may enable only one of them.
.It Fl txtls
If the driver supports transmit TLS offload,
disable transmit TLS offload on the interface.
It will always disable TLS for
.Xr ip 4
and
.Xr ip6 4 .
.It Cm nomap
If the driver supports unmapped network buffers,
enable them on the interface.

4
sbin/ifconfig/ifconfig.c
View file

@ -1257,7 +1257,7 @@ unsetifdescr(const char *val, int value, int s, const struct afswtch *afp)
"\020\1RXCSUM\2TXCSUM\3NETCONS\4VLAN_MTU\5VLAN_HWTAGGING\6JUMBO_MTU\7POLLING" \
"\10VLAN_HWCSUM\11TSO4\12TSO6\13LRO\14WOL_UCAST\15WOL_MCAST\16WOL_MAGIC" \
"\17TOE4\20TOE6\21VLAN_HWFILTER\23VLAN_HWTSO\24LINKSTATE\25NETMAP" \
"\26RXCSUM_IPV6\27TXCSUM_IPV6\31TXRTLMT\32HWRXTSTMP\33NOMAP"
"\26RXCSUM_IPV6\27TXCSUM_IPV6\31TXRTLMT\32HWRXTSTMP\33NOMAP\34TXTLS4\35TXTLS6"
/*
* Print the status of the interface. If an address family was
@ -1585,6 +1585,8 @@ static struct cmd basic_cmds[] = {
DEF_CMD("-toe", -IFCAP_TOE, setifcap),
DEF_CMD("lro", IFCAP_LRO, setifcap),
DEF_CMD("-lro", -IFCAP_LRO, setifcap),
DEF_CMD("txtls", IFCAP_TXTLS, setifcap),
DEF_CMD("-txtls", -IFCAP_TXTLS, setifcap),
DEF_CMD("wol", IFCAP_WOL, setifcap),
DEF_CMD("-wol", -IFCAP_WOL, setifcap),
DEF_CMD("wol_ucast", IFCAP_WOL_UCAST, setifcap),

47
share/man/man4/tcp.4
View file

@ -34,7 +34,7 @@
.\" From: @(#)tcp.4 8.1 (Berkeley) 6/5/93
.\" $FreeBSD$
.\"
.Dd July 23, 2019
.Dd August 26, 2019
.Dt TCP 4
.Os
.Sh NAME
@ -293,6 +293,51 @@ If an SADB entry cannot be found for the destination,
the system does not send any outgoing segments and drops any inbound segments.
.Pp
Each dropped segment is taken into account in the TCP protocol statistics.
.It Dv TCP_TXTLS_ENABLE
Enable in-kernel Transport Layer Security (TLS) for data written to this
socket.
The
.Vt struct tls_so_enable
argument defines the encryption and authentication algorithms and keys
used to encrypt the socket data as well as the maximum TLS record
payload size.
.Pp
All data written to this socket will be encapsulated in TLS records
and subsequently encrypted.
By default all data written to this socket is treated as application data.
Individual TLS records with a type other than application data
(for example, handshake messages),
may be transmitted by invoking
.Xr sendmsg 2
with a custom TLS record type set in a
.Dv TLS_SET_RECORD_TYPE
control message.
The payload of this control message is a single byte holding the desired
TLS record type.
.Pp
Data read from this socket will still be encrypted and must be parsed by
a TLS-aware consumer.
.Pp
At present, only a single key may be set on a socket.
As such, users of this option must disable rekeying.
.It Dv TCP_TXTLS_MODE
The integer argument can be used to get or set the current TLS mode of a
socket.
Setting the mode can only used to toggle between software and NIC TLS after
TLS has been initially enabled via the
.Dv TCP_TXTLS_ENABLE
option.
The available modes are:
.Bl -tag -width "Dv TCP_TLS_MODE_IFNET"
.It Dv TCP_TLS_MODE_NONE
In-kernel TLS framing and encryption is not enabled for this socket.
.It Dv TCP_TLS_MODE_SW
TLS records are encrypted by the kernel prior to placing the data in the
socket buffer.
Typically this encryption is performed in software.
.It Dv TCP_TLS_MODE_IFNET
TLS records are encrypted by the network interface card (NIC).
.El
.El
.Pp
The option level for the

4
sys/conf/NOTES
View file

@ -654,6 +654,10 @@ options IPSEC #IP security (requires device crypto)
options IPSEC_SUPPORT
#options IPSEC_DEBUG #debug for IP security
# TLS framing and encryption of data transmitted over TCP sockets.
options KERN_TLS # TLS transmit offload
#
# SMB/CIFS requester
# NETSMB enables support for SMB protocol, it requires LIBMCHAIN and LIBICONV

1
sys/conf/files
View file

@ -3862,6 +3862,7 @@ kern/tty_ttydisc.c standard
kern/uipc_accf.c standard
kern/uipc_debug.c optional ddb
kern/uipc_domain.c standard
kern/uipc_ktls.c optional kern_tls
kern/uipc_mbuf.c standard
kern/uipc_mbuf2.c standard
kern/uipc_mbufhash.c standard

1
sys/conf/options
View file

@ -435,6 +435,7 @@ IPSEC opt_ipsec.h
IPSEC_DEBUG opt_ipsec.h
IPSEC_SUPPORT opt_ipsec.h
IPSTEALTH
KERN_TLS
KRPC
LIBALIAS
LIBMCHAIN

25
sys/kern/kern_mbuf.c
View file

@ -31,6 +31,7 @@
__FBSDID("$FreeBSD$");
#include "opt_param.h"
#include "opt_kern_tls.h"
#include <sys/param.h>
#include <sys/conf.h>
@ -41,10 +42,12 @@ __FBSDID("$FreeBSD$");
#include <sys/domain.h>
#include <sys/eventhandler.h>
#include <sys/kernel.h>
#include <sys/ktls.h>
#include <sys/limits.h>
#include <sys/lock.h>
#include <sys/mutex.h>
#include <sys/protosw.h>
#include <sys/refcount.h>
#include <sys/sf_buf.h>
#include <sys/smp.h>
#include <sys/socket.h>
@ -112,10 +115,10 @@ int nmbjumbop; /* limits number of page size jumbo clusters */
int nmbjumbo9; /* limits number of 9k jumbo clusters */
int nmbjumbo16; /* limits number of 16k jumbo clusters */
bool mb_use_ext_pgs; /* use EXT_PGS mbufs for sendfile */
bool mb_use_ext_pgs; /* use EXT_PGS mbufs for sendfile & TLS */
SYSCTL_BOOL(_kern_ipc, OID_AUTO, mb_use_ext_pgs, CTLFLAG_RWTUN,
&mb_use_ext_pgs, 0,
"Use unmapped mbufs for sendfile(2)");
"Use unmapped mbufs for sendfile(2) and TLS offload");
static quad_t maxmbufmem; /* overall real memory limit for all mbufs */
@ -1281,13 +1284,27 @@ mb_free_ext(struct mbuf *m)
uma_zfree(zone_jumbo16, m->m_ext.ext_buf);
uma_zfree(zone_mbuf, mref);
break;
case EXT_PGS:
case EXT_PGS: {
#ifdef KERN_TLS
struct mbuf_ext_pgs *pgs;
struct ktls_session *tls;
#endif
KASSERT(mref->m_ext.ext_free != NULL,
("%s: ext_free not set", __func__));
mref->m_ext.ext_free(mref);
uma_zfree(zone_extpgs, mref->m_ext.ext_pgs);
#ifdef KERN_TLS
pgs = mref->m_ext.ext_pgs;
tls = pgs->tls;
if (tls != NULL &&
!refcount_release_if_not_last(&tls->refcount))
ktls_enqueue_to_free(pgs);
else
#endif
uma_zfree(zone_extpgs, mref->m_ext.ext_pgs);
uma_zfree(zone_mbuf, mref);
break;
}
case EXT_SFBUF:
case EXT_NET_DRV:
case EXT_MOD_TYPE:

101
sys/kern/kern_sendfile.c
View file

@ -30,12 +30,15 @@
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
#include "opt_kern_tls.h"
#include <sys/param.h>
#include <sys/systm.h>
#include <sys/capsicum.h>
#include <sys/kernel.h>
#include <netinet/in.h>
#include <sys/lock.h>
#include <sys/ktls.h>
#include <sys/mutex.h>
#include <sys/sysproto.h>
#include <sys/malloc.h>
@ -85,6 +88,7 @@ struct sf_io {
int npages;
struct socket *so;
struct mbuf *m;
struct ktls_session *tls;
vm_page_t pa[];
};
@ -262,6 +266,15 @@ sendfile_iodone(void *arg, vm_page_t *pg, int count, int error)
if (!refcount_release(&sfio->nios))
return;
#ifdef INVARIANTS
if ((sfio->m->m_flags & M_EXT) != 0 &&
sfio->m->m_ext.ext_type == EXT_PGS)
KASSERT(sfio->tls == sfio->m->m_ext.ext_pgs->tls,
("TLS session mismatch"));
else
KASSERT(sfio->tls == NULL,
("non-ext_pgs mbuf with TLS session"));
#endif
CURVNET_SET(so->so_vnet);
if (sfio->error) {
/*
@ -279,12 +292,29 @@ sendfile_iodone(void *arg, vm_page_t *pg, int count, int error)
so->so_error = EIO;
mb_free_notready(sfio->m, sfio->npages);
#ifdef KERN_TLS
} else if (sfio->tls != NULL && sfio->tls->sw_encrypt != NULL) {
/*
* I/O operation is complete, but we still need to
* encrypt. We cannot do this in the interrupt thread
* of the disk controller, so forward the mbufs to a
* different thread.
*
* Donate the socket reference from sfio to rather
* than explicitly invoking soref().
*/
ktls_enqueue(sfio->m, so, sfio->npages);
goto out_with_ref;
#endif
} else
(void)(so->so_proto->pr_usrreqs->pru_ready)(so, sfio->m,
sfio->npages);
SOCK_LOCK(so);
sorele(so);
#ifdef KERN_TLS
out_with_ref:
#endif
CURVNET_RESTORE();
free(sfio, M_TEMP);
}
@ -526,6 +556,9 @@ vn_sendfile(struct file *fp, int sockfd, struct uio *hdr_uio,
struct vnode *vp;
struct vm_object *obj;
struct socket *so;
#ifdef KERN_TLS
struct ktls_session *tls;
#endif
struct mbuf_ext_pgs *ext_pgs;
struct mbuf *m, *mh, *mhtail;
struct sf_buf *sf;
@ -534,12 +567,18 @@ vn_sendfile(struct file *fp, int sockfd, struct uio *hdr_uio,
struct vattr va;
off_t off, sbytes, rem, obj_size;
int bsize, error, ext_pgs_idx, hdrlen, max_pgs, softerr;
#ifdef KERN_TLS
int tls_enq_cnt;
#endif
bool use_ext_pgs;
obj = NULL;
so = NULL;
m = mh = NULL;
sfs = NULL;
#ifdef KERN_TLS
tls = NULL;
#endif
hdrlen = sbytes = 0;
softerr = 0;
use_ext_pgs = false;
@ -576,6 +615,9 @@ vn_sendfile(struct file *fp, int sockfd, struct uio *hdr_uio,
* we implement that, but possibly shouldn't.
*/
(void)sblock(&so->so_snd, SBL_WAIT | SBL_NOINTR);
#ifdef KERN_TLS
tls = ktls_hold(so->so_snd.sb_tls_info);
#endif
/*
* Loop through the pages of the file, starting with the requested
@ -669,7 +711,14 @@ vn_sendfile(struct file *fp, int sockfd, struct uio *hdr_uio,
if (hdr_uio != NULL && hdr_uio->uio_resid > 0) {
hdr_uio->uio_td = td;
hdr_uio->uio_rw = UIO_WRITE;
mh = m_uiotombuf(hdr_uio, M_WAITOK, space, 0, 0);
#ifdef KERN_TLS
if (tls != NULL)
mh = m_uiotombuf(hdr_uio, M_WAITOK, space,
tls->params.max_frame_len, M_NOMAP);
else
#endif
mh = m_uiotombuf(hdr_uio, M_WAITOK,
space, 0, 0);
hdrlen = m_length(mh, &mhtail);
space -= hdrlen;
/*
@ -743,6 +792,15 @@ vn_sendfile(struct file *fp, int sockfd, struct uio *hdr_uio,
sfio->so = so;
sfio->error = 0;
#ifdef KERN_TLS
/*
* This doesn't use ktls_hold() because sfio->m will
* also have a reference on 'tls' that will be valid
* for all of sfio's lifetime.
*/
sfio->tls = tls;
#endif
error = sendfile_swapin(obj, sfio, &nios, off, space, npages,
rhpages, flags);
if (error != 0) {
@ -763,11 +821,22 @@ vn_sendfile(struct file *fp, int sockfd, struct uio *hdr_uio,
* bufs are restricted to TCP as that is what has been
* tested. In particular, unmapped mbufs have not
* been tested with UNIX-domain sockets.
*
* TLS frames always require unmapped mbufs.
*/
if (mb_use_ext_pgs &&
so->so_proto->pr_protocol == IPPROTO_TCP) {
if ((mb_use_ext_pgs &&
so->so_proto->pr_protocol == IPPROTO_TCP)
#ifdef KERN_TLS
|| tls != NULL
#endif
) {
use_ext_pgs = true;
max_pgs = MBUF_PEXT_MAX_PGS;
#ifdef KERN_TLS
if (tls != NULL)
max_pgs = num_pages(tls->params.max_frame_len);
else
#endif
max_pgs = MBUF_PEXT_MAX_PGS;
/* Start at last index, to wrap on first use. */
ext_pgs_idx = max_pgs - 1;
@ -946,6 +1015,14 @@ vn_sendfile(struct file *fp, int sockfd, struct uio *hdr_uio,
__func__, m_length(m, NULL), space, hdrlen));
CURVNET_SET(so->so_vnet);
#ifdef KERN_TLS
if (tls != NULL) {
error = ktls_frame(m, tls, &tls_enq_cnt,
TLS_RLTYPE_APP);
if (error != 0)
goto done;
}
#endif
if (nios == 0) {
/*
* If sendfile_swapin() didn't initiate any I/Os,
@ -954,8 +1031,16 @@ vn_sendfile(struct file *fp, int sockfd, struct uio *hdr_uio,
* PRUS_NOTREADY flag.
*/
free(sfio, M_TEMP);
error = (*so->so_proto->pr_usrreqs->pru_send)
(so, 0, m, NULL, NULL, td);
#ifdef KERN_TLS
if (tls != NULL && tls->sw_encrypt != NULL) {
error = (*so->so_proto->pr_usrreqs->pru_send)
(so, PRUS_NOTREADY, m, NULL, NULL, td);
soref(so);
ktls_enqueue(m, so, tls_enq_cnt);
} else
#endif
error = (*so->so_proto->pr_usrreqs->pru_send)
(so, 0, m, NULL, NULL, td);
} else {
sfio->npages = npages;
soref(so);
@ -1019,6 +1104,10 @@ vn_sendfile(struct file *fp, int sockfd, struct uio *hdr_uio,
mtx_destroy(&sfs->mtx);
free(sfs, M_TEMP);
}
#ifdef KERN_TLS
if (tls != NULL)
ktls_free(tls);
#endif
if (error == ERESTART)
error = EINTR;

1450
sys/kern/uipc_ktls.c Normal file
View file

File diff suppressed because it is too large Load diff

22
sys/kern/uipc_sockbuf.c
View file

@ -34,11 +34,13 @@
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
#include "opt_kern_tls.h"
#include "opt_param.h"
#include <sys/param.h>
#include <sys/aio.h> /* for aio_swake proto */
#include <sys/kernel.h>
#include <sys/ktls.h>
#include <sys/lock.h>
#include <sys/malloc.h>
#include <sys/mbuf.h>
@ -112,7 +114,8 @@ sbready_compress(struct sockbuf *sb, struct mbuf *m0, struct mbuf *end)
MPASS((m->m_flags & M_NOTREADY) == 0);
/* Compress small unmapped mbufs into plain mbufs. */
if ((m->m_flags & M_NOMAP) && m->m_len <= MLEN) {
if ((m->m_flags & M_NOMAP) && m->m_len <= MLEN &&
!mbuf_has_tls_session(m)) {
MPASS(m->m_flags & M_EXT);
ext_size = m->m_ext.ext_size;
if (mb_unmapped_compress(m) == 0) {
@ -133,6 +136,8 @@ sbready_compress(struct sockbuf *sb, struct mbuf *m0, struct mbuf *end)
while ((n != NULL) && (n != end) && (m->m_flags & M_EOR) == 0 &&
M_WRITABLE(m) &&
(m->m_flags & M_NOMAP) == 0 &&
!mbuf_has_tls_session(n) &&
!mbuf_has_tls_session(m) &&
n->m_len <= MCLBYTES / 4 && /* XXX: Don't copy too much */
n->m_len <= M_TRAILINGSPACE(m) &&
m->m_type == n->m_type) {
@ -668,6 +673,11 @@ sbdestroy(struct sockbuf *sb, struct socket *so)
{
sbrelease_internal(sb, so);
#ifdef KERN_TLS
if (sb->sb_tls_info != NULL)
ktls_free(sb->sb_tls_info);
sb->sb_tls_info = NULL;
#endif
}
/*
@ -831,6 +841,11 @@ sbappendstream_locked(struct sockbuf *sb, struct mbuf *m, int flags)
SBLASTMBUFCHK(sb);
#ifdef KERN_TLS
if (sb->sb_tls_info != NULL)
ktls_seq(sb, m);
#endif
/* Remove all packet headers and mbuf tags to get a pure data chain. */
m_demote(m, 1, flags & PRUS_NOTREADY ? M_NOTREADY : 0);
@ -1134,6 +1149,8 @@ sbcompress(struct sockbuf *sb, struct mbuf *m, struct mbuf *n)
((sb->sb_flags & SB_NOCOALESCE) == 0) &&
!(m->m_flags & M_NOTREADY) &&
!(n->m_flags & (M_NOTREADY | M_NOMAP)) &&
!mbuf_has_tls_session(m) &&
!mbuf_has_tls_session(n) &&
m->m_len <= MCLBYTES / 4 && /* XXX: Don't copy too much */
m->m_len <= M_TRAILINGSPACE(n) &&
n->m_type == m->m_type) {
@ -1149,7 +1166,8 @@ sbcompress(struct sockbuf *sb, struct mbuf *m, struct mbuf *n)
continue;
}
if (m->m_len <= MLEN && (m->m_flags & M_NOMAP) &&
(m->m_flags & M_NOTREADY) == 0)
(m->m_flags & M_NOTREADY) == 0 &&
!mbuf_has_tls_session(m))
(void)mb_unmapped_compress(m);
if (n)
n->m_next = m;

94
sys/kern/uipc_socket.c
View file

@ -107,6 +107,7 @@ __FBSDID("$FreeBSD$");
#include "opt_inet.h"
#include "opt_inet6.h"
#include "opt_kern_tls.h"
#include "opt_sctp.h"
#include <sys/param.h>
@ -123,6 +124,7 @@ __FBSDID("$FreeBSD$");
#include <sys/hhook.h>
#include <sys/kernel.h>
#include <sys/khelp.h>
#include <sys/ktls.h>
#include <sys/event.h>
#include <sys/eventhandler.h>
#include <sys/poll.h>
@ -141,6 +143,7 @@ __FBSDID("$FreeBSD$");
#include <sys/jail.h>
#include <sys/syslog.h>
#include <netinet/in.h>
#include <netinet/tcp.h>
#include <net/vnet.h>
@ -1442,7 +1445,15 @@ sosend_generic(struct socket *so, struct sockaddr *addr, struct uio *uio,
ssize_t resid;
int clen = 0, error, dontroute;
int atomic = sosendallatonce(so) || top;
int pru_flag;
#ifdef KERN_TLS
struct ktls_session *tls;
int tls_enq_cnt, tls_pruflag;
uint8_t tls_rtype;
tls = NULL;
tls_rtype = TLS_RLTYPE_APP;
#endif
if (uio != NULL)
resid = uio->uio_resid;
else
@ -1474,6 +1485,28 @@ sosend_generic(struct socket *so, struct sockaddr *addr, struct uio *uio,
if (error)
goto out;
#ifdef KERN_TLS
tls_pruflag = 0;
tls = ktls_hold(so->so_snd.sb_tls_info);
if (tls != NULL) {
if (tls->sw_encrypt != NULL)
tls_pruflag = PRUS_NOTREADY;
if (control != NULL) {
struct cmsghdr *cm = mtod(control, struct cmsghdr *);
if (clen >= sizeof(*cm) &&
cm->cmsg_type == TLS_SET_RECORD_TYPE) {
tls_rtype = *((uint8_t *)CMSG_DATA(cm));
clen = 0;
m_freem(control);
control = NULL;
atomic = 1;
}
}
}
#endif
restart:
do {
SOCKBUF_LOCK(&so->so_snd);
@ -1551,10 +1584,27 @@ sosend_generic(struct socket *so, struct sockaddr *addr, struct uio *uio,
* is a workaround to prevent protocol send
* methods to panic.
*/
top = m_uiotombuf(uio, M_WAITOK, space,
(atomic ? max_hdr : 0),
(atomic ? M_PKTHDR : 0) |
((flags & MSG_EOR) ? M_EOR : 0));
#ifdef KERN_TLS
if (tls != NULL) {
top = m_uiotombuf(uio, M_WAITOK, space,
tls->params.max_frame_len,
M_NOMAP |
((flags & MSG_EOR) ? M_EOR : 0));
if (top != NULL) {
error = ktls_frame(top, tls,
&tls_enq_cnt, tls_rtype);
if (error) {
m_freem(top);
goto release;
}
}
tls_rtype = TLS_RLTYPE_APP;
} else
#endif
top = m_uiotombuf(uio, M_WAITOK, space,
(atomic ? max_hdr : 0),
(atomic ? M_PKTHDR : 0) |
((flags & MSG_EOR) ? M_EOR : 0));
if (top == NULL) {
error = EFAULT; /* only possible error */
goto release;
@ -1578,8 +1628,8 @@ sosend_generic(struct socket *so, struct sockaddr *addr, struct uio *uio,
* this.
*/
VNET_SO_ASSERT(so);
error = (*so->so_proto->pr_usrreqs->pru_send)(so,
(flags & MSG_OOB) ? PRUS_OOB :
pru_flag = (flags & MSG_OOB) ? PRUS_OOB :
/*
* If the user set MSG_EOF, the protocol understands
* this flag and nothing left to send then use
@ -1591,13 +1641,37 @@ sosend_generic(struct socket *so, struct sockaddr *addr, struct uio *uio,
PRUS_EOF :
/* If there is more to send set PRUS_MORETOCOME. */
(flags & MSG_MORETOCOME) ||
(resid > 0 && space > 0) ? PRUS_MORETOCOME : 0,
top, addr, control, td);
(resid > 0 && space > 0) ? PRUS_MORETOCOME : 0;
#ifdef KERN_TLS
pru_flag |= tls_pruflag;
#endif
error = (*so->so_proto->pr_usrreqs->pru_send)(so,
pru_flag, top, addr, control, td);
if (dontroute) {
SOCK_LOCK(so);
so->so_options &= ~SO_DONTROUTE;
SOCK_UNLOCK(so);
}
#ifdef KERN_TLS
if (tls != NULL && tls->sw_encrypt != NULL) {
/*
* Note that error is intentionally
* ignored.
*
* Like sendfile(), we rely on the
* completion routine (pru_ready())
* to free the mbufs in the event that
* pru_send() encountered an error and
* did not append them to the sockbuf.
*/
soref(so);
ktls_enqueue(top, so, tls_enq_cnt);
}
#endif
clen = 0;
control = NULL;
top = NULL;
@ -1609,6 +1683,10 @@ sosend_generic(struct socket *so, struct sockaddr *addr, struct uio *uio,
release:
sbunlock(&so->so_snd);
out:
#ifdef KERN_TLS
if (tls != NULL)
ktls_free(tls);
#endif
if (top != NULL)
m_freem(top);
if (control != NULL)

2
sys/modules/Makefile
View file

@ -200,6 +200,7 @@ SUBDIR= \
khelp \
krpc \
ksyms \
${_ktls_ocf} \
le \
lge \
libalias \
@ -412,6 +413,7 @@ _crypto= crypto
_cryptodev= cryptodev
_random_fortuna=random_fortuna
_random_other= random_other
_ktls_ocf= ktls_ocf
.endif
.endif

8
sys/modules/ktls_ocf/Makefile Normal file
View file

@ -0,0 +1,8 @@
# $FreeBSD$
.PATH: ${SRCTOP}/sys/opencrypto
KMOD= ktls_ocf
SRCS= ktls_ocf.c
.include <bsd.kmod.mk>

3
sys/net/ieee8023ad_lacp.c
View file

@ -32,6 +32,7 @@
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
#include "opt_kern_tls.h"
#include "opt_ratelimit.h"
#include <sys/param.h>
@ -882,7 +883,7 @@ lacp_select_tx_port(struct lagg_softc *sc, struct mbuf *m)
return (lp->lp_lagg);
}
#ifdef RATELIMIT
#if defined(RATELIMIT) || defined(KERN_TLS)
struct lagg_port *
lacp_select_tx_port_by_hash(struct lagg_softc *sc, uint32_t flowid)
{

2
sys/net/ieee8023ad_lacp.h
View file

@ -293,7 +293,7 @@ struct lacp_softc {
struct mbuf *lacp_input(struct lagg_port *, struct mbuf *);
struct lagg_port *lacp_select_tx_port(struct lagg_softc *, struct mbuf *);
#ifdef RATELIMIT
#if defined(RATELIMIT) || defined(KERN_TLS)
struct lagg_port *lacp_select_tx_port_by_hash(struct lagg_softc *, uint32_t);
#endif
void lacp_attach(struct lagg_softc *);

3
sys/net/if.h
View file

@ -247,6 +247,8 @@ struct if_data {
#define IFCAP_TXRTLMT 0x1000000 /* hardware supports TX rate limiting */
#define IFCAP_HWRXTSTMP 0x2000000 /* hardware rx timestamping */
#define IFCAP_NOMAP 0x4000000 /* can TX unmapped mbufs */
#define IFCAP_TXTLS4 0x8000000 /* can do TLS encryption and segmentation for TCP */
#define IFCAP_TXTLS6 0x10000000 /* can do TLS encryption and segmentation for TCP6 */
#define IFCAP_HWCSUM_IPV6 (IFCAP_RXCSUM_IPV6 | IFCAP_TXCSUM_IPV6)
@ -254,6 +256,7 @@ struct if_data {
#define IFCAP_TSO (IFCAP_TSO4 | IFCAP_TSO6)
#define IFCAP_WOL (IFCAP_WOL_UCAST | IFCAP_WOL_MCAST | IFCAP_WOL_MAGIC)
#define IFCAP_TOE (IFCAP_TOE4 | IFCAP_TOE6)
#define IFCAP_TXTLS (IFCAP_TXTLS4 | IFCAP_TXTLS6)
#define IFCAP_CANTCHANGE (IFCAP_NETMAP)

11
sys/net/if_lagg.c
View file

@ -23,6 +23,7 @@ __FBSDID("$FreeBSD$");
#include "opt_inet.h"
#include "opt_inet6.h"
#include "opt_kern_tls.h"
#include "opt_ratelimit.h"
#include <sys/param.h>
@ -135,7 +136,7 @@ static void lagg_port2req(struct lagg_port *, struct lagg_reqport *);
static void lagg_init(void *);
static void lagg_stop(struct lagg_softc *);
static int lagg_ioctl(struct ifnet *, u_long, caddr_t);
#ifdef RATELIMIT
#if defined(KERN_TLS) || defined(RATELIMIT)
static int lagg_snd_tag_alloc(struct ifnet *,
union if_snd_tag_alloc_params *,
struct m_snd_tag **);
@ -534,7 +535,7 @@ lagg_clone_create(struct if_clone *ifc, int unit, caddr_t params)
ifp->if_ioctl = lagg_ioctl;
ifp->if_get_counter = lagg_get_counter;
ifp->if_flags = IFF_SIMPLEX | IFF_BROADCAST | IFF_MULTICAST;
#ifdef RATELIMIT
#if defined(KERN_TLS) || defined(RATELIMIT)
ifp->if_snd_tag_alloc = lagg_snd_tag_alloc;
ifp->if_snd_tag_modify = lagg_snd_tag_modify;
ifp->if_snd_tag_query = lagg_snd_tag_query;
@ -1550,7 +1551,7 @@ lagg_ioctl(struct ifnet *ifp, u_long cmd, caddr_t data)
return (error);
}
#ifdef RATELIMIT
#if defined(KERN_TLS) || defined(RATELIMIT)
static inline struct lagg_snd_tag *
mst_to_lst(struct m_snd_tag *mst)
{
@ -1811,7 +1812,7 @@ lagg_transmit(struct ifnet *ifp, struct mbuf *m)
struct lagg_softc *sc = (struct lagg_softc *)ifp->if_softc;
int error;
#ifdef RATELIMIT
#if defined(KERN_TLS) || defined(RATELIMIT)
if (m->m_pkthdr.csum_flags & CSUM_SND_TAG)
MPASS(m->m_pkthdr.snd_tag->ifp == ifp);
#endif
@ -2007,7 +2008,7 @@ int
lagg_enqueue(struct ifnet *ifp, struct mbuf *m)
{
#ifdef RATELIMIT
#if defined(KERN_TLS) || defined(RATELIMIT)
if (m->m_pkthdr.csum_flags & CSUM_SND_TAG) {
struct lagg_snd_tag *lst;
struct m_snd_tag *mst;

11
sys/net/if_var.h
View file

@ -188,11 +188,13 @@ struct if_encap_req {
* m_snd_tag" comes from the network driver and it is free to allocate
* as much additional space as it wants for its own use.
*/
struct ktls_session;
struct m_snd_tag;
#define IF_SND_TAG_TYPE_RATE_LIMIT 0
#define IF_SND_TAG_TYPE_UNLIMITED 1
#define IF_SND_TAG_TYPE_MAX 2
#define IF_SND_TAG_TYPE_TLS 2
#define IF_SND_TAG_TYPE_MAX 3
struct if_snd_tag_alloc_header {
uint32_t type; /* send tag type, see IF_SND_TAG_XXX */
@ -207,6 +209,12 @@ struct if_snd_tag_alloc_rate_limit {
uint32_t reserved; /* alignment */
};
struct if_snd_tag_alloc_tls {
struct if_snd_tag_alloc_header hdr;
struct inpcb *inp;
const struct ktls_session *tls;
};
struct if_snd_tag_rate_limit_params {
uint64_t max_rate; /* in bytes/s */
uint32_t queue_level; /* 0 (empty) .. 65535 (full) */
@ -219,6 +227,7 @@ union if_snd_tag_alloc_params {
struct if_snd_tag_alloc_header hdr;
struct if_snd_tag_alloc_rate_limit rate_limit;
struct if_snd_tag_alloc_rate_limit unlimited;
struct if_snd_tag_alloc_tls tls;
};
union if_snd_tag_modify_params {

25
sys/net/if_vlan.c
View file

@ -46,6 +46,7 @@
__FBSDID("$FreeBSD$");
#include "opt_inet.h"
#include "opt_kern_tls.h"
#include "opt_vlan.h"
#include "opt_ratelimit.h"
@ -103,7 +104,7 @@ struct ifvlantrunk {
int refcnt;
};
#ifdef RATELIMIT
#if defined(KERN_TLS) || defined(RATELIMIT)
struct vlan_snd_tag {
struct m_snd_tag com;
struct m_snd_tag *tag;
@ -278,7 +279,7 @@ static void trunk_destroy(struct ifvlantrunk *trunk);
static void vlan_init(void *foo);
static void vlan_input(struct ifnet *ifp, struct mbuf *m);
static int vlan_ioctl(struct ifnet *ifp, u_long cmd, caddr_t addr);
#ifdef RATELIMIT
#if defined(KERN_TLS) || defined(RATELIMIT)
static int vlan_snd_tag_alloc(struct ifnet *,
union if_snd_tag_alloc_params *, struct m_snd_tag **);
static int vlan_snd_tag_modify(struct m_snd_tag *,
@ -1064,7 +1065,7 @@ vlan_clone_create(struct if_clone *ifc, char *name, size_t len, caddr_t params)
ifp->if_transmit = vlan_transmit;
ifp->if_qflush = vlan_qflush;
ifp->if_ioctl = vlan_ioctl;
#ifdef RATELIMIT
#if defined(KERN_TLS) || defined(RATELIMIT)
ifp->if_snd_tag_alloc = vlan_snd_tag_alloc;
ifp->if_snd_tag_modify = vlan_snd_tag_modify;
ifp->if_snd_tag_query = vlan_snd_tag_query;
@ -1157,7 +1158,7 @@ vlan_transmit(struct ifnet *ifp, struct mbuf *m)
BPF_MTAP(ifp, m);
#ifdef RATELIMIT
#if defined(KERN_TLS) || defined(RATELIMIT)
if (m->m_pkthdr.csum_flags & CSUM_SND_TAG) {
struct vlan_snd_tag *vst;
struct m_snd_tag *mst;
@ -1741,6 +1742,20 @@ vlan_capabilities(struct ifvlan *ifv)
cap |= (p->if_capabilities & IFCAP_NOMAP);
ena |= (mena & IFCAP_NOMAP);
/*
* If the parent interface can offload encryption and segmentation
* of TLS records over TCP, propagate it's capability to the VLAN
* interface.
*
* All TLS drivers in the tree today can deal with VLANs. If
* this ever changes, then a new IFCAP_VLAN_TXTLS can be
* defined.
*/
if (p->if_capabilities & IFCAP_TXTLS)
cap |= p->if_capabilities & IFCAP_TXTLS;
if (p->if_capenable & IFCAP_TXTLS)
ena |= mena & IFCAP_TXTLS;
ifp->if_capabilities = cap;
ifp->if_capenable = ena;
ifp->if_hwassist = hwa;
@ -1972,7 +1987,7 @@ vlan_ioctl(struct ifnet *ifp, u_long cmd, caddr_t data)
return (error);
}
#ifdef RATELIMIT
#if defined(KERN_TLS) || defined(RATELIMIT)
static int
vlan_snd_tag_alloc(struct ifnet *ifp,
union if_snd_tag_alloc_params *params,

36
sys/netinet/ip_output.c
View file

@ -36,6 +36,7 @@ __FBSDID("$FreeBSD$");
#include "opt_inet.h"
#include "opt_ipsec.h"
#include "opt_kern_tls.h"
#include "opt_mbuf_stress_test.h"
#include "opt_mpath.h"
#include "opt_ratelimit.h"
@ -46,6 +47,7 @@ __FBSDID("$FreeBSD$");
#include <sys/param.h>
#include <sys/systm.h>
#include <sys/kernel.h>
#include <sys/ktls.h>
#include <sys/lock.h>
#include <sys/malloc.h>
#include <sys/mbuf.h>
@ -212,14 +214,39 @@ static int
ip_output_send(struct inpcb *inp, struct ifnet *ifp, struct mbuf *m,
const struct sockaddr_in *gw, struct route *ro)
{
#ifdef KERN_TLS
struct ktls_session *tls = NULL;
#endif
struct m_snd_tag *mst;
int error;
MPASS((m->m_pkthdr.csum_flags & CSUM_SND_TAG) == 0);
mst = NULL;
#ifdef KERN_TLS
/*
* If this is an unencrypted TLS record, save a reference to
* the record. This local reference is used to call
* ktls_output_eagain after the mbuf has been freed (thus
* dropping the mbuf's reference) in if_output.
*/
if (m->m_next != NULL && mbuf_has_tls_session(m->m_next)) {
tls = ktls_hold(m->m_next->m_ext.ext_pgs->tls);
mst = tls->snd_tag;
/*
* If a TLS session doesn't have a valid tag, it must
* have had an earlier ifp mismatch, so drop this
* packet.
*/
if (mst == NULL) {
error = EAGAIN;
goto done;
}
}
#endif
#ifdef RATELIMIT
if (inp != NULL) {
if (inp != NULL && mst == NULL) {
if ((inp->inp_flags2 & INP_RATE_LIMIT_CHANGED) != 0 ||
(inp->inp_snd_tag != NULL &&
inp->inp_snd_tag->ifp != ifp))
@ -246,6 +273,13 @@ ip_output_send(struct inpcb *inp, struct ifnet *ifp, struct mbuf *m,
done:
/* Check for route change invalidating send tags. */
#ifdef KERN_TLS
if (tls != NULL) {
if (error == EAGAIN)
error = ktls_output_eagain(inp, tls);
ktls_free(tls);
}
#endif
#ifdef RATELIMIT
if (error == EAGAIN)
in_pcboutput_eagain(inp);

12
sys/netinet/tcp.h
View file

@ -174,6 +174,8 @@ struct tcphdr {
#define TCP_LOGDUMP 37 /* dump connection log events to device */
#define TCP_LOGDUMPID 38 /* dump events from connections with same ID to
device */
#define TCP_TXTLS_ENABLE 39 /* TLS framing and encryption for transmit */
#define TCP_TXTLS_MODE 40 /* Transmit TLS mode */
#define TCP_CONGESTION 64 /* get/set congestion control algorithm */
#define TCP_CCALGOOPT 65 /* get/set cc algorithm specific options */
#define TCP_DELACK 72 /* socket option for delayed ack */
@ -350,4 +352,14 @@ struct tcp_function_set {
uint32_t pcbcnt;
};
/* TLS modes for TCP_TXTLS_MODE */
#define TCP_TLS_MODE_NONE 0
#define TCP_TLS_MODE_SW 1
#define TCP_TLS_MODE_IFNET 2
/*
* TCP Control message types
*/
#define TLS_SET_RECORD_TYPE 1
#endif /* !_NETINET_TCP_H_ */

58
sys/netinet/tcp_output.c
View file

@ -37,6 +37,7 @@ __FBSDID("$FreeBSD$");
#include "opt_inet.h"
#include "opt_inet6.h"
#include "opt_ipsec.h"
#include "opt_kern_tls.h"
#include "opt_tcpdebug.h"
#include <sys/param.h>
@ -46,6 +47,9 @@ __FBSDID("$FreeBSD$");
#include <sys/hhook.h>
#endif
#include <sys/kernel.h>
#ifdef KERN_TLS
#include <sys/ktls.h>
#endif
#include <sys/lock.h>
#include <sys/mbuf.h>
#include <sys/mutex.h>
@ -219,6 +223,11 @@ tcp_output(struct tcpcb *tp)
isipv6 = (tp->t_inpcb->inp_vflag & INP_IPV6) != 0;
#endif
#ifdef KERN_TLS
const bool hw_tls = (so->so_snd.sb_flags & SB_TLS_IFNET) != 0;
#else
const bool hw_tls = false;
#endif
INP_WLOCK_ASSERT(tp->t_inpcb);
@ -1000,7 +1009,7 @@ tcp_output(struct tcpcb *tp)
* to the offset in the socket buffer chain.
*/
mb = sbsndptr_noadv(&so->so_snd, off, &moff);
if (len <= MHLEN - hdrlen - max_linkhdr) {
if (len <= MHLEN - hdrlen - max_linkhdr && !hw_tls) {
m_copydata(mb, moff, len,
mtod(m, caddr_t) + hdrlen);
if (SEQ_LT(tp->snd_nxt, tp->snd_max))
@ -1013,7 +1022,7 @@ tcp_output(struct tcpcb *tp)
msb = &so->so_snd;
m->m_next = tcp_m_copym(mb, moff,
&len, if_hw_tsomaxsegcount,
if_hw_tsomaxsegsize, msb);
if_hw_tsomaxsegsize, msb, hw_tls);
if (len <= (tp->t_maxseg - optlen)) {
/*
* Must have ran out of mbufs for the copy
@ -1816,8 +1825,12 @@ tcp_addoptions(struct tcpopt *to, u_char *optp)
*/
struct mbuf *
tcp_m_copym(struct mbuf *m, int32_t off0, int32_t *plen,
int32_t seglimit, int32_t segsize, struct sockbuf *sb)
int32_t seglimit, int32_t segsize, struct sockbuf *sb, bool hw_tls)
{
#ifdef KERN_TLS
struct ktls_session *tls, *ntls;
struct mbuf *start;
#endif
struct mbuf *n, **np;
struct mbuf *top;
int32_t off = off0;
@ -1849,6 +1862,13 @@ tcp_m_copym(struct mbuf *m, int32_t off0, int32_t *plen,
np = &top;
top = NULL;
pkthdrlen = NULL;
#ifdef KERN_TLS
if (m->m_flags & M_NOMAP)
tls = m->m_ext.ext_pgs->tls;
else
tls = NULL;
start = m;
#endif
while (len > 0) {
if (m == NULL) {
KASSERT(len == M_COPYALL,
@ -1858,6 +1878,38 @@ tcp_m_copym(struct mbuf *m, int32_t off0, int32_t *plen,
*pkthdrlen = len_cp;
break;
}
#ifdef KERN_TLS
if (hw_tls) {
if (m->m_flags & M_NOMAP)
ntls = m->m_ext.ext_pgs->tls;
else
ntls = NULL;
/*
* Avoid mixing TLS records with handshake
* data or TLS records from different
* sessions.
*/
if (tls != ntls) {
MPASS(m != start);
*plen = len_cp;
if (pkthdrlen != NULL)
*pkthdrlen = len_cp;
break;
}
/*
* Don't end a send in the middle of a TLS
* record if it spans multiple TLS records.
*/
if (tls != NULL && (m != start) && len < m->m_len) {
*plen = len_cp;
if (pkthdrlen != NULL)
*pkthdrlen = len_cp;
break;
}
}
#endif
mlen = min(len, m->m_len - off);
if (seglimit) {
/*

11
sys/netinet/tcp_stacks/rack.c
View file

@ -6971,6 +6971,12 @@ rack_output(struct tcpcb *tp)
struct ip6_hdr *ip6 = NULL;
int32_t isipv6;
#endif
#ifdef KERN_TLS
const bool hw_tls = (so->so_snd.sb_flags & SB_TLS_IFNET) != 0;
#else
const bool hw_tls = false;
#endif
/* setup and take the cache hits here */
rack = (struct tcp_rack *)tp->t_fb_ptr;
inp = rack->rc_inp;
@ -7946,7 +7952,7 @@ rack_output(struct tcpcb *tp)
* sb_offset in the socket buffer chain.
*/
mb = sbsndptr_noadv(sb, sb_offset, &moff);
if (len <= MHLEN - hdrlen - max_linkhdr) {
if (len <= MHLEN - hdrlen - max_linkhdr && !hw_tls) {
m_copydata(mb, moff, (int)len,
mtod(m, caddr_t)+hdrlen);
if (SEQ_LT(tp->snd_nxt, tp->snd_max))
@ -7960,7 +7966,8 @@ rack_output(struct tcpcb *tp)
else
msb = sb;
m->m_next = tcp_m_copym(/*tp, */ mb, moff, &len,
if_hw_tsomaxsegcount, if_hw_tsomaxsegsize, msb /*, 0, NULL*/);
if_hw_tsomaxsegcount, if_hw_tsomaxsegsize, msb,
hw_tls /*, NULL */);
if (len <= (tp->t_maxseg - optlen)) {
/*
* Must have ran out of mbufs for the copy

118
sys/netinet/tcp_subr.c
View file

@ -37,6 +37,7 @@ __FBSDID("$FreeBSD$");
#include "opt_inet.h"
#include "opt_inet6.h"
#include "opt_ipsec.h"
#include "opt_kern_tls.h"
#include "opt_tcpdebug.h"
#include <sys/param.h>
@ -50,6 +51,9 @@ __FBSDID("$FreeBSD$");
#ifdef TCP_HHOOK
#include <sys/khelp.h>
#endif
#ifdef KERN_TLS
#include <sys/ktls.h>
#endif
#include <sys/sysctl.h>
#include <sys/jail.h>
#include <sys/malloc.h>
@ -3076,6 +3080,120 @@ SYSCTL_PROC(_net_inet_tcp, TCPCTL_DROP, drop,
CTLFLAG_VNET | CTLTYPE_STRUCT | CTLFLAG_WR | CTLFLAG_SKIP, NULL,
0, sysctl_drop, "", "Drop TCP connection");
#ifdef KERN_TLS
static int
sysctl_switch_tls(SYSCTL_HANDLER_ARGS)
{
/* addrs[0] is a foreign socket, addrs[1] is a local one. */
struct sockaddr_storage addrs[2];
struct inpcb *inp;
struct sockaddr_in *fin, *lin;
struct epoch_tracker et;
#ifdef INET6
struct sockaddr_in6 *fin6, *lin6;
#endif
int error;
inp = NULL;
fin = lin = NULL;
#ifdef INET6
fin6 = lin6 = NULL;
#endif
error = 0;
if (req->oldptr != NULL || req->oldlen != 0)
return (EINVAL);
if (req->newptr == NULL)
return (EPERM);
if (req->newlen < sizeof(addrs))
return (ENOMEM);
error = SYSCTL_IN(req, &addrs, sizeof(addrs));
if (error)
return (error);
switch (addrs[0].ss_family) {
#ifdef INET6
case AF_INET6:
fin6 = (struct sockaddr_in6 *)&addrs[0];
lin6 = (struct sockaddr_in6 *)&addrs[1];
if (fin6->sin6_len != sizeof(struct sockaddr_in6) ||
lin6->sin6_len != sizeof(struct sockaddr_in6))
return (EINVAL);
if (IN6_IS_ADDR_V4MAPPED(&fin6->sin6_addr)) {
if (!IN6_IS_ADDR_V4MAPPED(&lin6->sin6_addr))
return (EINVAL);
in6_sin6_2_sin_in_sock((struct sockaddr *)&addrs[0]);
in6_sin6_2_sin_in_sock((struct sockaddr *)&addrs[1]);
fin = (struct sockaddr_in *)&addrs[0];
lin = (struct sockaddr_in *)&addrs[1];
break;
}
error = sa6_embedscope(fin6, V_ip6_use_defzone);
if (error)
return (error);
error = sa6_embedscope(lin6, V_ip6_use_defzone);
if (error)
return (error);
break;
#endif
#ifdef INET
case AF_INET:
fin = (struct sockaddr_in *)&addrs[0];
lin = (struct sockaddr_in *)&addrs[1];
if (fin->sin_len != sizeof(struct sockaddr_in) ||
lin->sin_len != sizeof(struct sockaddr_in))
return (EINVAL);
break;
#endif
default:
return (EINVAL);
}
INP_INFO_RLOCK_ET(&V_tcbinfo, et);
switch (addrs[0].ss_family) {
#ifdef INET6
case AF_INET6:
inp = in6_pcblookup(&V_tcbinfo, &fin6->sin6_addr,
fin6->sin6_port, &lin6->sin6_addr, lin6->sin6_port,
INPLOOKUP_WLOCKPCB, NULL);
break;
#endif
#ifdef INET
case AF_INET:
inp = in_pcblookup(&V_tcbinfo, fin->sin_addr, fin->sin_port,
lin->sin_addr, lin->sin_port, INPLOOKUP_WLOCKPCB, NULL);
break;
#endif
}
INP_INFO_RUNLOCK_ET(&V_tcbinfo, et);
if (inp != NULL) {
if ((inp->inp_flags & (INP_TIMEWAIT | INP_DROPPED)) != 0 ||
inp->inp_socket == NULL) {
error = ECONNRESET;
INP_WUNLOCK(inp);
} else {
struct socket *so;
so = inp->inp_socket;
soref(so);
error = ktls_set_tx_mode(so,
arg2 == 0 ? TCP_TLS_MODE_SW : TCP_TLS_MODE_IFNET);
INP_WUNLOCK(inp);
SOCK_LOCK(so);
sorele(so);
}
} else
error = ESRCH;
return (error);
}
SYSCTL_PROC(_net_inet_tcp, OID_AUTO, switch_to_sw_tls,
CTLFLAG_VNET | CTLTYPE_STRUCT | CTLFLAG_WR | CTLFLAG_SKIP, NULL,
0, sysctl_switch_tls, "", "Switch TCP connection to SW TLS");
SYSCTL_PROC(_net_inet_tcp, OID_AUTO, switch_to_ifnet_tls,
CTLFLAG_VNET | CTLTYPE_STRUCT | CTLFLAG_WR | CTLFLAG_SKIP, NULL,
1, sysctl_switch_tls, "", "Switch TCP connection to ifnet TLS");
#endif
/*
* Generate a standardized TCP log line for use throughout the
* tcp subsystem. Memory allocation is done with M_NOWAIT to

35
sys/netinet/tcp_usrreq.c
View file

@ -44,6 +44,7 @@ __FBSDID("$FreeBSD$");
#include "opt_inet.h"
#include "opt_inet6.h"
#include "opt_ipsec.h"
#include "opt_kern_tls.h"
#include "opt_tcpdebug.h"
#include <sys/param.h>
@ -52,6 +53,7 @@ __FBSDID("$FreeBSD$");
#include <sys/malloc.h>
#include <sys/refcount.h>
#include <sys/kernel.h>
#include <sys/ktls.h>
#include <sys/sysctl.h>
#include <sys/mbuf.h>
#ifdef INET6
@ -1755,6 +1757,9 @@ tcp_default_ctloutput(struct socket *so, struct sockopt *sopt, struct inpcb *inp
int error, opt, optval;
u_int ui;
struct tcp_info ti;
#ifdef KERN_TLS
struct tls_enable tls;
#endif
struct cc_algo *algo;
char *pbuf, buf[TCP_LOG_ID_LEN];
size_t len;
@ -1917,6 +1922,29 @@ tcp_default_ctloutput(struct socket *so, struct sockopt *sopt, struct inpcb *inp
INP_WUNLOCK(inp);
break;
#ifdef KERN_TLS
case TCP_TXTLS_ENABLE:
INP_WUNLOCK(inp);
error = sooptcopyin(sopt, &tls, sizeof(tls),
sizeof(tls));
if (error)
break;
error = ktls_enable_tx(so, &tls);
break;
case TCP_TXTLS_MODE:
INP_WUNLOCK(inp);
error = sooptcopyin(sopt, &ui, sizeof(ui), sizeof(ui));
if (error)
return (error);
if (ui != TCP_TLS_MODE_SW && ui != TCP_TLS_MODE_IFNET)
return (EINVAL);
INP_WLOCK_RECHECK(inp);
error = ktls_set_tx_mode(so, ui);
INP_WUNLOCK(inp);
break;
#endif
case TCP_KEEPIDLE:
case TCP_KEEPINTVL:
case TCP_KEEPINIT:
@ -2197,6 +2225,13 @@ tcp_default_ctloutput(struct socket *so, struct sockopt *sopt, struct inpcb *inp
INP_WUNLOCK(inp);
error = EINVAL;
break;
#endif
#ifdef KERN_TLS
case TCP_TXTLS_MODE:
optval = ktls_get_tx_mode(so);
INP_WUNLOCK(inp);
error = sooptcopyout(sopt, &optval, sizeof(optval));
break;
#endif
default:
INP_WUNLOCK(inp);

2
sys/netinet/tcp_var.h
View file

@ -952,7 +952,7 @@ uint32_t tcp_compute_initwnd(uint32_t);
void tcp_sndbuf_autoscale(struct tcpcb *, struct socket *, uint32_t);
struct mbuf *
tcp_m_copym(struct mbuf *m, int32_t off0, int32_t *plen,
int32_t seglimit, int32_t segsize, struct sockbuf *sb);
int32_t seglimit, int32_t segsize, struct sockbuf *sb, bool hw_tls);
static inline void

36
sys/netinet6/ip6_output.c
View file

@ -68,6 +68,7 @@ __FBSDID("$FreeBSD$");
#include "opt_inet.h"
#include "opt_inet6.h"
#include "opt_ipsec.h"
#include "opt_kern_tls.h"
#include "opt_ratelimit.h"
#include "opt_route.h"
#include "opt_rss.h"
@ -75,6 +76,7 @@ __FBSDID("$FreeBSD$");
#include <sys/param.h>
#include <sys/kernel.h>
#include <sys/ktls.h>
#include <sys/malloc.h>
#include <sys/mbuf.h>
#include <sys/errno.h>
@ -280,14 +282,39 @@ static int
ip6_output_send(struct inpcb *inp, struct ifnet *ifp, struct ifnet *origifp,
struct mbuf *m, struct sockaddr_in6 *dst, struct route_in6 *ro)
{
#ifdef KERN_TLS
struct ktls_session *tls = NULL;
#endif
struct m_snd_tag *mst;
int error;
MPASS((m->m_pkthdr.csum_flags & CSUM_SND_TAG) == 0);
mst = NULL;
#ifdef KERN_TLS
/*
* If this is an unencrypted TLS record, save a reference to
* the record. This local reference is used to call
* ktls_output_eagain after the mbuf has been freed (thus
* dropping the mbuf's reference) in if_output.
*/
if (m->m_next != NULL && mbuf_has_tls_session(m->m_next)) {
tls = ktls_hold(m->m_next->m_ext.ext_pgs->tls);
mst = tls->snd_tag;
/*
* If a TLS session doesn't have a valid tag, it must
* have had an earlier ifp mismatch, so drop this
* packet.
*/
if (mst == NULL) {
error = EAGAIN;
goto done;
}
}
#endif
#ifdef RATELIMIT
if (inp != NULL) {
if (inp != NULL && mst == NULL) {
if ((inp->inp_flags2 & INP_RATE_LIMIT_CHANGED) != 0 ||
(inp->inp_snd_tag != NULL &&
inp->inp_snd_tag->ifp != ifp))
@ -314,6 +341,13 @@ ip6_output_send(struct inpcb *inp, struct ifnet *ifp, struct ifnet *origifp,
done:
/* Check for route change invalidating send tags. */
#ifdef KERN_TLS
if (tls != NULL) {
if (error == EAGAIN)
error = ktls_output_eagain(inp, tls);
ktls_free(tls);
}
#endif
#ifdef RATELIMIT
if (error == EAGAIN)
in_pcboutput_eagain(inp);

308
sys/opencrypto/ktls_ocf.c Normal file
View file

@ -0,0 +1,308 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2019 Netflix Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
#include <sys/param.h>
#include <sys/systm.h>
#include <sys/counter.h>
#include <sys/endian.h>
#include <sys/kernel.h>
#include <sys/ktls.h>
#include <sys/lock.h>
#include <sys/malloc.h>
#include <sys/module.h>
#include <sys/mutex.h>
#include <sys/sysctl.h>
#include <sys/uio.h>
#include <opencrypto/cryptodev.h>
struct ocf_session {
crypto_session_t sid;
int crda_alg;
struct mtx lock;
};
struct ocf_operation {
struct ocf_session *os;
bool done;
struct iovec iov[0];
};
static MALLOC_DEFINE(M_KTLS_OCF, "ktls_ocf", "OCF KTLS");
SYSCTL_DECL(_kern_ipc_tls);
SYSCTL_DECL(_kern_ipc_tls_stats);
static counter_u64_t ocf_gcm_crypts;
SYSCTL_COUNTER_U64(_kern_ipc_tls_stats, OID_AUTO, ocf_gcm_crypts, CTLFLAG_RD,
&ocf_gcm_crypts,
"Total number of OCF GCM encryption operations");
static counter_u64_t ocf_retries;
SYSCTL_COUNTER_U64(_kern_ipc_tls_stats, OID_AUTO, ocf_retries, CTLFLAG_RD,
&ocf_retries,
"Number of OCF encryption operation retries");
static int
ktls_ocf_callback(struct cryptop *crp)
{
struct ocf_operation *oo;
oo = crp->crp_opaque;
mtx_lock(&oo->os->lock);
oo->done = true;
mtx_unlock(&oo->os->lock);
wakeup(oo);
return (0);
}
static int
ktls_ocf_encrypt(struct ktls_session *tls, const struct tls_record_layer *hdr,
uint8_t *trailer, struct iovec *iniov, struct iovec *outiov, int iovcnt,
uint64_t seqno)
{
struct uio uio;
struct tls_aead_data ad;
struct tls_nonce_data nd;
struct cryptodesc *crde, *crda;
struct cryptop *crp;
struct ocf_session *os;
struct ocf_operation *oo;
struct iovec *iov;
int i, error;
uint16_t tls_comp_len;
os = tls->cipher;
oo = malloc(sizeof(*oo) + (iovcnt + 2) * sizeof(*iov), M_KTLS_OCF,
M_WAITOK | M_ZERO);
oo->os = os;
iov = oo->iov;
crp = crypto_getreq(2);
if (crp == NULL) {
free(oo, M_KTLS_OCF);
return (ENOMEM);
}
/* Setup the IV. */
memcpy(nd.fixed, tls->params.iv, TLS_AEAD_GCM_LEN);
memcpy(&nd.seq, hdr + 1, sizeof(nd.seq));
/* Setup the AAD. */
tls_comp_len = ntohs(hdr->tls_length) -
(AES_GMAC_HASH_LEN + sizeof(nd.seq));
ad.seq = htobe64(seqno);
ad.type = hdr->tls_type;
ad.tls_vmajor = hdr->tls_vmajor;
ad.tls_vminor = hdr->tls_vminor;
ad.tls_length = htons(tls_comp_len);
iov[0].iov_base = &ad;
iov[0].iov_len = sizeof(ad);
uio.uio_resid = sizeof(ad);
/*
* OCF always does encryption in place, so copy the data if
* needed. Ugh.
*/
for (i = 0; i < iovcnt; i++) {
iov[i + 1] = outiov[i];
if (iniov[i].iov_base != outiov[i].iov_base)
memcpy(outiov[i].iov_base, iniov[i].iov_base,
outiov[i].iov_len);
uio.uio_resid += outiov[i].iov_len;
}
iov[iovcnt + 1].iov_base = trailer;
iov[iovcnt + 1].iov_len = AES_GMAC_HASH_LEN;
uio.uio_resid += AES_GMAC_HASH_LEN;
uio.uio_iov = iov;
uio.uio_iovcnt = iovcnt + 2;
uio.uio_offset = 0;
uio.uio_segflg = UIO_SYSSPACE;
uio.uio_td = curthread;
crp->crp_session = os->sid;
crp->crp_flags = CRYPTO_F_IOV | CRYPTO_F_CBIMM;
crp->crp_uio = &uio;
crp->crp_ilen = uio.uio_resid;
crp->crp_opaque = oo;
crp->crp_callback = ktls_ocf_callback;
crde = crp->crp_desc;
crda = crde->crd_next;
crda->crd_alg = os->crda_alg;
crda->crd_skip = 0;
crda->crd_len = sizeof(ad);
crda->crd_inject = crp->crp_ilen - AES_GMAC_HASH_LEN;
crde->crd_alg = CRYPTO_AES_NIST_GCM_16;
crde->crd_skip = sizeof(ad);
crde->crd_len = crp->crp_ilen - (sizeof(ad) + AES_GMAC_HASH_LEN);
crde->crd_flags = CRD_F_ENCRYPT | CRD_F_IV_EXPLICIT | CRD_F_IV_PRESENT;
memcpy(crde->crd_iv, &nd, sizeof(nd));
counter_u64_add(ocf_gcm_crypts, 1);
for (;;) {
error = crypto_dispatch(crp);
if (error)
break;
mtx_lock(&os->lock);
while (!oo->done)
mtx_sleep(oo, &os->lock, 0, "ocfktls", 0);
mtx_unlock(&os->lock);
if (crp->crp_etype != EAGAIN) {
error = crp->crp_etype;
break;
}
crp->crp_etype = 0;
crp->crp_flags &= ~CRYPTO_F_DONE;
oo->done = false;
counter_u64_add(ocf_retries, 1);
}
crypto_freereq(crp);
free(oo, M_KTLS_OCF);
return (error);
}
static void
ktls_ocf_free(struct ktls_session *tls)
{
struct ocf_session *os;
os = tls->cipher;
mtx_destroy(&os->lock);
explicit_bzero(os, sizeof(*os));
free(os, M_KTLS_OCF);
}
static int
ktls_ocf_try(struct socket *so, struct ktls_session *tls)
{
struct cryptoini cria, crie;
struct ocf_session *os;
int error;
memset(&cria, 0, sizeof(cria));
memset(&crie, 0, sizeof(crie));
switch (tls->params.cipher_algorithm) {
case CRYPTO_AES_NIST_GCM_16:
if (tls->params.iv_len != TLS_AEAD_GCM_LEN)
return (EINVAL);
switch (tls->params.cipher_key_len) {
case 128 / 8:
cria.cri_alg = CRYPTO_AES_128_NIST_GMAC;
break;
case 256 / 8:
cria.cri_alg = CRYPTO_AES_256_NIST_GMAC;
break;
default:
return (EINVAL);
}
cria.cri_key = tls->params.cipher_key;
cria.cri_klen = tls->params.cipher_key_len * 8;
break;
default:
return (EPROTONOSUPPORT);
}
/* Only TLS 1.1 and TLS 1.2 are currently supported. */
if (tls->params.tls_vmajor != TLS_MAJOR_VER_ONE ||
tls->params.tls_vminor < TLS_MINOR_VER_ONE ||
tls->params.tls_vminor > TLS_MINOR_VER_TWO)
return (EPROTONOSUPPORT);
os = malloc(sizeof(*os), M_KTLS_OCF, M_NOWAIT | M_ZERO);
if (os == NULL)
return (ENOMEM);
crie.cri_alg = tls->params.cipher_algorithm;
crie.cri_key = tls->params.cipher_key;
crie.cri_klen = tls->params.cipher_key_len * 8;
crie.cri_next = &cria;
error = crypto_newsession(&os->sid, &crie,
CRYPTO_FLAG_HARDWARE | CRYPTO_FLAG_SOFTWARE);
if (error) {
free(os, M_KTLS_OCF);
return (error);
}
os->crda_alg = cria.cri_alg;
mtx_init(&os->lock, "ktls_ocf", NULL, MTX_DEF);
tls->cipher = os;
tls->sw_encrypt = ktls_ocf_encrypt;
tls->free = ktls_ocf_free;
return (0);
}
struct ktls_crypto_backend ocf_backend = {
.name = "OCF",
.prio = 5,
.api_version = KTLS_API_VERSION,
.try = ktls_ocf_try,
};
static int
ktls_ocf_modevent(module_t mod, int what, void *arg)
{
int error;
switch (what) {
case MOD_LOAD:
ocf_gcm_crypts = counter_u64_alloc(M_WAITOK);
ocf_retries = counter_u64_alloc(M_WAITOK);
return (ktls_crypto_backend_register(&ocf_backend));
case MOD_UNLOAD:
error = ktls_crypto_backend_deregister(&ocf_backend);
if (error)
return (error);
counter_u64_free(ocf_gcm_crypts);
counter_u64_free(ocf_retries);
return (0);
default:
return (EOPNOTSUPP);
}
}
static moduledata_t ktls_ocf_moduledata = {
"ktls_ocf",
ktls_ocf_modevent,
NULL
};
DECLARE_MODULE(ktls_ocf, ktls_ocf_moduledata, SI_SUB_PROTO_END, SI_ORDER_ANY);

194
sys/sys/ktls.h Normal file
View file

@ -0,0 +1,194 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
*
* Copyright (c) 2014-2019 Netflix Inc.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*
* $FreeBSD$
*/
#ifndef _SYS_KTLS_H_
#define _SYS_KTLS_H_
#include <sys/refcount.h>
#include <sys/_task.h>
struct tls_record_layer {
uint8_t tls_type;
uint8_t tls_vmajor;
uint8_t tls_vminor;
uint16_t tls_length;
uint8_t tls_data[0];
} __attribute__ ((packed));
#define TLS_MAX_MSG_SIZE_V10_2 16384
#define TLS_MAX_PARAM_SIZE 1024 /* Max key/mac/iv in sockopt */
#define TLS_AEAD_GCM_LEN 4
#define TLS_CBC_IMPLICIT_IV_LEN 16
/* Type values for the record layer */
#define TLS_RLTYPE_APP 23
/*
* Nonce for GCM.
*/
struct tls_nonce_data {
uint8_t fixed[TLS_AEAD_GCM_LEN];
uint64_t seq;
} __packed;
/*
* AEAD additional data format per RFC.
*/
struct tls_aead_data {
uint64_t seq; /* In network order */
uint8_t type;
uint8_t tls_vmajor;
uint8_t tls_vminor;
uint16_t tls_length;
} __packed;
/*
* Stream Cipher MAC additional data input. This does not match the
* exact data on the wire (the sequence number is not placed on the
* wire, and any explicit IV after the record header is not covered by
* the MAC).
*/
struct tls_mac_data {
uint64_t seq;
uint8_t type;
uint8_t tls_vmajor;
uint8_t tls_vminor;
uint16_t tls_length;
} __packed;
#define TLS_MAJOR_VER_ONE 3
#define TLS_MINOR_VER_ZERO 1 /* 3, 1 */
#define TLS_MINOR_VER_ONE 2 /* 3, 2 */
#define TLS_MINOR_VER_TWO 3 /* 3, 3 */
/* For TCP_TXTLS_ENABLE */
struct tls_enable {
const uint8_t *cipher_key;
const uint8_t *iv; /* Implicit IV. */
const uint8_t *auth_key;
int cipher_algorithm; /* e.g. CRYPTO_AES_CBC */
int cipher_key_len;
int iv_len;
int auth_algorithm; /* e.g. CRYPTO_SHA2_256_HMAC */
int auth_key_len;
int flags;
uint8_t tls_vmajor;
uint8_t tls_vminor;
};
struct tls_session_params {
uint8_t *cipher_key;
uint8_t *auth_key;
uint8_t iv[TLS_CBC_IMPLICIT_IV_LEN];
int cipher_algorithm;
int auth_algorithm;
uint16_t cipher_key_len;
uint16_t iv_len;
uint16_t auth_key_len;
uint16_t max_frame_len;
uint8_t tls_vmajor;
uint8_t tls_vminor;
uint8_t tls_hlen;
uint8_t tls_tlen;
uint8_t tls_bs;
uint8_t flags;
};
#ifdef _KERNEL
#define KTLS_API_VERSION 5
struct iovec;
struct ktls_session;
struct m_snd_tag;
struct mbuf;
struct mbuf_ext_pgs;
struct sockbuf;
struct socket;
struct ktls_crypto_backend {
LIST_ENTRY(ktls_crypto_backend) next;
int (*try)(struct socket *so, struct ktls_session *tls);
int prio;
int api_version;
int use_count;
const char *name;
};
struct ktls_session {
int (*sw_encrypt)(struct ktls_session *tls,
const struct tls_record_layer *hdr, uint8_t *trailer,
struct iovec *src, struct iovec *dst, int iovcnt,
uint64_t seqno);
union {
void *cipher;
struct m_snd_tag *snd_tag;
};
struct ktls_crypto_backend *be;
void (*free)(struct ktls_session *tls);
struct tls_session_params params;
u_int wq_index;
volatile u_int refcount;
struct task reset_tag_task;
struct inpcb *inp;
bool reset_pending;
} __aligned(CACHE_LINE_SIZE);
int ktls_crypto_backend_register(struct ktls_crypto_backend *be);
int ktls_crypto_backend_deregister(struct ktls_crypto_backend *be);
int ktls_enable_tx(struct socket *so, struct tls_enable *en);
void ktls_destroy(struct ktls_session *tls);
int ktls_frame(struct mbuf *m, struct ktls_session *tls, int *enqueue_cnt,
uint8_t record_type);
void ktls_seq(struct sockbuf *sb, struct mbuf *m);
void ktls_enqueue(struct mbuf *m, struct socket *so, int page_count);
void ktls_enqueue_to_free(struct mbuf_ext_pgs *pgs);
int ktls_set_tx_mode(struct socket *so, int mode);
int ktls_get_tx_mode(struct socket *so);
int ktls_output_eagain(struct inpcb *inp, struct ktls_session *tls);
static inline struct ktls_session *
ktls_hold(struct ktls_session *tls)
{
if (tls != NULL)
refcount_acquire(&tls->refcount);
return (tls);
}
static inline void
ktls_free(struct ktls_session *tls)
{
if (refcount_release(&tls->refcount))
ktls_destroy(tls);
}
#endif /* !_KERNEL */
#endif /* !_SYS_KTLS_H_ */

19
sys/sys/mbuf.h
View file

@ -301,6 +301,7 @@ struct mbuf {
};
};
struct ktls_session;
struct socket;
/*
@ -344,7 +345,7 @@ struct mbuf_ext_pgs {
uint16_t last_pg_len; /* Length of last page */
vm_paddr_t pa[MBUF_PEXT_MAX_PGS]; /* phys addrs of pages */
char hdr[MBUF_PEXT_HDR_LEN]; /* TLS header */
void *tls; /* TLS session */
struct ktls_session *tls; /* TLS session */
#if defined(__i386__) || \
(defined(__powerpc__) && !defined(__powerpc64__) && defined(BOOKE))
/*
@ -357,9 +358,10 @@ struct mbuf_ext_pgs {
char trail[MBUF_PEXT_TRAIL_LEN]; /* TLS trailer */
struct {
struct socket *so;
void *mbuf;
struct mbuf *mbuf;
uint64_t seqno;
STAILQ_ENTRY(mbuf_ext_pgs) stailq;
int enc_cnt;
};
};
};
@ -1506,5 +1508,18 @@ void netdump_mbuf_dump(void);
void netdump_mbuf_reinit(int nmbuf, int nclust, int clsize);
#endif
static inline bool
mbuf_has_tls_session(struct mbuf *m)
{
if (m->m_flags & M_NOMAP) {
MBUF_EXT_PGS_ASSERT(m);
if (m->m_ext.ext_pgs->tls != NULL) {
return (true);
}
}
return (false);
}
#endif /* _KERNEL */
#endif /* !_SYS_MBUF_H_ */

2
sys/sys/param.h
View file

@ -60,7 +60,7 @@
* in the range 5 to 9.
*/
#undef __FreeBSD_version
#define __FreeBSD_version 1300041 /* Master, propagated to newvers */
#define __FreeBSD_version 1300042 /* Master, propagated to newvers */
/*
* __FreeBSD_kernel__ indicates that this system uses the kernel of FreeBSD,

5
sys/sys/sockbuf.h
View file

@ -50,6 +50,7 @@
#define SB_AUTOSIZE 0x800 /* automatically size socket buffer */
#define SB_STOP 0x1000 /* backpressure indicator */
#define SB_AIO_RUNNING 0x2000 /* AIO operation running */
#define SB_TLS_IFNET 0x4000 /* has used / is using ifnet KTLS */
#define SBS_CANTSENDMORE 0x0010 /* can't send more data to peer */
#define SBS_CANTRCVMORE 0x0020 /* can't receive more data from peer */
@ -63,6 +64,7 @@
#define SB_MAX (2*1024*1024) /* default for max chars in sockbuf */
struct ktls_session;
struct mbuf;
struct sockaddr;
struct socket;
@ -74,6 +76,7 @@ struct selinfo;
*
* Locking key to struct sockbuf:
* (a) locked by SOCKBUF_LOCK().
* (b) locked by sblock()
*/
struct sockbuf {
struct mtx sb_mtx; /* sockbuf lock */
@ -98,6 +101,8 @@ struct sockbuf {
u_int sb_ctl; /* (a) non-data chars in buffer */
int sb_lowat; /* (a) low water mark */
sbintime_t sb_timeo; /* (a) timeout for read/write */
uint64_t sb_tls_seqno; /* (a) TLS seqno */
struct ktls_session *sb_tls_info; /* (a + b) TLS state */
short sb_flags; /* (a) flags, see above */
int (*sb_upcall)(struct socket *, void *, int); /* (a) */
void *sb_upcallarg; /* (a) */

6
tools/tools/switch_tls/Makefile Normal file
View file

@ -0,0 +1,6 @@
# $FreeBSD$
PROG= switch_tls
MAN=
.include <bsd.prog.mk>

381
tools/tools/switch_tls/switch_tls.c Normal file
View file

@ -0,0 +1,381 @@
/* $OpenBSD: tcpdrop.c,v 1.4 2004/05/22 23:55:22 deraadt Exp $ */
/*-
* Copyright (c) 2009 Juli Mallett <jmallett@FreeBSD.org>
* Copyright (c) 2004 Markus Friedl <markus@openbsd.org>
*
* Permission to use, copy, modify, and distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
* copyright notice and this permission notice appear in all copies.
*
* THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
* WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
* MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
* ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
* WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
* ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
* OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
#include <sys/cdefs.h>
__FBSDID("$FreeBSD$");
#include <sys/param.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/socketvar.h>
#include <sys/sysctl.h>
#include <netinet/in.h>
#include <netinet/in_pcb.h>
#define TCPSTATES
#include <netinet/tcp_fsm.h>
#include <netinet/tcp_var.h>
#include <err.h>
#include <netdb.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#define TCPDROP_FOREIGN 0
#define TCPDROP_LOCAL 1
#define SW_TLS 0
#define IFNET_TLS 1
struct host_service {
char hs_host[NI_MAXHOST];
char hs_service[NI_MAXSERV];
};
static bool tcpswitch_list_commands = false;
static char *findport(const char *);
static struct xinpgen *getxpcblist(const char *);
static void sockinfo(const struct sockaddr *, struct host_service *);
static bool tcpswitch(const struct sockaddr *, const struct sockaddr *, int);
static bool tcpswitchall(const char *, int);
static bool tcpswitchbyname(const char *, const char *, const char *,
const char *, int);
static bool tcpswitchconn(const struct in_conninfo *, int);
static void usage(void);
/*
* Switch a tcp connection.
*/
int
main(int argc, char *argv[])
{
char stack[TCP_FUNCTION_NAME_LEN_MAX];
char *lport, *fport;
bool switchall, switchallstack;
int ch, mode;
switchall = false;
switchallstack = false;
stack[0] = '\0';
mode = SW_TLS;
while ((ch = getopt(argc, argv, "ailS:s")) != -1) {
switch (ch) {
case 'a':
switchall = true;
break;
case 'i':
mode = IFNET_TLS;
break;
case 'l':
tcpswitch_list_commands = true;
break;
case 'S':
switchallstack = true;
strlcpy(stack, optarg, sizeof(stack));
break;
case 's':
mode = SW_TLS;
break;
default:
usage();
}
}
argc -= optind;
argv += optind;
if (switchall && switchallstack)
usage();
if (switchall || switchallstack) {
if (argc != 0)
usage();
if (!tcpswitchall(stack, mode))
exit(1);
exit(0);
}
if ((argc != 2 && argc != 4) || tcpswitch_list_commands)
usage();
if (argc == 2) {
lport = findport(argv[0]);
fport = findport(argv[1]);
if (lport == NULL || lport[1] == '\0' || fport == NULL ||
fport[1] == '\0')
usage();
*lport++ = '\0';
*fport++ = '\0';
if (!tcpswitchbyname(argv[0], lport, argv[1], fport, mode))
exit(1);
} else if (!tcpswitchbyname(argv[0], argv[1], argv[2], argv[3], mode))
exit(1);
exit(0);
}
static char *
findport(const char *arg)
{
char *dot, *colon;
/* A strrspn() or strrpbrk() would be nice. */
dot = strrchr(arg, '.');
colon = strrchr(arg, ':');
if (dot == NULL)
return (colon);
if (colon == NULL)
return (dot);
if (dot < colon)
return (colon);
else
return (dot);
}
static struct xinpgen *
getxpcblist(const char *name)
{
struct xinpgen *xinp;
size_t len;
int rv;
len = 0;
rv = sysctlbyname(name, NULL, &len, NULL, 0);
if (rv == -1)
err(1, "sysctlbyname %s", name);
if (len == 0)
errx(1, "%s is empty", name);
xinp = malloc(len);
if (xinp == NULL)
errx(1, "malloc failed");
rv = sysctlbyname(name, xinp, &len, NULL, 0);
if (rv == -1)
err(1, "sysctlbyname %s", name);
return (xinp);
}
static void
sockinfo(const struct sockaddr *sa, struct host_service *hs)
{
static const int flags = NI_NUMERICHOST | NI_NUMERICSERV;
int rv;
rv = getnameinfo(sa, sa->sa_len, hs->hs_host, sizeof hs->hs_host,
hs->hs_service, sizeof hs->hs_service, flags);
if (rv == -1)
err(1, "getnameinfo");
}
static bool
tcpswitch(const struct sockaddr *lsa, const struct sockaddr *fsa, int mode)
{
struct host_service local, foreign;
struct sockaddr_storage addrs[2];
int rv;
memcpy(&addrs[TCPDROP_FOREIGN], fsa, fsa->sa_len);
memcpy(&addrs[TCPDROP_LOCAL], lsa, lsa->sa_len);
sockinfo(lsa, &local);
sockinfo(fsa, &foreign);
if (tcpswitch_list_commands) {
printf("switch_tls %s %s %s %s %s\n",
mode == SW_TLS ? "-s" : "-i",
local.hs_host, local.hs_service,
foreign.hs_host, foreign.hs_service);
return (true);
}
rv = sysctlbyname(mode == SW_TLS ? "net.inet.tcp.switch_to_sw_tls" :
"net.inet.tcp.switch_to_ifnet_tls", NULL, NULL, &addrs,
sizeof addrs);
if (rv == -1) {
warn("%s %s %s %s", local.hs_host, local.hs_service,
foreign.hs_host, foreign.hs_service);
return (false);
}
printf("%s %s %s %s: switched\n", local.hs_host, local.hs_service,
foreign.hs_host, foreign.hs_service);
return (true);
}
static bool
tcpswitchall(const char *stack, int mode)
{
struct xinpgen *head, *xinp;
struct xtcpcb *xtp;
struct xinpcb *xip;
bool ok;
ok = true;
head = getxpcblist("net.inet.tcp.pcblist");
#define XINP_NEXT(xinp) \
((struct xinpgen *)(uintptr_t)((uintptr_t)(xinp) + (xinp)->xig_len))
for (xinp = XINP_NEXT(head); xinp->xig_len > sizeof *xinp;
xinp = XINP_NEXT(xinp)) {
xtp = (struct xtcpcb *)xinp;
xip = &xtp->xt_inp;
/*
* XXX
* Check protocol, support just v4 or v6, etc.
*/
/* Ignore PCBs which were freed during copyout. */
if (xip->inp_gencnt > head->xig_gen)
continue;
/* Skip listening sockets. */
if (xtp->t_state == TCPS_LISTEN)
continue;
/* If requested, skip sockets not having the requested stack. */
if (stack[0] != '\0' &&
strncmp(xtp->xt_stack, stack, TCP_FUNCTION_NAME_LEN_MAX))
continue;
if (!tcpswitchconn(&xip->inp_inc, mode))
ok = false;
}
free(head);
return (ok);
}
static bool
tcpswitchbyname(const char *lhost, const char *lport, const char *fhost,
const char *fport, int mode)
{
static const struct addrinfo hints = {
/*
* Look for streams in all domains.
*/
.ai_family = AF_UNSPEC,
.ai_socktype = SOCK_STREAM,
};
struct addrinfo *ail, *local, *aif, *foreign;
int error;
bool ok, infamily;
error = getaddrinfo(lhost, lport, &hints, &local);
if (error != 0)
errx(1, "getaddrinfo: %s port %s: %s", lhost, lport,
gai_strerror(error));
error = getaddrinfo(fhost, fport, &hints, &foreign);
if (error != 0) {
freeaddrinfo(local); /* XXX gratuitous */
errx(1, "getaddrinfo: %s port %s: %s", fhost, fport,
gai_strerror(error));
}
ok = true;
infamily = false;
/*
* Try every combination of local and foreign address pairs.
*/
for (ail = local; ail != NULL; ail = ail->ai_next) {
for (aif = foreign; aif != NULL; aif = aif->ai_next) {
if (ail->ai_family != aif->ai_family)
continue;
infamily = true;
if (!tcpswitch(ail->ai_addr, aif->ai_addr, mode))
ok = false;
}
}
if (!infamily) {
warnx("%s %s %s %s: different address families", lhost, lport,
fhost, fport);
ok = false;
}
freeaddrinfo(local);
freeaddrinfo(foreign);
return (ok);
}
static bool
tcpswitchconn(const struct in_conninfo *inc, int mode)
{
struct sockaddr *local, *foreign;
struct sockaddr_in6 sin6[2];
struct sockaddr_in sin4[2];
if ((inc->inc_flags & INC_ISIPV6) != 0) {
memset(sin6, 0, sizeof sin6);
sin6[TCPDROP_LOCAL].sin6_len = sizeof sin6[TCPDROP_LOCAL];
sin6[TCPDROP_LOCAL].sin6_family = AF_INET6;
sin6[TCPDROP_LOCAL].sin6_port = inc->inc_lport;
memcpy(&sin6[TCPDROP_LOCAL].sin6_addr, &inc->inc6_laddr,
sizeof inc->inc6_laddr);
local = (struct sockaddr *)&sin6[TCPDROP_LOCAL];
sin6[TCPDROP_FOREIGN].sin6_len = sizeof sin6[TCPDROP_FOREIGN];
sin6[TCPDROP_FOREIGN].sin6_family = AF_INET6;
sin6[TCPDROP_FOREIGN].sin6_port = inc->inc_fport;
memcpy(&sin6[TCPDROP_FOREIGN].sin6_addr, &inc->inc6_faddr,
sizeof inc->inc6_faddr);
foreign = (struct sockaddr *)&sin6[TCPDROP_FOREIGN];
} else {
memset(sin4, 0, sizeof sin4);
sin4[TCPDROP_LOCAL].sin_len = sizeof sin4[TCPDROP_LOCAL];
sin4[TCPDROP_LOCAL].sin_family = AF_INET;
sin4[TCPDROP_LOCAL].sin_port = inc->inc_lport;
memcpy(&sin4[TCPDROP_LOCAL].sin_addr, &inc->inc_laddr,
sizeof inc->inc_laddr);
local = (struct sockaddr *)&sin4[TCPDROP_LOCAL];
sin4[TCPDROP_FOREIGN].sin_len = sizeof sin4[TCPDROP_FOREIGN];
sin4[TCPDROP_FOREIGN].sin_family = AF_INET;
sin4[TCPDROP_FOREIGN].sin_port = inc->inc_fport;
memcpy(&sin4[TCPDROP_FOREIGN].sin_addr, &inc->inc_faddr,
sizeof inc->inc_faddr);
foreign = (struct sockaddr *)&sin4[TCPDROP_FOREIGN];
}
return (tcpswitch(local, foreign, mode));
}
static void
usage(void)
{
fprintf(stderr,
"usage: switch_tls [-i | -s] local-address local-port foreign-address foreign-port\n"
" switch_tls [-i | -s] local-address:local-port foreign-address:foreign-port\n"
" switch_tls [-i | -s] local-address.local-port foreign-address.foreign-port\n"
" switch_tls [-l | -i | -s] -a\n"
" switch_tls [-l | -i | -s] -S stack\n");
exit(1);
}