system/freebsd-src
system/freebsd-src
1
0
Fork 0
mirror of https://github.com/freebsd/freebsd-src synced 2024-10-01 14:14:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Disable high priority ZIO threads on FreeBSD and Linux

Browse source 
High priority threads are handling ZIL writes.  While there is no
ZIL compression, there is encryption, checksuming and RAIDZ math.
We've found that on large systems 1 taskq with 5 threads can be
a bottleneck for throughput, IOPS or both. Instead of just bumping
number of threads with a risk of overloading CPUs and increasing
latency, switch to using TQ_FRONT mechanism to increase sync write
requests priority within standard write threads.  Do not do it on
Illumos, since its TQ_FRONT implementation is inherently unfair.
FreeBSD and Linux don't have this problem, so we can do it there.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Rob Norris <robn@despairlabs.com>
Signed-off-by: Alexander Motin <mav@FreeBSD.org>
Sponsored-By: iXsystems, Inc.
Closes #16146
This commit is contained in:
Alexander Motin 2024-05-03 12:53:34 -04:00 committed by GitHub
parent 8f1b7a6fa6
commit 04bae5ec95
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 16 additions and 9 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
man/man4/zfs.4
View file

@ -2367,7 +2367,7 @@ This is an advanced debugging parameter.
Don't change this unless you understand what it does.
Set values only apply to pools imported/created after that.
.
.It Sy zio_taskq_write Ns = Ns Sy sync fixed,1,5 scale fixed,1,5 Pq charp
.It Sy zio_taskq_write Ns = Ns Sy sync null scale null Pq charp
Set the queue and thread configuration for the IO write queues.
This is an advanced debugging parameter.
Don't change this unless you understand what it does.

11
module/zfs/spa.c
View file

@ -170,14 +170,19 @@ static const char *const zio_taskq_types[ZIO_TASKQ_TYPES] = {
* that scales with the number of CPUs.
*
* The different taskq priorities are to handle the different contexts (issue
* and interrupt) and then to reserve threads for ZIO_PRIORITY_NOW I/Os that
* need to be handled with minimum delay.
* and interrupt) and then to reserve threads for high priority I/Os that
* need to be handled with minimum delay. Illumos taskq has unfair TQ_FRONT
* implementation, so separate high priority threads are used there.
*/
static zio_taskq_info_t zio_taskqs[ZIO_TYPES][ZIO_TASKQ_TYPES] = {
/* ISSUE ISSUE_HIGH INTR INTR_HIGH */
{ ZTI_ONE, ZTI_NULL, ZTI_ONE, ZTI_NULL }, /* NULL */
{ ZTI_N(8), ZTI_NULL, ZTI_SCALE, ZTI_NULL }, /* READ */
#ifdef illumos
{ ZTI_SYNC, ZTI_N(5), ZTI_SCALE, ZTI_N(5) }, /* WRITE */
#else
{ ZTI_SYNC, ZTI_NULL, ZTI_SCALE, ZTI_NULL }, /* WRITE */
#endif
{ ZTI_SCALE, ZTI_NULL, ZTI_ONE, ZTI_NULL }, /* FREE */
{ ZTI_ONE, ZTI_NULL, ZTI_ONE, ZTI_NULL }, /* CLAIM */
{ ZTI_ONE, ZTI_NULL, ZTI_ONE, ZTI_NULL }, /* FLUSH */
@ -1217,7 +1222,7 @@ spa_taskqs_fini(spa_t *spa, zio_type_t t, zio_taskq_type_t q)
*
* Example (the defaults for READ and WRITE)
* zio_taskq_read='fixed,1,8 null scale null'
* zio_taskq_write='sync fixed,1,5 scale fixed,1,5'
* zio_taskq_write='sync null scale null'
*
* Each sets the entire row at a time.
*

10
module/zfs/zio.c
View file

@ -2041,12 +2041,14 @@ zio_taskq_dispatch(zio_t *zio, zio_taskq_type_t q, boolean_t cutinline)
/*
* If this is a high priority I/O, then use the high priority taskq if
* available.
* available or cut the line otherwise.
*/
if ((zio->io_priority == ZIO_PRIORITY_NOW ||
zio->io_priority == ZIO_PRIORITY_SYNC_WRITE) &&
spa->spa_zio_taskq[t][q + 1].stqs_count != 0)
if (zio->io_priority == ZIO_PRIORITY_SYNC_WRITE) {
if (spa->spa_zio_taskq[t][q + 1].stqs_count != 0)
q++;
else
flags |= TQ_FRONT;
}
ASSERT3U(q, <, ZIO_TASKQ_TYPES);