Logan Gunthorpe 7e55c60acf md/raid5: Pivot raid5_make_request() ... raid5_make_request() loops through every page in the request, finds the appropriate stripe and adds the bio for that page in the disk. This causes a great deal of contention on the hash_lock and extra work seeing each stripe must be found once for every data disk. The number of times a stripe must be found can be reduced by pivoting raid5_make_request() so that it loops through every stripe and then loops through every disk in that stripe to see if the bio must be added. This reduces the number of times the hash lock must be taken by a factor equal to the number of data disks. To accomplish this, the logical sectors that have already been added must be tracked. Tracking them is done with a bitmap: the bits for all pages are set at the start of the request and each bit is cleared once the bio is added to a stripe. Finding the next sector to be done is then just a call to find_first_bit() so that sectors that have been done can simply be skipped. One minor downside is that the maximum sectors for a request must be limited so that the bitmap can be appropriately sized on the stack. This limit is arbitrarily chosen to be 256 stripe pages which works out to 1MB if PAGE_SIZE == DEFAULT_STRIPE_SIZE. This doesn't actually restrict the maximum request further seeing the default block queue settings are used which restricts the number of segments to 128 (which results in request sizes that are approximately 512KB). Signed-off-by: Logan Gunthorpe <logang@deltatee.com> Signed-off-by: Song Liu <song@kernel.org> Signed-off-by: Jens Axboe <axboe@kernel.dk>		2022-08-02 17:14:42 -06:00
..
bcache	md/bcache: Combine two prio_io() arguments	2022-07-14 12:14:31 -06:00
persistent-data	dm space map common: add bounds check to sm_ll_lookup_bitmap()	2022-01-04 13:58:19 -05:00
dm-audit.c	dm: introduce audit event module for device mapper	2021-10-27 16:53:47 -04:00
dm-audit.h	dm: introduce audit event module for device mapper	2021-10-27 16:53:47 -04:00
dm-bio-prison-v1.c
dm-bio-prison-v1.h
dm-bio-prison-v2.c
dm-bio-prison-v2.h
dm-bio-record.h	block: move integrity handling out of <linux/blkdev.h>	2021-10-18 06:17:02 -06:00
dm-bufio.c	dm/core: Combine request operation type and flags	2022-07-14 12:14:31 -06:00
dm-builtin.c
dm-cache-background-tracker.c
dm-cache-background-tracker.h
dm-cache-block-types.h
dm-cache-metadata.c	dm cache metadata: remove unnecessary variable in __dump_mapping	2022-05-09 15:40:10 -04:00
dm-cache-metadata.h	dm cache: fix typo in 2 comment blocks	2022-07-07 11:49:37 -04:00
dm-cache-policy-internal.h
dm-cache-policy-smq.c	dm cache policy smq: make static read-only array table const	2022-02-22 10:35:53 -05:00
dm-cache-policy.c
dm-cache-policy.h
dm-cache-target.c	dm cache: fix typo in 2 comment blocks	2022-07-07 11:49:37 -04:00
dm-clone-metadata.c	dm clone metadata: remove unused function	2021-04-19 13:20:31 -04:00
dm-clone-metadata.h
dm-clone-target.c	block: remove QUEUE_FLAG_DISCARD	2022-04-17 19:49:59 -06:00
dm-core.h	dm table: audit all dm_table_get_target() callers	2022-07-07 11:49:34 -04:00
dm-crypt.c	dm crypt: make printing of the key constant-time	2022-05-09 12:34:03 -04:00
dm-delay.c	dm: simplify basic targets	2022-05-05 17:31:35 -04:00
dm-dust.c	dm: use bdev_nr_sectors and bdev_nr_bytes instead of open coding them	2021-10-18 14:43:22 -06:00
dm-ebs-target.c	dm/ebs: Change 'int rw' into 'enum req_op op'	2022-07-14 12:14:31 -06:00
dm-era-target.c	dm era: commit metadata in postsuspend after worker stops	2022-06-21 13:35:01 -04:00
dm-exception-store.c
dm-exception-store.h	dm: use bdev_nr_sectors and bdev_nr_bytes instead of open coding them	2021-10-18 14:43:22 -06:00
dm-flakey.c	dm/dm-flakey: Use the new blk_opf_t type	2022-07-14 12:14:31 -06:00
dm-ima.c	dm table: audit all dm_table_get_target() callers	2022-07-07 11:49:34 -04:00
dm-ima.h	dm ima: add version info to dm related events in ima log	2021-08-20 15:59:47 -04:00
dm-init.c
dm-integrity.c	dm/dm-integrity: Combine request operation and flags	2022-07-14 12:14:31 -06:00
dm-io-rewind.c	dm: add two stage requeue mechanism	2022-07-07 11:49:32 -04:00
dm-io-tracker.h	dm writecache: make writeback pause configurable	2021-06-28 16:30:13 -04:00
dm-io.c	dm/core: Combine request operation type and flags	2022-07-14 12:14:31 -06:00
dm-ioctl.c	dm table: remove dm_table_get_num_targets() wrapper	2022-07-07 11:49:33 -04:00
dm-kcopyd.c	- Refactor DM core's mempool allocation so that it clearer by not	2022-08-02 14:21:25 -07:00
dm-linear.c	libnvdimm for 5.19	2022-05-27 15:49:30 -07:00
dm-log-userspace-base.c	dm: update target status functions to support IMA measurement	2021-08-10 13:34:23 -04:00
dm-log-userspace-transfer.c
dm-log-userspace-transfer.h
dm-log-writes.c	libnvdimm for 5.19	2022-05-27 15:49:30 -07:00
dm-log.c	dm mirror log: Use the new blk_opf_t type	2022-07-14 12:14:31 -06:00
dm-mpath.c	dm mpath: provide high-resolution timer to HST for bio-based	2022-05-09 15:39:23 -04:00
dm-mpath.h
dm-path-selector.c
dm-path-selector.h	dm mpath: provide high-resolution timer to HST for bio-based	2022-05-09 15:39:23 -04:00
dm-ps-historical-service-time.c	dm mpath: provide high-resolution timer to HST for bio-based	2022-05-09 15:39:23 -04:00
dm-ps-io-affinity.c	dm: update target status functions to support IMA measurement	2021-08-10 13:34:23 -04:00
dm-ps-queue-length.c	dm: update target status functions to support IMA measurement	2021-08-10 13:34:23 -04:00
dm-ps-round-robin.c	dm: update target status functions to support IMA measurement	2021-08-10 13:34:23 -04:00
dm-ps-service-time.c	dm: update target status functions to support IMA measurement	2021-08-10 13:34:23 -04:00
dm-raid.c	md: unlock mddev before reap sync_thread in action_store	2022-08-02 17:14:40 -06:00
dm-raid1.c	dm/core: Reduce the size of struct dm_io_request	2022-07-14 12:14:31 -06:00
dm-region-hash.c
dm-rq.c	dm: unexport dm_get_reserved_rq_based_ios	2022-06-29 12:46:05 -04:00
dm-rq.h
dm-snap-persistent.c	dm-snap: Combine request operation type and flags	2022-07-14 12:14:31 -06:00
dm-snap-transient.c	dm: update target status functions to support IMA measurement	2021-08-10 13:34:23 -04:00
dm-snap.c	dm snapshot: fix typo in snapshot_map() comment	2022-07-07 11:49:39 -04:00
dm-stats.c	dm stats: add cond_resched when looping over entries	2022-05-09 12:11:07 -04:00
dm-stats.h	dm stats: fix too short end duration_ns when using precise_timestamps	2022-02-21 15:35:39 -05:00
dm-stripe.c	dax: add .recovery_write dax_operation	2022-05-16 13:37:59 -07:00
dm-switch.c	dm: use bdev_nr_sectors and bdev_nr_bytes instead of open coding them	2021-10-18 14:43:22 -06:00
dm-sysfs.c	dm sysfs: use default_groups in kobj_type	2022-01-06 09:48:55 -05:00
dm-table.c	- Refactor DM core's mempool allocation so that it clearer by not	2022-08-02 14:21:25 -07:00
dm-target.c	dax: introduce DAX_RECOVERY_WRITE dax access mode	2022-05-16 13:35:56 -07:00
dm-thin-metadata.c	dm thin: fix use-after-free crash in dm_sm_register_threshold_callback	2022-07-15 18:09:14 -04:00
dm-thin-metadata.h	dm thin metadata: remove unused dm_thin_remove_block and __remove	2022-02-22 13:55:50 -05:00
dm-thin.c	dm thin: fix use-after-free crash in dm_sm_register_threshold_callback	2022-07-15 18:09:14 -04:00
dm-uevent.c
dm-uevent.h
dm-unstripe.c	dm: update target status functions to support IMA measurement	2021-08-10 13:34:23 -04:00
dm-verity-fec.c	dm verity fec: fix misaligned RS roots IO	2021-04-14 14:28:29 -04:00
dm-verity-fec.h	dm verity fec: fix misaligned RS roots IO	2021-04-14 14:28:29 -04:00
dm-verity-target.c	dm verity: fix checkpatch close brace error	2022-07-07 11:49:36 -04:00
dm-verity-verify-sig.c	dm verity: fix require_signatures module_param permissions	2021-05-25 16:14:05 -04:00
dm-verity-verify-sig.h
dm-verity.h
dm-writecache.c	- Refactor DM core's mempool allocation so that it clearer by not	2022-08-02 14:21:25 -07:00
dm-zero.c	dm: add support for REQ_NOWAIT to various targets	2020-12-04 18:04:35 -05:00
dm-zone.c	- Refactor DM core's mempool allocation so that it clearer by not	2022-08-02 14:21:25 -07:00
dm-zoned-metadata.c	dm/dm-zoned: Use the enum req_op type	2022-07-14 12:14:31 -06:00
dm-zoned-reclaim.c	dm kcopyd: avoid useless atomic operations	2021-06-04 12:07:24 -04:00
dm-zoned-target.c	dm-zoned: cleanup dmz_fixup_devices	2022-07-06 06:46:26 -06:00
dm-zoned.h	dm/dm-zoned: Use the enum req_op type	2022-07-14 12:14:31 -06:00
dm.c	- Refactor DM core's mempool allocation so that it clearer by not	2022-08-02 14:21:25 -07:00
dm.h	dm table: audit all dm_table_get_target() callers	2022-07-07 11:49:34 -04:00
Kconfig	blk-mq: make the blk-mq stacking code optional	2022-02-16 19:39:09 -07:00
Makefile	dm: add dm_bio_rewind() API to DM core	2022-07-07 11:26:55 -04:00
md-autodetect.c	md: Explicitly create command-line configured devices	2022-08-02 17:14:40 -06:00
md-bitmap.c	fs/buffer: Combine two submit_bh() and ll_rw_block() arguments	2022-07-14 12:14:32 -06:00
md-bitmap.h
md-cluster.c	md: replace deprecated strlcpy & remove duplicated line	2022-04-25 14:00:36 -07:00
md-cluster.h
md-faulty.c	block: pass a block_device to bio_clone_fast	2022-02-04 07:43:18 -07:00
md-linear.c	md: remove most calls to bdevname	2022-05-22 23:07:21 -07:00
md-linear.h
md-multipath.c	md: remove most calls to bdevname	2022-05-22 23:07:21 -07:00
md-multipath.h
md.c	md: unlock mddev before reap sync_thread in action_store	2022-08-02 17:14:40 -06:00
md.h	md: Explicitly create command-line configured devices	2022-08-02 17:14:40 -06:00
raid0.c	md: Don't set mddev private to NULL in raid0 pers->free	2022-05-22 23:07:21 -07:00
raid0.h
raid1-10.c	md: raid1/raid10: drop pending_cnt	2022-03-08 15:16:54 -08:00
raid1.c	md/raid1: Use the new blk_opf_t type	2022-07-14 12:14:31 -06:00
raid1.h	md: raid1/raid10: drop pending_cnt	2022-03-08 15:16:54 -08:00
raid5-cache.c	md/raid5-cache: Annotate pslot with __rcu notation	2022-08-02 17:14:32 -06:00
raid5-log.h	md/raid5-ppl: Drop unused argument from ppl_handle_flush_request()	2022-08-02 17:14:31 -06:00
raid5-ppl.c	md/raid5-ppl: Drop unused argument from ppl_handle_flush_request()	2022-08-02 17:14:31 -06:00
raid5.c	md/raid5: Pivot raid5_make_request()	2022-08-02 17:14:42 -06:00
raid5.h	md/raid5: Add __rcu annotation to struct disk_info	2022-04-25 14:00:36 -07:00
raid10.c	md/raid10: Use the new blk_opf_t type	2022-07-14 12:14:32 -06:00
raid10.h	md: raid1/raid10: drop pending_cnt	2022-03-08 15:16:54 -08:00