linux/drivers
Huang Ying 362d37a106 mm, pcp: reduce lock contention for draining high-order pages
In commit f26b3fa046 ("mm/page_alloc: limit number of high-order pages
on PCP during bulk free"), the PCP (Per-CPU Pageset) will be drained when
PCP is mostly used for high-order pages freeing to improve the cache-hot
pages reusing between page allocating and freeing CPUs.

On system with small per-CPU data cache slice, pages shouldn't be cached
before draining to guarantee cache-hot.  But on a system with large
per-CPU data cache slice, some pages can be cached before draining to
reduce zone lock contention.

So, in this patch, instead of draining without any caching, "pcp->batch"
pages will be cached in PCP before draining if the size of the per-CPU
data cache slice is more than "3 * batch".

In theory, if the size of per-CPU data cache slice is more than "2 *
batch", we can reuse cache-hot pages between CPUs.  But considering the
other usage of cache (code, other data accessing, etc.), "3 * batch" is
used.

Note: "3 * batch" is chosen to make sure the optimization works on recent
x86_64 server CPUs.  If you want to increase it, please check whether it
breaks the optimization.

On a 2-socket Intel server with 128 logical CPU, with the patch, the
network bandwidth of the UNIX (AF_UNIX) test case of lmbench test suite
with 16-pair processes increase 70.5%.  The cycles% of the spinlock
contention (mostly for zone lock) decreases from 46.1% to 21.3%.  The
number of PCP draining for high order pages freeing (free_high) decreases
89.9%.  The cache miss rate keeps 0.2%.

Link: https://lkml.kernel.org/r/20231016053002.756205-4-ying.huang@intel.com
Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
Acked-by: Mel Gorman <mgorman@techsingularity.net>
Cc: Sudeep Holla <sudeep.holla@arm.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: David Hildenbrand <david@redhat.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2023-10-25 16:47:10 -07:00
..
accel accel/ivpu: Use cached buffers for FW loading 2023-09-27 07:40:43 +02:00
accessibility
acpi acpi, hmat: calculate abstract distance with HMAT 2023-10-16 15:44:39 -07:00
amba amba: bus: fix refcount leak 2023-08-22 15:50:57 +02:00
android binder: dynamically allocate the android-binder shrinker 2023-10-04 10:32:23 -07:00
ata ATA fixes for 6.6-rc4 2023-09-29 13:38:34 -07:00
atm
auxdisplay drm for 6.6-rc1 2023-08-30 13:34:34 -07:00
base mm, pcp: reduce lock contention for draining high-order pages 2023-10-25 16:47:10 -07:00
bcma
block rbd: take header_rwsem in rbd_dev_refresh() only when updating 2023-09-26 10:33:19 +02:00
bluetooth TTY/Serial driver changes for 6.6-rc1 2023-09-01 09:38:00 -07:00
bus bus: ti-sysc: Fix SYSC_QUIRK_SWSUP_SIDLE_ACT handling for uart wake-up 2023-09-13 10:24:53 +03:00
cache cache: Add L2 cache management for Andes AX45MP RISC-V core 2023-09-01 09:08:59 -07:00
cdrom
cdx
char Hi, 2023-09-13 11:44:20 -07:00
clk clk: tegra: fix error return case for recalc_rate 2023-09-12 10:56:05 -07:00
clocksource Updates for clocksource/clockevent drivers: 2023-09-04 13:15:57 -07:00
comedi Revert "comedi: add HAS_IOPORT dependencies" 2023-09-12 15:49:20 +02:00
connector
counter - New Drivers 2023-09-04 13:47:59 -07:00
cpufreq cpufreq: Support per-policy performance boost 2023-08-29 20:51:40 +02:00
cpuidle powerpc updates for 6.6 2023-08-31 12:43:10 -07:00
crypto This update includes the following changes: 2023-08-29 11:23:29 -07:00
cxl cxl/acpi: Annotate struct cxl_cxims_data with __counted_by 2023-09-22 14:31:04 -07:00
dax dax, kmem: calculate abstract distance with general interface 2023-10-16 15:44:39 -07:00
dca
devfreq
dio
dma dmaengine updates for v6.6 2023-09-03 10:49:42 -07:00
dma-buf drm for 6.6-rc1 2023-08-30 13:34:34 -07:00
edac Intel EDAC fixes: 2023-08-30 19:23:00 -07:00
eisa
extcon
firewire scsi: sd: Differentiate system and runtime start/stop management 2023-09-28 21:23:00 +09:00
firmware efi/unaccepted: do not let /proc/vmcore try to access unaccepted memory 2023-10-04 10:32:22 -07:00
fpga
fsi fsi: i2cr: Switch to use struct i2c_driver's .probe() 2023-08-22 15:51:33 +02:00
gnss
gpio gpio: pmic-eic-sprd: Add can_sleep flag for PMIC EIC chip 2023-09-27 09:12:09 +02:00
gpu drm/panfrost: dynamically allocate the drm-panfrost shrinker 2023-10-04 10:32:25 -07:00
greybus
hid for-linus-2023083101 2023-09-01 12:31:44 -07:00
hsi
hte hte: Explicitly include correct DT includes 2023-08-28 13:31:06 -05:00
hv hyperv-next for v6.6 2023-09-04 11:26:29 -07:00
hwmon hwmon: (nct6775) Fix non-existent ALARM warning 2023-09-18 11:52:18 -07:00
hwspinlock
hwtracing coresight: trbe: Fix TRBE potential sleep in atomic context 2023-08-18 16:42:26 +01:00
i2c i2c: npcm7xx: Fix callback completion ordering 2023-09-27 21:32:06 +02:00
i3c i3c: master: svc: fix probe failure when no i3c device exist 2023-09-06 01:21:47 +02:00
idle Perf events changes for v6.6: 2023-08-28 16:35:01 -07:00
iio Char/Misc driver changes for 6.6-rc1 2023-09-01 09:53:54 -07:00
infiniband SCSI misc on 20230902 2023-09-02 12:02:41 -07:00
input Input updates for 6.6 merge window: 2023-09-06 09:24:25 -07:00
interconnect This pull request is full of clk driver changes. In fact, there aren't any 2023-08-30 19:53:39 -07:00
iommu IOMMU Updates for Linux v6.6 2023-09-01 16:54:25 -07:00
ipack
irqchip irqchip: irq-xtensa-mx: include header for missing prototype 2023-09-20 05:03:20 -07:00
isdn Merge commit b320441c04 ("Merge tag 'tty-6.5-rc7' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty") into tty-next 2023-08-20 14:29:37 +02:00
leds - Core Frameworks 2023-09-04 13:52:58 -07:00
macintosh powerpc updates for 6.6 2023-08-31 12:43:10 -07:00
mailbox mailbox: qcom-ipcc: fix incorrect num_chans counting 2023-09-05 10:11:01 -05:00
mcb
md mm: shrinker: convert shrinker_rwsem to mutex 2023-10-04 10:32:26 -07:00
media media: imx-mipi-csis: Remove an incorrect fwnode_handle_put() call 2023-09-19 09:03:21 +02:00
memory
memstick
message
mfd mfd: cs42l43: Use correct macro for new-style PM runtime ops 2023-09-23 11:10:23 -07:00
misc vmw_balloon: dynamically allocate the vmw-balloon shrinker 2023-10-04 10:32:25 -07:00
mmc TTY/Serial driver changes for 6.6-rc1 2023-09-01 09:38:00 -07:00
most
mtd - New Drivers 2023-09-04 13:47:59 -07:00
mux mux: Explicitly include correct DT includes 2023-08-28 13:36:24 -05:00
net sfc: handle error pointers returned by rhashtable_lookup_get_insert_fast() 2023-09-21 10:37:16 +02:00
nfc NFC: nxp: add NXP1002 2023-08-30 18:32:24 -07:00
ntb ntb: Check tx descriptors outstanding instead of head/tail for tx queue 2023-08-22 12:38:19 -04:00
nubus
nvdimm nvdimm changes for v6.6 merge window 2023-08-30 20:52:08 -07:00
nvme nvme fixes for Linux 6.6 2023-09-14 16:20:31 -06:00
nvmem nvmem: core: Notify when a new layout is registered 2023-08-23 16:34:02 +02:00
of Devicetree updates for v6.6: 2023-08-30 16:59:03 -07:00
opp
parisc parisc: iosapic.c: Fix sparse warnings 2023-08-31 21:42:42 +02:00
parport TTY/Serial driver changes for 6.6-rc1 2023-09-01 09:38:00 -07:00
pci PCI/AER: Export pcie_aer_is_native() 2023-09-11 15:24:16 -07:00
pcmcia
peci
perf arm64 fixes for -rc1 2023-09-08 12:48:37 -07:00
phy phy-for-6.6 2023-09-03 10:38:02 -07:00
pinctrl Pin control bulk changes for the v6.6 kernel cycle: 2023-08-30 19:36:19 -07:00
platform platform/x86: thinkpad_acpi: Take mutex in hotkey_resume 2023-09-18 15:16:19 +02:00
pmdomain pmdomain: Rename the genpd subsystem to pmdomain 2023-09-13 11:09:21 +02:00
pnp
power power: supply: rk817: Fix node refcount leak 2023-09-20 19:37:15 +02:00
powercap powercap: intel_rapl: Fix invalid setting of Power Limit 4 2023-09-06 22:21:22 +02:00
pps
ps3
ptp
pwm pwm: Changes for v6.6-rc1 2023-09-07 18:05:58 -07:00
rapidio
ras
regulator regulator: Fix voltage range selection 2023-09-11 13:51:36 +01:00
remoteproc remoteproc updates for v6.6 2023-09-04 15:12:26 -07:00
reset This pull request is full of clk driver changes. In fact, there aren't any 2023-08-30 19:53:39 -07:00
rpmsg rpmsg updates for v6.6 2023-09-04 15:08:52 -07:00
rtc RTC for 6.6 2023-09-07 16:07:35 -07:00
s390 block-6.6-2023-09-08 2023-09-08 21:39:54 -07:00
sbus sbus: Explicitly include correct DT includes 2023-08-28 13:36:24 -05:00
scsi ATA fixes for 6.6-rc4 2023-09-29 13:38:34 -07:00
sh
siox
slimbus
soc soc: loongson: loongson2_guts: Remove unneeded semicolon 2023-09-27 11:05:47 +02:00
soundwire soundwire updates for 6.6 2023-09-03 10:20:57 -07:00
spi spi: spi-gxp: BUG: Correct spi write return value 2023-09-27 17:06:36 +02:00
spmi
ssb
staging media: dvb: symbol fixup for dvb_attach() 2023-09-09 08:15:11 +01:00
target scsi: target: core: Fix target_cmd_counter leak 2023-09-13 20:09:56 -04:00
tc
tee tee: Remove unused declarations 2023-09-13 08:16:24 +02:00
thermal thermal: sysfs: Fix trip_point_hyst_store() 2023-09-18 13:13:05 +02:00
thunderbolt thunderbolt: Changes for v6.6 merge window 2023-08-22 14:22:35 +02:00
tty Revert "tty: n_gsm: fix UAF in gsm_cleanup_mux" 2023-09-18 10:12:11 +02:00
ufs scsi: ufs: core: Poll HCS.UCRDY before issuing a UIC command 2023-09-05 06:10:24 -04:00
uio uio: pruss: fix missing iounmap() in pruss_probe() 2023-08-22 13:41:55 +02:00
usb usb: typec: ucsi: Fix NULL pointer dereference 2023-09-11 13:52:16 +02:00
vdpa virtio: features 2023-09-04 10:43:44 -07:00
vfio vfio/mdev: Fix a null-ptr-deref bug for mdev_unregister_parent() 2023-09-22 12:48:04 -06:00
vhost vdpa: add get_backend_features vdpa operation 2023-09-03 18:10:22 -04:00
video fbdev/sh7760fb: Depend on FB=y 2023-09-21 10:33:49 +02:00
virt minmax: add in_range() macro 2023-08-24 16:20:18 -07:00
virtio virtio_balloon: dynamically allocate the virtio-balloon shrinker 2023-10-04 10:32:25 -07:00
vlynq
w1 w1: ds2482: Switch back to use struct i2c_driver's .probe() 2023-09-13 10:48:42 +02:00
watchdog linux-watchdog 6.6-rc1 tag 2023-09-06 09:19:12 -07:00
xen xenbus/backend: dynamically allocate the xen-backend shrinker 2023-10-04 10:32:23 -07:00
zorro zorro: Include zorro.h in names.c 2023-08-21 13:27:44 +02:00
Kconfig Merge patch series "Add non-coherent DMA support for AX45MP" 2023-09-08 11:24:34 -07:00
Makefile pmdomain: Rename the genpd subsystem to pmdomain 2023-09-13 11:09:21 +02:00