linux/include/asm-ia64
akpm@osdl.org 198e2f1811 [PATCH] scheduler cache-hot-autodetect
)

From: Ingo Molnar <mingo@elte.hu>

This is the latest version of the scheduler cache-hot-auto-tune patch.

The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:

- I've added a 'domain distance' function, which is used to cache
  measurement results. Each distance is only measured once. This means
  that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
  distances 0 and 1, and on SMP distance 0 is measured. The code walks
  the domain tree to determine the distance, so it automatically follows
  whatever hierarchy an architecture sets up. This cuts down on the boot
  time significantly and removes the O(N^2) limit. The only assumption
  is that migration costs can be expressed as a function of domain
  distance - this covers the overwhelming majority of existing systems,
  and is a good guess even for more assymetric systems.

  [ People hacking systems that have assymetries that break this
    assumption (e.g. different CPU speeds) should experiment a bit with
    the cpu_distance() function. Adding a ->migration_distance factor to
    the domain structure would be one possible solution - but lets first
    see the problem systems, if they exist at all. Lets not overdesign. ]

Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:

- Instead of relying on a single cache-size provided by the platform and
  sticking to it, the code now auto-detects the 'effective migration
  cost' between two measured CPUs, via iterating through a wide range of
  cachesizes. The code searches for the maximum migration cost, which
  occurs when the working set of the test-workload falls just below the
  'effective cache size'. I.e. real-life optimized search is done for
  the maximum migration cost, between two real CPUs.

  This, amongst other things, has the positive effect hat if e.g. two
  CPUs share a L2/L3 cache, a different (and accurate) migration cost
  will be found than between two CPUs on the same system that dont share
  any caches.

(The reliable measurement of migration costs is tricky - see the source
for details.)

Furthermore i've added various boot-time options to override/tune
migration behavior.

Firstly, there's a blanket override for autodetection:

	migration_cost=1000,2000,3000

will override the depth 0/1/2 values with 1msec/2msec/3msec values.

Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:

	migration_factor=120

will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.

I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:

Dual Celeron (128K L2 cache):

 ---------------------
 migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
 ---------------------
           [00]    [01]
 [00]:     -     1.7(1)
 [01]:   1.7(1)    -
 ---------------------
 cacheflush times [2]: 0.0 (0) 1.7 (1784008)
 ---------------------

Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.

Dual HT P4 (512K L2 cache):

 ---------------------
 migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
 ---------------------
           [00]    [01]    [02]    [03]
 [00]:     -     0.4(1)  0.0(0)  0.4(1)
 [01]:   0.4(1)    -     0.4(1)  0.0(0)
 [02]:   0.0(0)  0.4(1)    -     0.4(1)
 [03]:   0.4(1)  0.0(0)  0.4(1)    -
 ---------------------
 cacheflush times [2]: 0.0 (33900) 0.4 (448514)
 ---------------------

Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.

8-way P3/Xeon [2MB L2 cache]:

 ---------------------
 migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
 ---------------------
           [00]    [01]    [02]    [03]    [04]    [05]    [06]    [07]
 [00]:     -    19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [01]:  19.2(1)    -    19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [02]:  19.2(1) 19.2(1)    -    19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [03]:  19.2(1) 19.2(1) 19.2(1)    -    19.2(1) 19.2(1) 19.2(1) 19.2(1)
 [04]:  19.2(1) 19.2(1) 19.2(1) 19.2(1)    -    19.2(1) 19.2(1) 19.2(1)
 [05]:  19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)    -    19.2(1) 19.2(1)
 [06]:  19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)    -    19.2(1)
 [07]:  19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)    -
 ---------------------
 cacheflush times [2]: 0.0 (0) 19.2 (19281756)
 ---------------------

This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.

Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 09:08:50 -08:00
..
sn [IA64-SGI] altix: pci_window fixup 2005-12-06 09:09:23 -08:00
a.out.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
acpi-ext.h [ACPI] fix IA64 build warning 2005-08-04 22:29:34 -04:00
acpi.h [IA64] Add ACPI based P-state support 2005-08-26 15:09:24 -07:00
agp.h [PATCH] AGP fix for Xen VMM 2005-06-07 12:35:43 -07:00
asmmacro.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
atomic.h [PATCH] mutex subsystem, add atomic_xchg() to all arches 2006-01-09 15:59:17 -08:00
auxvec.h [PATCH] auxiliary vector cleanups 2005-09-07 16:57:21 -07:00
bitops.h [FLS64]: generic version 2006-01-03 13:11:06 -08:00
break.h [PATCH] Kprobes/IA64: kdebug die notification mechanism 2005-06-23 09:45:22 -07:00
bug.h [PATCH] remove gcc-2 checks 2006-01-08 20:14:02 -08:00
bugs.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
byteorder.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
cache.h [PATCH] Kill L1_CACHE_SHIFT_MAX 2006-01-08 20:13:39 -08:00
cacheflush.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
checksum.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
compat.h [PATCH] compat: be more consistent about [ug]id_t 2005-09-07 16:57:19 -07:00
cpu.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
cputime.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
current.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
cyclone.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
delay.h [IA64] disable preemption in udelay() 2005-12-16 10:00:24 -08:00
div64.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
dma-mapping.h [PATCH] ia64: re-implement dma_get_cache_alignment to avoid EXPORT_SYMBOL 2005-11-07 07:53:23 -08:00
dma.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
elf.h [PATCH] auxiliary vector cleanups 2005-09-07 16:57:21 -07:00
emergency-restart.h [PATCH] Add emergency_restart() 2005-07-26 14:35:41 -07:00
errno.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
fcntl.h [PATCH] Clean up struct flock definitions 2005-09-07 16:57:38 -07:00
fpswa.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
fpu.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
futex.h [PATCH] consolidate asm/futex.h 2006-01-08 20:13:39 -08:00
gcc_intrin.h [IA64] fix fls() 2005-04-21 11:07:59 -07:00
hardirq.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
hw_irq.h [PATCH] x86/x86_64: deferred handling of writes to /proc/irqxx/smp_affinity 2005-09-07 16:57:15 -07:00
ia32.h [IA64] Change SET_PERSONALITY to comply with comment in binfmt_elf.c. 2005-12-06 09:12:34 -08:00
ia64regs.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
ide.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
intel_intrin.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
intrinsics.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
io.h [PATCH] /dev/mem: validate mmap requests 2006-01-08 20:14:02 -08:00
ioctl.h [PATCH] Generic ioctl.h 2006-01-10 08:01:34 -08:00
ioctls.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
iosapic.h [IA64] Minor cleanups - remove unnecessary function prototype in iosapic.h 2005-09-07 14:00:40 -07:00
ipcbuf.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
irq.h [IA64] Manual merge fix for 3 files 2005-09-08 14:27:13 -07:00
kdebug.h [IA64] Extend notify_die() hooks for IA64 2005-11-07 11:27:13 -08:00
kmap_types.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
kprobes.h [PATCH] kprobes: fix build breakage 2006-01-10 08:01:40 -08:00
kregs.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
linkage.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
local.h [IA64] Fix missing parameter for local_add/sub 2005-12-07 11:30:11 -08:00
machvec.h Merge branch 'release' of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux-2.6 2005-10-28 21:09:26 -07:00
machvec_dig.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
machvec_hpsim.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
machvec_hpzx1.h [IA64] more robust zx1/sx1000 machvec support 2005-09-14 16:22:11 -07:00
machvec_hpzx1_swiotlb.h [IA64] more robust zx1/sx1000 machvec support 2005-09-14 16:22:11 -07:00
machvec_init.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
machvec_sn2.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
mc146818rtc.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
mca.h [IA64] Wire in the MCA/INIT handler stacks 2005-09-22 13:24:19 -07:00
mca_asm.h [PATCH] MCA/INIT: use per cpu stacks 2005-09-11 14:08:41 -07:00
meminit.h Pull sparsemem-v5 into release branch 2005-10-28 14:32:56 -07:00
mman.h [PATCH] madvise(MADV_REMOVE): remove pages from tmpfs shm backing store 2006-01-06 08:33:22 -08:00
mmu.h [IA64] Fix race in mm-context wrap-around logic. 2005-08-12 15:05:21 -07:00
mmu_context.h [IA64] make mmu_context.h and tlb.c 80-column friendly 2005-11-03 14:43:50 -08:00
mmzone.h [PATCH] V5 ia64 SPARSEMEM - conditional changes for SPARSEMEM 2005-10-04 13:21:13 -07:00
module.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
msgbuf.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
msi.h [PATCH] PCI: Change MSI to use physical delivery mode always 2005-11-10 16:09:18 -08:00
mutex.h [PATCH] mutex subsystem, add default include/asm-*/mutex.h files 2006-01-09 15:59:19 -08:00
namei.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
nodedata.h [PATCH] V5 ia64 SPARSEMEM - conditional changes for SPARSEMEM 2005-10-04 13:21:13 -07:00
numa.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
numnodes.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
page.h [IA64] - Make pfn_valid more precise for SGI Altix systems 2005-11-29 09:24:10 -08:00
pal.h [IA64] Add ACPI based P-state support 2005-08-26 15:09:24 -07:00
param.h [PATCH] ia64: Selectable Timer Interrupt Frequency 2005-06-23 09:45:10 -07:00
parport.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
patch.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
pci.h [PATCH] Make sparc64 use setup-res.c 2005-09-08 14:57:25 -07:00
percpu.h [PATCH] adjust per_cpu definition in non-SMP case 2005-06-23 09:45:28 -07:00
perfmon.h [IA64] alternate perfmon handler 2005-05-18 16:14:30 -07:00
perfmon_default_smpl.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
pgalloc.h [IA64] 4-level page tables 2005-11-11 09:37:29 -08:00
pgtable.h [IA64] 4-level page tables 2005-11-11 09:37:29 -08:00
poll.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
posix_types.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
processor.h [PATCH] cpusets: Move the ia64 domain setup code to the generic code 2005-09-07 16:57:40 -07:00
ptrace.h [PATCH] consolidate sys_ptrace() 2005-11-07 07:53:42 -08:00
ptrace_offsets.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
resource.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
rse.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
rwsem.h [PATCH] add sem_is_read/write_locked() 2005-10-29 21:40:35 -07:00
sal.h [IA64] Split 16-bit severity field in sal_log_record_header 2005-12-13 10:41:49 -08:00
scatterlist.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
sections.h [PATCH] kprobes/ia64: refuse kprobe on ivt code 2005-06-27 15:23:54 -07:00
segment.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
semaphore.h [PATCH] semaphore: Remove __MUTEX_INITIALIZER() 2005-10-30 17:37:27 -08:00
sembuf.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
serial.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
setup.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
shmbuf.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
shmparam.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
sigcontext.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
siginfo.h [PATCH] consolidate SIGEV_PAD_SIZE 2005-05-01 08:59:08 -07:00
signal.h [PATCH] asm/signal.h unification 2005-05-04 07:33:15 -07:00
smp.h [PATCH] smp_processor_id() cleanup 2005-06-21 18:46:13 -07:00
socket.h [NET]: Introduce SO_{SND,RCV}BUFFORCE socket options 2005-08-29 15:31:35 -07:00
sockios.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
sparsemem.h [PATCH] V5 ia64 SPARSEMEM - conditional changes for SPARSEMEM 2005-10-04 13:21:13 -07:00
spinlock.h [PATCH] remove gcc-2 checks 2006-01-08 20:14:02 -08:00
spinlock_types.h [PATCH] spinlock consolidation 2005-09-10 10:06:21 -07:00
stat.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
statfs.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
string.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
suspend.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
system.h [PATCH] sched: add cacheflush() asm 2006-01-12 09:08:49 -08:00
termbits.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
termios.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
thread_info.h [IA64] fix circular dependency on generation of asm-offsets.h 2005-09-13 08:50:39 -07:00
timex.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
tlb.h [PATCH] mm: tlb_finish_mmu forget rss 2005-10-29 21:40:37 -07:00
tlbflush.h [IA64] Use bitmaps for efficient context allocation/free 2005-10-31 14:36:05 -08:00
topology.h [PATCH] scheduler cache-hot-autodetect 2006-01-12 09:08:50 -08:00
types.h [PATCH] sab: consolidate kmem_bufctl_t 2005-09-05 00:05:48 -07:00
uaccess.h [PATCH] ia64 basic __user annotations 2005-09-29 08:46:27 -07:00
ucontext.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
unaligned.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
uncached.h [PATCH] ia64 uncached alloc 2005-06-21 18:46:18 -07:00
unistd.h [PATCH] Swap Migration V5: sys_migrate_pages interface 2006-01-08 20:12:42 -08:00
unwind.h [IA64] MCA/INIT: remove obsolete unwind code 2005-09-11 14:09:34 -07:00
user.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
ustack.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00
vga.h [IA64-SGI] pcdp: add PCDP pci interface support 2005-06-28 09:09:06 -07:00
xor.h Linux-2.6.12-rc2 2005-04-16 15:20:36 -07:00