An upcoming change to include bitset(9) macros from vm_page.h
causes a macro name collision with vchi's custom bitset macros.
This change was performed mechanically by:
sed -i .orig s/BITSET/VCHI_BITSET/g $(grep -rl BITSET sys/contrib/vchiq)
Reviewed by: andrew
Approved by: scottl (implicit)
MFC after: 1 week
Sponsored by: Ampere Computing, Inc.
Differential Revision: https://reviews.freebsd.org/D26177
There are several mechanisms by which a vm_page reference is held,
preventing the page from being freed back to the page allocator. In
particular, holding the page's object lock is sufficient to prevent the
page from being freed; holding the busy lock or a wiring is sufficent as
well. These references are protected by the page lock, which must
therefore be acquired for many per-page operations. This results in
false sharing since the page locks are external to the vm_page
structures themselves and each lock protects multiple structures.
Transition to using an atomically updated per-page reference counter.
The object's reference is counted using a flag bit in the counter. A
second flag bit is used to atomically block new references via
pmap_extract_and_hold() while removing managed mappings of a page.
Thus, the reference count of a page is guaranteed not to increase if the
page is unbusied, unmapped, and the object's write lock is held. As
a consequence of this, the page lock no longer protects a page's
identity; operations which move pages between objects are now
synchronized solely by the objects' locks.
The vm_page_wire() and vm_page_unwire() KPIs are changed. The former
requires that either the object lock or the busy lock is held. The
latter no longer has a return value and may free the page if it releases
the last reference to that page. vm_page_unwire_noq() behaves the same
as before; the caller is responsible for checking its return value and
freeing or enqueuing the page as appropriate. vm_page_wire_mapped() is
introduced for use in pmap_extract_and_hold(). It fails if the page is
concurrently being unmapped, typically triggering a fallback to the
fault handler. vm_page_wire() no longer requires the page lock and
vm_page_unwire() now internally acquires the page lock when releasing
the last wiring of a page (since the page lock still protects a page's
queue state). In particular, synchronization details are no longer
leaked into the caller.
The change excises the page lock from several frequently executed code
paths. In particular, vm_object_terminate() no longer bounces between
page locks as it releases an object's pages, and direct I/O and
sendfile(SF_NOCACHE) completions no longer require the page lock. In
these latter cases we now get linear scalability in the common scenario
where different threads are operating on different files.
__FreeBSD_version is bumped. The DRM ports have been updated to
accomodate the KPI changes.
Reviewed by: jeff (earlier version)
Tested by: gallatin (earlier version), pho
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D20486
The hold_count and wire_count fields of struct vm_page are separate
reference counters with similar semantics. The remaining essential
differences are that holds are not counted as a reference with respect
to LRU, and holds have an implicit free-on-last unhold semantic whereas
vm_page_unwire() callers must explicitly determine whether to free the
page once the last reference to the page is released.
This change removes the KPIs which directly manipulate hold_count.
Functions such as vm_fault_quick_hold_pages() now return wired pages
instead. Since r328977 the overhead of maintaining LRU for wired pages
is lower, and in many cases vm_fault_quick_hold_pages() callers would
swap holds for wirings on the returned pages anyway, so with this change
we remove a number of page lock acquisitions.
No functional change is intended. __FreeBSD_version is bumped.
Reviewed by: alc, kib
Discussed with: jeff
Discussed with: jhb, np (cxgbe)
Tested by: pho (previous version)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D19247
This allows replacing "sys/eventfilter.h" includes with "sys/_eventfilter.h"
in other header files (e.g., sys/{bus,conf,cpu}.h) and reduces header
pollution substantially.
EVENTHANDLER_DECLARE and EVENTHANDLER_LIST_DECLAREs were moved out of .c
files into appropriate headers (e.g., sys/proc.h, powernv/opal.h).
As a side effect of reduced header pollution, many .c files and headers no
longer contain needed definitions. The remainder of the patch addresses
adding appropriate includes to fix those files.
LOCK_DEBUG and LOCK_FILE_LINE_ARG are moved to sys/_lock.h, as required by
sys/mutex.h since r326106 (but silently protected by header pollution prior
to this change).
No functional change (intended). Of course, any out of tree modules that
relied on header pollution for sys/eventhandler.h, sys/lock.h, or
sys/mutex.h inclusion need to be fixed. __FreeBSD_version has been bumped.
The flag is not used by anything for years and supporting it requires an
explicit read from the lock when entering slow path.
Flag value is left unused on purpose.
Sponsored by: The FreeBSD Foundation
Although vchiq_build_date and vchiq_build_time are not used in current
vchi driver at the moment, make sure these value will not leak into
build later on if at some point they will be refered in some new
imported code
PR: 215494
Reported by: emaste
MFC after: 1 week
by bus_dmamem_alloc() which creates associated bus_dmamap_t map for us.
When this memory is freed by bus_dmamem_free(), the map is freed as well.
Thus there is no need to free it explicitly by bus_dmamap_destroy(),
which leads to double freeing.
Discussed with: gonzo
Approved by: kib (mentor)
- Emulate Linux mutex API using sx(9) locks with only exclusive operations
instead of mutex(9), in Linux mutexes are sleepable.
- Emulate Linux rwlock_t using rwlock(9) instead of sx(9). rwlock_t
in Linux are spin locks
- Use pmap_quick_enter_page/pmap_quick_remove_page to bounce non-cacheline
aligned head and tail fragments
- Switch from static fragment size to configurable one, newer firmware
passes cache line size as cache_line_size DTB parameter.
With these changes both RPi and RPi2 pass functinal part of vchiq_test
We can't use copyout because destination memory is userland address
in another process but we have reference to respective page so map
the page into kernel address space and copy fragments there
years for head. However, it is continuously misused as the mpsafe argument
for callout_init(9). Deprecate the flag and clean up callout_init() calls
to make them more consistent.
Differential Revision: https://reviews.freebsd.org/D2613
Reviewed by: jhb
MFC after: 2 weeks
If structure packed as __packed clang (and probably gcc) generates
code that loads word fields (e.g. tx_pos) byte-by-byte and if it's
modified by VideoCore in the same time as ARM loads the value result
is going to be mixed combination of bytes from previous value and
new one.
