The kernel source contains several definitions of an ilog2 function;
some are slower than necessary, and one of them is incorrect.
Elimininate them all and define an ilog2 macro in libkern to replace
them, in a way that is fast, correct for all argument types, and, in a
GENERIC kernel, includes a check for an invalid zero parameter.
Folks at Microsoft have verified that having a correct ilog2
definition for their MANA driver doesn't break it.
Reviewed by: alc, markj, mhorne (older version), jhibbits (older version)
Differential Revision: https://reviews.freebsd.org/D45170
Differential Revision: https://reviews.freebsd.org/D45235
These are missing changes after 1228b93b41 when ref_count was
removed from bus_dma_tag_common and 1e3f42b6ba, when the address
arguments were switched to pointers.
Reviewed by: jhb
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D45289
Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.
Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/
Sponsored by: Netflix
This function needs to unmap all memory in a given SMMU context. Have
it iterate over all page table entries to find what has been mapped
rather than looking at virtual addresses.
While here use SMMU specific macros.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39185
This is not managed through the VM subsystem so only needs to hold the
data the SMMU driver needs.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39184
Now the PCI drivers have a common interface to read the IOMMU xref
and SID create a common function to read it. This fixes an issue where
we will call into an ACPI specific function when booting with FDT when
both are enabled.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39183
Use the xref from OF_xref_from_node for the smmu xref. We already have
a valid xref ID, there is no need to convert this to a memory address.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D39181
We only ever build a 4 level page table for the Arm SMMU. Remove the
support for a 3 level table.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38949
These are SMMU (and MALI GPU) specific. Give them a SMMU specific name.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D38948
Since IOMMU map entries store a reference to the domain in which they
reside, there is no need to pass the domain to iommu_gas_free_entry(),
iommu_gas_free_space(), and iommu_gas_free_region().
Push down the acquisition and release of the IOMMU domain lock into
iommu_gas_free_space() and iommu_gas_free_region().
Both of these changes allow for simplifications in the callers of the
functions without really complicating the functions themselves.
Moreover, the latter change eliminates the direct use of the IOMMU
domain lock from the x86-specific DMAR code.
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35995
Eliminate a possible case of use-after-free in an error handling path
after a mapping failure. Specifically, eliminate IOMMU_MAP_ENTRY_QI_NF
and instead perform the IOTLB invalidation synchronously. Otherwise,
when iommu_domain_unload_entry() is called and told not to free the
IOMMU map entry, the caller could free the entry before dmar_qi_task()
is finished with it.
Reviewed by: kib
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35878
This is needed to support non-PCI devices like memory-mapped
display controllers.
Split-out some initialization code from iommu_ctx_alloc() into
iommu_ctx_init() method so we could pass controller's MD-data
obtained from DTS to the driver prior to a CTX initialization.
Tested on Morello SoC.
Sponsored by: UKRI
For PCI devices we have entire L1 descriptor for every session ID (SID),
but for non-PCI (e.g. Display Processing Unit DPU), a single L1
descriptor serves multiple SIDs.
So prevent re-initialization of L1 descriptor if already initialized.
Don't free entire L1 descriptor on every STE removal.
Sponsored by: UKRI
One of the SMMU interrupt lines (priq) is optional and may be ommited in FDT.
Tested on ARM Morello Board, which has three SMMU units: first two have four
interrupt lines, last one has three interrupt lines.
Sponsored by: UKRI
Remove page zeroing code from consumers and stop specifying
VM_ALLOC_NOOBJ. In a few places, also convert an allocation loop to
simply use VM_ALLOC_WAITOK.
Similarly, convert vm_page_alloc_domain() callers.
Note that callers are now responsible for assigning the pindex.
Reviewed by: alc, hselasky, kib
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31986
This adds an arm64 iommu interface and a driver for Arm System Memory
Management Unit version 3.2 (ARM SMMU v3.2) specified in ARM IHI 0070C
document.
Hardware overview is provided in the header of smmu.c file.
The support is disabled by default. To enable add 'options IOMMU' to your
kernel configuration file.
The support was developed on Arm Neoverse N1 System Development Platform
(ARM N1SDP), kindly provided by ARM Ltd.
Currently, PCI-based devices and ACPI platforms are supported only.
The support was tested on IOMMU-enabled Marvell SATA controller,
Realtek Ethernet controller and a TI xHCI USB controller with a low to
medium load only.
Many thanks to Konstantin Belousov for help forming the generic IOMMU
framework that is vital for this project; to Andrew Turner for adding
IOMMU support to MSI interrupt code; to Mark Johnston for help with SMMU
page management; to John Baldwin for explaining various IOMMU bits.
Reviewed by: mmel
Relnotes: yes
Sponsored by: DARPA / AFRL
Sponsored by: Innovate UK (Digital Security by Design programme)
Differential Revision: https://reviews.freebsd.org/D24618
