if it was above 4GB. This was seen with CentOS 6.5 guests with
large RAM, since the block drivers are loaded late in the
boot sequence and end up allocating descriptor memory from
high addresses.
Reported by: Michael Dexter
MFC after: 3 days
LPC devices. Among other things, the LPC serial ports now appear as
ACPI devices.
- Move the info for the top-level PCI bus into the PCI emulation code and
add ResourceProducer entries for the memory ranges decoded by the bus
for memory BARs.
- Add a framework to allow each PCI emulation driver to optionally write
an entry into the DSDT under the \_SB_.PCI0 namespace. The LPC driver
uses this to write a node for the LPC bus (\_SB_.PCI0.ISA).
- Add a linker set to allow any LPC devices to write entries into the
DSDT below the LPC node.
- Move the existing DSDT block for the RTC to the RTC driver.
- Add DSDT nodes for the AT PIC, the 8254 ISA timer, and the LPC UART
devices.
- Add a "SuperIO" device under the LPC node to claim "system resources"
aling with a linker set to allow various drivers to add IO or memory
ranges that should be claimed as a system resource.
- Add system resource entries for the extended RTC IO range, the registers
used for ACPI power management, the ELCR, PCI interrupt routing register,
and post data register.
- Add various helper routines for generating DSDT entries.
Reviewed by: neel (earlier version)
hides the setjmp/longjmp semantics of VM enter/exit. vmx_enter_guest() is used
to enter guest context and vmx_exit_guest() is used to transition back into
host context.
Fix a longstanding race where a vcpu interrupt notification might be ignored
if it happens after vmx_inject_interrupts() but before host interrupts are
disabled in vmx_resume/vmx_launch. We now called vmx_inject_interrupts() with
host interrupts disabled to prevent this.
Suggested by: grehan@
to SIGTERM when ACPI is enabled. Sending SIGTERM to the hypervisor when an
ACPI-aware OS is running will now trigger a soft-off allowing for a graceful
shutdown of the guest.
- Move constants for ACPI-related registers to acpi.h.
- Implement an SMI_CMD register with commands to enable and disable ACPI.
Currently the only change when ACPI is enabled is to enable the virtual
power button via SIGTERM.
- Implement a fixed-feature power button when ACPI is enabled by asserting
PWRBTN_STS in PM1_EVT when SIGTERM is received.
- Add support for EVFILT_SIGNAL events to mevent.
- Implement support for the ACPI system command interrupt (SCI) and assert
it when needed based on the values in PM1_EVT. Mark the SCI as active-low
and level triggered in the MADT and MP Table.
- Mark PCI interrupts in the MP Table as active-low in addition to level
triggered.
Reviewed by: neel
- Implement the PM1_EVT and PM1_CTL registers required by ACPI.
The PM1_EVT register is mostly a dummy as bhyve doesn't support any
of the hardware-initiated events. The only bit of PM1_CNT that is
implemented are the sleep request bits (SPL_EN and SLP_TYP) which
request a graceful power off for S5. In particular, for S5, bhyve
exits with a non-zero value which terminates the loop in vmrun.sh.
- Emulate the Reset Control register at I/O port 0xcf9 and advertise
it as the reset register via ACPI.
- Advertise an _S5 package.
- Extend the in/out interface to allow an in/out handler to request
that the hypervisor trigger a reset or power-off.
- While here, note that all vCPUs in a guest support C1 ("hlt").
Reviewed by: neel (earlier version)
- Add a generic routine to trigger an LVT interrupt that supports both
fixed and NMI delivery modes.
- Add an ioctl and bhyvectl command to trigger local interrupts inside a
guest. In particular, a global NMI similar to that raised by SERR# or
PERR# can be simulated by asserting LINT1 on all vCPUs.
- Extend the LVT table in the vCPU local APIC to support CMCI.
- Flesh out the local APIC error reporting a bit to cache errors and
report them via ESR when ESR is written to. Add support for asserting
the error LVT when an error occurs. Raise illegal vector errors when
attempting to signal an invalid vector for an interrupt or when sending
an IPI.
- Ignore writes to reserved bits in LVT entries.
- Export table entries the MADT and MP Table advertising the stock x86
config of LINT0 set to ExtInt and LINT1 wired to NMI.
Reviewed by: neel (earlier version)
state before the requested state transition. This guarantees that there is
exactly one ioctl() operating on a vcpu at any point in time and prevents
unintended state transitions.
More details available here:
http://lists.freebsd.org/pipermail/freebsd-virtualization/2013-December/001825.html
Reviewed by: grehan
Reported by: Markiyan Kushnir (markiyan.kushnir at gmail.com)
MFC after: 3 days
callers treat the MSI 'addr' and 'data' fields as opaque and also lets
bhyve implement multiple destination modes: physical, flat and clustered.
Submitted by: Tycho Nightingale (tycho.nightingale@pluribusnetworks.com)
Reviewed by: grehan@
vcpu and destroy its thread context. Also modify the 'HLT' processing to ignore
pending interrupts in the IRR if interrupts have been disabled by the guest.
The interrupt cannot be injected into the guest in any case so resuming it
is futile.
With this change "halt" from a Linux guest works correctly.
Reviewed by: grehan@
Tested by: Tycho Nightingale (tycho.nightingale@pluribusnetworks.com)
commit level triggered interrupts would work as long as the pin was not shared
among multiple interrupt sources.
The vlapic now keeps track of level triggered interrupts in the trigger mode
register and will forward the EOI for a level triggered interrupt to the
vioapic. The vioapic in turn uses the EOI to sample the level on the pin and
re-inject the vector if the pin is still asserted.
The vhpet is the first consumer of level triggered interrupts and advertises
that it can generate interrupts on pins 20 through 23 of the vioapic.
Discussed with: grehan@
bhyveload: introduce the -c <device> parameter
to select a tty for output (or "stdio")
bhyve: allow the puc and lpc-com backends to
accept a tty in addition to "stdio"
When used in conjunction with the null-modem device,
nmdm(4), this allows attach/detach to the guest console
and multiple concurrent serial ports. kgdb on a serial
port is now functional.
Reviewed by: neel
Requested by: Almost everyone that has used bhyve
MFC after: 10.0
Table is 22 bits, with the bit 31 being the interrupt-on-completion
bit.
OpenBSD and UEFI set this bit, resulting in large block i/o lengths
being sent to bhyve and coredumping the process. Fix by masking off
the relevant 22 bits when using the DBC field as a length.
Reviewed by: Zhixiang Yu
Discussed with: Tycho Nightingale (tycho.nightingale@pluribusnetworks.com)
MFC after: 10.0
actual value read by the guest from the device. The IOAPIC ID is now set to
zero in both MPtable/ACPI tables as well as in the ioapic device emulation.
Pointed out by: grehan@
bhyve supports a single timer block with 8 timers. The timers are all 32-bit
and capable of being operated in periodic mode. All timers support interrupt
delivery using MSI. Timers 0 and 1 also support legacy interrupt routing.
At the moment the timers are not connected to any ioapic pins but that will
be addressed in a subsequent commit.
This change is based on a patch from Tycho Nightingale (tycho.nightingale@pluribusnetworks.com).
to inject edge triggered legacy interrupts into the guest.
Start using the new API in device models that use edge triggered interrupts:
viz. the 8254 timer and the LPC/uart device emulation.
Submitted by: Tycho Nightingale (tycho.nightingale@pluribusnetworks.com)
upcoming in-kernel device emulations like the HPET.
The ioctls VM_IOAPIC_ASSERT_IRQ and VM_IOAPIC_DEASSERT_IRQ are used to
manipulate the ioapic pin state.
Discussed with: grehan@
Submitted by: Tycho Nightingale (tycho.nightingale@pluribusnetworks.com)
pin 2 of the IOAPIC.
Add an 'Interrupt Source Override' entry to the MADT to describe this
and start asserting interrupts on pin 2 in the 8254 device model.
Submitted by: Tycho Nightingale (tycho.nightingale@pluribusnetworks.com)
Remove the VM name from some of the thread-naming calls
since it is now in the proc title.
Slightly modify the thread-naming for the net and block
threads.
This improves readability when using top/ps with the -a
and -H options on a system with a large number of bhyve VMs.
Requested by: Michael Dexter
Reviewed by: neel
MFC after: 4 weeks
to a virtual machine then we implicitly create COM1 and COM2 ISA devices.
Prior to this change the only way of attaching a COM port to the virtual
machine was by presenting it as a PCI device that is mapped at the legacy
I/O address 0x3F8 or 0x2F8.
There were some issues with the original approach:
- It did not work at all with UEFI because UEFI will reprogram the PCI device
BARs and remap the COM1/COM2 ports at non-legacy addresses.
- OpenBSD GENERIC kernel does not create a /dev/console because it expects
the uart device at the legacy 0x3F8/0x2F8 address to be an ISA device.
- It was functional with a FreeBSD guest but caused the console to appear
on /dev/ttyu2 which was not intuitive.
The uart emulation is now independent of the bus on which it resides. Thus it
is possible to have uart devices on the PCI bus in addition to the legacy
COM1/COM2 devices behind the LPC bus.
The command line option to attach ISA COM1/COM2 ports to a virtual machine is
"-s <bus>,lpc -l com1,stdio".
The command line option to create a PCI-attached uart device is:
"-s <bus>,uart[,stdio]"
The command line option to create PCI-attached COM1/COM2 device is:
"-S <bus>,uart[,stdio]". This style of creating COM ports is deprecated.
Discussed with: grehan
Reviewed by: grehan
Submitted by: Tycho Nightingale (tycho.nightingale@pluribusnetworks.com)
M share/examples/bhyve/vmrun.sh
AM usr.sbin/bhyve/legacy_irq.c
AM usr.sbin/bhyve/legacy_irq.h
M usr.sbin/bhyve/Makefile
AM usr.sbin/bhyve/uart_emul.c
M usr.sbin/bhyve/bhyverun.c
AM usr.sbin/bhyve/uart_emul.h
M usr.sbin/bhyve/pci_uart.c
M usr.sbin/bhyve/pci_emul.c
M usr.sbin/bhyve/inout.c
M usr.sbin/bhyve/pci_emul.h
M usr.sbin/bhyve/inout.h
AM usr.sbin/bhyve/pci_lpc.c
AM usr.sbin/bhyve/pci_lpc.h
where a pin assertion while a source was masked would result in
the interrupt being lost, with the symptom being a console hang.
The condition is now recorded, and the interrupt generated when
the source is unmasked.
Discovered by: OpenBSD 5.4 MP
Reviewed by: neel
MFC after: 3 days
- Use #defines for capability bits
- Export the VTBLK_F_BLK_SIZE capability
- Fix bug in calculating capacity: it is in
512-byte units, not the underlying sector size
This allows virtio-blk to have backing devices
with non 512-byte sector sizes e.g. /dev/cd0, and
4K-block harddrives.
Reviewed by: neel
MFC after: 3 days
- remove assumption that the backing file/device had
512-byte sectors
- fix incorrect iovec size variable that would result
in a buffer overrun when an o/s issued an i/o request
with more s/g elements than the blockif api
Reviewed by: Zhixiang Yu (zxyu.core@gmail.com)
MFC after: 3 days
- Allow a hostbridge to be created with AMD as a vendor.
This passes the OpenBSD check to allow the use of MSI
on a PCI bus.
- Enable the i/o interrupt section of the mptable, and
populate it with unity ISA mappings. This allows the
'legacy' IRQ mappings of the PCI serial port to be
set up. Delete unused print routine that was obscuring code.
- Use the '-W' option to enable virtio single-vector MSI
rather than an environment variable. Update the virtio
net/block drivers to query this flag when setting up
interrupts.: bhyverun.c
- Fix the arithmetic used to derive the century byte in
RTC CMOS, as well as encoding it in BCD.
Reviewed by: neel
MFC after: 3 days
Linux writes to these nominally read-only registers,
so avoid having bhyve write warning messages to stdout
when the reg writes can be safely ignored. Change the
WPRINTF to DPRINTF which is conditional.
Reviewed by: mav
Discussed with: mav, Zhixiang Yu
MFC after: 3 days
'invpcid' instruction to the guest. Currently bhyve will try to enable this
capability unconditionally if it is available.
Consolidate code in bhyve to set the capabilities so it is no longer
duplicated in BSP and AP bringup.
Add a sysctl 'vm.pmap.invpcid_works' to display whether the 'invpcid'
instruction is available.
Reviewed by: grehan
MFC after: 3 days
annoying verbose boot error of the form
g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT
The ident returned by bhyve is a text string 'BHYVE-XXXX-XXXX', where
the X's are the first bytes of the md5 hash of the backing filename.
Reviewed by: neel
Approved by: re (gjb)
the 2 read-only bytes at the start of a PCI capability.
This is the sequence that OpenBSD uses when enabling
MSI interrupts, and works fine on real h/w.
In bhyve, convert the 4 byte write to a 2-byte write to
the r/w area past the first 2 r/o bytes of a capability.
Reviewed by: neel
Approved by: re@ (blanket)
"assignment makes pointer from integer without a cast", by changing the
cmd_lst and rbis members of struct ahci_port from integers to pointers.
Also surround a pow-of-2 test expression with parentheses to clarify it,
and avoid another gcc warning.
Approved by: re (glebius)
Reviewed by: grehan, mav
rather than opt-out.
Prior to this change if the "-g" option was not specified then a listening
socket for tunneling gdb packets would be opened at port 6466. If a second
virtual machine is fired up, also without the "-g" option, then that would
fail because there is already a listener on port 6466.
After this change if a gdb tunnel port needs to be created it needs to be
explicitly specified with a "-g <portnum>" command line option.
Reviewed by: grehan@
Approved by: re@ (blanket)
Make the amd64/pmap code aware of nested page table mappings used by bhyve
guests. This allows bhyve to associate each guest with its own vmspace and
deal with nested page faults in the context of that vmspace. This also
enables features like accessed/dirty bit tracking, swapping to disk and
transparent superpage promotions of guest memory.
Guest vmspace:
Each bhyve guest has a unique vmspace to represent the physical memory
allocated to the guest. Each memory segment allocated by the guest is
mapped into the guest's address space via the 'vmspace->vm_map' and is
backed by an object of type OBJT_DEFAULT.
pmap types:
The amd64/pmap now understands two types of pmaps: PT_X86 and PT_EPT.
The PT_X86 pmap type is used by the vmspace associated with the host kernel
as well as user processes executing on the host. The PT_EPT pmap is used by
the vmspace associated with a bhyve guest.
Page Table Entries:
The EPT page table entries as mostly similar in functionality to regular
page table entries although there are some differences in terms of what
bits are used to express that functionality. For e.g. the dirty bit is
represented by bit 9 in the nested PTE as opposed to bit 6 in the regular
x86 PTE. Therefore the bitmask representing the dirty bit is now computed
at runtime based on the type of the pmap. Thus PG_M that was previously a
macro now becomes a local variable that is initialized at runtime using
'pmap_modified_bit(pmap)'.
An additional wrinkle associated with EPT mappings is that older Intel
processors don't have hardware support for tracking accessed/dirty bits in
the PTE. This means that the amd64/pmap code needs to emulate these bits to
provide proper accounting to the VM subsystem. This is achieved by using
the following mapping for EPT entries that need emulation of A/D bits:
Bit Position Interpreted By
PG_V 52 software (accessed bit emulation handler)
PG_RW 53 software (dirty bit emulation handler)
PG_A 0 hardware (aka EPT_PG_RD)
PG_M 1 hardware (aka EPT_PG_WR)
The idea to use the mapping listed above for A/D bit emulation came from
Alan Cox (alc@).
The final difference with respect to x86 PTEs is that some EPT implementations
do not support superpage mappings. This is recorded in the 'pm_flags' field
of the pmap.
TLB invalidation:
The amd64/pmap code has a number of ways to do invalidation of mappings
that may be cached in the TLB: single page, multiple pages in a range or the
entire TLB. All of these funnel into a single EPT invalidation routine called
'pmap_invalidate_ept()'. This routine bumps up the EPT generation number and
sends an IPI to the host cpus that are executing the guest's vcpus. On a
subsequent entry into the guest it will detect that the EPT has changed and
invalidate the mappings from the TLB.
Guest memory access:
Since the guest memory is no longer wired we need to hold the host physical
page that backs the guest physical page before we can access it. The helper
functions 'vm_gpa_hold()/vm_gpa_release()' are available for this purpose.
PCI passthru:
Guest's with PCI passthru devices will wire the entire guest physical address
space. The MMIO BAR associated with the passthru device is backed by a
vm_object of type OBJT_SG. An IOMMU domain is created only for guest's that
have one or more PCI passthru devices attached to them.
Limitations:
There isn't a way to map a guest physical page without execute permissions.
This is because the amd64/pmap code interprets the guest physical mappings as
user mappings since they are numerically below VM_MAXUSER_ADDRESS. Since PG_U
shares the same bit position as EPT_PG_EXECUTE all guest mappings become
automatically executable.
Thanks to Alan Cox and Konstantin Belousov for their rigorous code reviews
as well as their support and encouragement.
Thanks for John Baldwin for reviewing the use of OBJT_SG as the backing
object for pci passthru mmio regions.
Special thanks to Peter Holm for testing the patch on short notice.
Approved by: re
Discussed with: grehan
Reviewed by: alc, kib
Tested by: pho
these.
The mux-vcpus option may return at some point, given it's utility
in finding bhyve (and FreeBSD) bugs.
Approved by: re@ (blanket)
Discussed with: neel@
https://wiki.freebsd.org/SummerOfCode2013/bhyveAHCI
This provides ICH8 SATA disk and ATAPI ports, selectable
via the bhyve slot command-line parameter:
SATA
-s <slot>,ahci-hd,<image-file>
ATAPI
-s <slot>,ahci-cd,<image-file>
Slight modifications by: grehan@
Approved by: re@ (blanket)
Obtained from: FreeBSD GSoC'13
