| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
| |
Now we can handle EFI runtime service faults on arm64 do so.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46818
|
|
|
|
|
|
|
|
|
|
| |
This is a function that calls into the EFI Runtime Services, but can
handle a fault. To support this add a handler in assembly that can
restore the kernel state on an exception and return a failure to the
caller.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46817
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When we receive an exception while in a spinlock we shouldn't enable
interrupts. When entering a spinlock we disable interrupts so enabling
them here could cause surprising results.
The three cases that could cause this are:
1. A break-before-make sequence
2. Accessing possibly unmapped code with a fault handler
3. Buggy code
1 and 2 are supported later in the data abort handler, and 3 should be
fixed when found.
Reviewed by: mmel, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46816
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
sys/intr.h originally started life as an extract of arm's intr.h, and
this include was dropped in its place. Changes in flight want to add
some MD definitions that we'll use in the more MI parts of INTRNG.
Let's formally reverse the dependency now since this is way more
common in general. All of the includes switched in this change that I
spot-checked were in-fact wanting declarations historically included in
sys/intr.h anyways.
Reviewed by: andrew, imp, jrtc27, mhorne, mmel, olce
Differential Revision: https://reviews.freebsd.org/D47002
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
uint32_t is handy for directly interfacing with assembly-language. For
the C portion, enum is much handier. In particular there is no need to
count the number of roots by hand. This also works better for being
able to build kernels with varying numbers of roots.
Switch to INTR_ROOT_COUNT as this better matches the purpose of the
value. Switch to root_type, rather than rootnum for similar reasons.
Remove the default from the core. Better to require the architectures
to declare the type since they will routinely deviate and a default
chosen now will likely be suboptimal.
Leave intr_irq_handler() taking a register type as that better matches
for interfacing with assembly-language.
|
|
|
|
|
|
|
| |
Rather than keeping the entire machine interrupt header assembly-safe,
switch to use of the existing genassym.c program to generate them. This
will be handier in the long-term, most of the header never needs to be
exposed to assembly-language.
|
|
|
|
|
|
|
|
|
|
|
|
| |
Use the same method to decode CTR_EL0 as for the ID registers. This
will allow us to create a common view over all CPUs.
This will also allow us to create a common view for userspace and the
kernel if we detect a difference on some CPUs, or to handle errata.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47124
|
|
|
|
|
|
|
|
|
| |
In preperation for supporting non-4 bit field width to identcpu.c,
e.g. for ctr_el0, add the width of the existing ID register fields.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47118
|
|
|
|
|
|
|
|
|
| |
The VPIPT and AIVIVT cache types are reserved from Armv8.0. Remove
them as nothing will report these values.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47117
|
|
|
|
|
|
|
|
|
|
|
| |
The VPIPT I-cache policy has been removed from the architecture.
Remove support for it from pmap. The vmm side was not imported so
calls to pmap_invalidate_vpipt_icache would cause the kernel to branch
to a NULL pointer.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D47116
|
|
|
|
|
|
|
| |
While here make it return void, we don't set any useful return value
and nothing checks for it.
Sponsored by: Arm Ltd
|
|
|
|
|
|
|
|
|
| |
Report when SVE is present and allow it to be used by calling
sve_restore_state on an SVE exception from userspace.
Reviewed by: kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43310
|
|
|
|
|
|
|
|
|
| |
As with floating point instructions don't trap SVE instructions to the
hypervisor. This lets us handle then in the kernel.
Reviewed by: imp (earlier version)
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43303
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add the NT_ARM_SVE note type and use it to access the SVE registers
from ptrace. This allows userspace to modify the full SVE register
values.
Try to follow the Linux semantics to allow debuggers to use this with
minimal changes.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43309
|
|
|
|
|
|
|
|
|
| |
To allow for user space to read the SVE vector length add a sysarch
handler to return the value to userspace.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43308
|
|
|
|
|
|
|
|
|
|
|
|
| |
Use the new extended register support in the arm64 signal frame to
handle the SVE registers.
As these registers alias the VFP registers we use the floating point
register values to restore the lower bits of the SVE registers. This
is to support software that doesn't understand SVE to continue working.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43307
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add initial kernel support for SVE. This detects if SVE is present on
all CPUs, and if so allows for the use of SVE in the future.
As the SVE registers are a superset of the VFP registers we don't need
to restore the VFP registers when SVE is enabled.
Ths interface to enable SVE is provided, but not used until SVE is
supported in signals and with ptrace.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43306
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
arm64 supports FIQs, fast interrupt requests, which are required by
Apple silicon which hardwires the ARM timers to FIQs. This is needed by
the upcoming Apple Interrupt Controller. Based on work by andrew@ and
kevans@ in https://reviews.freebsd.org/D40161.
Signed-off-by: Ayrton Munoz <a.munoz3327@gmail.com>
Co-authored-by: Kyle Evans <kevans@FreeBSD.org>
Co-authored-by: Andrew Turner <andrew@FreeBSD.org>
Reviewed-by: imp,mmel,mhorne
Pull-Request: https://github.com/freebsd/freebsd-src/pull/1363
|
|
|
|
|
|
|
|
| |
Add a new macro that enables all CPTR_EL2 traps. This helps ensure we
trap all extensions we don't support.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46516
|
|
|
|
|
|
|
|
| |
The TTA field moves depending on the HCR_EL2.E2H field. Add a macro to
hold the E2H == 1 case.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46515
|
|
|
|
|
|
|
|
| |
Rename register fields that are only valid when HCR_EL2.E2H == 1. Some
fields move around depending on the value of the E2H field.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46514
|
|
|
|
|
|
| |
Reviewed by: emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46513
|
|
|
|
|
|
|
|
|
| |
Monitor Debug Configuration Register provides EL2 configuration options
for self-hosted debug and the Performance Monitors Extension.
Reviewed by: andrew
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46191
|
|
|
|
|
|
|
|
|
|
| |
When LPA2 is enabled the shareability attribute in the page table are
replaces with output address bits. To support a larger physical address
space make this attribute dynamic so we only set it when appropriate.
Reviewed by: alc, kib
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46394
|
|
|
|
|
|
|
|
|
|
|
|
| |
ATTR_SH(ATTR_SH_IS) will soon be dynamic as the field is moved out of
the page tables in FEAT_LPA2. When this happens ATTR_DEFAULT will just
be ATTR_AF.
Rather than keeping ATTR_DEFAULT with one attribute remove it.
Reviewed by: alc, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46466
|
|
|
|
|
|
|
|
|
|
|
| |
These are only used in one place so expand them there.
While here always set TCR_SH0_IS and TCR_SH1_IS. There is no advantage
to not set them in a UP kernel.
Reviewed by: alc, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46393
|
|
|
|
|
|
|
|
|
| |
This will be used to support FEAT_LPA2 to allow more than 48 bits of
physical address space.
Reviewed by: alc, kib, markj
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46392
|
|
|
|
|
|
|
|
|
|
|
| |
vmm.h is required for VM_MAX_SUFFIXLEN. vmm_snapshot.h is required for
struct vm_snapshot_meta.
This is a prerequisite for including vmm_dev.h in the headers parsed by
libsysdecode.
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D46485
|
|
|
|
|
|
|
|
|
|
| |
The maximum VM name length has changed since the arm64 vmm code was
forked. For now, just sync with the amd64 definitions, since they
permit longer VM names. As arm64/vmm is implemented only in main, I
won't bother dealing with backward compat.
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D46484
|
|
|
|
|
|
|
| |
This leads to an unallocated register. Fix the value to point to
ESR_EL1.
Sponsored by: Arm Ltd
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This file contains the vmm device file implementation. Most of this
code is not machine-dependent and so shouldn't be duplicated this way.
Move most of it into a generic dev/vmm/vmm_dev.c. This will make it
easier to introduce a cdev-based interface for VM creation, which in
turn makes it possible to implement support for running bhyve as an
unprivileged user.
Machine-dependent ioctls continue to be handled in machine-dependent
code. To make the split a bit easier to handle, introduce a pair of
tables which define MI and MD ioctls. Each table entry can set flags
which determine which locks need to be held in order to execute the
handler. vmmdev_ioctl() now looks up the ioctl in one of the tables,
acquires locks and either handles the ioctl directly or calls
vmmdev_machdep_ioctl() to handle it.
No functional change intended. There is a lot of churn in this change
but the underlying logic in the ioctl handlers is the same. For now,
vmm_dev.h is still mostly separate, even though some parts could be
merged in principle. This would involve changing include paths for
userspace, though.
Reviewed by: corvink, jhb
Differential Revision: https://reviews.freebsd.org/D46431
|
|
|
|
|
|
|
|
|
| |
This was added to not use the physical timer when E2H was set. As we
now use the correct timer in this case we can remove this extra check.
Tested by: kevans
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46086
|
|
|
|
|
|
|
|
|
| |
Teach the virtual timer about the cnthctl_el2 field layout under VHE.
As with non-VHE we need to trap the physical timer and not trap the
virtual timer.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46074
|
|
|
|
|
|
| |
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46071
|
|
|
|
|
|
|
|
|
|
| |
When targeting Armv8.1 we can assume FEAT_LSE is available and can use
the atomic instructions this provides without needing to check for
support first.
Reviewed by: imp, markj, emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D46159
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As of 9e6544dd6e02c46b805d11ab925c4f3b18ad7a4b contigfree(9) is no longer
needed and should not be used anymore. We leave a wrapper for 3rd party
code in at least 15.x but remove (almost) all other cases from the tree.
This leaves one use of contigfree(9) untouched; that was the original
trigger for 9e6544dd6e02 and is handled in D45813 (to be committed
seperately later).
Sponsored by: The FreeBSD Foundation
Reviewed by: markj, kib
Tested by: pho (10h stress test run)
Differential Revision: https://reviews.freebsd.org/D46099
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Before entering the kernel exception level ensure sctlr_el2 and
sctlr_el1 are in a known state. The EOS flag needs to be set to ensure
an eret instruction is a context synchronization event.
Set spcr_el1 when entering the kernel from EL1 and use an eret
instruction to return to the caller. This ensures the CPU pstate is
consistent with the value in spcr_el1 as it is the only way to set it
directly.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45528
|
|
|
|
|
|
|
|
|
| |
When entering the kernel with the E2H field set the layout of the
cnthctl_el2 register changes. Use the correct field locations to enable
access to the counter and timer registers from EL1.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45529
|
|
|
|
|
|
|
|
|
|
|
| |
When adding VHE support we may need to access these registers using
the _EL12 or _EL1 name depending on if VHE is enabled or not.
Add the registers we will likely need to access from VHE so these can
be updated in the guest switcher code.
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45525
|
|
|
|
|
|
|
|
|
|
|
| |
All architectures enable NEW_PCIB in DEFAULTS (arm being the most recent
to do so in 121be555997b (arm: Set NEW_PCIB in DEFAULTS rather than a
subset of kernel configs")), so it's time we removed the legacy code
that no longer sees much testing and has a significant maintenance
burden.
Reviewed by: jhb, andrew, emaste
Differential Revision: https://reviews.freebsd.org/D32954
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size,
which is 64KB when the base page size is 4KB and 2MB when the base page
size is 16KB.
Add support for L3C pages to shm_create_largepage().
Add support for creating L3C page mappings to pmap_enter(psind=1).
Add support for reporting L3C page mappings to mincore(2) and
procstat(8).
Update vm_fault_soft_fast() and vm_fault_populate() to handle multiple
superpage sizes.
Declare arm64 as supporting two superpage reservation sizes, and
simulate two superpage reservation sizes, updating the vm_page's psind
field to reflect the correct page size from pagesizes[]. (The next
patch in this series will replace this simulation. This patch is
already big enough.)
Co-authored-by: Eliot Solomon <ehs3@rice.edu>
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D45766
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The idea here is to avoid a memory access and conditional branch per
probe site. Instead, the probe is represented by an "unreachable"
unconditional function call. asm goto is used to store the address of
the probe site (represented by a no-op sled) and the address of the
function call into a tracepoint record. Each SDT probe carries a list
of tracepoints.
When the probe is enabled, the no-op sled corresponding to each
tracepoint is overwritten with a jmp to the corresponding label. The
implementation uses smp_rendezvous() to park all other CPUs while the
instruction is being overwritten, as this can't be done atomically in
general. The compiler moves argument marshalling code and the
sdt_probe() function call out-of-line, i.e., to the end of the function.
Per gallatin@ in D43504, this approach has less overhead when probes are
disabled. To make the implementation a bit simpler, I removed support
for probes with 7 arguments; nothing makes use of this except a
regression test case. It could be re-added later if need be.
The approach taken in this patch enables some more improvements:
1. We can now automatically fill out the "function" field of SDT probe
names. The SDT macros let the programmer specify the function and
module names, but this is really a bug and shouldn't have been
allowed. The intent was to be able to have the same probe in
multiple functions and to let the user restrict which probes actually
get enabled by specifying a function name or glob.
2. We can avoid branching on SDT_PROBES_ENABLED() by adding the ability
to include blocks of code in the out-of-line path. For example:
if (SDT_PROBES_ENABLED()) {
int reason = CLD_EXITED;
if (WCOREDUMP(signo))
reason = CLD_DUMPED;
else if (WIFSIGNALED(signo))
reason = CLD_KILLED;
SDT_PROBE1(proc, , , exit, reason);
}
could be written
SDT_PROBE1_EXT(proc, , , exit, reason,
int reason;
reason = CLD_EXITED;
if (WCOREDUMP(signo))
reason = CLD_DUMPED;
else if (WIFSIGNALED(signo))
reason = CLD_KILLED;
);
In the future I would like to use this mechanism more generally, e.g.,
to remove branches and marshalling code used by hwpmc, and generally to
make it easier to add new tracepoint consumers without having to add
more conditional branches to hot code paths.
Reviewed by: Domagoj Stolfa, avg
MFC after: 2 months
Differential Revision: https://reviews.freebsd.org/D44483
|
|
|
|
|
| |
Fixes: b16b4c22d2d1 ("vm_page: Implement lazy page initialization")
Reported by: Steffen Nurpmeso <steffen@sdaoden.eu>
|
|
|
|
|
|
|
|
|
| |
Most of vmm.h is machine-independent. Simplify merging amd64 and arm64
vmm code by removing this machine-dependent routine from arm64's vmm.h.
No functional change intended.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D45557
|
|
|
|
|
|
|
| |
No functional change intended.
Reviewed by: andrew
Differential Revision: https://reviews.freebsd.org/D45556
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
FreeBSD's boot times have decreased to the point where vm_page array
initialization represents a significant fraction of the total boot time.
For example, when booting FreeBSD in Firecracker (a VMM designed to
support lightweight VMs) with 128MB and 1GB of RAM, vm_page
initialization consumes 9% (3ms) and 37% (21.5ms) of the kernel boot
time, respectively. This is generally relevant in cloud environments,
where one wants to be able to spin up VMs as quickly as possible.
This patch implements lazy initialization of (most) page structures,
following a suggestion from cperciva@. The idea is to introduce a new
free pool, VM_FREEPOOL_LAZYINIT, into which all vm_page structures are
initially placed. For this to work, we need only initialize the first
free page of each chunk placed into the buddy allocator. Then, early
page allocations draw from the lazy init pool and initialize vm_page
chunks (up to 16MB, 4096 pages) on demand. Once APs are started, an
idle-priority thread drains the lazy init pool in the background to
avoid introducing extra latency in the allocator. With this scheme,
almost all of the initialization work is moved out of the critical path.
A couple of vm_phys operations require the pool to be drained before
they can run: vm_phys_find_range() and vm_phys_unfree_page(). However,
these are rare operations. I believe that
vm_phys_find_freelist_contig() does not require any special treatment,
as it only ever accesses the first page in a power-of-2-sized free page
chunk, which is always initialized.
For now the new pool is only used on amd64 and arm64, since that's where
I can easily test and those platforms would get the most benefit.
Reviewed by: alc, kib
Differential Revision: https://reviews.freebsd.org/D40403
|
|
|
|
|
|
|
|
|
| |
This will be used when we add SVE support to reduce the registers
needed to be saved on context switch.
Reviewed by: imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D43305
|
|
|
|
|
|
|
|
|
| |
Adjust the mair_el1 macro indentation to be consistent with the
surrounding macros.
Reviewed by: emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D45524
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This will be used to implement parts of bhyve's gdb stub.
Three VM capabilities are added, similar to amd64 without monitor mode.
Two cause breakpoint and single-step exceptions to be raised to EL2 and
then down to bhyve. One lets the gdb stub mask hardware interrupts
while single-stepping, since otherwise the guest will handle a timer
interrupt before executing the target instruction and thus fail
to make progress.
Reviewed by: bnovkov, andrew
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D44739
|
|
|
|
|
|
|
|
|
|
| |
This commit introduces the MINIDUMP_STARTUP_PAGE_TRACKING symbol and
uses it to simplify several instances of a complex preprocessor conditional
for adding pages allocated when bootstraping the kernel to minidumps.
Reviewed by: markj, mhorne
Approved by: markj (mentor)
Differential Revision: https://reviews.freebsd.org/D45085
|