aboutsummaryrefslogtreecommitdiff
path: root/sys/arm64/include
Commit message (Collapse)AuthorAgeFilesLines
* arm64: Enable handling EFI runtime service faultsAndrew Turner9 hours1-1/+1
| | | | | | | Now we can handle EFI runtime service faults on arm64 do so. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46818
* arm64: Implement efi_rt_arch_callAndrew Turner9 hours1-1/+5
| | | | | | | | | | This is a function that calls into the EFI Runtime Services, but can handle a fault. To support this add a handler in assembly that can restore the kernel state on an exception and return a failure to the caller. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46817
* arm64: Don't enable interrupts when in a spinlockAndrew Turner9 hours1-0/+1
| | | | | | | | | | | | | | | | | | When we receive an exception while in a spinlock we shouldn't enable interrupts. When entering a spinlock we disable interrupts so enabling them here could cause surprising results. The three cases that could cause this are: 1. A break-before-make sequence 2. Accessing possibly unmapped code with a fault handler 3. Buggy code 1 and 2 are supported later in the data abort handler, and 3 should be fixed when found. Reviewed by: mmel, kib, markj Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46816
* sys/intr.h: formally depend on machine/intr.hKyle Evans15 hours1-2/+0
| | | | | | | | | | | | | | sys/intr.h originally started life as an extract of arm's intr.h, and this include was dropped in its place. Changes in flight want to add some MD definitions that we'll use in the more MI parts of INTRNG. Let's formally reverse the dependency now since this is way more common in general. All of the includes switched in this change that I spot-checked were in-fact wanting declarations historically included in sys/intr.h anyways. Reviewed by: andrew, imp, jrtc27, mhorne, mmel, olce Differential Revision: https://reviews.freebsd.org/D47002
* intrng: change multi-interrupt root support type to enumElliott Mitchell15 hours1-3/+7
| | | | | | | | | | | | | | | | | uint32_t is handy for directly interfacing with assembly-language. For the C portion, enum is much handier. In particular there is no need to count the number of roots by hand. This also works better for being able to build kernels with varying numbers of roots. Switch to INTR_ROOT_COUNT as this better matches the purpose of the value. Switch to root_type, rather than rootnum for similar reasons. Remove the default from the core. Better to require the architectures to declare the type since they will routinely deviate and a default chosen now will likely be suboptimal. Leave intr_irq_handler() taking a register type as that better matches for interfacing with assembly-language.
* arm{,64}: use genassym for INTR_ROOT_* valuesElliott Mitchell15 hours1-3/+0
| | | | | | | Rather than keeping the entire machine interrupt header assembly-safe, switch to use of the existing genassym.c program to generate them. This will be handier in the long-term, most of the header never needs to be exposed to assembly-language.
* arm64: Decode CTR_EL0 via a tableAndrew Turner3 days1-0/+11
| | | | | | | | | | | | Use the same method to decode CTR_EL0 as for the ID registers. This will allow us to create a common view over all CPUs. This will also allow us to create a common view for userspace and the kernel if we detect a difference on some CPUs, or to handle errata. Reviewed by: imp Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D47124
* arm64: Add ID register field width valuesAndrew Turner3 days1-0/+157
| | | | | | | | | In preperation for supporting non-4 bit field width to identcpu.c, e.g. for ctr_el0, add the width of the existing ID register fields. Reviewed by: imp Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D47118
* arm64: Remove old I-Cache typesAndrew Turner3 days1-2/+0
| | | | | | | | | The VPIPT and AIVIVT cache types are reserved from Armv8.0. Remove them as nothing will report these values. Reviewed by: imp Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D47117
* arm64: Remove VPIPT I-cache support from pmapAndrew Turner3 days1-1/+0
| | | | | | | | | | | The VPIPT I-cache policy has been removed from the architecture. Remove support for it from pmap. The vmm side was not imported so calls to pmap_invalidate_vpipt_icache would cause the kernel to branch to a NULL pointer. Reviewed by: imp Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D47116
* arm64: Stop trashing x28 in savectxAndrew Turner9 days1-1/+1
| | | | | | | While here make it return void, we don't set any useful return value and nothing checks for it. Sponsored by: Arm Ltd
* arm64: Enable SVE in userspaceAndrew Turner2024-09-301-0/+1
| | | | | | | | | Report when SVE is present and allow it to be used by calling sve_restore_state on an SVE exception from userspace. Reviewed by: kib Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43310
* arm64: Don't trap SVE to EL2Andrew Turner2024-09-301-1/+2
| | | | | | | | | As with floating point instructions don't trap SVE instructions to the hypervisor. This lets us handle then in the kernel. Reviewed by: imp (earlier version) Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43303
* arm64: Support SVE in ptrace and core dumpsAndrew Turner2024-09-301-0/+13
| | | | | | | | | | | | Add the NT_ARM_SVE note type and use it to access the SVE registers from ptrace. This allows userspace to modify the full SVE register values. Try to follow the Linux semantics to allow debuggers to use this with minimal changes. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43309
* arm64: Add an SVE sysarchAndrew Turner2024-09-301-0/+3
| | | | | | | | | To allow for user space to read the SVE vector length add a sysarch handler to return the value to userspace. Reviewed by: imp Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43308
* arm64: Add the SVE registers to the signal frameAndrew Turner2024-09-301-0/+8
| | | | | | | | | | | | Use the new extended register support in the arm64 signal frame to handle the SVE registers. As these registers alias the VFP registers we use the floating point register values to restore the lower bits of the SVE registers. This is to support software that doesn't understand SVE to continue working. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43307
* arm64: Initial SVE supportAndrew Turner2024-09-303-3/+18
| | | | | | | | | | | | | | Add initial kernel support for SVE. This detects if SVE is present on all CPUs, and if so allows for the use of SVE in the future. As the SVE registers are a superset of the VFP registers we don't need to restore the VFP registers when SVE is enabled. Ths interface to enable SVE is provided, but not used until SVE is supported in signals and with ptrace. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43306
* arm64: Add support for FIQsAyrton Munoz2024-09-222-2/+8
| | | | | | | | | | | | | arm64 supports FIQs, fast interrupt requests, which are required by Apple silicon which hardwires the ARM timers to FIQs. This is needed by the upcoming Apple Interrupt Controller. Based on work by andrew@ and kevans@ in https://reviews.freebsd.org/D40161. Signed-off-by: Ayrton Munoz <a.munoz3327@gmail.com> Co-authored-by: Kyle Evans <kevans@FreeBSD.org> Co-authored-by: Andrew Turner <andrew@FreeBSD.org> Reviewed-by: imp,mmel,mhorne Pull-Request: https://github.com/freebsd/freebsd-src/pull/1363
* arm64: Add CPTR_TRAP_ALL and use it in vmmAndrew Turner2024-09-111-0/+2
| | | | | | | | Add a new macro that enables all CPTR_EL2 traps. This helps ensure we trap all extensions we don't support. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46516
* arm64: Add CPTR_E2H_TTAAndrew Turner2024-09-111-1/+2
| | | | | | | | The TTA field moves depending on the HCR_EL2.E2H field. Add a macro to hold the E2H == 1 case. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46515
* arm64: Add E2H to CPTR_EL2 register valuesAndrew Turner2024-09-111-1/+1
| | | | | | | | Rename register fields that are only valid when HCR_EL2.E2H == 1. Some fields move around depending on the value of the E2H field. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46514
* arm64: Adjust the indentation of CPTR_EL2 valuesAndrew Turner2024-09-111-6/+6
| | | | | | Reviewed by: emaste Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46513
* arm64: add additional MDCR_EL2 fieldsZachary Leaf2024-09-091-0/+30
| | | | | | | | | Monitor Debug Configuration Register provides EL2 configuration options for self-hosted debug and the Performance Monitors Extension. Reviewed by: andrew Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46191
* arm64: Make shareability attributes dynamicAndrew Turner2024-09-052-0/+4
| | | | | | | | | | When LPA2 is enabled the shareability attribute in the page table are replaces with output address bits. To support a larger physical address space make this attribute dynamic so we only set it when appropriate. Reviewed by: alc, kib Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46394
* arm64: Remove ATTR_DEFAULT from pte.hAndrew Turner2024-09-051-2/+0
| | | | | | | | | | | | ATTR_SH(ATTR_SH_IS) will soon be dynamic as the field is moved out of the page tables in FEAT_LPA2. When this happens ATTR_DEFAULT will just be ATTR_AF. Rather than keeping ATTR_DEFAULT with one attribute remove it. Reviewed by: alc, kib, markj Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46466
* arm64: Remove TCR_CACHE_ATTRS and TCR_SMP_ATTRSAndrew Turner2024-09-051-8/+0
| | | | | | | | | | | These are only used in one place so expand them there. While here always set TCR_SH0_IS and TCR_SH1_IS. There is no advantage to not set them in a UP kernel. Reviewed by: alc, kib, markj Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46393
* arm64: Add the tcr_el2 ds fieldAndrew Turner2024-09-051-0/+2
| | | | | | | | | This will be used to support FEAT_LPA2 to allow more than 48 bits of physical address space. Reviewed by: alc, kib, markj Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46392
* vmm: Make vmm_dev.h more self-containedMark Johnston2024-09-011-0/+2
| | | | | | | | | | | vmm.h is required for VM_MAX_SUFFIXLEN. vmm_snapshot.h is required for struct vm_snapshot_meta. This is a prerequisite for including vmm_dev.h in the headers parsed by libsysdecode. Reviewed by: jhb Differential Revision: https://reviews.freebsd.org/D46485
* arm64: Fix the VM_MAX_NAMELEN definitionMark Johnston2024-09-011-5/+21
| | | | | | | | | | The maximum VM name length has changed since the arm64 vmm code was forked. For now, just sync with the amd64 definitions, since they permit longer VM names. As arm64/vmm is implemented only in main, I won't bother dealing with backward compat. Reviewed by: jhb Differential Revision: https://reviews.freebsd.org/D46484
* arm64: Fix the ESR_EL1_op2 valueAndrew Turner2024-08-291-1/+1
| | | | | | | This leads to an unallocated register. Fix the value to point to ESR_EL1. Sponsored by: Arm Ltd
* vmm: Merge vmm_dev.cMark Johnston2024-08-261-5/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | This file contains the vmm device file implementation. Most of this code is not machine-dependent and so shouldn't be duplicated this way. Move most of it into a generic dev/vmm/vmm_dev.c. This will make it easier to introduce a cdev-based interface for VM creation, which in turn makes it possible to implement support for running bhyve as an unprivileged user. Machine-dependent ioctls continue to be handled in machine-dependent code. To make the split a bit easier to handle, introduce a pair of tables which define MI and MD ioctls. Each table entry can set flags which determine which locks need to be held in order to execute the handler. vmmdev_ioctl() now looks up the ioctl in one of the tables, acquires locks and either handles the ioctl directly or calls vmmdev_machdep_ioctl() to handle it. No functional change intended. There is a lot of churn in this change but the underlying logic in the ioctl handlers is the same. For now, vmm_dev.h is still mostly separate, even though some parts could be merged in principle. This would involve changing include paths for userspace, though. Reviewed by: corvink, jhb Differential Revision: https://reviews.freebsd.org/D46431
* arm64: Remove the E2H check from has_hypAndrew Turner2024-08-211-1/+0
| | | | | | | | | This was added to not use the physical timer when E2H was set. As we now use the correct timer in this case we can remove this extra check. Tested by: kevans Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46086
* arm64/vmm: Teach the vtimer about VHEAndrew Turner2024-08-201-0/+4
| | | | | | | | | Teach the virtual timer about the cnthctl_el2 field layout under VHE. As with non-VHE we need to trap the physical timer and not trap the virtual timer. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46074
* arm64: Add counter timer registers to armreg.hAndrew Turner2024-08-201-0/+48
| | | | | | Reviewed by: imp Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46071
* arm64: Expand the use of Armv8.1-A atomicsAndrew Turner2024-08-191-2/+3
| | | | | | | | | | When targeting Armv8.1 we can assume FEAT_LSE is available and can use the atomic instructions this provides without needing to check for support first. Reviewed by: imp, markj, emaste Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D46159
* Deprecate contigfree(9) in favour of free(9)Bjoern A. Zeeb2024-07-261-1/+1
| | | | | | | | | | | | | | | As of 9e6544dd6e02c46b805d11ab925c4f3b18ad7a4b contigfree(9) is no longer needed and should not be used anymore. We leave a wrapper for 3rd party code in at least 15.x but remove (almost) all other cases from the tree. This leaves one use of contigfree(9) untouched; that was the original trigger for 9e6544dd6e02 and is handled in D45813 (to be committed seperately later). Sponsored by: The FreeBSD Foundation Reviewed by: markj, kib Tested by: pho (10h stress test run) Differential Revision: https://reviews.freebsd.org/D46099
* arm64: Ensure sctlr and pstate are in known statesAndrew Turner2024-07-231-0/+4
| | | | | | | | | | | | | | Before entering the kernel exception level ensure sctlr_el2 and sctlr_el1 are in a known state. The EOS flag needs to be set to ensure an eret instruction is a context synchronization event. Set spcr_el1 when entering the kernel from EL1 and use an eret instruction to return to the caller. This ensures the CPU pstate is consistent with the value in spcr_el1 as it is the only way to set it directly. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45528
* arm64: Support counter access with E2HAndrew Turner2024-07-231-2/+7
| | | | | | | | | When entering the kernel with the E2H field set the layout of the cnthctl_el2 register changes. Use the correct field locations to enable access to the counter and timer registers from EL1. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45529
* arm64: Add EL1 and EL12 register alt namesAndrew Turner2024-07-231-0/+222
| | | | | | | | | | | When adding VHE support we may need to access these registers using the _EL12 or _EL1 name depending on if VHE is enabled or not. Add the registers we will likely need to access from VHE so these can be updated in the guest switcher code. Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45525
* Retire non-NEW_PCIB code and remove config optionJessica Clarke2024-07-181-2/+0
| | | | | | | | | | | All architectures enable NEW_PCIB in DEFAULTS (arm being the most recent to do so in 121be555997b (arm: Set NEW_PCIB in DEFAULTS rather than a subset of kernel configs")), so it's time we removed the legacy code that no longer sees much testing and has a significant maintenance burden. Reviewed by: jhb, andrew, emaste Differential Revision: https://reviews.freebsd.org/D32954
* arm64: Support the L3 ATTR_CONTIGUOUS page size in pagesizes[]Alan Cox2024-07-132-9/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | Update pagesizes[] to include the L3 ATTR_CONTIGUOUS (L3C) page size, which is 64KB when the base page size is 4KB and 2MB when the base page size is 16KB. Add support for L3C pages to shm_create_largepage(). Add support for creating L3C page mappings to pmap_enter(psind=1). Add support for reporting L3C page mappings to mincore(2) and procstat(8). Update vm_fault_soft_fast() and vm_fault_populate() to handle multiple superpage sizes. Declare arm64 as supporting two superpage reservation sizes, and simulate two superpage reservation sizes, updating the vm_page's psind field to reflect the correct page size from pagesizes[]. (The next patch in this series will replace this simulation. This patch is already big enough.) Co-authored-by: Eliot Solomon <ehs3@rice.edu> Reviewed by: kib Differential Revision: https://reviews.freebsd.org/D45766
* sdt: Implement SDT probes using hot-patchingMark Johnston2024-06-191-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The idea here is to avoid a memory access and conditional branch per probe site. Instead, the probe is represented by an "unreachable" unconditional function call. asm goto is used to store the address of the probe site (represented by a no-op sled) and the address of the function call into a tracepoint record. Each SDT probe carries a list of tracepoints. When the probe is enabled, the no-op sled corresponding to each tracepoint is overwritten with a jmp to the corresponding label. The implementation uses smp_rendezvous() to park all other CPUs while the instruction is being overwritten, as this can't be done atomically in general. The compiler moves argument marshalling code and the sdt_probe() function call out-of-line, i.e., to the end of the function. Per gallatin@ in D43504, this approach has less overhead when probes are disabled. To make the implementation a bit simpler, I removed support for probes with 7 arguments; nothing makes use of this except a regression test case. It could be re-added later if need be. The approach taken in this patch enables some more improvements: 1. We can now automatically fill out the "function" field of SDT probe names. The SDT macros let the programmer specify the function and module names, but this is really a bug and shouldn't have been allowed. The intent was to be able to have the same probe in multiple functions and to let the user restrict which probes actually get enabled by specifying a function name or glob. 2. We can avoid branching on SDT_PROBES_ENABLED() by adding the ability to include blocks of code in the out-of-line path. For example: if (SDT_PROBES_ENABLED()) { int reason = CLD_EXITED; if (WCOREDUMP(signo)) reason = CLD_DUMPED; else if (WIFSIGNALED(signo)) reason = CLD_KILLED; SDT_PROBE1(proc, , , exit, reason); } could be written SDT_PROBE1_EXT(proc, , , exit, reason, int reason; reason = CLD_EXITED; if (WCOREDUMP(signo)) reason = CLD_DUMPED; else if (WIFSIGNALED(signo)) reason = CLD_KILLED; ); In the future I would like to use this mechanism more generally, e.g., to remove branches and marshalling code used by hwpmc, and generally to make it easier to add new tracepoint consumers without having to add more conditional branches to hot code paths. Reviewed by: Domagoj Stolfa, avg MFC after: 2 months Differential Revision: https://reviews.freebsd.org/D44483
* vm_phys: Fix a typoMark Johnston2024-06-161-1/+1
| | | | | Fixes: b16b4c22d2d1 ("vm_page: Implement lazy page initialization") Reported by: Steffen Nurpmeso <steffen@sdaoden.eu>
* arm64/vmm: Avoid unnecessary indirection in vmmops_modinit()Mark Johnston2024-06-141-7/+0
| | | | | | | | | Most of vmm.h is machine-independent. Simplify merging amd64 and arm64 vmm code by removing this machine-dependent routine from arm64's vmm.h. No functional change intended. Reviewed by: andrew Differential Revision: https://reviews.freebsd.org/D45557
* arm64/vmm: Implement vm_disable_vcpu_creation()Mark Johnston2024-06-141-0/+1
| | | | | | | No functional change intended. Reviewed by: andrew Differential Revision: https://reviews.freebsd.org/D45556
* vm_page: Implement lazy page initializationMark Johnston2024-06-141-7/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | FreeBSD's boot times have decreased to the point where vm_page array initialization represents a significant fraction of the total boot time. For example, when booting FreeBSD in Firecracker (a VMM designed to support lightweight VMs) with 128MB and 1GB of RAM, vm_page initialization consumes 9% (3ms) and 37% (21.5ms) of the kernel boot time, respectively. This is generally relevant in cloud environments, where one wants to be able to spin up VMs as quickly as possible. This patch implements lazy initialization of (most) page structures, following a suggestion from cperciva@. The idea is to introduce a new free pool, VM_FREEPOOL_LAZYINIT, into which all vm_page structures are initially placed. For this to work, we need only initialize the first free page of each chunk placed into the buddy allocator. Then, early page allocations draw from the lazy init pool and initialize vm_page chunks (up to 16MB, 4096 pages) on demand. Once APs are started, an idle-priority thread drains the lazy init pool in the background to avoid introducing extra latency in the allocator. With this scheme, almost all of the initialization work is moved out of the critical path. A couple of vm_phys operations require the pool to be drained before they can run: vm_phys_find_range() and vm_phys_unfree_page(). However, these are rare operations. I believe that vm_phys_find_freelist_contig() does not require any special treatment, as it only ever accesses the first page in a power-of-2-sized free page chunk, which is always initialized. For now the new pool is only used on amd64 and arm64, since that's where I can easily test and those platforms would get the most benefit. Reviewed by: alc, kib Differential Revision: https://reviews.freebsd.org/D40403
* arm64: Create a version of vfp_save_state for cpu_switchAndrew Turner2024-06-121-0/+1
| | | | | | | | | This will be used when we add SVE support to reduce the registers needed to be saved on context switch. Reviewed by: imp Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D43305
* arm64: Fix indentation to be consistentAndrew Turner2024-06-101-7/+7
| | | | | | | | | Adjust the mair_el1 macro indentation to be consistent with the surrounding macros. Reviewed by: emaste Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D45524
* arm64/vmm: Add breakpoint and single-stepping supportMark Johnston2024-06-041-1/+5
| | | | | | | | | | | | | | | This will be used to implement parts of bhyve's gdb stub. Three VM capabilities are added, similar to amd64 without monitor mode. Two cause breakpoint and single-step exceptions to be raised to EL2 and then down to bhyve. One lets the gdb stub mask hardware interrupts while single-stepping, since otherwise the guest will handle a timer interrupt before executing the target instruction and thus fail to make progress. Reviewed by: bnovkov, andrew Sponsored by: Innovate UK Differential Revision: https://reviews.freebsd.org/D44739
* vm: Simplify startup page dumping conditionalBojan Novković2024-05-251-0/+1
| | | | | | | | | | This commit introduces the MINIDUMP_STARTUP_PAGE_TRACKING symbol and uses it to simplify several instances of a complex preprocessor conditional for adding pages allocated when bootstraping the kernel to minidumps. Reviewed by: markj, mhorne Approved by: markj (mentor) Differential Revision: https://reviews.freebsd.org/D45085
generated by cgit v1.2.3 (git 2.25.1) at 2024-10-24 19:16:06 +0000