Skip to content

L
linux
Commit 6803bd79 authored by Linus Torvalds's avatar Linus Torvalds
Browse files
Options

Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm 

Pull kvm updates from Paolo Bonzini:
 "ARM:

   - Generalized infrastructure for 'writable' ID registers, effectively
     allowing userspace to opt-out of certain vCPU features for its
     guest

   - Optimization for vSGI injection, opportunistically compressing
     MPIDR to vCPU mapping into a table

   - Improvements to KVM's PMU emulation, allowing userspace to select
     the number of PMCs available to a VM

   - Guest support for memory operation instructions (FEAT_MOPS)

   - Cleanups to handling feature flags in KVM_ARM_VCPU_INIT, squashing
     bugs and getting rid of useless code

   - Changes to the way the SMCCC filter is constructed, avoiding wasted
     memory allocations when not in use

   - Load the stage-2 MMU context at vcpu_load() for VHE systems,
     reducing the overhead of errata mitigations

   - Miscellaneous kernel and selftest fixes

  LoongArch:

   - New architecture for kvm.

     The hardware uses the same model as x86, s390 and RISC-V, where
     guest/host mode is orthogonal to supervisor/user mode. The
     virtualization extensions are very similar to MIPS, therefore the
     code also has some similarities but it's been cleaned up to avoid
     some of the historical bogosities that are found in arch/mips. The
     kernel emulates MMU, timer and CSR accesses, while interrupt
     controllers are only emulated in userspace, at least for now.

  RISC-V:

   - Support for the Smstateen and Zicond extensions

   - Support for virtualizing senvcfg

   - Support for virtualized SBI debug console (DBCN)

  S390:

   - Nested page table management can be monitored through tracepoints
     and statistics

  x86:

   - Fix incorrect handling of VMX posted interrupt descriptor in
     KVM_SET_LAPIC, which could result in a dropped timer IRQ

   - Avoid WARN on systems with Intel IPI virtualization

   - Add CONFIG_KVM_MAX_NR_VCPUS, to allow supporting up to 4096 vCPUs
     without forcing more common use cases to eat the extra memory
     overhead.

   - Add virtualization support for AMD SRSO mitigation (IBPB_BRTYPE and
     SBPB, aka Selective Branch Predictor Barrier).

   - Fix a bug where restoring a vCPU snapshot that was taken within 1
     second of creating the original vCPU would cause KVM to try to
     synchronize the vCPU's TSC and thus clobber the correct TSC being
     set by userspace.

   - Compute guest wall clock using a single TSC read to avoid
     generating an inaccurate time, e.g. if the vCPU is preempted
     between multiple TSC reads.

   - "Virtualize" HWCR.TscFreqSel to make Linux guests happy, which
     complain about a "Firmware Bug" if the bit isn't set for select
     F/M/S combos. Likewise "virtualize" (ignore) MSR_AMD64_TW_CFG to
     appease Windows Server 2022.

   - Don't apply side effects to Hyper-V's synthetic timer on writes
     from userspace to fix an issue where the auto-enable behavior can
     trigger spurious interrupts, i.e. do auto-enabling only for guest
     writes.

   - Remove an unnecessary kick of all vCPUs when synchronizing the
     dirty log without PML enabled.

   - Advertise "support" for non-serializing FS/GS base MSR writes as
     appropriate.

   - Harden the fast page fault path to guard against encountering an
     invalid root when walking SPTEs.

   - Omit "struct kvm_vcpu_xen" entirely when CONFIG_KVM_XEN=n.

   - Use the fast path directly from the timer callback when delivering
     Xen timer events, instead of waiting for the next iteration of the
     run loop. This was not done so far because previously proposed code
     had races, but now care is taken to stop the hrtimer at critical
     points such as restarting the timer or saving the timer information
     for userspace.

   - Follow the lead of upstream Xen and ignore the VCPU_SSHOTTMR_future
     flag.

   - Optimize injection of PMU interrupts that are simultaneous with
     NMIs.

   - Usual handful of fixes for typos and other warts.

  x86 - MTRR/PAT fixes and optimizations:

   - Clean up code that deals with honoring guest MTRRs when the VM has
     non-coherent DMA and host MTRRs are ignored, i.e. EPT is enabled.

   - Zap EPT entries when non-coherent DMA assignment stops/start to
     prevent using stale entries with the wrong memtype.

   - Don't ignore guest PAT for CR0.CD=1 && KVM_X86_QUIRK_CD_NW_CLEARED=y

     This was done as a workaround for virtual machine BIOSes that did
     not bother to clear CR0.CD (because ancient KVM/QEMU did not bother
     to set it, in turn), and there's zero reason to extend the quirk to
     also ignore guest PAT.

  x86 - SEV fixes:

   - Report KVM_EXIT_SHUTDOWN instead of EINVAL if KVM intercepts
     SHUTDOWN while running an SEV-ES guest.

   - Clean up the recognition of emulation failures on SEV guests, when
     KVM would like to "skip" the instruction but it had already been
     partially emulated. This makes it possible to drop a hack that
     second guessed the (insufficient) information provided by the
     emulator, and just do the right thing.

  Documentation:

   - Various updates and fixes, mostly for x86

   - MTRR and PAT fixes and optimizations"

* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (164 commits)
  KVM: selftests: Avoid using forced target for generating arm64 headers
  tools headers arm64: Fix references to top srcdir in Makefile
  KVM: arm64: Add tracepoint for MMIO accesses where ISV==0
  KVM: arm64: selftest: Perform ISB before reading PAR_EL1
  KVM: arm64: selftest: Add the missing .guest_prepare()
  KVM: arm64: Always invalidate TLB for stage-2 permission faults
  KVM: x86: Service NMI requests after PMI requests in VM-Enter path
  KVM: arm64: Handle AArch32 SPSR_{irq,abt,und,fiq} as RAZ/WI
  KVM: arm64: Do not let a L1 hypervisor access the *32_EL2 sysregs
  KVM: arm64: Refine _EL2 system register list that require trap reinjection
  arm64: Add missing _EL2 encodings
  arm64: Add missing _EL12 encodings
  KVM: selftests: aarch64: vPMU test for validating user accesses
  KVM: selftests: aarch64: vPMU register test for unimplemented counters
  KVM: selftests: aarch64: vPMU register test for implemented counters
  KVM: selftests: aarch64: Introduce vpmu_counter_access test
  tools: Import arm_pmuv3.h
  KVM: arm64: PMU: Allow userspace to limit PMCR_EL0.N for the guest
  KVM: arm64: Sanitize PM{C,I}NTEN{SET,CLR}, PMOVS{SET,CLR} before first run
  KVM: arm64: Add {get,set}_user for PM{C,I}NTEN{SET,CLR}, PMOVS{SET,CLR}
  ...
parents 5be99114 45b890f7
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 8885 additions and 1503 deletions
Documentation/devicetree/bindings/riscv/extensions.yaml
View file @ 6803bd79
......@@ -128,6 +128,12 @@ properties:
changes to interrupts as frozen at commit ccbddab ("Merge pull
request #42 from riscv/jhauser-2023-RC4") of riscv-aia.
- const: smstateen
description: |
The standard Smstateen extension for controlling access to CSRs
added by other RISC-V extensions in H/S/VS/U/VU modes and as
ratified at commit a28bfae (Ratified (#7)) of riscv-state-enable.
- const: ssaia
description: |
The standard Ssaia supervisor-level extension for the advanced
......@@ -212,6 +218,12 @@ properties:
ratified in the 20191213 version of the unprivileged ISA
specification.
- const: zicond
description:
The standard Zicond extension for conditional arithmetic and
conditional-select/move operations as ratified in commit 95cf1f9
("Add changes requested by Ved during signoff") of riscv-zicond.
- const: zicsr
description: |
The standard Zicsr extension for control and status register
......
Documentation/virt/kvm/api.rst
View file @ 6803bd79
This diff is collapsed.
Documentation/virt/kvm/arm/index.rst
View file @ 6803bd79
......@@ -11,3 +11,4 @@ ARM
hypercalls
pvtime
ptp_kvm
vcpu-features
Documentation/virt/kvm/arm/vcpu-features.rst 0 → 100644
View file @ 6803bd79
.. SPDX-License-Identifier: GPL-2.0
===============================
vCPU feature selection on arm64
===============================
KVM/arm64 provides two mechanisms that allow userspace to configure
the CPU features presented to the guest.
KVM_ARM_VCPU_INIT
=================
The ``KVM_ARM_VCPU_INIT`` ioctl accepts a bitmap of feature flags
(``struct kvm_vcpu_init::features``). Features enabled by this interface are
*opt-in* and may change/extend UAPI. See :ref:`KVM_ARM_VCPU_INIT` for complete
documentation of the features controlled by the ioctl.
Otherwise, all CPU features supported by KVM are described by the architected
ID registers.
The ID Registers
================
The Arm architecture specifies a range of *ID Registers* that describe the set
of architectural features supported by the CPU implementation. KVM initializes
the guest's ID registers to the maximum set of CPU features supported by the
system. The ID register values may be VM-scoped in KVM, meaning that the
values could be shared for all vCPUs in a VM.
KVM allows userspace to *opt-out* of certain CPU features described by the ID
registers by writing values to them via the ``KVM_SET_ONE_REG`` ioctl. The ID
registers are mutable until the VM has started, i.e. userspace has called
``KVM_RUN`` on at least one vCPU in the VM. Userspace can discover what fields
are mutable in the ID registers using the ``KVM_ARM_GET_REG_WRITABLE_MASKS``.
See the :ref:`ioctl documentation <KVM_ARM_GET_REG_WRITABLE_MASKS>` for more
details.
Userspace is allowed to *limit* or *mask* CPU features according to the rules
outlined by the architecture in DDI0487J.a D19.1.3 'Principles of the ID
scheme for fields in ID register'. KVM does not allow ID register values that
exceed the capabilities of the system.
.. warning::
It is **strongly recommended** that userspace modify the ID register values
before accessing the rest of the vCPU's CPU register state. KVM may use the
ID register values to control feature emulation. Interleaving ID register
modification with other system register accesses may lead to unpredictable
behavior.
Documentation/virt/kvm/devices/arm-vgic-v3.rst
View file @ 6803bd79
......@@ -59,6 +59,13 @@ Groups:
It is invalid to mix calls with KVM_VGIC_V3_ADDR_TYPE_REDIST and
KVM_VGIC_V3_ADDR_TYPE_REDIST_REGION attributes.
Note that to obtain reproducible results (the same VCPU being associated
with the same redistributor across a save/restore operation), VCPU creation
order, redistributor region creation order as well as the respective
interleaves of VCPU and region creation MUST be preserved. Any change in
either ordering may result in a different vcpu_id/redistributor association,
resulting in a VM that will fail to run at restore time.
Errors:
======= =============================================================
......
Documentation/virt/kvm/x86/mmu.rst
View file @ 6803bd79
......@@ -202,10 +202,22 @@ Shadow pages contain the following information:
Is 1 if the MMU instance cannot use A/D bits. EPT did not have A/D
bits before Haswell; shadow EPT page tables also cannot use A/D bits
if the L1 hypervisor does not enable them.
role.guest_mode:
Indicates the shadow page is created for a nested guest.
role.passthrough:
The page is not backed by a guest page table, but its first entry
points to one. This is set if NPT uses 5-level page tables (host
CR4.LA57=1) and is shadowing L1's 4-level NPT (L1 CR4.LA57=0).
mmu_valid_gen:
The MMU generation of this page, used to fast zap of all MMU pages within a
VM without blocking vCPUs too long. Specifically, KVM updates the per-VM
valid MMU generation which causes the mismatch of mmu_valid_gen for each mmu
page. This makes all existing MMU pages obsolete. Obsolete pages can't be
used. Therefore, vCPUs must load a new, valid root before re-entering the
guest. The MMU generation is only ever '0' or '1'. Note, the TDP MMU doesn't
use this field as non-root TDP MMU pages are reachable only from their
owning root. Thus it suffices for TDP MMU to use role.invalid in root pages
to invalidate all MMU pages.
gfn:
Either the guest page table containing the translations shadowed by this
page, or the base page frame for linear translations. See role.direct.
......@@ -219,21 +231,30 @@ Shadow pages contain the following information:
at __pa(sp2->spt). sp2 will point back at sp1 through parent_pte.
The spt array forms a DAG structure with the shadow page as a node, and
guest pages as leaves.
gfns:
An array of 512 guest frame numbers, one for each present pte. Used to
perform a reverse map from a pte to a gfn. When role.direct is set, any
element of this array can be calculated from the gfn field when used, in
this case, the array of gfns is not allocated. See role.direct and gfn.
root_count:
A counter keeping track of how many hardware registers (guest cr3 or
pdptrs) are now pointing at the page. While this counter is nonzero, the
page cannot be destroyed. See role.invalid.
shadowed_translation:
An array of 512 shadow translation entries, one for each present pte. Used
to perform a reverse map from a pte to a gfn as well as its access
permission. When role.direct is set, the shadow_translation array is not
allocated. This is because the gfn contained in any element of this array
can be calculated from the gfn field when used. In addition, when
role.direct is set, KVM does not track access permission for each of the
gfn. See role.direct and gfn.
root_count / tdp_mmu_root_count:
root_count is a reference counter for root shadow pages in Shadow MMU.
vCPUs elevate the refcount when getting a shadow page that will be used as
a root page, i.e. page that will be loaded into hardware directly (CR3,
PDPTRs, nCR3 EPTP). Root pages cannot be destroyed while their refcount is
non-zero. See role.invalid. tdp_mmu_root_count is similar but exclusively
used in TDP MMU as an atomic refcount.
parent_ptes:
The reverse mapping for the pte/ptes pointing at this page's spt. If
parent_ptes bit 0 is zero, only one spte points at this page and
parent_ptes points at this single spte, otherwise, there exists multiple
sptes pointing at this page and (parent_ptes & ~0x1) points at a data
structure with a list of parent sptes.
ptep:
The kernel virtual address of the SPTE that points at this shadow page.
Used exclusively by the TDP MMU, this field is a union with parent_ptes.
unsync:
If true, then the translations in this page may not match the guest's
translation. This is equivalent to the state of the tlb when a pte is
......@@ -261,6 +282,10 @@ Shadow pages contain the following information:
since the last time the page table was actually used; if emulation
is triggered too frequently on this page, KVM will unmap the page
to avoid emulation in the future.
tdp_mmu_page:
Is 1 if the shadow page is a TDP MMU page. This variable is used to
bifurcate the control flows for KVM when walking any data structure that
may contain pages from both TDP MMU and shadow MMU.
Reverse map
===========
......
MAINTAINERS
View file @ 6803bd79
......@@ -11604,6 +11604,18 @@ F: include/kvm/arm_*
F: tools/testing/selftests/kvm/*/aarch64/
F: tools/testing/selftests/kvm/aarch64/
KERNEL VIRTUAL MACHINE FOR LOONGARCH (KVM/LoongArch)
M: Tianrui Zhao <zhaotianrui@loongson.cn>
M: Bibo Mao <maobibo@loongson.cn>
M: Huacai Chen <chenhuacai@kernel.org>
L: kvm@vger.kernel.org
L: loongarch@lists.linux.dev
S: Maintained
T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git
F: arch/loongarch/include/asm/kvm*
F: arch/loongarch/include/uapi/asm/kvm*
F: arch/loongarch/kvm/
KERNEL VIRTUAL MACHINE FOR MIPS (KVM/mips)
M: Huacai Chen <chenhuacai@kernel.org>
L: linux-mips@vger.kernel.org
......@@ -11640,6 +11652,7 @@ F: arch/riscv/include/asm/kvm*
F: arch/riscv/include/uapi/asm/kvm*
F: arch/riscv/kvm/
F: tools/testing/selftests/kvm/*/riscv/
F: tools/testing/selftests/kvm/riscv/
KERNEL VIRTUAL MACHINE for s390 (KVM/s390)
M: Christian Borntraeger <borntraeger@linux.ibm.com>
......
arch/arm64/include/asm/kvm_arm.h
View file @ 6803bd79
......@@ -102,7 +102,9 @@
#define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC)
#define HCR_HOST_VHE_FLAGS (HCR_RW | HCR_TGE | HCR_E2H)
#define HCRX_GUEST_FLAGS (HCRX_EL2_SMPME | HCRX_EL2_TCR2En)
#define HCRX_GUEST_FLAGS \
(HCRX_EL2_SMPME | HCRX_EL2_TCR2En | \
(cpus_have_final_cap(ARM64_HAS_MOPS) ? (HCRX_EL2_MSCEn | HCRX_EL2_MCE2) : 0))
#define HCRX_HOST_FLAGS (HCRX_EL2_MSCEn | HCRX_EL2_TCR2En)
/* TCR_EL2 Registers bits */
......
arch/arm64/include/asm/kvm_emulate.h
View file @ 6803bd79
......@@ -54,6 +54,11 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu);
int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2);
int kvm_inject_nested_irq(struct kvm_vcpu *vcpu);
static inline bool vcpu_has_feature(const struct kvm_vcpu *vcpu, int feature)
{
return test_bit(feature, vcpu->kvm->arch.vcpu_features);
}
#if defined(__KVM_VHE_HYPERVISOR__) || defined(__KVM_NVHE_HYPERVISOR__)
static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
{
......@@ -62,7 +67,7 @@ static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
#else
static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
{
return test_bit(KVM_ARM_VCPU_EL1_32BIT, vcpu->arch.features);
return vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
}
#endif
......@@ -465,7 +470,7 @@ static inline bool kvm_is_write_fault(struct kvm_vcpu *vcpu)
static inline unsigned long kvm_vcpu_get_mpidr_aff(struct kvm_vcpu *vcpu)
{
return vcpu_read_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK;
return __vcpu_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK;
}
static inline void kvm_vcpu_set_be(struct kvm_vcpu *vcpu)
......@@ -565,12 +570,6 @@ static __always_inline void kvm_incr_pc(struct kvm_vcpu *vcpu)
vcpu_set_flag((v), e); \
} while (0)
static inline bool vcpu_has_feature(struct kvm_vcpu *vcpu, int feature)
{
return test_bit(feature, vcpu->arch.features);
}
static __always_inline void kvm_write_cptr_el2(u64 val)
{
if (has_vhe() || has_hvhe())
......
arch/arm64/include/asm/kvm_host.h
View file @ 6803bd79
......@@ -78,7 +78,7 @@ extern unsigned int __ro_after_init kvm_sve_max_vl;
int __init kvm_arm_init_sve(void);
u32 __attribute_const__ kvm_target_cpu(void);
int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
void kvm_reset_vcpu(struct kvm_vcpu *vcpu);
void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu);
struct kvm_hyp_memcache {
......@@ -158,6 +158,16 @@ struct kvm_s2_mmu {
phys_addr_t pgd_phys;
struct kvm_pgtable *pgt;
/*
* VTCR value used on the host. For a non-NV guest (or a NV
* guest that runs in a context where its own S2 doesn't
* apply), its T0SZ value reflects that of the IPA size.
*
* For a shadow S2 MMU, T0SZ reflects the PARange exposed to
* the guest.
*/
u64 vtcr;
/* The last vcpu id that ran on each physical CPU */
int __percpu *last_vcpu_ran;
......@@ -202,12 +212,34 @@ struct kvm_protected_vm {
struct kvm_hyp_memcache teardown_mc;
};
struct kvm_mpidr_data {
u64 mpidr_mask;
DECLARE_FLEX_ARRAY(u16, cmpidr_to_idx);
};
static inline u16 kvm_mpidr_index(struct kvm_mpidr_data *data, u64 mpidr)
{
unsigned long mask = data->mpidr_mask;
u64 aff = mpidr & MPIDR_HWID_BITMASK;
int nbits, bit, bit_idx = 0;
u16 index = 0;
/*
* If this looks like RISC-V's BEXT or x86's PEXT
* instructions, it isn't by accident.
*/
nbits = fls(mask);
for_each_set_bit(bit, &mask, nbits) {
index |= (aff & BIT(bit)) >> (bit - bit_idx);
bit_idx++;
}
return index;
}
struct kvm_arch {
struct kvm_s2_mmu mmu;
/* VTCR_EL2 value for this VM */
u64 vtcr;
/* Interrupt controller */
struct vgic_dist vgic;
......@@ -239,15 +271,16 @@ struct kvm_arch {
#define KVM_ARCH_FLAG_VM_COUNTER_OFFSET 5
/* Timer PPIs made immutable */
#define KVM_ARCH_FLAG_TIMER_PPIS_IMMUTABLE 6
/* SMCCC filter initialized for the VM */
#define KVM_ARCH_FLAG_SMCCC_FILTER_CONFIGURED 7
/* Initial ID reg values loaded */
#define KVM_ARCH_FLAG_ID_REGS_INITIALIZED 8
#define KVM_ARCH_FLAG_ID_REGS_INITIALIZED 7
unsigned long flags;
/* VM-wide vCPU feature set */
DECLARE_BITMAP(vcpu_features, KVM_VCPU_MAX_FEATURES);
/* MPIDR to vcpu index mapping, optional */
struct kvm_mpidr_data *mpidr_data;
/*
* VM-wide PMU filter, implemented as a bitmap and big enough for
* up to 2^10 events (ARMv8.0) or 2^16 events (ARMv8.1+).
......@@ -257,6 +290,9 @@ struct kvm_arch {
cpumask_var_t supported_cpus;
/* PMCR_EL0.N value for the guest */
u8 pmcr_n;
/* Hypercall features firmware registers' descriptor */
struct kvm_smccc_features smccc_feat;
struct maple_tree smccc_filter;
......@@ -574,9 +610,6 @@ struct kvm_vcpu_arch {
/* Cache some mmu pages needed inside spinlock regions */
struct kvm_mmu_memory_cache mmu_page_cache;
/* feature flags */
DECLARE_BITMAP(features, KVM_VCPU_MAX_FEATURES);
/* Virtual SError ESR to restore when HCR_EL2.VSE is set */
u64 vsesr_el2;
......@@ -1025,7 +1058,7 @@ int kvm_arm_pvtime_has_attr(struct kvm_vcpu *vcpu,
extern unsigned int __ro_after_init kvm_arm_vmid_bits;
int __init kvm_arm_vmid_alloc_init(void);
void __init kvm_arm_vmid_alloc_free(void);
void kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid);
bool kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid);
void kvm_arm_vmid_clear_active(void);
static inline void kvm_arm_pvtime_vcpu_init(struct kvm_vcpu_arch *vcpu_arch)
......@@ -1078,6 +1111,8 @@ int kvm_vm_ioctl_mte_copy_tags(struct kvm *kvm,
struct kvm_arm_copy_mte_tags *copy_tags);
int kvm_vm_ioctl_set_counter_offset(struct kvm *kvm,
struct kvm_arm_counter_offset *offset);
int kvm_vm_ioctl_get_reg_writable_masks(struct kvm *kvm,
struct reg_mask_range *range);
/* Guest/host FPSIMD coordination helpers */
int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu);
......@@ -1109,8 +1144,8 @@ static inline bool kvm_set_pmuserenr(u64 val)
}
#endif
void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu);
void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu);
void kvm_vcpu_load_vhe(struct kvm_vcpu *vcpu);
void kvm_vcpu_put_vhe(struct kvm_vcpu *vcpu);
int __init kvm_set_ipa_limit(void);
......
arch/arm64/include/asm/kvm_hyp.h
View file @ 6803bd79
......@@ -93,6 +93,8 @@ void __timer_disable_traps(struct kvm_vcpu *vcpu);
void __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt);
void __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt);
#else
void __vcpu_load_switch_sysregs(struct kvm_vcpu *vcpu);
void __vcpu_put_switch_sysregs(struct kvm_vcpu *vcpu);
void sysreg_save_host_state_vhe(struct kvm_cpu_context *ctxt);
void sysreg_restore_host_state_vhe(struct kvm_cpu_context *ctxt);
void sysreg_save_guest_state_vhe(struct kvm_cpu_context *ctxt);
......@@ -111,11 +113,6 @@ void __fpsimd_save_state(struct user_fpsimd_state *fp_regs);
void __fpsimd_restore_state(struct user_fpsimd_state *fp_regs);
void __sve_restore_state(void *sve_pffr, u32 *fpsr);
#ifndef __KVM_NVHE_HYPERVISOR__
void activate_traps_vhe_load(struct kvm_vcpu *vcpu);
void deactivate_traps_vhe_put(struct kvm_vcpu *vcpu);
#endif
u64 __guest_enter(struct kvm_vcpu *vcpu);
bool kvm_host_psci_handler(struct kvm_cpu_context *host_ctxt, u32 func_id);
......
arch/arm64/include/asm/kvm_mmu.h
View file @ 6803bd79
......@@ -150,9 +150,9 @@ static __always_inline unsigned long __kern_hyp_va(unsigned long v)
*/
#define KVM_PHYS_SHIFT (40)
#define kvm_phys_shift(kvm) VTCR_EL2_IPA(kvm->arch.vtcr)
#define kvm_phys_size(kvm) (_AC(1, ULL) << kvm_phys_shift(kvm))
#define kvm_phys_mask(kvm) (kvm_phys_size(kvm) - _AC(1, ULL))
#define kvm_phys_shift(mmu) VTCR_EL2_IPA((mmu)->vtcr)
#define kvm_phys_size(mmu) (_AC(1, ULL) << kvm_phys_shift(mmu))
#define kvm_phys_mask(mmu) (kvm_phys_size(mmu) - _AC(1, ULL))
#include <asm/kvm_pgtable.h>
#include <asm/stage2_pgtable.h>
......@@ -224,16 +224,41 @@ static inline void __clean_dcache_guest_page(void *va, size_t size)
kvm_flush_dcache_to_poc(va, size);
}
static inline size_t __invalidate_icache_max_range(void)
{
u8 iminline;
u64 ctr;
asm volatile(ALTERNATIVE_CB("movz %0, #0\n"
"movk %0, #0, lsl #16\n"
"movk %0, #0, lsl #32\n"
"movk %0, #0, lsl #48\n",
ARM64_ALWAYS_SYSTEM,
kvm_compute_final_ctr_el0)
: "=r" (ctr));
iminline = SYS_FIELD_GET(CTR_EL0, IminLine, ctr) + 2;
return MAX_DVM_OPS << iminline;
}
static inline void __invalidate_icache_guest_page(void *va, size_t size)
{
if (icache_is_aliasing()) {
/* any kind of VIPT cache */
/*
* VPIPT I-cache maintenance must be done from EL2. See comment in the
* nVHE flavor of __kvm_tlb_flush_vmid_ipa().
*/
if (icache_is_vpipt() && read_sysreg(CurrentEL) != CurrentEL_EL2)
return;
/*
* Blow the whole I-cache if it is aliasing (i.e. VIPT) or the
* invalidation range exceeds our arbitrary limit on invadations by
* cache line.
*/
if (icache_is_aliasing() || size > __invalidate_icache_max_range())
icache_inval_all_pou();
} else if (read_sysreg(CurrentEL) != CurrentEL_EL1 ||
!icache_is_vpipt()) {
/* PIPT or VPIPT at EL2 (see comment in __kvm_tlb_flush_vmid_ipa) */
else
icache_inval_pou((unsigned long)va, (unsigned long)va + size);
}
}
void kvm_set_way_flush(struct kvm_vcpu *vcpu);
......@@ -299,7 +324,7 @@ static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu)
static __always_inline void __load_stage2(struct kvm_s2_mmu *mmu,
struct kvm_arch *arch)
{
write_sysreg(arch->vtcr, vtcr_el2);
write_sysreg(mmu->vtcr, vtcr_el2);
write_sysreg(kvm_get_vttbr(mmu), vttbr_el2);
/*
......
arch/arm64/include/asm/kvm_nested.h
View file @ 6803bd79
......@@ -2,13 +2,14 @@
#ifndef __ARM64_KVM_NESTED_H
#define __ARM64_KVM_NESTED_H
#include <asm/kvm_emulate.h>
#include <linux/kvm_host.h>
static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu)
{
return (!__is_defined(__KVM_NVHE_HYPERVISOR__) &&
cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) &&
test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features));
vcpu_has_feature(vcpu, KVM_ARM_VCPU_HAS_EL2));
}
extern bool __check_nv_sr_forward(struct kvm_vcpu *vcpu);
......
arch/arm64/include/asm/stage2_pgtable.h
View file @ 6803bd79
......@@ -21,13 +21,13 @@
* (IPA_SHIFT - 4).
*/
#define stage2_pgtable_levels(ipa) ARM64_HW_PGTABLE_LEVELS((ipa) - 4)
#define kvm_stage2_levels(kvm) VTCR_EL2_LVLS(kvm->arch.vtcr)
#define kvm_stage2_levels(mmu) VTCR_EL2_LVLS((mmu)->vtcr)
/*
* kvm_mmmu_cache_min_pages() is the number of pages required to install
* a stage-2 translation. We pre-allocate the entry level page table at
* the VM creation.
*/
#define kvm_mmu_cache_min_pages(kvm) (kvm_stage2_levels(kvm) - 1)
#define kvm_mmu_cache_min_pages(mmu) (kvm_stage2_levels(mmu) - 1)
#endif /* __ARM64_S2_PGTABLE_H_ */
arch/arm64/include/asm/sysreg.h
View file @ 6803bd79
......@@ -270,6 +270,8 @@
/* ETM */
#define SYS_TRCOSLAR sys_reg(2, 1, 1, 0, 4)
#define SYS_BRBCR_EL2 sys_reg(2, 4, 9, 0, 0)
#define SYS_MIDR_EL1 sys_reg(3, 0, 0, 0, 0)
#define SYS_MPIDR_EL1 sys_reg(3, 0, 0, 0, 5)
#define SYS_REVIDR_EL1 sys_reg(3, 0, 0, 0, 6)
......@@ -484,6 +486,7 @@
#define SYS_SCTLR_EL2 sys_reg(3, 4, 1, 0, 0)
#define SYS_ACTLR_EL2 sys_reg(3, 4, 1, 0, 1)
#define SYS_SCTLR2_EL2 sys_reg(3, 4, 1, 0, 3)
#define SYS_HCR_EL2 sys_reg(3, 4, 1, 1, 0)
#define SYS_MDCR_EL2 sys_reg(3, 4, 1, 1, 1)
#define SYS_CPTR_EL2 sys_reg(3, 4, 1, 1, 2)
......@@ -497,10 +500,15 @@
#define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2)
#define SYS_TRFCR_EL2 sys_reg(3, 4, 1, 2, 1)
#define SYS_VNCR_EL2 sys_reg(3, 4, 2, 2, 0)
#define SYS_HAFGRTR_EL2 sys_reg(3, 4, 3, 1, 6)
#define SYS_SPSR_EL2 sys_reg(3, 4, 4, 0, 0)
#define SYS_ELR_EL2 sys_reg(3, 4, 4, 0, 1)
#define SYS_SP_EL1 sys_reg(3, 4, 4, 1, 0)
#define SYS_SPSR_irq sys_reg(3, 4, 4, 3, 0)
#define SYS_SPSR_abt sys_reg(3, 4, 4, 3, 1)
#define SYS_SPSR_und sys_reg(3, 4, 4, 3, 2)
#define SYS_SPSR_fiq sys_reg(3, 4, 4, 3, 3)
#define SYS_IFSR32_EL2 sys_reg(3, 4, 5, 0, 1)
#define SYS_AFSR0_EL2 sys_reg(3, 4, 5, 1, 0)
#define SYS_AFSR1_EL2 sys_reg(3, 4, 5, 1, 1)
......@@ -514,6 +522,18 @@
#define SYS_MAIR_EL2 sys_reg(3, 4, 10, 2, 0)
#define SYS_AMAIR_EL2 sys_reg(3, 4, 10, 3, 0)
#define SYS_MPAMHCR_EL2 sys_reg(3, 4, 10, 4, 0)
#define SYS_MPAMVPMV_EL2 sys_reg(3, 4, 10, 4, 1)
#define SYS_MPAM2_EL2 sys_reg(3, 4, 10, 5, 0)
#define __SYS__MPAMVPMx_EL2(x) sys_reg(3, 4, 10, 6, x)
#define SYS_MPAMVPM0_EL2 __SYS__MPAMVPMx_EL2(0)
#define SYS_MPAMVPM1_EL2 __SYS__MPAMVPMx_EL2(1)
#define SYS_MPAMVPM2_EL2 __SYS__MPAMVPMx_EL2(2)
#define SYS_MPAMVPM3_EL2 __SYS__MPAMVPMx_EL2(3)
#define SYS_MPAMVPM4_EL2 __SYS__MPAMVPMx_EL2(4)
#define SYS_MPAMVPM5_EL2 __SYS__MPAMVPMx_EL2(5)
#define SYS_MPAMVPM6_EL2 __SYS__MPAMVPMx_EL2(6)
#define SYS_MPAMVPM7_EL2 __SYS__MPAMVPMx_EL2(7)
#define SYS_VBAR_EL2 sys_reg(3, 4, 12, 0, 0)
#define SYS_RVBAR_EL2 sys_reg(3, 4, 12, 0, 1)
......@@ -562,24 +582,49 @@
#define SYS_CONTEXTIDR_EL2 sys_reg(3, 4, 13, 0, 1)
#define SYS_TPIDR_EL2 sys_reg(3, 4, 13, 0, 2)
#define SYS_SCXTNUM_EL2 sys_reg(3, 4, 13, 0, 7)
#define __AMEV_op2(m) (m & 0x7)
#define __AMEV_CRm(n, m) (n | ((m & 0x8) >> 3))
#define __SYS__AMEVCNTVOFF0n_EL2(m) sys_reg(3, 4, 13, __AMEV_CRm(0x8, m), __AMEV_op2(m))
#define SYS_AMEVCNTVOFF0n_EL2(m) __SYS__AMEVCNTVOFF0n_EL2(m)
#define __SYS__AMEVCNTVOFF1n_EL2(m) sys_reg(3, 4, 13, __AMEV_CRm(0xA, m), __AMEV_op2(m))
#define SYS_AMEVCNTVOFF1n_EL2(m) __SYS__AMEVCNTVOFF1n_EL2(m)
#define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3)
#define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0)
#define SYS_CNTHP_TVAL_EL2 sys_reg(3, 4, 14, 2, 0)
#define SYS_CNTHP_CTL_EL2 sys_reg(3, 4, 14, 2, 1)
#define SYS_CNTHP_CVAL_EL2 sys_reg(3, 4, 14, 2, 2)
#define SYS_CNTHV_TVAL_EL2 sys_reg(3, 4, 14, 3, 0)
#define SYS_CNTHV_CTL_EL2 sys_reg(3, 4, 14, 3, 1)
#define SYS_CNTHV_CVAL_EL2 sys_reg(3, 4, 14, 3, 2)
/* VHE encodings for architectural EL0/1 system registers */
#define SYS_BRBCR_EL12 sys_reg(2, 5, 9, 0, 0)
#define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0)
#define SYS_CPACR_EL12 sys_reg(3, 5, 1, 0, 2)
#define SYS_SCTLR2_EL12 sys_reg(3, 5, 1, 0, 3)
#define SYS_ZCR_EL12 sys_reg(3, 5, 1, 2, 0)
#define SYS_TRFCR_EL12 sys_reg(3, 5, 1, 2, 1)
#define SYS_SMCR_EL12 sys_reg(3, 5, 1, 2, 6)
#define SYS_TTBR0_EL12 sys_reg(3, 5, 2, 0, 0)
#define SYS_TTBR1_EL12 sys_reg(3, 5, 2, 0, 1)
#define SYS_TCR_EL12 sys_reg(3, 5, 2, 0, 2)
#define SYS_TCR2_EL12 sys_reg(3, 5, 2, 0, 3)
#define SYS_SPSR_EL12 sys_reg(3, 5, 4, 0, 0)
#define SYS_ELR_EL12 sys_reg(3, 5, 4, 0, 1)
#define SYS_AFSR0_EL12 sys_reg(3, 5, 5, 1, 0)
#define SYS_AFSR1_EL12 sys_reg(3, 5, 5, 1, 1)
#define SYS_ESR_EL12 sys_reg(3, 5, 5, 2, 0)
#define SYS_TFSR_EL12 sys_reg(3, 5, 5, 6, 0)
#define SYS_FAR_EL12 sys_reg(3, 5, 6, 0, 0)
#define SYS_PMSCR_EL12 sys_reg(3, 5, 9, 9, 0)
#define SYS_MAIR_EL12 sys_reg(3, 5, 10, 2, 0)
#define SYS_AMAIR_EL12 sys_reg(3, 5, 10, 3, 0)
#define SYS_VBAR_EL12 sys_reg(3, 5, 12, 0, 0)
#define SYS_CONTEXTIDR_EL12 sys_reg(3, 5, 13, 0, 1)
#define SYS_SCXTNUM_EL12 sys_reg(3, 5, 13, 0, 7)
#define SYS_CNTKCTL_EL12 sys_reg(3, 5, 14, 1, 0)
#define SYS_CNTP_TVAL_EL02 sys_reg(3, 5, 14, 2, 0)
#define SYS_CNTP_CTL_EL02 sys_reg(3, 5, 14, 2, 1)
......
arch/arm64/include/asm/tlbflush.h
View file @ 6803bd79
......@@ -332,7 +332,7 @@ static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
* This is meant to avoid soft lock-ups on large TLB flushing ranges and not
* necessarily a performance improvement.
*/
#define MAX_TLBI_OPS PTRS_PER_PTE
#define MAX_DVM_OPS PTRS_PER_PTE
/*
* __flush_tlb_range_op - Perform TLBI operation upon a range
......@@ -412,12 +412,12 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma,
/*
* When not uses TLB range ops, we can handle up to
* (MAX_TLBI_OPS - 1) pages;
* (MAX_DVM_OPS - 1) pages;
* When uses TLB range ops, we can handle up to
* (MAX_TLBI_RANGE_PAGES - 1) pages.
*/
if ((!system_supports_tlb_range() &&
(end - start) >= (MAX_TLBI_OPS * stride)) ||
(end - start) >= (MAX_DVM_OPS * stride)) ||
pages >= MAX_TLBI_RANGE_PAGES) {
flush_tlb_mm(vma->vm_mm);
return;
......@@ -450,7 +450,7 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
{
unsigned long addr;
if ((end - start) > (MAX_TLBI_OPS * PAGE_SIZE)) {
if ((end - start) > (MAX_DVM_OPS * PAGE_SIZE)) {
flush_tlb_all();
return;
}
......
arch/arm64/include/asm/traps.h
View file @ 6803bd79
......@@ -9,10 +9,9 @@
#include <linux/list.h>
#include <asm/esr.h>
#include <asm/ptrace.h>
#include <asm/sections.h>
struct pt_regs;
#ifdef CONFIG_ARMV8_DEPRECATED
bool try_emulate_armv8_deprecated(struct pt_regs *regs, u32 insn);
#else
......@@ -101,4 +100,55 @@ static inline unsigned long arm64_ras_serror_get_severity(unsigned long esr)
bool arm64_is_fatal_ras_serror(struct pt_regs *regs, unsigned long esr);
void __noreturn arm64_serror_panic(struct pt_regs *regs, unsigned long esr);
static inline void arm64_mops_reset_regs(struct user_pt_regs *regs, unsigned long esr)
{
bool wrong_option = esr & ESR_ELx_MOPS_ISS_WRONG_OPTION;
bool option_a = esr & ESR_ELx_MOPS_ISS_OPTION_A;
int dstreg = ESR_ELx_MOPS_ISS_DESTREG(esr);
int srcreg = ESR_ELx_MOPS_ISS_SRCREG(esr);
int sizereg = ESR_ELx_MOPS_ISS_SIZEREG(esr);
unsigned long dst, src, size;
dst = regs->regs[dstreg];
src = regs->regs[srcreg];
size = regs->regs[sizereg];
/*
* Put the registers back in the original format suitable for a
* prologue instruction, using the generic return routine from the
* Arm ARM (DDI 0487I.a) rules CNTMJ and MWFQH.
*/
if (esr & ESR_ELx_MOPS_ISS_MEM_INST) {
/* SET* instruction */
if (option_a ^ wrong_option) {
/* Format is from Option A; forward set */
regs->regs[dstreg] = dst + size;
regs->regs[sizereg] = -size;
}
} else {
/* CPY* instruction */
if (!(option_a ^ wrong_option)) {
/* Format is from Option B */
if (regs->pstate & PSR_N_BIT) {
/* Backward copy */
regs->regs[dstreg] = dst - size;
regs->regs[srcreg] = src - size;
}
} else {
/* Format is from Option A */
if (size & BIT(63)) {
/* Forward copy */
regs->regs[dstreg] = dst + size;
regs->regs[srcreg] = src + size;
regs->regs[sizereg] = -size;
}
}
}
if (esr & ESR_ELx_MOPS_ISS_FROM_EPILOGUE)
regs->pc -= 8;
else
regs->pc -= 4;
}
#endif
arch/arm64/include/uapi/asm/kvm.h
View file @ 6803bd79
......@@ -505,6 +505,38 @@ struct kvm_smccc_filter {
#define KVM_HYPERCALL_EXIT_SMC (1U << 0)
#define KVM_HYPERCALL_EXIT_16BIT (1U << 1)
/*
* Get feature ID registers userspace writable mask.
*
* From DDI0487J.a, D19.2.66 ("ID_AA64MMFR2_EL1, AArch64 Memory Model
* Feature Register 2"):
*
* "The Feature ID space is defined as the System register space in
* AArch64 with op0==3, op1=={0, 1, 3}, CRn==0, CRm=={0-7},
* op2=={0-7}."
*
* This covers all currently known R/O registers that indicate
* anything useful feature wise, including the ID registers.
*
* If we ever need to introduce a new range, it will be described as
* such in the range field.
*/
#define KVM_ARM_FEATURE_ID_RANGE_IDX(op0, op1, crn, crm, op2) \
({ \
__u64 __op1 = (op1) & 3; \
__op1 -= (__op1 == 3); \
(__op1 << 6 | ((crm) & 7) << 3 | (op2)); \
})
#define KVM_ARM_FEATURE_ID_RANGE 0
#define KVM_ARM_FEATURE_ID_RANGE_SIZE (3 * 8 * 8)
struct reg_mask_range {
__u64 addr; /* Pointer to mask array */
__u32 range; /* Requested range */
__u32 reserved[13];
};
#endif
#endif /* __ARM_KVM_H__ */
arch/arm64/kernel/traps.c
View file @ 6803bd79
......@@ -516,53 +516,7 @@ void do_el1_fpac(struct pt_regs *regs, unsigned long esr)
void do_el0_mops(struct pt_regs *regs, unsigned long esr)
{
bool wrong_option = esr & ESR_ELx_MOPS_ISS_WRONG_OPTION;
bool option_a = esr & ESR_ELx_MOPS_ISS_OPTION_A;
int dstreg = ESR_ELx_MOPS_ISS_DESTREG(esr);
int srcreg = ESR_ELx_MOPS_ISS_SRCREG(esr);
int sizereg = ESR_ELx_MOPS_ISS_SIZEREG(esr);
unsigned long dst, src, size;
dst = pt_regs_read_reg(regs, dstreg);
src = pt_regs_read_reg(regs, srcreg);
size = pt_regs_read_reg(regs, sizereg);
/*
* Put the registers back in the original format suitable for a
* prologue instruction, using the generic return routine from the
* Arm ARM (DDI 0487I.a) rules CNTMJ and MWFQH.
*/
if (esr & ESR_ELx_MOPS_ISS_MEM_INST) {
/* SET* instruction */
if (option_a ^ wrong_option) {
/* Format is from Option A; forward set */
pt_regs_write_reg(regs, dstreg, dst + size);
pt_regs_write_reg(regs, sizereg, -size);
}
} else {
/* CPY* instruction */
if (!(option_a ^ wrong_option)) {
/* Format is from Option B */
if (regs->pstate & PSR_N_BIT) {
/* Backward copy */
pt_regs_write_reg(regs, dstreg, dst - size);
pt_regs_write_reg(regs, srcreg, src - size);
}
} else {
/* Format is from Option A */
if (size & BIT(63)) {
/* Forward copy */
pt_regs_write_reg(regs, dstreg, dst + size);
pt_regs_write_reg(regs, srcreg, src + size);
pt_regs_write_reg(regs, sizereg, -size);
}
}
}
if (esr & ESR_ELx_MOPS_ISS_FROM_EPILOGUE)
regs->pc -= 8;
else
regs->pc -= 4;
arm64_mops_reset_regs(&regs->user_regs, esr);
/*
* If single stepping then finish the step before executing the
......
arch/arm64/kvm/arch_timer.c
View file @ 6803bd79
......@@ -453,7 +453,7 @@ static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level,
timer_ctx->irq.level);
if (!userspace_irqchip(vcpu->kvm)) {
ret = kvm_vgic_inject_irq(vcpu->kvm, vcpu->vcpu_id,
ret = kvm_vgic_inject_irq(vcpu->kvm, vcpu,
timer_irq(timer_ctx),
timer_ctx->irq.level,
timer_ctx);
......@@ -936,7 +936,7 @@ void kvm_timer_sync_user(struct kvm_vcpu *vcpu)
unmask_vtimer_irq_user(vcpu);
}
int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu)
void kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu)
{
struct arch_timer_cpu *timer = vcpu_timer(vcpu);
struct timer_map map;
......@@ -980,8 +980,6 @@ int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu)
soft_timer_cancel(&map.emul_vtimer->hrtimer);
if (map.emul_ptimer)
soft_timer_cancel(&map.emul_ptimer->hrtimer);
return 0;
}
static void timer_context_init(struct kvm_vcpu *vcpu, int timerid)
......
arch/arm64/kvm/arm.c
View file @ 6803bd79
This diff is collapsed.
arch/arm64/kvm/emulate-nested.c
View file @ 6803bd79
......@@ -648,15 +648,80 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initconst = {
SR_TRAP(SYS_APGAKEYLO_EL1, CGT_HCR_APK),
SR_TRAP(SYS_APGAKEYHI_EL1, CGT_HCR_APK),
/* All _EL2 registers */
SR_RANGE_TRAP(sys_reg(3, 4, 0, 0, 0),
sys_reg(3, 4, 3, 15, 7), CGT_HCR_NV),
SR_TRAP(SYS_BRBCR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VPIDR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VMPIDR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_SCTLR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_ACTLR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_SCTLR2_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_HCR_EL2,
SYS_HCRX_EL2, CGT_HCR_NV),
SR_TRAP(SYS_SMPRIMAP_EL2, CGT_HCR_NV),
SR_TRAP(SYS_SMCR_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_TTBR0_EL2,
SYS_TCR2_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VTTBR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VTCR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VNCR_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_HDFGRTR_EL2,
SYS_HAFGRTR_EL2, CGT_HCR_NV),
/* Skip the SP_EL1 encoding... */
SR_TRAP(SYS_SPSR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_ELR_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(sys_reg(3, 4, 4, 1, 1),
sys_reg(3, 4, 10, 15, 7), CGT_HCR_NV),
SR_RANGE_TRAP(sys_reg(3, 4, 12, 0, 0),
sys_reg(3, 4, 14, 15, 7), CGT_HCR_NV),
/* Skip SPSR_irq, SPSR_abt, SPSR_und, SPSR_fiq */
SR_TRAP(SYS_AFSR0_EL2, CGT_HCR_NV),
SR_TRAP(SYS_AFSR1_EL2, CGT_HCR_NV),
SR_TRAP(SYS_ESR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VSESR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_TFSR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_FAR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_HPFAR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_PMSCR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_MAIR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_AMAIR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_MPAMHCR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_MPAMVPMV_EL2, CGT_HCR_NV),
SR_TRAP(SYS_MPAM2_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_MPAMVPM0_EL2,
SYS_MPAMVPM7_EL2, CGT_HCR_NV),
/*
* Note that the spec. describes a group of MEC registers
* whose access should not trap, therefore skip the following:
* MECID_A0_EL2, MECID_A1_EL2, MECID_P0_EL2,
* MECID_P1_EL2, MECIDR_EL2, VMECID_A_EL2,
* VMECID_P_EL2.
*/
SR_RANGE_TRAP(SYS_VBAR_EL2,
SYS_RMR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_VDISR_EL2, CGT_HCR_NV),
/* ICH_AP0R<m>_EL2 */
SR_RANGE_TRAP(SYS_ICH_AP0R0_EL2,
SYS_ICH_AP0R3_EL2, CGT_HCR_NV),
/* ICH_AP1R<m>_EL2 */
SR_RANGE_TRAP(SYS_ICH_AP1R0_EL2,
SYS_ICH_AP1R3_EL2, CGT_HCR_NV),
SR_TRAP(SYS_ICC_SRE_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_ICH_HCR_EL2,
SYS_ICH_EISR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_ICH_ELRSR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_ICH_VMCR_EL2, CGT_HCR_NV),
/* ICH_LR<m>_EL2 */
SR_RANGE_TRAP(SYS_ICH_LR0_EL2,
SYS_ICH_LR15_EL2, CGT_HCR_NV),
SR_TRAP(SYS_CONTEXTIDR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_TPIDR_EL2, CGT_HCR_NV),
SR_TRAP(SYS_SCXTNUM_EL2, CGT_HCR_NV),
/* AMEVCNTVOFF0<n>_EL2, AMEVCNTVOFF1<n>_EL2 */
SR_RANGE_TRAP(SYS_AMEVCNTVOFF0n_EL2(0),
SYS_AMEVCNTVOFF1n_EL2(15), CGT_HCR_NV),
/* CNT*_EL2 */
SR_TRAP(SYS_CNTVOFF_EL2, CGT_HCR_NV),
SR_TRAP(SYS_CNTPOFF_EL2, CGT_HCR_NV),
SR_TRAP(SYS_CNTHCTL_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_CNTHP_TVAL_EL2,
SYS_CNTHP_CVAL_EL2, CGT_HCR_NV),
SR_RANGE_TRAP(SYS_CNTHV_TVAL_EL2,
SYS_CNTHV_CVAL_EL2, CGT_HCR_NV),
/* All _EL02, _EL12 registers */
SR_RANGE_TRAP(sys_reg(3, 5, 0, 0, 0),
sys_reg(3, 5, 10, 15, 7), CGT_HCR_NV),
......
arch/arm64/kvm/hyp/include/hyp/switch.h
View file @ 6803bd79
......@@ -30,6 +30,7 @@
#include <asm/fpsimd.h>
#include <asm/debug-monitors.h>
#include <asm/processor.h>
#include <asm/traps.h>
struct kvm_exception_table_entry {
int insn, fixup;
......@@ -265,6 +266,22 @@ static inline bool __populate_fault_info(struct kvm_vcpu *vcpu)
return __get_fault_info(vcpu->arch.fault.esr_el2, &vcpu->arch.fault);
}
static bool kvm_hyp_handle_mops(struct kvm_vcpu *vcpu, u64 *exit_code)
{
*vcpu_pc(vcpu) = read_sysreg_el2(SYS_ELR);
arm64_mops_reset_regs(vcpu_gp_regs(vcpu), vcpu->arch.fault.esr_el2);
write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR);
/*
* Finish potential single step before executing the prologue
* instruction.
*/
*vcpu_cpsr(vcpu) &= ~DBG_SPSR_SS;
write_sysreg_el2(*vcpu_cpsr(vcpu), SYS_SPSR);
return true;
}
static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu)
{
sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1, SYS_ZCR_EL2);
......
arch/arm64/kvm/hyp/include/nvhe/fixed_config.h
View file @ 6803bd79
......@@ -197,7 +197,8 @@
#define PVM_ID_AA64ISAR2_ALLOW (\
ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_GPA3) | \
ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_APA3) \
ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_APA3) | \
ARM64_FEATURE_MASK(ID_AA64ISAR2_EL1_MOPS) \
)
u64 pvm_read_id_reg(const struct kvm_vcpu *vcpu, u32 id);
......
arch/arm64/kvm/hyp/nvhe/mem_protect.c
View file @ 6803bd79
......@@ -129,8 +129,8 @@ static void prepare_host_vtcr(void)
parange = kvm_get_parange(id_aa64mmfr0_el1_sys_val);
phys_shift = id_aa64mmfr0_parange_to_phys_shift(parange);
host_mmu.arch.vtcr = kvm_get_vtcr(id_aa64mmfr0_el1_sys_val,
id_aa64mmfr1_el1_sys_val, phys_shift);
host_mmu.arch.mmu.vtcr = kvm_get_vtcr(id_aa64mmfr0_el1_sys_val,
id_aa64mmfr1_el1_sys_val, phys_shift);
}
static bool host_stage2_force_pte_cb(u64 addr, u64 end, enum kvm_pgtable_prot prot);
......@@ -235,7 +235,7 @@ int kvm_guest_prepare_stage2(struct pkvm_hyp_vm *vm, void *pgd)
unsigned long nr_pages;
int ret;
nr_pages = kvm_pgtable_stage2_pgd_size(vm->kvm.arch.vtcr) >> PAGE_SHIFT;
nr_pages = kvm_pgtable_stage2_pgd_size(mmu->vtcr) >> PAGE_SHIFT;
ret = hyp_pool_init(&vm->pool, hyp_virt_to_pfn(pgd), nr_pages, 0);
if (ret)
return ret;
......@@ -295,7 +295,7 @@ int __pkvm_prot_finalize(void)
return -EPERM;
params->vttbr = kvm_get_vttbr(mmu);
params->vtcr = host_mmu.arch.vtcr;
params->vtcr = mmu->vtcr;
params->hcr_el2 |= HCR_VM;
/*
......
arch/arm64/kvm/hyp/nvhe/pkvm.c
View file @ 6803bd79
......@@ -303,7 +303,7 @@ static void init_pkvm_hyp_vm(struct kvm *host_kvm, struct pkvm_hyp_vm *hyp_vm,
{
hyp_vm->host_kvm = host_kvm;
hyp_vm->kvm.created_vcpus = nr_vcpus;
hyp_vm->kvm.arch.vtcr = host_mmu.arch.vtcr;
hyp_vm->kvm.arch.mmu.vtcr = host_mmu.arch.mmu.vtcr;
}
static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu,
......@@ -483,7 +483,7 @@ int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva,
}
vm_size = pkvm_get_hyp_vm_size(nr_vcpus);
pgd_size = kvm_pgtable_stage2_pgd_size(host_mmu.arch.vtcr);
pgd_size = kvm_pgtable_stage2_pgd_size(host_mmu.arch.mmu.vtcr);
ret = -ENOMEM;
......
arch/arm64/kvm/hyp/nvhe/switch.c
View file @ 6803bd79
......@@ -192,6 +192,7 @@ static const exit_handler_fn hyp_exit_handlers[] = {
[ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low,
[ESR_ELx_EC_WATCHPT_LOW] = kvm_hyp_handle_watchpt_low,
[ESR_ELx_EC_PAC] = kvm_hyp_handle_ptrauth,
[ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops,
};
static const exit_handler_fn pvm_exit_handlers[] = {
......@@ -203,6 +204,7 @@ static const exit_handler_fn pvm_exit_handlers[] = {
[ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low,
[ESR_ELx_EC_WATCHPT_LOW] = kvm_hyp_handle_watchpt_low,
[ESR_ELx_EC_PAC] = kvm_hyp_handle_ptrauth,
[ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops,
};
static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu)
......
arch/arm64/kvm/hyp/pgtable.c
View file @ 6803bd79
......@@ -1314,7 +1314,7 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level,
KVM_PGTABLE_WALK_HANDLE_FAULT |
KVM_PGTABLE_WALK_SHARED);
if (!ret)
if (!ret || ret == -EAGAIN)
kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, pgt->mmu, addr, level);
return ret;
}
......@@ -1511,7 +1511,7 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
kvm_pgtable_force_pte_cb_t force_pte_cb)
{
size_t pgd_sz;
u64 vtcr = mmu->arch->vtcr;
u64 vtcr = mmu->vtcr;
u32 ia_bits = VTCR_EL2_IPA(vtcr);
u32 sl0 = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr);
u32 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0;
......
arch/arm64/kvm/hyp/vhe/switch.c
View file @ 6803bd79
......@@ -137,12 +137,12 @@ static void __deactivate_traps(struct kvm_vcpu *vcpu)
NOKPROBE_SYMBOL(__deactivate_traps);
/*
* Disable IRQs in {activate,deactivate}_traps_vhe_{load,put}() to
* Disable IRQs in __vcpu_{load,put}_{activate,deactivate}_traps() to
* prevent a race condition between context switching of PMUSERENR_EL0
* in __{activate,deactivate}_traps_common() and IPIs that attempts to
* update PMUSERENR_EL0. See also kvm_set_pmuserenr().
*/
void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
static void __vcpu_load_activate_traps(struct kvm_vcpu *vcpu)
{
unsigned long flags;
......@@ -151,7 +151,7 @@ void activate_traps_vhe_load(struct kvm_vcpu *vcpu)
local_irq_restore(flags);
}
void deactivate_traps_vhe_put(struct kvm_vcpu *vcpu)
static void __vcpu_put_deactivate_traps(struct kvm_vcpu *vcpu)
{
unsigned long flags;
......@@ -160,6 +160,19 @@ void deactivate_traps_vhe_put(struct kvm_vcpu *vcpu)
local_irq_restore(flags);
}
void kvm_vcpu_load_vhe(struct kvm_vcpu *vcpu)
{
__vcpu_load_switch_sysregs(vcpu);
__vcpu_load_activate_traps(vcpu);
__load_stage2(vcpu->arch.hw_mmu, vcpu->arch.hw_mmu->arch);
}
void kvm_vcpu_put_vhe(struct kvm_vcpu *vcpu)
{
__vcpu_put_deactivate_traps(vcpu);
__vcpu_put_switch_sysregs(vcpu);
}
static const exit_handler_fn hyp_exit_handlers[] = {
[0 ... ESR_ELx_EC_MAX] = NULL,
[ESR_ELx_EC_CP15_32] = kvm_hyp_handle_cp15_32,
......@@ -170,6 +183,7 @@ static const exit_handler_fn hyp_exit_handlers[] = {
[ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low,
[ESR_ELx_EC_WATCHPT_LOW] = kvm_hyp_handle_watchpt_low,
[ESR_ELx_EC_PAC] = kvm_hyp_handle_ptrauth,
[ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops,
};
static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu)
......@@ -214,17 +228,11 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
sysreg_save_host_state_vhe(host_ctxt);
/*
* ARM erratum 1165522 requires us to configure both stage 1 and
* stage 2 translation for the guest context before we clear
* HCR_EL2.TGE.
*
* We have already configured the guest's stage 1 translation in
* kvm_vcpu_load_sysregs_vhe above. We must now call
* __load_stage2 before __activate_traps, because
* __load_stage2 configures stage 2 translation, and
* __activate_traps clear HCR_EL2.TGE (among other things).
* Note that ARM erratum 1165522 requires us to configure both stage 1
* and stage 2 translation for the guest context before we clear
* HCR_EL2.TGE. The stage 1 and stage 2 guest context has already been
* loaded on the CPU in kvm_vcpu_load_vhe().
*/
__load_stage2(vcpu->arch.hw_mmu, vcpu->arch.hw_mmu->arch);
__activate_traps(vcpu);
__kvm_adjust_pc(vcpu);
......
arch/arm64/kvm/hyp/vhe/sysreg-sr.c
View file @ 6803bd79
......@@ -52,7 +52,7 @@ void sysreg_restore_guest_state_vhe(struct kvm_cpu_context *ctxt)
NOKPROBE_SYMBOL(sysreg_restore_guest_state_vhe);
/**
* kvm_vcpu_load_sysregs_vhe - Load guest system registers to the physical CPU
* __vcpu_load_switch_sysregs - Load guest system registers to the physical CPU
*
* @vcpu: The VCPU pointer
*
......@@ -62,7 +62,7 @@ NOKPROBE_SYMBOL(sysreg_restore_guest_state_vhe);
* and loading system register state early avoids having to load them on
* every entry to the VM.
*/
void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu)
void __vcpu_load_switch_sysregs(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
struct kvm_cpu_context *host_ctxt;
......@@ -92,12 +92,10 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu)
__sysreg_restore_el1_state(guest_ctxt);
vcpu_set_flag(vcpu, SYSREGS_ON_CPU);
activate_traps_vhe_load(vcpu);
}
/**
* kvm_vcpu_put_sysregs_vhe - Restore host system registers to the physical CPU
* __vcpu_put_switch_syregs - Restore host system registers to the physical CPU
*
* @vcpu: The VCPU pointer
*
......@@ -107,13 +105,12 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu)
* and deferring saving system register state until we're no longer running the
* VCPU avoids having to save them on every exit from the VM.
*/
void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu)
void __vcpu_put_switch_sysregs(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
struct kvm_cpu_context *host_ctxt;
host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
deactivate_traps_vhe_put(vcpu);
__sysreg_save_el1_state(guest_ctxt);
__sysreg_save_user_state(guest_ctxt);
......
arch/arm64/kvm/hyp/vhe/tlb.c
View file @ 6803bd79
......@@ -11,18 +11,25 @@
#include <asm/tlbflush.h>
struct tlb_inv_context {
unsigned long flags;
u64 tcr;
u64 sctlr;
struct kvm_s2_mmu *mmu;
unsigned long flags;
u64 tcr;
u64 sctlr;
};
static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu,
struct tlb_inv_context *cxt)
{
struct kvm_vcpu *vcpu = kvm_get_running_vcpu();
u64 val;
local_irq_save(cxt->flags);
if (vcpu && mmu != vcpu->arch.hw_mmu)
cxt->mmu = vcpu->arch.hw_mmu;
else
cxt->mmu = NULL;
if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
/*
* For CPUs that are affected by ARM errata 1165522 or 1530923,
......@@ -66,10 +73,13 @@ static void __tlb_switch_to_host(struct tlb_inv_context *cxt)
* We're done with the TLB operation, let's restore the host's
* view of HCR_EL2.
*/
write_sysreg(0, vttbr_el2);
write_sysreg(HCR_HOST_VHE_FLAGS, hcr_el2);
isb();
/* ... and the stage-2 MMU context that we switched away from */
if (cxt->mmu)
__load_stage2(cxt->mmu, cxt->mmu->arch);
if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
/* Restore the registers to what they were */
write_sysreg_el1(cxt->tcr, SYS_TCR);
......
arch/arm64/kvm/hypercalls.c
View file @ 6803bd79
......@@ -133,12 +133,10 @@ static bool kvm_smccc_test_fw_bmap(struct kvm_vcpu *vcpu, u32 func_id)
ARM_SMCCC_SMC_64, \
0, ARM_SMCCC_FUNC_MASK)
static void init_smccc_filter(struct kvm *kvm)
static int kvm_smccc_filter_insert_reserved(struct kvm *kvm)
{
int r;
mt_init(&kvm->arch.smccc_filter);
/*
* Prevent userspace from handling any SMCCC calls in the architecture
* range, avoiding the risk of misrepresenting Spectre mitigation status
......@@ -148,14 +146,25 @@ static void init_smccc_filter(struct kvm *kvm)
SMC32_ARCH_RANGE_BEGIN, SMC32_ARCH_RANGE_END,
xa_mk_value(KVM_SMCCC_FILTER_HANDLE),
GFP_KERNEL_ACCOUNT);
WARN_ON_ONCE(r);
if (r)
goto out_destroy;
r = mtree_insert_range(&kvm->arch.smccc_filter,
SMC64_ARCH_RANGE_BEGIN, SMC64_ARCH_RANGE_END,
xa_mk_value(KVM_SMCCC_FILTER_HANDLE),
GFP_KERNEL_ACCOUNT);
WARN_ON_ONCE(r);
if (r)
goto out_destroy;
return 0;
out_destroy:
mtree_destroy(&kvm->arch.smccc_filter);
return r;
}
static bool kvm_smccc_filter_configured(struct kvm *kvm)
{
return !mtree_empty(&kvm->arch.smccc_filter);
}
static int kvm_smccc_set_filter(struct kvm *kvm, struct kvm_smccc_filter __user *uaddr)
......@@ -184,13 +193,14 @@ static int kvm_smccc_set_filter(struct kvm *kvm, struct kvm_smccc_filter __user
goto out_unlock;
}
if (!kvm_smccc_filter_configured(kvm)) {
r = kvm_smccc_filter_insert_reserved(kvm);
if (WARN_ON_ONCE(r))
goto out_unlock;
}
r = mtree_insert_range(&kvm->arch.smccc_filter, start, end,
xa_mk_value(filter.action), GFP_KERNEL_ACCOUNT);
if (r)
goto out_unlock;
set_bit(KVM_ARCH_FLAG_SMCCC_FILTER_CONFIGURED, &kvm->arch.flags);
out_unlock:
mutex_unlock(&kvm->arch.config_lock);
return r;
......@@ -201,7 +211,7 @@ static u8 kvm_smccc_filter_get_action(struct kvm *kvm, u32 func_id)
unsigned long idx = func_id;
void *val;
if (!test_bit(KVM_ARCH_FLAG_SMCCC_FILTER_CONFIGURED, &kvm->arch.flags))
if (!kvm_smccc_filter_configured(kvm))
return KVM_SMCCC_FILTER_HANDLE;
/*
......@@ -387,7 +397,7 @@ void kvm_arm_init_hypercalls(struct kvm *kvm)
smccc_feat->std_hyp_bmap = KVM_ARM_SMCCC_STD_HYP_FEATURES;
smccc_feat->vendor_hyp_bmap = KVM_ARM_SMCCC_VENDOR_HYP_FEATURES;
init_smccc_filter(kvm);
mt_init(&kvm->arch.smccc_filter);
}
void kvm_arm_teardown_hypercalls(struct kvm *kvm)
......@@ -554,7 +564,7 @@ int kvm_arm_set_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
{
bool wants_02;
wants_02 = test_bit(KVM_ARM_VCPU_PSCI_0_2, vcpu->arch.features);
wants_02 = vcpu_has_feature(vcpu, KVM_ARM_VCPU_PSCI_0_2);
switch (val) {
case KVM_ARM_PSCI_0_1:
......
arch/arm64/kvm/mmio.c
View file @ 6803bd79
......@@ -135,6 +135,9 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
* volunteered to do so, and bail out otherwise.
*/
if (!kvm_vcpu_dabt_isvalid(vcpu)) {
trace_kvm_mmio_nisv(*vcpu_pc(vcpu), kvm_vcpu_get_esr(vcpu),
kvm_vcpu_get_hfar(vcpu), fault_ipa);
if (test_bit(KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER,
&vcpu->kvm->arch.flags)) {
run->exit_reason = KVM_EXIT_ARM_NISV;
......@@ -143,7 +146,6 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
return 0;
}
kvm_pr_unimpl("Data abort outside memslots with no valid syndrome info\n");
return -ENOSYS;
}
......
arch/arm64/kvm/mmu.c
View file @ 6803bd79
......@@ -892,7 +892,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu, unsigned long t
mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
mmfr1 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR1_EL1);
kvm->arch.vtcr = kvm_get_vtcr(mmfr0, mmfr1, phys_shift);
mmu->vtcr = kvm_get_vtcr(mmfr0, mmfr1, phys_shift);
if (mmu->pgt != NULL) {
kvm_err("kvm_arch already initialized?\n");
......@@ -1067,7 +1067,8 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
phys_addr_t addr;
int ret = 0;
struct kvm_mmu_memory_cache cache = { .gfp_zero = __GFP_ZERO };
struct kvm_pgtable *pgt = kvm->arch.mmu.pgt;
struct kvm_s2_mmu *mmu = &kvm->arch.mmu;
struct kvm_pgtable *pgt = mmu->pgt;
enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_DEVICE |
KVM_PGTABLE_PROT_R |
(writable ? KVM_PGTABLE_PROT_W : 0);
......@@ -1080,7 +1081,7 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
for (addr = guest_ipa; addr < guest_ipa + size; addr += PAGE_SIZE) {
ret = kvm_mmu_topup_memory_cache(&cache,
kvm_mmu_cache_min_pages(kvm));
kvm_mmu_cache_min_pages(mmu));
if (ret)
break;
......@@ -1298,28 +1299,8 @@ transparent_hugepage_adjust(struct kvm *kvm, struct kvm_memory_slot *memslot,
if (sz < PMD_SIZE)
return PAGE_SIZE;
/*
* The address we faulted on is backed by a transparent huge
* page. However, because we map the compound huge page and
* not the individual tail page, we need to transfer the
* refcount to the head page. We have to be careful that the
* THP doesn't start to split while we are adjusting the
* refcounts.
*
* We are sure this doesn't happen, because mmu_invalidate_retry
* was successful and we are holding the mmu_lock, so if this
* THP is trying to split, it will be blocked in the mmu
* notifier before touching any of the pages, specifically
* before being able to call __split_huge_page_refcount().
*
* We can therefore safely transfer the refcount from PG_tail
* to PG_head and switch the pfn from a tail page to the head
* page accordingly.
*/
*ipap &= PMD_MASK;
kvm_release_pfn_clean(pfn);
pfn &= ~(PTRS_PER_PMD - 1);
get_page(pfn_to_page(pfn));
*pfnp = pfn;
return PMD_SIZE;
......@@ -1431,7 +1412,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
if (fault_status != ESR_ELx_FSC_PERM ||
(logging_active && write_fault)) {
ret = kvm_mmu_topup_memory_cache(memcache,
kvm_mmu_cache_min_pages(kvm));
kvm_mmu_cache_min_pages(vcpu->arch.hw_mmu));
if (ret)
return ret;
}
......@@ -1747,7 +1728,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu)
}
/* Userspace should not be able to register out-of-bounds IPAs */
VM_BUG_ON(fault_ipa >= kvm_phys_size(vcpu->kvm));
VM_BUG_ON(fault_ipa >= kvm_phys_size(vcpu->arch.hw_mmu));
if (fault_status == ESR_ELx_FSC_ACCESS) {
handle_access_fault(vcpu, fault_ipa);
......@@ -2021,7 +2002,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
* Prevent userspace from creating a memory region outside of the IPA
* space addressable by the KVM guest IPA space.
*/
if ((new->base_gfn + new->npages) > (kvm_phys_size(kvm) >> PAGE_SHIFT))
if ((new->base_gfn + new->npages) > (kvm_phys_size(&kvm->arch.mmu) >> PAGE_SHIFT))
return -EFAULT;
hva = new->userspace_addr;
......
arch/arm64/kvm/pkvm.c
View file @ 6803bd79
......@@ -123,7 +123,7 @@ static int __pkvm_create_hyp_vm(struct kvm *host_kvm)
if (host_kvm->created_vcpus < 1)
return -EINVAL;
pgd_sz = kvm_pgtable_stage2_pgd_size(host_kvm->arch.vtcr);
pgd_sz = kvm_pgtable_stage2_pgd_size(host_kvm->arch.mmu.vtcr);
/*
* The PGD pages will be reclaimed using a hyp_memcache which implies
......
arch/arm64/kvm/pmu-emul.c
View file @ 6803bd79
......@@ -60,6 +60,23 @@ static u32 kvm_pmu_event_mask(struct kvm *kvm)
return __kvm_pmu_event_mask(pmuver);
}
u64 kvm_pmu_evtyper_mask(struct kvm *kvm)
{
u64 mask = ARMV8_PMU_EXCLUDE_EL1 | ARMV8_PMU_EXCLUDE_EL0 |
kvm_pmu_event_mask(kvm);
u64 pfr0 = IDREG(kvm, SYS_ID_AA64PFR0_EL1);
if (SYS_FIELD_GET(ID_AA64PFR0_EL1, EL2, pfr0))
mask |= ARMV8_PMU_INCLUDE_EL2;
if (SYS_FIELD_GET(ID_AA64PFR0_EL1, EL3, pfr0))
mask |= ARMV8_PMU_EXCLUDE_NS_EL0 |
ARMV8_PMU_EXCLUDE_NS_EL1 |
ARMV8_PMU_EXCLUDE_EL3;
return mask;
}
/**
* kvm_pmc_is_64bit - determine if counter is 64bit
* @pmc: counter context
......@@ -72,7 +89,7 @@ static bool kvm_pmc_is_64bit(struct kvm_pmc *pmc)
static bool kvm_pmc_has_64bit_overflow(struct kvm_pmc *pmc)
{
u64 val = __vcpu_sys_reg(kvm_pmc_to_vcpu(pmc), PMCR_EL0);
u64 val = kvm_vcpu_read_pmcr(kvm_pmc_to_vcpu(pmc));
return (pmc->idx < ARMV8_PMU_CYCLE_IDX && (val & ARMV8_PMU_PMCR_LP)) ||
(pmc->idx == ARMV8_PMU_CYCLE_IDX && (val & ARMV8_PMU_PMCR_LC));
......@@ -250,7 +267,7 @@ void kvm_pmu_vcpu_destroy(struct kvm_vcpu *vcpu)
u64 kvm_pmu_valid_counter_mask(struct kvm_vcpu *vcpu)
{
u64 val = __vcpu_sys_reg(vcpu, PMCR_EL0) >> ARMV8_PMU_PMCR_N_SHIFT;
u64 val = kvm_vcpu_read_pmcr(vcpu) >> ARMV8_PMU_PMCR_N_SHIFT;
val &= ARMV8_PMU_PMCR_N_MASK;
if (val == 0)
......@@ -272,7 +289,7 @@ void kvm_pmu_enable_counter_mask(struct kvm_vcpu *vcpu, u64 val)
if (!kvm_vcpu_has_pmu(vcpu))
return;
if (!(__vcpu_sys_reg(vcpu, PMCR_EL0) & ARMV8_PMU_PMCR_E) || !val)
if (!(kvm_vcpu_read_pmcr(vcpu) & ARMV8_PMU_PMCR_E) || !val)
return;
for (i = 0; i < ARMV8_PMU_MAX_COUNTERS; i++) {
......@@ -324,7 +341,7 @@ static u64 kvm_pmu_overflow_status(struct kvm_vcpu *vcpu)
{
u64 reg = 0;
if ((__vcpu_sys_reg(vcpu, PMCR_EL0) & ARMV8_PMU_PMCR_E)) {
if ((kvm_vcpu_read_pmcr(vcpu) & ARMV8_PMU_PMCR_E)) {
reg = __vcpu_sys_reg(vcpu, PMOVSSET_EL0);
reg &= __vcpu_sys_reg(vcpu, PMCNTENSET_EL0);
reg &= __vcpu_sys_reg(vcpu, PMINTENSET_EL1);
......@@ -348,7 +365,7 @@ static void kvm_pmu_update_state(struct kvm_vcpu *vcpu)
pmu->irq_level = overflow;
if (likely(irqchip_in_kernel(vcpu->kvm))) {
int ret = kvm_vgic_inject_irq(vcpu->kvm, vcpu->vcpu_id,
int ret = kvm_vgic_inject_irq(vcpu->kvm, vcpu,
pmu->irq_num, overflow, pmu);
WARN_ON(ret);
}
......@@ -426,7 +443,7 @@ static void kvm_pmu_counter_increment(struct kvm_vcpu *vcpu,
{
int i;
if (!(__vcpu_sys_reg(vcpu, PMCR_EL0) & ARMV8_PMU_PMCR_E))
if (!(kvm_vcpu_read_pmcr(vcpu) & ARMV8_PMU_PMCR_E))
return;
/* Weed out disabled counters */
......@@ -569,7 +586,7 @@ void kvm_pmu_handle_pmcr(struct kvm_vcpu *vcpu, u64 val)
static bool kvm_pmu_counter_is_enabled(struct kvm_pmc *pmc)
{
struct kvm_vcpu *vcpu = kvm_pmc_to_vcpu(pmc);
return (__vcpu_sys_reg(vcpu, PMCR_EL0) & ARMV8_PMU_PMCR_E) &&
return (kvm_vcpu_read_pmcr(vcpu) & ARMV8_PMU_PMCR_E) &&
(__vcpu_sys_reg(vcpu, PMCNTENSET_EL0) & BIT(pmc->idx));
}
......@@ -584,6 +601,7 @@ static void kvm_pmu_create_perf_event(struct kvm_pmc *pmc)
struct perf_event *event;
struct perf_event_attr attr;
u64 eventsel, reg, data;
bool p, u, nsk, nsu;
reg = counter_index_to_evtreg(pmc->idx);
data = __vcpu_sys_reg(vcpu, reg);
......@@ -610,13 +628,18 @@ static void kvm_pmu_create_perf_event(struct kvm_pmc *pmc)
!test_bit(eventsel, vcpu->kvm->arch.pmu_filter))
return;
p = data & ARMV8_PMU_EXCLUDE_EL1;
u = data & ARMV8_PMU_EXCLUDE_EL0;
nsk = data & ARMV8_PMU_EXCLUDE_NS_EL1;
nsu = data & ARMV8_PMU_EXCLUDE_NS_EL0;
memset(&attr, 0, sizeof(struct perf_event_attr));
attr.type = arm_pmu->pmu.type;
attr.size = sizeof(attr);
attr.pinned = 1;
attr.disabled = !kvm_pmu_counter_is_enabled(pmc);
attr.exclude_user = data & ARMV8_PMU_EXCLUDE_EL0 ? 1 : 0;
attr.exclude_kernel = data & ARMV8_PMU_EXCLUDE_EL1 ? 1 : 0;
attr.exclude_user = (u != nsu);
attr.exclude_kernel = (p != nsk);
attr.exclude_hv = 1; /* Don't count EL2 events */
attr.exclude_host = 1; /* Don't count host events */
attr.config = eventsel;
......@@ -657,18 +680,13 @@ void kvm_pmu_set_counter_event_type(struct kvm_vcpu *vcpu, u64 data,
u64 select_idx)
{
struct kvm_pmc *pmc = kvm_vcpu_idx_to_pmc(vcpu, select_idx);
u64 reg, mask;
u64 reg;
if (!kvm_vcpu_has_pmu(vcpu))
return;
mask = ARMV8_PMU_EVTYPE_MASK;
mask &= ~ARMV8_PMU_EVTYPE_EVENT;
mask |= kvm_pmu_event_mask(vcpu->kvm);
reg = counter_index_to_evtreg(pmc->idx);
__vcpu_sys_reg(vcpu, reg) = data & mask;
__vcpu_sys_reg(vcpu, reg) = data & kvm_pmu_evtyper_mask(vcpu->kvm);
kvm_pmu_create_perf_event(pmc);
}
......@@ -717,10 +735,9 @@ static struct arm_pmu *kvm_pmu_probe_armpmu(void)
* It is still necessary to get a valid cpu, though, to probe for the
* default PMU instance as userspace is not required to specify a PMU
* type. In order to uphold the preexisting behavior KVM selects the
* PMU instance for the core where the first call to the
* KVM_ARM_VCPU_PMU_V3_CTRL attribute group occurs. A dependent use case
* would be a user with disdain of all things big.LITTLE that affines
* the VMM to a particular cluster of cores.
* PMU instance for the core during vcpu init. A dependent use
* case would be a user with disdain of all things big.LITTLE that
* affines the VMM to a particular cluster of cores.
*
* In any case, userspace should just do the sane thing and use the UAPI
* to select a PMU type directly. But, be wary of the baggage being
......@@ -786,6 +803,17 @@ u64 kvm_pmu_get_pmceid(struct kvm_vcpu *vcpu, bool pmceid1)
return val & mask;
}
void kvm_vcpu_reload_pmu(struct kvm_vcpu *vcpu)
{
u64 mask = kvm_pmu_valid_counter_mask(vcpu);
kvm_pmu_handle_pmcr(vcpu, kvm_vcpu_read_pmcr(vcpu));
__vcpu_sys_reg(vcpu, PMOVSSET_EL0) &= mask;
__vcpu_sys_reg(vcpu, PMINTENSET_EL1) &= mask;
__vcpu_sys_reg(vcpu, PMCNTENSET_EL0) &= mask;
}
int kvm_arm_pmu_v3_enable(struct kvm_vcpu *vcpu)
{
if (!kvm_vcpu_has_pmu(vcpu))
......@@ -874,6 +902,52 @@ static bool pmu_irq_is_valid(struct kvm *kvm, int irq)
return true;
}
/**
* kvm_arm_pmu_get_max_counters - Return the max number of PMU counters.
* @kvm: The kvm pointer
*/
u8 kvm_arm_pmu_get_max_counters(struct kvm *kvm)
{
struct arm_pmu *arm_pmu = kvm->arch.arm_pmu;
/*
* The arm_pmu->num_events considers the cycle counter as well.
* Ignore that and return only the general-purpose counters.
*/
return arm_pmu->num_events - 1;
}
static void kvm_arm_set_pmu(struct kvm *kvm, struct arm_pmu *arm_pmu)
{
lockdep_assert_held(&kvm->arch.config_lock);
kvm->arch.arm_pmu = arm_pmu;
kvm->arch.pmcr_n = kvm_arm_pmu_get_max_counters(kvm);
}
/**
* kvm_arm_set_default_pmu - No PMU set, get the default one.
* @kvm: The kvm pointer
*
* The observant among you will notice that the supported_cpus
* mask does not get updated for the default PMU even though it
* is quite possible the selected instance supports only a
* subset of cores in the system. This is intentional, and
* upholds the preexisting behavior on heterogeneous systems
* where vCPUs can be scheduled on any core but the guest
* counters could stop working.
*/
int kvm_arm_set_default_pmu(struct kvm *kvm)
{
struct arm_pmu *arm_pmu = kvm_pmu_probe_armpmu();
if (!arm_pmu)
return -ENODEV;
kvm_arm_set_pmu(kvm, arm_pmu);
return 0;
}
static int kvm_arm_pmu_v3_set_pmu(struct kvm_vcpu *vcpu, int pmu_id)
{
struct kvm *kvm = vcpu->kvm;
......@@ -893,7 +967,7 @@ static int kvm_arm_pmu_v3_set_pmu(struct kvm_vcpu *vcpu, int pmu_id)
break;
}
kvm->arch.arm_pmu = arm_pmu;
kvm_arm_set_pmu(kvm, arm_pmu);
cpumask_copy(kvm->arch.supported_cpus, &arm_pmu->supported_cpus);
ret = 0;
break;
......@@ -916,23 +990,6 @@ int kvm_arm_pmu_v3_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
if (vcpu->arch.pmu.created)
return -EBUSY;
if (!kvm->arch.arm_pmu) {
/*
* No PMU set, get the default one.
*
* The observant among you will notice that the supported_cpus
* mask does not get updated for the default PMU even though it
* is quite possible the selected instance supports only a
* subset of cores in the system. This is intentional, and
* upholds the preexisting behavior on heterogeneous systems
* where vCPUs can be scheduled on any core but the guest
* counters could stop working.
*/
kvm->arch.arm_pmu = kvm_pmu_probe_armpmu();
if (!kvm->arch.arm_pmu)
return -ENODEV;
}
switch (attr->attr) {
case KVM_ARM_VCPU_PMU_V3_IRQ: {
int __user *uaddr = (int __user *)(long)attr->addr;
......@@ -1072,3 +1129,15 @@ u8 kvm_arm_pmu_get_pmuver_limit(void)
ID_AA64DFR0_EL1_PMUVer_V3P5);
return FIELD_GET(ARM64_FEATURE_MASK(ID_AA64DFR0_EL1_PMUVer), tmp);
}
/**
* kvm_vcpu_read_pmcr - Read PMCR_EL0 register for the vCPU
* @vcpu: The vcpu pointer
*/
u64 kvm_vcpu_read_pmcr(struct kvm_vcpu *vcpu)
{
u64 pmcr = __vcpu_sys_reg(vcpu, PMCR_EL0) &
~(ARMV8_PMU_PMCR_N_MASK << ARMV8_PMU_PMCR_N_SHIFT);
return pmcr | ((u64)vcpu->kvm->arch.pmcr_n << ARMV8_PMU_PMCR_N_SHIFT);
}
arch/arm64/kvm/reset.c
View file @ 6803bd79
......@@ -73,11 +73,8 @@ int __init kvm_arm_init_sve(void)
return 0;
}
static int kvm_vcpu_enable_sve(struct kvm_vcpu *vcpu)
static void kvm_vcpu_enable_sve(struct kvm_vcpu *vcpu)
{
if (!system_supports_sve())
return -EINVAL;
vcpu->arch.sve_max_vl = kvm_sve_max_vl;
/*
......@@ -86,8 +83,6 @@ static int kvm_vcpu_enable_sve(struct kvm_vcpu *vcpu)
* kvm_arm_vcpu_finalize(), which freezes the configuration.
*/
vcpu_set_flag(vcpu, GUEST_HAS_SVE);
return 0;
}
/*
......@@ -170,20 +165,9 @@ static void kvm_vcpu_reset_sve(struct kvm_vcpu *vcpu)
memset(vcpu->arch.sve_state, 0, vcpu_sve_state_size(vcpu));
}
static int kvm_vcpu_enable_ptrauth(struct kvm_vcpu *vcpu)
static void kvm_vcpu_enable_ptrauth(struct kvm_vcpu *vcpu)
{
/*
* For now make sure that both address/generic pointer authentication
* features are requested by the userspace together and the system
* supports these capabilities.
*/
if (!test_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, vcpu->arch.features) ||
!test_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, vcpu->arch.features) ||
!system_has_full_ptr_auth())
return -EINVAL;
vcpu_set_flag(vcpu, GUEST_HAS_PTRAUTH);
return 0;
}
/**
......@@ -204,10 +188,9 @@ static int kvm_vcpu_enable_ptrauth(struct kvm_vcpu *vcpu)
* disable preemption around the vcpu reset as we would otherwise race with
* preempt notifiers which also call put/load.
*/
int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
void kvm_reset_vcpu(struct kvm_vcpu *vcpu)
{
struct vcpu_reset_state reset_state;
int ret;
bool loaded;
u32 pstate;
......@@ -224,29 +207,16 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
if (loaded)
kvm_arch_vcpu_put(vcpu);
/* Disallow NV+SVE for the time being */
if (vcpu_has_nv(vcpu) && vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE)) {
ret = -EINVAL;
goto out;
}
if (!kvm_arm_vcpu_sve_finalized(vcpu)) {
if (test_bit(KVM_ARM_VCPU_SVE, vcpu->arch.features)) {
ret = kvm_vcpu_enable_sve(vcpu);
if (ret)
goto out;
}
if (vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE))
kvm_vcpu_enable_sve(vcpu);
} else {
kvm_vcpu_reset_sve(vcpu);
}
if (test_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, vcpu->arch.features) ||
test_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, vcpu->arch.features)) {
if (kvm_vcpu_enable_ptrauth(vcpu)) {
ret = -EINVAL;
goto out;
}
}
if (vcpu_has_feature(vcpu, KVM_ARM_VCPU_PTRAUTH_ADDRESS) ||
vcpu_has_feature(vcpu, KVM_ARM_VCPU_PTRAUTH_GENERIC))
kvm_vcpu_enable_ptrauth(vcpu);
if (vcpu_el1_is_32bit(vcpu))
pstate = VCPU_RESET_PSTATE_SVC;
......@@ -255,11 +225,6 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
else
pstate = VCPU_RESET_PSTATE_EL1;
if (kvm_vcpu_has_pmu(vcpu) && !kvm_arm_support_pmu_v3()) {
ret = -EINVAL;
goto out;
}
/* Reset core registers */
memset(vcpu_gp_regs(vcpu), 0, sizeof(*vcpu_gp_regs(vcpu)));
memset(&vcpu->arch.ctxt.fp_regs, 0, sizeof(vcpu->arch.ctxt.fp_regs));
......@@ -294,12 +259,11 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
}
/* Reset timer */
ret = kvm_timer_vcpu_reset(vcpu);
out:
kvm_timer_vcpu_reset(vcpu);
if (loaded)
kvm_arch_vcpu_load(vcpu, smp_processor_id());
preempt_enable();
return ret;
}
u32 get_kvm_ipa_limit(void)
......
arch/arm64/kvm/sys_regs.c
View file @ 6803bd79
This diff is collapsed.
arch/arm64/kvm/trace_arm.h
View file @ 6803bd79
......@@ -136,6 +136,31 @@ TRACE_EVENT(kvm_mmio_emulate,
__entry->vcpu_pc, __entry->instr, __entry->cpsr)
);
TRACE_EVENT(kvm_mmio_nisv,
TP_PROTO(unsigned long vcpu_pc, unsigned long esr,
unsigned long far, unsigned long ipa),
TP_ARGS(vcpu_pc, esr, far, ipa),
TP_STRUCT__entry(
__field( unsigned long, vcpu_pc )
__field( unsigned long, esr )
__field( unsigned long, far )
__field( unsigned long, ipa )
),
TP_fast_assign(
__entry->vcpu_pc = vcpu_pc;
__entry->esr = esr;
__entry->far = far;
__entry->ipa = ipa;
),
TP_printk("ipa %#016lx, esr %#016lx, far %#016lx, pc %#016lx",
__entry->ipa, __entry->esr,
__entry->far, __entry->vcpu_pc)
);
TRACE_EVENT(kvm_set_way_flush,
TP_PROTO(unsigned long vcpu_pc, bool cache),
TP_ARGS(vcpu_pc, cache),
......
arch/arm64/kvm/vgic/vgic-debug.c
View file @ 6803bd79
......@@ -166,7 +166,7 @@ static void print_header(struct seq_file *s, struct vgic_irq *irq,
if (vcpu) {
hdr = "VCPU";
id = vcpu->vcpu_id;
id = vcpu->vcpu_idx;
}
seq_printf(s, "\n");
......@@ -212,7 +212,7 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
" %2d "
"\n",
type, irq->intid,
(irq->target_vcpu) ? irq->target_vcpu->vcpu_id : -1,
(irq->target_vcpu) ? irq->target_vcpu->vcpu_idx : -1,
pending,
irq->line_level,
irq->active,
......@@ -224,7 +224,7 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
irq->mpidr,
irq->source,
irq->priority,
(irq->vcpu) ? irq->vcpu->vcpu_id : -1);
(irq->vcpu) ? irq->vcpu->vcpu_idx : -1);
}
static int vgic_debug_show(struct seq_file *s, void *v)
......
arch/arm64/kvm/vgic/vgic-irqfd.c
View file @ 6803bd79
......@@ -23,7 +23,7 @@ static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
if (!vgic_valid_spi(kvm, spi_id))
return -EINVAL;
return kvm_vgic_inject_irq(kvm, 0, spi_id, level, NULL);
return kvm_vgic_inject_irq(kvm, NULL, spi_id, level, NULL);
}
/**
......
arch/arm64/kvm/vgic/vgic-its.c
View file @ 6803bd79
......@@ -378,6 +378,12 @@ static int update_affinity(struct vgic_irq *irq, struct kvm_vcpu *vcpu)
return ret;
}
static struct kvm_vcpu *collection_to_vcpu(struct kvm *kvm,
struct its_collection *col)
{
return kvm_get_vcpu_by_id(kvm, col->target_addr);
}
/*
* Promotes the ITS view of affinity of an ITTE (which redistributor this LPI
* is targeting) to the VGIC's view, which deals with target VCPUs.
......@@ -391,7 +397,7 @@ static void update_affinity_ite(struct kvm *kvm, struct its_ite *ite)
if (!its_is_collection_mapped(ite->collection))
return;
vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
vcpu = collection_to_vcpu(kvm, ite->collection);
update_affinity(ite->irq, vcpu);
}
......@@ -679,7 +685,7 @@ int vgic_its_resolve_lpi(struct kvm *kvm, struct vgic_its *its,
if (!ite || !its_is_collection_mapped(ite->collection))
return E_ITS_INT_UNMAPPED_INTERRUPT;
vcpu = kvm_get_vcpu(kvm, ite->collection->target_addr);
vcpu = collection_to_vcpu(kvm, ite->collection);
if (!vcpu)
return E_ITS_INT_UNMAPPED_INTERRUPT;
......@@ -887,7 +893,7 @@ static int vgic_its_cmd_handle_movi(struct kvm *kvm, struct vgic_its *its,
return E_ITS_MOVI_UNMAPPED_COLLECTION;
ite->collection = collection;
vcpu = kvm_get_vcpu(kvm, collection->target_addr);
vcpu = collection_to_vcpu(kvm, collection);
vgic_its_invalidate_cache(kvm);
......@@ -1121,7 +1127,7 @@ static int vgic_its_cmd_handle_mapi(struct kvm *kvm, struct vgic_its *its,
}
if (its_is_collection_mapped(collection))
vcpu = kvm_get_vcpu(kvm, collection->target_addr);
vcpu = collection_to_vcpu(kvm, collection);
irq = vgic_add_lpi(kvm, lpi_nr, vcpu);
if (IS_ERR(irq)) {
......@@ -1242,21 +1248,22 @@ static int vgic_its_cmd_handle_mapc(struct kvm *kvm, struct vgic_its *its,
u64 *its_cmd)
{
u16 coll_id;
u32 target_addr;
struct its_collection *collection;
bool valid;
valid = its_cmd_get_validbit(its_cmd);
coll_id = its_cmd_get_collection(its_cmd);
target_addr = its_cmd_get_target_addr(its_cmd);
if (target_addr >= atomic_read(&kvm->online_vcpus))
return E_ITS_MAPC_PROCNUM_OOR;
if (!valid) {
vgic_its_free_collection(its, coll_id);
vgic_its_invalidate_cache(kvm);
} else {
struct kvm_vcpu *vcpu;
vcpu = kvm_get_vcpu_by_id(kvm, its_cmd_get_target_addr(its_cmd));
if (!vcpu)
return E_ITS_MAPC_PROCNUM_OOR;
collection = find_collection(its, coll_id);
if (!collection) {
......@@ -1270,9 +1277,9 @@ static int vgic_its_cmd_handle_mapc(struct kvm *kvm, struct vgic_its *its,
coll_id);
if (ret)
return ret;
collection->target_addr = target_addr;
collection->target_addr = vcpu->vcpu_id;
} else {
collection->target_addr = target_addr;
collection->target_addr = vcpu->vcpu_id;
update_affinity_collection(kvm, its, collection);
}
}
......@@ -1382,7 +1389,7 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,
if (!its_is_collection_mapped(collection))
return E_ITS_INVALL_UNMAPPED_COLLECTION;
vcpu = kvm_get_vcpu(kvm, collection->target_addr);
vcpu = collection_to_vcpu(kvm, collection);
vgic_its_invall(vcpu);
return 0;
......@@ -1399,23 +1406,21 @@ static int vgic_its_cmd_handle_invall(struct kvm *kvm, struct vgic_its *its,
static int vgic_its_cmd_handle_movall(struct kvm *kvm, struct vgic_its *its,
u64 *its_cmd)
{
u32 target1_addr = its_cmd_get_target_addr(its_cmd);
u32 target2_addr = its_cmd_mask_field(its_cmd, 3, 16, 32);
struct kvm_vcpu *vcpu1, *vcpu2;
struct vgic_irq *irq;
u32 *intids;
int irq_count, i;
if (target1_addr >= atomic_read(&kvm->online_vcpus) ||
target2_addr >= atomic_read(&kvm->online_vcpus))
/* We advertise GITS_TYPER.PTA==0, making the address the vcpu ID */
vcpu1 = kvm_get_vcpu_by_id(kvm, its_cmd_get_target_addr(its_cmd));
vcpu2 = kvm_get_vcpu_by_id(kvm, its_cmd_mask_field(its_cmd, 3, 16, 32));
if (!vcpu1 || !vcpu2)
return E_ITS_MOVALL_PROCNUM_OOR;
if (target1_addr == target2_addr)
if (vcpu1 == vcpu2)
return 0;
vcpu1 = kvm_get_vcpu(kvm, target1_addr);
vcpu2 = kvm_get_vcpu(kvm, target2_addr);
irq_count = vgic_copy_lpi_list(kvm, vcpu1, &intids);
if (irq_count < 0)
return irq_count;
......@@ -2258,7 +2263,7 @@ static int vgic_its_restore_ite(struct vgic_its *its, u32 event_id,
return PTR_ERR(ite);
if (its_is_collection_mapped(collection))
vcpu = kvm_get_vcpu(kvm, collection->target_addr);
vcpu = kvm_get_vcpu_by_id(kvm, collection->target_addr);
irq = vgic_add_lpi(kvm, lpi_id, vcpu);
if (IS_ERR(irq)) {
......@@ -2573,7 +2578,7 @@ static int vgic_its_restore_cte(struct vgic_its *its, gpa_t gpa, int esz)
coll_id = val & KVM_ITS_CTE_ICID_MASK;
if (target_addr != COLLECTION_NOT_MAPPED &&
target_addr >= atomic_read(&kvm->online_vcpus))
!kvm_get_vcpu_by_id(kvm, target_addr))
return -EINVAL;
collection = find_collection(its, coll_id);
......
arch/arm64/kvm/vgic/vgic-kvm-device.c
View file @ 6803bd79
......@@ -27,7 +27,8 @@ int vgic_check_iorange(struct kvm *kvm, phys_addr_t ioaddr,
if (addr + size < addr)
return -EINVAL;
if (addr & ~kvm_phys_mask(kvm) || addr + size > kvm_phys_size(kvm))
if (addr & ~kvm_phys_mask(&kvm->arch.mmu) ||
(addr + size) > kvm_phys_size(&kvm->arch.mmu))
return -E2BIG;
return 0;
......@@ -339,13 +340,9 @@ int vgic_v2_parse_attr(struct kvm_device *dev, struct kvm_device_attr *attr,
{
int cpuid;
cpuid = (attr->attr & KVM_DEV_ARM_VGIC_CPUID_MASK) >>
KVM_DEV_ARM_VGIC_CPUID_SHIFT;
cpuid = FIELD_GET(KVM_DEV_ARM_VGIC_CPUID_MASK, attr->attr);
if (cpuid >= atomic_read(&dev->kvm->online_vcpus))
return -EINVAL;
reg_attr->vcpu = kvm_get_vcpu(dev->kvm, cpuid);
reg_attr->vcpu = kvm_get_vcpu_by_id(dev->kvm, cpuid);
reg_attr->addr = attr->attr & KVM_DEV_ARM_VGIC_OFFSET_MASK;
return 0;
......
arch/arm64/kvm/vgic/vgic-mmio-v3.c
View file @ 6803bd79
......@@ -1013,35 +1013,6 @@ int vgic_v3_has_attr_regs(struct kvm_device *dev, struct kvm_device_attr *attr)
return 0;
}
/*
* Compare a given affinity (level 1-3 and a level 0 mask, from the SGI
* generation register ICC_SGI1R_EL1) with a given VCPU.
* If the VCPU's MPIDR matches, return the level0 affinity, otherwise
* return -1.
*/
static int match_mpidr(u64 sgi_aff, u16 sgi_cpu_mask, struct kvm_vcpu *vcpu)
{
unsigned long affinity;
int level0;
/*
* Split the current VCPU's MPIDR into affinity level 0 and the
* rest as this is what we have to compare against.
*/
affinity = kvm_vcpu_get_mpidr_aff(vcpu);
level0 = MPIDR_AFFINITY_LEVEL(affinity, 0);
affinity &= ~MPIDR_LEVEL_MASK;
/* bail out if the upper three levels don't match */
if (sgi_aff != affinity)
return -1;
/* Is this VCPU's bit set in the mask ? */
if (!(sgi_cpu_mask & BIT(level0)))
return -1;
return level0;
}
/*
* The ICC_SGI* registers encode the affinity differently from the MPIDR,
......@@ -1052,6 +1023,38 @@ static int match_mpidr(u64 sgi_aff, u16 sgi_cpu_mask, struct kvm_vcpu *vcpu)
((((reg) & ICC_SGI1R_AFFINITY_## level ##_MASK) \
>> ICC_SGI1R_AFFINITY_## level ##_SHIFT) << MPIDR_LEVEL_SHIFT(level))
static void vgic_v3_queue_sgi(struct kvm_vcpu *vcpu, u32 sgi, bool allow_group1)
{
struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, sgi);
unsigned long flags;
raw_spin_lock_irqsave(&irq->irq_lock, flags);
/*
* An access targeting Group0 SGIs can only generate
* those, while an access targeting Group1 SGIs can
* generate interrupts of either group.
*/
if (!irq->group || allow_group1) {
if (!irq->hw) {
irq->pending_latch = true;
vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
} else {
/* HW SGI? Ask the GIC to inject it */
int err;
err = irq_set_irqchip_state(irq->host_irq,
IRQCHIP_STATE_PENDING,
true);
WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
}
} else {
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
}
vgic_put_irq(vcpu->kvm, irq);
}
/**
* vgic_v3_dispatch_sgi - handle SGI requests from VCPUs
* @vcpu: The VCPU requesting a SGI
......@@ -1062,83 +1065,46 @@ static int match_mpidr(u64 sgi_aff, u16 sgi_cpu_mask, struct kvm_vcpu *vcpu)
* This will trap in sys_regs.c and call this function.
* This ICC_SGI1R_EL1 register contains the upper three affinity levels of the
* target processors as well as a bitmask of 16 Aff0 CPUs.
* If the interrupt routing mode bit is not set, we iterate over all VCPUs to
* check for matching ones. If this bit is set, we signal all, but not the
* calling VCPU.
*
* If the interrupt routing mode bit is not set, we iterate over the Aff0
* bits and signal the VCPUs matching the provided Aff{3,2,1}.
*
* If this bit is set, we signal all, but not the calling VCPU.
*/
void vgic_v3_dispatch_sgi(struct kvm_vcpu *vcpu, u64 reg, bool allow_group1)
{
struct kvm *kvm = vcpu->kvm;
struct kvm_vcpu *c_vcpu;
u16 target_cpus;
unsigned long target_cpus;
u64 mpidr;
int sgi;
int vcpu_id = vcpu->vcpu_id;
bool broadcast;
unsigned long c, flags;
sgi = (reg & ICC_SGI1R_SGI_ID_MASK) >> ICC_SGI1R_SGI_ID_SHIFT;
broadcast = reg & BIT_ULL(ICC_SGI1R_IRQ_ROUTING_MODE_BIT);
target_cpus = (reg & ICC_SGI1R_TARGET_LIST_MASK) >> ICC_SGI1R_TARGET_LIST_SHIFT;
mpidr = SGI_AFFINITY_LEVEL(reg, 3);
mpidr |= SGI_AFFINITY_LEVEL(reg, 2);
mpidr |= SGI_AFFINITY_LEVEL(reg, 1);
/*
* We iterate over all VCPUs to find the MPIDRs matching the request.
* If we have handled one CPU, we clear its bit to detect early
* if we are already finished. This avoids iterating through all
* VCPUs when most of the times we just signal a single VCPU.
*/
kvm_for_each_vcpu(c, c_vcpu, kvm) {
struct vgic_irq *irq;
/* Exit early if we have dealt with all requested CPUs */
if (!broadcast && target_cpus == 0)
break;
/* Don't signal the calling VCPU */
if (broadcast && c == vcpu_id)
continue;
u32 sgi, aff0;
unsigned long c;
if (!broadcast) {
int level0;
sgi = FIELD_GET(ICC_SGI1R_SGI_ID_MASK, reg);
level0 = match_mpidr(mpidr, target_cpus, c_vcpu);
if (level0 == -1)
/* Broadcast */
if (unlikely(reg & BIT_ULL(ICC_SGI1R_IRQ_ROUTING_MODE_BIT))) {
kvm_for_each_vcpu(c, c_vcpu, kvm) {
/* Don't signal the calling VCPU */
if (c_vcpu == vcpu)
continue;
/* remove this matching VCPU from the mask */
target_cpus &= ~BIT(level0);
vgic_v3_queue_sgi(c_vcpu, sgi, allow_group1);
}
irq = vgic_get_irq(vcpu->kvm, c_vcpu, sgi);
raw_spin_lock_irqsave(&irq->irq_lock, flags);
return;
}
/*
* An access targeting Group0 SGIs can only generate
* those, while an access targeting Group1 SGIs can
* generate interrupts of either group.
*/
if (!irq->group || allow_group1) {
if (!irq->hw) {
irq->pending_latch = true;
vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
} else {
/* HW SGI? Ask the GIC to inject it */
int err;
err = irq_set_irqchip_state(irq->host_irq,
IRQCHIP_STATE_PENDING,
true);
WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
}
} else {
raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
}
/* We iterate over affinities to find the corresponding vcpus */
mpidr = SGI_AFFINITY_LEVEL(reg, 3);
mpidr |= SGI_AFFINITY_LEVEL(reg, 2);
mpidr |= SGI_AFFINITY_LEVEL(reg, 1);
target_cpus = FIELD_GET(ICC_SGI1R_TARGET_LIST_MASK, reg);
vgic_put_irq(vcpu->kvm, irq);
for_each_set_bit(aff0, &target_cpus, hweight_long(ICC_SGI1R_TARGET_LIST_MASK)) {
c_vcpu = kvm_mpidr_to_vcpu(kvm, mpidr | aff0);
if (c_vcpu)
vgic_v3_queue_sgi(c_vcpu, sgi, allow_group1);
}
}
......
arch/arm64/kvm/vgic/vgic.c
View file @ 6803bd79
......@@ -422,7 +422,7 @@ bool vgic_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq,
/**
* kvm_vgic_inject_irq - Inject an IRQ from a device to the vgic
* @kvm: The VM structure pointer
* @cpuid: The CPU for PPIs
* @vcpu: The CPU for PPIs or NULL for global interrupts
* @intid: The INTID to inject a new state to.
* @level: Edge-triggered: true: to trigger the interrupt
* false: to ignore the call
......@@ -436,24 +436,22 @@ bool vgic_queue_irq_unlock(struct kvm *kvm, struct vgic_irq *irq,
* level-sensitive interrupts. You can think of the level parameter as 1
* being HIGH and 0 being LOW and all devices being active-HIGH.
*/
int kvm_vgic_inject_irq(struct kvm *kvm, int cpuid, unsigned int intid,
bool level, void *owner)
int kvm_vgic_inject_irq(struct kvm *kvm, struct kvm_vcpu *vcpu,
unsigned int intid, bool level, void *owner)
{
struct kvm_vcpu *vcpu;
struct vgic_irq *irq;
unsigned long flags;
int ret;
trace_vgic_update_irq_pending(cpuid, intid, level);
ret = vgic_lazy_init(kvm);
if (ret)
return ret;
vcpu = kvm_get_vcpu(kvm, cpuid);
if (!vcpu && intid < VGIC_NR_PRIVATE_IRQS)
return -EINVAL;
trace_vgic_update_irq_pending(vcpu ? vcpu->vcpu_idx : 0, intid, level);
irq = vgic_get_irq(kvm, vcpu, intid);
if (!irq)
return -EINVAL;
......
arch/arm64/kvm/vmid.c
View file @ 6803bd79
......@@ -135,10 +135,11 @@ void kvm_arm_vmid_clear_active(void)
atomic64_set(this_cpu_ptr(&active_vmids), VMID_ACTIVE_INVALID);
}
void kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid)
bool kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid)
{
unsigned long flags;
u64 vmid, old_active_vmid;
bool updated = false;
vmid = atomic64_read(&kvm_vmid->id);
......@@ -156,17 +157,21 @@ void kvm_arm_vmid_update(struct kvm_vmid *kvm_vmid)
if (old_active_vmid != 0 && vmid_gen_match(vmid) &&
0 != atomic64_cmpxchg_relaxed(this_cpu_ptr(&active_vmids),
old_active_vmid, vmid))
return;
return false;
raw_spin_lock_irqsave(&cpu_vmid_lock, flags);
/* Check that our VMID belongs to the current generation. */
vmid = atomic64_read(&kvm_vmid->id);
if (!vmid_gen_match(vmid))
if (!vmid_gen_match(vmid)) {
vmid = new_vmid(kvm_vmid);
updated = true;
}
atomic64_set(this_cpu_ptr(&active_vmids), vmid);
raw_spin_unlock_irqrestore(&cpu_vmid_lock, flags);
return updated;
}
/*
......
arch/loongarch/Kbuild
View file @ 6803bd79
......@@ -3,5 +3,7 @@ obj-y += mm/
obj-y += net/
obj-y += vdso/
obj-$(CONFIG_KVM) += kvm/
# for cleaning
subdir- += boot
arch/loongarch/Kconfig
View file @ 6803bd79
......@@ -129,6 +129,7 @@ config LOONGARCH
select HAVE_KPROBES
select HAVE_KPROBES_ON_FTRACE
select HAVE_KRETPROBES
select HAVE_KVM
select HAVE_MOD_ARCH_SPECIFIC
select HAVE_NMI
select HAVE_PCI
......@@ -263,6 +264,9 @@ config AS_HAS_LASX_EXTENSION
config AS_HAS_LBT_EXTENSION
def_bool $(as-instr,movscr2gr \$a0$(comma)\$scr0)
config AS_HAS_LVZ_EXTENSION
def_bool $(as-instr,hvcl 0)
menu "Kernel type and options"
source "kernel/Kconfig.hz"
......@@ -676,3 +680,5 @@ source "kernel/power/Kconfig"
source "drivers/acpi/Kconfig"
endmenu
source "arch/loongarch/kvm/Kconfig"
arch/loongarch/configs/loongson3_defconfig
View file @ 6803bd79
......@@ -66,6 +66,8 @@ CONFIG_EFI_ZBOOT=y
CONFIG_EFI_GENERIC_STUB_INITRD_CMDLINE_LOADER=y
CONFIG_EFI_CAPSULE_LOADER=m
CONFIG_EFI_TEST=m
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_JUMP_LABEL=y
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
......
arch/loongarch/include/asm/inst.h
View file @ 6803bd79
......@@ -65,6 +65,14 @@ enum reg2_op {
revbd_op = 0x0f,
revh2w_op = 0x10,
revhd_op = 0x11,
iocsrrdb_op = 0x19200,
iocsrrdh_op = 0x19201,
iocsrrdw_op = 0x19202,
iocsrrdd_op = 0x19203,
iocsrwrb_op = 0x19204,
iocsrwrh_op = 0x19205,
iocsrwrw_op = 0x19206,
iocsrwrd_op = 0x19207,
};
enum reg2i5_op {
......@@ -318,6 +326,13 @@ struct reg2bstrd_format {
unsigned int opcode : 10;
};
struct reg2csr_format {
unsigned int rd : 5;
unsigned int rj : 5;
unsigned int csr : 14;
unsigned int opcode : 8;
};
struct reg3_format {
unsigned int rd : 5;
unsigned int rj : 5;
......@@ -346,6 +361,7 @@ union loongarch_instruction {
struct reg2i14_format reg2i14_format;
struct reg2i16_format reg2i16_format;
struct reg2bstrd_format reg2bstrd_format;
struct reg2csr_format reg2csr_format;
struct reg3_format reg3_format;
struct reg3sa2_format reg3sa2_format;
};
......
arch/loongarch/include/asm/kvm_csr.h 0 → 100644
View file @ 6803bd79
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#ifndef __ASM_LOONGARCH_KVM_CSR_H__
#define __ASM_LOONGARCH_KVM_CSR_H__
#include <linux/uaccess.h>
#include <linux/kvm_host.h>
#include <asm/loongarch.h>
#include <asm/kvm_vcpu.h>
#define gcsr_read(csr) \
({ \
register unsigned long __v; \
__asm__ __volatile__( \
" gcsrrd %[val], %[reg]\n\t" \
: [val] "=r" (__v) \
: [reg] "i" (csr) \
: "memory"); \
__v; \
})
#define gcsr_write(v, csr) \
({ \
register unsigned long __v = v; \
__asm__ __volatile__ ( \
" gcsrwr %[val], %[reg]\n\t" \
: [val] "+r" (__v) \
: [reg] "i" (csr) \
: "memory"); \
})
#define gcsr_xchg(v, m, csr) \
({ \
register unsigned long __v = v; \
__asm__ __volatile__( \
" gcsrxchg %[val], %[mask], %[reg]\n\t" \
: [val] "+r" (__v) \
: [mask] "r" (m), [reg] "i" (csr) \
: "memory"); \
__v; \
})
/* Guest CSRS read and write */
#define read_gcsr_crmd() gcsr_read(LOONGARCH_CSR_CRMD)
#define write_gcsr_crmd(val) gcsr_write(val, LOONGARCH_CSR_CRMD)
#define read_gcsr_prmd() gcsr_read(LOONGARCH_CSR_PRMD)
#define write_gcsr_prmd(val) gcsr_write(val, LOONGARCH_CSR_PRMD)
#define read_gcsr_euen() gcsr_read(LOONGARCH_CSR_EUEN)
#define write_gcsr_euen(val) gcsr_write(val, LOONGARCH_CSR_EUEN)
#define read_gcsr_misc() gcsr_read(LOONGARCH_CSR_MISC)
#define write_gcsr_misc(val) gcsr_write(val, LOONGARCH_CSR_MISC)
#define read_gcsr_ecfg() gcsr_read(LOONGARCH_CSR_ECFG)
#define write_gcsr_ecfg(val) gcsr_write(val, LOONGARCH_CSR_ECFG)
#define read_gcsr_estat() gcsr_read(LOONGARCH_CSR_ESTAT)
#define write_gcsr_estat(val) gcsr_write(val, LOONGARCH_CSR_ESTAT)
#define read_gcsr_era() gcsr_read(LOONGARCH_CSR_ERA)
#define write_gcsr_era(val) gcsr_write(val, LOONGARCH_CSR_ERA)
#define read_gcsr_badv() gcsr_read(LOONGARCH_CSR_BADV)
#define write_gcsr_badv(val) gcsr_write(val, LOONGARCH_CSR_BADV)
#define read_gcsr_badi() gcsr_read(LOONGARCH_CSR_BADI)
#define write_gcsr_badi(val) gcsr_write(val, LOONGARCH_CSR_BADI)
#define read_gcsr_eentry() gcsr_read(LOONGARCH_CSR_EENTRY)
#define write_gcsr_eentry(val) gcsr_write(val, LOONGARCH_CSR_EENTRY)
#define read_gcsr_asid() gcsr_read(LOONGARCH_CSR_ASID)
#define write_gcsr_asid(val) gcsr_write(val, LOONGARCH_CSR_ASID)
#define read_gcsr_pgdl() gcsr_read(LOONGARCH_CSR_PGDL)
#define write_gcsr_pgdl(val) gcsr_write(val, LOONGARCH_CSR_PGDL)
#define read_gcsr_pgdh() gcsr_read(LOONGARCH_CSR_PGDH)
#define write_gcsr_pgdh(val) gcsr_write(val, LOONGARCH_CSR_PGDH)
#define write_gcsr_pgd(val) gcsr_write(val, LOONGARCH_CSR_PGD)
#define read_gcsr_pgd() gcsr_read(LOONGARCH_CSR_PGD)
#define read_gcsr_pwctl0() gcsr_read(LOONGARCH_CSR_PWCTL0)
#define write_gcsr_pwctl0(val) gcsr_write(val, LOONGARCH_CSR_PWCTL0)
#define read_gcsr_pwctl1() gcsr_read(LOONGARCH_CSR_PWCTL1)
#define write_gcsr_pwctl1(val) gcsr_write(val, LOONGARCH_CSR_PWCTL1)
#define read_gcsr_stlbpgsize() gcsr_read(LOONGARCH_CSR_STLBPGSIZE)
#define write_gcsr_stlbpgsize(val) gcsr_write(val, LOONGARCH_CSR_STLBPGSIZE)
#define read_gcsr_rvacfg() gcsr_read(LOONGARCH_CSR_RVACFG)
#define write_gcsr_rvacfg(val) gcsr_write(val, LOONGARCH_CSR_RVACFG)
#define read_gcsr_cpuid() gcsr_read(LOONGARCH_CSR_CPUID)
#define write_gcsr_cpuid(val) gcsr_write(val, LOONGARCH_CSR_CPUID)
#define read_gcsr_prcfg1() gcsr_read(LOONGARCH_CSR_PRCFG1)
#define write_gcsr_prcfg1(val) gcsr_write(val, LOONGARCH_CSR_PRCFG1)
#define read_gcsr_prcfg2() gcsr_read(LOONGARCH_CSR_PRCFG2)
#define write_gcsr_prcfg2(val) gcsr_write(val, LOONGARCH_CSR_PRCFG2)
#define read_gcsr_prcfg3() gcsr_read(LOONGARCH_CSR_PRCFG3)
#define write_gcsr_prcfg3(val) gcsr_write(val, LOONGARCH_CSR_PRCFG3)
#define read_gcsr_kscratch0() gcsr_read(LOONGARCH_CSR_KS0)
#define write_gcsr_kscratch0(val) gcsr_write(val, LOONGARCH_CSR_KS0)
#define read_gcsr_kscratch1() gcsr_read(LOONGARCH_CSR_KS1)
#define write_gcsr_kscratch1(val) gcsr_write(val, LOONGARCH_CSR_KS1)
#define read_gcsr_kscratch2() gcsr_read(LOONGARCH_CSR_KS2)
#define write_gcsr_kscratch2(val) gcsr_write(val, LOONGARCH_CSR_KS2)
#define read_gcsr_kscratch3() gcsr_read(LOONGARCH_CSR_KS3)
#define write_gcsr_kscratch3(val) gcsr_write(val, LOONGARCH_CSR_KS3)
#define read_gcsr_kscratch4() gcsr_read(LOONGARCH_CSR_KS4)
#define write_gcsr_kscratch4(val) gcsr_write(val, LOONGARCH_CSR_KS4)
#define read_gcsr_kscratch5() gcsr_read(LOONGARCH_CSR_KS5)
#define write_gcsr_kscratch5(val) gcsr_write(val, LOONGARCH_CSR_KS5)
#define read_gcsr_kscratch6() gcsr_read(LOONGARCH_CSR_KS6)
#define write_gcsr_kscratch6(val) gcsr_write(val, LOONGARCH_CSR_KS6)
#define read_gcsr_kscratch7() gcsr_read(LOONGARCH_CSR_KS7)
#define write_gcsr_kscratch7(val) gcsr_write(val, LOONGARCH_CSR_KS7)
#define read_gcsr_timerid() gcsr_read(LOONGARCH_CSR_TMID)
#define write_gcsr_timerid(val) gcsr_write(val, LOONGARCH_CSR_TMID)
#define read_gcsr_timercfg() gcsr_read(LOONGARCH_CSR_TCFG)
#define write_gcsr_timercfg(val) gcsr_write(val, LOONGARCH_CSR_TCFG)
#define read_gcsr_timertick() gcsr_read(LOONGARCH_CSR_TVAL)
#define write_gcsr_timertick(val) gcsr_write(val, LOONGARCH_CSR_TVAL)
#define read_gcsr_timeroffset() gcsr_read(LOONGARCH_CSR_CNTC)
#define write_gcsr_timeroffset(val) gcsr_write(val, LOONGARCH_CSR_CNTC)
#define read_gcsr_llbctl() gcsr_read(LOONGARCH_CSR_LLBCTL)
#define write_gcsr_llbctl(val) gcsr_write(val, LOONGARCH_CSR_LLBCTL)
#define read_gcsr_tlbidx() gcsr_read(LOONGARCH_CSR_TLBIDX)
#define write_gcsr_tlbidx(val) gcsr_write(val, LOONGARCH_CSR_TLBIDX)
#define read_gcsr_tlbrentry() gcsr_read(LOONGARCH_CSR_TLBRENTRY)
#define write_gcsr_tlbrentry(val) gcsr_write(val, LOONGARCH_CSR_TLBRENTRY)
#define read_gcsr_tlbrbadv() gcsr_read(LOONGARCH_CSR_TLBRBADV)
#define write_gcsr_tlbrbadv(val) gcsr_write(val, LOONGARCH_CSR_TLBRBADV)
#define read_gcsr_tlbrera() gcsr_read(LOONGARCH_CSR_TLBRERA)
#define write_gcsr_tlbrera(val) gcsr_write(val, LOONGARCH_CSR_TLBRERA)
#define read_gcsr_tlbrsave() gcsr_read(LOONGARCH_CSR_TLBRSAVE)
#define write_gcsr_tlbrsave(val) gcsr_write(val, LOONGARCH_CSR_TLBRSAVE)
#define read_gcsr_tlbrelo0() gcsr_read(LOONGARCH_CSR_TLBRELO0)
#define write_gcsr_tlbrelo0(val) gcsr_write(val, LOONGARCH_CSR_TLBRELO0)
#define read_gcsr_tlbrelo1() gcsr_read(LOONGARCH_CSR_TLBRELO1)
#define write_gcsr_tlbrelo1(val) gcsr_write(val, LOONGARCH_CSR_TLBRELO1)
#define read_gcsr_tlbrehi() gcsr_read(LOONGARCH_CSR_TLBREHI)
#define write_gcsr_tlbrehi(val) gcsr_write(val, LOONGARCH_CSR_TLBREHI)
#define read_gcsr_tlbrprmd() gcsr_read(LOONGARCH_CSR_TLBRPRMD)
#define write_gcsr_tlbrprmd(val) gcsr_write(val, LOONGARCH_CSR_TLBRPRMD)
#define read_gcsr_directwin0() gcsr_read(LOONGARCH_CSR_DMWIN0)
#define write_gcsr_directwin0(val) gcsr_write(val, LOONGARCH_CSR_DMWIN0)
#define read_gcsr_directwin1() gcsr_read(LOONGARCH_CSR_DMWIN1)
#define write_gcsr_directwin1(val) gcsr_write(val, LOONGARCH_CSR_DMWIN1)
#define read_gcsr_directwin2() gcsr_read(LOONGARCH_CSR_DMWIN2)
#define write_gcsr_directwin2(val) gcsr_write(val, LOONGARCH_CSR_DMWIN2)
#define read_gcsr_directwin3() gcsr_read(LOONGARCH_CSR_DMWIN3)
#define write_gcsr_directwin3(val) gcsr_write(val, LOONGARCH_CSR_DMWIN3)
/* Guest related CSRs */
#define read_csr_gtlbc() csr_read64(LOONGARCH_CSR_GTLBC)
#define write_csr_gtlbc(val) csr_write64(val, LOONGARCH_CSR_GTLBC)
#define read_csr_trgp() csr_read64(LOONGARCH_CSR_TRGP)
#define read_csr_gcfg() csr_read64(LOONGARCH_CSR_GCFG)
#define write_csr_gcfg(val) csr_write64(val, LOONGARCH_CSR_GCFG)
#define read_csr_gstat() csr_read64(LOONGARCH_CSR_GSTAT)
#define write_csr_gstat(val) csr_write64(val, LOONGARCH_CSR_GSTAT)
#define read_csr_gintc() csr_read64(LOONGARCH_CSR_GINTC)
#define write_csr_gintc(val) csr_write64(val, LOONGARCH_CSR_GINTC)
#define read_csr_gcntc() csr_read64(LOONGARCH_CSR_GCNTC)
#define write_csr_gcntc(val) csr_write64(val, LOONGARCH_CSR_GCNTC)
#define __BUILD_GCSR_OP(name) __BUILD_CSR_COMMON(gcsr_##name)
__BUILD_CSR_OP(gcfg)
__BUILD_CSR_OP(gstat)
__BUILD_CSR_OP(gtlbc)
__BUILD_CSR_OP(gintc)
__BUILD_GCSR_OP(llbctl)
__BUILD_GCSR_OP(tlbidx)
#define set_gcsr_estat(val) \
gcsr_xchg(val, val, LOONGARCH_CSR_ESTAT)
#define clear_gcsr_estat(val) \
gcsr_xchg(~(val), val, LOONGARCH_CSR_ESTAT)
#define kvm_read_hw_gcsr(id) gcsr_read(id)
#define kvm_write_hw_gcsr(id, val) gcsr_write(val, id)
#define kvm_save_hw_gcsr(csr, gid) (csr->csrs[gid] = gcsr_read(gid))
#define kvm_restore_hw_gcsr(csr, gid) (gcsr_write(csr->csrs[gid], gid))
int kvm_emu_iocsr(larch_inst inst, struct kvm_run *run, struct kvm_vcpu *vcpu);
static __always_inline unsigned long kvm_read_sw_gcsr(struct loongarch_csrs *csr, int gid)
{
return csr->csrs[gid];
}
static __always_inline void kvm_write_sw_gcsr(struct loongarch_csrs *csr, int gid, unsigned long val)
{
csr->csrs[gid] = val;
}
static __always_inline void kvm_set_sw_gcsr(struct loongarch_csrs *csr,
int gid, unsigned long val)
{
csr->csrs[gid] |= val;
}
static __always_inline void kvm_change_sw_gcsr(struct loongarch_csrs *csr,
int gid, unsigned long mask, unsigned long val)
{
unsigned long _mask = mask;
csr->csrs[gid] &= ~_mask;
csr->csrs[gid] |= val & _mask;
}
#endif /* __ASM_LOONGARCH_KVM_CSR_H__ */
arch/loongarch/include/asm/kvm_host.h 0 → 100644
View file @ 6803bd79
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#ifndef __ASM_LOONGARCH_KVM_HOST_H__
#define __ASM_LOONGARCH_KVM_HOST_H__
#include <linux/cpumask.h>
#include <linux/hrtimer.h>
#include <linux/interrupt.h>
#include <linux/kvm.h>
#include <linux/kvm_types.h>
#include <linux/mutex.h>
#include <linux/spinlock.h>
#include <linux/threads.h>
#include <linux/types.h>
#include <asm/inst.h>
#include <asm/kvm_mmu.h>
#include <asm/loongarch.h>
/* Loongarch KVM register ids */
#define KVM_GET_IOC_CSR_IDX(id) ((id & KVM_CSR_IDX_MASK) >> LOONGARCH_REG_SHIFT)
#define KVM_GET_IOC_CPUCFG_IDX(id) ((id & KVM_CPUCFG_IDX_MASK) >> LOONGARCH_REG_SHIFT)
#define KVM_MAX_VCPUS 256
#define KVM_MAX_CPUCFG_REGS 21
/* memory slots that does not exposed to userspace */
#define KVM_PRIVATE_MEM_SLOTS 0
#define KVM_HALT_POLL_NS_DEFAULT 500000
struct kvm_vm_stat {
struct kvm_vm_stat_generic generic;
u64 pages;
u64 hugepages;
};
struct kvm_vcpu_stat {
struct kvm_vcpu_stat_generic generic;
u64 int_exits;
u64 idle_exits;
u64 cpucfg_exits;
u64 signal_exits;
};
struct kvm_arch_memory_slot {
};
struct kvm_context {
unsigned long vpid_cache;
struct kvm_vcpu *last_vcpu;
};
struct kvm_world_switch {
int (*exc_entry)(void);
int (*enter_guest)(struct kvm_run *run, struct kvm_vcpu *vcpu);
unsigned long page_order;
};
#define MAX_PGTABLE_LEVELS 4
struct kvm_arch {
/* Guest physical mm */
kvm_pte_t *pgd;
unsigned long gpa_size;
unsigned long invalid_ptes[MAX_PGTABLE_LEVELS];
unsigned int pte_shifts[MAX_PGTABLE_LEVELS];
unsigned int root_level;
s64 time_offset;
struct kvm_context __percpu *vmcs;
};
#define CSR_MAX_NUMS 0x800
struct loongarch_csrs {
unsigned long csrs[CSR_MAX_NUMS];
};
/* Resume Flags */
#define RESUME_HOST 0
#define RESUME_GUEST 1
enum emulation_result {
EMULATE_DONE, /* no further processing */
EMULATE_DO_MMIO, /* kvm_run filled with MMIO request */
EMULATE_DO_IOCSR, /* handle IOCSR request */
EMULATE_FAIL, /* can't emulate this instruction */
EMULATE_EXCEPT, /* A guest exception has been generated */
};
#define KVM_LARCH_FPU (0x1 << 0)
#define KVM_LARCH_SWCSR_LATEST (0x1 << 1)
#define KVM_LARCH_HWCSR_USABLE (0x1 << 2)
struct kvm_vcpu_arch {
/*
* Switch pointer-to-function type to unsigned long
* for loading the value into register directly.
*/
unsigned long host_eentry;
unsigned long guest_eentry;
/* Pointers stored here for easy accessing from assembly code */
int (*handle_exit)(struct kvm_run *run, struct kvm_vcpu *vcpu);
/* Host registers preserved across guest mode execution */
unsigned long host_sp;
unsigned long host_tp;
unsigned long host_pgd;
/* Host CSRs are used when handling exits from guest */
unsigned long badi;
unsigned long badv;
unsigned long host_ecfg;
unsigned long host_estat;
unsigned long host_percpu;
/* GPRs */
unsigned long gprs[32];
unsigned long pc;
/* Which auxiliary state is loaded (KVM_LARCH_*) */
unsigned int aux_inuse;
/* FPU state */
struct loongarch_fpu fpu FPU_ALIGN;
/* CSR state */
struct loongarch_csrs *csr;
/* GPR used as IO source/target */
u32 io_gpr;
/* KVM register to control count timer */
u32 count_ctl;
struct hrtimer swtimer;
/* Bitmask of intr that are pending */
unsigned long irq_pending;
/* Bitmask of pending intr to be cleared */
unsigned long irq_clear;
/* Bitmask of exceptions that are pending */
unsigned long exception_pending;
unsigned int esubcode;
/* Cache for pages needed inside spinlock regions */
struct kvm_mmu_memory_cache mmu_page_cache;
/* vcpu's vpid */
u64 vpid;
/* Frequency of stable timer in Hz */
u64 timer_mhz;
ktime_t expire;
/* Last CPU the vCPU state was loaded on */
int last_sched_cpu;
/* mp state */
struct kvm_mp_state mp_state;
/* cpucfg */
u32 cpucfg[KVM_MAX_CPUCFG_REGS];
};
static inline unsigned long readl_sw_gcsr(struct loongarch_csrs *csr, int reg)
{
return csr->csrs[reg];
}
static inline void writel_sw_gcsr(struct loongarch_csrs *csr, int reg, unsigned long val)
{
csr->csrs[reg] = val;
}
/* Debug: dump vcpu state */
int kvm_arch_vcpu_dump_regs(struct kvm_vcpu *vcpu);
/* MMU handling */
void kvm_flush_tlb_all(void);
void kvm_flush_tlb_gpa(struct kvm_vcpu *vcpu, unsigned long gpa);
int kvm_handle_mm_fault(struct kvm_vcpu *vcpu, unsigned long badv, bool write);
#define KVM_ARCH_WANT_MMU_NOTIFIER
void kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte);
int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end, bool blockable);
int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end);
int kvm_test_age_hva(struct kvm *kvm, unsigned long hva);
static inline void update_pc(struct kvm_vcpu_arch *arch)
{
arch->pc += 4;
}
/*
* kvm_is_ifetch_fault() - Find whether a TLBL exception is due to ifetch fault.
* @vcpu: Virtual CPU.
*
* Returns: Whether the TLBL exception was likely due to an instruction
* fetch fault rather than a data load fault.
*/
static inline bool kvm_is_ifetch_fault(struct kvm_vcpu_arch *arch)
{
return arch->pc == arch->badv;
}
/* Misc */
static inline void kvm_arch_hardware_unsetup(void) {}
static inline void kvm_arch_sync_events(struct kvm *kvm) {}
static inline void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) {}
static inline void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu) {}
static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
static inline void kvm_arch_vcpu_block_finish(struct kvm_vcpu *vcpu) {}
static inline void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) {}
void kvm_check_vpid(struct kvm_vcpu *vcpu);
enum hrtimer_restart kvm_swtimer_wakeup(struct hrtimer *timer);
void kvm_arch_flush_remote_tlbs_memslot(struct kvm *kvm, const struct kvm_memory_slot *memslot);
void kvm_init_vmcs(struct kvm *kvm);
void kvm_exc_entry(void);
int kvm_enter_guest(struct kvm_run *run, struct kvm_vcpu *vcpu);
extern unsigned long vpid_mask;
extern const unsigned long kvm_exception_size;
extern const unsigned long kvm_enter_guest_size;
extern struct kvm_world_switch *kvm_loongarch_ops;
#define SW_GCSR (1 << 0)
#define HW_GCSR (1 << 1)
#define INVALID_GCSR (1 << 2)
int get_gcsr_flag(int csr);
void set_hw_gcsr(int csr_id, unsigned long val);
#endif /* __ASM_LOONGARCH_KVM_HOST_H__ */
arch/loongarch/include/asm/kvm_mmu.h 0 → 100644
View file @ 6803bd79
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#ifndef __ASM_LOONGARCH_KVM_MMU_H__
#define __ASM_LOONGARCH_KVM_MMU_H__
#include <linux/kvm_host.h>
#include <asm/pgalloc.h>
#include <asm/tlb.h>
/*
* KVM_MMU_CACHE_MIN_PAGES is the number of GPA page table translation levels
* for which pages need to be cached.
*/
#define KVM_MMU_CACHE_MIN_PAGES (CONFIG_PGTABLE_LEVELS - 1)
#define _KVM_FLUSH_PGTABLE 0x1
#define _KVM_HAS_PGMASK 0x2
#define kvm_pfn_pte(pfn, prot) (((pfn) << PFN_PTE_SHIFT) | pgprot_val(prot))
#define kvm_pte_pfn(x) ((phys_addr_t)((x & _PFN_MASK) >> PFN_PTE_SHIFT))
typedef unsigned long kvm_pte_t;
typedef struct kvm_ptw_ctx kvm_ptw_ctx;
typedef int (*kvm_pte_ops)(kvm_pte_t *pte, phys_addr_t addr, kvm_ptw_ctx *ctx);
struct kvm_ptw_ctx {
kvm_pte_ops ops;
unsigned long flag;
/* for kvm_arch_mmu_enable_log_dirty_pt_masked use */
unsigned long mask;
unsigned long gfn;
/* page walk mmu info */
unsigned int level;
unsigned long pgtable_shift;
unsigned long invalid_entry;
unsigned long *invalid_ptes;
unsigned int *pte_shifts;
void *opaque;
/* free pte table page list */
struct list_head list;
};
kvm_pte_t *kvm_pgd_alloc(void);
static inline void kvm_set_pte(kvm_pte_t *ptep, kvm_pte_t val)
{
WRITE_ONCE(*ptep, val);
}
static inline int kvm_pte_write(kvm_pte_t pte) { return pte & _PAGE_WRITE; }
static inline int kvm_pte_dirty(kvm_pte_t pte) { return pte & _PAGE_DIRTY; }
static inline int kvm_pte_young(kvm_pte_t pte) { return pte & _PAGE_ACCESSED; }
static inline int kvm_pte_huge(kvm_pte_t pte) { return pte & _PAGE_HUGE; }
static inline kvm_pte_t kvm_pte_mkyoung(kvm_pte_t pte)
{
return pte | _PAGE_ACCESSED;
}
static inline kvm_pte_t kvm_pte_mkold(kvm_pte_t pte)
{
return pte & ~_PAGE_ACCESSED;
}
static inline kvm_pte_t kvm_pte_mkdirty(kvm_pte_t pte)
{
return pte | _PAGE_DIRTY;
}
static inline kvm_pte_t kvm_pte_mkclean(kvm_pte_t pte)
{
return pte & ~_PAGE_DIRTY;
}
static inline kvm_pte_t kvm_pte_mkhuge(kvm_pte_t pte)
{
return pte | _PAGE_HUGE;
}
static inline kvm_pte_t kvm_pte_mksmall(kvm_pte_t pte)
{
return pte & ~_PAGE_HUGE;
}
static inline int kvm_need_flush(kvm_ptw_ctx *ctx)
{
return ctx->flag & _KVM_FLUSH_PGTABLE;
}
static inline kvm_pte_t *kvm_pgtable_offset(kvm_ptw_ctx *ctx, kvm_pte_t *table,
phys_addr_t addr)
{
return table + ((addr >> ctx->pgtable_shift) & (PTRS_PER_PTE - 1));
}
static inline phys_addr_t kvm_pgtable_addr_end(kvm_ptw_ctx *ctx,
phys_addr_t addr, phys_addr_t end)
{
phys_addr_t boundary, size;
size = 0x1UL << ctx->pgtable_shift;
boundary = (addr + size) & ~(size - 1);
return (boundary - 1 < end - 1) ? boundary : end;
}
static inline int kvm_pte_present(kvm_ptw_ctx *ctx, kvm_pte_t *entry)
{
if (!ctx || ctx->level == 0)
return !!(*entry & _PAGE_PRESENT);
return *entry != ctx->invalid_entry;
}
static inline int kvm_pte_none(kvm_ptw_ctx *ctx, kvm_pte_t *entry)
{
return *entry == ctx->invalid_entry;
}
static inline void kvm_ptw_enter(kvm_ptw_ctx *ctx)
{
ctx->level--;
ctx->pgtable_shift = ctx->pte_shifts[ctx->level];
ctx->invalid_entry = ctx->invalid_ptes[ctx->level];
}
static inline void kvm_ptw_exit(kvm_ptw_ctx *ctx)
{
ctx->level++;
ctx->pgtable_shift = ctx->pte_shifts[ctx->level];
ctx->invalid_entry = ctx->invalid_ptes[ctx->level];
}
#endif /* __ASM_LOONGARCH_KVM_MMU_H__ */
arch/loongarch/include/asm/kvm_types.h 0 → 100644
View file @ 6803bd79
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#ifndef _ASM_LOONGARCH_KVM_TYPES_H
#define _ASM_LOONGARCH_KVM_TYPES_H
#define KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE 40
#endif /* _ASM_LOONGARCH_KVM_TYPES_H */
arch/loongarch/include/asm/kvm_vcpu.h 0 → 100644
View file @ 6803bd79
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#ifndef __ASM_LOONGARCH_KVM_VCPU_H__
#define __ASM_LOONGARCH_KVM_VCPU_H__
#include <linux/kvm_host.h>
#include <asm/loongarch.h>
/* Controlled by 0x5 guest estat */
#define CPU_SIP0 (_ULCAST_(1))
#define CPU_SIP1 (_ULCAST_(1) << 1)
#define CPU_PMU (_ULCAST_(1) << 10)
#define CPU_TIMER (_ULCAST_(1) << 11)
#define CPU_IPI (_ULCAST_(1) << 12)
/* Controlled by 0x52 guest exception VIP aligned to estat bit 5~12 */
#define CPU_IP0 (_ULCAST_(1))
#define CPU_IP1 (_ULCAST_(1) << 1)
#define CPU_IP2 (_ULCAST_(1) << 2)
#define CPU_IP3 (_ULCAST_(1) << 3)
#define CPU_IP4 (_ULCAST_(1) << 4)
#define CPU_IP5 (_ULCAST_(1) << 5)
#define CPU_IP6 (_ULCAST_(1) << 6)
#define CPU_IP7 (_ULCAST_(1) << 7)
#define MNSEC_PER_SEC (NSEC_PER_SEC >> 20)
/* KVM_IRQ_LINE irq field index values */
#define KVM_LOONGSON_IRQ_TYPE_SHIFT 24
#define KVM_LOONGSON_IRQ_TYPE_MASK 0xff
#define KVM_LOONGSON_IRQ_VCPU_SHIFT 16
#define KVM_LOONGSON_IRQ_VCPU_MASK 0xff
#define KVM_LOONGSON_IRQ_NUM_SHIFT 0
#define KVM_LOONGSON_IRQ_NUM_MASK 0xffff
typedef union loongarch_instruction larch_inst;
typedef int (*exit_handle_fn)(struct kvm_vcpu *);
int kvm_emu_mmio_read(struct kvm_vcpu *vcpu, larch_inst inst);
int kvm_emu_mmio_write(struct kvm_vcpu *vcpu, larch_inst inst);
int kvm_complete_mmio_read(struct kvm_vcpu *vcpu, struct kvm_run *run);
int kvm_complete_iocsr_read(struct kvm_vcpu *vcpu, struct kvm_run *run);
int kvm_emu_idle(struct kvm_vcpu *vcpu);
int kvm_pending_timer(struct kvm_vcpu *vcpu);
int kvm_handle_fault(struct kvm_vcpu *vcpu, int fault);
void kvm_deliver_intr(struct kvm_vcpu *vcpu);
void kvm_deliver_exception(struct kvm_vcpu *vcpu);
void kvm_own_fpu(struct kvm_vcpu *vcpu);
void kvm_lose_fpu(struct kvm_vcpu *vcpu);
void kvm_save_fpu(struct loongarch_fpu *fpu);
void kvm_restore_fpu(struct loongarch_fpu *fpu);
void kvm_restore_fcsr(struct loongarch_fpu *fpu);
void kvm_acquire_timer(struct kvm_vcpu *vcpu);
void kvm_init_timer(struct kvm_vcpu *vcpu, unsigned long hz);
void kvm_reset_timer(struct kvm_vcpu *vcpu);
void kvm_save_timer(struct kvm_vcpu *vcpu);
void kvm_restore_timer(struct kvm_vcpu *vcpu);
int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq);
/*
* Loongarch KVM guest interrupt handling
*/
static inline void kvm_queue_irq(struct kvm_vcpu *vcpu, unsigned int irq)
{
set_bit(irq, &vcpu->arch.irq_pending);
clear_bit(irq, &vcpu->arch.irq_clear);
}
static inline void kvm_dequeue_irq(struct kvm_vcpu *vcpu, unsigned int irq)
{
clear_bit(irq, &vcpu->arch.irq_pending);
set_bit(irq, &vcpu->arch.irq_clear);
}
static inline int kvm_queue_exception(struct kvm_vcpu *vcpu,
unsigned int code, unsigned int subcode)
{
/* only one exception can be injected */
if (!vcpu->arch.exception_pending) {
set_bit(code, &vcpu->arch.exception_pending);
vcpu->arch.esubcode = subcode;
return 0;
} else
return -1;
}
#endif /* __ASM_LOONGARCH_KVM_VCPU_H__ */
arch/loongarch/include/asm/loongarch.h
View file @ 6803bd79
......@@ -226,6 +226,7 @@
#define LOONGARCH_CSR_ECFG 0x4 /* Exception config */
#define CSR_ECFG_VS_SHIFT 16
#define CSR_ECFG_VS_WIDTH 3
#define CSR_ECFG_VS_SHIFT_END (CSR_ECFG_VS_SHIFT + CSR_ECFG_VS_WIDTH - 1)
#define CSR_ECFG_VS (_ULCAST_(0x7) << CSR_ECFG_VS_SHIFT)
#define CSR_ECFG_IM_SHIFT 0
#define CSR_ECFG_IM_WIDTH 14
......@@ -314,13 +315,14 @@
#define CSR_TLBLO1_V (_ULCAST_(0x1) << CSR_TLBLO1_V_SHIFT)
#define LOONGARCH_CSR_GTLBC 0x15 /* Guest TLB control */
#define CSR_GTLBC_RID_SHIFT 16
#define CSR_GTLBC_RID_WIDTH 8
#define CSR_GTLBC_RID (_ULCAST_(0xff) << CSR_GTLBC_RID_SHIFT)
#define CSR_GTLBC_TGID_SHIFT 16
#define CSR_GTLBC_TGID_WIDTH 8
#define CSR_GTLBC_TGID_SHIFT_END (CSR_GTLBC_TGID_SHIFT + CSR_GTLBC_TGID_WIDTH - 1)
#define CSR_GTLBC_TGID (_ULCAST_(0xff) << CSR_GTLBC_TGID_SHIFT)
#define CSR_GTLBC_TOTI_SHIFT 13
#define CSR_GTLBC_TOTI (_ULCAST_(0x1) << CSR_GTLBC_TOTI_SHIFT)
#define CSR_GTLBC_USERID_SHIFT 12
#define CSR_GTLBC_USERID (_ULCAST_(0x1) << CSR_GTLBC_USERID_SHIFT)
#define CSR_GTLBC_USETGID_SHIFT 12
#define CSR_GTLBC_USETGID (_ULCAST_(0x1) << CSR_GTLBC_USETGID_SHIFT)
#define CSR_GTLBC_GMTLBSZ_SHIFT 0
#define CSR_GTLBC_GMTLBSZ_WIDTH 6
#define CSR_GTLBC_GMTLBSZ (_ULCAST_(0x3f) << CSR_GTLBC_GMTLBSZ_SHIFT)
......@@ -475,6 +477,7 @@
#define LOONGARCH_CSR_GSTAT 0x50 /* Guest status */
#define CSR_GSTAT_GID_SHIFT 16
#define CSR_GSTAT_GID_WIDTH 8
#define CSR_GSTAT_GID_SHIFT_END (CSR_GSTAT_GID_SHIFT + CSR_GSTAT_GID_WIDTH - 1)
#define CSR_GSTAT_GID (_ULCAST_(0xff) << CSR_GSTAT_GID_SHIFT)
#define CSR_GSTAT_GIDBIT_SHIFT 4
#define CSR_GSTAT_GIDBIT_WIDTH 6
......@@ -525,6 +528,12 @@
#define CSR_GCFG_MATC_GUEST (_ULCAST_(0x0) << CSR_GCFG_MATC_SHITF)
#define CSR_GCFG_MATC_ROOT (_ULCAST_(0x1) << CSR_GCFG_MATC_SHITF)
#define CSR_GCFG_MATC_NEST (_ULCAST_(0x2) << CSR_GCFG_MATC_SHITF)
#define CSR_GCFG_MATP_NEST_SHIFT 2
#define CSR_GCFG_MATP_NEST (_ULCAST_(0x1) << CSR_GCFG_MATP_NEST_SHIFT)
#define CSR_GCFG_MATP_ROOT_SHIFT 1
#define CSR_GCFG_MATP_ROOT (_ULCAST_(0x1) << CSR_GCFG_MATP_ROOT_SHIFT)
#define CSR_GCFG_MATP_GUEST_SHIFT 0
#define CSR_GCFG_MATP_GUEST (_ULCAST_(0x1) << CSR_GCFG_MATP_GUEST_SHIFT)
#define LOONGARCH_CSR_GINTC 0x52 /* Guest interrupt control */
#define CSR_GINTC_HC_SHIFT 16
......
arch/loongarch/include/uapi/asm/kvm.h 0 → 100644
View file @ 6803bd79
/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#ifndef __UAPI_ASM_LOONGARCH_KVM_H
#define __UAPI_ASM_LOONGARCH_KVM_H
#include <linux/types.h>
/*
* KVM LoongArch specific structures and definitions.
*
* Some parts derived from the x86 version of this file.
*/
#define __KVM_HAVE_READONLY_MEM
#define KVM_COALESCED_MMIO_PAGE_OFFSET 1
#define KVM_DIRTY_LOG_PAGE_OFFSET 64
/*
* for KVM_GET_REGS and KVM_SET_REGS
*/
struct kvm_regs {
/* out (KVM_GET_REGS) / in (KVM_SET_REGS) */
__u64 gpr[32];
__u64 pc;
};
/*
* for KVM_GET_FPU and KVM_SET_FPU
*/
struct kvm_fpu {
__u32 fcsr;
__u64 fcc; /* 8x8 */
struct kvm_fpureg {
__u64 val64[4];
} fpr[32];
};
/*
* For LoongArch, we use KVM_SET_ONE_REG and KVM_GET_ONE_REG to access various
* registers. The id field is broken down as follows:
*
* bits[63..52] - As per linux/kvm.h
* bits[51..32] - Must be zero.
* bits[31..16] - Register set.
*
* Register set = 0: GP registers from kvm_regs (see definitions below).
*
* Register set = 1: CSR registers.
*
* Register set = 2: KVM specific registers (see definitions below).
*
* Register set = 3: FPU / SIMD registers (see definitions below).
*
* Other sets registers may be added in the future. Each set would
* have its own identifier in bits[31..16].
*/
#define KVM_REG_LOONGARCH_GPR (KVM_REG_LOONGARCH | 0x00000ULL)
#define KVM_REG_LOONGARCH_CSR (KVM_REG_LOONGARCH | 0x10000ULL)
#define KVM_REG_LOONGARCH_KVM (KVM_REG_LOONGARCH | 0x20000ULL)
#define KVM_REG_LOONGARCH_FPSIMD (KVM_REG_LOONGARCH | 0x30000ULL)
#define KVM_REG_LOONGARCH_CPUCFG (KVM_REG_LOONGARCH | 0x40000ULL)
#define KVM_REG_LOONGARCH_MASK (KVM_REG_LOONGARCH | 0x70000ULL)
#define KVM_CSR_IDX_MASK 0x7fff
#define KVM_CPUCFG_IDX_MASK 0x7fff
/*
* KVM_REG_LOONGARCH_KVM - KVM specific control registers.
*/
#define KVM_REG_LOONGARCH_COUNTER (KVM_REG_LOONGARCH_KVM | KVM_REG_SIZE_U64 | 1)
#define KVM_REG_LOONGARCH_VCPU_RESET (KVM_REG_LOONGARCH_KVM | KVM_REG_SIZE_U64 | 2)
#define LOONGARCH_REG_SHIFT 3
#define LOONGARCH_REG_64(TYPE, REG) (TYPE | KVM_REG_SIZE_U64 | (REG << LOONGARCH_REG_SHIFT))
#define KVM_IOC_CSRID(REG) LOONGARCH_REG_64(KVM_REG_LOONGARCH_CSR, REG)
#define KVM_IOC_CPUCFG(REG) LOONGARCH_REG_64(KVM_REG_LOONGARCH_CPUCFG, REG)
struct kvm_debug_exit_arch {
};
/* for KVM_SET_GUEST_DEBUG */
struct kvm_guest_debug_arch {
};
/* definition of registers in kvm_run */
struct kvm_sync_regs {
};
/* dummy definition */
struct kvm_sregs {
};
struct kvm_iocsr_entry {
__u32 addr;
__u32 pad;
__u64 data;
};
#define KVM_NR_IRQCHIPS 1
#define KVM_IRQCHIP_NUM_PINS 64
#define KVM_MAX_CORES 256
#endif /* __UAPI_ASM_LOONGARCH_KVM_H */
arch/loongarch/kernel/asm-offsets.c
View file @ 6803bd79
......@@ -9,6 +9,7 @@
#include <linux/mm.h>
#include <linux/kbuild.h>
#include <linux/suspend.h>
#include <linux/kvm_host.h>
#include <asm/cpu-info.h>
#include <asm/ptrace.h>
#include <asm/processor.h>
......@@ -289,3 +290,34 @@ void output_fgraph_ret_regs_defines(void)
BLANK();
}
#endif
void output_kvm_defines(void)
{
COMMENT("KVM/LoongArch Specific offsets.");
OFFSET(VCPU_FCC, kvm_vcpu_arch, fpu.fcc);
OFFSET(VCPU_FCSR0, kvm_vcpu_arch, fpu.fcsr);
BLANK();
OFFSET(KVM_VCPU_ARCH, kvm_vcpu, arch);
OFFSET(KVM_VCPU_KVM, kvm_vcpu, kvm);
OFFSET(KVM_VCPU_RUN, kvm_vcpu, run);
BLANK();
OFFSET(KVM_ARCH_HSP, kvm_vcpu_arch, host_sp);
OFFSET(KVM_ARCH_HTP, kvm_vcpu_arch, host_tp);
OFFSET(KVM_ARCH_HPGD, kvm_vcpu_arch, host_pgd);
OFFSET(KVM_ARCH_HANDLE_EXIT, kvm_vcpu_arch, handle_exit);
OFFSET(KVM_ARCH_HEENTRY, kvm_vcpu_arch, host_eentry);
OFFSET(KVM_ARCH_GEENTRY, kvm_vcpu_arch, guest_eentry);
OFFSET(KVM_ARCH_GPC, kvm_vcpu_arch, pc);
OFFSET(KVM_ARCH_GGPR, kvm_vcpu_arch, gprs);
OFFSET(KVM_ARCH_HBADI, kvm_vcpu_arch, badi);
OFFSET(KVM_ARCH_HBADV, kvm_vcpu_arch, badv);
OFFSET(KVM_ARCH_HECFG, kvm_vcpu_arch, host_ecfg);
OFFSET(KVM_ARCH_HESTAT, kvm_vcpu_arch, host_estat);
OFFSET(KVM_ARCH_HPERCPU, kvm_vcpu_arch, host_percpu);
OFFSET(KVM_GPGD, kvm, arch.pgd);
BLANK();
}
arch/loongarch/kvm/Kconfig 0 → 100644
View file @ 6803bd79
# SPDX-License-Identifier: GPL-2.0
#
# KVM configuration
#
source "virt/kvm/Kconfig"
menuconfig VIRTUALIZATION
bool "Virtualization"
help
Say Y here to get to see options for using your Linux host to run
other operating systems inside virtual machines (guests).
This option alone does not add any kernel code.
If you say N, all options in this submenu will be skipped and
disabled.
if VIRTUALIZATION
config KVM
tristate "Kernel-based Virtual Machine (KVM) support"
depends on AS_HAS_LVZ_EXTENSION
depends on HAVE_KVM
select HAVE_KVM_DIRTY_RING_ACQ_REL
select HAVE_KVM_EVENTFD
select HAVE_KVM_VCPU_ASYNC_IOCTL
select KVM_GENERIC_DIRTYLOG_READ_PROTECT
select KVM_GENERIC_HARDWARE_ENABLING
select KVM_MMIO
select KVM_XFER_TO_GUEST_WORK
select MMU_NOTIFIER
select PREEMPT_NOTIFIERS
help
Support hosting virtualized guest machines using
hardware virtualization extensions. You will need
a processor equipped with virtualization extensions.
If unsure, say N.
endif # VIRTUALIZATION
arch/loongarch/kvm/Makefile 0 → 100644
View file @ 6803bd79
# SPDX-License-Identifier: GPL-2.0
#
# Makefile for LoongArch KVM support
#
ccflags-y += -I $(srctree)/$(src)
include $(srctree)/virt/kvm/Makefile.kvm
obj-$(CONFIG_KVM) += kvm.o
kvm-y += exit.o
kvm-y += interrupt.o
kvm-y += main.o
kvm-y += mmu.o
kvm-y += switch.o
kvm-y += timer.o
kvm-y += tlb.o
kvm-y += vcpu.o
kvm-y += vm.o
CFLAGS_exit.o += $(call cc-option,-Wno-override-init,)
arch/loongarch/kvm/exit.c 0 → 100644
View file @ 6803bd79
This diff is collapsed.
arch/loongarch/kvm/interrupt.c 0 → 100644
View file @ 6803bd79
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <linux/err.h>
#include <linux/errno.h>
#include <asm/kvm_csr.h>
#include <asm/kvm_vcpu.h>
static unsigned int priority_to_irq[EXCCODE_INT_NUM] = {
[INT_TI] = CPU_TIMER,
[INT_IPI] = CPU_IPI,
[INT_SWI0] = CPU_SIP0,
[INT_SWI1] = CPU_SIP1,
[INT_HWI0] = CPU_IP0,
[INT_HWI1] = CPU_IP1,
[INT_HWI2] = CPU_IP2,
[INT_HWI3] = CPU_IP3,
[INT_HWI4] = CPU_IP4,
[INT_HWI5] = CPU_IP5,
[INT_HWI6] = CPU_IP6,
[INT_HWI7] = CPU_IP7,
};
static int kvm_irq_deliver(struct kvm_vcpu *vcpu, unsigned int priority)
{
unsigned int irq = 0;
clear_bit(priority, &vcpu->arch.irq_pending);
if (priority < EXCCODE_INT_NUM)
irq = priority_to_irq[priority];
switch (priority) {
case INT_TI:
case INT_IPI:
case INT_SWI0:
case INT_SWI1:
set_gcsr_estat(irq);
break;
case INT_HWI0 ... INT_HWI7:
set_csr_gintc(irq);
break;
default:
break;
}
return 1;
}
static int kvm_irq_clear(struct kvm_vcpu *vcpu, unsigned int priority)
{
unsigned int irq = 0;
clear_bit(priority, &vcpu->arch.irq_clear);
if (priority < EXCCODE_INT_NUM)
irq = priority_to_irq[priority];
switch (priority) {
case INT_TI:
case INT_IPI:
case INT_SWI0:
case INT_SWI1:
clear_gcsr_estat(irq);
break;
case INT_HWI0 ... INT_HWI7:
clear_csr_gintc(irq);
break;
default:
break;
}
return 1;
}
void kvm_deliver_intr(struct kvm_vcpu *vcpu)
{
unsigned int priority;
unsigned long *pending = &vcpu->arch.irq_pending;
unsigned long *pending_clr = &vcpu->arch.irq_clear;
if (!(*pending) && !(*pending_clr))
return;
if (*pending_clr) {
priority = __ffs(*pending_clr);
while (priority <= INT_IPI) {
kvm_irq_clear(vcpu, priority);
priority = find_next_bit(pending_clr,
BITS_PER_BYTE * sizeof(*pending_clr),
priority + 1);
}
}
if (*pending) {
priority = __ffs(*pending);
while (priority <= INT_IPI) {
kvm_irq_deliver(vcpu, priority);
priority = find_next_bit(pending,
BITS_PER_BYTE * sizeof(*pending),
priority + 1);
}
}
}
int kvm_pending_timer(struct kvm_vcpu *vcpu)
{
return test_bit(INT_TI, &vcpu->arch.irq_pending);
}
/*
* Only support illegal instruction or illegal Address Error exception,
* Other exceptions are injected by hardware in kvm mode
*/
static void _kvm_deliver_exception(struct kvm_vcpu *vcpu,
unsigned int code, unsigned int subcode)
{
unsigned long val, vec_size;
/*
* BADV is added for EXCCODE_ADE exception
* Use PC register (GVA address) if it is instruction exeception
* Else use BADV from host side (GPA address) for data exeception
*/
if (code == EXCCODE_ADE) {
if (subcode == EXSUBCODE_ADEF)
val = vcpu->arch.pc;
else
val = vcpu->arch.badv;
kvm_write_hw_gcsr(LOONGARCH_CSR_BADV, val);
}
/* Set exception instruction */
kvm_write_hw_gcsr(LOONGARCH_CSR_BADI, vcpu->arch.badi);
/*
* Save CRMD in PRMD
* Set IRQ disabled and PLV0 with CRMD
*/
val = kvm_read_hw_gcsr(LOONGARCH_CSR_CRMD);
kvm_write_hw_gcsr(LOONGARCH_CSR_PRMD, val);
val = val & ~(CSR_CRMD_PLV | CSR_CRMD_IE);
kvm_write_hw_gcsr(LOONGARCH_CSR_CRMD, val);
/* Set exception PC address */
kvm_write_hw_gcsr(LOONGARCH_CSR_ERA, vcpu->arch.pc);
/*
* Set exception code
* Exception and interrupt can be inject at the same time
* Hardware will handle exception first and then extern interrupt
* Exception code is Ecode in ESTAT[16:21]
* Interrupt code in ESTAT[0:12]
*/
val = kvm_read_hw_gcsr(LOONGARCH_CSR_ESTAT);
val = (val & ~CSR_ESTAT_EXC) | code;
kvm_write_hw_gcsr(LOONGARCH_CSR_ESTAT, val);
/* Calculate expcetion entry address */
val = kvm_read_hw_gcsr(LOONGARCH_CSR_ECFG);
vec_size = (val & CSR_ECFG_VS) >> CSR_ECFG_VS_SHIFT;
if (vec_size)
vec_size = (1 << vec_size) * 4;
val = kvm_read_hw_gcsr(LOONGARCH_CSR_EENTRY);
vcpu->arch.pc = val + code * vec_size;
}
void kvm_deliver_exception(struct kvm_vcpu *vcpu)
{
unsigned int code;
unsigned long *pending = &vcpu->arch.exception_pending;
if (*pending) {
code = __ffs(*pending);
_kvm_deliver_exception(vcpu, code, vcpu->arch.esubcode);
*pending = 0;
vcpu->arch.esubcode = 0;
}
}
arch/loongarch/kvm/main.c 0 → 100644
View file @ 6803bd79
This diff is collapsed.
arch/loongarch/kvm/mmu.c 0 → 100644
View file @ 6803bd79
This diff is collapsed.
arch/loongarch/kvm/switch.S 0 → 100644
View file @ 6803bd79
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <linux/linkage.h>
#include <asm/asm.h>
#include <asm/asmmacro.h>
#include <asm/loongarch.h>
#include <asm/regdef.h>
#include <asm/stackframe.h>
#define HGPR_OFFSET(x) (PT_R0 + 8*x)
#define GGPR_OFFSET(x) (KVM_ARCH_GGPR + 8*x)
.macro kvm_save_host_gpr base
.irp n,1,2,3,22,23,24,25,26,27,28,29,30,31
st.d $r\n, \base, HGPR_OFFSET(\n)
.endr
.endm
.macro kvm_restore_host_gpr base
.irp n,1,2,3,22,23,24,25,26,27,28,29,30,31
ld.d $r\n, \base, HGPR_OFFSET(\n)
.endr
.endm
/*
* Save and restore all GPRs except base register,
* and default value of base register is a2.
*/
.macro kvm_save_guest_gprs base
.irp n,1,2,3,4,5,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31
st.d $r\n, \base, GGPR_OFFSET(\n)
.endr
.endm
.macro kvm_restore_guest_gprs base
.irp n,1,2,3,4,5,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31
ld.d $r\n, \base, GGPR_OFFSET(\n)
.endr
.endm
/*
* Prepare switch to guest, save host regs and restore guest regs.
* a2: kvm_vcpu_arch, don't touch it until 'ertn'
* t0, t1: temp register
*/
.macro kvm_switch_to_guest
/* Set host ECFG.VS=0, all exceptions share one exception entry */
csrrd t0, LOONGARCH_CSR_ECFG
bstrins.w t0, zero, CSR_ECFG_VS_SHIFT_END, CSR_ECFG_VS_SHIFT
csrwr t0, LOONGARCH_CSR_ECFG
/* Load up the new EENTRY */
ld.d t0, a2, KVM_ARCH_GEENTRY
csrwr t0, LOONGARCH_CSR_EENTRY
/* Set Guest ERA */
ld.d t0, a2, KVM_ARCH_GPC
csrwr t0, LOONGARCH_CSR_ERA
/* Save host PGDL */
csrrd t0, LOONGARCH_CSR_PGDL
st.d t0, a2, KVM_ARCH_HPGD
/* Switch to kvm */
ld.d t1, a2, KVM_VCPU_KVM - KVM_VCPU_ARCH
/* Load guest PGDL */
li.w t0, KVM_GPGD
ldx.d t0, t1, t0
csrwr t0, LOONGARCH_CSR_PGDL
/* Mix GID and RID */
csrrd t1, LOONGARCH_CSR_GSTAT
bstrpick.w t1, t1, CSR_GSTAT_GID_SHIFT_END, CSR_GSTAT_GID_SHIFT
csrrd t0, LOONGARCH_CSR_GTLBC
bstrins.w t0, t1, CSR_GTLBC_TGID_SHIFT_END, CSR_GTLBC_TGID_SHIFT
csrwr t0, LOONGARCH_CSR_GTLBC
/*
* Enable intr in root mode with future ertn so that host interrupt
* can be responsed during VM runs
* Guest CRMD comes from separate GCSR_CRMD register
*/
ori t0, zero, CSR_PRMD_PIE
csrxchg t0, t0, LOONGARCH_CSR_PRMD
/* Set PVM bit to setup ertn to guest context */
ori t0, zero, CSR_GSTAT_PVM
csrxchg t0, t0, LOONGARCH_CSR_GSTAT
/* Load Guest GPRs */
kvm_restore_guest_gprs a2
/* Load KVM_ARCH register */
ld.d a2, a2, (KVM_ARCH_GGPR + 8 * REG_A2)
ertn /* Switch to guest: GSTAT.PGM = 1, ERRCTL.ISERR = 0, TLBRPRMD.ISTLBR = 0 */
.endm
/*
* Exception entry for general exception from guest mode
* - IRQ is disabled
* - kernel privilege in root mode
* - page mode keep unchanged from previous PRMD in root mode
* - Fixme: tlb exception cannot happen since registers relative with TLB
* - is still in guest mode, such as pgd table/vmid registers etc,
* - will fix with hw page walk enabled in future
* load kvm_vcpu from reserved CSR KVM_VCPU_KS, and save a2 to KVM_TEMP_KS
*/
.text
.cfi_sections .debug_frame
SYM_CODE_START(kvm_exc_entry)
csrwr a2, KVM_TEMP_KS
csrrd a2, KVM_VCPU_KS
addi.d a2, a2, KVM_VCPU_ARCH
/* After save GPRs, free to use any GPR */
kvm_save_guest_gprs a2
/* Save guest A2 */
csrrd t0, KVM_TEMP_KS
st.d t0, a2, (KVM_ARCH_GGPR + 8 * REG_A2)
/* A2 is kvm_vcpu_arch, A1 is free to use */
csrrd s1, KVM_VCPU_KS
ld.d s0, s1, KVM_VCPU_RUN
csrrd t0, LOONGARCH_CSR_ESTAT
st.d t0, a2, KVM_ARCH_HESTAT
csrrd t0, LOONGARCH_CSR_ERA
st.d t0, a2, KVM_ARCH_GPC
csrrd t0, LOONGARCH_CSR_BADV
st.d t0, a2, KVM_ARCH_HBADV
csrrd t0, LOONGARCH_CSR_BADI
st.d t0, a2, KVM_ARCH_HBADI
/* Restore host ECFG.VS */
csrrd t0, LOONGARCH_CSR_ECFG
ld.d t1, a2, KVM_ARCH_HECFG
or t0, t0, t1
csrwr t0, LOONGARCH_CSR_ECFG
/* Restore host EENTRY */
ld.d t0, a2, KVM_ARCH_HEENTRY
csrwr t0, LOONGARCH_CSR_EENTRY
/* Restore host pgd table */
ld.d t0, a2, KVM_ARCH_HPGD
csrwr t0, LOONGARCH_CSR_PGDL
/*
* Disable PGM bit to enter root mode by default with next ertn
*/
ori t0, zero, CSR_GSTAT_PVM
csrxchg zero, t0, LOONGARCH_CSR_GSTAT
/*
* Clear GTLBC.TGID field
* 0: for root tlb update in future tlb instr
* others: for guest tlb update like gpa to hpa in future tlb instr
*/
csrrd t0, LOONGARCH_CSR_GTLBC
bstrins.w t0, zero, CSR_GTLBC_TGID_SHIFT_END, CSR_GTLBC_TGID_SHIFT
csrwr t0, LOONGARCH_CSR_GTLBC
ld.d tp, a2, KVM_ARCH_HTP
ld.d sp, a2, KVM_ARCH_HSP
/* restore per cpu register */
ld.d u0, a2, KVM_ARCH_HPERCPU
addi.d sp, sp, -PT_SIZE
/* Prepare handle exception */
or a0, s0, zero
or a1, s1, zero
ld.d t8, a2, KVM_ARCH_HANDLE_EXIT
jirl ra, t8, 0
or a2, s1, zero
addi.d a2, a2, KVM_VCPU_ARCH
/* Resume host when ret <= 0 */
blez a0, ret_to_host
/*
* Return to guest
* Save per cpu register again, maybe switched to another cpu
*/
st.d u0, a2, KVM_ARCH_HPERCPU
/* Save kvm_vcpu to kscratch */
csrwr s1, KVM_VCPU_KS
kvm_switch_to_guest
ret_to_host:
ld.d a2, a2, KVM_ARCH_HSP
addi.d a2, a2, -PT_SIZE
kvm_restore_host_gpr a2
jr ra
SYM_INNER_LABEL(kvm_exc_entry_end, SYM_L_LOCAL)
SYM_CODE_END(kvm_exc_entry)
/*
* int kvm_enter_guest(struct kvm_run *run, struct kvm_vcpu *vcpu)
*
* @register_param:
* a0: kvm_run* run
* a1: kvm_vcpu* vcpu
*/
SYM_FUNC_START(kvm_enter_guest)
/* Allocate space in stack bottom */
addi.d a2, sp, -PT_SIZE
/* Save host GPRs */
kvm_save_host_gpr a2
/* Save host CRMD, PRMD to stack */
csrrd a3, LOONGARCH_CSR_CRMD
st.d a3, a2, PT_CRMD
csrrd a3, LOONGARCH_CSR_PRMD
st.d a3, a2, PT_PRMD
addi.d a2, a1, KVM_VCPU_ARCH
st.d sp, a2, KVM_ARCH_HSP
st.d tp, a2, KVM_ARCH_HTP
/* Save per cpu register */
st.d u0, a2, KVM_ARCH_HPERCPU
/* Save kvm_vcpu to kscratch */
csrwr a1, KVM_VCPU_KS
kvm_switch_to_guest
SYM_INNER_LABEL(kvm_enter_guest_end, SYM_L_LOCAL)
SYM_FUNC_END(kvm_enter_guest)
SYM_FUNC_START(kvm_save_fpu)
fpu_save_csr a0 t1
fpu_save_double a0 t1
fpu_save_cc a0 t1 t2
jr ra
SYM_FUNC_END(kvm_save_fpu)
SYM_FUNC_START(kvm_restore_fpu)
fpu_restore_double a0 t1
fpu_restore_csr a0 t1 t2
fpu_restore_cc a0 t1 t2
jr ra
SYM_FUNC_END(kvm_restore_fpu)
.section ".rodata"
SYM_DATA(kvm_exception_size, .quad kvm_exc_entry_end - kvm_exc_entry)
SYM_DATA(kvm_enter_guest_size, .quad kvm_enter_guest_end - kvm_enter_guest)
arch/loongarch/kvm/timer.c 0 → 100644
View file @ 6803bd79
This diff is collapsed.
arch/loongarch/kvm/tlb.c 0 → 100644
View file @ 6803bd79
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <linux/kvm_host.h>
#include <asm/tlb.h>
#include <asm/kvm_csr.h>
/*
* kvm_flush_tlb_all() - Flush all root TLB entries for guests.
*
* Invalidate all entries including GVA-->GPA and GPA-->HPA mappings.
*/
void kvm_flush_tlb_all(void)
{
unsigned long flags;
local_irq_save(flags);
invtlb_all(INVTLB_ALLGID, 0, 0);
local_irq_restore(flags);
}
void kvm_flush_tlb_gpa(struct kvm_vcpu *vcpu, unsigned long gpa)
{
unsigned long flags;
local_irq_save(flags);
gpa &= (PAGE_MASK << 1);
invtlb(INVTLB_GID_ADDR, read_csr_gstat() & CSR_GSTAT_GID, gpa);
local_irq_restore(flags);
}
arch/loongarch/kvm/trace.h 0 → 100644
View file @ 6803bd79
This diff is collapsed.
arch/loongarch/kvm/vcpu.c 0 → 100644
View file @ 6803bd79
This diff is collapsed.
arch/loongarch/kvm/vm.c 0 → 100644
View file @ 6803bd79
This diff is collapsed.
arch/riscv/include/asm/csr.h
View file @ 6803bd79
......@@ -203,6 +203,18 @@
#define ENVCFG_CBIE_INV _AC(0x3, UL)
#define ENVCFG_FIOM _AC(0x1, UL)
/* Smstateen bits */
#define SMSTATEEN0_AIA_IMSIC_SHIFT 58
#define SMSTATEEN0_AIA_IMSIC (_ULL(1) << SMSTATEEN0_AIA_IMSIC_SHIFT)
#define SMSTATEEN0_AIA_SHIFT 59
#define SMSTATEEN0_AIA (_ULL(1) << SMSTATEEN0_AIA_SHIFT)
#define SMSTATEEN0_AIA_ISEL_SHIFT 60
#define SMSTATEEN0_AIA_ISEL (_ULL(1) << SMSTATEEN0_AIA_ISEL_SHIFT)
#define SMSTATEEN0_HSENVCFG_SHIFT 62
#define SMSTATEEN0_HSENVCFG (_ULL(1) << SMSTATEEN0_HSENVCFG_SHIFT)
#define SMSTATEEN0_SSTATEEN0_SHIFT 63
#define SMSTATEEN0_SSTATEEN0 (_ULL(1) << SMSTATEEN0_SSTATEEN0_SHIFT)
/* symbolic CSR names: */
#define CSR_CYCLE 0xc00
#define CSR_TIME 0xc01
......@@ -275,6 +287,8 @@
#define CSR_SIE 0x104
#define CSR_STVEC 0x105
#define CSR_SCOUNTEREN 0x106
#define CSR_SENVCFG 0x10a
#define CSR_SSTATEEN0 0x10c
#define CSR_SSCRATCH 0x140
#define CSR_SEPC 0x141
#define CSR_SCAUSE 0x142
......@@ -349,6 +363,10 @@
#define CSR_VSIEH 0x214
#define CSR_VSIPH 0x254
/* Hypervisor stateen CSRs */
#define CSR_HSTATEEN0 0x60c
#define CSR_HSTATEEN0H 0x61c
#define CSR_MSTATUS 0x300
#define CSR_MISA 0x301
#define CSR_MIDELEG 0x303
......
arch/riscv/include/asm/hwcap.h
View file @ 6803bd79
......@@ -58,6 +58,8 @@
#define RISCV_ISA_EXT_ZICSR 40
#define RISCV_ISA_EXT_ZIFENCEI 41
#define RISCV_ISA_EXT_ZIHPM 42
#define RISCV_ISA_EXT_SMSTATEEN 43
#define RISCV_ISA_EXT_ZICOND 44
#define RISCV_ISA_EXT_MAX 64
......
arch/riscv/include/asm/kvm_host.h
View file @ 6803bd79
This diff is collapsed.
arch/riscv/include/asm/kvm_vcpu_sbi.h
View file @ 6803bd79
......@@ -11,7 +11,7 @@
#define KVM_SBI_IMPID 3
#define KVM_SBI_VERSION_MAJOR 1
#define KVM_SBI_VERSION_MAJOR 2
#define KVM_SBI_VERSION_MINOR 0
enum kvm_riscv_sbi_ext_status {
......@@ -35,6 +35,9 @@ struct kvm_vcpu_sbi_return {
struct kvm_vcpu_sbi_extension {
unsigned long extid_start;
unsigned long extid_end;
bool default_unavail;
/**
* SBI extension handler. It can be defined for a given extension or group of
* extension. But it should always return linux error codes rather than SBI
......@@ -59,6 +62,7 @@ int kvm_riscv_vcpu_get_reg_sbi_ext(struct kvm_vcpu *vcpu,
const struct kvm_vcpu_sbi_extension *kvm_vcpu_sbi_find_ext(
struct kvm_vcpu *vcpu, unsigned long extid);
int kvm_riscv_vcpu_sbi_ecall(struct kvm_vcpu *vcpu, struct kvm_run *run);
void kvm_riscv_vcpu_sbi_init(struct kvm_vcpu *vcpu);
#ifdef CONFIG_RISCV_SBI_V01
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_v01;
......@@ -69,6 +73,7 @@ extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_ipi;
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_rfence;
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_srst;
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_hsm;
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_dbcn;
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_experimental;
extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_vendor;
......
arch/riscv/include/asm/sbi.h
View file @ 6803bd79
This diff is collapsed.
arch/riscv/include/uapi/asm/kvm.h
View file @ 6803bd79
This diff is collapsed.
arch/riscv/kernel/cpufeature.c
View file @ 6803bd79
This diff is collapsed.
arch/riscv/kvm/vcpu.c
View file @ 6803bd79
This diff is collapsed.
arch/riscv/kvm/vcpu_onereg.c
View file @ 6803bd79
This diff is collapsed.
arch/riscv/kvm/vcpu_sbi.c
View file @ 6803bd79
This diff is collapsed.
arch/riscv/kvm/vcpu_sbi_replace.c
View file @ 6803bd79
This diff is collapsed.
arch/s390/include/asm/kvm_host.h
View file @ 6803bd79
This diff is collapsed.
arch/s390/kvm/gaccess.c
View file @ 6803bd79
This diff is collapsed.
arch/s390/kvm/kvm-s390.c
View file @ 6803bd79
This diff is collapsed.
arch/s390/kvm/trace-s390.h
View file @ 6803bd79
This diff is collapsed.
arch/s390/kvm/vsie.c
View file @ 6803bd79
This diff is collapsed.
arch/x86/include/asm/cpufeatures.h
View file @ 6803bd79
......@@ -443,6 +443,7 @@
/* AMD-defined Extended Feature 2 EAX, CPUID level 0x80000021 (EAX), word 20 */
#define X86_FEATURE_NO_NESTED_DATA_BP (20*32+ 0) /* "" No Nested Data Breakpoints */
#define X86_FEATURE_WRMSR_XX_BASE_NS (20*32+ 1) /* "" WRMSR to {FS,GS,KERNEL_GS}_BASE is non-serializing */
#define X86_FEATURE_LFENCE_RDTSC (20*32+ 2) /* "" LFENCE always serializing / synchronizes RDTSC */
#define X86_FEATURE_NULL_SEL_CLR_BASE (20*32+ 6) /* "" Null Selector Clears Base */
#define X86_FEATURE_AUTOIBRS (20*32+ 8) /* "" Automatic IBRS */
......
arch/x86/include/asm/kvm-x86-ops.h
View file @ 6803bd79
This diff is collapsed.
arch/x86/include/asm/kvm_host.h
View file @ 6803bd79
This diff is collapsed.
arch/x86/include/asm/msr-index.h
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/Kconfig
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/cpuid.c
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/cpuid.h
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/hyperv.c
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/lapic.c
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/mmu.h
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/mmu/mmu.c
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/mtrr.c
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/smm.c
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/svm/svm.c
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/vmx/vmx.c
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/x86.c
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/x86.h
View file @ 6803bd79
This diff is collapsed.
arch/x86/kvm/xen.c
View file @ 6803bd79
This diff is collapsed.
include/kvm/arm_arch_timer.h
View file @ 6803bd79
This diff is collapsed.
include/kvm/arm_pmu.h
View file @ 6803bd79
This diff is collapsed.
include/kvm/arm_psci.h
View file @ 6803bd79
This diff is collapsed.
include/kvm/arm_vgic.h
View file @ 6803bd79
This diff is collapsed.
include/linux/perf/arm_pmuv3.h
View file @ 6803bd79
This diff is collapsed.
include/uapi/linux/kvm.h
View file @ 6803bd79
This diff is collapsed.
tools/arch/arm64/include/.gitignore 0 → 100644
View file @ 6803bd79
This diff is collapsed.
tools/arch/arm64/include/asm/gpr-num.h 0 → 100644
View file @ 6803bd79
This diff is collapsed.
tools/arch/arm64/include/asm/sysreg.h
View file @ 6803bd79
This diff is collapsed.
tools/arch/arm64/tools/Makefile 0 → 100644
View file @ 6803bd79
This diff is collapsed.
tools/include/perf/arm_pmuv3.h 0 → 100644
View file @ 6803bd79
This diff is collapsed.
tools/perf/Makefile.perf
View file @ 6803bd79
This diff is collapsed.
tools/perf/util/Build
View file @ 6803bd79
This diff is collapsed.
tools/testing/selftests/kvm/Makefile
View file @ 6803bd79
This diff is collapsed.
tools/testing/selftests/kvm/aarch64/set_id_regs.c 0 → 100644
View file @ 6803bd79
This diff is collapsed.
tools/testing/selftests/kvm/aarch64/vpmu_counter_access.c 0 → 100644
View file @ 6803bd79
This diff is collapsed.
tools/testing/selftests/kvm/x86_64/hwcr_msr_test.c 0 → 100644
View file @ 6803bd79
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7