You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On ARMv7 CPUs that cache first level page table entries (like the
Cortex-A15), using a reserved ASID while changing the TTBR or flushing
the TLB is unsafe.
This is because the CPU may cache the first level entry as the result of
a speculative memory access while the reserved ASID is assigned. After
the process owning the page tables dies, the memory will be reallocated
and may be written with junk values which can be interpreted as global,
valid PTEs by the processor. This will result in the TLB being populated
with bogus global entries.
This patch avoids the use of a reserved context ID in the v7 switch_mm
and ASID rollover code by temporarily using the swapper_pg_dir pointed
at by TTBR1, which contains only global entries that are not tagged
with ASIDs.
Reviewed-by: Frank Rowand <[email protected]>
Tested-by: Marc Zyngier <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
[[email protected]: add LPAE support]
Signed-off-by: Catalin Marinas <[email protected]>
Change-Id: I2dd82501dac5ee402765aaa0ffb3f7f577a603c9
ARM: Remove __ARCH_WANT_INTERRUPTS_ON_CTXSW on ASID-capable CPUs
Since the ASIDs must be unique to an mm across all the CPUs in a system,
the __new_context() function needs to broadcast a context reset event to
all the CPUs during ASID allocation if a roll-over occurred. Such IPIs
cannot be issued with interrupts disabled and ARM had to define
__ARCH_WANT_INTERRUPTS_ON_CTXSW.
This patch changes the check_context() function to
check_and_switch_context() called from switch_mm(). In case of
ASID-capable CPUs (ARMv6 onwards), if a new ASID is needed and the
interrupts are disabled, it defers the __new_context() and
cpu_switch_mm() calls to the post-lock switch hook where the interrupts
are enabled. Setting the reserved TTBR0 was also moved to
check_and_switch_context() from cpu_v7_switch_mm().
Reviewed-by: Will Deacon <[email protected]>
Tested-by: Will Deacon <[email protected]>
Reviewed-by: Frank Rowand <[email protected]>
Tested-by: Marc Zyngier <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
Conflicts:
arch/arm/mm/proc-v7-2level.S
Change-Id: I48e39730c49ff8d30ce566e8a03cf54557869a52
ARM: Remove current_mm per-cpu variable
The current_mm variable was used to store the new mm between the
switch_mm() and switch_to() calls where an IPI to reset the context
could have set the wrong mm. Since the interrupts are disabled during
context switch, there is no need for this variable, current->active_mm
already points to the current mm when interrupts are re-enabled.
Reviewed-by: Will Deacon <[email protected]>
Tested-by: Will Deacon <[email protected]>
Reviewed-by: Frank Rowand <[email protected]>
Tested-by: Marc Zyngier <[email protected]>
Signed-off-by: Catalin Marinas <[email protected]>
ARM: 7502/1: contextidr: avoid using bfi instruction during notifier
The bfi instruction is not available on ARMv6, so instead use an and/orr
sequence in the contextidr_notifier. This gets rid of the assembler
error:
Assembler messages:
Error: selected processor does not support ARM mode `bfi r3,r2,#0,lg-devs#8'
Reported-by: Arnd Bergmann <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Signed-off-by: Russell King <[email protected]>
Conflicts:
arch/arm/mm/context.c
Change-Id: Id64c0145d0dcd1cfe9e2aba59f86c08ec5fbf649
ARM: mm: remove IPI broadcasting on ASID rollover
ASIDs are allocated to MMU contexts based on a rolling counter. This
means that after 255 allocations we must invalidate all existing ASIDs
via an expensive IPI mechanism to synchronise all of the online CPUs and
ensure that all tasks execute with an ASID from the new generation.
This patch changes the rollover behaviour so that we rely instead on the
hardware broadcasting of the TLB invalidation to avoid the IPI calls.
This works by keeping track of the active ASID on each core, which is
then reserved in the case of a rollover so that currently scheduled
tasks can continue to run. For cores without hardware TLB broadcasting,
we keep track of pending flushes in a cpumask, so cores can flush their
local TLB before scheduling a new mm.
Reviewed-by: Catalin Marinas <[email protected]>
Tested-by: Marc Zyngier <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Conflicts:
arch/arm/mm/context.c
Change-Id: I58990400aaaaef35319f7b3fb2f84fe7e46cb581
ARM: mm: avoid taking ASID spinlock on fastpath
When scheduling a new mm, we take a spinlock so that we can:
1. Safely allocate a new ASID, if required
2. Update our active_asids field without worrying about parallel
updates to reserved_asids
3. Ensure that we flush our local TLB, if required
However, this has the nasty affect of serialising context-switch across
all CPUs in the system. The usual (fast) case is where the next mm has
a valid ASID for the current generation. In such a scenario, we can
avoid taking the lock and instead use atomic64_xchg to update the
active_asids variable for the current CPU. If a rollover occurs on
another CPU (which would take the lock), when copying the active_asids
into the reserved_asids another atomic64_xchg is used to replace each
active_asids with 0. The fast path can then detect this case and fall
back to spinning on the lock.
Tested-by: Marc Zyngier <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
ARM: mm: use bitmap operations when allocating new ASIDs
When allocating a new ASID, we must take care not to re-assign a
reserved ASID-value to a new mm. This requires us to check each
candidate ASID against those currently reserved by other cores before
assigning a new ASID to the current mm.
This patch improves the ASID allocation algorithm by using a
bitmap-based approach. Rather than iterating over the reserved ASID
array for each candidate ASID, we simply find the first zero bit,
ensuring that those indices corresponding to reserved ASIDs are set
when flushing during a rollover event.
Tested-by: Marc Zyngier <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
ARM: 7649/1: mm: mm->context.id fix for big-endian
Since the new ASID code in b5466f8728527a05a493cc4abe9e6f034a1bbaab
("ARM: mm: remove IPI broadcasting on ASID rollover") was changed to
use 64bit operations it has broken the BE operation due to an issue
with the MM code accessing sub-fields of mm->context.id.
When running in BE mode we see the values in mm->context.id are stored
with the highest value first, so the LDR in the arch/arm/mm/proc-macros.S
reads the wrong part of this field. To resolve this, change the LDR in
the mmid macro to load from +4.
Acked-by: Will Deacon <[email protected]>
Signed-off-by: Ben Dooks <[email protected]>
Signed-off-by: Russell King <[email protected]>
ARM: 7658/1: mm: fix race updating mm->context.id on ASID rollover
If a thread triggers an ASID rollover, other threads of the same process
must be made to wait until the mm->context.id for the shared mm_struct
has been updated to new generation and associated book-keeping (e.g.
TLB invalidation) has ben performed.
However, there is a *tiny* window where both mm->context.id and the
relevant active_asids entry are updated to the new generation, but the
TLB flush has not been performed, which could allow another thread to
return to userspace with a dirty TLB, potentially leading to data
corruption. In reality this will never occur because one CPU would need
to perform a context-switch in the time it takes another to do a couple
of atomic test/set operations but we should plug the race anyway.
This patch moves the active_asids update until after the potential TLB
flush on context-switch.
Cc: <[email protected]> # 3.8
Reviewed-by: Catalin Marinas <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Signed-off-by: Russell King <[email protected]>
ARM: 7659/1: mm: make mm->context.id an atomic64_t variable
mm->context.id is updated under asid_lock when a new ASID is allocated
to an mm_struct. However, it is also read without the lock when a task
is being scheduled and checking whether or not the current ASID
generation is up-to-date.
If two threads of the same process are being scheduled in parallel and
the bottom bits of the generation in their mm->context.id match the
current generation (that is, the mm_struct has not been used for ~2^24
rollovers) then the non-atomic, lockless access to mm->context.id may
yield the incorrect ASID.
This patch fixes this issue by making mm->context.id and atomic64_t,
ensuring that the generation is always read consistently. For code that
only requires access to the ASID bits (e.g. TLB flushing by mm), then
the value is accessed directly, which GCC converts to an ldrb.
Cc: <[email protected]> # 3.8
Reviewed-by: Catalin Marinas <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Signed-off-by: Russell King <[email protected]>
Conflicts:
arch/arm/include/asm/mmu.h
Change-Id: I682895d6357a91ecc439c8543fa94f1aecbfcb4c
ARM: 7661/1: mm: perform explicit branch predictor maintenance when required
The ARM ARM requires branch predictor maintenance if, for a given ASID,
the instructions at a specific virtual address appear to change.
From the kernel's point of view, that means:
- Changing the kernel's view of memory (e.g. switching to the
identity map)
- ASID rollover (since ASIDs will be re-allocated to new tasks)
This patch adds explicit branch predictor maintenance when either of the
two conditions above are met.
Reviewed-by: Catalin Marinas <[email protected]>
Signed-off-by: Will Deacon <[email protected]>
Signed-off-by: Russell King <[email protected]>
ARM: 7684/1: errata: Workaround for Cortex-A15 erratum 798181 (TLBI/DSB operations)
On Cortex-A15 (r0p0..r3p2) the TLBI/DSB are not adequately shooting down
all use of the old entries. This patch implements the erratum workaround
which consists of:
1. Dummy TLBIMVAIS and DSB on the CPU doing the TLBI operation.
2. Send IPI to the CPUs that are running the same mm (and ASID) as the
one being invalidated (or all the online CPUs for global pages).
3. CPU receiving the IPI executes a DMB and CLREX (part of the exception
return code already).
Signed-off-by: Catalin Marinas <[email protected]>
Signed-off-by: Russell King <[email protected]>
Conflicts:
arch/arm/Kconfig
Change-Id: I4513d042301a1faad817b83434280462cc176df1
msm: rtb: Log the context id in the rtb
Store the context id in the register trace buffer.
The process id can be derived from the context id.
This gives a general idea about what process was last
running when the RTB stopped.
Change-Id: I2fb8934d008b8cf3666f1df2652846c15faca776
Signed-off-by: Laura Abbott <[email protected]>
(cherry picked from commit 445eb9a)
Conflicts:
arch/arm/mach-msm/include/mach/msm_rtb.h
ARM: 7767/1: let the ASID allocator handle suspended animation
commit ae120d9edfe96628f03d87634acda0bfa7110632 upstream.
When a CPU is running a process, the ASID for that process is
held in a per-CPU variable (the "active ASIDs" array). When
the ASID allocator handles a rollover, it copies the active
ASIDs into a "reserved ASIDs" array to ensure that a process
currently running on another CPU will continue to run unaffected.
The active array is zero-ed to indicate that a rollover occurred.
Because of this mechanism, a reserved ASID is only remembered for
a single rollover. A subsequent rollover will completely refill
the reserved ASIDs array.
In a severely oversubscribed environment where a CPU can be
prevented from running for extended periods of time (think virtual
machines), the above has a horrible side effect:
[P{a} denotes process P running with ASID a]
CPU-0 CPU-1
A{x} [active = <x 0>]
[suspended] runs B{y} [active = <x y>]
[rollover:
active = <0 0>
reserved = <x y>]
runs B{y} [active = <0 y>
reserved = <x y>]
[rollover:
active = <0 0>
reserved = <0 y>]
runs C{x} [active = <0 x>]
[resumes]
runs A{x}
At that stage, both A and C have the same ASID, with deadly
consequences.
The fix is to preserve reserved ASIDs across rollovers if
the CPU doesn't have an active ASID when the rollover occurs.
Acked-by: Will Deacon <[email protected]>
Acked-by: Catalin Carinas <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Signed-off-by: Russell King <[email protected]>
Signed-off-by: Greg Kroah-Hartman <[email protected]>
ARM: 7768/1: prevent risks of out-of-bound access in ASID allocator
commit b8e4a4740fa2b17c0a447b3ab783b3dc10702e27 upstream.
On a CPU that never ran anything, both the active and reserved ASID
fields are set to zero. In this case the ASID_TO_IDX() macro will
return -1, which is not a very useful value to index a bitmap.
Instead of trying to offset the ASID so that ASID lg-devs#1 is actually
bit 0 in the asid_map bitmap, just always ignore bit 0 and start
the search from bit 1. This makes the code a bit more readable,
and without risk of OoB access.
Acked-by: Will Deacon <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Reported-by: Catalin Marinas <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Signed-off-by: Russell King <[email protected]>
Signed-off-by: Greg Kroah-Hartman <[email protected]>
ARM: 7703/1: Disable preemption in broadcast_tlb*_a15_erratum()
Commit 93dc688 (ARM: 7684/1: errata: Workaround for Cortex-A15 erratum
798181 (TLBI/DSB operations)) introduces calls to smp_processor_id() and
smp_call_function_many() with preemption enabled. This patch disables
preemption and also optimises the smp_processor_id() call in
broadcast_tlb_mm_a15_erratum(). The broadcast_tlb_a15_erratum() function
is changed to use smp_call_function() which disables preemption.
Signed-off-by: Catalin Marinas <[email protected]>
Reported-by: Geoff Levand <[email protected]>
Reported-by: Nicolas Pitre <[email protected]>
Signed-off-by: Russell King <[email protected]>
ARM: 7769/1: Cortex-A15: fix erratum 798181 implementation
commit 0d0752bca1f9a91fb646647aa4abbb21156f316c upstream.
Looking into the active_asids array is not enough, as we also need
to look into the reserved_asids array (they both represent processes
that are currently running).
Also, not holding the ASID allocator lock is racy, as another CPU
could schedule that process and trigger a rollover, making the erratum
workaround miss an IPI.
Exposing this outside of context.c is a little ugly on the side, so
let's define a new entry point that the erratum workaround can call
to obtain the cpumask.
Acked-by: Will Deacon <[email protected]>
Acked-by: Catalin Marinas <[email protected]>
Signed-off-by: Marc Zyngier <[email protected]>
Signed-off-by: Russell King <[email protected]>
Signed-off-by: Greg Kroah-Hartman <[email protected]>
arm: mm: Clean ASID patchset
Change-Id: Id5b0cc6d5300d293e33baf6603453bde0df6d6d8
android/lowmemorykiller: Ignore tasks with freed mm
A killed task can stay in the task list long after its
memory has been returned to the system, therefore
ignore any tasks whose mm struct has been freed.
Change-Id: I76394b203b4ab2312437c839976f0ecb7b6dde4e
CRs-fixed: 450383
Signed-off-by: Liam Mark <[email protected]>
Signed-off-by: Pranav Vashi <[email protected]>
0 commit comments