1. 03 Oct, 2009 11 commits
    • Christoph Lameter's avatar
      this_cpu: Use this_cpu operations in RCU · e800879d
      Christoph Lameter authored
      RCU does not do dynamic allocations but it increments per cpu variables
      a lot. These instructions results in a move to a register and then back
      to memory. This patch will make it use the inc/dec instructions on x86
      that do not need a register.
      Acked-by: default avatarTejun Heo <tj@kernel.org>
      Acked-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      Signed-off-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      e800879d
    • Christoph Lameter's avatar
      this_cpu: Use this_cpu ops for VM statistics · 4dac3e98
      Christoph Lameter authored
      Using per cpu atomics for the vm statistics reduces their overhead.
      And in the case of x86 we are guaranteed that they will never race even
      in the lax form used for vm statistics.
      Acked-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      4dac3e98
    • Christoph Lameter's avatar
      this_cpu: Use this_cpu_ptr in crypto subsystem · 0b44f486
      Christoph Lameter authored
      Just a slight optimization that removes one array lookup.
      The processor number is needed for other things as well so the
      get/put_cpu cannot be removed.
      Acked-by: default avatarTejun Heo <tj@kernel.org>
      Cc: Huang Ying <ying.huang@intel.com>
      Signed-off-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      0b44f486
    • Christoph Lameter's avatar
      this_cpu: xfs_icsb_modify_counters does not need "cpu" variable · 7a9e02d6
      Christoph Lameter authored
      The xfs_icsb_modify_counters() function no longer needs the cpu variable
      if we use this_cpu_ptr() and we can get rid of get/put_cpu().
      Acked-by: default avatarTejun Heo <tj@kernel.org>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      Acked-by: default avatarOlaf Weber <olaf@sgi.com>
      Signed-off-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      7a9e02d6
    • Christoph Lameter's avatar
      this_cpu: Eliminate get/put_cpu · e7dcaa47
      Christoph Lameter authored
      There are cases where we can use this_cpu_ptr and as the result
      of using this_cpu_ptr() we no longer need to determine the
      currently executing cpu.
      
      In those places no get/put_cpu combination is needed anymore.
      The local cpu variable can be eliminated.
      
      Preemption still needs to be disabled and enabled since the
      modifications of the per cpu variables is not atomic. There may
      be multiple per cpu variables modified and those must all
      be from the same processor.
      Acked-by: default avatarMaciej Sosnowski <maciej.sosnowski@intel.com>
      Acked-by: default avatarDan Williams <dan.j.williams@intel.com>
      Acked-by: default avatarTejun Heo <tj@kernel.org>
      cc: Eric Biederman <ebiederm@aristanetworks.com>
      cc: Stephen Hemminger <shemminger@vyatta.com>
      cc: David L Stevens <dlstevens@us.ibm.com>
      Signed-off-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      e7dcaa47
    • Christoph Lameter's avatar
      this_cpu: Straight transformations · ca0c9584
      Christoph Lameter authored
      Use this_cpu_ptr and __this_cpu_ptr in locations where straight
      transformations are possible because per_cpu_ptr is used with
      either smp_processor_id() or raw_smp_processor_id().
      
      cc: David Howells <dhowells@redhat.com>
      Acked-by: default avatarTejun Heo <tj@kernel.org>
      cc: Ingo Molnar <mingo@elte.hu>
      cc: Rusty Russell <rusty@rustcorp.com.au>
      cc: Eric Dumazet <dada1@cosmosbay.com>
      Signed-off-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      ca0c9584
    • Christoph Lameter's avatar
    • Christoph Lameter's avatar
      this_cpu: Use this_cpu operations for NFS statistics · fce22848
      Christoph Lameter authored
      Simplify NFS statistics and allow the use of optimized
      arch instructions.
      Acked-by: default avatarTejun Heo <tj@kernel.org>
      CC: Trond Myklebust <trond.myklebust@fys.uio.no>
      Signed-off-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      fce22848
    • Christoph Lameter's avatar
      this_cpu: Use this_cpu operations for SNMP statistics · 4eb41d10
      Christoph Lameter authored
      SNMP statistic macros can be signficantly simplified.
      This will also reduce code size if the arch supports these operations
      in hardware.
      Acked-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      4eb41d10
    • Christoph Lameter's avatar
      this_cpu: Implement X86 optimized this_cpu operations · 30ed1a79
      Christoph Lameter authored
      Basically the existing percpu ops can be used for this_cpu variants that allow
      operations also on dynamically allocated percpu data. However, we do not pass a
      reference to a percpu variable in. Instead a dynamically or statically
      allocated percpu variable is provided.
      
      Preempt, the non preempt and the irqsafe operations generate the same code.
      It will always be possible to have the requires per cpu atomicness in a single
      RMW instruction with segment override on x86.
      
      64 bit this_cpu operations are not supported on 32 bit.
      Signed-off-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      30ed1a79
    • Christoph Lameter's avatar
      this_cpu: Introduce this_cpu_ptr() and generic this_cpu_* operations · 7340a0b1
      Christoph Lameter authored
      This patch introduces two things: First this_cpu_ptr and then per cpu
      atomic operations.
      
      this_cpu_ptr
      ------------
      
      A common operation when dealing with cpu data is to get the instance of the
      cpu data associated with the currently executing processor. This can be
      optimized by
      
      this_cpu_ptr(xx) = per_cpu_ptr(xx, smp_processor_id).
      
      The problem with per_cpu_ptr(x, smp_processor_id) is that it requires
      an array lookup to find the offset for the cpu. Processors typically
      have the offset for the current cpu area in some kind of (arch dependent)
      efficiently accessible register or memory location.
      
      We can use that instead of doing the array lookup to speed up the
      determination of the address of the percpu variable. This is particularly
      significant because these lookups occur in performance critical paths
      of the core kernel. this_cpu_ptr() can avoid memory accesses and
      
      this_cpu_ptr comes in two flavors. The preemption context matters since we
      are referring the the currently executing processor. In many cases we must
      insure that the processor does not change while a code segment is executed.
      
      __this_cpu_ptr 	-> Do not check for preemption context
      this_cpu_ptr	-> Check preemption context
      
      The parameter to these operations is a per cpu pointer. This can be the
      address of a statically defined per cpu variable (&per_cpu_var(xxx)) or
      the address of a per cpu variable allocated with the per cpu allocator.
      
      per cpu atomic operations: this_cpu_*(var, val)
      -----------------------------------------------
      this_cpu_* operations (like this_cpu_add(struct->y, value) operate on
      abitrary scalars that are members of structures allocated with the new
      per cpu allocator. They can also operate on static per_cpu variables
      if they are passed to per_cpu_var() (See patch to use this_cpu_*
      operations for vm statistics).
      
      These operations are guaranteed to be atomic vs preemption when modifying
      the scalar. The calculation of the per cpu offset is also guaranteed to
      be atomic at the same time. This means that a this_cpu_* operation can be
      safely used to modify a per cpu variable in a context where interrupts are
      enabled and preemption is allowed. Many architectures can perform such
      a per cpu atomic operation with a single instruction.
      
      Note that the atomicity here is different from regular atomic operations.
      Atomicity is only guaranteed for data accessed from the currently executing
      processor. Modifications from other processors are still possible. There
      must be other guarantees that the per cpu data is not modified from another
      processor when using these instruction. The per cpu atomicity is created
      by the fact that the processor either executes and instruction or not.
      Embedded in the instruction is the relocation of the per cpu address to
      the are reserved for the current processor and the RMW action. Therefore
      interrupts or preemption cannot occur in the mids of this processing.
      
      Generic fallback functions are used if an arch does not define optimized
      this_cpu operations. The functions come also come in the two flavors used
      for this_cpu_ptr().
      
      The firstparameter is a scalar that is a member of a structure allocated
      through allocpercpu or a per cpu variable (use per_cpu_var(xxx)). The
      operations are similar to what percpu_add() and friends do.
      
      this_cpu_read(scalar)
      this_cpu_write(scalar, value)
      this_cpu_add(scale, value)
      this_cpu_sub(scalar, value)
      this_cpu_inc(scalar)
      this_cpu_dec(scalar)
      this_cpu_and(scalar, value)
      this_cpu_or(scalar, value)
      this_cpu_xor(scalar, value)
      
      Arch code can override the generic functions and provide optimized atomic
      per cpu operations. These atomic operations must provide both the relocation
      (x86 does it through a segment override) and the operation on the data in a
      single instruction. Otherwise preempt needs to be disabled and there is no
      gain from providing arch implementations.
      
      A third variant is provided prefixed by irqsafe_. These variants are safe
      against hardware interrupts on the *same* processor (all per cpu atomic
      primitives are *always* *only* providing safety for code running on the
      *same* processor!). The increment needs to be implemented by the hardware
      in such a way that it is a single RMW instruction that is either processed
      before or after an interrupt.
      
      cc: David Howells <dhowells@redhat.com>
      cc: Ingo Molnar <mingo@elte.hu>
      cc: Rusty Russell <rusty@rustcorp.com.au>
      cc: Eric Dumazet <dada1@cosmosbay.com>
      Signed-off-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      7340a0b1
  2. 02 Oct, 2009 6 commits
    • Tejun Heo's avatar
      percpu: kill legacy percpu allocator · 23fb064b
      Tejun Heo authored
      With ia64 converted, there's no arch left which still uses legacy
      percpu allocator.  Kill it.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Delightedly-acked-by: default avatarRusty Russell <rusty@rustcorp.com.au>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      23fb064b
    • Tejun Heo's avatar
      ia64: convert to dynamic percpu allocator · 52594762
      Tejun Heo authored
      Unlike other archs, ia64 reserves space for percpu areas during early
      memory initialization.  These areas occupy a contiguous region indexed
      by cpu number on contiguous memory model or are grouped by node on
      discontiguous memory model.
      
      As allocation and initialization are done by the arch code, all that
      setup_per_cpu_areas() needs to do is communicating the determined
      layout to the percpu allocator.  This patch implements
      setup_per_cpu_areas() for both contig and discontig memory models and
      drops HAVE_LEGACY_PER_CPU_AREA.
      
      Please note that for contig model, the allocation itself is modified
      only to allocate for possible cpus instead of NR_CPUS.  As dynamic
      percpu allocator can handle non-direct mapping, there's no reason to
      allocate memory for cpus which aren't possible.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Acked-by: default avatarTony Luck <tony.luck@intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: linux-ia64 <linux-ia64@vger.kernel.org>
      52594762
    • Tejun Heo's avatar
      ia64: allocate percpu area for cpu0 like percpu areas for other cpus · 36886478
      Tejun Heo authored
      cpu0 used special percpu area reserved by the linker, __cpu0_per_cpu,
      which is set up early in boot by head.S.  However, this doesn't
      guarantee that the area will be on the same node as cpu0 and the
      percpu area for cpu0 ends up very far away from percpu areas for other
      cpus which cause problems for congruent percpu allocator.
      
      This patch makes percpu area initialization allocate percpu area for
      cpu0 like any other cpus and copy it from __cpu0_per_cpu which now
      resides in the __init area.  This means that for cpu0, percpu area is
      first setup at __cpu0_per_cpu early by head.S and then moved to an
      area in the linear mapping during memory initialization and it's not
      allowed to take a pointer to percpu variables between head.S and
      memory initialization.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Acked-by: default avatarTony Luck <tony.luck@intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: linux-ia64 <linux-ia64@vger.kernel.org>
      36886478
    • Tejun Heo's avatar
      ia64: initialize cpu maps early · 12cda817
      Tejun Heo authored
      All information necessary to initialize cpu possible and present maps
      are available once early_acpi_boot_init() is complete.  Reorganize
      setup_arch() and acpi init functions such that,
      
      * CPU information is printed after LAPIC entries are parsed in
        early_acpi_boot_init().
      
      * smp_build_cpu_map() is called by setup_arch() instead of acpi
        functions.
      
      * smp_build_cpu_map() is called once all CPU related information is
        available before memory is initialized.
      
      This is primarily to allow find_memory() to use cpu maps but is also a
      general cleanup.  Please note that with this change, the somewhat
      ad-hoc early_cpu_possible_map defined and used for NUMA configurations
      is probably unnecessary.  Something to clean up another day.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Acked-by: default avatarTony Luck <tony.luck@intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: linux-ia64 <linux-ia64@vger.kernel.org>
      12cda817
    • Tejun Heo's avatar
      ia64: don't alias VMALLOC_END to vmalloc_end · 126b3fcd
      Tejun Heo authored
      If CONFIG_VIRTUAL_MEM_MAP is enabled, ia64 defines macro VMALLOC_END
      as unsigned long variable vmalloc_end which is adjusted to prepare
      room for vmemmap.  This becomes probnlematic if a local variables
      vmalloc_end is defined in some function (not very unlikely) and
      VMALLOC_END is used in the function - the function thinks its
      referencing the global VMALLOC_END value but would be referencing its
      own local vmalloc_end variable.
      
      There's no reason VMALLOC_END should be a macro.  Just define it as an
      unsigned long variable if CONFIG_VIRTUAL_MEM_MAP is set to avoid nasty
      surprises.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Acked-by: default avatarTony Luck <tony.luck@intel.com>
      Cc: Fenghua Yu <fenghua.yu@intel.com>
      Cc: linux-ia64 <linux-ia64@vger.kernel.org>
      Cc: Christoph Lameter <cl@linux-foundation.org>
      126b3fcd
    • Linus Torvalds's avatar
      Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable · 0efe5e32
      Linus Torvalds authored
      * 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable:
        Btrfs: fix data space leak fix
        Btrfs: remove duplicates of filemap_ helpers
        Btrfs: take i_mutex before generic_write_checks
        Btrfs: fix arguments to btrfs_wait_on_page_writeback_range
        Btrfs: fix deadlock with free space handling and user transactions
        Btrfs: fix error cases for ioctl transactions
        Btrfs: Use CONFIG_BTRFS_POSIX_ACL to enable ACL code
        Btrfs: introduce missing kfree
        Btrfs: Fix setting umask when POSIX ACLs are not enabled
        Btrfs: proper -ENOSPC handling
      0efe5e32
  3. 01 Oct, 2009 23 commits