1. 21 Jan, 2009 4 commits
  2. 20 Jan, 2009 9 commits
  3. 19 Jan, 2009 2 commits
  4. 18 Jan, 2009 14 commits
  5. 17 Jan, 2009 2 commits
  6. 16 Jan, 2009 9 commits
    • Tejun Heo's avatar
      x86_64: initialize this_cpu_off to __per_cpu_load · cd3adf52
      Tejun Heo authored
      On x86_64, if get_per_cpu_var() is used before per cpu area is setup
      (if lockdep is turned on, it happens), it needs this_cpu_off to point
      to __per_cpu_load.  Initialize accordingly.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      cd3adf52
    • Tejun Heo's avatar
      x86: fix build bug introduced during merge · a338af2c
      Tejun Heo authored
      EXPORT_PER_CPU_SYMBOL() got misplaced during merge leading to build
      failure.  Fix it.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      a338af2c
    • Ingo Molnar's avatar
      percpu: add optimized generic percpu accessors · 6dbde353
      Ingo Molnar authored
      It is an optimization and a cleanup, and adds the following new
      generic percpu methods:
      
        percpu_read()
        percpu_write()
        percpu_add()
        percpu_sub()
        percpu_and()
        percpu_or()
        percpu_xor()
      
      and implements support for them on x86. (other architectures will fall
      back to a default implementation)
      
      The advantage is that for example to read a local percpu variable,
      instead of this sequence:
      
       return __get_cpu_var(var);
      
       ffffffff8102ca2b:	48 8b 14 fd 80 09 74 	mov    -0x7e8bf680(,%rdi,8),%rdx
       ffffffff8102ca32:	81
       ffffffff8102ca33:	48 c7 c0 d8 59 00 00 	mov    $0x59d8,%rax
       ffffffff8102ca3a:	48 8b 04 10          	mov    (%rax,%rdx,1),%rax
      
      We can get a single instruction by using the optimized variants:
      
       return percpu_read(var);
      
       ffffffff8102ca3f:	65 48 8b 05 91 8f fd 	mov    %gs:0x7efd8f91(%rip),%rax
      
      I also cleaned up the x86-specific APIs and made the x86 code use
      these new generic percpu primitives.
      
      tj: * fixed generic percpu_sub() definition as Roel Kluin pointed out
          * added percpu_and() for completeness's sake
          * made generic percpu ops atomic against preemption
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      6dbde353
    • Tejun Heo's avatar
      x86: misc clean up after the percpu update · 004aa322
      Tejun Heo authored
      Do the following cleanups:
      
      * kill x86_64_init_pda() which now is equivalent to pda_init()
      
      * use per_cpu_offset() instead of cpu_pda() when initializing
        initial_gs
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      004aa322
    • Tejun Heo's avatar
      x86: convert pda ops to wrappers around x86 percpu accessors · 49357d19
      Tejun Heo authored
      pda is now a percpu variable and there's no reason it can't use plain
      x86 percpu accessors.  Add x86_test_and_clear_bit_percpu() and replace
      pda op implementations with wrappers around x86 percpu accessors.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      49357d19
    • Tejun Heo's avatar
      x86: make pda a percpu variable · b12d8db8
      Tejun Heo authored
      [ Based on original patch from Christoph Lameter and Mike Travis. ]
      
      As pda is now allocated in percpu area, it can easily be made a proper
      percpu variable.  Make it so by defining per cpu symbol from linker
      script and declaring it in C code for SMP and simply defining it for
      UP.  This change cleans up code and brings SMP and UP closer a bit.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      b12d8db8
    • Tejun Heo's avatar
      x86: merge 64 and 32 SMP percpu handling · 9939ddaf
      Tejun Heo authored
      Now that pda is allocated as part of percpu, percpu doesn't need to be
      accessed through pda.  Unify x86_64 SMP percpu access with x86_32 SMP
      one.  Other than the segment register, operand size and the base of
      percpu symbols, they behave identical now.
      
      This patch replaces now unnecessary pda->data_offset with a dummy
      field which is necessary to keep stack_canary at its place.  This
      patch also moves per_cpu_offset initialization out of init_gdt() into
      setup_per_cpu_areas().  Note that this change also necessitates
      explicit per_cpu_offset initializations in voyager_smp.c.
      
      With this change, x86_OP_percpu()'s are as efficient on x86_64 as on
      x86_32 and also x86_64 can use assembly PER_CPU macros.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      9939ddaf
    • Tejun Heo's avatar
      x86: fold pda into percpu area on SMP · 1a51e3a0
      Tejun Heo authored
      [ Based on original patch from Christoph Lameter and Mike Travis. ]
      
      Currently pdas and percpu areas are allocated separately.  %gs points
      to local pda and percpu area can be reached using pda->data_offset.
      This patch folds pda into percpu area.
      
      Due to strange gcc requirement, pda needs to be at the beginning of
      the percpu area so that pda->stack_canary is at %gs:40.  To achieve
      this, a new percpu output section macro - PERCPU_VADDR_PREALLOC() - is
      added and used to reserve pda sized chunk at the start of the percpu
      area.
      
      After this change, for boot cpu, %gs first points to pda in the
      data.init area and later during setup_per_cpu_areas() gets updated to
      point to the actual pda.  This means that setup_per_cpu_areas() need
      to reload %gs for CPU0 while clearing pda area for other cpus as cpu0
      already has modified it when control reaches setup_per_cpu_areas().
      
      This patch also removes now unnecessary get_local_pda() and its call
      sites.
      
      A lot of this patch is taken from Mike Travis' "x86_64: Fold pda into
      per cpu area" patch.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      1a51e3a0
    • Tejun Heo's avatar
      x86: use static _cpu_pda array · c8f3329a
      Tejun Heo authored
      _cpu_pda array first uses statically allocated storage in data.init
      and then switches to allocated bootmem to conserve space.  However,
      after folding pda area into percpu area, _cpu_pda array will be
      removed completely.  Drop the reallocation part to simplify the code
      for soon-to-follow changes.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      c8f3329a