1. 20 Jan, 2009 9 commits
  2. 19 Jan, 2009 2 commits
  3. 18 Jan, 2009 14 commits
  4. 17 Jan, 2009 2 commits
  5. 16 Jan, 2009 13 commits
    • Tejun Heo's avatar
      x86_64: initialize this_cpu_off to __per_cpu_load · cd3adf52
      Tejun Heo authored
      On x86_64, if get_per_cpu_var() is used before per cpu area is setup
      (if lockdep is turned on, it happens), it needs this_cpu_off to point
      to __per_cpu_load.  Initialize accordingly.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      cd3adf52
    • Tejun Heo's avatar
      x86: fix build bug introduced during merge · a338af2c
      Tejun Heo authored
      EXPORT_PER_CPU_SYMBOL() got misplaced during merge leading to build
      failure.  Fix it.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      a338af2c
    • Ingo Molnar's avatar
      percpu: add optimized generic percpu accessors · 6dbde353
      Ingo Molnar authored
      It is an optimization and a cleanup, and adds the following new
      generic percpu methods:
      
        percpu_read()
        percpu_write()
        percpu_add()
        percpu_sub()
        percpu_and()
        percpu_or()
        percpu_xor()
      
      and implements support for them on x86. (other architectures will fall
      back to a default implementation)
      
      The advantage is that for example to read a local percpu variable,
      instead of this sequence:
      
       return __get_cpu_var(var);
      
       ffffffff8102ca2b:	48 8b 14 fd 80 09 74 	mov    -0x7e8bf680(,%rdi,8),%rdx
       ffffffff8102ca32:	81
       ffffffff8102ca33:	48 c7 c0 d8 59 00 00 	mov    $0x59d8,%rax
       ffffffff8102ca3a:	48 8b 04 10          	mov    (%rax,%rdx,1),%rax
      
      We can get a single instruction by using the optimized variants:
      
       return percpu_read(var);
      
       ffffffff8102ca3f:	65 48 8b 05 91 8f fd 	mov    %gs:0x7efd8f91(%rip),%rax
      
      I also cleaned up the x86-specific APIs and made the x86 code use
      these new generic percpu primitives.
      
      tj: * fixed generic percpu_sub() definition as Roel Kluin pointed out
          * added percpu_and() for completeness's sake
          * made generic percpu ops atomic against preemption
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      6dbde353
    • Tejun Heo's avatar
      x86: misc clean up after the percpu update · 004aa322
      Tejun Heo authored
      Do the following cleanups:
      
      * kill x86_64_init_pda() which now is equivalent to pda_init()
      
      * use per_cpu_offset() instead of cpu_pda() when initializing
        initial_gs
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      004aa322
    • Tejun Heo's avatar
      x86: convert pda ops to wrappers around x86 percpu accessors · 49357d19
      Tejun Heo authored
      pda is now a percpu variable and there's no reason it can't use plain
      x86 percpu accessors.  Add x86_test_and_clear_bit_percpu() and replace
      pda op implementations with wrappers around x86 percpu accessors.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      49357d19
    • Tejun Heo's avatar
      x86: make pda a percpu variable · b12d8db8
      Tejun Heo authored
      [ Based on original patch from Christoph Lameter and Mike Travis. ]
      
      As pda is now allocated in percpu area, it can easily be made a proper
      percpu variable.  Make it so by defining per cpu symbol from linker
      script and declaring it in C code for SMP and simply defining it for
      UP.  This change cleans up code and brings SMP and UP closer a bit.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      b12d8db8
    • Tejun Heo's avatar
      x86: merge 64 and 32 SMP percpu handling · 9939ddaf
      Tejun Heo authored
      Now that pda is allocated as part of percpu, percpu doesn't need to be
      accessed through pda.  Unify x86_64 SMP percpu access with x86_32 SMP
      one.  Other than the segment register, operand size and the base of
      percpu symbols, they behave identical now.
      
      This patch replaces now unnecessary pda->data_offset with a dummy
      field which is necessary to keep stack_canary at its place.  This
      patch also moves per_cpu_offset initialization out of init_gdt() into
      setup_per_cpu_areas().  Note that this change also necessitates
      explicit per_cpu_offset initializations in voyager_smp.c.
      
      With this change, x86_OP_percpu()'s are as efficient on x86_64 as on
      x86_32 and also x86_64 can use assembly PER_CPU macros.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      9939ddaf
    • Tejun Heo's avatar
      x86: fold pda into percpu area on SMP · 1a51e3a0
      Tejun Heo authored
      [ Based on original patch from Christoph Lameter and Mike Travis. ]
      
      Currently pdas and percpu areas are allocated separately.  %gs points
      to local pda and percpu area can be reached using pda->data_offset.
      This patch folds pda into percpu area.
      
      Due to strange gcc requirement, pda needs to be at the beginning of
      the percpu area so that pda->stack_canary is at %gs:40.  To achieve
      this, a new percpu output section macro - PERCPU_VADDR_PREALLOC() - is
      added and used to reserve pda sized chunk at the start of the percpu
      area.
      
      After this change, for boot cpu, %gs first points to pda in the
      data.init area and later during setup_per_cpu_areas() gets updated to
      point to the actual pda.  This means that setup_per_cpu_areas() need
      to reload %gs for CPU0 while clearing pda area for other cpus as cpu0
      already has modified it when control reaches setup_per_cpu_areas().
      
      This patch also removes now unnecessary get_local_pda() and its call
      sites.
      
      A lot of this patch is taken from Mike Travis' "x86_64: Fold pda into
      per cpu area" patch.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      1a51e3a0
    • Tejun Heo's avatar
      x86: use static _cpu_pda array · c8f3329a
      Tejun Heo authored
      _cpu_pda array first uses statically allocated storage in data.init
      and then switches to allocated bootmem to conserve space.  However,
      after folding pda area into percpu area, _cpu_pda array will be
      removed completely.  Drop the reallocation part to simplify the code
      for soon-to-follow changes.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      c8f3329a
    • Tejun Heo's avatar
      x86: load pointer to pda into %gs while brining up a CPU · f32ff538
      Tejun Heo authored
      [ Based on original patch from Christoph Lameter and Mike Travis. ]
      
      CPU startup code in head_64.S loaded address of a zero page into %gs
      for temporary use till pda is loaded but address to the actual pda is
      available at the point.  Load the real address directly instead.
      
      This will help unifying percpu and pda handling later on.
      
      This patch is mostly taken from Mike Travis' "x86_64: Fold pda into
      per cpu area" patch.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      f32ff538
    • Tejun Heo's avatar
      x86: make percpu symbols zerobased on SMP · 3e5d8f97
      Tejun Heo authored
      [ Based on original patch from Christoph Lameter and Mike Travis. ]
      
      This patch makes percpu symbols zerobased on x86_64 SMP by adding
      PERCPU_VADDR() to vmlinux.lds.h which helps setting explicit vaddr on
      the percpu output section and using it in vmlinux_64.lds.S.  A new
      PHDR is added as existing ones cannot contain sections near address
      zero.  PERCPU_VADDR() also adds a new symbol __per_cpu_load which
      always points to the vaddr of the loaded percpu data.init region.
      
      The following adjustments have been made to accomodate the address
      change.
      
      * code to locate percpu gdt_page in head_64.S is updated to add the
        load address to the gdt_page offset.
      
      * __per_cpu_load is used in places where access to the init data area
        is necessary.
      
      * pda->data_offset is initialized soon after C code is entered as zero
        value doesn't work anymore.
      
      This patch is mostly taken from Mike Travis' "x86_64: Base percpu
      variables at zero" patch.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      3e5d8f97
    • Tejun Heo's avatar
      x86: make vmlinux_32.lds.S use PERCPU() macro · a698c823
      Tejun Heo authored
      Make vmlinux_32.lds.S use the generic PERCPU() macro instead of open
      coding it.  This will ease future changes.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      a698c823
    • Mike Travis's avatar
      x86: cleanup early setup_percpu references · c90aa894
      Mike Travis authored
      [ Based on original patch from Christoph Lameter and Mike Travis. ]
      
        * Ruggedize some calls in setup_percpu.c to prevent mishaps
          in early calls, particularly for non-critical functions.
      
        * Cleanup DEBUG_PER_CPU_MAPS usages and some comments.
      Signed-off-by: default avatarMike Travis <travis@sgi.com>
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
      c90aa894