• Peter Zijlstra's avatar
    sched/fair: Fix post_init_entity_util_avg() serialization · b7fa30c9
    Peter Zijlstra authored
    Chris Wilson reported a divide by 0 at:
    
     post_init_entity_util_avg():
    
     >    725	if (cfs_rq->avg.util_avg != 0) {
     >    726		sa->util_avg  = cfs_rq->avg.util_avg * se->load.weight;
     > -> 727		sa->util_avg /= (cfs_rq->avg.load_avg + 1);
     >    728
     >    729		if (sa->util_avg > cap)
     >    730			sa->util_avg = cap;
     >    731	} else {
    
    Which given the lack of serialization, and the code generated from
    update_cfs_rq_load_avg() is entirely possible:
    
    	if (atomic_long_read(&cfs_rq->removed_load_avg)) {
    		s64 r = atomic_long_xchg(&cfs_rq->removed_load_avg, 0);
    		sa->load_avg = max_t(long, sa->load_avg - r, 0);
    		sa->load_sum = max_t(s64, sa->load_sum - r * LOAD_AVG_MAX, 0);
    		removed_load = 1;
    	}
    
    turns into:
    
      ffffffff81087064:       49 8b 85 98 00 00 00    mov    0x98(%r13),%rax
      ffffffff8108706b:       48 85 c0                test   %rax,%rax
      ffffffff8108706e:       74 40                   je     ffffffff810870b0
      ffffffff81087070:       4c 89 f8                mov    %r15,%rax
      ffffffff81087073:       49 87 85 98 00 00 00    xchg   %rax,0x98(%r13)
      ffffffff8108707a:       49 29 45 70             sub    %rax,0x70(%r13)
      ffffffff8108707e:       4c 89 f9                mov    %r15,%rcx
      ffffffff81087081:       bb 01 00 00 00          mov    $0x1,%ebx
      ffffffff81087086:       49 83 7d 70 00          cmpq   $0x0,0x70(%r13)
      ffffffff8108708b:       49 0f 49 4d 70          cmovns 0x70(%r13),%rcx
    
    Which you'll note ends up with 'sa->load_avg - r' in memory at
    ffffffff8108707a.
    
    By calling post_init_entity_util_avg() under rq->lock we're sure to be
    fully serialized against PELT updates and cannot observe intermediate
    state like this.
    Reported-by: default avatarChris Wilson <chris@chris-wilson.co.uk>
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Mike Galbraith <efault@gmx.de>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Yuyang Du <yuyang.du@intel.com>
    Cc: bsegall@google.com
    Cc: morten.rasmussen@arm.com
    Cc: pjt@google.com
    Cc: steve.muckle@linaro.org
    Fixes: 2b8c41da ("sched/fair: Initiate a new task's util avg to a bounded value")
    Link: http://lkml.kernel.org/r/20160609130750.GQ30909@twins.programming.kicks-ass.netSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
    b7fa30c9
core.c 210 KB