• David Howells's avatar
    CacheFiles: Catch an overly long wait for an old active object · fee096de
    David Howells authored
    Catch an overly long wait for an old, dying active object when we want to
    replace it with a new one.  The probability is that all the slow-work threads
    are hogged, and the delete can't get a look in.
    
    What we do instead is:
    
     (1) if there's nothing in the slow work queue, we sleep until either the dying
         object has finished dying or there is something in the slow work queue
         behind which we can queue our object.
    
     (2) if there is something in the slow work queue, we return ETIMEDOUT to
         fscache_lookup_object(), which then puts us back on the slow work queue,
         presumably behind the deletion that we're blocked by.  We are then
         deferred for a while until we work our way back through the queue -
         without blocking a slow-work thread unnecessarily.
    
    A backtrace similar to the following may appear in the log without this patch:
    
    	INFO: task kslowd004:5711 blocked for more than 120 seconds.
    	"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    	kslowd004     D 0000000000000000     0  5711      2 0x00000080
    	 ffff88000340bb80 0000000000000046 ffff88002550d000 0000000000000000
    	 ffff88002550d000 0000000000000007 ffff88000340bfd8 ffff88002550d2a8
    	 000000000000ddf0 00000000000118c0 00000000000118c0 ffff88002550d2a8
    	Call Trace:
    	 [<ffffffff81058e21>] ? trace_hardirqs_on+0xd/0xf
    	 [<ffffffffa011c4d8>] ? cachefiles_wait_bit+0x0/0xd [cachefiles]
    	 [<ffffffffa011c4e1>] cachefiles_wait_bit+0x9/0xd [cachefiles]
    	 [<ffffffff81353153>] __wait_on_bit+0x43/0x76
    	 [<ffffffff8111ae39>] ? ext3_xattr_get+0x1ec/0x270
    	 [<ffffffff813531ef>] out_of_line_wait_on_bit+0x69/0x74
    	 [<ffffffffa011c4d8>] ? cachefiles_wait_bit+0x0/0xd [cachefiles]
    	 [<ffffffff8104c125>] ? wake_bit_function+0x0/0x2e
    	 [<ffffffffa011bc79>] cachefiles_mark_object_active+0x203/0x23b [cachefiles]
    	 [<ffffffffa011c209>] cachefiles_walk_to_object+0x558/0x827 [cachefiles]
    	 [<ffffffffa011a429>] cachefiles_lookup_object+0xac/0x12a [cachefiles]
    	 [<ffffffffa00aa1e9>] fscache_lookup_object+0x1c7/0x214 [fscache]
    	 [<ffffffffa00aafc5>] fscache_object_state_machine+0xa5/0x52d [fscache]
    	 [<ffffffffa00ab4ac>] fscache_object_slow_work_execute+0x5f/0xa0 [fscache]
    	 [<ffffffff81082093>] slow_work_execute+0x18f/0x2d1
    	 [<ffffffff8108239a>] slow_work_thread+0x1c5/0x308
    	 [<ffffffff8104c0f1>] ? autoremove_wake_function+0x0/0x34
    	 [<ffffffff810821d5>] ? slow_work_thread+0x0/0x308
    	 [<ffffffff8104be91>] kthread+0x7a/0x82
    	 [<ffffffff8100beda>] child_rip+0xa/0x20
    	 [<ffffffff8100b87c>] ? restore_args+0x0/0x30
    	 [<ffffffff8104be17>] ? kthread+0x0/0x82
    	 [<ffffffff8100bed0>] ? child_rip+0x0/0x20
    	1 lock held by kslowd004/5711:
    	 #0:  (&sb->s_type->i_mutex_key#7/1){+.+.+.}, at: [<ffffffffa011be64>] cachefiles_walk_to_object+0x1b3/0x827 [cachefiles]
    Signed-off-by: default avatarDavid Howells <dhowells@redhat.com>
    fee096de
object.c 25.5 KB