1. 29 Jan, 2014 9 commits
    • Wang Shilong's avatar
      Btrfs: rework ulist with list+rb_tree · 4c7a6f74
      Wang Shilong authored
      We are really suffering from now ulist's implementation, some developers
      gave their try, and i just gave some of my ideas for things:
      
       1. use list+rb_tree instead of arrary+rb_tree
      
       2. add cur_list to iterator rather than ulist structure.
      
       3. add seqnum into every node when they are added, this is
       used to do selfcheck when iterating node.
      
      I noticed Zach Brown's comments before, long term is to kick off
      ulist implementation, however, for now, we need at least avoid
      arrary from ulist.
      
      Cc: Liu Bo <bo.li.liu@oracle.com>
      Cc: Josef Bacik <jbacik@fb.com>
      Cc: Zach Brown <zab@redhat.com>
      Signed-off-by: default avatarWang Shilong <wangsl.fnst@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      4c7a6f74
    • Wang Shilong's avatar
      Btrfs: fix memory leaks on walking backrefs failure · f05c4746
      Wang Shilong authored
      When walking backrefs, we may iterate every inode's extent
      and add/merge them into ulist, and the caller will free memory
      from ulist.
      
      However, if we fail to allocate inode's extents element
      memory or ulist_add() fail to allocate memory, we won't
      add allocated memory into ulist, and the caller won't
      free some allocated memory thus memory leaks happen.
      Signed-off-by: default avatarWang Shilong <wangsl.fnst@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      f05c4746
    • Filipe David Borba Manana's avatar
      Btrfs: fix send file hole detection leading to data corruption · bf54f412
      Filipe David Borba Manana authored
      There was a case where file hole detection was incorrect and it would
      cause an incremental send to override a section of a file with zeroes.
      
      This happened in the case where between the last leaf we processed which
      contained a file extent item for our current inode and the leaf we're
      currently are at (and has a file extent item for our current inode) there
      are only leafs containing exclusively file extent items for our current
      inode, and none of them was updated since the previous send operation.
      The file hole detection code would incorrectly consider the file range
      covered by these leafs as a hole.
      
      A test case for xfstests follows soon.
      Signed-off-by: default avatarFilipe David Borba Manana <fdmanana@gmail.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      bf54f412
    • Wang Shilong's avatar
      Btrfs: add a reschedule point in btrfs_find_all_roots() · bca1a290
      Wang Shilong authored
      I can easily trigger the following warnings when enabling quota
      in my virtual machine(running Opensuse), Steps are firstly creating
      a subvolume full of fragment extents, and then create many snapshots
      (500 in my test case).
      
      [ 2362.808459] BUG: soft lockup - CPU#0 stuck for 22s! [btrfs-qgroup-re:1970]
      
      [ 2362.809023] task: e4af8450 ti: e371c000 task.ti: e371c000
      [ 2362.809026] EIP: 0060:[<fa38f4ae>] EFLAGS: 00000246 CPU: 0
      [ 2362.809049] EIP is at __merge_refs+0x5e/0x100 [btrfs]
      [ 2362.809051] EAX: 00000000 EBX: cfadbcf0 ECX: 00000000 EDX: cfadbcb0
      [ 2362.809052] ESI: dd8d3370 EDI: e371dde0 EBP: e371dd6c ESP: e371dd5c
      [ 2362.809054]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
      [ 2362.809055] CR0: 80050033 CR2: ac454d50 CR3: 009a9000 CR4: 001407d0
      [ 2362.809099] Stack:
      [ 2362.809100]  00000001 e371dde0 dfcc6890 f29f8000 e371de28 fa39016d 00000011 00000001
      [ 2362.809105]  99bfc000 00000000 93928000 00000000 00000001 00000050 e371dda8 00000001
      [ 2362.809109]  f3a31000 f3413000 00000001 e371ddb8 000040a8 00000202 00000000 00000023
      [ 2362.809113] Call Trace:
      [ 2362.809136]  [<fa39016d>] find_parent_nodes+0x34d/0x1280 [btrfs]
      [ 2362.809156]  [<fa391172>] btrfs_find_all_roots+0xb2/0x110 [btrfs]
      [ 2362.809174]  [<fa3934a8>] btrfs_qgroup_rescan_worker+0x358/0x7a0 [btrfs]
      [ 2362.809180]  [<c024d0ce>] ? lock_timer_base.isra.39+0x1e/0x40
      [ 2362.809199]  [<fa3648df>] worker_loop+0xff/0x470 [btrfs]
      [ 2362.809204]  [<c027a88a>] ? __wake_up_locked+0x1a/0x20
      [ 2362.809221]  [<fa3647e0>] ? btrfs_queue_worker+0x2b0/0x2b0 [btrfs]
      [ 2362.809225]  [<c025ebbc>] kthread+0x9c/0xb0
      [ 2362.809229]  [<c06b487b>] ret_from_kernel_thread+0x1b/0x30
      [ 2362.809233]  [<c025eb20>] ? kthread_create_on_node+0x110/0x110
      
      By adding a reschedule point at the end of btrfs_find_all_roots(), i no longer
      hit these warnings.
      
      Cc: Josef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarWang Shilong <wangsl.fnst@cn.fujitsu.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.cz>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      bca1a290
    • Filipe David Borba Manana's avatar
      Btrfs: make send's file extent item search more efficient · 7fdd29d0
      Filipe David Borba Manana authored
      Instead of looking for a file extent item, process it, release the path
      and do a btree search for the next file extent item, just process all
      file extent items in a leaf without intermediate btree searches. This way
      we save cpu and we're not blocking other tasks or affecting concurrency on
      the btree, because send's paths use the commit root and skip btree node/leaf
      locking.
      Signed-off-by: default avatarFilipe David Borba Manana <fdmanana@gmail.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      7fdd29d0
    • Wang Shilong's avatar
      Btrfs: fix to catch all errors when resolving indirect ref · 95def2ed
      Wang Shilong authored
      We can only tolerate ENOENT here, for other errors, we should
      return directly.
      Signed-off-by: default avatarWang Shilong <wangsl.fnst@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      95def2ed
    • Wang Shilong's avatar
      Btrfs: fix protection between walking backrefs and root deletion · 538f72cd
      Wang Shilong authored
      There is a race condition between resolving indirect ref and root deletion,
      and we should gurantee that root can not be destroyed to avoid accessing
      broken tree here.
      
      Here we fix it by holding @subvol_srcu, and we will release it as soon
      as we have held root node lock.
      Signed-off-by: default avatarWang Shilong <wangsl.fnst@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      538f72cd
    • Gui Hecheng's avatar
      btrfs: fix warning while merging two adjacent extents · 3c9665df
      Gui Hecheng authored
      When we have two adjacent extents in relink_extent_backref,
      we try to merge them. When we use btrfs_search_slot to locate the
      slot for the current extent, we shouldn't set "ins_len = 1",
      because we will merge it into the previous extent rather than
      insert a new item. Otherwise, we may happen to create a new leaf
      in btrfs_search_slot and path->slot[0] will be 0. Then we try to
      fetch the previous item using "path->slots[0]--", and it will cause
      a warning as follows:
      
      	[  145.713385] WARNING: CPU: 3 PID: 1796 at fs/btrfs/extent_io.c:5043 map_private_extent_buffer+0xd4/0xe0
      	[  145.713387] btrfs bad mapping eb start 53370886 len 4096, wanted 167772306 8
      	...
      	[  145.713462]  [<ffffffffa034b1f4>] map_private_extent_buffer+0xd4/0xe0
      	[  145.713476]  [<ffffffffa030097a>] ? btrfs_free_path+0x2a/0x40
      	[  145.713485]  [<ffffffffa0340864>] btrfs_get_token_64+0x64/0xf0
      	[  145.713498]  [<ffffffffa033472c>] relink_extent_backref+0x41c/0x820
      	[  145.713508]  [<ffffffffa0334d69>] btrfs_finish_ordered_io+0x239/0xa80
      
      I encounter this warning when running defrag having mkfs.btrfs
      with option -M. At the same time there are read/writes & snapshots
      running at background.
      Signed-off-by: default avatarGui Hecheng <guihc.fnst@cn.fujitsu.com>
      Reviewed-by: default avatarLiu Bo <bo.li.liu@oracle.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      3c9665df
    • Filipe David Borba Manana's avatar
      Btrfs: fix infinite path build loops in incremental send · 9f03740a
      Filipe David Borba Manana authored
      The send operation processes inodes by their ascending number, and assumes
      that any rename/move operation can be successfully performed (sent to the
      caller) once all previous inodes (those with a smaller inode number than the
      one we're currently processing) were processed.
      
      This is not true when an incremental send had to process an hierarchical change
      between 2 snapshots where the parent-children relationship between directory
      inodes was reversed - that is, parents became children and children became
      parents. This situation made the path building code go into an infinite loop,
      which kept allocating more and more memory that eventually lead to a krealloc
      warning being displayed in dmesg:
      
        WARNING: CPU: 1 PID: 5705 at mm/page_alloc.c:2477 __alloc_pages_nodemask+0x365/0xad0()
        Modules linked in: btrfs raid6_pq xor pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) snd_hda_codec_hdmi snd_hda_codec_realtek joydev radeon snd_hda_intel snd_hda_codec snd_hwdep snd_seq_midi snd_pcm psmouse i915 snd_rawmidi serio_raw snd_seq_midi_event lpc_ich snd_seq snd_timer ttm snd_seq_device rfcomm drm_kms_helper parport_pc bnep bluetooth drm ppdev snd soundcore i2c_algo_bit snd_page_alloc binfmt_misc video lp parport r8169 mii hid_generic usbhid hid
        CPU: 1 PID: 5705 Comm: btrfs Tainted: G           O 3.13.0-rc7-fdm-btrfs-next-18+ #3
        Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77 Pro4, BIOS P1.50 09/04/2012
        [ 5381.660441]  00000000000009ad ffff8806f6f2f4e8 ffffffff81777434 0000000000000007
        [ 5381.660447]  0000000000000000 ffff8806f6f2f528 ffffffff8104a9ec ffff8807038f36f0
        [ 5381.660452]  0000000000000000 0000000000000206 ffff8807038f2490 ffff8807038f36f0
        [ 5381.660457] Call Trace:
        [ 5381.660464]  [<ffffffff81777434>] dump_stack+0x4e/0x68
        [ 5381.660471]  [<ffffffff8104a9ec>] warn_slowpath_common+0x8c/0xc0
        [ 5381.660476]  [<ffffffff8104aa3a>] warn_slowpath_null+0x1a/0x20
        [ 5381.660480]  [<ffffffff81144995>] __alloc_pages_nodemask+0x365/0xad0
        [ 5381.660487]  [<ffffffff8108313f>] ? local_clock+0x4f/0x60
        [ 5381.660491]  [<ffffffff811430e8>] ? free_one_page+0x98/0x440
        [ 5381.660495]  [<ffffffff8108313f>] ? local_clock+0x4f/0x60
        [ 5381.660502]  [<ffffffff8113fae4>] ? __get_free_pages+0x14/0x50
        [ 5381.660508]  [<ffffffff81095fb8>] ? trace_hardirqs_off_caller+0x28/0xd0
        [ 5381.660515]  [<ffffffff81183caf>] alloc_pages_current+0x10f/0x1f0
        [ 5381.660520]  [<ffffffff8113fae4>] ? __get_free_pages+0x14/0x50
        [ 5381.660524]  [<ffffffff8113fae4>] __get_free_pages+0x14/0x50
        [ 5381.660530]  [<ffffffff8115dace>] kmalloc_order_trace+0x3e/0x100
        [ 5381.660536]  [<ffffffff81191ea0>] __kmalloc_track_caller+0x220/0x230
        [ 5381.660560]  [<ffffffffa0729fdb>] ? fs_path_ensure_buf.part.12+0x6b/0x200 [btrfs]
        [ 5381.660564]  [<ffffffff8178085c>] ? retint_restore_args+0xe/0xe
        [ 5381.660569]  [<ffffffff811580ef>] krealloc+0x6f/0xb0
        [ 5381.660586]  [<ffffffffa0729fdb>] fs_path_ensure_buf.part.12+0x6b/0x200 [btrfs]
        [ 5381.660601]  [<ffffffffa072a208>] fs_path_prepare_for_add+0x98/0xb0 [btrfs]
        [ 5381.660615]  [<ffffffffa072a2bc>] fs_path_add_path+0x2c/0x60 [btrfs]
        [ 5381.660628]  [<ffffffffa072c55c>] get_cur_path+0x7c/0x1c0 [btrfs]
      
      Even without this loop, the incremental send couldn't succeed, because it would attempt
      to send a rename/move operation for the lower inode before the highest inode number was
      renamed/move. This issue is easy to trigger with the following steps:
      
        $ mkfs.btrfs -f /dev/sdb3
        $ mount /dev/sdb3 /mnt/btrfs
        $ mkdir -p /mnt/btrfs/a/b/c/d
        $ mkdir /mnt/btrfs/a/b/c2
        $ btrfs subvol snapshot -r /mnt/btrfs /mnt/btrfs/snap1
        $ mv /mnt/btrfs/a/b/c/d /mnt/btrfs/a/b/c2/d2
        $ mv /mnt/btrfs/a/b/c /mnt/btrfs/a/b/c2/d2/cc
        $ btrfs subvol snapshot -r /mnt/btrfs /mnt/btrfs/snap2
        $ btrfs send -p /mnt/btrfs/snap1 /mnt/btrfs/snap2 > /tmp/incremental.send
      
      The structure of the filesystem when the first snapshot is taken is:
      
      	 .                       (ino 256)
      	 |-- a                   (ino 257)
      	     |-- b               (ino 258)
      	         |-- c           (ino 259)
      	         |   |-- d       (ino 260)
                       |
      	         |-- c2          (ino 261)
      
      And its structure when the second snapshot is taken is:
      
      	 .                       (ino 256)
      	 |-- a                   (ino 257)
      	     |-- b               (ino 258)
      	         |-- c2          (ino 261)
      	             |-- d2      (ino 260)
      	                 |-- cc  (ino 259)
      
      Before the move/rename operation is performed for the inode 259, the
      move/rename for inode 260 must be performed, since 259 is now a child
      of 260.
      
      A test case for xfstests, with a more complex scenario, will follow soon.
      Signed-off-by: default avatarFilipe David Borba Manana <fdmanana@gmail.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      9f03740a
  2. 28 Jan, 2014 31 commits
    • Anand Jain's avatar
      btrfs: undo sysfs when open_ctree() fails · 2365dd3c
      Anand Jain authored
      reproducer:
      mkfs.btrfs -f /dev/sdb &&\
      mount /dev/sdb /btrfs &&\
      btrfs dev add -f /dev/sdc /btrfs &&\
      umount /btrfs &&\
      wipefs -a /dev/sdc &&\
      mount -o degraded /dev/sdb /btrfs
      //above mount fails so try with RO
      mount -o degraded,ro /dev/sdb /btrfs
      
      ------
      sysfs: cannot create duplicate filename '/fs/btrfs/3f48c79e-5ed0-4e87-b189-86e749e503f4'
      ::
      
      dump_stack+0x49/0x5e
      warn_slowpath_common+0x87/0xb0
      warn_slowpath_fmt+0x41/0x50
      strlcat+0x69/0x80
      sysfs_warn_dup+0x87/0xa0
      sysfs_add_one+0x40/0x50
      create_dir+0x76/0xc0
      sysfs_create_dir_ns+0x7a/0xc0
      kobject_add_internal+0xad/0x220
      kobject_add_varg+0x38/0x60
      kobject_init_and_add+0x53/0x70
      mutex_lock+0x11/0x40
      __free_pages+0x25/0x30
      free_pages+0x41/0x50
      selinux_sb_copy_data+0x14e/0x1e0
      mount_fs+0x3e/0x1a0
      vfs_kern_mount+0x71/0x120
      do_mount+0x3f7/0x980
      SyS_mount+0x8b/0xe0
      system_call_fastpath+0x16/0x1b
      ------
      
      further 'modprobe -r btrfs' fails as well
      Signed-off-by: default avatarAnand Jain <anand.jain@oracle.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      2365dd3c
    • Filipe David Borba Manana's avatar
      Btrfs: fix snprintf usage by send's gen_unique_name · f74b86d8
      Filipe David Borba Manana authored
      The buffer size argument passed to snprintf must account for the
      trailing null byte added by snprintf, and it returns a value >= then
      sizeof(buffer) when the string can't fit in the buffer.
      
      Since our buffer has a size of 64 characters, and the maximum orphan
      name we can generate is 63 characters wide, we must pass 64 as the
      buffer size to snprintf, and not 63.
      Signed-off-by: default avatarFilipe David Borba Manana <fdmanana@gmail.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.cz>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      f74b86d8
    • Justin Maggard's avatar
      btrfs: fix defrag 32-bit integer overflow · c41570c9
      Justin Maggard authored
      When defragging a very large file, the cluster variable can wrap its 32-bit
      signed int type and become negative, which eventually gets passed to
      btrfs_force_ra() as a very large unsigned long value.  On 32-bit platforms,
      this eventually results in an Oops from the SLAB allocator.
      
      Change the cluster and max_cluster signed int variables to unsigned long to
      match the readahead functions.  This also allows the min() comparison in
      btrfs_defrag_file() to work as intended.
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      c41570c9
    • David Sterba's avatar
      btrfs: sysfs: list the NO_HOLES feature · c736c095
      David Sterba authored
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.cz>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      c736c095
    • David Sterba's avatar
      btrfs: sysfs: don't show reserved incompat feature · 66b4bbd4
      David Sterba authored
      The COMPRESS_LZOv2 incompat featue is currently not implemented, the bit
      is only reserved, no point to list it in sysfs.
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.cz>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      66b4bbd4
    • David Sterba's avatar
      btrfs: call permission checks earlier in ioctls and return EPERM · bd60ea0f
      David Sterba authored
      The owner and capability checks in IOC_SUBVOL_SETFLAGS and
      SET_RECEIVED_SUBVOL should be called before any other checks are done.
      
      Also unify the error code to EPERM.
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.cz>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      bd60ea0f
    • David Sterba's avatar
      btrfs: restrict snapshotting to own subvolumes · d0242061
      David Sterba authored
      Currently, any user can snapshot any subvolume if the path is accessible and
      thus indirectly create and keep files he does not own under his direcotries.
      This is not possible with traditional directories.
      
      In security context, a user can snapshot root filesystem and pin any
      potentially buggy binaries, even if the updates are applied.
      
      All the snapshots are visible to the administrator, so it's possible to
      verify if there are suspicious snapshots.
      
      Another more practical problem is that any user can pin the space used
      by eg. root and cause ENOSPC.
      
      Original report:
      https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/484786
      
      CC: stable@vger.kernel.org
      Signed-off-by: default avatarDavid Sterba <dsterba@suse.cz>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      d0242061
    • Miao Xie's avatar
      Btrfs: fix wrong block group in trace during the free space allocation · 89d4346a
      Miao Xie authored
      We allocate the free space from the former block group, not the current
      one, so should use the former one to output the trace information.
      Signed-off-by: default avatarMiao Xie <miaox@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      89d4346a
    • Miao Xie's avatar
      Btrfs: cleanup the code of used_block_group in find_free_extent() · 215a63d1
      Miao Xie authored
      used_block_group is just used for the space cluster which doesn't
      belong to the current block group, the other place needn't use it.
      Or the logic of code seems unclear.
      Signed-off-by: default avatarMiao Xie <miaox@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      215a63d1
    • Miao Xie's avatar
      920e4a58
    • Miao Xie's avatar
      Btrfs: change the members' order of btrfs_space_info structure to reduce the cache miss · 26b47ff6
      Miao Xie authored
      It is better that the position of the lock is close to the data which is
      protected by it, because they may be in the same cache line, we will load
      less cache lines when we access them. So we rearrange the members' position
      of btrfs_space_info structure to make the lock be closer to the its data.
      Signed-off-by: default avatarMiao Xie <miaox@cn.fujitsu.com>
      Reviewed-by: default avatarDavid Sterba <dsterba@suse.cz>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      26b47ff6
    • Wang Shilong's avatar
      Btrfs: fix wrong search path initialization before searching tree root · ffcfaf81
      Wang Shilong authored
      To search tree root without transaction protection, we should neither search commit
      root nor skip locking here, fix it.
      Signed-off-by: default avatarWang Shilong <wangsl.fnst@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      ffcfaf81
    • Miao Xie's avatar
      Btrfs: flush the dirty pages of the ordered extent aggressively during logging csum · 23c671a5
      Miao Xie authored
      The performance of fsync dropped down suddenly sometimes, the main reason
      of this problem was that we might only flush part dirty pages in a ordered
      extent, then got that ordered extent, wait for the csum calcucation. But if
      no task flushed the left part, we would wait until the flusher flushed them,
      sometimes we need wait for several seconds, it made the performance drop
      down suddenly. (On my box, it drop down from 56MB/s to 4-10MB/s)
      
      This patch improves the above problem by flushing left dirty pages aggressively.
      
      Test Environment:
      CPU:		2CPU * 2Cores
      Memory:		4GB
      Partition:	20GB(HDD)
      
      Test Command:
       # sysbench --num-threads=8 --test=fileio --file-num=1 \
       > --file-total-size=8G --file-block-size=32768 \
       > --file-io-mode=sync --file-fsync-freq=100 \
       > --file-fsync-end=no --max-requests=10000 \
       > --file-test-mode=rndwr run
      Signed-off-by: default avatarMiao Xie <miaox@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      23c671a5
    • Wang Shilong's avatar
      Btrfs: fix transaction abortion when remounting btrfs from RW to RO · 2c21b4d7
      Wang Shilong authored
      Steps to reproduce:
       # mkfs.btrfs -f /dev/sda8
       # mount /dev/sda8 /mnt -o flushoncommit
       # dd if=/dev/zero of=/mnt/data bs=4k count=102400 &
       # mount /dev/sda8 /mnt -o remount, ro
      
      When remounting RW to RO, the logic is to firstly set flag
      to RO and then commit transaction, however with option
      flushoncommit enabled,we will do RO check within committing
      transaction, so we get a transaction abortion here.
      
      Actually,here check is wrong, we should check if FS_STATE_ERROR
      is set, fix it.
      Reported-by: default avatarQu Wenruo <quwenruo@cn.fujitsu.com>
      Suggested-by: default avatarMiao Xie <miaox@cn.fujitsu.com>
      Signed-off-by: default avatarWang Shilong <wangsl.fnst@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      2c21b4d7
    • Filipe David Borba Manana's avatar
      Btrfs: faster file extent item search in clone ioctl · e4355f34
      Filipe David Borba Manana authored
      When we are looking for file extent items that intersect the cloning
      range, for each one that falls completely outside the range, don't
      release the path and do another full tree search - just move on
      to the next slot and copy the file extent item into our buffer only
      if the item intersects the cloning range.
      Signed-off-by: default avatarFilipe David Borba Manana <fdmanana@gmail.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      e4355f34
    • Liu Bo's avatar
      Btrfs: fix extent state leak on transaction abortion · 1a4319cc
      Liu Bo authored
      When transaction is aborted, we fail to commit transaction, instead we do
      cleanup work.  After that when we umount btrfs, we get to free fs roots' log
      trees respectively, but that happens after we unpin extents, so those extents
      pinned by freeing log trees will remain in memory and lead to the leak.
      Signed-off-by: default avatarLiu Bo <bo.li.liu@oracle.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      1a4319cc
    • Qu Wenruo's avatar
      btrfs: Cleanup the btrfs_parse_options for remount. · 07802534
      Qu Wenruo authored
      Since remount will pending the new mount options to the original mount
      options, which will make btrfs_parse_options check the old options then
      new options, causing some stupid output like "enabling XXX" following by
      "disable XXX".
      
      This patch will add extra check before every btrfs_info to skip the
      output from old options checking.
      Signed-off-by: default avatarQu Wenruo <quwenruo@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      07802534
    • Qu Wenruo's avatar
      btrfs: Add noinode_cache mount option · 3818aea2
      Qu Wenruo authored
      Add noinode_cache mount option for btrfs.
      
      Since inode map cache involves all the btrfs_find_free_ino/return_ino
      things and if just trigger the mount_opt,
      an inode number get from inode map cache will not returned to inode map
      cache.
      
      To keep the find and return inode both in the same behavior,
      a new bit in mount_opt, CHANGE_INODE_CACHE, is introduced for this idea.
      CHANGE_INODE_CACHE is set/cleared in remounting, and the original
      INODE_MAP_CACHE is set/cleared according to CHANGE_INODE_CACHE after a
      success transaction.
      Since find/return inode is all done between btrfs_start_transaction and
      btrfs_commit_transaction, this will keep consistent behavior.
      
      Also noinode_cache mount option will not stop the caching_kthread.
      
      Cc: David Sterba <dsterba@suse.cz>
      Signed-off-by: default avatarMiao Xie <miaox@cn.fujitsu.com>
      Signed-off-by: default avatarQu Wenruo <quwenruo@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      3818aea2
    • Wang Shilong's avatar
      Btrfs: fix to search previous metadata extent item since skinny metadata · ade2e0b3
      Wang Shilong authored
      There is a bug that using btrfs_previous_item() to search metadata extent item.
      This is because in btrfs_previous_item(), we need type match, however, since
      skinny metada was introduced by josef, we may mix this two types. So just
      use btrfs_previous_item() is not working right.
      
      To keep btrfs_previous_item() like normal tree search, i introduce another
      function btrfs_previous_extent_item().
      Signed-off-by: default avatarWang Shilong <wangsl.fnst@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      ade2e0b3
    • Wang Shilong's avatar
      Btrfs: fix missing skinny metadata check in scrub_stripe() · 7c76edb7
      Wang Shilong authored
      Check if we support skinny metadata firstly and fix to use
      right type to search.
      Signed-off-by: default avatarWang Shilong <wangsl.fnst@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      7c76edb7
    • Filipe David Borba Manana's avatar
      Btrfs: fix send to not send non-aligned clone operations · 28e5dd8f
      Filipe David Borba Manana authored
      It is possible for the send feature to send clone operations that
      request a cloning range (offset + length) that is not aligned with
      the block size. This makes the btrfs receive command send issue a
      clone ioctl call that will fail, as the ioctl will return an -EINVAL
      error because of the unaligned range.
      
      Fix this by not sending clone operations for non block aligned ranges,
      and instead send regular write operation for these (less common) cases.
      
      The following xfstest reproduces this issue, which fails on the second
      btrfs receive command without this change:
      
        seq=`basename $0`
        seqres=$RESULT_DIR/$seq
        echo "QA output created by $seq"
      
        tmp=`mktemp -d`
      
        status=1	# failure is the default!
        trap "_cleanup; exit \$status" 0 1 2 3 15
      
        _cleanup()
        {
            rm -fr $tmp
        }
      
        # get standard environment, filters and checks
        . ./common/rc
        . ./common/filter
      
        # real QA test starts here
        _supported_fs btrfs
        _supported_os Linux
        _require_scratch
        _need_to_be_root
      
        rm -f $seqres.full
      
        _scratch_mkfs >/dev/null 2>&1
        _scratch_mount
      
        $XFS_IO_PROG -f -c "truncate 819200" $SCRATCH_MNT/foo | _filter_xfs_io
        $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT | _filter_scratch
      
        $XFS_IO_PROG -c "falloc -k 819200 667648" $SCRATCH_MNT/foo | _filter_xfs_io
        $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT | _filter_scratch
      
        $XFS_IO_PROG -f -c "pwrite 1482752 2978" $SCRATCH_MNT/foo | _filter_xfs_io
        $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT | _filter_scratch
      
        $BTRFS_UTIL_PROG subvol snapshot -r $SCRATCH_MNT $SCRATCH_MNT/mysnap1 | \
            _filter_scratch
      
        $XFS_IO_PROG -f -c "truncate 883305" $SCRATCH_MNT/foo | _filter_xfs_io
        $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT | _filter_scratch
      
        $BTRFS_UTIL_PROG subvol snapshot -r $SCRATCH_MNT $SCRATCH_MNT/mysnap2 | \
            _filter_scratch
      
        $BTRFS_UTIL_PROG send $SCRATCH_MNT/mysnap1 -f $tmp/1.snap 2>&1 | _filter_scratch
        $BTRFS_UTIL_PROG send -p $SCRATCH_MNT/mysnap1 $SCRATCH_MNT/mysnap2 \
            -f $tmp/2.snap 2>&1 | _filter_scratch
      
        md5sum $SCRATCH_MNT/foo | _filter_scratch
        md5sum $SCRATCH_MNT/mysnap1/foo | _filter_scratch
        md5sum $SCRATCH_MNT/mysnap2/foo | _filter_scratch
      
        _scratch_unmount
        _check_btrfs_filesystem $SCRATCH_DEV
        _scratch_mkfs >/dev/null 2>&1
        _scratch_mount
      
        $BTRFS_UTIL_PROG receive $SCRATCH_MNT -f $tmp/1.snap
        md5sum $SCRATCH_MNT/mysnap1/foo | _filter_scratch
      
        $BTRFS_UTIL_PROG receive $SCRATCH_MNT -f $tmp/2.snap
        md5sum $SCRATCH_MNT/mysnap2/foo | _filter_scratch
      
        _scratch_unmount
        _check_btrfs_filesystem $SCRATCH_DEV
      
        status=0
        exit
      
      The tests expected output is:
      
        QA output created by 025
        FSSync 'SCRATCH_MNT'
        FSSync 'SCRATCH_MNT'
        wrote 2978/2978 bytes at offset 1482752
        XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
        FSSync 'SCRATCH_MNT'
        Create a readonly snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/mysnap1'
        FSSync 'SCRATCH_MNT'
        Create a readonly snapshot of 'SCRATCH_MNT' in 'SCRATCH_MNT/mysnap2'
        At subvol SCRATCH_MNT/mysnap1
        At subvol SCRATCH_MNT/mysnap2
        129b8eaee8d3c2bcad49bec596591cb3  SCRATCH_MNT/foo
        42b6369eae2a8725c1aacc0440e597aa  SCRATCH_MNT/mysnap1/foo
        129b8eaee8d3c2bcad49bec596591cb3  SCRATCH_MNT/mysnap2/foo
        At subvol mysnap1
        42b6369eae2a8725c1aacc0440e597aa  SCRATCH_MNT/mysnap1/foo
        At snapshot mysnap2
        129b8eaee8d3c2bcad49bec596591cb3  SCRATCH_MNT/mysnap2/foo
      Signed-off-by: default avatarFilipe David Borba Manana <fdmanana@gmail.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      28e5dd8f
    • Filipe David Borba Manana's avatar
      Btrfs: fix btrfs boot when compiled as built-in · 14a958e6
      Filipe David Borba Manana authored
      After the change titled "Btrfs: add support for inode properties", if
      btrfs was built-in the kernel (i.e. not as a module), it would cause a
      kernel panic, as reported recently by Fengguang:
      
      [    2.024722] BUG: unable to handle kernel NULL pointer dereference at           (null)
      [    2.027814] IP: [<ffffffff81501594>] crc32c+0xc/0x6b
      [    2.028684] PGD 0
      [    2.028684] Oops: 0000 [#1] SMP
      [    2.028684] Modules linked in:
      [    2.028684] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.13.0-rc7-04795-ga7b57c2 #1
      [    2.028684] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
      [    2.028684] task: ffff88000edba100 ti: ffff88000edd6000 task.ti: ffff88000edd6000
      [    2.028684] RIP: 0010:[<ffffffff81501594>]  [<ffffffff81501594>] crc32c+0xc/0x6b
      [    2.028684] RSP: 0000:ffff88000edd7e58  EFLAGS: 00010246
      [    2.028684] RAX: 0000000000000000 RBX: ffffffff82295550 RCX: 0000000000000000
      [    2.028684] RDX: 0000000000000011 RSI: ffffffff81efe393 RDI: 00000000fffffffe
      [    2.028684] RBP: ffff88000edd7e60 R08: 0000000000000003 R09: 0000000000015d20
      [    2.028684] R10: ffffffff81ef225e R11: ffffffff811b0222 R12: ffffffffffffffff
      [    2.028684] R13: 0000000000000239 R14: 0000000000000000 R15: 0000000000000000
      [    2.028684] FS:  0000000000000000(0000) GS:ffff88000fa00000(0000) knlGS:0000000000000000
      [    2.028684] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
      [    2.028684] CR2: 0000000000000000 CR3: 000000000220c000 CR4: 00000000000006f0
      [    2.028684] Stack:
      [    2.028684]  ffffffff82295550 ffff88000edd7e80 ffffffff8238af62 ffffffff8238ac05
      [    2.028684]  0000000000000000 ffff88000edd7e98 ffffffff8238ac0f ffffffff8238ac05
      [    2.028684]  ffff88000edd7f08 ffffffff810002ba ffff88000edd7f00 ffffffff810e2404
      [    2.028684] Call Trace:
      [    2.028684]  [<ffffffff8238af62>] btrfs_props_init+0x4f/0x96
      [    2.028684]  [<ffffffff8238ac05>] ? ftrace_define_fields_btrfs_space_reservation+0x145/0x145
      [    2.028684]  [<ffffffff8238ac0f>] init_btrfs_fs+0xa/0xf0
      [    2.028684]  [<ffffffff8238ac05>] ? ftrace_define_fields_btrfs_space_reservation+0x145/0x145
      [    2.028684]  [<ffffffff810002ba>] do_one_initcall+0xa4/0x13a
      [    2.028684]  [<ffffffff810e2404>] ? parse_args+0x25f/0x33d
      [    2.028684]  [<ffffffff8234cf75>] kernel_init_freeable+0x1aa/0x230
      [    2.028684]  [<ffffffff8234c785>] ? do_early_param+0x88/0x88
      [    2.028684]  [<ffffffff819f61b5>] ? rest_init+0x89/0x89
      [    2.028684]  [<ffffffff819f61c3>] kernel_init+0xe/0x109
      
      The issue here is that the initialization function of btrfs (super.c:init_btrfs_fs)
      started using crc32c (from lib/libcrc32c.c). But when it needs to call crc32c (as
      part of the properties initialization routine), the libcrc32c is not yet initialized,
      so crc32c derreferenced a NULL pointer (lib/libcrc32c.c:tfm), causing the kernel
      panic on boot.
      
      The approach to fix this is to use crypto component directly to use its crc32c (which
      is basically what lib/libcrc32c.c is, a wrapper around crypto). This is what ext4 is
      doing as well, it uses crypto directly to get crc32c functionality.
      
      Verified this works both when btrfs is built-in and when it's loadable kernel module.
      Signed-off-by: default avatarFilipe David Borba Manana <fdmanana@gmail.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      14a958e6
    • Filipe David Borba Manana's avatar
      Btrfs: unlock inodes in correct order in clone ioctl · c57c2b3e
      Filipe David Borba Manana authored
      In the clone ioctl, when the source and target inodes are different,
      we can acquire their mutexes in 2 possible different orders. After
      we're done cloning, we were releasing the mutexes always in the same
      order - the most correct way of doing it is to release them by the
      reverse order they were acquired.
      Signed-off-by: default avatarFilipe David Borba Manana <fdmanana@gmail.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      c57c2b3e
    • Wang Shilong's avatar
      Btrfs: optimize to remove unnecessary removal with ulist reallocation · f499e40f
      Wang Shilong authored
      Here we are not going to free memory, no need to remove every node
      one by one, just init root node here is ok.
      
      Cc:  Liu Bo <bo.li.liu@oracle.com>
      Signed-off-by: default avatarWang Shilong <wangsl.fnst@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      f499e40f
    • Liu Bo's avatar
      Btrfs: release subvolume's block_rsv before transaction commit · de6e8200
      Liu Bo authored
      We don't have to keep subvolume's block_rsv during transaction commit,
      and within transaction commit, we may also need the free space reclaimed
      from this block_rsv to process delayed refs.
      Signed-off-by: default avatarLiu Bo <bo.li.liu@oracle.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      de6e8200
    • Miao Xie's avatar
      Btrfs: fix the race between write back and nocow buffered write · f1de9683
      Miao Xie authored
      When we ran the 274th case of xfstests with nodatacow mount option,
      We met the following warning message:
      WARNING: CPU: 1 PID: 14185 at fs/btrfs/extent-tree.c:3734 btrfs_free_reserved_data_space+0xa6/0xd0
      
      It is caused by the race between the write back and nocow buffered
      write:
        Task1				Task2
        __btrfs_buffered_write()
          skip data reservation
          reserve the metadata space
          copy the data
          dirty the pages
          unlock the pages
      				write back the pages
      				release the data space
         				  becasue there is no
      				  noreserve flag
         set the noreserve flag
      
      This patch fixes this problem by unlocking the pages after
      the noreserve flag is set.
      Reported-by: default avatarTsutomu Itoh <t-itoh@jp.fujitsu.com>
      Signed-off-by: default avatarMiao Xie <miaox@cn.fujitsu.com>
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      f1de9683
    • Josef Bacik's avatar
      Btrfs: only process as many file extents as there are refs · 7ef81ac8
      Josef Bacik authored
      The backref walking code will search down to the key it is looking for and then
      proceed to walk _all_ of the extents on the file until it hits the end.  This is
      suboptimal with large files, we only need to look for as many extents as we have
      references for that inode.  I have a testcase that creates a randomly written 4
      gig file and before this patch it took 6min 30sec to do the initial send, with
      this patch it takes 2min 30sec to do the intial send.  Thanks,
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      7ef81ac8
    • Josef Bacik's avatar
      Btrfs: fix qgroup rescan to work with skinny metadata · 3a6d75e8
      Josef Bacik authored
      Could have sworn I fixed this before but apparently not.  This makes us pass
      btrfs/022 with skinny metadata enabled.  Thanks,
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      3a6d75e8
    • Josef Bacik's avatar
      Btrfs: fix extent_from_logical to deal with skinny metadata · 580f0a67
      Josef Bacik authored
      I don't think this is an issue and I've not seen it in practice but
      extent_from_logical will fail to find a skinny extent because it uses
      btrfs_previous_item and gives it the normal extent item type.  This is just not
      a place to use btrfs_previous_item since we care about either normal extents or
      skinny extents, so open code btrfs_previous_item to properly check.  This would
      only affect metadata and the only place this is used for metadata is scrub and
      I'm pretty sure it's just for printing stuff out, not actually doing any work so
      hopefully it was never a problem other than a cosmetic one.  Thanks,
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      580f0a67
    • Josef Bacik's avatar
      Btrfs: throttle delayed refs better · 0a2b2a84
      Josef Bacik authored
      On one of our gluster clusters we noticed some pretty big lag spikes.  This
      turned out to be because our transaction commit was taking like 3 minutes to
      complete.  This is because we have like 30 gigs of metadata, so our global
      reserve would end up being the max which is like 512 mb.  So our throttling code
      would allow a ridiculous amount of delayed refs to build up and then they'd all
      get run at transaction commit time, and for a cold mounted file system that
      could take up to 3 minutes to run.  So fix the throttling to be based on both
      the size of the global reserve and how long it takes us to run delayed refs.
      This patch tracks the time it takes to run delayed refs and then only allows 1
      seconds worth of outstanding delayed refs at a time.  This way it will auto-tune
      itself from cold cache up to when everything is in memory and it no longer has
      to go to disk.  This makes our transaction commits take much less time to run.
      Thanks,
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      0a2b2a84
    • Josef Bacik's avatar
      Btrfs: attach delayed ref updates to delayed ref heads · d7df2c79
      Josef Bacik authored
      Currently we have two rb-trees, one for delayed ref heads and one for all of the
      delayed refs, including the delayed ref heads.  When we process the delayed refs
      we have to hold onto the delayed ref lock for all of the selecting and merging
      and such, which results in quite a bit of lock contention.  This was solved by
      having a waitqueue and only one flusher at a time, however this hurts if we get
      a lot of delayed refs queued up.
      
      So instead just have an rb tree for the delayed ref heads, and then attach the
      delayed ref updates to an rb tree that is per delayed ref head.  Then we only
      need to take the delayed ref lock when adding new delayed refs and when
      selecting a delayed ref head to process, all the rest of the time we deal with a
      per delayed ref head lock which will be much less contentious.
      
      The locking rules for this get a little more complicated since we have to lock
      up to 3 things to properly process delayed refs, but I will address that problem
      later.  For now this passes all of xfstests and my overnight stress tests.
      Thanks,
      Signed-off-by: default avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: default avatarChris Mason <clm@fb.com>
      d7df2c79