- 12 Oct, 2023 40 commits
-
-
Boris Burkov authored
Consider the following sequence: - enable quotas - create subvol S id 256 at dir outer/ - create a qgroup 1/100 - add 0/256 (S's auto qgroup) to 1/100 - create subvol T id 257 at dir outer/inner/ With full qgroups, there is no relationship between 0/257 and either of 0/256 or 1/100. There is an inherit feature that the creator of inner/ can use to specify it ought to be in 1/100. Simple quotas are targeted at container isolation, where such automatic inheritance for not necessarily trusted/controlled nested subvol creation would be quite helpful. Therefore, add a new default behavior for simple quotas: when you create a nested subvol, automatically inherit as parents any parents of the qgroup of the subvol the new inode is going in. In our example, 257/0 would also be under 1/100, allowing easy control of a total quota over an arbitrary hierarchy of subvolumes. I think this _might_ be a generally useful behavior, so it could be interesting to put it behind a new inheritance flag that simple quotas always use while traditional quotas let the user specify, but this is a minimally intrusive change to start. Signed-off-by:
Boris Burkov <boris@bur.io> Signed-off-by:
David Sterba <dsterba@suse.com>
-
Boris Burkov authored
At the moment that we run delayed refs, we make the final ref-count based decision on creating/removing extent (and metadata) items. Therefore, it is exactly the spot to hook up simple quotas. There are a few important subtleties to the fields we must collect to accurately track simple quotas, particularly when removing an extent. When removing a data extent, the ref could be in any tree (due to reflink, for example) and so we need to recover the owning root id from the owner ref item. When removing a metadata extent, we know the owning root from the owner field in the header when we create the delayed ref, so we can recover it from there. We must also be careful to handle reservations properly to not leaked reserved space. The happy path is freeing the reservation when the simple quota delta runs on a data extent. If that doesn't happen, due to refs canceling out or some error, the ref head already has the must_insert_reserved machinery to handle this, so we piggy back on that and use it to clean up the reserved data. Signed-off-by:
-
Boris Burkov authored
Inline ref parsing is a bit tricky and relies on a decent amount of implicit information, so I think it is beneficial to have a helper function for reading the owner ref, if only to "document" the format, along with the write path. The main subtlety of note which I was missing by open-coding this was that it is important to check whether or not inline refs are present *at all*. i.e., if we are writing out a new extent under squotas, we will always use a big enough item for the inline ref and have it. However, it is possible that some random item predating squotas will not have any inline refs. In that case, trying to read the "type" field of the first inline ref will just be reading garbage in the form of whatever is in the next item. This will be used by the extent free-ing path, which looks up data extent owners as well as a relocation path which needs to grab the owner before relocating an extent. Signed-off-by:
Josef Bacik <josef@toxicpanda.com> Signed-off-by:
-
Boris Burkov authored
In order to implement simple quota groups, we need to be able to associate a data extent with the subvolume that created it. Once you account for reflink, this information cannot be recovered without explicitly storing it. Options for storing it are: - a new key/item - a new extent inline ref item The former is backwards compatible, but wastes space, the latter is incompat, but is efficient in space and reuses the existing inline ref machinery, while only abusing it a tiny amount -- specifically, the new item is not a ref, per-se. Signed-off-by:
-
Boris Burkov authored
Simple quotas requires tracking the original creating root of any given extent. This gets complicated when multiple subvolumes create overlapping/contradictory refs in the same transaction. For example, due to modifying or deleting an extent while also snapshotting it. To resolve this in a general way, take advantage of the fact that we are essentially already tracking this for handling releasing reservations. The head ref coalesces the various refs and uses must_insert_reserved to check if it needs to create an extent/free reservation. Store the ref that set must_insert_reserved as the owning ref on the head ref. Note that this can result in writing an extent for the very first time with an owner different from its only ref, but it will look the same as if you first created it with the original owning ref, then added the other ref, then removed the owning ref. Signed-off-by:
-
Boris Burkov authored
While data extents require us to store additional inline refs to track the original owner on free, this information is available implicitly for metadata. It is found in the owner field of the header of the tree block. Even if other trees refer to this block and the original ref goes away, we will not rewrite that header field, so it will reliably give the original owner. In addition, there is a relocation case where a new data extent needs to have an owning root separate from the referring root wired through delayed refs. To use it for recording simple quota deltas, we need to wire this root id through from when we create the delayed ref until we fully process it. Store it in the generic btrfs_ref struct of the delayed ref. Signed-off-by:
-
Boris Burkov authored
commit 113479d5 ("btrfs: rename root fields in delayed refs structs") changed these from ref_root to owning_root. However, there are many circumstances where that name is not really accurate and the root on the ref struct _is_ the referring root. In general, these are not the owning root, though it does happen in some ref merging cases involving overwrites during snapshots and similar. Simple quotas cares quite a bit about tracking the original owner of an extent through delayed refs, so rename these back to free up the name for the real owning root (which will live on the generic btrfs_ref and the head ref) Reviewed-by:
-
Boris Burkov authored
Rather than re-computing shared/exclusive ownership based on backrefs and walking roots for implicit backrefs, simple quotas does an increment when creating an extent and a decrement when deleting it. Add the API for the extent item code to use to track those events. Signed-off-by:
-
Boris Burkov authored
Pull creating the qgroup earlier in the snapshot. This allows simple quotas qgroups to see all the metadata writes related to the snapshot being created and to be born with the root node accounted. Note this has an impact on transaction commit where the qgroup creation can do a lot of work, allocate memory and take locks. The change is done for correctness, potential performance issues will be fixed in the future. Signed-off-by:
-
Boris Burkov authored
The following sequence: enable simple quotas do some writes reserve space create ordered_extent release rsv (store rsv_bytes in OE, mark QGROUP_RESERVED bits) disable quotas enable simple quotas set qgroup rsv to 0 on all subvolumes ordered_extent finishes create delayed ref with rsv_bytes from before run delayed ref record_simple_quota_delta free rsv_bytes (0 -> -rsv_delta) results in us reliably underflowing the subvolume's qgroup rsv counter, because disabling/re-enabling quotas toggles reservation counters down to 0, but does not remove other file system state which represents successful acquisition of qgroup rsv space. Specifically metadata rsv counters on the root object and rsv_bytes on ordered_extent objects that have released their reservation as well as the corresponding QGROUP_RESERVED extent bits. Normal qgroups gets away with this, I believe because it forces more work to happen on transaction commit, but I am not certain it is totally safe from the ordered_extent/leaked extent bit variant. Simple quotas hits this reliably. The intent of the fix is to make disable take the time to clear that external to qgroups state as well: after flipping off the quota bit on fs_info, flush delalloc and ordered extents, clearing the extent bits along the way. This makes it so there are no ordered extents or meta prealloc hanging around from the first enablement period during the second. Reviewed-by: -
Boris Burkov authored
Add an entry in the features directory for the new incompat flag Signed-off-by:
-
Boris Burkov authored
Add a new sysfs file /sys/fs/btrfs/<uuid>/qgroups/mode which prints out the mode qgroups is running in. The possible modes are qgroup, and squota. If quotas are not enabled, then the qgroups directory will not exist, so don't handle that mode. Signed-off-by:
-
Boris Burkov authored
Add a new quota mode called "simple quotas". It can be enabled by the existing quota enable ioctl via a new command, and sets an incompat bit, as the implementation of simple quotas will make backwards incompatible changes to the disk format of the extent tree. Signed-off-by:
-
Boris Burkov authored
In preparation for introducing simple quotas, change from a binary setting for quotas to an enum based mode. Initially, the possible modes are disabled/full. Full quotas is normal btrfs qgroups. Reviewed-by:
-
David Sterba authored
There are two callbacks defined in btrfs_work but only two actually make use of them, otherwise there are NULLs. We can get rid of the freeing callback making it a special case of the normal work. This reduces the size of btrfs_work by 8 bytes, final layout: struct btrfs_work { btrfs_func_t func; /* 0 8 */ btrfs_ordered_func_t ordered_func; /* 8 8 */ struct work_struct normal_work; /* 16 32 */ struct list_head ordered_list; /* 48 16 */ /* --- cacheline 1 boundary (64 bytes) --- */ struct btrfs_workqueue * wq; /* 64 8 */ long unsigned int flags; /* 72 8 */ /* size: 80, cachelines: 2, members: 6 */ /* last cacheline: 16 bytes */ }; This in turn reduces size of other structures (on a release config): - async_chunk 160 -> 152 - async_submit_bio 152 -> 144 - btrfs_async_delayed_work 104 -> 96 - btrfs_caching_control 176 -> 168 - btrfs_delalloc_work 144 -> 136 - btrfs_fs_info 3608 -> 3600 - btrfs_ordered_extent 440 -> 424 - btrfs_writepage_fixup 104 -> 96 Signed-off-by: -
Johannes Thumshirn authored
Until the raid stripe tree code is well enough tested and feature complete, "hide" it behind CONFIG_BTRFS_DEBUG so only people who want to use it are actually using it. The scrub support may still fail some tests (btrfs/060 and up) and will be fixed, RAID5/6 is not supported. Reviewed-by:
Johannes Thumshirn <johannes.thumshirn@wdc.com> Reviewed-by:
-
Johannes Thumshirn authored
Add a tree checker support for RAID stripe tree items, verify: - alignment - presence of the incompat bit - supported encoding Signed-off-by:
-
Johannes Thumshirn authored
Add trace events for raid-stripe-tree operations. Signed-off-by:
-
Johannes Thumshirn authored
If a filesystem with a raid-stripe-tree is mounted, show the RST feature in sysfs, currently still under the CONFIG_BTRFS_DEBUG option. Reviewed-by:
-
Johannes Thumshirn authored
Decode raid-stripe-tree entries on btrfs_print_tree(). Signed-off-by:
-
Johannes Thumshirn authored
When we have a raid-stripe-tree, we can do RAID0/1/10 on zoned devices for data block groups. For metadata block groups, we don't actually need anything special, as all metadata I/O is protected by the btrfs_zoned_meta_io_lock() already. Signed-off-by:
-
Johannes Thumshirn authored
A filesystem that uses the raid stripe tree for logical to physical address translation can't use the regular scrub path, that reads all stripes and then checks if a sector is unused afterwards. When using the raid stripe tree, this will result in lookup errors, as the stripe tree doesn't know the requested logical addresses. In case we're scrubbing a filesystem which uses the RAID stripe tree for multi-device logical to physical address translation, perform an extra block mapping step to get the real on-disk stripe length from the stripe tree when scrubbing the sectors. This prevents a double completion of the btrfs_bio caused by splitting the underlying bio and ultimately a use-after-free. Signed-off-by:
-
Johannes Thumshirn authored
Lookup the physical address from the raid stripe tree when a read on an RAID volume formatted with the raid stripe tree was attempted. Signed-off-by:
-
Johannes Thumshirn authored
As each stripe extent is tied to an extent item, delete the stripe extent once the corresponding extent item is deleted. Signed-off-by:
-
Johannes Thumshirn authored
Add support for inserting stripe extents into the raid stripe tree on completion of every write that needs an extra logical-to-physical translation when using RAID. Inserting the stripe extents happens after the data I/O has completed, this is done to a) support zone-append and b) rule out the possibility of a RAID-write-hole. Signed-off-by:
-
Johannes Thumshirn authored
If we find the raid-stripe-tree on mount, read it from disk. This is a backward incompatible feature. The rescue=ignorebadroots mount option will skip this tree. Signed-off-by:
-
Johannes Thumshirn authored
Add definitions for the raid stripe tree. This tree will hold information about the on-disk layout of the stripes in a RAID set. Each stripe extent has a 1:1 relationship with an on-disk extent item and is doing the logical to per-drive physical address translation for the extent item in question. Signed-off-by:
-
Qu Wenruo authored
A long time ago, we had some metadata chunks which started at sector boundary but not aligned to nodesize boundary. This led to some older filesystems which can have tree blocks only aligned to sectorsize, but not nodesize. Later 'btrfs check' gained the ability to detect and warn about such tree blocks, and kernel fixed the chunk allocation behavior, nowadays those tree blocks should be pretty rare. But in the future, if we want to migrate metadata to folio, we cannot have such tree blocks, as filemap_add_folio() requires the page index to be aligned with the folio number of pages. Such unaligned tree blocks can lead to VM_BUG_ON(). So this patch adds extra warning for those unaligned tree blocks, as a preparation for the future folio migration. Reviewed-by:
Anand Jain <anand.jain@oracle.com> Signed-off-by:
Qu Wenruo <wqu@suse.com> Reviewed-by:
-
Josef Bacik authored
We have a random schedule_timeout() if the current transaction is committing, which seems to be a holdover from the original delalloc reservation code. Remove this, we have the proper flushing stuff, we shouldn't be hoping for random timing things to make everything work. This just induces latency for no reason. CC: stable@vger.kernel.org # 5.4+ Signed-off-by:
-
Filipe Manana authored
The comment on top of btrfs_pin_extent_for_log_replay() mentioning that the function must be called within a transaction is pointless as of commit 9fce5704 ("btrfs: Make btrfs_pin_extent_for_log_replay take transaction handle"), since the function now takes a transaction handle as its first argument. So remove the comment because it's completely useless now. Signed-off-by:
Filipe Manana <fdmanana@suse.com> Reviewed-by:
-
Filipe Manana authored
A comment at btrfs_free_extent() mentions the call to btrfs_pin_extent() unlocks the pinned mutex, however that mutex is long gone, it was removed in 2009 by commit 04018de5 ("Btrfs: kill the pinned_mutex"). So just delete the comment. Signed-off-by:
-
Christoph Hellwig authored
Split the code handling a type DUP block group from btrfs_load_block_group_zone_info to make the code more readable. Reviewed-by:
Christoph Hellwig <hch@lst.de> Reviewed-by:
-
Christoph Hellwig authored
Split the code handling a type single block group from btrfs_load_block_group_zone_info to make the code more readable. Reviewed-by:
-
Christoph Hellwig authored
Split out a helper for the body of the per-zone loop in btrfs_load_block_group_zone_info to make the function easier to read and modify. Reviewed-by:
-
Christoph Hellwig authored
Add a new zone_info structure to hold per-zone information in btrfs_load_block_group_zone_info and prepare for breaking out helpers from it. Reviewed-by:
-
Filipe Manana authored
When an extent is allocated or freed, we call btrfs_update_block_group() to update its block group and space info. An extent always belongs to a single block group, it can never span multiple block groups, so the loop we have at btrfs_update_block_group() is pointless, as it always has a single iteration. The loop was added in the very early days, 2007, when the block group code was added in commit 9078a3e1 ("Btrfs: start of block group code"), but even back then it seemed pointless. So remove the loop and assert the block group containing the start offset of the extent also contains the whole extent. Signed-off-by:
-
Filipe Manana authored
At btrfs_mark_buffer_dirty(), having a transaction id mismatch is never expected to happen and it usually means there's a bug or some memory corruption due to a bitflip for example. So mark the condition as unlikely to optimize code generation as well as to make it obvious for human readers that it is a very unexpected condition. Signed-off-by:
-
Filipe Manana authored
There's no need to use WARN() at btrfs_mark_buffer_dirty() to print an error message, as we have the fs_info pointer we can use btrfs_crit() which prints device information and makes the message have a more uniform format. As we are already aborting the transaction we already have a stack trace printed as well. So replace the use of WARN() with btrfs_crit(). Also slightly reword the message to use 'logical' instead of 'block' as it's what is used in other error/warning messages. Signed-off-by:
-
Filipe Manana authored
When marking an extent buffer as dirty, at btrfs_mark_buffer_dirty(), we check if its generation matches the running transaction and if not we just print a warning. Such mismatch is an indicator that something really went wrong and only printing a warning message (and stack trace) is not enough to prevent a corruption. Allowing a transaction to commit with such an extent buffer will trigger an error if we ever try to read it from disk due to a generation mismatch with its parent generation. So abort the current transaction with -EUCLEAN if we notice a generation mismatch. For this we need to pass a transaction handle to btrfs_mark_buffer_dirty() which is always available except in test code, in which case we can pass NULL since it operates on dummy extent buffers and all test roots have a single node/leaf (root node at level 0). Signed-off-by:
-
Anand Jain authored
After the commit 5f58d783 ("btrfs: free device in btrfs_close_devices for a single device filesystem") we unregister the device from the kernel memory upon unmounting for a single device. So, device registration that was performed before mounting if any is no longer in the kernel memory. However, in fact, note that device registration is unnecessary for a single-device btrfs filesystem unless it's a seed device. So for commands like 'btrfs device scan' or 'btrfs device ready' with a non-seed single-device btrfs filesystem, they can return success just after superblock verification and without the actual device scan. When 'device scan --forget' is called on such device no error is returned. The seed device must remain in the kernel memory to allow the sprout device to mount without the need to specify the seed device explicitly. Signed-off-by:
-
