• David Chinner's avatar
    [XFS] Concurrent Multi-File Data Streams · 2a82b8be
    David Chinner authored
    In media spaces, video is often stored in a frame-per-file format. When
    dealing with uncompressed realtime HD video streams in this format, it is
    crucial that files do not get fragmented and that multiple files a placed
    contiguously on disk.
    
    When multiple streams are being ingested and played out at the same time,
    it is critical that the filesystem does not cross the streams and
    interleave them together as this creates seek and readahead cache miss
    latency and prevents both ingest and playout from meeting frame rate
    targets.
    
    This patch set creates a "stream of files" concept into the allocator to
    place all the data from a single stream contiguously on disk so that RAID
    array readahead can be used effectively. Each additional stream gets
    placed in different allocation groups within the filesystem, thereby
    ensuring that we don't cross any streams. When an AG fills up, we select a
    new AG for the stream that is not in use.
    
    The core of the functionality is the stream tracking - each inode that we
    create in a directory needs to be associated with the directories' stream.
    Hence every time we create a file, we look up the directories' stream
    object and associate the new file with that object.
    
    Once we have a stream object for a file, we use the AG that the stream
    object point to for allocations. If we can't allocate in that AG (e.g. it
    is full) we move the entire stream to another AG. Other inodes in the same
    stream are moved to the new AG on their next allocation (i.e. lazy
    update).
    
    Stream objects are kept in a cache and hold a reference on the inode.
    Hence the inode cannot be reclaimed while there is an outstanding stream
    reference. This means that on unlink we need to remove the stream
    association and we also need to flush all the associations on certain
    events that want to reclaim all unreferenced inodes (e.g. filesystem
    freeze).
    
    SGI-PV: 964469
    SGI-Modid: xfs-linux-melb:xfs-kern:29096a
    Signed-off-by: default avatarDavid Chinner <dgc@sgi.com>
    Signed-off-by: default avatarBarry Naujok <bnaujok@sgi.com>
    Signed-off-by: default avatarDonald Douwsma <donaldd@sgi.com>
    Signed-off-by: default avatarChristoph Hellwig <hch@infradead.org>
    Signed-off-by: default avatarTim Shimmin <tes@sgi.com>
    Signed-off-by: default avatarVlad Apostolov <vapo@sgi.com>
    2a82b8be
xfs_filestream.h 4.77 KB