Commit 29c8c4ac authored by Mauro Carvalho Chehab's avatar Mauro Carvalho Chehab Committed by Jonathan Corbet

xz.txt: standardize document format

Each text file under Documentation follows a different
format. Some doesn't even have titles!

Change its representation to follow the adopted standard,
using ReST markups for it to be parseable by Sphinx:
- Use marks for titles;
- Adjust indentation.
Signed-off-by: default avatarMauro Carvalho Chehab <mchehab@s-opensource.com>
Signed-off-by: default avatarJonathan Corbet <corbet@lwn.net>
parent 05a07392
============================
XZ data compression in Linux XZ data compression in Linux
============================ ============================
Introduction Introduction
============
XZ is a general purpose data compression format with high compression XZ is a general purpose data compression format with high compression
ratio and relatively fast decompression. The primary compression ratio and relatively fast decompression. The primary compression
algorithm (filter) is LZMA2. Additional filters can be used to improve algorithm (filter) is LZMA2. Additional filters can be used to improve
compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters
improve compression ratio of executable data. improve compression ratio of executable data.
The XZ decompressor in Linux is called XZ Embedded. It supports The XZ decompressor in Linux is called XZ Embedded. It supports
the LZMA2 filter and optionally also BCJ filters. CRC32 is supported the LZMA2 filter and optionally also BCJ filters. CRC32 is supported
for integrity checking. The home page of XZ Embedded is at for integrity checking. The home page of XZ Embedded is at
<http://tukaani.org/xz/embedded.html>, where you can find the <http://tukaani.org/xz/embedded.html>, where you can find the
latest version and also information about using the code outside latest version and also information about using the code outside
the Linux kernel. the Linux kernel.
For userspace, XZ Utils provide a zlib-like compression library For userspace, XZ Utils provide a zlib-like compression library
and a gzip-like command line tool. XZ Utils can be downloaded from and a gzip-like command line tool. XZ Utils can be downloaded from
<http://tukaani.org/xz/>. <http://tukaani.org/xz/>.
XZ related components in the kernel XZ related components in the kernel
===================================
The xz_dec module provides XZ decompressor with single-call (buffer
to buffer) and multi-call (stateful) APIs. The usage of the xz_dec The xz_dec module provides XZ decompressor with single-call (buffer
module is documented in include/linux/xz.h. to buffer) and multi-call (stateful) APIs. The usage of the xz_dec
module is documented in include/linux/xz.h.
The xz_dec_test module is for testing xz_dec. xz_dec_test is not
useful unless you are hacking the XZ decompressor. xz_dec_test The xz_dec_test module is for testing xz_dec. xz_dec_test is not
allocates a char device major dynamically to which one can write useful unless you are hacking the XZ decompressor. xz_dec_test
.xz files from userspace. The decompressed output is thrown away. allocates a char device major dynamically to which one can write
Keep an eye on dmesg to see diagnostics printed by xz_dec_test. .xz files from userspace. The decompressed output is thrown away.
See the xz_dec_test source code for the details. Keep an eye on dmesg to see diagnostics printed by xz_dec_test.
See the xz_dec_test source code for the details.
For decompressing the kernel image, initramfs, and initrd, there
is a wrapper function in lib/decompress_unxz.c. Its API is the For decompressing the kernel image, initramfs, and initrd, there
same as in other decompress_*.c files, which is defined in is a wrapper function in lib/decompress_unxz.c. Its API is the
include/linux/decompress/generic.h. same as in other decompress_*.c files, which is defined in
include/linux/decompress/generic.h.
scripts/xz_wrap.sh is a wrapper for the xz command line tool found
from XZ Utils. The wrapper sets compression options to values suitable scripts/xz_wrap.sh is a wrapper for the xz command line tool found
for compressing the kernel image. from XZ Utils. The wrapper sets compression options to values suitable
for compressing the kernel image.
For kernel makefiles, two commands are provided for use with
$(call if_needed). The kernel image should be compressed with For kernel makefiles, two commands are provided for use with
$(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2 $(call if_needed). The kernel image should be compressed with
dictionary. It will also append a four-byte trailer containing the $(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2
uncompressed size of the file, which is needed by the boot code. dictionary. It will also append a four-byte trailer containing the
Other things should be compressed with $(call if_needed,xzmisc) uncompressed size of the file, which is needed by the boot code.
which will use no BCJ filter and 1 MiB LZMA2 dictionary. Other things should be compressed with $(call if_needed,xzmisc)
which will use no BCJ filter and 1 MiB LZMA2 dictionary.
Notes on compression options Notes on compression options
============================
Since the XZ Embedded supports only streams with no integrity check or Since the XZ Embedded supports only streams with no integrity check or
CRC32, make sure that you don't use some other integrity check type CRC32, make sure that you don't use some other integrity check type
when encoding files that are supposed to be decoded by the kernel. With when encoding files that are supposed to be decoded by the kernel. With
liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32 liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32
when encoding. With the xz command line tool, use --check=none or when encoding. With the xz command line tool, use --check=none or
--check=crc32. --check=crc32.
Using CRC32 is strongly recommended unless there is some other layer Using CRC32 is strongly recommended unless there is some other layer
which will verify the integrity of the uncompressed data anyway. which will verify the integrity of the uncompressed data anyway.
Double checking the integrity would probably be waste of CPU cycles. Double checking the integrity would probably be waste of CPU cycles.
Note that the headers will always have a CRC32 which will be validated Note that the headers will always have a CRC32 which will be validated
by the decoder; you can only change the integrity check type (or by the decoder; you can only change the integrity check type (or
disable it) for the actual uncompressed data. disable it) for the actual uncompressed data.
In userspace, LZMA2 is typically used with dictionary sizes of several In userspace, LZMA2 is typically used with dictionary sizes of several
megabytes. The decoder needs to have the dictionary in RAM, thus big megabytes. The decoder needs to have the dictionary in RAM, thus big
dictionaries cannot be used for files that are intended to be decoded dictionaries cannot be used for files that are intended to be decoded
by the kernel. 1 MiB is probably the maximum reasonable dictionary by the kernel. 1 MiB is probably the maximum reasonable dictionary
size for in-kernel use (maybe more is OK for initramfs). The presets size for in-kernel use (maybe more is OK for initramfs). The presets
in XZ Utils may not be optimal when creating files for the kernel, in XZ Utils may not be optimal when creating files for the kernel,
so don't hesitate to use custom settings. Example: so don't hesitate to use custom settings. Example::
xz --check=crc32 --lzma2=dict=512KiB inputfile xz --check=crc32 --lzma2=dict=512KiB inputfile
An exception to above dictionary size limitation is when the decoder An exception to above dictionary size limitation is when the decoder
is used in single-call mode. Decompressing the kernel itself is an is used in single-call mode. Decompressing the kernel itself is an
example of this situation. In single-call mode, the memory usage example of this situation. In single-call mode, the memory usage
doesn't depend on the dictionary size, and it is perfectly fine to doesn't depend on the dictionary size, and it is perfectly fine to
use a big dictionary: for maximum compression, the dictionary should use a big dictionary: for maximum compression, the dictionary should
be at least as big as the uncompressed data itself. be at least as big as the uncompressed data itself.
Future plans Future plans
============
Creating a limited XZ encoder may be considered if people think it is Creating a limited XZ encoder may be considered if people think it is
useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at
the fastest settings, so it isn't clear if LZMA2 encoder is wanted the fastest settings, so it isn't clear if LZMA2 encoder is wanted
into the kernel. into the kernel.
Support for limited random-access reading is planned for the Support for limited random-access reading is planned for the
decompression code. I don't know if it could have any use in the decompression code. I don't know if it could have any use in the
kernel, but I know that it would be useful in some embedded projects kernel, but I know that it would be useful in some embedded projects
outside the Linux kernel. outside the Linux kernel.
Conformance to the .xz file format specification Conformance to the .xz file format specification
================================================
There are a couple of corner cases where things have been simplified There are a couple of corner cases where things have been simplified
at expense of detecting errors as early as possible. These should not at expense of detecting errors as early as possible. These should not
matter in practice all, since they don't cause security issues. But matter in practice all, since they don't cause security issues. But
it is good to know this if testing the code e.g. with the test files it is good to know this if testing the code e.g. with the test files
from XZ Utils. from XZ Utils.
Reporting bugs Reporting bugs
==============
Before reporting a bug, please check that it's not fixed already Before reporting a bug, please check that it's not fixed already
at upstream. See <http://tukaani.org/xz/embedded.html> to get the at upstream. See <http://tukaani.org/xz/embedded.html> to get the
latest code. latest code.
Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on Report bugs to <lasse.collin@tukaani.org> or visit #tukaani on
Freenode and talk to Larhzu. I don't actively read LKML or other Freenode and talk to Larhzu. I don't actively read LKML or other
kernel-related mailing lists, so if there's something I should know, kernel-related mailing lists, so if there's something I should know,
you should email to me personally or use IRC. you should email to me personally or use IRC.
Don't bother Igor Pavlov with questions about the XZ implementation Don't bother Igor Pavlov with questions about the XZ implementation
in the kernel or about XZ Utils. While these two implementations in the kernel or about XZ Utils. While these two implementations
include essential code that is directly based on Igor Pavlov's code, include essential code that is directly based on Igor Pavlov's code,
these implementations aren't maintained nor supported by him. these implementations aren't maintained nor supported by him.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment