Split the 'flush_list', which contained a mix of both 'mark' and 'clear'
requests, into two distinct lists ('mark_list' and 'clear_list').

The device mapper log implementations (used by various DM targets) are
allowed to cache 'mark' and 'clear' requests until a 'flush' is
received.  Until now, these cached requests were kept in the same list.
They will now be put into distinct lists to facilitate group processing
of these requests (in the next patch).
Signed-off-by: Jonathan Brassow <jbrassow@redhat.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: Alasdair G Kergon <agk@redhat.com>

Make kcopyd merge more I/O requests by using device unplugging.

Without this patch, each I/O request is dispatched separately to the device.
If the device supports tagged queuing, there are many small requests sent
to the device. To improve performance, this patch will batch as many requests
as possible, allowing the queue to merge consecutive requests, and send them
to the device at once.

In my tests (15k SCSI disk), this patch improves sequential write throughput:

  Sequential write throughput (chunksize of 4k, 32k, 512k)
  unpatched: 15.2, 18.5, 17.5 MB/s
  patched:   14.4, 22.6, 23.0 MB/s

In most common uses (snapshot or two-way mirror), kcopyd is only used for
two devices, one for reading and the other for writing, thus this optimization
is implemented only for two devices. The optimization may be extended to n-way
mirrors with some code complexity increase.

We keep track of two block devices to unplug (one for read and the
other for write) and unplug them when exiting "do_work" thread.  If
there are more devices used (in theory it could happen, in practice it
is rare), we unplug immediately.
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: Alasdair G Kergon <agk@redhat.com>

When constructing a mirror log, it is possible for the initial request
to fail for other reasons besides -ESRCH.  These must be handled too.
Signed-off-by: Jonathan Brassow <jbrassow@redhat.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: Alasdair G Kergon <agk@redhat.com>

Simplify key size verification (hexadecimal string) and
set key size early in constructor.

(Patch required by later changes.)
Signed-off-by: Milan Broz <mbroz@redhat.com>
Signed-off-by: Alasdair G Kergon <agk@redhat.com>

This patch replaces dm_mutex with _minor_lock in dm_blk_close()
and then removes it.

During the BKL conversion, commit 6e9624b8
(block: push down BKL into .open and .release) pushed lock_kernel()
down into dm_blk_open/close calls.
Commit 2a48fc0a
(block: autoconvert trivial BKL users to private mutex) converted it to a
local mutex, but _minor_lock is sufficient.
Signed-off-by: Milan Broz <mbroz@redhat.com>
Signed-off-by: Alasdair G Kergon <agk@redhat.com>

Enable discard support in the DM mirror target.
Also change an existing use of 'bvec' to 'addr' in the union.
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: Alasdair G Kergon <agk@redhat.com>

Allow the uuid of a mapped device to be set after device creation.
Previously the uuid (which is optional) could only be set by
DM_DEV_CREATE.  If no uuid was supplied it could not be set later.

Sometimes it's necessary to create the device before the uuid is known,
and in such cases the uuid must be filled in after the creation.

This patch extends DM_DEV_RENAME to accept a uuid accompanied by
a new flag DM_UUID_FLAG.  This can only be done once and if no
uuid was previously supplied.  It cannot be used to change an
existing uuid.

DM_VERSION_MINOR is also bumped to 19 to indicate this interface
extension is available.
Signed-off-by: Peter Jones <pjones@redhat.com>
Signed-off-by: Jonathan Brassow <jbrassow@redhat.com>
Signed-off-by: Alasdair G Kergon <agk@redhat.com>

Remove the REQ_SYNC flag to improve write throughput when writing
to the origin with a snapshot on the same device (using the CFQ I/O
scheduler).

Sequential write throughput (chunksize of 4k, 32k, 512k)
  unpatched:  8.5,  8.6,  9.3 MB/s
  patched:   15.2, 18.5, 17.5 MB/s

Snapshot exception reallocations are triggered by writes that are
usually async, so mark the associated dm_io_request as async as well.
This helps when using the CFQ I/O scheduler because it has separate
queues for sync and async I/O.  Async is optimized for throughput; sync
for latency.  With this change we're consciously favoring throughput over
latency.
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Signed-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: Alasdair G Kergon <agk@redhat.com>

Revert commit 224cb3e9
  dm: Call blk_abort_queue on failed paths

Multipath began to use blk_abort_queue() to allow for
lower latency path deactivation.  This was found to
cause list corruption:

   the cmd gets blk_abort_queued/timedout run on it and the scsi eh
   somehow is able to complete and run scsi_queue_insert while
   scsi_request_fn is still trying to process the request.

   https://www.redhat.com/archives/dm-devel/2010-November/msg00085.htmlSigned-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: Alasdair G Kergon <agk@redhat.com>
Cc: Mike Anderson <andmike@linux.vnet.ibm.com>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: stable@kernel.org

No longer needlessly hold md->bdev->bd_inode->i_mutex when changing the
size of a DM device.  This additional locking is unnecessary because
i_size_write() is already protected by the existing critical section in
dm_swap_table().  DM already has a reference on md->bdev so the
associated bd_inode may be changed without lifetime concerns.

A negative side-effect of having held md->bdev->bd_inode->i_mutex was
that a concurrent DM device resize and flush (via fsync) would deadlock.
Dropping md->bdev->bd_inode->i_mutex eliminates this potential for
deadlock.  The following reproducer no longer deadlocks:
  https://www.redhat.com/archives/dm-devel/2009-July/msg00284.htmlSigned-off-by: Mike Snitzer <snitzer@redhat.com>
Signed-off-by: Mikulas Patocka <mpatocka@redhat.com>
Signed-off-by: Alasdair G Kergon <agk@redhat.com>
Cc: stable@kernel.org

* 'release' of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux-2.6:
  [IA64] Fix format warning in arch/ia64/kernel/acpi.c

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/ieee1394/linux1394-2.6:
  firewire: ohci: fix compilation on arches without PAGE_KERNEL_RO

* 'for-2.6.38/drivers' of git://git.kernel.dk/linux-2.6-block:
  cciss: reinstate proper FIFO order of command queue list
  floppy: replace NO_GEOM macro with a function

* 'for-2.6.38/core' of git://git.kernel.dk/linux-2.6-block: (43 commits)
  block: ensure that completion error gets properly traced
  blktrace: add missing probe argument to block_bio_complete
  block cfq: don't use atomic_t for cfq_group
  block cfq: don't use atomic_t for cfq_queue
  block: trace event block fix unassigned field
  block: add internal hd part table references
  block: fix accounting bug on cross partition merges
  kref: add kref_test_and_get
  bio-integrity: mark kintegrityd_wq highpri and CPU intensive
  block: make kblockd_workqueue smarter
  Revert "sd: implement sd_check_events()"
  block: Clean up exit_io_context() source code.
  Fix compile warnings due to missing removal of a 'ret' variable
  fs/block: type signature of major_to_index(int) to major_to_index(unsigned)
  block: convert !IS_ERR(p) && p to !IS_ERR_NOR_NULL(p)
  cfq-iosched: don't check cfqg in choose_service_tree()
  fs/splice: Pull buf->ops->confirm() from splice_from_pipe actors
  cdrom: export cdrom_check_events()
  sd: implement sd_check_events()
  sr: implement sr_check_events()
  ...

* 'for-linus/i2c-2638' of git://git.fluff.org/bjdooks/linux:
  i2c-bfin-twi: move setup to the earlier subsys initcall
  i2c-bfin-twi: handle faulty slave devices better
  i2c-mv64xxx: send repeated START between messages in xfer
  i2c-nomadik: fix regression on adapter name
  i2c-omap: Set latency requirements only once for several messages
  i2c-eg20t: add driver for Intel EG20T
  i2c-ocores: add some device tree documentation
  i2c-ocores: Use devres for resource allocation
  i2c-ocores: Adapt for device tree
  i2c-iop3xx: add iomem annotation

* 'rmobile-latest' of git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6:
  ARM: mach-shmobile: Kill off unused !gpio_is_valid() case
  ARM: mach-shmobile: sh7372 Enable SDIO IRQs for Mackerel
  ARM: mach-shmobile: sh7377 Enable SDIO IRQs
  ARM: mach-shmobile: sh7367 Enable SDIO IRQs
  ARM: mach-shmobile: sh7372 Enable SDIO IRQs
  ARM: mach-shmobile: mackerel: Add touchscreen ST1232 support
  ARM: mach-shmobile: ap4eb: SCIF port for earlyprintk when using zboot
  ARM: mach-shmobile: mackerel: SCIF port for earlyprintk when using zboot
  ARM: mach-shmobile: mackerel: Add support get_cd in CN23

* 'sh-latest' of git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6: (31 commits)
  sh: Add support for AP-SH4AD-0A board.
  sh: Add support for AP-SH4A-3A board.
  sh: Add a new mach type for alpha project boards.
  serial: sh-sci: build fixes.
  sh: sh7372 SH4AL-DSP probe support
  sh: sh7366 Enable SDIO IRQs
  sh: sh7343 Enable SDIO IRQs
  sh: mach-ecovec24: enable runtime PM for SDHI
  sh: sh7723 / ap325rxa enable SDIO IRQs
  sh: sh7722 Enable SDIO IRQs
  sh: sh7724 Enable SDIO IRQs
  sh: Fix up legacy PTEA space attribute mapping.
  sh: Stub out legacy PCC pgprot encoding for X2 TLBs.
  sh: constify prefetch pointers.
  sh: Add a machvec callback for early memblock reservations.
  sh: update sh7757lcr_defconfig
  sh: add PVR probing for SH7757 3rd cut
  sh: Use device_initcall() instead of __initcall()
  sh: intc - convert board specific landisk code
  sh: Move init_landisk_IRQ to header file
  ...

* git://git.kernel.org/pub/scm/linux/kernel/git/lethal/fbdev-2.6: (29 commits)
  video: move SH_MIPI_DSI/SH_LCD_MIPI_DSI to the top of menu
  fbdev: Implement simple blanking in pseudocolor modes for vt8500lcdfb
  video: imx: Update the manufacturer's name
  nuc900fb: don't treat NULL clk as an error
  s3c2410fb: don't treat NULL clk as an error
  video: tidy up modedb formatting.
  video: matroxfb: Correct video option in comments and kernel config help.
  fbdev: sh_mobile_hdmi: simplify pointer handling
  fbdev: sh_mobile_hdmi: framebuffer notifiers have to be registered
  fbdev: sh_mobile_hdmi: add command line option to use the preferred EDID mode
  OMAP: DSS2: Introduce omap_channel as an omap_dss_device parameter, add new overlay manager.
  OMAP: DSS2: Use dss_features to handle DISPC bits removed on OMAP4
  OMAP: DSS2: LCD2 Channel Changes for DISPC
  OMAP: DSS2: Change remaining DISPC functions for new omap_channel argument
  OMAP: DSS2: Introduce omap_channel argument to DISPC functions used by interface drivers
  OMAP: DSS2: Represent DISPC register defines with channel as parameter
  OMAP: DSS2: Add dss_features for omap4 and overlay manager related features
  OMAP: DSS2: Clean up DISPC color mode validation checks
  OMAP: DSS2: Add back authors of panel-generic.c based drivers
  OMAP: DSS2: remove generic DPI panel driver duplicated panel drivers
  ...

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound-2.6: (348 commits)
  ALSA: hda - Fix NULL-derefence with a single mic in STAC auto-mic detection
  ALSA: hda - Add missing NID 0x19 fixup for Sony VAIO
  ALSA: hda - Fix ALC275 enable hardware EQ for SONY VAIO
  ALSA: oxygen: fix Xonar DG input
  ALSA: hda - Fix EAPD on Lenovo NB ALC269 to low
  ALSA: hda - Fix missing EAPD for Acer 4930G
  ALSA: hda: Disable 4/6 channels on some NVIDIA GPUs.
  ALSA: hda - Add static_hdmi_pcm option to HDMI codec parser
  ALSA: hda - Don't refer ELD when unplugged
  ASoC: tpa6130a2: Fix compiler warning
  ASoC: tlv320dac33: Add DAPM selection for LOM invert
  ASoC: DMIC codec: Adding a generic DMIC codec
  ALSA: snd-usb-us122l: Fix missing NULL checks
  ALSA: snd-usb-us122l: Fix MIDI output
  ASoC: soc-cache: Fix invalid memory access during snd_soc_lzo_cache_sync()
  ASoC: Fix section mismatch in wm8995.c
  ALSA: oxygen: add S/PDIF source selection for Claro cards
  ALSA: oxygen: fix CD/MIDI for X-Meridian (2G)
  ASoC: fix migor audio build
  ALSA: include delay.h for msleep in Xonar DG support
  ...

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs-2.6: (41 commits)
  fs: add documentation on fallocate hole punching
  Gfs2: fail if we try to use hole punch
  Btrfs: fail if we try to use hole punch
  Ext4: fail if we try to use hole punch
  Ocfs2: handle hole punching via fallocate properly
  XFS: handle hole punching via fallocate properly
  fs: add hole punching to fallocate
  vfs: pass struct file to do_truncate on O_TRUNC opens (try #2)
  fix signedness mess in rw_verify_area() on 64bit architectures
  fs: fix kernel-doc for dcache::prepend_path
  fs: fix kernel-doc for dcache::d_validate
  sanitize ecryptfs ->mount()
  switch afs
  move internal-only parts of ncpfs headers to fs/ncpfs
  switch ncpfs
  switch 9p
  pass default dentry_operations to mount_pseudo()
  switch hostfs
  switch affs
  switch configfs
  ...

* git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (46 commits)
  hwrng: via_rng - Fix memory scribbling on some CPUs
  crypto: padlock - Move padlock.h into include/crypto
  hwrng: via_rng - Fix asm constraints
  crypto: n2 - use __devexit not __exit in n2_unregister_algs
  crypto: mark crypto workqueues CPU_INTENSIVE
  crypto: mv_cesa - dont return PTR_ERR() of wrong pointer
  crypto: ripemd - Set module author and update email address
  crypto: omap-sham - backlog handling fix
  crypto: gf128mul - Remove experimental tag
  crypto: af_alg - fix af_alg memory_allocated data type
  crypto: aesni-intel - Fixed build with binutils 2.16
  crypto: af_alg - Make sure sk_security is initialized on accept()ed sockets
  net: Add missing lockdep class names for af_alg
  include: Install linux/if_alg.h for user-space crypto API
  crypto: omap-aes - checkpatch --file warning fixes
  crypto: omap-aes - initialize aes module once per request
  crypto: omap-aes - unnecessary code removed
  crypto: omap-aes - error handling implementation improved
  crypto: omap-aes - redundant locking is removed
  crypto: omap-aes - DMA initialization fixes for OMAP off mode
  ...

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/sage/ceph-client:
  rbd: fix cleanup when trying to mount inexistent image
  net/ceph: make ceph_msgr_wq non-reentrant
  ceph: fsc->*_wq's aren't used in memory reclaim path
  ceph: Always free allocated memory in osdmap_decode()
  ceph: Makefile: Remove unnessary code
  ceph: associate requests with opening sessions
  ceph: drop redundant r_mds field
  ceph: implement DIRLAYOUTHASH feature to get dir layout from MDS
  ceph: add dir_layout to inode

* git://git.kernel.org/pub/scm/linux/kernel/git/wim/linux-2.6-watchdog:
  watchdog: Add MCF548x watchdog driver.
  watchdog: add driver for the Atheros AR71XX/AR724X/AR913X SoCs
  watchdog: Add TCO support for nVidia chipsets
  watchdog: Add support for sp5100 chipset TCO
  watchdog: f71808e_wdt: add F71862FG, F71869 to Kconfig
  watchdog: iTCO_wdt: TCO Watchdog patch for Intel DH89xxCC PCH
  watchdog: iTCO_wdt: TCO Watchdog patch for Intel NM10 DeviceIDs
  watchdog: ks8695_wdt: include mach/hardware.h instead of mach/timex.h.
  watchdog: Propagate Book E WDT period changes to all cores
  watchdog: add CONFIG_WATCHDOG_NOWAYOUT support to PowerPC Book-E watchdog driver
  watchdog: alim7101_wdt: fix compiler warning on alim7101_pci_tbl
  watchdog: alim1535_wdt: fix compiler warning on ali_pci_tbl
  watchdog: Fix reboot on W83627ehf chipset.
  watchdog: Add watchdog support for W83627DHG chip
  watchdog: f71808e_wdt: Add Fintek F71869 watchdog
  watchdog: add f71862fg support
  watchdog: clean-up f71808e_wdt.c

* 'hwmon-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jdelvare/staging:
  hwmon: (dme1737) Minor cleanups
  hwmon: (dme1737) Add support for in7 for SCH5127
  hwmon: (emc1403) Add EMC1423 support
  hwmon: (w83627hf) Document W83627THF voltage pin mapping
  hwmon: (w83793) Drop useless mutex
  hwmon: (fschmd) Drop useless mutex
  hwmon: (w83781d) Use pr_fmt and pr_<level>
  hwmon: (pc87427) Use pr_fmt and pr_<level>
  hwmon: (pc87360) Use pr_fmt and pr_<level>
  hwmon: (lm78) Use pr_fmt and pr_<level>
  hwmon: (it87) Use pr_fmt and pr_<level>
  hwmon: Schedule the removal of the old intrusion detection interfaces
  hwmon: (w83793) Implement the standard intrusion detection interface
  hwmon: (w83792d) Implement the standard intrusion detection interface
  hwmon: (adm9240) Implement the standard intrusion detection interface
  hwmon: (via686a) Initialize fan_div values
  hwmon: (w83795) Silent false warning from gcc
  hwmon: (ads7828) Update email contact details

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/lrg/voltage-2.6: (45 commits)
  regulator: missing index in PTR_ERR() in isl6271a_probe()
  regulator: Assign return value of mc13xxx_reg_rmw to ret
  regulator: Add initial per-regulator debugfs support
  regulator: Make regulator_has_full_constraints a bool
  regulator: Clean up logging a bit
  regulator: Optimise out noop voltage changes
  regulator: Add API to re-apply voltage to hardware
  regulator: Staticise non-exported functions in mc13892
  regulator: Only notify voltage changes when they succeed
  regulator: Provide a selector based set_voltage_sel() operation
  regulator: Factor out voltage set operation into a separate function
  regulator: Convert WM8994 to use get_voltage_sel()
  regulator: Convert WM835x to use get_voltage_sel()
  regulator: Allow modular build of mc13xxx-core
  regulator: support PMIC mc13892
  make mc13783 regulator code generic
  Change the register name definitions for mc13783
  mach-ux500: Updated and connected ab8500 regulator board configuration
  regulators: Removed macros for initialization of ab8500 regulators
  regulators: Added verbose debug messages to ab8500 regulators
  ...

Merge branch 'x86-olpc-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip

* 'x86-olpc-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip:
  x86, olpc: Speed up device tree creation during boot
  x86, olpc: Add OLPC device-tree support
  x86, of: Define irq functions to allow drivers/of/* to build on x86

* 'kvm-updates/2.6.38' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (142 commits)
  KVM: Initialize fpu state in preemptible context
  KVM: VMX: when entering real mode align segment base to 16 bytes
  KVM: MMU: handle 'map_writable' in set_spte() function
  KVM: MMU: audit: allow audit more guests at the same time
  KVM: Fetch guest cr3 from hardware on demand
  KVM: Replace reads of vcpu->arch.cr3 by an accessor
  KVM: MMU: only write protect mappings at pagetable level
  KVM: VMX: Correct asm constraint in vmcs_load()/vmcs_clear()
  KVM: MMU: Initialize base_role for tdp mmus
  KVM: VMX: Optimize atomic EFER load
  KVM: VMX: Add definitions for more vm entry/exit control bits
  KVM: SVM: copy instruction bytes from VMCB
  KVM: SVM: implement enhanced INVLPG intercept
  KVM: SVM: enhance mov DR intercept handler
  KVM: SVM: enhance MOV CR intercept handler
  KVM: SVM: add new SVM feature bit names
  KVM: cleanup emulate_instruction
  KVM: move complete_insn_gp() into x86.c
  KVM: x86: fix CR8 handling
  KVM guest: Fix kvm clock initialization when it's configured out
  ...

* 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/trivial: (43 commits)
  Documentation/trace/events.txt: Remove obsolete sched_signal_send.
  writeback: fix global_dirty_limits comment runtime -> real-time
  ppc: fix comment typo singal -> signal
  drivers: fix comment typo diable -> disable.
  m68k: fix comment typo diable -> disable.
  wireless: comment typo fix diable -> disable.
  media: comment typo fix diable -> disable.
  remove doc for obsolete dynamic-printk kernel-parameter
  remove extraneous 'is' from Documentation/iostats.txt
  Fix spelling milisec -> ms in snd_ps3 module parameter description
  Fix spelling mistakes in comments
  Revert conflicting V4L changes
  i7core_edac: fix typos in comments
  mm/rmap.c: fix comment
  sound, ca0106: Fix assignment to 'channel'.
  hrtimer: fix a typo in comment
  init/Kconfig: fix typo
  anon_inodes: fix wrong function name in comment
  fix comment typos concerning "consistent"
  poll: fix a typo in comment
  ...

Fix up trivial conflicts in:
 - drivers/net/wireless/iwlwifi/iwl-core.c (moved to iwl-legacy.c)
 - fs/ext4/ext4.h

Also fix missed 'diabled' typo in drivers/net/bnx2x/bnx2x.h while at it.

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/hid:
  HID: hid-multitouch: minor fixes based on additional review
  HID: Switch turbox/mosart touchscreen to hid-mosart
  HID: add Add Cando touch screen 10.1-inch product id
  HID: hid-mulitouch: add support for the 'Sensing Win7-TwoFinger'
  HID: hid-multitouch: add support for Cypress TrueTouch panels
  HID: hid-multitouch: support for PixCir-based panels
  HID: set HID_MAX_FIELD at 128
  HID: add feature_mapping callback

* 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input:
  Input: add SW_ROTATE_LOCK switch type
  Input: fix force feedback capability query example
  Input: wacom_w8001 - add single-touch support
  Input: add Austria Microsystem AS5011 joystick driver
  Input: remove aaed2000 keyboard driver
  Input: i8042 - introduce 'notimeout' blacklist for Dell Vostro V13
  Input: cy8ctmg110_ts - Convert to dev_pm_ops
  Input: migor_ts - convert to dev_pm_ops
  Input: mcs5000_ts - convert to dev_pm_ops
  Input: eeti_ts - convert to dev_pm_ops
  Input: ad7879 - convert I2C to dev_pm_ops

Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Check for end of the input buffer when skipping over the filename field in
the .gz file header.
Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Alain Knaff <alain@knaff.lu>
Cc: Albin Tonnerre <albin.tonnerre@free-electrons.com>
Cc: Phillip Lougher <phillip@lougher.demon.co.uk>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

This integrates the XZ decompression code to the x86 pre-boot code.

mkpiggy.c is updated to reserve about 32 KiB more buffer safety margin for
kernel decompression.  It is done unconditionally for all decompressors to
keep the code simpler.

The XZ decompressor needs around 30 KiB of heap, so the heap size is
increased to 32 KiB on both x86-32 and x86-64.

Documentation/x86/boot.txt is updated to list the XZ magic number.

With the x86 BCJ filter in XZ, XZ-compressed x86 kernel tends to be a few
percent smaller than the equivalent LZMA-compressed kernel.
Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Alain Knaff <alain@knaff.lu>
Cc: Albin Tonnerre <albin.tonnerre@free-electrons.com>
Cc: Phillip Lougher <phillip@lougher.demon.co.uk>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

This implements the API defined in <linux/decompress/generic.h> which is
used for kernel, initramfs, and initrd decompression.  This patch together
with the first patch is enough for XZ-compressed initramfs and initrd;
XZ-compressed kernel will need arch-specific changes.

The buffering requirements described in decompress_unxz.c are stricter
than with gzip, so the relevant changes should be done to the
arch-specific code when adding support for XZ-compressed kernel.
Similarly, the heap size in arch-specific pre-boot code may need to be
increased (30 KiB is enough).

The XZ decompressor needs memmove(), memeq() (memcmp() == 0), and
memzero() (memset(ptr, 0, size)), which aren't available in all
arch-specific pre-boot environments.  I'm including simple versions in
decompress_unxz.c, but a cleaner solution would naturally be nicer.
Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Alain Knaff <alain@knaff.lu>
Cc: Albin Tonnerre <albin.tonnerre@free-electrons.com>
Cc: Phillip Lougher <phillip@lougher.demon.co.uk>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

In userspace, the .lzma format has become mostly a legacy file format that
got superseded by the .xz format.  Similarly, LZMA Utils was superseded by
XZ Utils.

These patches add support for XZ decompression into the kernel.  Most of
the code is as is from XZ Embedded <http://tukaani.org/xz/embedded.html>.
It was written for the Linux kernel but is usable in other projects too.

Advantages of XZ over the current LZMA code in the kernel:
  - Nice API that can be used by other kernel modules; it's
    not limited to kernel, initramfs, and initrd decompression.
  - Integrity check support (CRC32)
  - BCJ filters improve compression of executable code on
    certain architectures. These together with LZMA2 can
    produce a few percent smaller kernel or Squashfs images
    than plain LZMA without making the decompression slower.

This patch: Add the main decompression code (xz_dec), testing module
(xz_dec_test), wrapper script (xz_wrap.sh) for the xz command line tool,
and documentation.  The xz_dec module is enough to have a usable XZ
decompressor e.g.  for Squashfs.
Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Alain Knaff <alain@knaff.lu>
Cc: Albin Tonnerre <albin.tonnerre@free-electrons.com>
Cc: Phillip Lougher <phillip@lougher.demon.co.uk>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Callback-to-callback decompression mode is used for initrd (not
initramfs).  The LZO wrapper is broken for this use case for two reasons:

  - The argument validation is needlessly too strict by
    requiring that "posp" is non-NULL when "fill" is non-NULL.

  - The buffer handling code didn't work at all for this
    use case.

I tested with LZO-compressed kernel, initramfs, initrd, and corrupt
(truncated) initramfs and initrd images.
Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Alain Knaff <alain@knaff.lu>
Cc: Albin Tonnerre <albin.tonnerre@free-electrons.com>
Cc: Phillip Lougher <phillip@lougher.demon.co.uk>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

The code assumes that the input is valid and not truncated.  Add checks to
avoid reading past the end of the input buffer.  Change the type of "skip"
from u8 to int to fix a possible integer overflow.
Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Alain Knaff <alain@knaff.lu>
Cc: Albin Tonnerre <albin.tonnerre@free-electrons.com>
Cc: Phillip Lougher <phillip@lougher.demon.co.uk>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

The return value of flush() is not checked in unlzo().  This means that
the decompressor won't stop even if the caller doesn't want more data.
This can happen e.g.  with a corrupt LZO-compressed initramfs image.
Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Alain Knaff <alain@knaff.lu>
Cc: Albin Tonnerre <albin.tonnerre@free-electrons.com>
Cc: Phillip Lougher <phillip@lougher.demon.co.uk>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

Validate the newly decoded distance (rep0) in process_bit1().  This is to
detect corrupt LZMA data quickly.  The old code can run for long time
producing garbage until it hits the end of the input.
Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Alain Knaff <alain@knaff.lu>
Cc: Albin Tonnerre <albin.tonnerre@free-electrons.com>
Cc: Phillip Lougher <phillip@lougher.demon.co.uk>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

The return value of wr->flush() is not checked in write_byte().  This
means that the decompressor won't stop even if the caller doesn't want
more data.  This can happen e.g.  with corrupt LZMA-compressed initramfs.
Returning the error quickly allows the user to see the error message
quicker.

There is a similar missing check for wr.flush() near the end of unlzma().
Signed-off-by: Lasse Collin <lasse.collin@tukaani.org>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Alain Knaff <alain@knaff.lu>
Cc: Albin Tonnerre <albin.tonnerre@free-electrons.com>
Cc: Phillip Lougher <phillip@lougher.demon.co.uk>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>