Thanks Matt and Jens for the debug help on the multipath problem. 

This solves the problem and makes multipath work properly.

There are two types of "flags" that are used in a block io request,
bi_flags, and bi_rw.  bi_flags is used for flags to the block level
code, and bi_rw is used for flags to the low level device drivers.

The code in the multipath driver used the wrong flag in the wrong field.
In this case, the flag FASTFAIL (value 3) was being set to the bi_flags
field.  FASTFAIL is a hint to the low level driver that it should try to
fail out quickly.  Unfortunately, the value 3 is also BIO_SEG_VALID,
which is a flag to the block subsystem that the segments shouldn't be
recalculated.  The result was that the wrong field was set, telling the
block layer not to recalculate the segments resulting in phys and hw
segments of 0.  Not good.