- Loosen up test as Fedora has implicit xattrs
- Actually test the destination for xattrs

Should make Fedora buildbot green again.