Al Viro pointed out that I broke some acl functionality...

 * ACLs could not be fully removed
 * posix_acl_chmod would be called while the old ACL was still cached
 * new mode propagated to orangefs server before ACL.

... when I tried to make sure that modes that got changed as a
result of ACL-sets would be sent back to the orangefs server.

Not wanting to try and change the code without having some cases to
test it with, I began to hunt for setfacl examples that were expressible
in pure mode. Along the way I found examples like the following
which confused me:

  user A had a file (/home/A/asdf) with mode 740
  user B was in user A's group
  user C was not in user A's group

  setfacl -m u:C:rwx /home/A/asdf

  The above setfacl caused ls -l /home/A/asdf to show a mode of 770,
  making it appear that all users in user A's group now had full access
  to /home/A/asdf, however, user B still only had read acces. Madness.

Anywho, I finally found that the above (whacky as it is) appears to
be "posixly on purpose" and explained in acl(5):

  If the ACL has an ACL_MASK entry, the group permissions correspond
  to the permissions of the ACL_MASK entry.
Signed-off-by: Mike Marshall <hubcap@omnibond.com>