we're in the pre-2.2.0 series now, I'm all synched up with Alan, and I
don't have anything pending any more. Over the internet nobody can hear
you all scream in pain over all your favourite features that didn't make
it.

	Linus "another year older and wise as hell by now" Torvalds

There's a new pre-patch out there. I'm back from Finland, and have caught
up with just about half the email that I got during the stay. However,
even the part I caught up with I may have partly missed something in,
because (for obvious reasons) I didn't read them as carefully (*) as I
usually do.

This should fix at least part of the NFS problems people have reported:
there was code to completely incorrectly invalidate quite valid write
requests under some circumstances. The pre-patch also contains the first
batch of patches merged in from Alan, and the "rmdir" problems should be
fixed (mostly thanks to Al Viro).

This pre-patch also gets rid of some imho completely unnecessary
complexity in some of the VM memory freeing routines. There have been
patches floating around that added more heuristics on when to do
something, and this tries to get the same result by just removing old
heuristics that didn't make much sense.

	Linus

(*) Even my usual "careful" is not very careful by other peoples
standards. So when _I_ say that I wasn't very careful, you should just
assume that I was reading my email about as carefully as a hyper-active
hedgehog on some serious uppers. Can you say "ignored email" three times
quickly while chewing on an apple?

Ok,
 I've made a new pre-patch-2.1.131-3.

The basic problem (that Alexander Viro correctly diagnosed) is that the
inode locking was horribly and subtly wrong for the case of a "rmdir()"
call. What rmdir() did was essentially something like

 - VFS: lock the directory that contains the directory to remove
   (this is normal and required to make sure that the name updates are
   completely atomic - so removing or adding anything requires you to hold
   the lock on the directory that contains the removee/addee)
 - low-level filesystem: lock the directory you're going to remove, in
   order to atomically check that it's empty.

So far so good, the above makes tons of sense. HOWEVER, the problem is
fairly obvious if anybody before Alexander had actually bothered to think
about it: when we hold two locks, we had better make sure that we get the
locks in the right order, or we may end up deadlocked with two (or more)
processes getting the locks in the wrong order and waiting on each other.

Now, if it was only rmdir(), things would be fine, because the directory
hierarchy itself imposes a lock order for rmdir(). But we have another
case that needs to lock two directories: "rename()". And that one doesn't
have the same kind of obvious order, and uses a different way to order the
two locks it gets. BOOM.

As far as I can tell, this is a problem in 2.0.x too, but while it's a
potential really nasty DoS-opening, it does have the saving grace that the
window to trigger it is really really small. I don't know if you can
actually make an exploit for it that has any real chance of hitting it,
but it's at least conceptually possible.

Now, the only sane fix was to actually make the VFS layer do all the
locking for rmdir(), and thus let the VFS layer make sure the order is
correct, so that low-level filesystems don't need to worry their pretty
heads. I tried to do that in the previous pre-patch, and it worked well
for ext2, but not all that much else. The problem was that too many
filesystems "knew" what the rmdir() downcall used to do. Oh, well.

Anyway, I've fixed the low-level filesystems as far as I can tell, and the
end result is a much cleaner interface (and one less bug). But it's an
interface change at a fairly late date, and while the fixes to smbfs etc
looked for the most part obvious, I haven't been able to test them, so
I've done most of them "blind".

Sadly, this bug couldn't just be glossed over, because a normal user could
(by knowing the exact right incantation) force tons of unkillable
processes that held critical filesystem resources (any lookup on a
directory that was locked would in turn also lock up). So I'd ask people
who have done filesystems for Linux to look over my changes, and if the
filesystems are not part of the standard distribution please look over
your own locally maintained fs code. I think we can ignore 2.0.x by virtue
of it probably being virtually impossible to trigger. I'll leave the
decision up to Alan.

Most specially, I'd like to have people who use/maintain vfat and umsdos
filesystems to test out that I actually made those filesystems happy with
my changes. The other filesystems were more straightforward.

Oh, and thanks to Alexander. Not that I really needed another bug to fix,
but it feels good to plug holes.

                        Linus

The change is basically:
 - the VFS layer locks the directory to be removed for you (as opposed to
   just the directory that contains the directory to be removed as it used
   to). A lot of filesystems didn't actually do this, and it is required,
   because otherwise the test for an empty directory may be subverted by a
   clever hacker.
 - the VFS layer will have done a dcache "prune" operation on the
   directory, and if there were no other uses for that dcache entry, it
   will have done a "d_drop()" on it too.
 - the above essentially means that any filesystem can do a
        if (!list_empty(&dentry->d_hash))
                return -EBUSY;
   to test whether there are other users of this directory. No need to do
   any extra pruning etc - if it's been dropped there won't be any new
   users of the dentry afterwards, so there are no races. So after doing
   the above test you know that you'll have exclusive access to the dentry
   forever.
   Most notably, the low-level filesystem should _not_ look at the
   dentry->d_count member to see how many users there are. The VFS layer
   currently artificially raises the dentry count to make sure
   "d_delete()" doesn't get rid of the inode early.
 - however: traditional local UNIX-type filesystems tend to want to allow
   removing of the directory even if it is in use by something else. This
   requires that the inode be accessible even after the rmdir() - even
   though it doesn't necessarily need to actually _do_ anything.
   For a normal UNIX-like filesystem this tends to be trivial and quite
   automatic behaviour, but you need to think about whether your
   filesystem is of the kind where the inode stays around even after the
   delete until we locally do the final "iput()". For example, on
   networked filesystems this is generally not true, simply because the
   server will have de-allocated the inode even if we still have a
   reference to it locally.

There's a new pre-patch for people who want to test these things out: I'll
probably make a real 2.1.130 soon just to make sure all the silly problems
in 2.1.129 are left behind (ie the UP flu in particular that people are
still discussing even though there's a known cure).

The pre-patch fixes a rather serious problem with wall-clock itimer
functions, that admittedly was very very hard to trigger in real life (the
only reason we found it was due to the diligent help from John Taves that
saw sporadic problems under some very specific circumstances - thanks
John).

It also fixes a very silly NFS path revalidation issue: when we
revalidated a cached NFS path component, we didn't update the revalidation
time, so we ended up doing a lookup over the wire every time after the
first time - essentially making the dcache useless for path component
caching of NFS. If you use NFS heavily, you _will_ notice this change (it
also fixes some rather ugly uses of dentries and inodes in the NFS code
where we didn't update the counter so the inode wasn't guaranteed to even
be there any more!).

Also, thanks to Richard Gooch &co, who found the rather nasty race
condition when a kernel thread was started from an init-region. The
trivial fix was to not have the kernel thread function be inlined, but
while fixing it was trivial, it wasn't trivial to notice in the first
place. Good debugging.

And the UP flu is obviously fixed here (as it was in earlier pre-patches
and in various other patches floating around).

                        Linus