1. 30 Jan, 2019 13 commits
    • Han-Wen Nienhuys's avatar
      unionfs: copy fuse.Attr before modifying return value of branch cache · 32b02fb6
      Han-Wen Nienhuys authored
      Fixes a race condition.
      32b02fb6
    • Han-Wen Nienhuys's avatar
    • Han-Wen Nienhuys's avatar
      unionfs: clarify test error message · 40675bc3
      Han-Wen Nienhuys authored
      40675bc3
    • Han-Wen Nienhuys's avatar
    • Jakob Unterwurzacher's avatar
      fuse: improve BATCH_FORGET debug log output · 8020c188
      Jakob Unterwurzacher authored
      Node numbers are now written as "i1234" everywhere except in the
      BATCH_FORGET output, fix that. Also show the transaction number
      so we can follow where each forget comes from.
      
      As a normal FORGET looks like this:
      
      	2019/01/27 14:38:07 rx 1488: FORGET i822 {Nlookup=1}
      
      with the new output, "grep FORGET i1234 " will catch both single forgets
      and batch forgets.
      
      Before:
      
      	2019/01/20 18:15:31 rx 1372: BATCH_FORGET i0 {Count=3}  48b
      	2019/01/20 18:15:31 doBatchForget: forgetting 1 of 3: NodeId: 40, Nlookup: 1
      	2019/01/20 18:15:31 doBatchForget: forgetting 2 of 3: NodeId: 41, Nlookup: 1
      	2019/01/20 18:15:31 doBatchForget: forgetting 3 of 3: NodeId: 42, Nlookup: 1
      
      After:
      
      	2019/01/27 14:48:35 rx 2982: BATCH_FORGET i0 {Count=2}  32b
      	2019/01/27 14:48:35 doBatchForget: rx 2982 1/2: FORGET i804 {Nlookup=1}
      	2019/01/27 14:48:35 doBatchForget: rx 2982 2/2: FORGET i803 {Nlookup=1}
      8020c188
    • Kirill Smelkov's avatar
      nodefs: Don't crash on handleless WithFlags opens · 402331d4
      Kirill Smelkov authored
      Since https://git.kernel.org/linus/7678ac5061 FUSE filesystems were allowed to
      return ENOSYS from open and, open seeing this, the kernel will not send any
      open request anymore and just use Fh=0 for all opened files. This is handy for
      filesystems that don't have any per-opened-file state.
      
      However if there is at least one node for which opened files should have
      their own state, it is not correct to return ENOSYS from open _ever_, as that
      would prevent the kernel to call open on all nodes - in particular on the node
      where open needs to create a state associated with the file handle(*).
      
      It is already explicitly documented that Node.Open can return File=nil
      in which case filesystem operations like Read and Write will be called
      on the node directly. This way a filesystem could try to simulate an
      Open that was returning ENOSYS with Open that returns File=nil. However
      it is not the same as ENOSYS open also prevents the kernel from dropping
      the file cache:
      
      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/fuse/file.c?id=v5.0-rc3-241-g7c2614bf7a1f#n131
      
      From this point of view Opens that were returning ENOSYS should be
      replaced with Open that returns WithFlags{File=nil, flags=FOPEN_KEEP_CACHE}.
      
      Unfortunately if one tries to do that, go-fuse segfaults as the test
      included in this patch demonstrates:
      
      	13:36:53.238706 rx 13: LOOKUP i1 ["world.txt"] 10b
      	13:36:53.238745 tx 13:     OK, {i4 g3 tE=1s tA=1s {M0100644 SZ=5 L=1 1000:1000 B0*0 i0:4 A 0.000000 M 0.000000 C 0.000000}}
      	13:36:53.238780 rx 14: OPEN i4 {O_RDONLY,0x8000}
      	13:36:53.238791 tx 14:     OK, {Fh 2 CACHE}				<-- NOTE Fh != 0
      	13:36:53.238706 rx 12: RELEASE i3 {Fh 0 NONBLOCK,0x8000  L0}
      	13:36:53.238804 tx 12:     OK
      	13:36:53.238823 rx 15: READ i4 {Fh 2 [0 +4096)  L 0 NONBLOCK,0x8000}
      	13:36:53.238830 tx 15:     OK,  5b data "world"
      	13:36:53.238846 rx 16: GETATTR i4 {Fh 2}
      	13:36:53.238865 tx 16:     OK, {tA=1s {M0100644 SZ=5 L=1 1000:1000 B0*0 i0:4 A 0.000000 M 0.000000 C 0.000000}}
      	13:36:53.238879 rx 17: FLUSH i4 {Fh 2}
      	panic: runtime error: invalid memory address or nil pointer dereference
      	[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x539f9f]
      
      	goroutine 6 [running]:
      	github.com/hanwen/go-fuse/fuse/nodefs.(*rawBridge).Flush(0xc000010960, 0xc000104180, 0x0)
      	        /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/nodefs/fsops.go:490 +0x6f
      	github.com/hanwen/go-fuse/fuse.doFlush(0xc0000de000, 0xc000104000)
      	        /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/opcode.go:373 +0x42
      	github.com/hanwen/go-fuse/fuse.(*Server).handleRequest(0xc0000de000, 0xc000104000, 0xc000104000)
      	        /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/server.go:431 +0x26b
      	github.com/hanwen/go-fuse/fuse.(*Server).loop(0xc0000de000, 0x0)
      	        /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/server.go:403 +0x18f
      	github.com/hanwen/go-fuse/fuse.(*Server).Serve(0xc0000de000)
      	        /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/server.go:331 +0x6d
      	created by github.com/hanwen/go-fuse/fuse/test.TestNoFile
      	        /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/test/nofile_test.go:73 +0x442
      
      Fix it by teaching registerFileHandle not to register any handle at all
      and return Fh=0, if the inner file (file itself, or the file that was wrapped
      with WithFlags) is nil.
      
      After the patch the trace for world.txt open/read (that is opened with
      WithFlags{File=nil, Flags=FOPEN_KEEP_CACHE} is as follows.
      
      	14:59:31.714048 rx 13: LOOKUP i1 ["world.txt"] 10b
      	14:59:31.714062 tx 13:     OK, {i4 g3 tE=1s tA=1s {M0100644 SZ=5 L=1 1000:1000 B0*0 i0:4 A 0.000000 M 0.000000 C 0.000000}}
      	14:59:31.714081 rx 14: OPEN i4 {O_RDONLY,0x8000}
      	14:59:31.714091 tx 14:     OK, {Fh 0 CACHE}				<-- NOTE Fh = 0
      	14:59:31.714123 rx 15: READ i4 {Fh 0 [0 +4096)  L 0 NONBLOCK,0x8000}
      	14:59:31.714132 tx 15:     OK,  5b data "world"
      	14:59:31.714150 rx 16: GETATTR i4 {Fh 0}
      	14:59:31.714159 tx 16:     OK, {tA=1s {M0100644 SZ=5 L=1 1000:1000 B0*0 i0:4 A 0.000000 M 0.000000 C 0.000000}}
      	14:59:31.714181 rx 17: FLUSH i4 {Fh 0}
      	14:59:31.714186 tx 17:     OK
      	14:59:31.714202 rx 18: RELEASE i4 {Fh 0 NONBLOCK,0x8000  L0}
      	14:59:31.714208 tx 18:     OK
      
      (*) my particular case is filesystem where many nodes are just data, but
      additionally there are files that behave like sockets - for every client
      who opens such file the filesystem establishes separate bidirectional
      channel for control exchange via that stream:
      
      https://lab.nexedi.com/kirr/wendelin.core/blob/a50da567fd/wcfs/misc.go#L205
      402331d4
    • Han-Wen Nienhuys's avatar
      unionfs: fix capitalization of TTL · f45e18d6
      Han-Wen Nienhuys authored
      f45e18d6
    • Han-Wen Nienhuys's avatar
      fuse/nodefs: protect mutable memnode fields with lock · 7f7894d2
      Han-Wen Nienhuys authored
      This fixes race conditions found by the race detector.
      7f7894d2
    • Han-Wen Nienhuys's avatar
      unionfs: update in-memory dircache as the last step. · e6fa1a12
      Han-Wen Nienhuys authored
      If we call AddEntry/RemoveEntry before the update to the deletion list
      in storage, a poorly timed cache refill that happens just before the
      storage update will result in storing the old state for as long as the
      cache's TTL.
      e6fa1a12
    • Han-Wen Nienhuys's avatar
      e00a5f2a
    • Han-Wen Nienhuys's avatar
      aed5935e
    • Han-Wen Nienhuys's avatar
    • Han-Wen Nienhuys's avatar
      Use subtests for rename tests · 33c1cab5
      Han-Wen Nienhuys authored
      33c1cab5
  2. 23 Jan, 2019 1 commit
    • Jakob Unterwurzacher's avatar
      server: print debug message on ENODEV · dfd0ceb2
      Jakob Unterwurzacher authored
      This makes it clear when (and why) the filesystem was unmounted.
      
      Example output:
      
      $ gocryptfs -fg -nosyslog -fusedebug a b
      Password:
      Decrypting master key
      Filesystem mounted and ready.
      2019/01/23 21:52:59 rx 2: ACCESS i1 {r}
      2019/01/23 21:52:59 tx 2:     OK
      2019/01/23 21:52:59 rx 3: LOOKUP i1 [".Trash"] 7b
      2019/01/23 21:52:59 tx 3:     OK, {i0 g0 tE=1s tA=0s {M00 SZ=0 L=0 0:0 B0*0 i0:0 A 0.000000 M 0.000000 C 0.000000}}
      2019/01/23 21:52:59 rx 4: LOOKUP i1 [".Trash-1026"] 12b
      2019/01/23 21:52:59 tx 4:     OK, {i0 g0 tE=1s tA=0s {M00 SZ=0 L=0 0:0 B0*0 i0:0 A 0.000000 M 0.000000 C 0.000000}}
      2019/01/23 21:53:02 received ENODEV (unmount request), thread exiting
      2019/01/23 21:53:02 received ENODEV (unmount request), thread exiting
      2019/01/23 21:53:02 received ENODEV (unmount request), thread exiting
      dfd0ceb2
  3. 11 Jan, 2019 1 commit
    • Kirill Smelkov's avatar
      readme: Add appendix about how to read Go-FUSE debug log output · 425e8d53
      Kirill Smelkov authored
      d1c826d1 ("fuse: increase signal/noise in log messages",
      https://github.com/hanwen/go-fuse/pull/249) changed log to use
      abbreviations in the name of improving signal/noise ratio. However it
      might be not evident offhand what an abbreviation means.
      
      Add a short reference in the readme with the summary on how to read the
      log and a short example.
      
      The summary is incomplete - for example I don't describe Fh and other
      attributes that d1c826d1 did not change. I propose we review logging
      more, e.g. consider doing `Fh X` -> `fX` and maybe similarly for other
      fields, and while doing so, also populate/update the summary and example
      in the readme. In other words it is good to document current state, but
      it is not fixed in stone and can be evolved.
      425e8d53
  4. 08 Jan, 2019 1 commit
    • Kirill Smelkov's avatar
      fuse: increase signal/noise in log messages · d1c826d1
      Kirill Smelkov authored
      - use tA=...s instead of `A...` or `AttrValid=...` for time(Attr valid);
      - use tE=...s instead of `EntryValid` for time(Entry valid);
      - use iX instead of `NodeId: X` for inode X;
      - use gX instead of `Generation=X` for generation X;
      - use `[<off> +<size>)` instead of `off <off> sz <sz>`;
      
      - use
      
              rx uniq: Op iX opargs
              tx uniq: status reply...
      
        instead of
      
              Dispatch uniq: Op NodeId: X opargs
              Serialize uniq: Op code: status reply...
      
        for received/replied messages;
      
      - use `["file1", "file2"]` instead of `names: [file1, file2]` for filenames.
      
      - use `xb` instead of `x bytes`
      
      - for replied data, also log data content, but only up to first 8 bytes.
        This does not flood the log, but at the same time makes filesystem
        diagnostics easier.
      d1c826d1
  5. 17 Dec, 2018 2 commits
  6. 12 Dec, 2018 2 commits
  7. 11 Dec, 2018 1 commit
  8. 27 Oct, 2018 1 commit
  9. 26 Oct, 2018 1 commit
  10. 15 Oct, 2018 1 commit
  11. 11 Oct, 2018 2 commits
    • Kirill Smelkov's avatar
      fuse: Correct NOTIFY constants/operation names · b760b557
      Kirill Smelkov authored
      - it is not NOTIFY_INVAL_DELETE, but just NOTIFY_DELETE as the kernel is
        notified of entry being deleted and there is no invalidation here.
        uapi/linux/fuse.h does not use "_INVAL_" for NOTIFY_DELETE neither.
      
      - similarly for "notify store/retrieve" _INVAL_ is not appropriate and
        neither is used in uapi/linux/fuse.h . However since we took more
        explicit approach for "notify store/retrieve" naming (see bdca0e6a
        "Add support for store notify") let's also add _CACHE suffix for
        "notify store/retrieve" operation names for consistency and for being
        less ambiguous.
      
      - for inode/entry invalidation operations, let's use _INVAL_ in internal
        _OP_NOTIFY_*, similarly to how it is used in corresponding NOTIFY_*
        constant. For example:
      
      	_OP_NOTIFY_ENTRY -> _OP_NOTIFY_INVAL_ENTRY	(corresponds to NOTIFY_INVAL_ENTRY)
      b760b557
    • Han-Wen Nienhuys's avatar
      fuse/nodefs: fix some lint errors · 6df8ddc4
      Han-Wen Nienhuys authored
      6df8ddc4
  12. 10 Oct, 2018 1 commit
  13. 09 Oct, 2018 9 commits
  14. 08 Oct, 2018 3 commits
    • Jakob Unterwurzacher's avatar
      pathfs: fix possible nil pointer dereference in GetAttr · 47211c2b
      Jakob Unterwurzacher authored
      gocryptfs user Felix Lechner reported a nil pointer dereference
      in GetAttr: https://github.com/rfjakob/gocryptfs/issues/260
      
      The crash is in line
      
      	n.setClientInode(fi.Ino)
      
      because fi is nil.
      
      This can happen when file.GetAttr() returns an error code other than
      ENOSYS and EBADF. For gocryptfs, this can only happen when an open
      file descriptor breaks. In this case it was triggered by a failing
      NFS volume.
      
      Fix the crash by erroring out for error codes that are not handled
      by the retry-by-path logic.
      47211c2b
    • Kirill Smelkov's avatar
      fuse: A bit better package documentation rendering in godoc · f4f7205f
      Kirill Smelkov authored
      Current state is: fuse's godoc package-level description is completely
      empty because there is a blank link between package-level description
      and package clause.
      
      Yes, a user still has to go through api.go and read it in full, but
      having at least something for the description is a better start, and
      removing one line is easy.
      
      We already do it this way e.g. for nodefs and in other places.
      f4f7205f
    • Kirill Smelkov's avatar
      Add support for store notify · bdca0e6a
      Kirill Smelkov authored
      I'm writing a networked filesystem which reads data in 2MB blocks from
      remote database. However read requests from the kernel come in much
      smaller sizes - for example 4K-128K.
      
      Since it would be very slow to refetch the 2MB block for e.g. every
      consecutive 4K reads, a cache for fetched data is needed. A custom cache
      would do, however since the kernel already implements pagecache to cache
      file data, it is logical to use it directly.
      
      FUSE protocol provides primitives for pagecache control. We already have
      support for e.g. invalidating a data region for inode (InodeNotify), but
      there is more there. In particular it is possible to store and
      retrieve data into/from the kernel cache with "store notify" and
      "retrieve notify" messages:
      
      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fuse.h?id=v4.19-rc6-177-gcec4de302c5f#n68
      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fuse.h?id=v4.19-rc6-177-gcec4de302c5f#n756
      
      https://git.kernel.org/linus/a1d75f2582
      https://git.kernel.org/linus/2d45ba381a
      
      This patch adds support for "store notify". Adding support for "retrieve
      notify" might be added later since a) it is not needed right now, and b)
      supporting it is a bit more work since the kernel sends a separate reply
      with data and infrastructure has to be added to glue it back to original
      "retrieve notify" server-originated request.
      
      For user-visible API I decided not to duplicate FUSE-protocol naming
      1-1 and to be more explicit in names - emphasizing it is about cache
      control - it is e.g. InodeNotifyStoreCache instead of
      InodeNotifyStore, and for retrieving it (hopefully) should be just
      InodeRetrieveCache instead of InodeNotifyRetrieveCache and a
      separate callback.
      
      Thanks beforehand,
      Kirill
      bdca0e6a
  15. 01 Oct, 2018 1 commit
    • Kirill Smelkov's avatar
      fuse: Fix thinko in SupportsVersion · 8ccb625b
      Kirill Smelkov authored
      Current code was requiring that both
      
      	in.Major >= maj		_and_
      	in.Minor >= min
      
      this way e.g. kernel's 7.27 would satisfy requirement of e.g. 7.25, but
      8.1 won't do. Fix it.
      8ccb625b