1. 25 Aug, 2021 4 commits
    • Jakob Unterwurzacher's avatar
      Rewrite defunct all.bash · 45c0d70f
      Jakob Unterwurzacher authored
      all.bash is broken for some time now (since the move to v2?).
      Rewrite and simplify it using "./..." syntax.
      
      And add the binaries that are generated to gitignore.
      
      all.bash will be used in the upcomming patch for Github Actions CI.
      
      Change-Id: Ia1b2e6bc60486b419408d4efd50bef4b8dd62bae
      45c0d70f
    • Jakob Unterwurzacher's avatar
      fuse: move parseFuseFd() to unbreak darwin build · 61df8108
      Jakob Unterwurzacher authored
      Unfortunately, I broke darwin with the last commit:
      
      + GOOS=darwin
      + GOARCH=amd64
      + go build -tags without_openssl -o /dev/null
      ../../../../pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.1-0.20210825070001-74a933d6/fuse/server.go:122:5: undefined: parseFuseFd
      ../../../../pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.1-0.20210825070001-74a933d6/fuse/server.go:887:5: undefined: parseFuseFd
      
      Move parseFuseFd() to build on all archs to unbreak the build.
      
      Change-Id: Ice173cc70a6a95765b56b444623b68ed92382052
      61df8108
    • Jakob Unterwurzacher's avatar
      fuse: support special /dev/fd/N mountpoint · 74a933d6
      Jakob Unterwurzacher authored
      libfuse introduced [1] a special `/dev/fd/N` syntax for the mountpoint:
      It means that a privileged parent process:
      
       * Opened /dev/fuse
       * Called mount() on a real mountpoint directory
       * Inherited the fd to /dev/fuse to us
       * Informs us about the fd number via /dev/fd/N
      
      This functionality is used to allow FUSE mounts inside containers
      that have neither root permissions nor suid binaries [2], and
      for the --drop_privileges flag of mount.fuse3 [4]
      
      Tested with singularity and gocryptfs and actually works [3].
      
      v2: Added doccomment for NewServer.
      v3: Added specific error message on Server.Unmount().
      v4: Moved mount details to package comment
      
      [1] https://github.com/libfuse/libfuse/commit/64e11073b9347fcf9c6d1eea143763ba9e946f70
      [2] https://github.com/rfjakob/gocryptfs/issues/590
      [3] $ singularity run --fusemount "host:gocryptfs --extpass echo --extpass test /tmp/a /mnt" docker://ubuntu
          INFO:    Using cached SIF image
          Reading password from extpass program "echo", arguments: ["test"]
          Decrypting master key
          bash: /home/jakob/.cargo/env: No such file or directory
          bash: /home/jakob/.cargo/env: No such file or directory
          bash: /home/jakob/.cargo/env: No such file or directory
          Singularity> Filesystem mounted and ready.
      [4] man mount.fuse3
      
      Change-Id: Ibcc2464b0ef1e3d236207981b487fd9a7d94c910
      74a933d6
    • Jakob Unterwurzacher's avatar
      fuse: refer to fs instead of nodefs & pathfs · 4ba0b1e3
      Jakob Unterwurzacher authored
      Update the package comment to refer to the fs package
      instead of nodefs and pathfs, which are deprecated.
      
      Change-Id: I317e53152b63b982298a7de29120191488dcd362
      4ba0b1e3
  2. 12 Aug, 2021 1 commit
  3. 02 Aug, 2021 2 commits
  4. 01 Aug, 2021 1 commit
    • Jakob Unterwurzacher's avatar
      Run "go mod tidy" · efbe2d7d
      Jakob Unterwurzacher authored
      Drops an obsolete entry pointing to go-fuse v1.0.0.
      
      Change-Id: Iaf8e880d6f39e63b593a7929b8b4ff649c1d7807
      efbe2d7d
  5. 30 Jul, 2021 2 commits
    • Jakob Unterwurzacher's avatar
      fs: fix Inode.IsDir logic · 458f01e0
      Jakob Unterwurzacher authored
      The file type is not a bitmap, and looking at
      
      	S_IFBLK                          = 0x6000
      	S_IFCHR                          = 0x2000
      	S_IFDIR                          = 0x4000
      	S_IFIFO                          = 0x1000
      	S_IFLNK                          = 0xa000
      	S_IFREG                          = 0x8000
      	S_IFSOCK                         = 0xc000
      
      and confirmed by the added test, we used to classify
      block devices and sockets as directories:
      
       --- FAIL: TestInodeIsDir (0.00s)
           inode_test.go:25: wrong result for case struct { mode uint32; dir bool }{mode:0x6000, dir:false}
           inode_test.go:25: wrong result for case struct { mode uint32; dir bool }{mode:0xc000, dir:false}
      
      The check is fixed now and the test passes.
      
      Change-Id: I6490992d1fecc8a6bea7c2c4b2f1bca765d03d4c
      458f01e0
    • Jakob Unterwurzacher's avatar
      fuse: fix deadlock in TestDeleteNotify · bba1094c
      Jakob Unterwurzacher authored
      This
      
      	GOMAXPROCS=1 go test -v -count=1 -run TestDeleteNotify ./fuse/test
      
      deadlocked reliably, apparently in the chdir() from cmd.Start().
      
      We can do without the chdir() by using an absolte path.
      This avoids the issue.
      
      Fixes https://github.com/hanwen/go-fuse/issues/261
      
      Change-Id: Ia9c8f15819c125c5bf94b085fa4c5f2977a6789a
      bba1094c
  6. 11 Jun, 2021 2 commits
  7. 08 Jun, 2021 1 commit
    • Tommy Lindgren's avatar
      fuse: suppress ENOENT warnings for INTERRUPT responses · eefea5e0
      Tommy Lindgren authored
      If writing the INTERRUPT response returns ENOENT it means that the
      referred request is no longer known by the kernel. This is a normal if
      the referred request already has completed. We thus suppress the scary
      warning unless debugging is enabled.
      
      Fixes #375.
      
      Change-Id: I3266dbd8ad53a42db9e0e746e4734f284746b76c
      eefea5e0
  8. 03 Jun, 2021 1 commit
  9. 08 May, 2021 2 commits
  10. 23 Apr, 2021 2 commits
    • Jakob Unterwurzacher's avatar
      fs: track most recent parent · a90e1f46
      Jakob Unterwurzacher authored
      Introduce the new struct inodeParents that wraps a map
      and one special slot for the most recent parent. Unit tests
      included.
      
      Because the map is lazily initialized, we should save some
      memory on the common single-parent case (= file with no hard links)
      compared to before.
      
      Benchmarking with gocryptfs shows no discernible change
      in performance. fsstress testing with gocryptfs shows no issues.
      
      TestStaleHardlinks the previous commit passes now.
      
      Change-Id: I8d69093abc906addde751a9e70dbd78a3a61371a
      a90e1f46
    • Jakob Unterwurzacher's avatar
      fs: tests: add TestStaleHardlinks (fails at the moment) · c3186132
      Jakob Unterwurzacher authored
      This test shows a problem in selecting a random path
      for a node when there is more than one.
      
      The failure looks like this:
      
        18:41:50.796468 rx 136: LOOKUP n1 ["link0"] 6b
        18:41:50.796489 tx 136:     OK, {n2 g1 tE=0s tA=0s {M0100600 SZ=0 L=1 1026:1026 B0*4096 i0:269663 A 1616348510.793212 M 1616348510.793212 C 1616348510.795212}}
        18:41:50.796535 rx 138: OPEN n2 {O_RDONLY,0x8000}
        18:41:50.796557 tx 138:     2=no such file or directory, {Fh 0 }
      
      The LOOKUP succeeds (because the file "link0" is there).
      But the OPEN fails because go-fuse chooses another (stale) path.
      
      I will try to make the behavoir more robust against changes
      behind our back, but this patch only adds a test to show the
      problem.
      
      Change-Id: I39b31ba717ddaaad7dda6ecd86707c75cd25102e
      c3186132
  11. 12 Apr, 2021 1 commit
  12. 10 Apr, 2021 2 commits
  13. 22 Mar, 2021 1 commit
  14. 01 Feb, 2021 1 commit
    • Tommy Lindgren's avatar
      fuse: increase max readers for better throughput · 0f728ba1
      Tommy Lindgren authored
      Previously maximum number of requests readers was hard-coded to 2.
      Setting it to max(2, min(16, GOMAXPROCS)) improves the throughput on
      systems with more cores available. Going beyond 16 readers seems to hurt
      performance even if there are more cores available.
      
      Benchmark GoFuseRead can be used for demonstrating the effects of this
      variable. The benchmark reads 32 streams in parallel from a dummy
      filesystem (read operations immediately return zeros). Example results
      from an i7-8550U (8 cores):
      
      | Max readers | Total throughput |
      | ----------: | ---------------: |
      |           2 |       13217 MB/s |
      |           4 |       19202 MB/s |
      |           8 |       19973 MB/s |
      |          16 |       18994 MB/s |
      
      On a 96 core system:
      
      | Max readers | Total throughput |
      | ----------: | ---------------: |
      |           2 |       11490 MB/s |
      |           4 |       16129 MB/s |
      |           8 |       24263 MB/s |
      |          16 |       29568 MB/s |
      |          32 |       28262 MB/s |
      
      Note that improvements won't be as dramatic for real filesytem
      implementations. In benchmarks for a filesystem doing real work I see a
      30-40% improvement (8.3 -> 11.4 GB/s) on the 96 core system.
      
      Also tweaked some of the other benchmarks so they don't leave behind
      mountpoints.
      
      Fixes #388.
      
      Change-Id: Ibff17d7fc92195f078a9ccff818a31f3a58873f2
      0f728ba1
  15. 25 Jan, 2021 1 commit
  16. 21 Jan, 2021 1 commit
    • Han-Wen Nienhuys's avatar
      fs: allow extending LoopbackNode · e29d3106
      Han-Wen Nienhuys authored
      Make LoopbackNode and LoopbackRoot public, and add an example of how
      to extend them.
      
      Fixes #364.
      
      Change-Id: Idcce1bf3765e93ece3bcf9d14de8a957b8c6b63a
      e29d3106
  17. 04 Jan, 2021 1 commit
  18. 31 Dec, 2020 1 commit
  19. 27 Dec, 2020 4 commits
  20. 09 Dec, 2020 1 commit
    • Han-Wen Nienhuys's avatar
      fuse: fix pollHack response to SETATTR · 9473722e
      Han-Wen Nienhuys authored
      On OSX, syscall.O_TRUNC on the Create call apparently triggers a
      SETATTR call with size 0, which must be responded to correctly.
      
      Change-Id: Icb99891eae99ced6ffbd5fb1dadf42ffad687534
      9473722e
  21. 08 Dec, 2020 1 commit
    • Han-Wen Nienhuys's avatar
      fuse: try to fix mount errors on OSX · 4a458845
      Han-Wen Nienhuys authored
      Some of the error codes due to the poll hack get returned back to the
      OSX mount process, which leads to failing mounts.
      
      Change-Id: I5b10509c3b8173343caf3f67c0e12a25ddfae32b
      4a458845
  22. 01 Dec, 2020 1 commit
    • Han-Wen Nienhuys's avatar
      fs: redeclare syscall.Dirent locally · 588c83d1
      Han-Wen Nienhuys authored
      syscall.Dirent declares a 256-byte "name" field. In reality, this
      field is only as long as necessary to transmit the name, so more
      entries can be packed in a page.
      
      When casting the result to syscall.Dirent, the result might run over
      the end of the byte buffer, which the race detector doesn't like.
      
      Address this by using our own flavor of syscall.Dirent. All linux
      flavors use the same structure layout, except for the amount of
      padding at the end.
      
      Change-Id: I618f7ef9e167894e1ab25b2113dcd20d7b76ceb4
      588c83d1
  23. 04 Nov, 2020 5 commits
    • Jakob Unterwurzacher's avatar
      fuse: debug logging: mark nodeids as "nXXX" · be8e5f4a
      Jakob Unterwurzacher authored
      nodeid and inode number used to be identical, which is
      why both could be marked as "iXXX". This is no longer the
      case, so make nodeids distinguishable.
      
      Example:
      
      23:47:17.637736 rx 9156: MKDIR n28 {0777 (00)} ["d90X"] 5b
      23:47:17.637801 tx 9156:     OK, {n112 g1 tE=1s tA=1s {M040755 SZ=4096 L=2 1026:1026 B8*4096 i0:2769795 A 1601761637.637385 M 1601761637.637385 C 1601761637.637385}}
      
      v4: also update orphaned inode warning
      
      Change-Id: Ib29dabfd4cf6692cf4118191f8dc73529562426f
      be8e5f4a
    • Jakob Unterwurzacher's avatar
      fs: split inode number tracking and kernel nodeid tracking · 1d0096e5
      Jakob Unterwurzacher authored
      Using the inode numbers as the nodeid causes problems
      when the fs reuses inode numbers. This is the case with
      any overlay filesystem that is backed by ext4 like the
      loopback example or gocryptfs.
      
      We already had the expSleep() and re-add hack,
      
      	7090b027 fs: wait out apparent inode type change
      	68f70527 fs: addNewChild(): handle concurrent FORGETs
      
      to mitigate some of the problems (at the risk of deadlocking
      forever), but I could not find a way to work around the following
      case uncovered by fsstress:
      
      The kernel expects a fresh nodeid from MKDIR (see #372 for a
      debug log).
      
      This is now guaranteed by passing the O_EXCL to addNewChild().
      However, this also means that the hard link detection must
      happen in addNewChild() as opposed to newInodeUnlocked()
      before. The expSleep and re-add hacks are no longer needed
      and have been dropped.
      
      This survived 24 hours (42587 iterations) of fsstress testing.
      Tested was the loopback example running on top of ext4 on
      Linux 5.8.10.
      
      Fixes https://github.com/hanwen/go-fuse/issues/372 .
      
      v2: Rename inoMap -> stableAttrs, nodeidMap -> kernelNodeIds
          acc. to feedback from Han-Wen.
      v3: initialize Inode.nodeId early
      v4: drop leftover file stash.txt
      
      Change-Id: Ibb36a886f15d48727daa10b9717ea88e45a6b8af
      1d0096e5
    • Jakob Unterwurzacher's avatar
      fs: TestFsstress: add rename, link ; reduce runtime to 1 sec · 6a785da2
      Jakob Unterwurzacher authored
      With rename and link, it hangs within 20ms, so we can reduce the
      runtime.
      
      Also add deadlock detection & statistics printout.
      
      A good run looks like this:
      
      	$ TMPDIR=/var/tmp go test -run TestFsstress -v
      	=== RUN   TestFsstress
      	    simple_test.go:601: Operation statistics:
      	    simple_test.go:604:      creat:    33 ok, 132978 fail,  0 hung
      	    simple_test.go:604:       link:    85 ok,  37196 fail,  0 hung
      	    simple_test.go:604:      mkdir:    60 ok, 155273 fail,  0 hung
      	    simple_test.go:604:     mkfifo:    45 ok, 161937 fail,  0 hung
      	    simple_test.go:604: mknod_fifo:    55 ok, 154459 fail,  0 hung
      	    simple_test.go:604:  mknod_reg:    77 ok, 180614 fail,  0 hung
      	    simple_test.go:604: mknod_sock:    83 ok, 139479 fail,  0 hung
      	    simple_test.go:604:    readdir:    13 ok,      0 fail,  0 hung
      	    simple_test.go:604:     remove:   180 ok,     40 fail,  0 hung
      	    simple_test.go:604:     rename:   646 ok,    137 fail,  0 hung
      	    simple_test.go:604:      rmdir:    48 ok,  48342 fail,  0 hung
      	    simple_test.go:604:    symlink:    30 ok, 119954 fail,  0 hung
      	    simple_test.go:604:     unlink:   150 ok,   6121 fail,  0 hung
      
      	--- PASS: TestFsstress (1.02s)
      	PASS
      	ok  	github.com/hanwen/go-fuse/v2/fs	1.022s
      
      A bad run like this:
      
      	go-fuse/fs$ TMPDIR=/var/tmp go test -run TestFsstress -v
      	=== RUN   TestFsstress
      	    simple_test.go:592: timeout waiting for goroutines to exit (deadlocked?)
      	    simple_test.go:601: Operation statistics:
      	    simple_test.go:604:      creat:     0 ok,     44 fail, 10 hung
      	    simple_test.go:604:       link:     0 ok,      6 fail, 10 hung
      	    simple_test.go:604:      mkdir:     1 ok,    283 fail, 10 hung
      	    simple_test.go:604:     mkfifo:     1 ok,    140 fail, 10 hung
      	    simple_test.go:604: mknod_fifo:     1 ok,   1246 fail, 10 hung
      	    simple_test.go:604:  mknod_reg:     0 ok,     19 fail, 10 hung
      	    simple_test.go:604: mknod_sock:     0 ok,    110 fail, 10 hung
      	    simple_test.go:604:    readdir:     1 ok,      0 fail,  1 hung
      	    simple_test.go:604:     remove:     1 ok,      0 fail, 10 hung
      	    simple_test.go:604:     rename:     0 ok,     12 fail, 10 hung
      	    simple_test.go:604:      rmdir:     1 ok,    346 fail, 10 hung
      	    simple_test.go:604:    symlink:     1 ok,   1456 fail, 10 hung
      	    simple_test.go:604:     unlink:     2 ok,     15 fail, 10 hung
      	    simple_test.go:51: /usr/bin/fusermount: failed to unmount /var/tmp/TestFsstress302646914/mnt: Device or resource busy
      	         (code exit status 1)
      
      	--- FAIL: TestFsstress (2.30s)
      	FAIL
      
      	[hangs]
      
      Change-Id: I9eb0ce02eb1df766be257d6a87de53c6ffa6d21e
      6a785da2
    • Jakob Unterwurzacher's avatar
      fs: add TestFsstress · e5a9167d
      Jakob Unterwurzacher authored
      TestFsstress is loosely modeled after xfstest's fsstress. It performs rapid
      parallel removes / creates / readdirs. Coupled with inode reuse, this test
      used to deadlock go-fuse quite quickly.
      
      Change-Id: Ia3af099e08c74d3c160062c1ed414430b6c1e7c9
      e5a9167d
    • Jakob Unterwurzacher's avatar
      fs: detect lookupCount underflow · e9983956
      Jakob Unterwurzacher authored
      If this happens, it means our node accounting is messed up.
      Abort the process by panicing instead of wrapping around
      to UINT_MAX.
      
      Change-Id: Id662ab03877d44a16785bfd164a96173a3b7519f
      e9983956
  24. 07 Oct, 2020 1 commit
    • Kirill Smelkov's avatar
      nodefs: Skip TestNodeParallelLookup on kernels that don't have CAP_PARALLEL_DIROPS · 3a8e4783
      Kirill Smelkov authored
      If kernel internally serializes lookup requests for a directory, the
      test will deadlock, becaue it tries to issue two such request and pause
      lookup handlers, untill both requests come in.
      
      For example here is how this test hangs on Ubuntu Xenial 16.04:
      
      kirr@kubu:~/t/go-fuse$ go test -count=1 -v -run TestNodeParallelLookup ./fuse/test/
      === RUN   TestNodeParallelLookup
      18:24:38.494651 rx 1: INIT i0 {7.23 Ra 0x20000 POSIX_LOCKS,ATOMIC_O_TRUNC,EXPORT_SUPPORT,DONT_MASK,FLOCK_LOCKS,READDIRPLUS_AUTO,ASYNC_DIO,WRITEBACK_CACHE,SPLICE_WRITE,SPLICE_READ,IOCTL_DIR,AUTO_INVAL_DATA,READDIRPLUS,NO_OPEN_SUPPORT,ASYNC_READ,BIG_WRITES,SPLICE_MOVE}
      18:24:38.494713 tx 1:     OK, {7.23 Ra 0x20000 ASYNC_READ,BIG_WRITES,AUTO_INVAL_DATA,READDIRPLUS,NO_OPEN_SUPPORT 0/0 Wr 0x10000 Tg 0x0}
      18:24:38.495351 rx 2: LOOKUP i1 [".go-fuse-epoll-hack"] 20b
      18:24:38.495467 tx 2:     OK, {i18446744073709551615 g0 tE=0s tA=0s {M0100644 SZ=0 L=1 0:0 B0*0 i0:18446744073709551615 A 0.000000 M 0.000000 C 0.000000}}
      18:24:38.495530 rx 3: OPEN i18446744073709551615 {O_RDONLY,0x8000}
      18:24:38.495575 tx 3:     OK, {Fh 18446744073709551615 }
      18:24:38.495624 rx 4: POLL i18446744073709551615
      18:24:38.495659 tx 4:     38=function not implemented
      18:24:38.495706 rx 5: FLUSH i18446744073709551615 {Fh 18446744073709551615}
      18:24:38.495741 tx 5:     34=numerical result out of range
      18:24:38.495812 rx 6: RELEASE i18446744073709551615 {Fh 18446744073709551615 0x8000  L0}
      18:24:38.495849 tx 6:     34=numerical result out of range
      18:24:38.495904 rx 7: LOOKUP i1 ["world"] 6b
      18:24:38.495957 I: <- lookup "world"
      < hang >
      
      Amends: 17c0c400 (nodefs: Allow for several Lookup requests to be served simultaneously)
      Updates: https://github.com/hanwen/go-fuse/issues/261
      
      Change-Id: I5d0a76d1190af805351326c378f69cf8ff781a37
      3a8e4783