- 30 Jan, 2019 13 commits
-
-
Han-Wen Nienhuys authored
Fixes a race condition.
-
Han-Wen Nienhuys authored
-
Han-Wen Nienhuys authored
-
Han-Wen Nienhuys authored
-
Jakob Unterwurzacher authored
Node numbers are now written as "i1234" everywhere except in the BATCH_FORGET output, fix that. Also show the transaction number so we can follow where each forget comes from. As a normal FORGET looks like this: 2019/01/27 14:38:07 rx 1488: FORGET i822 {Nlookup=1} with the new output, "grep FORGET i1234 " will catch both single forgets and batch forgets. Before: 2019/01/20 18:15:31 rx 1372: BATCH_FORGET i0 {Count=3} 48b 2019/01/20 18:15:31 doBatchForget: forgetting 1 of 3: NodeId: 40, Nlookup: 1 2019/01/20 18:15:31 doBatchForget: forgetting 2 of 3: NodeId: 41, Nlookup: 1 2019/01/20 18:15:31 doBatchForget: forgetting 3 of 3: NodeId: 42, Nlookup: 1 After: 2019/01/27 14:48:35 rx 2982: BATCH_FORGET i0 {Count=2} 32b 2019/01/27 14:48:35 doBatchForget: rx 2982 1/2: FORGET i804 {Nlookup=1} 2019/01/27 14:48:35 doBatchForget: rx 2982 2/2: FORGET i803 {Nlookup=1}
-
Kirill Smelkov authored
Since https://git.kernel.org/linus/7678ac5061 FUSE filesystems were allowed to return ENOSYS from open and, open seeing this, the kernel will not send any open request anymore and just use Fh=0 for all opened files. This is handy for filesystems that don't have any per-opened-file state. However if there is at least one node for which opened files should have their own state, it is not correct to return ENOSYS from open _ever_, as that would prevent the kernel to call open on all nodes - in particular on the node where open needs to create a state associated with the file handle(*). It is already explicitly documented that Node.Open can return File=nil in which case filesystem operations like Read and Write will be called on the node directly. This way a filesystem could try to simulate an Open that was returning ENOSYS with Open that returns File=nil. However it is not the same as ENOSYS open also prevents the kernel from dropping the file cache: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/fuse/file.c?id=v5.0-rc3-241-g7c2614bf7a1f#n131 From this point of view Opens that were returning ENOSYS should be replaced with Open that returns WithFlags{File=nil, flags=FOPEN_KEEP_CACHE}. Unfortunately if one tries to do that, go-fuse segfaults as the test included in this patch demonstrates: 13:36:53.238706 rx 13: LOOKUP i1 ["world.txt"] 10b 13:36:53.238745 tx 13: OK, {i4 g3 tE=1s tA=1s {M0100644 SZ=5 L=1 1000:1000 B0*0 i0:4 A 0.000000 M 0.000000 C 0.000000}} 13:36:53.238780 rx 14: OPEN i4 {O_RDONLY,0x8000} 13:36:53.238791 tx 14: OK, {Fh 2 CACHE} <-- NOTE Fh != 0 13:36:53.238706 rx 12: RELEASE i3 {Fh 0 NONBLOCK,0x8000 L0} 13:36:53.238804 tx 12: OK 13:36:53.238823 rx 15: READ i4 {Fh 2 [0 +4096) L 0 NONBLOCK,0x8000} 13:36:53.238830 tx 15: OK, 5b data "world" 13:36:53.238846 rx 16: GETATTR i4 {Fh 2} 13:36:53.238865 tx 16: OK, {tA=1s {M0100644 SZ=5 L=1 1000:1000 B0*0 i0:4 A 0.000000 M 0.000000 C 0.000000}} 13:36:53.238879 rx 17: FLUSH i4 {Fh 2} panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x539f9f] goroutine 6 [running]: github.com/hanwen/go-fuse/fuse/nodefs.(*rawBridge).Flush(0xc000010960, 0xc000104180, 0x0) /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/nodefs/fsops.go:490 +0x6f github.com/hanwen/go-fuse/fuse.doFlush(0xc0000de000, 0xc000104000) /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/opcode.go:373 +0x42 github.com/hanwen/go-fuse/fuse.(*Server).handleRequest(0xc0000de000, 0xc000104000, 0xc000104000) /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/server.go:431 +0x26b github.com/hanwen/go-fuse/fuse.(*Server).loop(0xc0000de000, 0x0) /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/server.go:403 +0x18f github.com/hanwen/go-fuse/fuse.(*Server).Serve(0xc0000de000) /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/server.go:331 +0x6d created by github.com/hanwen/go-fuse/fuse/test.TestNoFile /home/kirr/src/neo/src/github.com/hanwen/go-fuse/fuse/test/nofile_test.go:73 +0x442 Fix it by teaching registerFileHandle not to register any handle at all and return Fh=0, if the inner file (file itself, or the file that was wrapped with WithFlags) is nil. After the patch the trace for world.txt open/read (that is opened with WithFlags{File=nil, Flags=FOPEN_KEEP_CACHE} is as follows. 14:59:31.714048 rx 13: LOOKUP i1 ["world.txt"] 10b 14:59:31.714062 tx 13: OK, {i4 g3 tE=1s tA=1s {M0100644 SZ=5 L=1 1000:1000 B0*0 i0:4 A 0.000000 M 0.000000 C 0.000000}} 14:59:31.714081 rx 14: OPEN i4 {O_RDONLY,0x8000} 14:59:31.714091 tx 14: OK, {Fh 0 CACHE} <-- NOTE Fh = 0 14:59:31.714123 rx 15: READ i4 {Fh 0 [0 +4096) L 0 NONBLOCK,0x8000} 14:59:31.714132 tx 15: OK, 5b data "world" 14:59:31.714150 rx 16: GETATTR i4 {Fh 0} 14:59:31.714159 tx 16: OK, {tA=1s {M0100644 SZ=5 L=1 1000:1000 B0*0 i0:4 A 0.000000 M 0.000000 C 0.000000}} 14:59:31.714181 rx 17: FLUSH i4 {Fh 0} 14:59:31.714186 tx 17: OK 14:59:31.714202 rx 18: RELEASE i4 {Fh 0 NONBLOCK,0x8000 L0} 14:59:31.714208 tx 18: OK (*) my particular case is filesystem where many nodes are just data, but additionally there are files that behave like sockets - for every client who opens such file the filesystem establishes separate bidirectional channel for control exchange via that stream: https://lab.nexedi.com/kirr/wendelin.core/blob/a50da567fd/wcfs/misc.go#L205
-
Han-Wen Nienhuys authored
-
Han-Wen Nienhuys authored
This fixes race conditions found by the race detector.
-
Han-Wen Nienhuys authored
If we call AddEntry/RemoveEntry before the update to the deletion list in storage, a poorly timed cache refill that happens just before the storage update will result in storing the old state for as long as the cache's TTL.
-
Han-Wen Nienhuys authored
-
Han-Wen Nienhuys authored
-
Han-Wen Nienhuys authored
-
Han-Wen Nienhuys authored
-
- 23 Jan, 2019 1 commit
-
-
Jakob Unterwurzacher authored
This makes it clear when (and why) the filesystem was unmounted. Example output: $ gocryptfs -fg -nosyslog -fusedebug a b Password: Decrypting master key Filesystem mounted and ready. 2019/01/23 21:52:59 rx 2: ACCESS i1 {r} 2019/01/23 21:52:59 tx 2: OK 2019/01/23 21:52:59 rx 3: LOOKUP i1 [".Trash"] 7b 2019/01/23 21:52:59 tx 3: OK, {i0 g0 tE=1s tA=0s {M00 SZ=0 L=0 0:0 B0*0 i0:0 A 0.000000 M 0.000000 C 0.000000}} 2019/01/23 21:52:59 rx 4: LOOKUP i1 [".Trash-1026"] 12b 2019/01/23 21:52:59 tx 4: OK, {i0 g0 tE=1s tA=0s {M00 SZ=0 L=0 0:0 B0*0 i0:0 A 0.000000 M 0.000000 C 0.000000}} 2019/01/23 21:53:02 received ENODEV (unmount request), thread exiting 2019/01/23 21:53:02 received ENODEV (unmount request), thread exiting 2019/01/23 21:53:02 received ENODEV (unmount request), thread exiting
-
- 11 Jan, 2019 1 commit
-
-
Kirill Smelkov authored
d1c826d1 ("fuse: increase signal/noise in log messages", https://github.com/hanwen/go-fuse/pull/249) changed log to use abbreviations in the name of improving signal/noise ratio. However it might be not evident offhand what an abbreviation means. Add a short reference in the readme with the summary on how to read the log and a short example. The summary is incomplete - for example I don't describe Fh and other attributes that d1c826d1 did not change. I propose we review logging more, e.g. consider doing `Fh X` -> `fX` and maybe similarly for other fields, and while doing so, also populate/update the summary and example in the readme. In other words it is good to document current state, but it is not fixed in stone and can be evolved.
-
- 08 Jan, 2019 1 commit
-
-
Kirill Smelkov authored
- use tA=...s instead of `A...` or `AttrValid=...` for time(Attr valid); - use tE=...s instead of `EntryValid` for time(Entry valid); - use iX instead of `NodeId: X` for inode X; - use gX instead of `Generation=X` for generation X; - use `[<off> +<size>)` instead of `off <off> sz <sz>`; - use rx uniq: Op iX opargs tx uniq: status reply... instead of Dispatch uniq: Op NodeId: X opargs Serialize uniq: Op code: status reply... for received/replied messages; - use `["file1", "file2"]` instead of `names: [file1, file2]` for filenames. - use `xb` instead of `x bytes` - for replied data, also log data content, but only up to first 8 bytes. This does not flood the log, but at the same time makes filesystem diagnostics easier.
-
- 17 Dec, 2018 2 commits
-
-
Kirill Smelkov authored
nodeMap.Handle(&node) == 0 means the kernel does not currently have an entry for this node in its dentry cache. And the kernel would return ENOENT if we would try to send it notify store message with nodeID it does not currently know. So return proper ENOENT when we see that currently there is no handle for the inode, instead of returning EINVAL, which means that call arguments - e.g. off is invalid. NOTE contrary to cache retrieve (where we could turn such error into empty OK read; see previous patch) for cache store we cannot ignore the error. Fixes bdca0e6a (Add support for store notify). See also: https://github.com/hanwen/go-fuse/pull/246#discussion_r242050595
-
Kirill Smelkov authored
This patch continues bdca0e6a (Add support for store notify) and adds corresponding support for counterpart to cache-store - to retrieve an inode data from kernel cache. As it was already noted in bdca0e6a, FUSE protocol provides primitives for pagecache control: to invalidate a data region for inode (notify_inval_inode), to store data into inode kernel's cache (notify_store), and to retrieve data from inode kernel's cache (notify_retrieve). For the latter 2 FUSE protocol messages and brief documentation about semantic can be seen here: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fuse.h?id=v4.19-rc6-177-gcec4de302c5f#n68 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fuse.h?id=v4.19-rc6-177-gcec4de302c5f#n756 https://git.kernel.org/linus/a1d75f2582 https://git.kernel.org/linus/2d45ba381a In short, to retrieve data from kernel cache, filesystem server sends S > C NOTIFY_RETRIEVE_CACHE{notifyUnique, inode, offset, size} and if that message was sent correctly, the kernel sends back another write-style message with unique=notifyUnique S < C NOTIFY_REPLY{inode, offset, size, data} Since so far there were no cases when a server was querying the kernel, and the reply comes as separate kernel "request", we have to add infrastructure for tracking such in-flight queries. This is done by adding Server.retrieveTab and friends. Otherwise the implementation is straightforward. A particular note is that from a user-level API point of view we are not following e.g. libfuse to register a callback to be invoked upon reply, but instead provide {Inode,File}RetrieveCache that synchronously send notify query and wait for kernel's reply. This fits more naturally to Go and is easier to use.
-
- 12 Dec, 2018 2 commits
-
-
Han-Wen Nienhuys authored
-
Han-Wen Nienhuys authored
-
- 11 Dec, 2018 1 commit
-
-
Natalie Fioretti authored
-
- 27 Oct, 2018 1 commit
-
-
Tsuyoshi Hombashi authored
-
- 26 Oct, 2018 1 commit
-
-
Natalie Fioretti authored
-
- 15 Oct, 2018 1 commit
-
-
Kirill Smelkov authored
I've made the notes for myself, but then though it could be maybe useful as the package documentation.
-
- 11 Oct, 2018 2 commits
-
-
Kirill Smelkov authored
- it is not NOTIFY_INVAL_DELETE, but just NOTIFY_DELETE as the kernel is notified of entry being deleted and there is no invalidation here. uapi/linux/fuse.h does not use "_INVAL_" for NOTIFY_DELETE neither. - similarly for "notify store/retrieve" _INVAL_ is not appropriate and neither is used in uapi/linux/fuse.h . However since we took more explicit approach for "notify store/retrieve" naming (see bdca0e6a "Add support for store notify") let's also add _CACHE suffix for "notify store/retrieve" operation names for consistency and for being less ambiguous. - for inode/entry invalidation operations, let's use _INVAL_ in internal _OP_NOTIFY_*, similarly to how it is used in corresponding NOTIFY_* constant. For example: _OP_NOTIFY_ENTRY -> _OP_NOTIFY_INVAL_ENTRY (corresponds to NOTIFY_INVAL_ENTRY)
-
Han-Wen Nienhuys authored
-
- 10 Oct, 2018 1 commit
-
-
Han-Wen Nienhuys authored
The debug printout in the -race tests interferes with timing, so it makes race conditions less likely to appear.
-
- 09 Oct, 2018 9 commits
-
-
Han-Wen Nienhuys authored
-
Han-Wen Nienhuys authored
There was a thinko in PathFS.Lookup: if GetAttr returns OK, we did not copy the attribute if there was a known child with that name. This lead returning status OK, with a zeroed out Attr structure. The 0 mode would sometimes lead to I/O errors, as files without type are not allowed. Fixes #239
-
Han-Wen Nienhuys authored
Otherwise, printing inodes generates panics, because of concurrent writes.
-
Han-Wen Nienhuys authored
-
Han-Wen Nienhuys authored
The CAP_POSIX_ACL and CAP_HANDLE_KILLPRIV opcodes were swapped.
-
Kirill Smelkov authored
@rfjakob reports that after bdca0e6a (Add support for store notify) build got broken on ARM: $ GOARCH=arm go build ./example/loopback # github.com/hanwen/go-fuse/fuse fuse/server.go:490:11: constant 4294967295 overflows int fuse/server.go:492:9: constant 4294967295 overflows int Fix it by amending "store notify" patch by chunking data into 2GB instead of full-32bit-range 4GB.
-
Han-Wen Nienhuys authored
These two tests failed in recent travis runs.
-
Han-Wen Nienhuys authored
This helps debugging timing issues
-
Han-Wen Nienhuys authored
-
- 08 Oct, 2018 3 commits
-
-
Jakob Unterwurzacher authored
gocryptfs user Felix Lechner reported a nil pointer dereference in GetAttr: https://github.com/rfjakob/gocryptfs/issues/260 The crash is in line n.setClientInode(fi.Ino) because fi is nil. This can happen when file.GetAttr() returns an error code other than ENOSYS and EBADF. For gocryptfs, this can only happen when an open file descriptor breaks. In this case it was triggered by a failing NFS volume. Fix the crash by erroring out for error codes that are not handled by the retry-by-path logic.
-
Kirill Smelkov authored
Current state is: fuse's godoc package-level description is completely empty because there is a blank link between package-level description and package clause. Yes, a user still has to go through api.go and read it in full, but having at least something for the description is a better start, and removing one line is easy. We already do it this way e.g. for nodefs and in other places.
-
Kirill Smelkov authored
I'm writing a networked filesystem which reads data in 2MB blocks from remote database. However read requests from the kernel come in much smaller sizes - for example 4K-128K. Since it would be very slow to refetch the 2MB block for e.g. every consecutive 4K reads, a cache for fetched data is needed. A custom cache would do, however since the kernel already implements pagecache to cache file data, it is logical to use it directly. FUSE protocol provides primitives for pagecache control. We already have support for e.g. invalidating a data region for inode (InodeNotify), but there is more there. In particular it is possible to store and retrieve data into/from the kernel cache with "store notify" and "retrieve notify" messages: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fuse.h?id=v4.19-rc6-177-gcec4de302c5f#n68 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/fuse.h?id=v4.19-rc6-177-gcec4de302c5f#n756 https://git.kernel.org/linus/a1d75f2582 https://git.kernel.org/linus/2d45ba381a This patch adds support for "store notify". Adding support for "retrieve notify" might be added later since a) it is not needed right now, and b) supporting it is a bit more work since the kernel sends a separate reply with data and infrastructure has to be added to glue it back to original "retrieve notify" server-originated request. For user-visible API I decided not to duplicate FUSE-protocol naming 1-1 and to be more explicit in names - emphasizing it is about cache control - it is e.g. InodeNotifyStoreCache instead of InodeNotifyStore, and for retrieving it (hopefully) should be just InodeRetrieveCache instead of InodeNotifyRetrieveCache and a separate callback. Thanks beforehand, Kirill
-
- 01 Oct, 2018 1 commit
-
-
Kirill Smelkov authored
Current code was requiring that both in.Major >= maj _and_ in.Minor >= min this way e.g. kernel's 7.27 would satisfy requirement of e.g. 7.25, but 8.1 won't do. Fix it.
-