1. 17 Dec, 2020 1 commit
    • Kirill Smelkov's avatar
      xnet: Adjust Networker.Listen to return listener that can handle cancellation in Accept · 3354b401
      Kirill Smelkov authored
      We already handle cancellation in Dial, but Accepting was out of luck
      until now. This makes it more difficult for clients to implement and
      wrap acceptors where they need to handle cancellations. This also makes
      it possible for a test or program to get stuck in Accept loop if it is
      not careful enough to manually handle ctx cancel around Accept calls.
      
      -> Fix it in one place - here, in xnet - so that users are offloaded
      from all this and can just call Accept(ctx) and rely on underlying
      implementation to handle ctx cancel.
      
      This patch:
      
      - introduces xnet.Listener interface, which is like net.Listener, but
        Accept goes with ctx argument.
      - changes Networker.Listen signature to return xnet.Listener instead of
        net.Listener. While we are here - changing it - also add ctx argument
        to Listen call itself.
      - Adds listenerCtx - which, given net.Listener, provides xnet.Listener
        by wrapping some logic around original.
      - Adapts NetPlain, NetTLS and NetTrace to provide updated interface.
      
      We'll fix up everything in other packages to match/use updated interface
      in the next patch.
      3354b401
  2. 01 Dec, 2020 1 commit
  3. 18 Oct, 2020 1 commit
  4. 16 Sep, 2020 1 commit
    • Kirill Smelkov's avatar
      exc: Contextf · 31661766
      Kirill Smelkov authored
      A shorthand for when exception context is just formatted string.
      Simplifies
      
      	func doSomething(path string) {
      		defer exc.Context(func() interface{} {
      			return fmt.Sprintf("doing something %s", path)
      		})()
      		...
      
      to
      
      	func doSomething(path string) {
      		defer exc.Contextf("doing something %s", path)
      		...
      31661766
  5. 15 Sep, 2020 1 commit
  6. 27 May, 2020 4 commits
  7. 03 Apr, 2020 1 commit
  8. 16 Jan, 2020 1 commit
  9. 13 Jan, 2020 1 commit
  10. 26 Dec, 2019 6 commits
  11. 20 Dec, 2019 1 commit
    • Kirill Smelkov's avatar
      tracing/runtime: Add support for Go1.14 (preliminary) · 5f6ae15b
      Kirill Smelkov authored
      Generate g for ~ Go 1.14beta1 (go1.14beta1-10-g5c6f42773c)
      Compared to Go1.13 there are several changed to g, _defer and timer
      related to:
      
      - non-cooperative preemption (https://golang.org/issue/10958, https://golang.org/issue/24543);
      - inlined defers (https://golang.org/issue/14939, https://golang.org/issue/34481);
      - timers rework to be integrated with network poller (https://golang.org/issue/6239, https://golang.org/issue/27707):
      
      Regenerated files stay without changes for Go1.13 and previous releases.
      
      ---- 8< ----
      diff --git a/zruntime_g_go1.13.go b/zruntime_g_go1.14.go
      index 76851fd..19cebae 100644
      --- a/zruntime_g_go1.13.go
      +++ b/zruntime_g_go1.14.go
      @@ -1,6 +1,6 @@
       // Code generated by g_typedef; DO NOT EDIT.
      
      -// +build go1.13,!go1.14
      +// +build go1.14,!go1.15
      
       package xruntime
      
      @@ -32,12 +32,25 @@ type g struct {
       	schedlink    guintptr
       	waitsince    int64      // approx time when the g become blocked
       	waitreason   waitReason // if status==Gwaiting
      +
       	preempt       bool // preemption signal, duplicates stackguard0 = stackpreempt
      +	preemptStop   bool // transition to _Gpreempted on preemption; otherwise, just deschedule
      +	preemptShrink bool // shrink stack at synchronous safe point
      +
      +	// asyncSafePoint is set if g is stopped at an asynchronous
      +	// safe point. This means there are frames on the stack
      +	// without precise pointer information.
      +	asyncSafePoint bool
      +
       	paniconfault bool // panic (instead of crash) on unexpected fault address
      -	preemptscan    bool       // preempted g does scan for gc
       	gcscandone   bool // g has scanned stack; protected by _Gscan bit in status
      -	gcscanvalid    bool       // false at start of gc cycle, true if G has not run since last scan; TODO: remove?
       	throwsplit   bool // must not split stack
      +	// activeStackChans indicates that there are unlocked channels
      +	// pointing into this goroutine's stack. If true, stack
      +	// copying needs to acquire channel locks to protect these
      +	// areas of the stack.
      +	activeStackChans bool
      +
       	raceignore     int8     // ignore race detection events
       	sysblocktraced bool     // StartTrace has emitted EvGoInSyscall about this goroutine
       	sysexitticks   int64    // cputicks when syscall has returned (for tracing)
      @@ -76,18 +89,37 @@ type _panic struct {
       	argp      unsafe.Pointer // pointer to arguments of deferred call run during panic; cannot move - known to liblink
       	arg       interface{}    // argument to panic
       	link      *_panic        // link to earlier panic
      +	pc        uintptr        // where to return to in runtime if this panic is bypassed
      +	sp        unsafe.Pointer // where to return to in runtime if this panic is bypassed
       	recovered bool           // whether this panic is over
       	aborted   bool           // the panic was aborted
      +	goexit    bool
       }
       type _defer struct {
       	siz     int32 // includes both arguments and results
       	started bool
       	heap    bool
      +	// openDefer indicates that this _defer is for a frame with open-coded
      +	// defers. We have only one defer record for the entire frame (which may
      +	// currently have 0, 1, or more defers active).
      +	openDefer bool
       	sp        uintptr  // sp at time of defer
      -	pc      uintptr
      -	fn      *funcval
      +	pc        uintptr  // pc at time of defer
      +	fn        *funcval // can be nil for open-coded defers
       	_panic    *_panic  // panic that is running defer
       	link      *_defer
      +
      +	// If openDefer is true, the fields below record values about the stack
      +	// frame and associated function that has the open-coded defer(s). sp
      +	// above will be the sp for the frame, and pc will be address of the
      +	// deferreturn call in the function.
      +	fd   unsafe.Pointer // funcdata for the function associated with the frame
      +	varp uintptr        // value of varp for the stack frame
      +	// framepc is the current pc associated with the stack frame. Together,
      +	// with sp above (which is the sp associated with the stack frame),
      +	// framepc/sp can be used as pc/sp pair to continue a stack trace via
      +	// gentraceback().
      +	framepc uintptr
       }
       type gobuf struct {
       	// The offsets of sp, pc, and g are known to (hard-coded in) libmach.
      @@ -114,8 +146,10 @@ type funcval struct {
       	fn uintptr
       }
       type timer struct {
      -	tb *timersBucket // the bucket the timer lives in
      -	i  int           // heap index
      +	// If this timer is on a heap, which P's heap it is on.
      +	// puintptr rather than *p to match uintptr in the versions
      +	// of this struct defined in other packages.
      +	pp puintptr
      
       	// Timer wakes up at when, and then at when+period, ... (period > 0 only)
       	// each time calling f(arg, now) in the timer goroutine, so f must be
      @@ -125,6 +159,12 @@ type timer struct {
       	f      func(interface{}, uintptr)
       	arg    interface{}
       	seq    uintptr
      +
      +	// What to set the when field to in timerModifiedXX status.
      +	nextwhen int64
      +
      +	// The status field holds one of the values below.
      +	status uint32
       }
       type guintptr uintptr
       type puintptr uintptr
      5f6ae15b
  12. 28 Jul, 2019 1 commit
  13. 19 May, 2019 2 commits
  14. 16 Apr, 2019 1 commit
    • Kirill Smelkov's avatar
      tracing: call/return overhead related to using trace functions is gone · 79711643
      Kirill Smelkov authored
      As of Go1.12 the check for whether particular trace event is enabled or
      not is inlined into caller. For example:
      
      	---- 8< ---- (connection.go)
      	//trace:event traceMsgSendPre(l *NodeLink, connId uint32, msg proto.Msg)
      
      	func (link *NodeLink) sendMsg(connId uint32, msg proto.Msg) error {
      		traceMsgSendPre(link, connId, msg)
      
      		buf := msgPack(connId, msg)
      		return link.sendPkt(buf)
      	}
      
      	---- 8< ---- (ztrace.go)
      	// traceevent: traceMsgSendPre(l *NodeLink, connId uint32, msg proto.Msg)
      
      	type _t_traceMsgSendPre struct {
      	        tracing.Probe
      	        probefunc     func(l *NodeLink, connId uint32, msg proto.Msg)
      	}
      
      	var _traceMsgSendPre *_t_traceMsgSendPre
      
      	func traceMsgSendPre(l *NodeLink, connId uint32, msg proto.Msg) {
      	        if _traceMsgSendPre != nil {
      	                _traceMsgSendPre_run(l, connId, msg)
      	        }
      	}
      
      	func _traceMsgSendPre_run(l *NodeLink, connId uint32, msg proto.Msg) {
      	        for p := _traceMsgSendPre; p != nil; p = (*_t_traceMsgSendPre)(unsafe.Pointer(p.Next())) {
      	                p.probefunc(l, connId, msg)
      	        }
      	}
      
      	---- 8< ---- (connection.o)
      	TEXT lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·(*NodeLink).sendMsg(SB), ABIInternal, $40-48 // connection.go:1368
      	        // MOVQ    (TLS), CX (stack growth prologue)
      	        // CMPQ    SP, 16(CX)
      	        // JLS     177
      	        // SUBQ    $40, SP
      	        // MOVQ    BP, 32(SP) (BP save)
      	        // LEAQ    32(SP), BP (BP init)
      	        // FUNCDATA $0, gclocals·dbae954985b577993f0eafab1347dd21(SB) (args)
      	        FUNCDATA   $1, gclocals·f6bd6b3389b872033d462029172c8612(SB) (locals)
      	        FUNCDATA   $3, gclocals·47f0f3578d0f16ef296c57af2564832c(SB)
      	        PCDATA     $2, $0             // connection.go:1369
      	        PCDATA     $0, $0
      	        XCHGL      AX, AX
      	        CMPQ       lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·_traceMsgSendPre(SB), $0	<-- inlined traceMsgSendPre
      	        JNE        pc130              // ztrace.go:50					<--
      	pc44:
      	        MOVL       connId+56(SP), AX  // connection.go:1371				<-- normal path
      	        MOVL       AX, (SP)
      	        MOVQ       msg+64(SP), AX
      	        MOVQ       AX, 8(SP)
      	        PCDATA     $2, $1
      	        PCDATA     $0, $1
      	        MOVQ       msg+72(SP), AX
      	        PCDATA     $2, $0
      	        MOVQ       AX, 16(SP)
      	        CALL       lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·msgPack(SB)
      	        PCDATA     $2, $1
      	        MOVQ       24(SP), AX
      	        PCDATA     $2, $2             // connection.go:1372
      	        PCDATA     $0, $2
      	        MOVQ       link+48(SP), CX
      	        PCDATA     $2, $1
      	        MOVQ       CX, (SP)
      	        PCDATA     $2, $0
      	        MOVQ       AX, 8(SP)
      	        CALL       lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·(*NodeLink).sendPkt(SB)
      	        PCDATA     $2, $1
      	        MOVQ       24(SP), AX
      	        MOVQ       16(SP), CX
      	        PCDATA     $0, $3
      	        MOVQ       CX, _r2+80(SP)
      	        PCDATA     $2, $0
      	        MOVQ       AX, _r2+88(SP)
      	        // MOVQ    32(SP), BP (BP restore)
      	        // ADDQ    $40, SP (SP restore)
      	        RET
      	pc130:											<-- slow path (tracepoint attached)
      	        PCDATA     $2, $1             // ztrace.go:51
      	        PCDATA     $0, $0
      	        MOVQ       link+48(SP), AX
      	        PCDATA     $2, $0
      	        MOVQ       AX, (SP)
      	        MOVL       connId+56(SP), CX
      	        MOVL       CX, 8(SP)
      	        MOVQ       msg+64(SP), DX
      	        MOVQ       DX, 16(SP)
      	        PCDATA     $2, $3
      	        MOVQ       msg+72(SP), BX
      	        PCDATA     $2, $0
      	        MOVQ       BX, 24(SP)
      	        CALL       lab·nexedi·com∕kirr∕neo∕go∕neo∕neonet·_traceMsgSendPre_run(SB)
      	        JMP        pc44
      	        // NOP (stack growth)
      	        // PCDATA  $0, $-1            // connection.go:1368
      	        // PCDATA  $2, $-1
      	        // CALL    runtime.morestack_noctxt(SB)
      	        // JMP     0
      79711643
  15. 12 Apr, 2019 1 commit
  16. 11 Apr, 2019 2 commits
  17. 24 Jan, 2019 1 commit
  18. 17 Jan, 2019 1 commit
  19. 16 Jan, 2019 1 commit
  20. 19 Nov, 2018 1 commit
  21. 06 Jul, 2018 1 commit
  22. 03 Jul, 2018 1 commit
    • Kirill Smelkov's avatar
      tracing/pyruntraced: New tool to run Python code with tracepoints activated (draft) · 7a476082
      Kirill Smelkov authored
      We will hopefully need it in the future to test external python code
      similarly to when testing in-process go code with the help of
      tracepoints. See module description for details (tracetest is not yet
      done).
      
      There is no yet Go client API for interacting with such-traced python
      program. For the record: manually inspecting traced python process could
      be done via e.g.
      
      	socat EXEC:"./pyruntraced 3 neo.trace.py -- ../../../t/backup-play/N1-writer",fdin=3,fdout=3 TCP:localhost:8888
      
      and telnetting to localhost:8888 from another xterm.
      7a476082
  23. 20 Jun, 2018 1 commit
  24. 19 Jun, 2018 7 commits
    • Kirill Smelkov's avatar
      xnet/lonet: Draft _Python_ counterpart · 5f16012d
      Kirill Smelkov authored
      This patch brings lonet implementation in Python with the idea that Go
      and Python programs could interoperate via lonet network and thus mixed
      Go/Python application cluster could be tested.
      
      Implementation quality is lower compared to Go version, but still it
      should be more or less ok.
      5f16012d
    • Kirill Smelkov's avatar
      xerr: Draft _Python_ counterpart · fa0f9048
      Kirill Smelkov authored
      Provide infrastructure, similar to what we already have in xerr, to Python
      programs: to wrap errors with error context and to extract the cause
      from a wrapped error.
      
      We will need it in the next patch.
      fa0f9048
    • Kirill Smelkov's avatar
      golang/gcompat: New _Python_ package that provides Go-compatibility layer for Python · 7c4f365c
      Kirill Smelkov authored
      Let's start with substitute for %q that is lacking in python.
      The implementation is copied from zodbtools from here:
      
      https://lab.nexedi.com/nexedi/zodbtools/blob/2801fae9/zodbtools/util.py#L39
      
      It is not good for golang to depend on zodbtools and the function is
      minor. In the future maybe zodbtools in turn will change to import qq
      from golang.
      7c4f365c
    • Kirill Smelkov's avatar
      golang: New _Python_ package to provide Go-like features to Python language · 9e1aa6ab
      Kirill Smelkov authored
      - `go` spawns lightweight thread.
      - `chan` and `select` provide channels with Go semantic.
      - `method` allows to define methods separate from class.
      - `gimport` allows to import python modules by full path in a Go workspace.
      
      The focus of first draft was on usage interface and on correctness, not speed.
      In particular select should be fully working.
      
      If there is a chance I will maybe try to followup with gevent-based
      implementation in the future.
      9e1aa6ab
    • Kirill Smelkov's avatar
      xnet/lonet: New package to provide TCP network simulated on top of localhost TCP loopback. · fd4f9a4e
      Kirill Smelkov authored
      Lonet is the virtnet network that, contrary to pipenet, could be used
      when there are several OS-level processes involved.
      
      It uses SQLite for its registry and native OS-level TCP over loopback
      for data exchange. There is small text-based connection handshake
      protocol prelude that have to be carried out when a connection is tried
      to be established to host via its subnetwork, but after it data exchange
      goes directly through OS TCP stack.
      
      Lonet documentation follows:
      
      """
      Package lonet provides TCP network simulated on top of localhost TCP loopback.
      
      For testing distributed systems it is sometimes handy to imitate network of
      several TCP hosts. It is also handy that ports allocated on Dial/Listen/Accept on
      that hosts be predictable - that would help tests to verify network events
      against expected sequence. When whole system could be imitated in 1 OS-level
      process, package lab.nexedi.com/kirr/go123/xnet/pipenet serves the task via
      providing TCP-like synchronous in-memory network of net.Pipes. However
      pipenet cannot be used for cases where tested system consists of 2 or more
      OS-level processes. This is where lonet comes into play:
      
      Similarly to pipenet addresses on lonet are host:port pairs and several
      hosts could be created with different names. A host is xnet.Networker and
      so can be worked with similarly to regular TCP network access-point with
      Dial/Listen/Accept. Host's ports allocation is predictable: ports of a host
      are contiguous integer sequence starting from 1 that are all initially free,
      and whenever autobind is requested the first free port of the host will be
      used.
      
      Internally lonet network maintains registry of hosts so that lonet
      addresses could be resolved to OS-level addresses, for example α:1 and β:1
      to 127.0.0.1:4567 and 127.0.0.1:8765, and once lonet connection is
      established it becomes served by OS-level TCP connection over loopback.
      
      Example:
      
      net, err := lonet.Join(ctx, "mynet")
      hα, err := net.NewHost(ctx, "α")
      hβ, err := net.NewHost(ctx, "β")
      
      // starts listening on address "α:10"
      l, err := hα.Listen(":10")
      go func() {
       csrv, err := l.Accept()   // csrv will have LocalAddr "α:1"
      }()
      ccli, err := hβ.Dial(ctx, "α:10") // ccli will be connection between "β:1" - "α:1"
      
      Once again lonet is similar to pipenet, but since it works via OS TCP stack
      it could be handy for testing networked application when there are several
      OS-level processes involved.
      """
      
      """
      Lonet organization
      
      For every lonet network there is a registry with information about hosts
      available on the network, and for each host its OS-level listening address.
      The registry is kept as SQLite database under
      
      /<tmp>/lonet/<network>/registry.db
      
      Whenever host α needs to establish connection to address on host β, it
      queries the registry for β and further talks to β on that address.
      Correspondingly when a host joins the network, it announces itself to the
      registry so that other hosts could see it.
      
      Handshake protocol
      
      After α establishes OS-level connection to β via main β address, it sends
      request to further establish lonet connection on top of that:
      
      > lonet "<network>" dial "<α:portα>" "<β:portβ>"\n
      
      β checks whether portβ is listening, and if yes, accepts the connection on
      corresponding on-β listener with giving feedback to α that connection was
      accepted:
      
      < lonet "<network>" connected "<β:portβ'>"\n
      
      After that connection is considered to be lonet-established and all further
      exchange on it is directly controlled by corresponding lonet-level
      Read/Write on α and β.
      
      If, on the other hand, lonet-level connection cannot be established, β replies:
      
      < lonet "<networkβ>" E "<error>"\n
      
      where <error> could be:
      
      - connection refused if <β:portβ> is not listening
      - network mismatch if β thinks it works on different lonet network than α
      - protocol error if β thinks that α send incorrect dial request
      - ...
      """
      fd4f9a4e
    • Kirill Smelkov's avatar
      xnet/pipenet: Generalize it into xnet/virtnet · 40120cb0
      Kirill Smelkov authored
      As we are going to implement another virtual network it would be good to
      share common code between implementations. For this generalize pipenet
      implementation to also cover the case when one does not own full network
      and owns only some hosts of it.
      
      An example of such situation is when one process handles one group of
      virtual hosts and another process handles another group of virtual
      hosts. Below a group of virtual hosts handled as part of network is
      called subnetwork.
      
      If hosts are not created in one place, we need a way to communicate
      information about new hosts in between subnetworks. This leads to using
      some kind of "registry" (see Registry interface).
      
      Then for the common code to be reused by a virtual network
      implementation it has to provide its working in the form of Engine
      interface to that common code. In turn the common code exposes another
      - Notifier - interface for particular network implementation to notify
      common code of events that come from outside to the subnetwork.
      
      Pipenet is reworked to be just a client of the common virtnet
      infrastructure.
      
      Virtnet documentation follows:
      
      """
      Package virtnet provides infrastructure for TCP-like virtual networks.
      
      For testing distributed systems it is sometimes handy to imitate network of
      several TCP hosts. It is also handy that ports allocated on Dial/Listen/Accept
      on that hosts be predictable - that would help tests to verify network
      events against expected sequence.
      
      Package virtnet provides infrastructure for using and implementing such
      TCP-like virtual networks.
      
      Using virtnet networks
      
      Addresses on a virtnet network are host:port pairs represented by Addr.
      A network conceptually consists of several SubNetworks each being home for
      multiple Hosts. Host is xnet.Networker and so can be worked with similarly
      to regular TCP network access-point with Dial/Listen/Accept. Host's ports
      allocation is predictable: ports of a host are contiguous integer sequence
      starting from 1 that are all initially free, and whenever autobind is
      requested the first free port of the host will be used.
      Virtnet ensures that host names are unique throughout whole network.
      
      To work with a virtnet network, one uses corresponding package for
      particular virtnet network implementation. Such packages provide a way to
      join particular network and after joining give back SubNetwork to user.
      Starting from SubNetwork one can create Hosts and from those exchange data
      throughout whole network.
      
      Please see package lab.nexedi.com/kirr/go123/xnet/pipenet for particular
      well-known virtnet-based network.
      
      Implementing virtnet networks
      
      To implement a virtnet-based network one need to implement Engine and Registry.
      
      A virtnet network implementation should provide Engine and Registry
      instances to SubNetwork when creating it. The subnetwork will use provided
      engine and registry for its operations. A virtnet network implementation
      receives instance of Notifier together with SubNetwork when creating it. The
      network implementation should use provided Notifier to notify the subnetwork
      to handle incoming events.
      
      Please see Engine, Registry and Notifier documentation for details.
      """
      
      Another virtnet-based network that is not limited to be used only in 1
      OS process will follow next.
      40120cb0
    • Kirill Smelkov's avatar
      xnet/pipenet: Polish a bit · f04d243b
      Kirill Smelkov authored
      - clarify docstrings;
      - unexport NetPrefix.
      f04d243b