I was too used to my old Python habit while originally writing this.
Using the occasion add package-level description which outlines the
package to a user.

Until https://github.com/golang/go/issues/7873 is fixed/implemented,
let's follow the currently advised scheme to indent lists and this way
put them as pre-formatted text.

We already do so mostly everywhere, but 2 of the places were not
following it and thus all items there were rendered as one line.

For example in ZODB FileStorage format reader routines are written
working with io.ReaderAt for the following reasons:

- for loads random-access is required,
- there can be several concurrent loads in flight simultaneously.

At the same time various database iterations (APIs additional to load)
use sequential access pattern and can be served by the same record
reading routines. However with them we cannot use e.g. bufio.Reader
because it works with plain io.Reader, not io.ReaderAt.

Here comes SeqReaderAt: it adds a buffer, by default 2·4K, on top of
original io.Reader, automatically detects direction of sequential
access which can be forward, backward, or interleaved forward-backward
patterns, and buffers data accordingly to avoid many syscalls e.g. in
os.File case.

Currently by direct & buffered readers that also can report current
offset in input stream.

This functionality is handy to have when for example one needs to report
an error after finding decoding problem and wants to include particular
stream position in the report.

Freelist of buffers is frequently needed to avoid GC/zeroing overhead in
e.g. networked servers. Making buffer reference-counted is then required
if buffer is shared between several users so that it is clear when it
can go back to its pool. In ZODB this situation was arising on returning
the same buffer from cached loads if several loads for same data are
issued in close to parallel.

[1] https://lab.nexedi.com/kirr/neo/tree/75a71514/go/zodb

Preliminary because Go1.10 has not been released yet, so
zruntime_g_go1.10.go will need to be potentially regenerated after
the release.

./gotrace.go:707: Errorf format %s arg pkgi.Pkg.Path is a func value, not called

xruntime/xruntime_test.go:47: Fatal call has possible formatting directive %v

./alloc_test.go:62: Fatalf format %d has arg aliases(s, sprev) of wrong type bool

This tool analyzes `go tool trace -d` output and take notices on
ProcStart and GoStart events to track G->P->M relation to check how
often a G changes M.

Unfortunately it seems to be a frequent event and changing M probably means
changing CPU and thus loosing CPU caches.

This patch adds pipenet - virtual network of net.Pipes.

Addresses on pipenet are host:port pairs. A host is xnet.Networker and so
can be worked with similarly to regular TCP network with Dial/Listen/Accept/...

Example:

    net := pipenet.New("")
    h1 := net.Host("abc")
    h2 := net.Host("def")

    l, err := h1.Listen(":10")       // starts listening on address "abc:10"
    go func() {
            csrv, err := l.Accept()  // csrv will have LocalAddr "abc:10"
    }()
    ccli, err := h2.Dial("abc:10")   // ccli will have RemoteAddr "def:10"

Pipenet might be handy for testing interaction of networked applications in 1
process without going to OS networking stack.

This patch adds xnet.NetTrace which wraps a networker calling
notification functions on Connect/Listen/Tx events.

The code is draft and I'm not sure adding this functionality is good
idea, but I still add it for completeness and because there is one user
for it.

Please do not expect the interface of xnet tracing to be stable.

Std net.Conn already can represent both plain TCP connections and
connections wrapped with TLS. However the connections itself need to be
created differently. This might become inconvenient when establishing
connections should be inside server logic where it is desirable to have
one codepath which works uniformly via interfaces.

This patch introduces Networker - new interface which represents
access-point to a streaming network. One can Dial or Listen on it and
get underlying network name.

Two functions are also provided to create networkers for plain TCP
and to wrap a networker with TLS layer.

This way one can initially decide and setup a networker, pass it to
server logic, and server inside uses just Networker interface
transparently either listening/connecting via regular sockets, or via
sockets wrapped with TLS layer.

XRun (added in db941f12) and Runx (added in 486ede30) run a function and
convert error <-> exception back and forth. However sometimes it is
handy to only convert a function but not run it - e.g. for passing into

	x/sync/errgroup.Group.Go

To do so this patch adds XFunc and Funcx - functional counterparts to
XRun and Runx.

No new tests are needed because now XRun and Runx are just tiny wrappers
around new functions and we already have tests for XRun and Runx.

Race-detector does not know Probe.Detach works under world stopped and
that this way it cannot break consistency of probes list attached to a
trace event - on event signalling either a probe will be run or not run
at all.

And we do not mind that e.g. while Detach was in progress a probe was
read from traceevent list and decided to be run and the probe
function was actually called just after Detach finished.

For this reason tell race-detector to not take into account all memory
read/write that are performed while the world is stopped.

If we do not it complains e.g. this way:

    ==================
    WARNING: DATA RACE
    Read at 0x00c42000d760 by goroutine 7:
      lab.nexedi.com/kirr/neo/go/zodb/storage._traceCacheGCFinish_run()
          /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/xcommon/tracing/tracing.go:265 +0x81
      lab.nexedi.com/kirr/neo/go/zodb/storage.traceCacheGCFinish()
          /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/ztrace.go:22 +0x63
      lab.nexedi.com/kirr/neo/go/zodb/storage.(*Cache).gc()
          /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/cache.go:497 +0x62c
      lab.nexedi.com/kirr/neo/go/zodb/storage.(*Cache).gcmain()
          /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/cache.go:478 +0x4c

    Previous write at 0x00c42000d760 by goroutine 6:
      lab.nexedi.com/kirr/neo/go/xcommon/tracing.(*Probe).Detach()
          /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/xcommon/tracing/tracing.go:319 +0x103
      lab.nexedi.com/kirr/neo/go/xcommon/tracing.(*ProbeGroup).Done()
          /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/xcommon/tracing/tracing.go:344 +0xa5
      lab.nexedi.com/kirr/neo/go/zodb/storage.TestCache()
          /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/cache_test.go:576 +0x7f94
      testing.tRunner()
          /home/kirr/src/tools/go/go/src/testing/testing.go:746 +0x16c

    Goroutine 7 (running) created at:
      lab.nexedi.com/kirr/neo/go/zodb/storage.NewCache()
          /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/cache.go:129 +0x227
      lab.nexedi.com/kirr/neo/go/zodb/storage.TestCache()
          /home/kirr/src/neo/src/lab.nexedi.com/kirr/neo/go/zodb/storage/cache_test.go:165 +0x7b1
      testing.tRunner()
          /home/kirr/src/tools/go/go/src/testing/testing.go:746 +0x16c

    Goroutine 6 (finished) created at:
      testing.(*T).Run()
          /home/kirr/src/tools/go/go/src/testing/testing.go:789 +0x568
      testing.runTests.func1()
          /home/kirr/src/tools/go/go/src/testing/testing.go:1004 +0xa7
      testing.tRunner()
          /home/kirr/src/tools/go/go/src/testing/testing.go:746 +0x16c
      testing.runTests()
          /home/kirr/src/tools/go/go/src/testing/testing.go:1002 +0x521
      testing.(*M).Run()
          /home/kirr/src/tools/go/go/src/testing/testing.go:921 +0x206
      main.main()
          lab.nexedi.com/kirr/neo/go/zodb/storage/_test/_testmain.go:44 +0x1d3
    ==================

As it was said in the previous patch here goes gotrace utility to
process `//trace:event ...` and other tracing related directives.

Related excerpt from the documentation:

---- 8< ----
Gotrace

The way //trace:event and //trace:import works is via additional code being
generated for them. Whenever a package uses any //trace: directive,
it has to organize to run `gotrace gen` on its sources for them to work,
usually with the help of //go:generate. For example:

	package hello

	//go:generate gotrace gen .

	//trace:event ...

Besides `gotrace gen` gotrace has other subcommands also related to tracing,
for example `gotrace list` lists trace events a package provides.
---- 8< ----

Gotrace works by parsing and typechecking go sources via go/parse &
go/types and then for special comments generating corresponding
additional code that is supported by tracing runtime.

Package tracing will provide usage and runtime support for Linux-style
traceevents/tracepoints for Go.

This patch comes with the runtime support to attach/detach probes to/from
trace-events and stop/restart the world so that attaching/detaching can
be done safely while nothing else is running. Having attach/detach under
STW allows regular probe list iteration to be done without locks.

The next patch will add gotrace utility which automatically turns
//trace:event in-source comments into proper trace-event definitions.

Below is excerpt from tracing usage. Please refer to tracing.go for full
text.

---- 8< ----
Package tracing provides usage and runtime support for Go tracing facilities.

Trace events

A Go package can define several events of interest to trace via special
comments. With such definition a tracing event becomes associated with trace
function that is used to signal when the event happens. For example:

	package hello

	//trace:event traceHelloPre(who string)
	//trace:event traceHello(who string)

	func SayHello(who string) {
		traceHelloPre(who)
		fmt.Println("Hello, %s", who)
		traceHello(who)
	}

By default trace function does nothing and has very small overhead.

Probes

However it is possible to attach probing functions to events. A probe, once
attached, is called whenever event is signalled in the context which triggered
the event and pauses original code execution until the probe is finished. It is
possible to attach several probing functions to the same event and dynamically
detach/(re-)attach them at runtime. Attaching/detaching probes must be done
under tracing.Lock. For example:

	type saidHelloT struct {
		who  string
		when time.Time
	}
	saidHello := make(chan saidHelloT)

	tracing.Lock()
	p := traceHello_Attach(nil, func(who string) {
		saidHello <- saidHelloT{who, time.Now()}
	})
	tracing.Unlock()

	go func() {
		for hello := range saidHello {
			fmt.Printf("Said hello to %v @ %v\n", hello.who, hello.when)
		}
	}()

	SayHello("JP")
	SayHello("Kirr")
	SayHello("Varya")

	tracing.Lock()
	p.Detach()
	tracing.Unlock()

	close(saidHello)

For convenience it is possible to keep group of attached probes and detach them
all at once using ProbeGroup:

	pg := &tracing.ProbeGroup{}

	tracing.Lock()
	traceHelloPre_Attach(pg, func(who string) { ... })
	traceHello_Attach(pg, func(who string) { ... })
	tracing.Unlock()

	// some activity

	// when probes needs to be detached (no explicit tracing.Lock needed):
	pg.Done()

Probes is general mechanism which allows various kinds of trace events usage.
Three ways particularly are well-understood and handy:

	- recording events stream
	- profiling
	- synchronous tracing

...

Initial draft. The implementation is modelled after `git` and `go`.

4d9a613c (Relicense to GPLv3+ with wide exception for all Free Software
/ Open Source projects + Business options.) added more lines to
my/my_test.go than removed (@@ -5,16 +5,18 @@) so the line number of

        myline := Line()

changed by 2 (= 18 - 16).

Fix the test accordingly.

Relicense to GPLv3+ with wide exception for all Free Software / Open Source projects + Business options.

Nexedi stack is licensed under Free Software licenses with various exceptions
that cover three business cases:

- Free Software
- Proprietary Software
- Rebranding

As long as one intends to develop Free Software based on Nexedi stack, no
license cost is involved. Developing proprietary software based on Nexedi stack
may require a proprietary exception license. Rebranding Nexedi stack is
prohibited unless rebranding license is acquired.

Through this licensing approach, Nexedi expects to encourage Free Software
development without restrictions and at the same time create a framework for
proprietary software to contribute to the long term sustainability of the
Nexedi stack.

Please see https://www.nexedi.com/licensing for details, rationale and options.

CeilLog2(x)	= min i: 2^i >= x
FloorLog2(x)	= max i: 2^i <= x

This way original error can still be retrieved via errors.Cause()

This continues c0bbd06e (xfmt: Qpy & friends to quote string the way
Python would do).

This are handy utilities to reduce several errors into only 1 either
picking the first error or merging, if there are several, into Errorv.

It is unfortunate but an issue with Errorv was realized that even though
it satisfies the error interface, it cannot be generally worked with as
error because it (being []error) is uncomparable. Thus e.g. the following
code, if err dynamic type is Errorv, will panic:

	if err == io.EOF

It is pity Go slices are uncomparable.

This are handy utilities to automatically prepend context on error
return. For example in

	func myfunc(...) (..., err error) {
		defer xerr.Context(&err, "error context")
		...
		if ... {
			return ..., errors.New("an error")
		}
		return ..., nil
	}

while preserving nil error return on successful execution, myfunc will
return error with string "error context: an error" on failure case.

This is somtimes needed for checking programs output bit-to-bit where on
python side repr(x), `x` or %r is used for output.

Std fmt works ok unless you need to do text formatting in hot codepaths.
There fmt becomes inappropriate as it is slow and does allocations on
every formatting.

strconv also does not have append routines for every needed case, e.g.
there is no strconv.AppendRune, no strconv.AppendHex etc.

So xfmt

1. provides append routines for builtin types lacking in strconv

2. introduces xfmt.Stringer interface which types can implement to hook
   into general formatting via xfmt.Append()

3. provides xfmt.Buffer which is []byte with syntatic sugar for
   formatting in a way similar to printf: For example if in fmt speak
   you have

	s := fmt.Sprintf("hello %q %d %x", "world", 1, []byte("data"))

   xfmt analog would be

	buf := xfmt.Buffer{}
	buf .S("hello ") .Q("world") .C(' ') .D(1) .C(' ') .Xb([]byte("data"))
	s := buf.Bytes()

   and xfmt.Buffer can be reused several times via Buffer.Reset() .

The above xfmt.Buffer usage is more uglier than fmt.Printf but much less uglier
than direct strconv.Append* and friends calls, and works faster and without
allocations compared to fmt.Printf:

	BenchmarkXFmt/%c(0x41)-4                20000000                65.4 ns/op             1 B/op          1 allocs/op
	BenchmarkXFmt/.Cb(0x41)-4               200000000                5.96 ns/op            0 B/op          0 allocs/op
	BenchmarkXFmt/%c(-1)-4                  20000000                70.1 ns/op             3 B/op          1 allocs/op
	BenchmarkXFmt/.C(-1)-4                  100000000               12.9 ns/op             0 B/op          0 allocs/op
	BenchmarkXFmt/%c(66)-4                  20000000                65.8 ns/op             1 B/op          1 allocs/op
	BenchmarkXFmt/.C(66)-4                  100000000               12.7 ns/op             0 B/op          0 allocs/op
	BenchmarkXFmt/%c(1080)-4                20000000                67.2 ns/op             2 B/op          1 allocs/op
	BenchmarkXFmt/.C(1080)-4                100000000               12.8 ns/op             0 B/op          0 allocs/op
	BenchmarkXFmt/%c(8364)-4                20000000                69.4 ns/op             3 B/op          1 allocs/op
	BenchmarkXFmt/.C(8364)-4                100000000               13.8 ns/op             0 B/op          0 allocs/op
	BenchmarkXFmt/%c(65537)-4               20000000                70.5 ns/op             4 B/op          1 allocs/op
	BenchmarkXFmt/.C(65537)-4               100000000               14.3 ns/op             0 B/op          0 allocs/op
	BenchmarkXFmt/%s("hello")-4             20000000                72.3 ns/op             5 B/op          1 allocs/op
	BenchmarkXFmt/.S("hello")-4             200000000                9.40 ns/op            0 B/op          0 allocs/op
	...

- small addons over std bytes package: xbytes.ContainsByte
- (re)allocation routines for byte slices

So far only one of them: CeilPow2 to return min(y) >= x: y = 2^i - i.e.
next power of two >= x. This is handy to have in reallocation routines
to allocate buffers from 2^i classes.

Two implementations:

- fast for go19
- slower fallback for go18

As it was planned switch to using runtime.Frame instead of our
local imitation.

Add test to make sure we are not breaking anything.
Adjust users in exc.

To determine current function's file name, line number and runtime.Frame

Because in general case runtime.CallersFrames is more accurate than
runtime.FuncForPC - e.g. the latter does not correctly work with inlined
functions.

This causes the following changes on client side:

	myname.Func	-> my.FuncName
	myname.Pkg	-> my.PkgName

Reason for the change is that we are going to introduce my.File and
my.Line which would not fit into myname package naming.

We want to make sure that the code can be used by Free Software and Open
Source projects without a problem.

So change the license to be GPLv3+ with wide exception for all
open-source licenses which practically cover them all.

exc += XRun() - utility to run a function which returns regular error, and raise exception if error is not nil

This might be frequently needed in tests, possibly with some kind of
"apply" syntatic sugar.

Complements 486ede30 (exc += Runx() - utility to run a function and
translate caught exception to regular error)

Previously Error() was giving "0 errors", but "" seems to be a better
choice.

Anyway proper usage of Errorv is to call .Err() when handing result to
some where so this way empty-vector case becomes nil.