Commit 5890e25b authored by Russ Cox's avatar Russ Cox

cmd/go: add new test script facility

The original cmd/go tests were tiny shell scripts
written against a library of shell functions.
They were okay to write but difficult to run:
you couldn't select individual tests (with -run)
they didn't run on Windows, they were slow, and so on.

CL 10464 introduced go_test.go's testgo framework
and later CLs translated the test shell script over to
individual go tests. This let us run tests selectively,
run tests on Windows, run tests in parallel, isolate
different tests, and so on. It was a big advance.

The tests had always been awkward to write.
Here was the first test in test.bash:

	TEST 'file:line in error messages'
	# Test that error messages have file:line information at beginning of
	# the line. Also test issue 4917: that the error is on stderr.
	d=$(TMPDIR=/var/tmp mktemp -d -t testgoXXX)
	fn=$d/err.go
	echo "package main" > $fn
	echo 'import "bar"' >> $fn
	./testgo run $fn 2>$d/err.out || true
	if ! grep -q "^$fn:" $d/err.out; then
		echo "missing file:line in error message"
		cat $d/err.out
		ok=false
	fi
	rm -r $d

The final Go version of this test was:

	func TestFileLineInErrorMessages(t *testing.T) {
		tg := testgo(t)
		defer tg.cleanup()
		tg.parallel()
		tg.tempFile("err.go", `package main; import "bar"`)
		path := tg.path("err.go")
		tg.runFail("run", path)
		shortPath := path
		if rel, err := filepath.Rel(tg.pwd(), path); err == nil && len(rel) < len(path) {
			shortPath = rel
		}
		tg.grepStderr("^"+regexp.QuoteMeta(shortPath)+":", "missing file:line in error message")
	}

It's better but still quite difficult to skim.

This CL introduces a new facility meant as a successor to the testgo
approach that brings back the style of writing tests as little scripts,
but they are now scripts in a built-for-purpose shell-like language,
not bash itself. In this new form, the test above is a single file,
testdata/script/fileline.txt:

	# look for short, relative file:line in error message
	! go run ../../gopath/x/y/z/err.go
	stderr ^..[\\/]x[\\/]y[\\/]z[\\/]err.go:

	-- ../x/y/z/err.go --
	package main; import "bar"

The file is a txtar text archive (see CL 123359) in which the leading comment
is the test script and the files are the initial state of the temporary file
system where the script runs.

Each script runs as a subtest, so that they can still be selected individually.

The scripts are kept isolated from each other by default,
so all script subtests are treated as parallel tests, for the
testing package to run in parallel. Even for the 15 tests in
this CL, that cuts the time for TestScript from 5.5s to 2.5s.

The scripts do not have access to the cmd/go source directory,
nor to cmd/go/testdata, so they are prevented from creating temporary
files in those places or modifying existing ones. (Many existing tests
scribble in testdata, unfortunately, especially testdata/pkg when
they run builds with GOPATH=testdata.)

This CL introduces the script facility and converts 15 tests.
The txtar archive form will allow us to delete the large trees of trivial
files in testdata; a few are deleted in this CL.

See testdata/script/README for details and a larger conversion example.

As part of converting testdata/script/test_badtest.txt,
I discovered that 'go test' was incorrectly printing a FAIL line
to stderr (not stdout) in one corner case. This CL fixes that
to keep the test passing.

Future CLs will convert more tests.

Change-Id: I11aa9e18dd2d4c7dcd8e310dbdc6a1ea5f7e54c1
Reviewed-on: https://go-review.googlesource.com/123577
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: default avatarIan Lance Taylor <iant@golang.org>
parent b59b42ce
This diff is collapsed.
......@@ -183,27 +183,27 @@ func MatchFile(name string, tags map[string]bool) bool {
l = l[:n-1]
}
n := len(l)
if n >= 2 && knownOS[l[n-2]] && knownArch[l[n-1]] {
if n >= 2 && KnownOS[l[n-2]] && KnownArch[l[n-1]] {
return tags[l[n-2]] && tags[l[n-1]]
}
if n >= 1 && knownOS[l[n-1]] {
if n >= 1 && KnownOS[l[n-1]] {
return tags[l[n-1]]
}
if n >= 1 && knownArch[l[n-1]] {
if n >= 1 && KnownArch[l[n-1]] {
return tags[l[n-1]]
}
return true
}
var knownOS = make(map[string]bool)
var knownArch = make(map[string]bool)
var KnownOS = make(map[string]bool)
var KnownArch = make(map[string]bool)
func init() {
for _, v := range strings.Fields(goosList) {
knownOS[v] = true
KnownOS[v] = true
}
for _, v := range strings.Fields(goarchList) {
knownArch[v] = true
KnownArch[v] = true
}
}
......
......@@ -716,13 +716,12 @@ func runTest(cmd *base.Command, args []string) {
if err != nil {
str := err.Error()
str = strings.TrimPrefix(str, "\n")
failed := fmt.Sprintf("FAIL\t%s [setup failed]\n", p.ImportPath)
if p.ImportPath != "" {
base.Errorf("# %s\n%s\n%s", p.ImportPath, str, failed)
base.Errorf("# %s\n%s", p.ImportPath, str)
} else {
base.Errorf("%s\n%s", str, failed)
base.Errorf("%s", str)
}
fmt.Printf("FAIL\t%s [setup failed]\n", p.ImportPath)
continue
}
builds = append(builds, buildTest)
......
......@@ -226,7 +226,7 @@ func (b *Builder) Init() {
} else {
tmp, err := ioutil.TempDir(os.Getenv("GOTMPDIR"), "go-build")
if err != nil {
base.Fatalf("%s", err)
base.Fatalf("go: creating work dir: %v", err)
}
if !filepath.IsAbs(tmp) {
abs, err := filepath.Abs(tmp)
......
This diff is collapsed.
This directory holds test scripts *.txt run during 'go test cmd/go'.
To run a specific script foo.txt
go test cmd/go -run=Script/^foo$
In general script files should have short names: a few words, not whole sentences.
The first word should be the general category of behavior being tested,
often the name of a go subcommand (list, build, test, ...) or concept (vendor, pattern).
Each script is a text archive (go doc cmd/go/internal/txtar).
The script begins with an actual command script to run
followed by the content of zero or more supporting files to
create in the script's temporary file system before it starts executing.
As an example, run_hello.txt says:
# hello world
go run hello.go
stderr 'hello world'
! stdout .
-- hello.go --
package main
func main() { println("hello world") }
Each script runs in a fresh temporary work directory tree, available to scripts as $WORK.
Scripts also have access to these other environment variables:
GOARCH=<target GOARCH>
GOCACHE=<actual GOCACHE being used outside the test>
GOOS=<target GOOS>
GOPATH=$WORK/gopath
GOROOT=<actual GOROOT>
HOME=/no-home
PATH=<actual PATH>
TMPDIR=$WORK/tmp
devnull=<value of os.DevNull>
The environment variable $exe (lowercase) is an empty string on most systems, ".exe" on Windows.
The scripts supporting files are unpacked relative to $GOPATH/src (aka $WORK/gopath/src)
and then the script begins execution in that directory as well. Thus the example above runs
in $WORK/gopath/src with GOPATH=$WORK/gopath and $WORK/gopath/src/hello.go
containing the listed contents.
The lines at the top of the script are a sequence of commands to be executed
by a tiny script engine in ../../script_test.go (not the system shell).
The script stops and the overall test fails if any particular command fails.
Each line is parsed into a sequence of space-separated command words,
with environment variable expansion. Adding single quotes around text
keeps spaces in that text from being treated as word separators and also
disables environment variable expansion. Inside a single-quoted block of
text, a repeated single quote indicates a literal single quote, as in:
'Don''t communicate by sharing memory.'
A line beginning with # is a comment and conventionally explains what is
being done or tested at the start of a new phase in the script.
The command prefix ! indicates that the command on the rest of the line
(typically go or a matching predicate) must fail, not succeed. Only certain
commands support this prefix. They are indicated below by [!] in the synopsis.
The command prefix [cond] indicates that the command on the rest of the line
should only run when the condition is satisfied. The available conditions are:
- GOOS and GOARCH values, like [386], [windows], and so on.
- Compiler names, like [gccgo], [gc].
- Test environment details:
- [short] for testing.Short()
- [cgo], [msan], [race] for whether cgo, msan, and the race detector can be used
- [net] for whether the external network can be used
- [link] for testenv.HasLink()
- [symlink] for testenv.HasSymlink()
- [exec:prog] for whether prog is available for execution (found by exec.LookPath)
A condition can be negated: [!short] means to run the rest of the line
when testing.Short() is false.
The commands are:
- cd dir
Change to the given directory for future commands.
- cp src... dst
Copy the listed files to the target file or existing directory.
- env [key=value...]
With no arguments, print the environment (useful for debugging).
Otherwise add the listed key=value pairs to the environment.
- [!] exec program [args...]
Run the given executable program with the arguments.
It must (or must not) succeed.
Note that 'exec' does not terminate the script (unlike in Unix shells).
- [!] exists file...
Each of the listed files must (or must not) exist. (Directories are allowed.)
- [!] go args...
Run the (test copy of the) go command with the given arguments.
It must (or must not) succeed.
- mkdir path...
Create the listed directories, if they do not already exists.
- rm file...
Remove the listed files or directories.
- skip [message]
Mark the test skipped, including the message if given.
- [!] stale path...
The packages named by the path arguments must (or must not)
be reported as "stale" by the go command.
- [!] stderr pattern
Standard error from the most recent exec or go command
must (or must not) match the regular expression pattern.
- [!] stdout pattern
Standard output from the most recent exec or go command
must (or must not) match the regular expression pattern.
- stop [message]
Stop the test early (marking it as passing), including the message if given.
When TestScript runs a script and the script fails, by default TestScript shows
the execution of the most recent phase of the script (since the last # comment)
and only shows the # comments for earlier phases. For example, here is a
multi-phase script with a bug in it:
# GOPATH with p1 in d2, p2 in d2
env GOPATH=$WORK/d1:$WORK/d2
# build & install p1
env
go install -i p1
! stale p1
! stale p2
# modify p2 - p1 should appear stale
cp $WORK/p2x.go $WORK/d2/src/p2/p2.go
stale p1 p2
# build & install p1 again
go install -i p11
! stale p1
! stale p2
-- $WORK/d1/src/p1/p1.go --
package p1
import "p2"
func F() { p2.F() }
-- $WORK/d2/src/p2/p2.go --
package p2
func F() {}
-- $WORK/p2x.go --
package p2
func F() {}
func G() {}
The bug is that the final phase installs p11 instead of p1. The test failure looks like:
$ go test -run=Script
--- FAIL: TestScript (3.75s)
--- FAIL: TestScript/install_rebuild_gopath (0.16s)
script_test.go:223:
# GOPATH with p1 in d2, p2 in d2 (0.000s)
# build & install p1 (0.087s)
# modify p2 - p1 should appear stale (0.029s)
# build & install p1 again (0.022s)
> go install -i p11
[stderr]
can't load package: package p11: cannot find package "p11" in any of:
/Users/rsc/go/src/p11 (from $GOROOT)
$WORK/d1/src/p11 (from $GOPATH)
$WORK/d2/src/p11
[exit status 1]
FAIL: unexpected go command failure
script_test.go:73: failed at testdata/script/install_rebuild_gopath.txt:15 in $WORK/gopath/src
FAIL
exit status 1
FAIL cmd/go 4.875s
$
Note that the commands in earlier phases have been hidden, so that the relevant
commands are more easily found, and the elapsed time for a completed phase
is shown next to the phase heading. To see the entire execution, use "go test -v",
which also adds an initial environment dump to the beginning of the log.
Note also that in reported output, the actual name of the per-script temporary directory
has been consistently replaced with the literal string $WORK.
The cmd/go test flag -testwork (which must appear on the "go test" command line after
standard test flags) causes each test to log the name of its $WORK directory and other
environment variable settings and also to leave that directory behind when it exits,
for manual debugging of failing tests:
$ go test -run=Script -work
--- FAIL: TestScript (3.75s)
--- FAIL: TestScript/install_rebuild_gopath (0.16s)
script_test.go:223:
WORK=/tmp/cmd-go-test-745953508/script-install_rebuild_gopath
GOARCH=
GOCACHE=/Users/rsc/Library/Caches/go-build
GOOS=
GOPATH=$WORK/gopath
GOROOT=/Users/rsc/go
HOME=/no-home
TMPDIR=$WORK/tmp
exe=
# GOPATH with p1 in d2, p2 in d2 (0.000s)
# build & install p1 (0.085s)
# modify p2 - p1 should appear stale (0.030s)
# build & install p1 again (0.019s)
> go install -i p11
[stderr]
can't load package: package p11: cannot find package "p11" in any of:
/Users/rsc/go/src/p11 (from $GOROOT)
$WORK/d1/src/p11 (from $GOPATH)
$WORK/d2/src/p11
[exit status 1]
FAIL: unexpected go command failure
script_test.go:73: failed at testdata/script/install_rebuild_gopath.txt:15 in $WORK/gopath/src
FAIL
exit status 1
FAIL cmd/go 4.875s
$
$ WORK=/tmp/cmd-go-test-745953508/script-install_rebuild_gopath
$ cd $WORK/d1/src/p1
$ cat p1.go
package p1
import "p2"
func F() { p2.F() }
$
# Build should use GOTMPDIR if set.
env GOTMPDIR=$WORK/my-favorite-tmpdir
env GOCACHE=off
mkdir $GOTMPDIR
go build -work hello.go
stderr ^WORK=.*my-favorite-tmpdir
-- hello.go --
package main
func main() { println("hello") }
# Set up fresh GOCACHE.
env GOCACHE=$WORK/gocache
mkdir $GOCACHE
# Building trivial non-main package should run compiler the first time.
go build -x lib.go
stderr '(compile|gccgo)( |\.exe).*lib\.go'
# ... but not again ...
go build -x lib.go
! stderr '(compile|gccgo)( |\.exe).*lib\.go'
# ... unless we use -a.
go build -a -x lib.go
stderr '(compile|gccgo)( |\.exe)'
-- lib.go --
package lib
# Set up fresh GOCACHE.
env GOCACHE=$WORK/gocache
mkdir $GOCACHE
# Building a main package should run the compiler and linker ...
go build -o $devnull -x main.go
stderr '(compile|gccgo)( |\.exe).*main\.go'
stderr '(link|gccgo)( |\.exe)'
# ... and then the linker again ...
go build -o $devnull -x main.go
! stderr '(compile|gccgo)( |\.exe).*main\.go'
stderr '(link|gccgo)( |\.exe)'
# ... but the output binary can serve as a cache.
go build -o main$exe -x main.go
stderr '(link|gccgo)( |\.exe)'
go build -o main$exe -x main.go
! stderr '(link|gccgo)( |\.exe)'
-- main.go --
package main
func main() {}
[!gc] skip
# Set up fresh GOCACHE.
env GOCACHE=$WORK/gocache
mkdir $GOCACHE
# Building a trivial non-main package should run compiler the first time.
go build -x -gcflags=-m lib.go
stderr 'compile( |\.exe)'
stderr 'lib.go:2.* can inline f'
# ... but not the second, even though it still prints the compiler output.
go build -x -gcflags=-m lib.go
! stderr 'compile( |\.exe)'
stderr 'lib.go:2.* can inline f'
-- lib.go --
package p
func f(x *int) *int { return x }
# look for short, relative file:line in error message
! go run ../../gopath/x/y/z/err.go
stderr ^..[\\/]x[\\/]y[\\/]z[\\/]err.go:
-- ../x/y/z/err.go --
package main; import "bar"
# 'go install' with no arguments should clean up after go build
cd mycmd
go build
exists mycmd$exe
go install
! exists mycmd$exe
# 'go install mycmd' does not clean up, even in the mycmd directory
go build
exists mycmd$exe
go install mycmd
exists mycmd$exe
# 'go install mycmd' should not clean up in an unrelated current directory either
cd ..
cp mycmd/mycmd$exe mycmd$exe
go install mycmd
exists mycmd$exe
-- mycmd/main.go --
package main
func main() {}
cd mycmd
go build mycmd
# cross-compile install with implicit GOBIN=$GOPATH/bin can make subdirectory
env GOARCH=386
[386] env GOARCH=amd64
env GOOS=linux
go install mycmd
exists $GOPATH/bin/linux_$GOARCH/mycmd
# cross-compile install with explicit GOBIN cannot make subdirectory
env GOBIN=$WORK/bin
! go install mycmd
! exists $GOBIN/linux_$GOARCH
# installing standard command should still work
# (should also be mtime update only if cmd/pack is up-to-date).
! stale cmd/pack
[!short] go install cmd/pack
-- mycmd/x.go --
package main
func main() {}
# GOPATH with p1 in d1, p2 in d2
[!windows] env GOPATH=$WORK/d1:$WORK/d2
[windows] env GOPATH=$WORK/d1;$WORK/d2
# build & install p1
go install -i p1
! stale p1 p2
# modify p2 - p1 should appear stale
cp $WORK/p2x.go $WORK/d2/src/p2/p2.go
stale p1 p2
# build & install p1 again
go install -i p1
! stale p1 p2
-- $WORK/d1/src/p1/p1.go --
package p1
import "p2"
func F() { p2.F() }
-- $WORK/d2/src/p2/p2.go --
package p2
func F() {}
-- $WORK/p2x.go --
package p2
func F() {}
func G() {}
# go command should detect package staleness as source file set changes
go install mypkg
! stale mypkg
# z.go was not compiled; removing it should NOT make mypkg stale
rm mypkg/z.go
! stale mypkg
# y.go was compiled; removing it should make mypkg stale
rm mypkg/y.go
stale mypkg
# go command should detect executable staleness too
go install mycmd
! stale mycmd
rm mycmd/z.go
! stale mycmd
rm mycmd/y.go
stale mycmd
-- mypkg/x.go --
package mypkg
-- mypkg/y.go --
package mypkg
-- mypkg/z.go --
// +build missingtag
package mypkg
-- mycmd/x.go --
package main
func main() {}
-- mycmd/y.go --
package main
-- mycmd/z.go --
// +build missingtag
package main
# check for linker name in error message about linker crash
[!gc] skip
! go build -ldflags=-crash_for_testing x.go
stderr [\\/]tool[\\/].*[\\/]link
-- x.go --
package main; func main() {}
[!gc] skip
# listing GOROOT should only find standard packages
cd $GOROOT/src
go list -f '{{if not .Standard}}{{.ImportPath}}{{end}}' ./...
! stdout .
# TODO: ignore _/blah/go/src in output
# our vendored packages should be reported as standard
go list std cmd
stdout golang_org/x/net/http2/hpack
stdout cmd/vendor/golang\.org/x/arch/x86/x86asm
# go list supports -tags
go list -tags=thetag ./my...
stdout mypkg
-- mypkg/x.go --
// +build thetag
package mypkg
# patterns match directories with syntax errors
! go list ./...
! go build ./...
! go install ./...
-- mypkg/x.go --
package mypkg
-- mypkg/y.go --
pkg mypackage
# hello world
go run hello.go
stderr 'hello world'
-- hello.go --
package main
func main() { println("hello world") }
! go test badtest/...
! stdout ^ok
stdout ^FAIL\tbadtest/badexec
stdout ^FAIL\tbadtest/badsyntax
stdout ^FAIL\tbadtest/badvar
-- badtest/badexec/x_test.go --
package badexec
func init() {
panic("badexec")
}
-- badtest/badsyntax/x.go --
package badsyntax
-- badtest/badsyntax/x_test.go --
package badsyntax
func func func func func!
-- badtest/badvar/x.go --
package badvar
-- badtest/badvar/x_test.go --
package badvar_test
func f() {
_ = notdefined
}
# smoke test for complex build configuration
go build -o complex.exe complex
[exec:gccgo] go build -compiler=gccgo -o complex.exe complex
-- complex/main.go --
package main
import (
_ "complex/nest/sub/test12"
_ "complex/nest/sub/test23"
"complex/w"
"v"
)
func main() {
println(v.Hello + " " + w.World)
}
-- complex/nest/sub/test12/p.go --
package test12
// Check that vendor/v1 is used but vendor/v2 is NOT used (sub/vendor/v2 wins).
import (
"v1"
"v2"
)
const x = v1.ComplexNestVendorV1
const y = v2.ComplexNestSubVendorV2
-- complex/nest/sub/test23/p.go --
package test23
// Check that vendor/v3 is used but vendor/v2 is NOT used (sub/vendor/v2 wins).
import (
"v2"
"v3"
)
const x = v3.ComplexNestVendorV3
const y = v2.ComplexNestSubVendorV2
-- complex/nest/sub/vendor/v2/v2.go --
package v2
const ComplexNestSubVendorV2 = true
-- complex/nest/vendor/v1/v1.go --
package v1
const ComplexNestVendorV1 = true
-- complex/nest/vendor/v2/v2.go --
package v2
const ComplexNestVendorV2 = true
-- complex/nest/vendor/v3/v3.go --
package v3
const ComplexNestVendorV3 = true
-- complex/vendor/v/v.go --
package v
const Hello = "hello"
-- complex/w/w.go --
package w
const World = "world"
package badexec
func init() {
panic("badexec")
}
package badsyntax
func func func func func!
package badvar_test
func f() {
_ = notdefined
}
package main
import (
_ "complex/nest/sub/test12"
_ "complex/nest/sub/test23"
"complex/w"
"v"
)
func main() {
println(v.Hello + " " + w.World)
}
package test12
// Check that vendor/v1 is used but vendor/v2 is NOT used (sub/vendor/v2 wins).
import (
"v1"
"v2"
)
const x = v1.ComplexNestVendorV1
const y = v2.ComplexNestSubVendorV2
package test23
// Check that vendor/v3 is used but vendor/v2 is NOT used (sub/vendor/v2 wins).
import (
"v2"
"v3"
)
const x = v3.ComplexNestVendorV3
const y = v2.ComplexNestSubVendorV2
package v2
const ComplexNestSubVendorV2 = true
package v1
const ComplexNestVendorV1 = true
package v2
const ComplexNestVendorV2 = true
package v3
const ComplexNestVendorV3 = true
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment