(allow no arguments of paths to protect)

* static files ending with an index were redirected improperly

* optimize requestPage

Distinguishable exit codes

0: normal or expected exit
1: error before server finished starting
2: double SIGINT (force quit)
3: error stopping with SIGQUIT
4: shutdown callback(s) returned error(s)

Only the outside function call is executed in a new goroutine when
invoking 'go'. Oops. Force quits (2 SIGINTs) now work again.

fastcgi: Fix for #1788. Align atomic fields to 64-bit word (Go bug)

Also clarified a comment in SiteConfig

Move fallbackHosts to vhostTrie

httpserver: Add global FallbackHosts for vhost matching

mitm: Fix out of bounds error when checking software version in UA

httpserver: Encapsulate WriteSiteNotFound error

proxy: fix hang on chunked websocket server

Signed-off-by: Tw <tw19881113@gmail.com>

Pushes for /index.html work when surfing to /

Use httpserver.IndexFile() to determine index files

Test if middleware pushes indexfile when requesting directory

Fix codereview issues

Serve original request first, push later

Revert "Serve original request first, push later"

This reverts commit 2c66f01115747e5665ba7f2d33e2fd551dc31877.

log,error: fix roller parser issue

Signed-off-by: Tw <tw19881113@gmail.com>

Include test for iOS 11 beta

* Revert fastcgi to emove persistent connections.

* Fix linting errors

* reintroduce timeout tests

* check for non-zero timeout

* ensure resp is not nil

* Adding nobots directive

* Moving nobots directive behind log one.

* Move nobots directive to a better position

gzip: change ETag to weak etag after gzip

According to https://tools.ietf.org/html/rfc7232#section-2.1
> Likewise, a validator is weak if it is shared by two or more
representations of a given resource at the same time, unless those
representations have identical representation data.  For example, if
the origin server sends the same validator for a representation with
a gzip content coding applied as it does for a representation with no
content coding, then that validator is weak.

Therefore, after gzip, we should change the original etag to weak etag.

Not much changed but it's a clean pull now; easier for future reference.