Validation happens on each configuration change, but for sure it is checked
each 2 hours.

State of configuration is calculated in separate script.

As some tools are not configurable with logrotate provide a script to
"rotate" files generated by those tools.
This script more or less follows logrotate behaviour:

 * moves rotated files to logrotate directory
 * compresses them with xz
 * have to be configured to keep only files for some days

There is no need anymore to have two processes for normal and nginx slaves,
as nginx ones are served by caddy anyway.

Also inform the requester that type:eventsource is not implemented.

This reverts commit 7993ff81.

Custom configuration checks are hard to be trusted, as they can impact too
many aspects of running frontend.

Frontend administrator knows the risks of custom configuration, and shall take
proper care.

/reviewed-on nexedi/slapos!543

Instead of fetching certificates on each slapos node instance use new
kedifa-updater, which is a tool to asynchronously fetch certificates and
has a hook to reload the server in case if new certificate is available.

custom_ssl_directory is NOT BBB

Adapted configuration and instantiation to ATS 7.

Deployment:
 * traffic_line has been replaced with traffic_ctl
 * access log, of squid style, is ascii instead of binary, to do so
   logging.config is generated
 * ip_allow.config is configured to allow access from any host
 * RFC 5861 (stale content on error or revalidate) is implemented with core
   instead with deprecated plugin
 * trafficserver-autoconf-port renamed to trafficserver-synthetic-port
 * proxy.config.system.mmap_max removed, as it is not used by the system anymore

Tests:
 * As Via header is not returned to the client, it is dropped from the
   tests, instead its existence in the backend is checked.
 * Promise plugin trafficserver-cache-availability.py is re enabled, as
   it is expected to work immediately.

csr_id is exposed over HTTPS with short living self signed certificate,
which is transmitted via SlapOS Master. Thanks to this, it is possible to
match csr_id with certificate of given partition and take decision if it shall
be signed or not.

This is "quite secure" apporach, a bit better than blidny trusting what CSR
to sign in KeDiFa. The bootstrap information, which is short living
(certificates are valid for 5 days), resides in SlapOS Master. The csr_id
is not directly known to SlapOS Master, and shall be consumed as fast as
possible by frontend cluster operator in order to sign CSR appearing in
KeDiFa caucase. The known possible attack vector requires that attacker knows
caucased HTTP listening port and can hijack HTTPS traffic to the csr_id-url
to get the human approve his own csr_id. The second is hoped to be overcomed
by publishing certificate of this endpoint via SlapOS Master.

Unfortunately caucase-updater prefix is directly used to find real CSR, as the
one generated is just a template for rerequest, thus csr_id would be different
from really used by caucase-updater.

Use KeDiFa to store keys, and transmit the url to the requester for master
and slave partitions.

Download keys on the slave partitions level.

Use caucase to fetch main caucase CA.

kedifa-caucase-url is published in order to have access to it.

Note: caucase is prepended with kedifa, as this is that one.

Use kedifa-csr tool to generate CSR and use caucase-updater macro.

Switch to KeDiFa with SSL Auth and updated goodies.

KeDiFa endpoint URLs are randomised.

Only one (first) user certificate is going to be automatically accepted. This
one shall be operated by the cluster owner, the requester of frontend master
partition.

Then he will be able to sign certificates for other users and also for
services - so each node in the cluster.

Special trick from https://security.stackexchange.com/questions/74345/provide-subjectaltname-to-openssl-directly-on-command-line
is used for one command generation of extensions in the certificate.
Note: We could upgrade to openssl 1.1.1 in order to have it really
simplified (see https://security.stackexchange.com/a/183973 )

Improve CSR readability by creating cluster-identification, which is master
partition title, and use it as Organization of the CSR.

Reserve slots for data exchange in KeDiFa.

Validate only if interesting files are changed.

Simplify validation and graceful scripts, have one template for all cases.

Aviod needless repetition.

Note: There is limit of arguments to be passed to commands, see
https://www.in-ulm.de/~mascheck/various/argmax/, but we are safe for now:

  $ getconf ARG_MAX
  2097152

So we are keeping shell expansion instead of playing with find or other tools.

Even if the master partition owner will authorise given slave for custom
configuration reject this slave in case if it does not pass validation for
snippet.

Create caddyprofiledeps egg with dummy noop recipe.

Thanks to setting dependencies of this egg and enabling it on the instance
profile, buildout will install eggs during software run and activate them
during instance run.

No existing egg (like slapos.cookbook) is used, as this technique is to allow
profile/software release developer to choose required eggs used during
instantiation.

Another apporach would be to add dependency for validators in
slapos.recipe.template (in install_requires).

Features:

 * jinja2 is used to generate instance templates
 * downloads are done the same way for all resources
 * create with shared content for all instance profiles
 * fill in instance-common with shared sections
 * render templates late in order to ease its extenension and development
 * drop not needd duplicated section
 * drop slap-parameter in frontend and replicate template
 * simplify monitor configuration
 * move instance-parameter to instance file
   Thanks to this only one and topmost profile is reponsible for parsing and
   passing through the information which comes from the network

The backend url can come from request in `url` and `https-url` strings.
It is validated using real caddy template configuration and by using
caddy's `-stdin`. It results with calling it on each slave having any of
those parameters.

Those kept are backward compatibility variables from the request.

Caddy is able to bind only to all or one interface
( https://github.com/mholt/caddy/issues/864 )

By using 6tunnel this limitation is workarounded, and in the result listen on IPv6.

Also drop needless "ipv6" keys across configuration.

Features:

 * shared place for Caddy configuration
 * gather a lot of parameters for caddy executable, as dislike Apache
   Caddy is configured from command line
 * dummy vhost for example.org
 * challanges (ACME SSL) are disabled
 * bind to interfaces are done per site
 * cache access is dummy, but working
 * /server-status redone in Caddy style
 * antiloris dropped, as this is apache specific
 * apache_custom_http and apache_custom_https
 * dropped not needed leftover access-control-string and protected-path
 * nginx replacement added
 * bin/caddy-wrapper is provided in order to allow parameterization of caddy
   over the network
 * access to log files over http is provided
   * username on log access is consistent, it is not uppercased like it was
     originally on apache-frontend
 * list of TODOs in TODO.rst

This will make it easier to track changes.

  Wait for 60 to reload apache configuration in order to accumulate
  several logrotate runs.

  If the amount of slaves are too high, the number of logs are high,
  so the entries on logrotate are also high. So it is enough to DDoS
  with a huge amount of 'kill -1', so delay is the only way to avoid
  to re-implement logrotate existing features.

  Only reload the apache configuration if the the apache configuration
  or the certificates contains a change, else don't reload it.

  Keep a command on bin folder to force reload of configuration in
  case it is required.

If not so, trafficserver configuration is removed and then replaced on
every slapos node instance call making traffic server unstable

  Introduce NGINX on the same partition of apache to handle websocket\
  and eventsource types.

  The NGINX will run on another port and it would require a second ip at
  the machine for enable it.

  This configuration is a working version with fully https support, but
  some additional adjustments might be required.

  Use a single apache server to handle cache and normal apache configurations. With trafficserver,
  the access would follow the model:

    Apache > Traffic Server > Apache 'Cache' > Backend

  Now the configuration changed to:

    Apache > Traffic Server > Apache (same as before) > Backend

  This simplify the amount of apache process to manage on the frontend.

  Reduce a bit the number of sections also for create directories
  Move all 'set's and 'do's to earliest as possible, to keep buildout syntax more evident
  Drop duplicated logics and 'if's reducing general code