nexedi / slapos.toolbox

  Follow up erp5@ccbf9007

 This bug was preventing to get a single test per group (computer) 
 per time. Causing multiple tests run on the same computer and no
 tests run on others. 

  Retry if git command fail (to fetch for example) and also retry if slapos report a connection error

  There is no need to be faster them 2 minutes between the queries.

  - Get precise state of all software instances from a Hosting Subscription
  - Get the connection parameters from  all software instances from a Hosting Subscription
  - Get the instance parameters from all software instances from a Hosting Subscription
  - Move all API to connect slapos master to HAL implementation
  - Get the state of the Software Installation
  - Try to connect to monitoring tool in case of error.
  - Use TaskDistributor to determinate the tests (instead local file)
  - Use erp5.util and few erp5testnode features to determinate test revisions.
  - Download informations from the Task distributor instead use parameters (it makes implementation more flexible).

Conflicts:
	slapos/agent/agent.py

This change also replace break by continue, in order to try next computer,
if others computers exist.

The tests runs always 1 test per available computer (no more
no less), this approach prevent pick several tests on the same
machine which leads to timeouts and lack of resources (Disk) on
some machines.

If a software release fail to install because it was not found, there is no
reason to tear down it.

Network can be quite unreliable, so play safe retrying the method in case
some exception related to network failure be raised. Usually they are
related to timeout on SSLError and/or xmlrpclib timeouts.

Make test agent delay more time for 2 reasons:

  - slapgrid-cp/sr takes more them 1 min to run
  - The default periodicity on the servers are 5 min, so nothing happens in
        less them 5 minutes.

This change also gives time for vifib master process the information with
activities.

This changes improve the log readability in order to identify when a test
start and stops (tears down.)

Reuse erp5.util.taskdistribution for better TaskDistributionTool
integration (adds a dependency, removes 2 entry points): it is possible to
create several test agents with same configuration, each testing a subset
of the entire test suite. Especially useful if/when test agents gets the
extra task of monitoring commits on software releases, and they locally
build & instanciate the SR they test.
Use a medium-lived process rather than a short-lived one, to drop state
loading & saving complexity.
Allow a single test agent to run several concurrent tests.
Cleanup pidfile, even when exceptions occur.
Factorise code used to test each step.
Rework configuration file layout, to better use magic "[DEFAULT]" section.
Take keys & certs from configuration file, allowing for per-test certs.
Remove the need for storing using login & password by reusing x509 auth.

This side-effect was removed in "Pass connection helper as a constructor
parameter, not as class property." commit on slapos.core repository
(hash = c2d5a1eb5d256ef580b1afd34633413adee23e8c).

Closes the gap between command-line arguments and configuration file.
Also, make key_file and cert_file optional, as they are only needed by
slap when master_url uses the "https" scheme.