See merge request nexedi/slapos.core!346

See merge request nexedi/slapos.core!347

Fix a missing dependency in 1.7.0 on python2

Minor version was increased because "slapgrid: Process promises with
instance python" brings full support of using slapos node on python3
also when using software on python2

Support a syntax like:

      slapos request \
        --node=computer_guid=local \
        --parameters-file=~/request.json \
        ERP5 \
        https://lab.nexedi.com/nexedi/slapos/raw/1.0.145/software/erp5/software.cfg

to request an instance with parameters in ~/request.json file.

This also teach slapos.core about the instance schema, which makes it possible to
understand the serialisation, so that we no longer need to use `_ = json.dump(params)`
to encode parameters.

This also emits a warning when requesting with parameters which do not match the schema.


See merge request nexedi/slapos.core!215

Because json is unicode in python2, pretty-printing the parameters would
display something like {u'foo': u'bar'}. Encode these unicode strings on
python2, so that it looks like {'foo': 'bar'} on both python2 and python3

This is more readable

Support a syntax like:

      slapos request \
        --node=computer_guid=local \
        --parameters-file=~/request.json \
        ERP5 \
        https://lab.nexedi.com/nexedi/slapos/raw/1.0.145/software/erp5/software.cfg

to request an instance with parameters in ~/request.json file.

This automatically understands the serialisation, when software uses json-in-xml
serialisation, no need to use `_` argument with the encoded json.

This allows easily serving a directory content with a server like this:

    from six.moves import SimpleHTTPServer

    class FileHTTPServer(ManagedHTTPServer):
      working_directory = profile_dir
      RequestHandler = SimpleHTTPServer.SimpleHTTPRequestHandler

See merge request nexedi/slapos.core!351

SlapOS ticket RSS have one entry for each ticket, with the content of
the latest event as description, but when using a feed reader application,
we'd like to have a new entry for each message posted to the ticket, to
be notified that a new message was posted.

Feed reader applications fetch the RSS and have to determine if each
entry is already in their database or not. In a scenario like below, we
observe different behaviors for different applications:

  1. a user open a ticket "help" with a first message "I need help !".
    The RSS entry for this ticket is:

    <item>
      <link>https://slapos.example.com/#/support_request_module/1</link>
      <title>Help</title>
      <pubDate>Mon, 01 Jan 2001 10:57:10 +0100</pubDate>
      <description>I need help !</description>
    </item>

   => feed reader fetches the RSS and add a new entry with title "Help"

  2. support operator reply with a second message "how can I help you ?"
    The RSS entry for the ticket becomes:

    <item>
      <link>https://slapos.example.com/#/support_request_module/1</link>
      <title>Help</title>
      <pubDate>Mon, 02 Jan 2001 08:32:12 +0100</pubDate>      <== this is different
      <description>How can I help you ?</description>           <== this is different
    </item>

now depending on the implementation of the feed reader, it will either:

 - add a new entry, this is the best behavior. In the RSS readers I tested,
  liferea and newsboat do this. Probably they consider that the entry is
   different because it has a new pubDate and/or description.
 - update the existing entry "in place" - this is not so good, because
  this does not appear as a new entry and the original message is no
  longer in the RSS reader. TinyTinyRSS behaves like this (at least in
  version 17.4, which is a bit old)
 - consider they already have the entry and don't do anything - this is
  even worse, because user can not see there was a reply. FreshRSS
  behaves like this.

A RSS for tickets will always have the problem that the feed reader must
refresh often enough if we want to have a copy of all messages in the
feed reader (and that's why ERP5 CRM's uses a RSS of Events and not
Tickets), but we can make this situation better by using guid on messages.

By constructing a guid [1] that will become different every time a new
message was posted, feed readers consider each item as a a new, different
entry. At least this is the case for all the feed reader I tested, except
ERP5 builtin feed reader which only consider the `link` attribute.

Strictly speaking this implementation has a "problem" that these guids
are not permalinks, but I believe that's something we should address in
erp5_rss_style (if guid does not look like an URL, set isPermalink="false"),
but in practice it does not seem to be a problem.

1: https://validator.w3.org/feed/docs/rss2.html#ltguidgtSubelementOfLtitemgt

It can happen that slapos can be terminated without removing the pid
file, and another long running process using the same pid from the one
in the pid file is running, which cause slapos node to never run.

This makes setRunning a little bit clever, by ignoring the process
unless it also has `slapos` in its name.

When using HTTPS, slapos.libnetworkcache uses exclusively the client certificate
(which must be configured) so https://*.nxdcdn.com can't work from slapos.libnetworkcache

And without download-dir-url, download-cache-url is quite useless.

Reorder options to clarify the difference between upload & https.

See merge request nexedi/slapos.core!345

   We just leave as it is, and let the user decide for himself the value.

  Same as Subscription Requests

On python3 slapos.core for python2 software release, the output was bytes:

```
2021-11-05 17:28:39 slapos[12654] INFO Error with promises for the following partitions:
2021-11-05 17:28:39 slapos[12654] INFO   slappart7[balancer]: b'Promise \'check-apachedex-result.py\' failed with output: ERROR \'"/srv/slapgrid/slappart3/srv/runner/software/cc0326f0dcb093f56c01291c300c8481/bin/check-apachedex-result" --apachedex_path "/srv/slapgrid/slappart3/srv/runner/instance/slappart7/srv/monitor/private/apachedex" --status_file /srv/slapgrid/slappart3/srv/runner/instance/slappart7/srv/monitor/private/apachedex.report.json --threshold "70"\' run with failure, output: \'Score too low: 0% - Threshold: 70.0%\\n\''
```

See merge request nexedi/slapos.core!344

    It add a script to be custom on project site, if you have short term users, that requires short password expiration timing.

   Included for the changes on the configured chinese taxation

1) when ansible is still python2 and the callback "json" was introduced
(e.g. in Ubuntu 18.04) we need to use "strict_import" to be sure to use
the json python module and not the json callback

2) when ansible is python3 the syntax must be python3 compatible

3) also remove an uneeded data.copy()

As reported on https://github.com/xolox/python-coloredlogs/issues/107
logging objects with a __str__ method returning non-ascii characters
raises UnicodeDecodeError.

We have vendored coloredlogs version 0.5 long time ago, so just
apply the suggested fix here for now.

Because we were reading subprocess output as bytes, the output in case
of promise errors was something like b'ERROR ...'

There was another problem writing string to an binary file in error path.

When using virtualenv, the module paths are not simple list, but
instances of importlib._bootstrap_external._NamespacePath.

According to docs [1] module's __path__ does is not a list, but an
iterator producing strings.

Cast to a list to consume these iterators.

[1]
https://docs.python.org/3/reference/import.html?highlight=__path__#path__

We fork a subprocess to serve requests, but the listening socket is bound
in server constructor and inherited by subprocess. With this pattern, we
have to explicitly close the socket from parent process to prevent the
leak.

Allow to have missing bin/buildout of the software release, especially to cover
the case, when SR is not installed for partition being processed, eg in order
to destroy it.

  This work is unfinished, so it will be reworked on a branch.