Hello up there,

This merge-requests primarily adjusts promises code to treat `data` field in `*.json.log` as real JSON. It is counterpart to slapos!1447 .

- patch 1 syncs tests to current state of ORS SR;
- patch 2 removes replacement of `'` with `"` in input data;
- patch 3 removes UTF-8 decoding when input is fed to json.loads.

Please see individual patches for details.

Kirill

/cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus, @jerome
/reviewed-on !120 and slapos!1447

Starting from py3.6 json.loads accepts all unicode, bytes and bytearray:

    https://docs.python.org/3/library/json.html#:~:text=json.loads(s
    https://github.com/python/cpython/commit/b161562f72a2
    https://bugs.python.org/issue17909

Adjust promises correspondingly to slapos!1447
which changed ors-amarisoft software-release to emit log payload via
json.dumps instead of py %s.

Do this for both promises code and JSON-loading utilities in slapos/promise/plugin/util.py also used inside ORS promises code.
We can adjust the utilities because there are two of them (get_json_log_data_interval and get_json_log_latest_timestamp) with
get_json_log_latest_timestamp being used only in ORS-promises:

    slapos.toolbox$ git grep get_json_log_latest_timestamp
    slapos/promise/plugin/check_amarisoft_stats_log.py:from .util import get_json_log_latest_timestamp
    slapos/promise/plugin/check_amarisoft_stats_log.py:    latest_timestamp = get_json_log_latest_timestamp(self.amarisoft_stats_log)
    slapos/promise/plugin/util.py:def get_json_log_latest_timestamp(json_log_file):

and get_json_log_data_interval being used in ORS promises and
check_cpu_temperature, check_network_transit and check_ram_usage:

    slapos.toolbox$ git grep get_json_log_data_interval
    slapos/promise/plugin/check_baseband_latency.py:from .util import get_json_log_data_interval
    slapos/promise/plugin/check_baseband_latency.py:    data_list = get_json_log_data_interval(self.amarisoft_stats_log, self.stats_period * 5)
    slapos/promise/plugin/check_cpu_temperature.py:      temp_list = self.get_json_log_data_interval(self.avg_temp_duration)
    slapos/promise/plugin/check_network_transit.py:      temp_list = self.get_json_log_data_interval(self.transit_period)
    slapos/promise/plugin/check_ram_usage.py:      temp_list = self.get_json_log_data_interval(self.avg_ram_period)
    slapos/promise/plugin/check_rx_saturated.py:from .util import get_json_log_data_interval
    slapos/promise/plugin/check_rx_saturated.py:    data_list = get_json_log_data_interval(self.amarisoft_stats_log, self.stats_period * 2)
    slapos/promise/plugin/util.py:def get_json_log_data_interval(json_log_file, interval):
    slapos/promise/plugin/util.py:  def get_json_log_data_interval(self, interval):
    slapos/promise/plugin/util.py:    return get_json_log_data_interval(self.__json_log_file, interval)

However all check_cpu_temperature, check_network_transit and
check_ram_usage produce their *.json.log files themselves and already
emit `data` via json.dumps:

https://lab.nexedi.com/nexedi/slapos.toolbox/blob/453dce5f/slapos/promise/plugin/check_cpu_temperature.py#L41-54
https://lab.nexedi.com/nexedi/slapos.toolbox/blob/453dce5f/slapos/promise/plugin/check_network_transit.py#L29-42
https://lab.nexedi.com/nexedi/slapos.toolbox/blob/453dce5f/slapos/promise/plugin/check_ram_usage.py#L34-46

So it is safe to adjust the utilities.

tests/promise/plugin: Adjust Amarisoft-related tests to match amarisoft_stats.json.log emitted by SR

In nexedi/slapos@b32b4a8e `data` field in
that log switched from emitting objects via py repr to emitting them via
JSON. As the result, in particular, strings are now emitted quoted with
" instead of with ' .

-> Adjust all tests that simulate amarisoft_stats.json.log to follow
actual software-release behaviour.

The tests continue to pass.

See merge request nexedi/slapos.toolbox!119

See https://pypi.org/help/#description-content-type

In some deployments it is important to being able to select only specific
senses on which the promise shall react.

See also nexedi/slapos@4230cd1b

Tests are rewritten for easier coverage of all cases, including all-ok case.
In the same time code has been improved while detecting inconsistencies.

Sort entry_list by known key, the one most important for the usage.

Instead of hiding unconfigured checks, emit OK message, which gives
information to the user regarding the full status.

The IP list comparision of http_query has been dropped, as the IP presence
shall be compared from DNS result.

Improvements:
 * use real dict in the test
 * merge test cases
 * increase test coverage
 * minimize and improve messages
 * use error state instead of error list
 * always shall full diff during test failure to ease debugging
 * fix ssl_certificate missing information
 * avoid cluttering code with test time

See merge request nexedi/slapos.toolbox!113

Also add a "Requires-Python" information so that dropping python2
support will be easier in the future. See https://packaging.python.org/en/latest/guides/dropping-older-python-versions/

Note that this promise is only available in Python3.