GitLab

ORS software-release emits many .json.log files with information about current state of eNB and radio units. Those files are used by code in slapos.toolbox to implement corresponding promises. The files, it seems, try to match JSON Lines format() with every line containing a JSON object describing timestamp, log level and log payload of corresponding log entry.

So far all is good. One peculiarity, however, is that, contrary to everything else, the content of the data field in the logs is not emitted via JSON-encoding. Initially, starting from 4968d55c (ors-amarisoft: get eNB stats and add promises), that data payload was being emitted via %s py formatting, which is similar to what JSON-encoding would produce, but is not it exactly. For example if log payload is {'abc': 123} then %s and JSON would quote abc differently:

{'abc': 123}  # %s
{"abc": 123}  # JSON

That would not be a big deal if what %s produces would be still valid JSON accepted by JSON-decoder, but unfortunately it is not so: JSONDecodeError is raised when {'abc': 123} is tried to be decoded.

The code in slapos.toolbox, so far, is handling this situation by feeding to JSON-decoder original input with ' characters replaced by "

json.loads(line.decode().replace("'", '"'))     (+)

which more or less works in basic cases, but breaks when log payload contains ' character itself. For example for {"a'bc": 123} replacing gives {"a"bc": 123}, which, being invalid JSON, raises

JSONDecodeError: Expecting ':' delimiter: line 1 column 5 (char 4)

when fed to json.loads() .

I've actually hit this problem for real with amarisoft-rf-info.json.log where the data part contains output of rf_info looking e.g. as

TX underflows=0 RX overflows=0
TRX SDR driver 2023-09-07, API v15/18
PCIe CPRI /dev/sdr0@0:
  Hardware ID: 0x4b12
  DNA: [0x0048248a334a7054]
  Serial: ''
  FPGA revision: 2023-06-23  10:05:24
  FPGA vccint: 0.98 V
  FPGA vccaux: 1.76 V
  FPGA vccbram: 0.98 V
  FPGA temperature: 71.9 °C
  Clock tune: 0.0 ppm
  NUMA: 0
  CPRI_option: '5' (x8) lock=no
  DMA0: TX fifo: 66.67us  Usage=16/32768 (0%)
  DMA0: RX fifo: 66.67us  Usage=16/32768 (0%)
  DMA0 Underflows: 0
  DMA0 Overflows: 0
PCIe CPRI /dev/sdr0@1:
  Hardware ID: 0x4b12
  DNA: [0x0048248a334a7054]
  Serial: ''
  FPGA revision: 2023-06-23  10:05:24
  FPGA vccint: 0.98 V
  FPGA vccaux: 1.77 V
  FPGA vccbram: 0.98 V
  FPGA temperature: 71.7 °C
  Clock tune: 0.0 ppm
  NUMA: 0
  CPRI_option: '5' (x8) lock=HW+SW rx/tx=46.606us
    Port #0: T14=46.606us
  DMA0: TX fifo: 66.67us  Usage=16/32768 (0%)
  DMA0: RX fifo: 66.67us  Usage=16/32768 (0%)
  DMA0 Underflows: 0
  DMA0 Overflows: 0

which lead to breakage when trying to load that log due to ' symbols.

The fix is simple: change the code, that emits *.json.log to emit data payload via json.dumps instead of %s. We can do that without breaking anything because, contrary to enb.xlog, those *.json.log are not uploaded to any separate system and currently are only used to implement promises without preserving log files in any storage for any non-small amount of time. In other words, currently those *.json.log are temporary files whose format can be adjusted without strongly caring about backward compatibility.

b32b4a8e (software/ors-amarisoft: general improvement for RU (logs/promises/input parameters)) already changed %s to json.dumps for amarisoft-stats.json.log , but left all other logs at %s.

-> Fix this for everything by replacing %s to json.dumps in all generated logs.
-> Corresponding slapos.toolbox adjustments are in slapos.toolbox!120 (merged).

/cc @jhuge, @lu.xu, @tomo, @xavier_thompson, @Daetalus, @jerome

(*) https://jsonlines.org/
(+) see e.g. https://lab.nexedi.com/nexedi/slapos.toolbox/blob/453dce5f/slapos/promise/plugin/util.py#L50