Teach slapproxy to handle non-ascii instance parameters

Several fixes to make slapproxy accept creating instances with non-ascii parameters. My use-case it to instantiate gitlab with ICP, e.g. this way:

    ICP = 沪ICP备14008524号

gitlab-ce@1e11409c  
slapos@66919597

/cc @rafael, @alain.takoudjou, @kazuhiko, @jerome, @arnau

See merge request !3

1. when forming instance parameters xml dict on client side, we cannot
   use str() to convert parameter_value to string - because it breaks e.g.
   this way:

   In [1]: s = u'Привет Мир!'
   In [2]: str(s)
   ---------------------------------------------------------------------------
   UnicodeEncodeError                        Traceback (most recent call

   UnicodeEncodeError: 'ascii' codec can't encode characters in position
   0-5: ordinal not in range(128)

   Fix is: convert arguments to unicode strings.

2. when decoding xml on server side it was erroneously assumed to be
   always unicode, and this way xml.encode('utf-8') fails if xml is str
   but with non-ascii characters:

   In [3]: r = 'hello world'

   In [4]: r.encode('utf-8')
   Out[4]: 'hello world'

   In [5]: t = s.encode('utf-8')

   In [6]: t
   Out[6]: '\xd0\x9f\xd1\x80\xd0\xb8\xd0\xb2\xd0\xb5\xd1\x82 \xd0\x9c\xd0\xb8\xd1\x80!'

   In [7]: t.encode('utf-8')
   ---------------------------------------------------------------------------
   UnicodeDecodeError                        Traceback (most recent call last)

   UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)

   Fix is: don't assume xml string is unicode and decode it only, if it
           is unicode, and leave as it is if it is just str.

Because e.g. for the following instance parameters

    <?xml version='1.0' encoding='utf-8'?>
    <instance>
      <parameter id="text">Привет Мир!</parameter>
    </instance>

we get the following DB query

    'UPDATE %s SET slap_state="busy" ,software_release=? ,partition_reference=? ,requested_by=? ,software_type=? ,xml=? WHERE
reference=? AND computer_reference=?',
    ['http://sr//', 'myinstance', 'slappart0', 'default',
     '<?xml version=\'1.0\' encoding=\'utf-8\'?>\n<instance>\n  <parameter id="text">\xd0\x9f\xd1\x80\xd0\xb8\xd0\xb2\xd0\xb5\xd1\x82\xd0\x9c\xd0\xb8\xd1\x80!</parameter>\n</instance>\n',
     'slappart6', 'computer']

which fails the following way:

  File ".../site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File ".../site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File ".../site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File ".../site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File ".../site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "slapos/proxy/views.py", line 393, in requestComputerPartition
    software_instance = requestNotSlave(**parsed_request_dict)
  File "slapos/proxy/views.py", line 609, in requestNotSlave
    execute_db('partition', q, args)
  File "slapos/proxy/views.py", line 142, in execute_db
    cur = db.execute(query, args)
ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

our instance parameters is always XML UTF-8 encoded, and thus it is not
unicode and we cannot go the 'use unicode always' way.

So autotune sqlite connections on open to support 8-bit strings.

Currently it is just sqlite3.connect(), but we'll want to add connection
autotuning in the next patch, so before that let's consolidate all
sqlite opening places in the code into one.

Partition folder snapshot use du to calculate folder size and return value in Mb.
This snapshot is enabled by default and will calculate size of each partition every 24hours.
Partition with size < 1Mb are not are not reported (considered as empty).
The behavior can be changed using some configurations in slapos.cfg:

[collect]
report_disk_usage = True

disk_snapshot_process_pid_foder = /srv/slapgrid/var/run

disk_snapshot_time_cycle = 86400

  There was an obvious check just after query.

  The order_by was already ignored, and it is not required to sort the results, and there is no meaning of process in a sorted order. 

A list of authorized ip can also be send

Tap nobridge



See merge request !2

firewall process is now controlled by supervisord

slapos firewall: speed up computer_partitions configuration

slapos firewall: block etablished and related connection which are not allowed

slapos_web: Improve the display of service list and hosting subscription (short title and description)

https://github.com/giampaolo/psutil/issues/451
( and other deprecated API were removed )