Commit da71a8ed authored by Thomas Gambier's avatar Thomas Gambier 🚴🏼 Committed by Julien Muchembled

download: don't use splitport and splituser

They are deprecated since Python 3.8

Rebase instructions:
- split and fixup commits
    171c2459
    c9e9b267
parent 2789ebd1
Pipeline #21139 passed with stage
in 0 seconds
...@@ -21,13 +21,13 @@ except ImportError: ...@@ -21,13 +21,13 @@ except ImportError:
try: try:
# Python 3 # Python 3
from urllib.error import HTTPError from urllib.error import HTTPError
from urllib.request import Request, splitport, splituser, urlopen from urllib.request import Request, urlopen
from urllib.parse import urlparse, urlunparse from urllib.parse import urlparse, urlunparse
except ImportError: except ImportError:
# Python 2 # Python 2
from urlparse import urlparse from urlparse import urlparse
from urlparse import urlunparse from urlparse import urlunparse
from urllib2 import HTTPError, Request, splitport, splituser, urlopen from urllib2 import HTTPError, Request, urlopen
from zc.buildout.easy_install import realpath from zc.buildout.easy_install import realpath
from base64 import b64encode from base64 import b64encode
...@@ -271,14 +271,15 @@ class Download(object): ...@@ -271,14 +271,15 @@ class Download(object):
return '%s:%s' % (url_host, url_port) return '%s:%s' % (url_host, url_port)
def urlretrieve(self, url, tmp_path): def urlretrieve(self, url, tmp_path):
scheme, netloc, path, params, query, frag = urlparse(url) parsed_url = urlparse(url)
req = url req = url
while scheme in ('http', 'https'): # not a loop while parsed_url.scheme in ('http', 'https'): # not a loop
auth, host = splituser(netloc) auth_host = parsed_url.netloc.rsplit('@', 1)
if auth: if len(auth_host) > 1:
url = urlunparse((scheme, host, path, params, query, frag)) auth = auth_host[0]
url = parsed_url._replace(netloc=auth_host[1]).geturl()
else: else:
auth = netrc.authenticators(splitport(host)[0]) auth = netrc.authenticators(parsed_url.hostname)
if not auth: if not auth:
break break
auth = '{0}:{2}'.format(*auth) auth = '{0}:{2}'.format(*auth)
......
  • This commit shows a general problem. For example, I'm working in a Theia that uses Python 3.9 and I can't instantiate anymore old SR that are already using Python 3. I'm sure this kind of issue will happen again in the future.

    For the moment, I have no other idea than bootstrapping with a dedicated Python that is as old as possible, problably 2.7 for still many years.

    /cc @jerome @xavier_thompson

  • In slapos!1207 (comment 174788) we described a similar problem that happens with networkcache ( the symptoms were that there is no error running buildout, but the generated bin/buildout does not use networkcache). For this, we had the idea of having slapos.core (or slapos-node package) include a version of buildout and slapos.libnetworkcache for bootstrap, so that we can make sure we have compatible versions. This was just a general idea, I don't know if we tried.

    slapos-node package is using python3.7 (which today is also EOL) for some compatibility reason, maybe for the same problem.

  • slapos-node package is using python3.7 (which today is also EOL) for some compatibility reason, maybe for the same problem.

    Yes, indeed, this is exactly for this same problem. And as you said the solution is to bundle a version of buildout and libnetworkcache compatible with slapos-node's python.

  • slapos-node can't have a version that's too old to run things like supervisord. More generally, we'll want the SR python of slapos-node or Theia (i.e. the python that runs buildout and any python software in the instance) to be as recent as possible, as we usually do. Not only for security reasons but also because it would be nightmare maintenance to keep a very old & full-featured version in slapos.git.

    For me, the only possible minimal change is to add a second python in slapos-node that's only used for bootstrapping. It would add only networkcache (and its dependencies). We'd start with Python 2.7. It would join the minimal set of binaries that the machine is expected to provide when starting to work with slapos; the only difference with /bin/sh, gcc & al. is that most OS stop shipping very old Python so we'd have to ship it ourselves (i.e. slapos-node).

    Then there's the case of Theia (or any SR that bundles a slapos node):

    • it could inherit this special python from the parent slapos node if there's any, using symlinks
    • or else build it, but that's not use case we currently have (I mean: build a slapos SR outside of a slapos node)
    Edited by Julien Muchembled
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment