download: don't use splitport and splituser

They are deprecated since Python 3.8

Rebase instructions:
- split and fixup commits
    171c2459
    c9e9b267

src/zc/buildout/download.py

@@ -21,13 +21,13 @@ except ImportError:
 try:
     # Python 3
     from urllib.error import HTTPError
-    from urllib.request import Request, splitport, splituser, urlopen
+    from urllib.request import Request, urlopen
     from urllib.parse import urlparse, urlunparse
 except ImportError:
     # Python 2
     from urlparse import urlparse
     from urlparse import urlunparse
-    from urllib2 import HTTPError, Request, splitport, splituser, urlopen
+    from urllib2 import HTTPError, Request, urlopen
 
 from zc.buildout.easy_install import realpath
 from base64 import b64encode
@@ -271,14 +271,15 @@ class Download(object):
             return '%s:%s' % (url_host, url_port)
 
     def urlretrieve(self, url, tmp_path):
-        scheme, netloc, path, params, query, frag = urlparse(url)
+        parsed_url = urlparse(url)
         req = url
-        while scheme in ('http', 'https'): # not a loop
-            auth, host = splituser(netloc)
-            if auth:
-                url = urlunparse((scheme, host, path, params, query, frag))
+        while parsed_url.scheme in ('http', 'https'): # not a loop
+            auth_host = parsed_url.netloc.rsplit('@', 1)
+            if len(auth_host) > 1:
+                auth = auth_host[0]
+                url = parsed_url._replace(netloc=auth_host[1]).geturl()
             else:
-                auth = netrc.authenticators(splitport(host)[0])
+                auth = netrc.authenticators(parsed_url.hostname)
                 if not auth:
                     break
                 auth = '{0}:{2}'.format(*auth)

[Julien Muchembled]

This commit shows a general problem. For example, I'm working in a Theia that uses Python 3.9 and I can't instantiate anymore old SR that are already using Python 3. I'm sure this kind of issue will happen again in the future.

For the moment, I have no other idea than bootstrapping with a dedicated Python that is as old as possible, problably 2.7 for still many years.

/cc @jerome @xavier_thompson

[Jérome Perrin]

In slapos!1207 (comment 174788) we described a similar problem that happens with networkcache ( the symptoms were that there is no error running buildout, but the generated bin/buildout does not use networkcache). For this, we had the idea of having slapos.core (or slapos-node package) include a version of buildout and slapos.libnetworkcache for bootstrap, so that we can make sure we have compatible versions. This was just a general idea, I don't know if we tried.

slapos-node package is using python3.7 (which today is also EOL) for some compatibility reason, maybe for the same problem.

[Thomas Gambier]

slapos-node package is using python3.7 (which today is also EOL) for some compatibility reason, maybe for the same problem.

Yes, indeed, this is exactly for this same problem. And as you said the solution is to bundle a version of buildout and libnetworkcache compatible with slapos-node's python.

[Julien Muchembled]

slapos-node can't have a version that's too old to run things like supervisord. More generally, we'll want the SR python of slapos-node or Theia (i.e. the python that runs buildout and any python software in the instance) to be as recent as possible, as we usually do. Not only for security reasons but also because it would be nightmare maintenance to keep a very old & full-featured version in slapos.git.

For me, the only possible minimal change is to add a second python in slapos-node that's only used for bootstrapping. It would add only networkcache (and its dependencies). We'd start with Python 2.7. It would join the minimal set of binaries that the machine is expected to provide when starting to work with slapos; the only difference with /bin/sh, gcc & al. is that most OS stop shipping very old Python so we'd have to ship it ourselves (i.e. slapos-node).

Then there's the case of Theia (or any SR that bundles a slapos node):

• it could inherit this special python from the parent slapos node if there's any, using symlinks
• or else build it, but that's not use case we currently have (I mean: build a slapos SR outside of a slapos node)