X Demo how to support private raw URLs on lab.nexedi.com with pristine gitlab

src/zc/buildout/download.py

@@ -22,12 +22,13 @@ try:
     # Python 3
     from urllib.error import HTTPError
     from urllib.request import Request, urlopen
-    from urllib.parse import urlparse, urlunparse
+    from urllib.parse import urlparse, urlunparse, quote, urlencode
 except ImportError:
     # Python 2
     from urlparse import urlparse
     from urlparse import urlunparse
-    from urllib2 import HTTPError, Request, urlopen
+    from urllib2 import HTTPError, Request, urlopen, quote
+    from urllib import urlencode
 
 from zc.buildout.easy_install import realpath
 from base64 import b64encode
@@ -284,7 +285,70 @@ class Download(object):
             if auth:
                 return '{0}:{2}'.format(*auth), url
 
+    # _labraw_authproxy rewrites URLs to raw files on lab.nexedi.com to contain
+    # private_token fetched from netrc database.
+    #
+    # For example if ~/.netrc contains
+    #
+    #    machine  lab.nexedi.com/kirr/pygolang
+    #    login    private_token
+    #    password XXXX
+    #
+    # then
+    #
+    #    https://lab.nexedi.com/kirr/pygolang/-/raw/master/golang/libgolang.h
+    #
+    # is rewritten as
+    #
+    #    https://lab.nexedi.com/api/v4/projects/kirr%2Fpygolang/repository/files/golang%2Flibgolang.h/raw/?ref=master&private_token=XXXX
+    #
+    # if the url is not of lab.nexedi.com/raw kind, or if no authentication is
+    # provided, then original url is returned as is.
+    #
+    # NOTE gitlab accepts both projectID and project string in /projects/:id
+    #
+    #      https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28342
+    #
+    # NOTE the machine section of netrc can contain both hostname and path
+    #      which is useful to be able to use project-specific tokens
+    #
+    # NOTE see the following links for the details of how to fetch raw file content via gitlab API
+    #
+    #      https://ericsysmin.com/2024/02/13/accessing-raw-files-on-authenticated-gitlab/
+    #      https://docs.gitlab.com/ee/api/repository_files.html#get-raw-file-from-repository
+    def _labraw_authproxy(self, url): # -> url'
+        p = urlparse(url)
+        if p.hostname != 'lab.nexedi.com':
+            return url
+        pathv = p.path.split('/')   # ['', 'kirr', 'pygolang', '-', 'raw', 'master', 'golang', 'libgolang.h']
+        if pathv[3:5] != ['-', 'raw']:
+            return url
+        repo = '/'.join(pathv[1:3]) # kirr/pygolang
+
+        auth = netrc.authenticators('%s/%s' % (p.hostname, repo))    # auth for lab.nexedi.com/kirr/pygolang
+        if auth is None:
+            return url
+
+        # FIXME this does not support refs like y/bstr.
+        # To support this we will need to do what
+        # https://lab.nexedi.com/nexedi/gitlab-workhorse/commit/5b8cf10e
+        # was doing - try to extract all variants for ref from longest to
+        # shortest and stop on the first variant thay yields good result.
+        ref  = pathv[5] # master
+
+        filepath = '/'.join(pathv[6:])  # golang/libgolang.h
+
+        qrepo     = quote(repo, '')     # kirr%2Fpygolang
+        qfilepath = quote(filepath, '') # golang%2Flibgolang.h
+
+        path  = '/api/v4/projects/%s/repository/files/%s/raw' % (qrepo, qfilepath)
+        query = urlencode({'ref': ref, auth[0]: auth[2]})
+
+        return urlunparse((p.scheme, p.netloc, path, p.params, query, p.fragment))
+
+
     def urlretrieve(self, url, tmp_path):
+        url  = self._labraw_authproxy(url)
         auth = self._auth(url)
         if auth:
             req = Request(auth[1])

[Rafael Monnerat]

I think for historical reasons, you could include 'lab.node.vifib.com', so older SR can work (no idea if this is useful or not, just saying)

However I wonder if we could not get it from environ, so we don't hardcore our lab.nexedi.com:

if p.hostname not in os.environ.get('GITLAB_COMPATIBLE_DOMAIN', 'lab.nexedi.com').split(':'):
  return url

or something similar.

Regards, Rafael Monnerat

PS: Not exactly like this but with some way to configure it.

[Kirill Smelkov]

Rafael, thanks for feedback. Sure my patch was only a demonstration and including lab.node.vifib.com and maybe also other our sites should be done. Speaking about environment - indeed the decision whether to go gitlab way or not could be made dependeing on what we see in netrc: if netrc for host/path comes with gitlab-specific private_token or similar, then gitlab mode is activated.

[Jérome Perrin]

An environment variable seems not really practical, because this happens when running non-interactively in slapos node software.

I feel that we don't even need to check the hostname, if the URL path looks like /user/repo/raw/... we can always assume it's a gitlab URL. We can do something like this:

• if path matches /user/repo/raw/*:
  • try to download using gitlab raw API
  • if it fails, fallback to downloading original URL

[Nicolas Wavrant]

Hi Jérôme,

An environment variable seems not really practical, because this happens when running non-interactively in slapos node software.

I share this feeling.

Also, I'm afraid that some other forges may use this URL pattern, and may not use the gitlab auth system. For exemple, my gitea follows this URL pattern. There may be more.

I feel that the fallback should be to gitlab auth, and not from gitlab auth. But in practice, for our use case, private downloads must happen almost exclusively from gitlab.

[Jérome Perrin]

Thanks, I was not sure in which order we should try and fallback, I wrote gitlab API first because I was thinking that this way we don't make 2 requests when we have to use gitlab API, but that sounds silly now, the typical pattern is that we have one or two extends that are protected URL and all other that are public.

My next idea was:

• try to download
• if it failed¹ and the path matches /user/repo/raw/* and we have credentials:
  • try to download using gitlab raw API

note ¹ : in the case of gitlab, an unauthorized /raw/ request is a 302 redirect response to login page . In the current production version, the redirect is on localhost ( http://localhost:8080/users/sign_in ), but in the upcoming version it is the correct login page ( https://lab.nexedi.com/users/sign_in ), served by a 200 OK response, so it's not a download error at HTTP level. I guess what we can do here is just be more clever when detecting failures, maybe checking the URL matches /user/repo/raw/* and the downloaded content looks like login page, or adjusting the redirect handler to raise an error when redirected to login page.

( cc @yusei not sure you see this thread )

[Jérome Perrin]

I think the - is still optional at this point, an URL like https://lab.nexedi.com/kirr/pygolang/raw/master/golang/libgolang.h is working and this is what we are still using.

Maybe we need to change to something like

if pathv[3:5] != ['-', 'raw'] or pathv[3:4] != ['raw']:

[Kirill Smelkov]

@jerome, thanks for feedback. Sure, this is only a draft patch illustrating the idea. It is better to support URLs without - and maybe it is also better to pass private token via headers instead of query so that credentials do not get logged somewhere accidentally. And going this way I thought that maybe there is also a way to just add a header without rewriting URL at all. This should work if unpatched gitlab supports /raw/ URLs with private cookie in the header somehow. But if not we can always stay with rewriting the URL while preserving backward compatibility for users.

My primary message with this patch is that there is a way to do things simply while preserving backward compatibility.

[Jérome Perrin]

+1 to pass private token via headers instead and thank you very much for this

[Kirill Smelkov]

Thanks, Jérome. You are welcome.

[Jérome Perrin]

This approach seems really good, I can give it a try next week, unless someone ( @alain.takoudjou ?) can also check

[Kirill Smelkov]

Thanks, Jérome.

[Alain Takoudjou]

Yes, @jerome I'll try. Thank you @kirr

[Kirill Smelkov]

You are welcome, Alain. Good luck with trying.

[Nicolas Wavrant]

Thank you very much for this patch, Kirill. It is always a pleasure to work with you.

[Klaus Wölfel]

Thank you, @kirr !

[Kirill Smelkov]

Nicolas, Klaus, thanks for feedback and for kind words. You are welcome. I'm glad if this little fix could be useful.

[Kazuhiko Shiozaki]

@kirr can you please support username/token inside URL rather than .netrc as we had before, like https://gitlab+deploy-token-x:deploy_token_here@lab.nexedi.com/... ? This way will be quite useful especially for configuring automated test suite for the private repository.

[Jérome Perrin]

@kazuhiko this was also my feeling (discussed on the forum), using netrc if found and falling back on credentials from url.

But @kirr will not finalize the patch (unless you changed your mind and want to ;)) it will probably be @alain.takoudjou or me as part of our work on updating lab.nexedi.com

[Jérome Perrin]

This way will be quite useful especially for configuring automated test suite for the private repository.

I was about to say that the patch only affect /raw/ get request and that automated test suite use git clone protocol which supports credentials, but I think you have a project where a private slapos profile extends another private slapos profile ( @Nicolas showed me one and we looked at it but it does not seem needed, the profiles just have xmlsec that is now in public slapos ). I was thinking we keep support for credentials in URL only for backward compatibility, with the idea that it is becoming a deprecated bad practice. In any case, it's not good to have credentials in URL, because they leak easily and it's hard to change them.

If we need to keep support, then maybe slapos node from testnode could also handle the netrc, but I hope we don't need it.

[Jérome Perrin]

( this is a public repository, if we need to discuss details we should use the forum )

[Kirill Smelkov]

@kazuhiko, thanks for asking. If needed credentials in the URL can be indeed supported and the amendment to hereby patch should be quite easy I think: first check netrc - if credentials are there, use those; if not - check credentials in the URL and, if present, use auth from there.

I believe it would be good to start from having "credentials in the URL" supported, if they are present, and emit warning, so that people know they will need to move secrets out of their URLs to external storage, but things continue to work without any breakage for some time.

As @jerome explained by default I'm not active anymore on the particular code development on hereby topic. I primarily wanted to show "there is a way" to do things with backward compatibility, but once shown and accepted - the details of this - I prefer to handover to lab people and switch back to focus on my nexedi/pygolang!21 and LTE topics.

If lab people really need help I think I should be able to help, but personally I believe for further steps Jérome and Alain can do on their own for now. Please let me know if it is not the case and we need my further involvement.

Kirill

[Jérome Perrin]

for reference, this is submitted in 08a21a2f