X Demo how to support private raw URLs on lab.nexedi.com with pristine gitlab
... | ... | @@ -22,12 +22,13 @@ try: |
# Python 3 | ||
from urllib.error import HTTPError | ||
from urllib.request import Request, urlopen | ||
from urllib.parse import urlparse, urlunparse | ||
from urllib.parse import urlparse, urlunparse, quote, urlencode | ||
except ImportError: | ||
# Python 2 | ||
from urlparse import urlparse | ||
from urlparse import urlunparse | ||
from urllib2 import HTTPError, Request, urlopen | ||
from urllib2 import HTTPError, Request, urlopen, quote | ||
from urllib import urlencode | ||
from zc.buildout.easy_install import realpath | ||
from base64 import b64encode | ||
... | ... | @@ -284,7 +285,70 @@ class Download(object): |
if auth: | ||
return '{0}:{2}'.format(*auth), url | ||
# _labraw_authproxy rewrites URLs to raw files on lab.nexedi.com to contain | ||
# private_token fetched from netrc database. | ||
# | ||
# For example if ~/.netrc contains | ||
# | ||
# machine lab.nexedi.com/kirr/pygolang | ||
# login private_token | ||
# password XXXX | ||
# | ||
# then | ||
# | ||
# https://lab.nexedi.com/kirr/pygolang/-/raw/master/golang/libgolang.h | ||
# | ||
# is rewritten as | ||
# | ||
# https://lab.nexedi.com/api/v4/projects/kirr%2Fpygolang/repository/files/golang%2Flibgolang.h/raw/?ref=master&private_token=XXXX | ||
# | ||
# if the url is not of lab.nexedi.com/raw kind, or if no authentication is | ||
# provided, then original url is returned as is. | ||
# | ||
# NOTE gitlab accepts both projectID and project string in /projects/:id | ||
# | ||
# https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28342 | ||
# | ||
# NOTE the machine section of netrc can contain both hostname and path | ||
# which is useful to be able to use project-specific tokens | ||
# | ||
# NOTE see the following links for the details of how to fetch raw file content via gitlab API | ||
# | ||
# https://ericsysmin.com/2024/02/13/accessing-raw-files-on-authenticated-gitlab/ | ||
# https://docs.gitlab.com/ee/api/repository_files.html#get-raw-file-from-repository | ||
def _labraw_authproxy(self, url): # -> url' | ||
p = urlparse(url) | ||
if p.hostname != 'lab.nexedi.com': | ||
|
||
return url | ||
pathv = p.path.split('/') # ['', 'kirr', 'pygolang', '-', 'raw', 'master', 'golang', 'libgolang.h'] | ||
if pathv[3:5] != ['-', 'raw']: | ||
return url | ||
|
||
repo = '/'.join(pathv[1:3]) # kirr/pygolang | ||
auth = netrc.authenticators('%s/%s' % (p.hostname, repo)) # auth for lab.nexedi.com/kirr/pygolang | ||
if auth is None: | ||
return url | ||
# FIXME this does not support refs like y/bstr. | ||
# To support this we will need to do what | ||
# https://lab.nexedi.com/nexedi/gitlab-workhorse/commit/5b8cf10e | ||
# was doing - try to extract all variants for ref from longest to | ||
# shortest and stop on the first variant thay yields good result. | ||
ref = pathv[5] # master | ||
filepath = '/'.join(pathv[6:]) # golang/libgolang.h | ||
qrepo = quote(repo, '') # kirr%2Fpygolang | ||
qfilepath = quote(filepath, '') # golang%2Flibgolang.h | ||
path = '/api/v4/projects/%s/repository/files/%s/raw' % (qrepo, qfilepath) | ||
query = urlencode({'ref': ref, auth[0]: auth[2]}) | ||
return urlunparse((p.scheme, p.netloc, path, p.params, query, p.fragment)) | ||
def urlretrieve(self, url, tmp_path): | ||
url = self._labraw_authproxy(url) | ||
auth = self._auth(url) | ||
if auth: | ||
req = Request(auth[1]) | ||
... | ... |
-
This approach seems really good, I can give it a try next week, unless someone ( @alain.takoudjou ?) can also check
-
Maintainer
Thanks, Jérome.
-
-
Maintainer
You are welcome, Alain. Good luck with trying.
-
Thank you very much for this patch, Kirill. It is always a pleasure to work with you.
-
Maintainer
Nicolas, Klaus, thanks for feedback and for kind words. You are welcome. I'm glad if this little fix could be useful.
-
@kazuhiko this was also my feeling (discussed on the forum), using netrc if found and falling back on credentials from url.
But @kirr will not finalize the patch (unless you changed your mind and want to ;)) it will probably be @alain.takoudjou or me as part of our work on updating lab.nexedi.com
-
This way will be quite useful especially for configuring automated test suite for the private repository.
I was about to say that the patch only affect /raw/ get request and that automated test suite use git clone protocol which supports credentials, but I think you have a project where a private slapos profile extends another private slapos profile ( @Nicolas showed me one and we looked at it but it does not seem needed, the profiles just have xmlsec that is now in public slapos ). I was thinking we keep support for credentials in URL only for backward compatibility, with the idea that it is becoming a deprecated bad practice. In any case, it's not good to have credentials in URL, because they leak easily and it's hard to change them.
If we need to keep support, then maybe
slapos node
from testnode could also handle the netrc, but I hope we don't need it. -
( this is a public repository, if we need to discuss details we should use the forum )
-
Maintainer
@kazuhiko, thanks for asking. If needed credentials in the URL can be indeed supported and the amendment to hereby patch should be quite easy I think: first check netrc - if credentials are there, use those; if not - check credentials in the URL and, if present, use auth from there.
I believe it would be good to start from having "credentials in the URL" supported, if they are present, and emit warning, so that people know they will need to move secrets out of their URLs to external storage, but things continue to work without any breakage for some time.
As @jerome explained by default I'm not active anymore on the particular code development on hereby topic. I primarily wanted to show "there is a way" to do things with backward compatibility, but once shown and accepted - the details of this - I prefer to handover to lab people and switch back to focus on my nexedi/pygolang!21 and LTE topics.
If lab people really need help I think I should be able to help, but personally I believe for further steps Jérome and Alain can do on their own for now. Please let me know if it is not the case and we need my further involvement.
Kirill
-
mentioned in merge request nexedi/slapos.buildout!34