• Alain Takoudjou's avatar
    [feat] allow to rewrite url before download using netrc and macdef · 9db7477c
    Alain Takoudjou authored
    This change is to rewrite URL of file to download through buildout.download.Download using macdef definition in netrc file.
    The rewrite is based on regular expression, captures are substituted after optionally urlencoding them..
    
    This is how macdef definition is used to rewrite URL:
    
    .netrc:
    
    machine HOSTNAME
    login foo
    password bar
    
    macdef buildout:HOSTNAME
      REGEX_STRING
        TEMPLATE_1 HEADER1=VALUE1 HEADER2=VALUE2 ...
      ANOTHER_REGEX
        TEMPLATE_2 HEADER1=VALUE1 HEADER2=VALUE2 ...
      ...
    
    macdef ...
    
    HEADER1=VALUE1 string are optional, they are used to set the header. Similar to
    the command `curl --header "HEADER1: VALUE1" ...`. Headers can
    be repeated as it's needed.
    
    REGEX_STRING is used to match the path and query (if present) of the url we are trying to download.
    for example: the regex `/(.*)/-/raw/([\w\-]+)/(.*)` for url
    'https://lab.nexedi.com/namespace/project/-/raw/master/README.md'
    
                    /(.*)/-/raw/([\w\-]+)/(.*)
          {0}         {1}          {2}     {3}
    
    TEMPLATE is the new full url with scheme and authority (netloc). All captures are used to format
    the template and the headers. It's possible to encode string while formatting, for now only quote
    method is supported. Captured groups from REGEX_STRING start at {1}; {0} is the base url.
    
    Example of template and headers:
    
    {0}/api/v4/projects/{1.quote}/repository/files/{3}/raw?ref={2} Authentication={2}
    
    With the regex and template above, the url 'https://lab.nexedi.com/namespace/project/-/raw/master/README.md'
    is rewritten to: https://lab.nexedi.com/api/v4/projects/namespace%2Fproject/repository/files/README.md/raw?ref=master
    
    {0} is the base URL (https://lab.nexedi.com)
    {1} match 'namespace/project'
    {2} match 'master' and
    {3} match 'README.md'
    
    `namespace/project` is changed to `namespace%2Fproject` since it's encoded encoded with `quote`.
    
    Some uses cases:
    
    machine lab.nexedi.com
    login ignored
    password <ACCESS_TOKEN>
    
    macdef buildout:lab.nexedi.com
      /(.*)/-/raw/([\w\-]+)/(.*)
        {0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} PRIVATE-TOKEN=<ACCESS_TOKEN>
    
    or
    
    macdef buildout:lab.nexedi.com
      /(.*)/raw/([\w\-]+)/(.*)
        {0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} Authorization="Bearer <OAUTH-TOKEN>"
      /(.*)/-/raw/([\w\-]+)/(.*)
        {0}/api/v4/projects/{1.quote}/repository/files/{3.quote}/raw?ref={2} PRIVATE-TOKEN=<ACCESS_TOKEN>
    9db7477c
download.py 13.7 KB