Merge branch '217486-gitlab-s-robots-txt-file-is-out-of-date' into 'master'

Resolve "GitLab's robots.txt file is out of date"

Closes #217486

See merge request gitlab-org/gitlab!31782

public/robots.txt

@@ -35,6 +35,7 @@ Disallow: /snippets/*/raw
 User-Agent: *
 Disallow: /*/*.git
 Disallow: /*/*/fork/new
+Disallow: /*/-/archive/
 Disallow: /*/*/repository/archive*
 Disallow: /*/*/activity
 Disallow: /*/*/new