Skip to content
GitLab
Explore
Sign in
Register
nexedi
gitlab-ce
Repository
gitlab-ce
lib
gitlab
testing
robots_blocker_middleware.rb
Find file
Blame
History
Permalink
Test that project homepage is accessible by crawlers
· dbf1b5af
Thong Kuah
authored
Sep 14, 2020
We simulate this by parsing robots.txt and blocking access to the Disallow lines in robots.txt
dbf1b5af
