Merge pull request #9693 from bbodenmiller/patch-1

allow crawling of commit page but not patch/diffs

Merge pull request #9693 from bbodenmiller/patch-1
allow crawling of commit page but not patch/diffs
94405891 · Stan Hu · 1115caf8 · abf18abb · 94405891
Commit 94405891 authored Oct 08, 2015 by Stan Hu
Show whitespace changes
Inline Side-by-side

Showing with 2 additions and 1 deletion

public/robots.txt public/robots.txt +2 -1

No files found.
--- a/public/robots.txt
+++ b/public/robots.txt
@@ -38,7 +38,8 @@ Disallow: /*/*/edit
 Disallow: /*/*/raw
 Disallow: /*/*/blame
 Disallow: /*/*/commits/*/*
-Disallow: /*/*/commit
+Disallow: /*/*/commit/*.patch
+Disallow: /*/*/commit/*.diff
 Disallow: /*/*/compare
 Disallow: /*/*/branches/new
 Disallow: /*/*/tags/new