Import erp5.utils.web_checker from svn.erp5.org:public/erp5/trunk/utils

CHANGES.erp5.util.txt

@@ -4,7 +4,10 @@ Changes
 0.3 (unreleased)
 ----------------
 
- * No changes yet
+ * Imported from https://svn.erp5.org/repos/public/erp5/trunk/utils/
+
+   - erp5.util.webchecker:
+     Utility to check caching policy of websites
 
 0.2 (2011-09-20)
 ----------------

erp5/util/CHANGES.webchecker.txt

+Changelog
+=========
+
+0.0.9 (unreleased)
+------------------
+
+0.0.8 (2010-10-19)
+------------------
+- same as 0.0.7
+  [Nicolas Delaby]
+
+0.0.7 (2010-10-19)
+------------------
+- Bug fix: typo
+
+0.0.6 (2010-10-19)
+------------------
+- Bug fix about RawConfigParser
+  [Nicolas Delaby]
+
+2010-08-13 Nicolas Delaby
+-------------------------
+small Bug fix.
+Add option to export outpout into file
+This option take precedence over mailing support.
+
+2010-08-04 Nicolas Delaby
+-------------------------
+Bug fix in status_search_in_wget_regex
+Enhance output report (fetch all errors per url)
+Add 2 options to check Urls
+  prohibited_file_name_list
+  prohibited_folder_name_list
+
+2010-08-03 Nicolas Delaby
+-------------------------
+First release

erp5/util/README.webchecker.txt

+Utility able to call wget and varnishlog to extract Headers and return all failures
+according expected caching policy.
+
+This utility is configurable through a configuration file like:
+
+[web_checker]
+url = http://www.example.com/
+working_directory = /home/me/tmp/crawled_content
+varnishlog_binary_path = varnishlog
+email_address = me@example.com
+smtp_host = localhost
+debug_level = debug
+
+[header_list]
+Last-Modified = True
+Cache-Control = max-age=300
+                max-age=3600
+Vary = Accept-Language, Cookie, Accept-Encoding
+       Accept-Language, Cookie
+       Accept-Language,Cookie,Accept-Encoding
+       Accept-Language,Cookie
+Expires = True
+
+[erp5_extension_list]
+prohibited_file_name_list = WebSection_viewAsWeb
+                            Base_viewHistory
+                            list
+prohibited_folder_name_list = web_page_module
+                              document_module
+
+
+
+with
+  url : website to check
+  working_directory : fetched data will be downloaded
+  varnishlog_binary_path :  path to varnishlog
+  email_address : email address to send result
+  smtp_host : smtp host to use
+  debug_level : log level of this utility (debug =>very verbose,
+                                          info=>normal,
+                                          warning=>nothing)
+
+  header_list : Key == Header id.
+                value: if equals to True, it means that header needs to be present in RESPONSE
+                      if it is a tuple, the Header value must sastify at least one of the proposed values
+
+  erp5_extension_list: Optional section.
+    prohibited_file_name_list: which check that any links redirect to prohibited forms
+      like WebSection_viewAsWeb, Base_viewHistory, list, ...
+    prohibited_folder_name_list: usefull to detect that links does not redirect to
+      specified modules like, web_page_module, document_module, ...
+
+
+This utility requires wget => 1.12
+And a callable varnishlog.
+The utility must be run on same server where varnish is running.
+
+web_checker reads varnishlogs to detect if a Query goes to the backend.

setup.py

@@ -61,6 +61,7 @@ setup(name=name,
           'performance_tester_erp5 = erp5.util.benchmark.performance_tester:main [benchmark]',
           'scalability_tester_erp5 = erp5.util.benchmark.scalability_tester:main [scalability_tester]',
           'generate_erp5_tester_report = erp5.util.benchmark.report:generateReport [benchmark-report]',
+          'web_checker_utility = erp5.util.webchecker:web_checker_utility [webchecker]'
         ],
       }
     )