Initial implementation of automated UI tests for TodoMVC using WebdriverJS (Selenium)

Minor documentation tweaks (+18 squashed commits)
Squashed commits:
[2ad3f9f] Added speed mode and optimist
[ee0843e] Added a small sleep for requireJS implementations
[c5fb9f7] Simplified the allTests script
[bdaab39] minor tidyup tasks
[a9a018b] Completed tests and updated documentation
[5f59664] Completed edit tests
[f6f53b3] Added a command line arg for running individual tests
[dc292a8] Added allTests that iterates over all tests cases
[0ef4410] Made double click work!
[34b9ba1] Removed unused dependencies
[bd708d9] Improved documentation
[50e77b0] Added persistence and completed items tests
[0311a0b] Added tests for the entire spec
[9a783b8] Added some words about these tests
[00bb1d4] Organising tests to match the specification
[18aa950] Added routing tests
[6ac96bc] Refactoring
[3b4b2e4] Initial test implementation

browser-tests/README.md

+#Overview
+
+The TodoMVC project has a great many implementations of exactly the same app using different MV* frameworks. The apps should be functionally identical. The goal of these tests is to provide a fully automated browser-based test that can be used to ensure that the specification is being followed by each and every TodoMVC app.
+
+##Todo
+
+ + Complete the test implementation (27 out of 28 are now complete). The only test that I am struggling with is to test that the delete button becomes visible on hover.
+ + Find a more elegant solution for TodoMVC apps that use RequireJS, currently there is a short 'sleep' statement in order to give the browser time to load dependencies. Yuck!
+ + Run JSHint over my code ;-)
+ + Find a mechanism for indicating which implementations have routing so that those that do not can be skipped.
+ + Make it work with PhantomJS. In practice, Phantom is only a little bit faster, but it should work. Currently there are a few Phantom specific failures.
+ + There are still a number of false negatives, with GWT, Dojo, Polymer and YUI being problematic. The tests are in a state where they certainly add value, but ideally they should not have any false positive or negative results.
+
+##Running the tests
+
+These tests use Selenium 2 (WebDriver), via the JavaScript API (WebdriverJS).  In order to run the tests you will need to install the dependencies. Run the following command from within the `browser-tests` folder:
+
+    npm install
+    
+The tests use mocha, which must be installed as a command line module:
+
+    sudo npm install -g mocha
+    
+You need to run a local server at the root of the TodoMVC project. On Mac OSX, you can do the following:
+
+	python -m SimpleHTTPServer 8000
+    
+To run the tests for all TodoMVC implementations, run the following:
+
+    mocha allTests.js  -R spec
+    
+Note that `-R spec` uses the mocha 'spec' reporter, which is quite informative. You can of course specify any other reported.
+    
+In order to run tests for a single TodoMVC implementation, supply a framework argument as follows:
+    
+    mocha allTests.js  -R spec --framework=angularjs
+    
+It can be useful send the results to the console and a file:
+
+    mocha allTests.js  -R spec 2>&1 | tee test.txt
+    
+Failed tests can be found using grep:
+
+	grep " [0-9]*)"  test.txt
+
+    
+###Chrome
+
+In order to run the tests using the Chrome browser, you need to install ChromeDriver. Instructions for download and installation can be found on the [ChromeDriver homepage](http://code.google.com/p/selenium/wiki/ChromeDriver), or a simpler set of instructions is available [here](http://damien.co/resources/how-to-install-chromedriver-mac-os-x-selenium-python-7406).
+
+###Example output
+
+A test run should look something like the following:
+
+    $ mocha allTests.js  -R spec --framework=angularjs
+    
+	  angularjs
+	    TodoMVC
+	      No Todos
+	        ✓ should hide #main and #footer (201ms)
+	      New Todo
+	        ✓ should allow me to add todo items (548ms)
+	        ✓ should clear text input field when an item is added (306ms)
+	        ✓ should trim text input (569ms)
+	        ✓ should show #main and #footer when items added (405ms)
+	      Mark all as completed
+	        ✓ should allow me to mark all items as completed (1040ms)
+	        ✓ should allow me to clear the completion state of all items (1014ms)
+	        ✓ complete all checkbox should update state when items are completed (1413ms)
+	      Item
+	        ✓ should allow me to mark items as complete (843ms)
+	        ✓ should allow me to un-mark items as complete (978ms)
+	        ✓ should allow me to edit an item (1155ms)
+	        ✓ should show the remove button on hover 
+	      Editing
+	        ✓ should hide other controls when editing (718ms)
+	        ✓ should save edits on enter (1093ms)
+	        ✓ should save edits on blur (1256ms)
+	        ✓ should trim entered text (1163ms)
+	        ✓ should remove the item if an empty text string was entered (1033ms)
+	        ✓ should cancel edits on escape (1115ms)
+	      Counter
+	        ✓ should display the current number of todo items (462ms)
+	      Clear completed button
+	        ✓ should display the number of completed items (873ms)
+	        ✓ should remove completed items when clicked (898ms)
+	        ✓ should be hidden when there are no items that are completed (893ms)
+	      Persistence
+	        ✓ should persist its data (3832ms)
+	      Routing
+	        ✓ should allow me to display active items (871ms)
+	        ✓ should allow me to display completed items (960ms)
+	        ✓ should allow me to display all items (1192ms)
+	        ✓ should highlight the currently applied filter (1095ms)
+	
+	
+	  27 passing (1m)
+    
+##Speed mode
+
+In order to keep each test case fully isolated, the browser is closed then re-opened in between each test. This does mean that the tests can take quite a long time to run. If you don't mind the risk of side-effects you can run the tests in speed mode by adding the `--speedMode` argument.
+
+	mocha allTests.js  -R spec --speedMode
+	
+Before each test all the todo items are checked as completed and the 'clear complete' button pressed. This make the tests run in around half the time, but with the obvious risk that the tear-down code may fail.
+
+##Test design
+
+Very briefly, the tests are designed as follows:
+
+ + `ToDoPage.js` - provides an abstraction layer for the HTML template. All the code required to access elements from the DOM is found within this file. The XPaths used to locate elements are based on the TodoMVC specification, using the required element classes / ids.
+ + `TestOperations.js` - provides common assertions and operations.
+ + `test.js` - Erm … the tests! These are written to closely match the TodoMVC spec.
+ + `allTest.js` - A simple file that locates all of the framework examples, and runs the tests for each.
+ 
+**NOTE:** All of the WebdriverJS methods return promises and are executed asynchronously. However, you do not have to 'chain' then using `then`, they are instead automagically added to a queue, then executed. This means that if you add non-WebdriverJS operations (asserts, log messages) these will not be executed at the point you might expect. This is why `TestOperations.js` uses an explicit `then` each time it asserts.
+
+	
+ 
+  
\ No newline at end of file

browser-tests/ToDoPage.js

+var webdriver = require('selenium-webdriver');
+
+function Page(browser) {
+
+    function xPathForItemAtIndex(index) {
+        // why is XPath the only language silly enough to be 1-indexed?
+        return "//ul[@id='todo-list']/li[" + (index + 1) + "]";
+    }
+
+    // ----------------- try / get methods
+
+    // unfortunately webdriver does not have a decent API for determining if an 
+    // element exists. The standard approach is to obtain an array of elements
+    // and test that the length is zero. These methods are used to obtain 
+    // elements which *might* be present in the DOM, hence the try/get name.
+
+    this.tryGetMainSectionElement = function() {
+        return browser.findElements(webdriver.By.xpath("//section[@id='main']"));
+    }
+
+    this.tryGetFooterElement = function() {
+        return browser.findElements(webdriver.By.xpath("//footer[@id='footer']"));
+    }
+
+    this.tryGetClearCompleteButton = function() {
+        return browser.findElements(webdriver.By.xpath("//button[@id='clear-completed']"));
+    }
+
+    this.tryGetToggleForItemAtIndex = function(index) {
+        return browser.findElements(webdriver.By.xpath(xPathForItemAtIndex(index) + "//input[contains(@class,'toggle')]"));
+    }
+
+    this.tryGetItemLabelAtIndex = function(index) {
+        return browser.findElements(webdriver.By.xpath(xPathForItemAtIndex(index) + "//label"));
+    }
+
+    // ----------------- DOM element access methods
+
+    this.getItemInputField = function() {
+        return browser.findElement(webdriver.By.xpath("//input[@id='new-todo']"));
+    }
+
+    this.getMarkAllCompletedCheckBox = function() {
+        return browser.findElement(webdriver.By.xpath("//input[@id='toggle-all']"));
+    }
+
+    this.getItemElements = function() {
+        return browser.findElements(webdriver.By.xpath("//ul[@id='todo-list']/li"));
+    }
+
+    this.getNonCompletedItemElements = function() {
+        return browser.findElements(webdriver.By.xpath("//ul[@id='todo-list']/li[not(contains(@class,'completed'))]"));
+    }
+
+    this.getItemsCountElement = function() {
+        return browser.findElement(webdriver.By.id("todo-count"));
+    }
+
+    this.getItemLabelAtIndex = function(index) {
+        return browser.findElement(webdriver.By.xpath(xPathForItemAtIndex(index) + "//label"));
+    }
+
+    this.getFilterElements = function() {
+        return browser.findElements(webdriver.By.xpath("//ul[@id='filters']//a"));
+    }
+
+    // ----------------- page actions
+
+    this.clickMarkAllCompletedCheckBox = function() {
+        return this.getMarkAllCompletedCheckBox().then(function(checkbox){
+            checkbox.click();
+        });
+    }
+
+    this.clickClearCompleteButton = function() {
+        return this.tryGetClearCompleteButton().then(function(elements) {
+            var button = elements[0];
+            button.click();
+        });
+    }
+
+    this.enterItem = function(itemText) {
+        var textField = this.getItemInputField();
+        textField.sendKeys(itemText);
+        textField.sendKeys(webdriver.Key.ENTER);
+    };
+
+    this.toggleItemAtIndex = function(index) {
+        return this.tryGetToggleForItemAtIndex(index).then(function(elements) {
+            var toggleElement = elements[0];
+            toggleElement.click();
+        });
+    }
+
+    this.editItemAtIndex = function(index, itemText) {
+        return browser.findElement(webdriver.By.xpath(xPathForItemAtIndex(index) + "//input[contains(@class,'edit')]"))
+            .then(function(itemEditField) {
+
+                // send 50 delete keypresses, just to be sure the item text is deleted
+                var deleteKeyPresses = "";
+                for (var i=0;i<50;i++) {
+                    deleteKeyPresses += webdriver.Key.BACK_SPACE
+                }
+                itemEditField.sendKeys(deleteKeyPresses);
+
+                // update the item with the new text.
+                itemEditField.sendKeys(itemText);
+            });
+    };
+
+    this.doubleClickItemAtIndex = function(index) {
+        return this.getItemLabelAtIndex(index).then(function(itemLabel) {
+            // double click is not 'natively' supported, so we need to send the event direct to the element
+            // see: http://stackoverflow.com/questions/3982442/selenium-2-webdriver-how-to-double-click-a-table-row-which-opens-a-new-window
+            browser.executeScript("var evt = document.createEvent('MouseEvents');" +
+                "evt.initMouseEvent('dblclick',true, true, window, 0, 0, 0, 0, 0, false, false, false, false, 0,null);" +
+                "arguments[0].dispatchEvent(evt);", itemLabel);
+
+        });
+    }
+
+    this.filterByActiveItems = function(index) {
+        return this.getFilterElements().then(function(filters) {
+            filters[1].click();
+        });
+    }
+
+    this.filterByCompletedItems = function(index) {
+        return this.getFilterElements().then(function(filters) {
+            filters[2].click();
+        });
+    }
+
+    this.filterByAllItems = function(index) {
+        return this.getFilterElements().then(function(filters) {
+            filters[0].click();
+        });
+    }
+}
+
+module.exports = Page;
\ No newline at end of file

browser-tests/allTests.js

+var testSuite = require('./test.js'),
+    fs = require('fs'),
+    argv = require('optimist').argv,
+    rootUrl = "http://localhost:8000/",
+    frameworkNamePattern = /^[a-z-_]+$/;
+
+// collect together the framework names from each of the subfolders
+var list = fs.readdirSync("../architecture-examples/")
+        .map(function(folderName) { return { name : folderName, path : "architecture-examples/" + folderName} });
+
+/*list = list.concat(fs.readdirSync("../labs/architecture-examples/")
+        .map(function(folderName) { return { name : folderName, path: "labs/architecture-examples/" + folderName} }));
+
+list = list.concat(fs.readdirSync("../labs/dependency-examples/")
+        .map(function(folderName) { return { name : folderName, path: "labs/dependency-examples/" + folderName} }));
+
+list = list.concat(fs.readdirSync("../dependency-examples/")
+        .map(function(folderName) { return { name : folderName, path: "dependency-examples/" + folderName} }));*/
+
+
+// filter out any folders that are not frameworks (.e.g  hidden files)
+list = list.filter(function(framework) { return frameworkNamePattern.test(framework.name); });
+
+// if a specific framework has been named, just run this one
+if (argv.framework) {
+    list = list.filter(function(framework) { return framework.name === argv.framework});
+}
+
+// run the tests for each framework
+var testIndex = 1;
+list.forEach(function(framework) {
+    testSuite.todoMVCTest(framework.name + " (" + testIndex++ + "/" + list.length + ")",
+        rootUrl + framework.path + "/index.html", argv.speedMode);
+});

browser-tests/package.json

+{
+  "name": "TodoMVC-Browser-Tests",
+  "description": "An automated test suite for TodoMVC",
+  "private": true,
+  "devDependencies": {
+    "grunt": "0.4.1",
+    "mocha": "1.14.0",
+    "selenium-webdriver": "2.37.0",
+    "optimist" : "0.6.0"
+  }
+}

browser-tests/testOperations.js

+var assert = require("assert");
+
+function TestOperations(page) {
+
+    // unfortunately webdriver does not have a decent API for determining if an 
+    // element exists. The standard approach is to obtain an array of elements
+    // and test that the length is zero. In this case the item is hidden if
+    // it is either not in the DOM, or is in the DOM but not visible.
+    function testIsHidden(elements) {
+        if (elements.length === 1) {
+            elements[0].isDisplayed().then(function(isDisplayed) {
+                assert(!isDisplayed);
+            })
+        }
+    }
+    function testIsVisible(elements) {
+        assert.equal(1, elements.length);
+        elements[0].isDisplayed().then(function(isDisplayed) {
+            assert(isDisplayed);
+        });
+    }
+
+    this.assertClearCompleteButtonIsHidden = function() {
+        page.tryGetClearCompleteButton().then(function(element) {
+            testIsHidden(element);
+        });
+    }
+
+    this.assertClearCompleteButtonIsVisible = function() {
+        page.tryGetClearCompleteButton().then(function(element) {
+            testIsVisible(element);
+        });
+    }
+
+    this.assertItemCount = function(itemCount) {
+        page.getItemElements().then(function(toDoItems){
+            assert.equal(itemCount, toDoItems.length);
+        });
+    }
+
+    this.assertClearCompleteButtonText = function(buttonText) {
+        page.tryGetClearCompleteButton().then(function(elements) {
+            var button = elements[0];
+            button.getText().then(function(text) {
+                assert.equal(buttonText, text);
+            });
+        });
+    }
+
+    this.assertMainSectionIsHidden = function() {
+        page.tryGetMainSectionElement().then(function(mainSection) {
+            testIsHidden(mainSection);
+        });
+    }
+
+    this.assertFooterIsHidden = function() {
+        page.tryGetFooterElement().then(function(footer) {
+            testIsHidden(footer);
+        });
+    }
+
+    this.assertMainSectionIsVisible = function() {
+        page.tryGetMainSectionElement().then(function(mainSection) {
+            testIsVisible(mainSection);
+        });
+    }
+
+    this.assertItemToggleIsHidden = function() {
+        page.tryGetToggleForItemAtIndex().then(function(toggleItem) {
+            testIsHidden(toggleItem);
+        });
+    }
+
+    this.assertItemLabelIsHidden = function() {
+        page.tryGetItemLabelAtIndex().then(function(toggleItem) {
+            testIsHidden(toggleItem);
+        });
+    }
+
+    this.assertFooterIsVisible = function() {
+        page.tryGetFooterElement().then(function(footer) {
+            testIsVisible(footer);
+        });
+    }
+
+    this.assertItemInputFieldText = function(text) {
+        page.getItemInputField().getText().then(function(inputFieldText) {
+            assert.equal(text, inputFieldText);
+        });
+    }
+
+    this.assertItemText = function(itemIndex, textToAssert) {
+        page.getItemLabelAtIndex(itemIndex).getText().then(function(text) {
+            assert.equal(textToAssert, text.trim());
+        });
+    }
+
+    this.assertItemCountText = function(textToAssert) {
+        page.getItemsCountElement().getText().then(function(text) {
+            assert.equal(textToAssert, text.trim());
+        });
+    }
+
+    // tests for the presence of the 'completed' CSS class for the item at the given index
+    this.assertItemAtIndexIsCompleted = function(index) {
+        page.getItemElements().then(function(toDoItems) {
+            toDoItems[index].getAttribute("class").then(function(cssClass) {
+                assert(cssClass.indexOf("completed") !== -1);
+            });
+        });
+    }
+
+    this.assertItemAtIndexIsNotCompleted = function(index) {
+        page.getItemElements().then(function(toDoItems) {
+            toDoItems[index].getAttribute("class").then(function(cssClass) {
+                assert(cssClass.indexOf("completed") === -1);
+            });
+        });
+    }
+
+    function isSelected(cssClass) {
+        return cssClass.indexOf("selected") !== -1;
+    }
+
+    this.assertFilterAtIndexIsSelected = function(index) {
+       page.getFilterElements().then(function(filterElements) {
+            filterElements[0].getAttribute("class").then(function(cssClass) {
+                assert(index == 0 ? isSelected(cssClass) : !isSelected(cssClass));
+            });
+
+            filterElements[1].getAttribute("class").then(function(cssClass) {
+                assert(index == 1 ? isSelected(cssClass) : !isSelected(cssClass));
+            });
+
+            filterElements[2].getAttribute("class").then(function(cssClass) {
+                assert(index == 2 ? isSelected(cssClass) : !isSelected(cssClass));
+            });
+        });
+    }
+
+    this.assertCompleteAllIsClear = function() {
+        page.getMarkAllCompletedCheckBox().then(function(markAllCompleted) {
+            markAllCompleted.isSelected().then(function(isSelected){
+                assert(!isSelected);
+            })
+        });
+    }
+
+    this.assertCompleteAllIsChecked = function() {
+        page.getMarkAllCompletedCheckBox().then(function(markAllCompleted) {
+            markAllCompleted.isSelected().then(function(isSelected){
+                assert(isSelected);
+            })
+        });
+    }
+}
+
+module.exports = TestOperations;
\ No newline at end of file