RunSauce ======== Run many kinds of simple tests with any [Sauce Labs](https://saucelabs.com) options. This is probably mostly useful for people who work at [Sauce](https://github.com/saucelabs). ## Requirements RunSauce requires Node.js version 7.6 or greater, and NPM. ## Install ```bash npm install -g runsauce ``` ## Configure RunSauce needs to know your Sauce credentials before it can do anything useful. There's a handy interactive setup that will pre-populate a config file from your environment variables if it can. ```bash # configure with your username / access key runsauce --setup ``` If you run RunSauce with `--setup` after this initial setup, you will be given the ability to add servers to your config file. This can be useful for targeting dev environments or even local environments. ## Basic Usage ```bash # run a basic test with all the default options runsauce # check out all the options runsauce --help runsauce --tests runsauce --shortcuts # run a test with some custom options, like a guinea pig page test on chrome on # linux runsauce -t web_guinea -b c -p l # do something craaazy, like run a test against a self-signed https cert # on a custom dev server (assuming you have the config set) in Mac OS X 10.9 # with Safari 7, doing this 10 times with a concurrency of 5 tests at a time runsauce -c dev -t selfsigned -p m9 -b s -v 7 -r 10 -n 5 ``` ## Running sets of tests You can also specify complex sets of tests and platforms to run. First, create a JSON file with your specification (notice the DSL you can use to restrict combinations of platforms): ``` { "tests": [ { "test": ["ios", "ios_hybrid", "ios_loc_serv"], "device": ["ip", "ipa|t=ios"], "version": ["6.1|a<=1.0", "7.0", "7.1", "8.0|a>=1.3.0", "8.1|a>=1.3.1"], "backendVersion": ["1.0", "1.1", "1.2", "1.3.4"] }, { "test": ["android_hybrid|v>=4.4", "selendroid"], "device": ["ae"], "version": ["2.3", "4.0", "4.2", "4.3", "4.4", "5.0|a>=1.2.2"], "backendVersion": ["1.0", "1.1", "1.2", "1.3.4"] } ], "config": "prod", "build": "appium-smoke-%t", "processes": 12 } ``` This will take each test set and run every possible combination of test type, device name, platform version, and Appium version (but it will filter out combinations deemed unsuitable by the restrictions---e.g., a platform version of `"6.1|a<=1.0"` means that it will run on 6.1, but only in the case where the Appium version is <= 1.0). It will run these tests against the `prod` config, and give a build name of `appium-smoke-%t` (where `%t` is replaced by a timestamp). Finally, it will run at a concurrency of 12! I can now run this "build" with the following command: ``` ./bin/runsauce -i my-build.json ``` (Note that you can also use the shortcut keys, e.g., `"t"` instead of `"test"` and `"d"` instead of `"device"` and so on). For a working example, see `demo-build.json`, included in this repo. ## Sauce Connect-based tests Some tests require you to start the Sauce Connect Proxy on your system in advance of running the test: * `connect` * `localname` Note that if running the `localname` test (which is a test of non-"localhost" local name proxying), you'll need to pass the `-l` or `--localname` option to runsauce with the 'interesting' local name (e.g., `-l MyComputer.local`) ## Runsauce as a library You can also build apps on top of runsauce. An example of this is in `lib/matrix` (made available as `./bin/appium-matrix.js`. This app reads a "build" configuration (see above), then after running the set of Appium tests puts together a 4-dimensional support matrix with stability annotations. It outputs to the command-line or as HTML. For example, let's say I have the desire to see how Appium fares across Appium versions and iOS versions with respect to the `safariIgnoreFraudWarning` capability. Then I can build a "build" configuration as follows in, say, "fraud-build.js": ``` module.exports = { config: 'prod', build: 'appium-fraud-matrix-%t', processes: 20, name: "Appium iOS fraud warning capability test" tests: { backendVersion: ['1.0.0', '1.1.0', '1.2.4', '1.3.6', '1.3.7-beta'], version: ['6.1', '7.0', '7.1', '8.0|a>=1.3.6', '8.1|a>=1.3.6', '8.2|a>=1.3.7-beta'], test: 'web_fraud', browser: 's', device: ['ip', 'ipa'] } }; ``` Now I can run the matrix app, specifying my input file and telling it to run each combination 5 times so we can generate meaningful reliability statistics: ``` ./bin/appium-matrix.js -d -r 5 -i fraud-build.js ``` Ultimately the app will output something like: ``` ┌───────────────────┬───────────────────────────────┬───────────────────────────────┬───────────────────────────────┬─────────┬───────────────────────────────┬─────────┐ │ │ iOS 6.1 │ iOS 7.0 │ iOS 7.1 │ iOS 8.0 │ iOS 8.1 │ iOS 8.2 │ ├───────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┼─────────┼───────────────────────────────┼─────────┤ │ Appium 1.0.0 │ ┌───────────┬──────┬────────┐ │ ✗ │ ✗ │ — │ — │ — │ │ │ │ 0.92 │ iPad │ iPhone │ │ │ │ │ │ │ │ │ ├───────────┼──────┼────────┤ │ │ │ │ │ │ │ │ │ web_fraud │ ✓ │ 0.83 │ │ │ │ │ │ │ │ │ └───────────┴──────┴────────┘ │ │ │ │ │ │ ├───────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┼─────────┼───────────────────────────────┼─────────┤ │ Appium 1.1.0 │ ✓ │ ✗ │ ✗ │ — │ — │ — │ ├───────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┼─────────┼───────────────────────────────┼─────────┤ │ Appium 1.2.4 │ ✓ │ ✓ │ ┌───────────┬────────┬──────┐ │ — │ — │ — │ │ │ │ │ │ 0.92 │ iPhone │ iPad │ │ │ │ │ │ │ │ │ ├───────────┼────────┼──────┤ │ │ │ │ │ │ │ │ │ web_fraud │ ✓ │ 0.83 │ │ │ │ │ │ │ │ │ └───────────┴────────┴──────┘ │ │ │ │ ├───────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┼─────────┼───────────────────────────────┼─────────┤ │ Appium 1.3.6 │ ✓ │ ┌───────────┬────────┬──────┐ │ ✓ │ ✗ │ ✓ │ — │ │ │ │ │ 0.83 │ iPhone │ iPad │ │ │ │ │ │ │ │ │ ├───────────┼────────┼──────┤ │ │ │ │ │ │ │ │ │ web_fraud │ ✓ │ 0.67 │ │ │ │ │ │ │ │ │ └───────────┴────────┴──────┘ │ │ │ │ │ ├───────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┼─────────┼───────────────────────────────┼─────────┤ │ Appium 1.3.7-beta │ ┌───────────┬────────┬──────┐ │ ┌───────────┬────────┬──────┐ │ ✓ │ ✗ │ ┌───────────┬────────┬──────┐ │ ✓ │ │ │ │ 0.92 │ iPhone │ iPad │ │ │ 0.83 │ iPhone │ iPad │ │ │ │ │ 0.92 │ iPhone │ iPad │ │ │ │ │ ├───────────┼────────┼──────┤ │ ├───────────┼────────┼──────┤ │ │ │ ├───────────┼────────┼──────┤ │ │ │ │ │ web_fraud │ ✓ │ 0.83 │ │ │ web_fraud │ 0.83 │ 0.83 │ │ │ │ │ web_fraud │ 0.83 │ ✓ │ │ │ │ │ └───────────┴────────┴──────┘ │ └───────────┴────────┴──────┘ │ │ │ └───────────┴────────┴──────┘ │ │ └───────────────────┴───────────────────────────────┴───────────────────────────────┴───────────────────────────────┴─────────┴───────────────────────────────┴─────────┘ ``` This depiction gives us a good idea of which platforms are supported and how reliable they are! Run `appium-matrix` for the list of parameters: ``` Options: -d, --detail Show detail view of failing tests [default: false] -i, --infile JSON or JS file that exports a build definition [required] [default: null] -f, --file File to store raw results [default: null] -r, --runs Number of runs for each test [default: 3] -s, --skip Skip actual test run and use results file [default: false] -h, --html Output html table instead of CLI table [default: false] ``` Since matrix builds often take a long time to run, it can be useful to dump the output into a format parseable by the tool, using the `-f` flag. Once this is done, there are two further useful options: ``` appium-matrix -i build.js -f /my/output.json -s ``` The above will re-generate the CLI report from the output file rather than actually re-running the matrix. ``` appium-matrix -i build.js -f /my/output.json -s -h > /my/output.hml ``` The above will generate a beautiful little HTML page that is much more readable especially for large matrices or ones with the `-d` option to show detailed information. (For a set of example matrix build definition files, check out `lib/matrix/builds`) ## Report results to Sumo Logic If you have a SumoLogic collector endpoint set up, Runsauce will report detailed information about test passes and failures, including timing information, which can be used for later statistical purposes. ```bash runsauce -i tests.json -j "https://endpoint1.collection.us2.sumologic.com/...." ``` ## Building from source ```bash git clone https://github.com/jlipps/runsauce.git cd runsauce npm install # run the (admittedly very few) tests npm test # configure runsauce, if you haven't already # ./bin/runsauce.js --setup # use runsauce to run a test on sauce! ./bin/runsauce.js ``` # Containerized runsauce * Build ```bash docker build -t runsauce . --no-cache ``` * Run ```bash docker run -e SAUCE_USERNAME= -e SAUCE_ACCESS_KEY= --rm -it runsauce -t web -v 11.2 -a 1.8.1 -d "iPhone Simulator" ```