Siesta launchers, reports.
=========================

This guide describes how you can run the <a href="#!/guide/getting_started_browser">Siesta browser project</a> from the command line.
This is a common requirement for the continuous integration process. It also describes which kind of reports can be generated for
the test suite results.

*Please note:* This functionality is only available in the Siesta Standard package.  


Launchers
----------

To run the browser project from the command line Siesta uses "launchers" - executable files, sharing a common API. All of them are placed
in the `bin` folder of the Siesta package. 


Common launcher API
---------

All browser project launchers accept the URL of the Siesta project HTML wrapper file, as the only required positional argument.
All other options should start with double dash: `--option=value` and are optional. 

On MacOS and Linux it looks like:

    > __SIESTA__/bin/__LAUNCHER__ http://your/siesta/project/wrapper/siesta.html [OPTIONS]
    
On Windows:

    > __SIESTA__\bin\__LAUNCHER__ http://your/siesta/project/wrapper/siesta.html [OPTIONS]
    
Here, the `__SIESTA__` placeholder is the path to your Siesta package. 

Most important, commonly supported for all launchers options are:

- `help`                - prints help message with all available options
- `max-workers`         - maximum number of parallel testing "threads" that can be opened simultaneously. 
- `headless`            - some launchers supports headless mode (browser tests are not visible and not interfering with the working desktop). 
- `simulation`          - some launchers supports native events simulation mode (real OS cursor movement). 
- `filter pattern`      - a pattern to filter the tests tree, same as in the web interface. Pattern can be:

```
WORD - any word, will match with the test file name (no directory part)
WORD | WORD - any of the alternative patterns
 
GROUP > TEST - first match GROUP with the group name, then TEST with tests inside that group
 
GROUP1 | GROUP2 > TEST1 | TEST2 - alternation support for groups too
```

For example, filtering pattern for "mouse tests in the 'unit tests' group" may look like: `unit>mou`.
This option is processed before the following two ones.  

- `include regexp`      - a regexp to match against the full urls of the test files. Only matching tests will be launched.
- `exclude regexp`      - a regexp to match against the full urls of the test files. Only non-matching tests will be launched.
- `report-format`       - the format of the report, see the "Reporting the results of a test suite execution in a structured format" section below
- `report-file`         - the file to save the report to
- `project-url`         - URL of the project file (instead of the positional argument).
- `verbose`             - will include the information about every individual assertions to the output. By default, only failed assertions will be shown
- `debug`               - will enable various debugging messages

Please also refer to the "Parallelization" section below.

In case of any failures in the test suite the command will exit with non-zero exit code. See "Exit codes" section for details.


Option groups
---------

Some command line options forms groups. Such "grouped" options starts with group name, then follows the dot, then option name. For example,
the `project.title` and `project.mouseMovePrecision` both belongs to the group `project`. Grouped options only supports assignments using
the `=` character: `project.title="Awesome test suite"`

Currently there are 2 options groups:

- `project.*` - Options in this group will override the {@link Siesta.Project#configure configuration options} of the Siesta project 
- `nyc.*` - Options in this group will be translated to the [NYC](https://github.com/istanbuljs/nyc) tool, which is used for 
[code coverage](#!/guide/code_coverage)


Selenium WebDriver launcher
---------

WebDriver launcher supports running your Siesta project in the following browsers: Chrome, Firefox, Safari, Edge, IE11. Older versions of IE will 
work too, but support for them is limited to best-effort. 

This launcher requires the presence of the actual browsers in the operation system, installed in the default locations.

Launcher executable is called `webdriver`, on MacOS and Linux it looks like:

    > __SIESTA__/bin/webdriver http://your/siesta/project/wrapper/siesta.html [--browser=chrome] [OPTIONS]
    
On Windows:

    > __SIESTA__\bin\webdriver http://your/siesta/project/wrapper/siesta.html [--browser=chrome] [OPTIONS]

The `--browser` option specifies the browser to open and should be one of: `chrome`, `firefox`, `safari`, `edge`, `ie`. Default value is `chrome`

The `--cap` option can specify various Selenium capabilities, for example `--cap browserName=firefox --cap platform=XP`,
for full list see <https://github.com/SeleniumHQ/selenium/wiki/DesiredCapabilities>

The `--headless` option is supported for Chrome and Firefox. It prevents opening of browser window and interfering
with your normal work. For other browsers you need to make sure to not use real mouse / keyboard during tests execution.

The `--simulation` option is supported, however the combination of `--simulation` and `--headless` option is only supported on Linux with Xvfb.   

**Important**. Using the WebDriver requires some manual configuration steps for IE and Safari. Please refer to these 
pages <https://github.com/SeleniumHQ/selenium/wiki/InternetExplorerDriver> and <https://github.com/SeleniumHQ/selenium/wiki/SafariDriver>. 
Also, when running test suite in IE, make sure the IE window is focused (on top of other windows) and mouse cursor is outside of the IE window.


Puppeteer launcher
---------

Puppeteer is an automation library for the headless Chrome browser. Puppeteer uses real rendering engine of the Chromium browser - 
you can trust the results from Puppeteer as if they were received from "real" Chrome.

Puppeteer is the recommended launcher for integration with your IDE. Currently its the only one, that supports all combinations of the `--max-workers`,
`--headless` and `--simulation` configuration options.

On MacOS and Linux:

    > __SIESTA__/bin/puppeteer http://your/siesta/project/wrapper/siesta.html [OPTIONS]
    
On Windows:

    > __SIESTA__\bin\puppeteer http://your/siesta/project/wrapper/siesta.html [OPTIONS]
    
Puppeteer will download a required Chrome version on the 1st launch. Further launches will be instant.


Remote Selenium WebDriver launcher
---------

You can launch the test suite (and receive the results) on one machine, but physically open the browsers on some other machine using the RemoteWebDriver server.
To do that, first start the RemoteWebDriver server on target machine:

On MacOS and Linux:

    > __SIESTA__/bin/webdriver-server [OPTIONS]
    
On Windows:

    > __SIESTA__\bin\webdriver-server [OPTIONS]
 
"webdriver-server" is a very thin wrapper around "selenium-server-standalone-xx.jar" which just specifies the location of binaries for various browsers. It bypass
any command line options to that jar file. For example, to specify the port of server (default value is 4444), specify it as:

    > __SIESTA__/bin/webdriver-server -port 4444

For a list of available options for server, launch it with "-help" switch:

    > __SIESTA__/bin/webdriver-server -help

Also, please refer to: <https://github.com/SeleniumHQ/selenium/wiki/RemoteWebDriverServer> 

Then, when launching the test suite with `webdriver` launcher, specify an additional `--host`, `--port` options. Launcher will connect to the
remote WebDriver server and opens a new browser window there:

    > __SIESTA__/bin/webdriver http://your/siesta/project/wrapper/siesta.html --host remote.webdriver.server.host --port 4444
    
The `--cap` option can be used to specify various Selenium capabilities, for example `--cap browserName=firefox --cap platform=XP`,
for full list see <https://github.com/SeleniumHQ/selenium/wiki/DesiredCapabilities>


Native events
-------------

By default Siesta uses synthetic events simulation, which is very fast. However, in some cases, you may need to test some intrinsic browser
behavior which synthetic events can not reproduce. Most notable example will be testing the `:hover` CSS styles of some DOM elements.
In such case you can use {@link Siesta.Project.Browser#simulation native events simulation}.


Parallelization
---------------

Siesta allows you to run the test suite in several parallel "threads" or "workers", speeding up the execution almost linearly by the
number of "workers". 

This can be done using the `--max-workers` option, which specifies the maximum number of "workers". 

There are some caveats though, related to focus handling. Even if several browser windows are opened simultaneously,
the focus can remain only in one of them, and all browsers "fights" for it (focus jumps randomly from one window to
another). This naturally leads to tests instability, since code often relies on some element being focused.
So, in general, the `--max-workers` can be used safely only for the non-visual tests. However see below.

- XVFB

Linux only. When using Linux with the "Xvfb" package installed, one can add special command-line switch `--xvfb`, which launch
every browser window inside the separate virtual X desktop (with own focus handling). This makes it safe to specify `--max-workers` 
more than 1. For this option to work, the `Xvfb` is expected to be in PATH, same for the browser binaries, 
`google-chrome-stable` and `firefox`. This feature does the same thing as `--headless` browser mode, plus, it can be combined
with `--simulation=native` option. 

- Headless

The headless mode of the webdriver launcher provides an isolated environment for every browser window as well. It is safe
to use several workers in this case. Note, however, that currently headless Chrome is known to be very slow on Windows 7.

- Cloud providers 

When using cloud-based infrastructure (SauceLabs or BrowserStack), each test page is running inside of the own VM, 
which guarantees the exclusive focus owning and makes it safe to use `--max-workers`. 

When value of `--max-workers` is more than 1, the order of tests execution is not defined. A test, that goes ealier
in the `project.plan()` list, can be executed after the following tests. This is simply because all tests are divided in several
"threads" and all threads are executed simultaneously. You should not rely on some test being run after another, instead, 
every test should execute standalone (allocate exclusive resources, perform all necessary setup).


Reports 
---------------------

You can export the results of a test suite execution in the structured format. To do that, provide the `--report-format` and `--report-file` 
options to the launcher. Supported formats are `HTML`, `JSON`, `JSONS` and `JUnit` (case insensitive). These options can be repeated 
several times, to generate several reports. 

The `JSON` and `JSONS` formats are almost identical, the `JSON` format contains the test results as a flat list, 
the `JSONS` contains results as a tree structure, based on the test groups information.

`HTML` report is basically a `JSONS` report, plus visualization files. It can be used to examine test suite execution results in the browser.
When using `HTML` report, the `--report-file` option actually specifies the *directory* for files (as this report consists from several files).
The `JSONS` report will be created as `report-data.json` file in that directory.

An example:

	siesta/bin/webdriver http://your/siesta/project/wrapper/siesta.html --report-format JSON --report-file report.json

And this will generate a file named `report.json` containing the following JSON structure similar to (please refer to the actual output,
as the sample below can be outdated):

    {
        "testSuiteName" : "Siesta self-hosting test suite",
        "startDate"     : 1343114314723,
        "endDate"       : 1343114315401,
        "passed"        : true,
        "testCases"     : [{
            "url"           : "010_sanity.t.js",
            "startDate"     : 1343114315390,
            "endDate"       : 1343114315396,
            "passed"        : true,
            "assertions"    : [{
                "passed"        : true,
                "description"   : "Siesta is here",
                "type"          : "Siesta.Result.Assertion"
            }, {
                "passed"        : true,
                "description"   : "Siesta.Test is here",
                "type"          : "Siesta.Result.Assertion"
            }, {
                "passed"        : true,
                "description"   : "Siesta.Project is here",
                "type"          : "Siesta.Result.Assertion"
            }, {
                "passed"        : false,
                "description"   : "Field 1 focused",
                "annotation"    : "Failed assertion `ok` at line 27 of keyevents/050_tab_key_focus2.t.js\nGot: false\nNeed \"truthy\" value",
                "sourceLine"    : "27",
                "name"          : "ok"
                "type"          : "Siesta.Result.Assertion"
            }]
        }]
    }

Using the `JUnit` report option looks like this:

	siesta/bin/webdriver http://localhost/YourApplication/tests --report-format junit --report-file report.xml

This will generate a file named `report.xml` containing the following JUnit XML structure:

	<testsuite errors="0" failures="1" hostname="localhost:8085" name="Ext Scheduler Test Suite" tests="2" time="3.594" timestamp="2012-06-06T08:55:21.520">
		
		<testcase classname="Bryntum.Test" name="lifecycle/040_schedulergrid.t.js" time="1.238">
			<failure message="Oops" type="FAIL"></failure>
		</testcase>

		<testcase classname="Bryntum.Test" name="lifecycle/042_schedulergrid_right_columns.t.js" time="0.818"></testcase>
	</testsuite>

 
Exit codes
----------

- 0 - All tests passed successfully
- 1 - Some tests failed
- 3 - No supported browsers available on this machine
- 4 - No tests to run (probably filter doesn't match any test url)
- 5 - Can't open project page
- 6 - Wrong command line arguments
- 8 - Exit after showing the Siesta version (when `--version` is provided on the command line)


Buy this product
---------

Visit our store: <https://bryntum.com/store/siesta>

Support
---------

Ask a question in our community forum: <https://www.bryntum.com/forum/viewforum.php?f=20>

Subscribers can post expedited questions in Premium Forum: <https://www.bryntum.com/forum/viewforum.php?f=21>

Please report any bugs through the web interface at <https://www.assembla.com/spaces/bryntum/support/tickets>


See also
---------

Siesta web-page: <https://bryntum.com/products/siesta>

Other Bryntum products: <https://bryntum.com/products>


COPYRIGHT AND LICENSE
---------

Copyright (c) 2009-2018, Bryntum AB & Nickolay Platonov

All rights reserved.
