There are many possibilities for testing on Sauce Labs, and in this module we will cover the specific case of testing a desktop browser web applications using the Sauce Labs Cloud of virtual machines.

Users who have their own local test environment and code set up can start on module 3.03

You can also use the Example Test Suite and follow the next module to get set up.

The examples here uses Java code using Selenium version 3.14.15, the Maven build tool, as well as the JUnit4 test runner. The examples are on a MacOS machine, however we do provide examples of environment setup on a Windows machine.

Skills & Knowledge

In this Module you will:

This module contains a list of resources your will need to clone & use the sample test code as well as get set up including:

Video

Local Test Project Code – Selenium Java and JUnit4

Test Code

If you have your own testing suite written in Java, using the JUnit4 test runner, with capabilities set up similar to the base test here or you understand the differences between the test runner you are using and how to structure capabilities, you jump to the next section. Otherwise, use the example code in the GitHub repo.

Use GitHub Repository (Optional)

If you are familiar with using GitHub to write your code, you can also fork/ branch this repository here and run the code in 3.02 as a boilerplate:

Example Starting Folder

Required Dependencies

To run a local test as shown, you will need to set up and install the following:

If you would like step-by-step instructions to help installing the dependencies above you can use the instructions in the Selenium Java course.

Windows

Follow these instructions to install and set up a JDK, Maven, and IntelliJ on Windows 10.

MacOS

See the lesson in the Selenium Java course to see how to set up your local environment on MacOS

Run the Test Code Locally

If you are cloning a project, run the command in IntelliJ (after you have updated pom.xml):

mvn clean test

WebDriver Manager

This sample of test code is using WebDriver Manager, which handles the downloading of the browser drivers that works with the version of the browser you are working with on your machine, and pointing your tests to the correct driver.

Update pom.xml

Add your configurations into pom.xml like the project here to get all the dependencies you will need for this module.

You may need to invalidate caches and restart IntelliJ IDE again to activate the imports specified in pom.xml. You can also search for the latest Maven package versions.

Troubleshoot Project Setup

If your test isn't running, try the following to troubleshoot to get the tests running locally:

Once you understand how the test suite functions, you need to update settings such as the capabilities, endpoint to run against the Sauce Labs Cloud, and your Sauce username and access key. In this module you will gain understanding about:

Video

Setup to Run Web App Tests On Sauce Labs

About Running Tests on Sauce Labs

Capabilities

When you run tests in any evvironment, using the W3C WebDriver protocol, which Selenium does, you need to pass along information about what environment you want your tests are run in, which are known as options or capabilities. W3C has a set of capabilities you can use, and you can create your own sets of capabilities using Mutable Capabilities to create your own capabilities as well.

Other vendors, such as Chrome, Firefox, and Sauce Labs have created other pre-defined subsets of capabilities that you can set, which are useful when you need to set specific capabilities such as extensions for Chrome, or the Sauce username.

Why Sauce Labs?

In this module you are going to learn how to move the test suite that you have on your local machine onto the Sauce Labs cloud platform.

Sauce Labs maintains a set of real and virtual devices, as well as a Selenium grid that you can use to run your test in almost any environment. There are many reasons this is advantageous:

When you run tests on Sauce Labs, you are using the Selenium Grid to test on multiple operating systems and the Remote Webdriver. The Sauce Labs Selenium Grid lets you distribute test execution across several machines and you connect to it with Selenium RemoteWebDriver.

You tell the Grid which browser and OS you want your test to run on through the use of Selenium's class object, MutableCapabilities, and its various subclasses for specific browser options (ChromeOptions, FirefoxOptions, etc.) Sauce Labs has specific language bindings that act as wrappers for supported programming languages.

Setup your Sauce Labs Account

You'll need an account to use Sauce Labs. Their free trial offers enough to get you started.

Once you have your account set up, go to Account> User Settings to find your username and access key.

Sauce Labs User Name Access Key

You will need to set up your username and access key on your machine's (or CI Tools') environment variables to use them in your test.

To learn more about setting up environment variables, you can see the article here.

Video

Watch This Video to See how to set up your Sauce Credentials as environment variables on MacOS.

Setup Tests for Sauce Labs

Update Config.java

In the Config.java file, you are going to communicate the settings for our test environment with the W3C Capabilities, required for every Selenium test.

Here, you will define some variables that you can use in your Base Test code, for your tests to be able to communicate with Sauce Labs:

// filename: tests/Config.java
package tests;

public class Config {
    public static final String baseUrl = System.getProperty("baseUrl", "http://the-internet.herokuapp.com");
    public static final String host = System.getProperty("host", "saucelabs");
    public static final String browserName = System.getProperty("browserName", "chrome");
    public static final String browserVersion = System.getProperty("browserVersion", "75.0");
    public static final String platformName = System.getProperty("platformName", "Windows 10");
    public static final String sauceUser = System.getenv("SAUCE_USERNAME");
    public static final String sauceKey = System.getenv("SAUCE_ACCESS_KEY");
}

Notice the new variables you have added:

Switch the host in BaseTest.java

Now that you have the variables you will you for your capabilities, you want to create switch statement with two different cases that you can use: one for running tests on you local machine, and one for running tests on Sauce Labs.

In the BaseTest.java class, within the before() method, add a switch() method with two different host options:

// filename: tests/BaseTest.java
// ...
       switch (host) {
             case "saucelabs": {
                  //...
                  break;
              }
              case "localhost":
                  //...
                  break;
              }

Inside of the first case, "saucelabs", type in the declaration of the sauceUrl variable, and create your MutableCapabilities. We will add more to these capabilities later.

// filename: tests/BaseTest.java
// ...
    switch (host) {
        case "saucelabs": {
            String sauceUrl = "https://ondemand.us-west-1.saucelabs.com/wd/hub";
            MutableCapabilities capabilities;
        break;
    }

Finally, move the if, else if statement that checks the browserName variable inside of the "localhost" case:

// filename: tests/BaseTest.java
// ...
    case "localhost": {
         if ("firefox".equals(browserName)) {
             WebDriverManager.firefoxdriver().setup();
             driver = new FirefoxDriver();
         } else if ("chrome".equals(browserName)) {
             WebDriverManager.chromedriver().setup();
             driver = new ChromeDriver();
         }
         break;
     }

Final Code

See and example of both BaseTest.java and Config.yml in the 3.03 example.

Sauce Labs Account

Notice how many of the variables for capabilities are grey in this example, since they aren't yet used in the test code:

Capabilities for your test

Now that you have the pieces in place, such as your Sauce Labs credentials and variables to pass in as capabilities, you can run the example test suite on Sauce Labs in a few simple steps:

Video

Run a Web App Test on Sauce Labs – Selenium Java

Update your URL and Capabilities

Now you need to update BaseTest.java to work with these new values and connect to Sauce Labs. Note that these are called capabilities, and the format they are in here is compatible with Selenium WebDriver 4.0, as well as all previous Selenium versions. They set the options for setting up the environment for your tests.

// filename: tests/BaseTest.java
// ...
  @Override
        protected void before() throws Exception {
            if (host.equals("saucelabs")) {
                MutableCapabilities sauceOptions = new MutableCapabilities();
                sauceOptions.setCapability("username", sauceUser);
                sauceOptions.setCapability("accessKey", sauceKey);
                MutableCapabilities capabilities = new MutableCapabilities();
                capabilities.setCapability("browserName", browserName);
                capabilities.setCapability("browserVersion", browserVersion);
                capabilities.setCapability("platformName", platformName);
                capabilities.setCapability("sauce:options", sauceOptions);
                String sauceUrl = String.format("https://ondemand.saucelabs.com/wd/hub");
                driver = new RemoteWebDriver(new URL(sauceUrl), capabilities);
            } else if (host.equals("localhost")) {
                if (browserName.equals("firefox")) {
                    System.setProperty("webdriver.gecko.driver",
                            System.getProperty("webdriver.gecko.driver", "src/test/java/drivers/geckodriver"));
                    driver = new FirefoxDriver();
                } else if (browserName.equals("chrome")) {
                    System.setProperty("webdriver.chrome.driver", "src/test/java/drivers/chromedriver");
                    ChromeOptions browserOptions = new ChromeOptions();
                    driver = new ChromeDriver();
                }
            }
        }


This has two if/ else statements:

Now you can import the MutableCapabilities and RemoteWebDriver Selenium classes, as well as the URL java class. Add these imports in BaseTest.java:

// filename: tests/BaseTest.java
// ...
import org.openqa.selenium.MutableCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import java.net.URL;
// ...

Run Your Tests

Now you can use terminal commands to run your tests on Sauce Labs while specifying the browserName, browserVersion, and platformName. As an example, if you run this command the test will be run in Sauce Labs in on MacOS 10.10 in the Chrome 75 browser:

mvn clean test -Dhost=saucelabs -DbrowserName=chrome -DbrowserVersion=75 -Dplatform="OS X 10.10"

You should also visit http://app.saucelabs.com/. Go to the left hand menu and choose Automated → Test Results. There you will see your tests with icons indicating they were run on the operating system & browser that you chose:

Jobs Run on Sauce

Final Code

The complete code can be found here. Your final code will look like this:

Updated Sauce Labs capabilitiesUpdated imports

In this lesson you will add in the test name to make it easier to understand which tests you are viewing results for on the Sauce Labs automated web testing platform. In this lesson you will learn to:

Video

Adding a Test Name on Sauce Labs – Selenium Java

Now that your tests are up and running on the Sauce Labs platform, you'll notice it's hard to tell one apart from the other. The tests you should have run will show up as Unnamed job with a hash identifier- not easy to use for testing and debugging.

Unnamed Job

To fix this issue, you can pull information from the test and send it to Sauce Labs.

Use TestWatcher to Add a Test Name

Not having test runs that are named makes it extremely challenging to know which tests were run in each job. The TestWatcher rule will allow you gather information at the right time, and pass the test name to Sauce Labs.

In BaseTest you will use another JUnit rule called TestWatcher(). First you will need to create a string variable called testName in the BaseTest class underneath where you instantiate the driver.

// filename: tests/BaseTest.java
// ...
private String testName;
// ...

Next, before the final closing bracket of the BaseTest class, create a new @Rule, after the @Override that quits the driver. This will create a TestRule using TestWatcher that pulls the display name of the test when the test is starting, so you can pass it in as a Sauce Option.

// filename: tests/BaseTest.java
// ...
@Rule
   public TestRule watcher; {
       watcher = new TestWatcher() {
           @Override
           protected void starting (Description description) {
               testName = description.getDisplayName();
           }
       };
   }
};

Add a Sauce Capability for Test Name

Now you can add it to Sauce Options in betweenplatformNameafter the accessKey and before the list of Mutable Capabilities:

// filename: tests/BaseTest.java
// ...
sauceOptions.setCapability("name", testName);
// ...

Run Your Test

Once you have all three elements added, make sure you have imported the JUnit dependencies:

// filename: tests/BaseTest.java
// ...
import org.junit.rules.TestRule;
import org.junit.rules.TestWatcher;
import org.junit.runner.Description;
// ...

Run mvn clean test -Dhost=saucelabs in your project in terminal to see if it works. Now when you run our tests in Sauce Labs, the account dashboard will show the tests running with the name of the test, test class, and package appearing on the dashboard:

Error or Complete

Final Code

See an example of the test with a name added.

Import assets and create test name variableCreate test name variable capabilityTest watcher for test name

Right now regardless of the outcome of a test, the job in Sauce Labs will register as Complete or Error. Ideally you want to know if the job was a Pass or a Fail. That way we can tell at a glance if a test failed or not. With a couple of tweaks we can make this happen easily enough. In this lesson you will learn to:

Video

Add a Test Status on Sauce Labs – Selenium Java

Create Session ID and Sauce Rest Functionality

In the variable list of the BaseTest class (below private string testName;) add in the following to create variables to store the RemoteWebDriver SessionId and the instance of the SauceREST client:

// filename: tests/BaseTest.java
// ...
    private String sessionId;
    private SauceREST sauceClient;
// ...

Under the saucelabs driver instantiation in the before() rule instantiate a sessionId and sauceClient for when you are running tests on Sauce Labs:

// filename: tests/BaseTest.java
// ...
  case "saucelabs":}
  // ...
      driver = new RemoteWebDriver(new URL(sauceUrl), capabilities);
      sessionId = ((RemoteWebDriver) driver).getSessionId().toString();
      sauceClient = new SauceREST(sauceUser, sauceKey, DataCenter.US);

// ...

The sessionId is retrieved from the RemoteWebDriver. The sauceClient creates an instance using the Sauce Labs REST API, passing in the username, access key, and data center location. You can change the data center on your Sauce Labs homepage. Once that is changed, if you would like, go into your code and change the DataCenter option in your code to reflect this.

Data Center

Now you can import the SauceREST and DataCenter libraries in the imports list of BaseTest.java:

// filename: tests/BaseTest.java
// ...
import com.saucelabs.saucerest.SauceREST;
import com.saucelabs.saucerest.DataCenter;
// ...

Using the Sauce REST API

In order to tell Sauce Labs that your test has passed or failed, you will need to use the SauceREST API

You'll first need install the saucerest library is a part of your pom.xml file within the dependencies tags.

// filename: pom.xml
// ...
    <dependencies>
    // ...
        <dependency>
            <groupId>com.saucelabs</groupId>
            <artifactId>saucerest</artifactId>
            <version>1.0.40</version>
            <scope>test</scope>
        </dependency>
// ...

Get Pass And Fail Status with TestWatcher

Now, go down to the TestWatcher rule. Under the first @Override annotation, add in two more:

// filename: tests/BaseTest.java
// ...
  public TestRule watcher;{
    // ...
          @Override
          protected void failed(Throwable throwable, Description description) {
              if (host.equals("saucelabs")) {
                  sauceClient.jobFailed(sessionId);
                  System.out.println(String.format("https://saucelabs.com/tests/%s", sessionId));
              }
          }

          @Override
          protected void succeeded(Description description) {
              if (host.equals("saucelabs")) {
                  sauceClient.jobPassed(sessionId);
              }
          }
        };
// ...

Once a Sauce Job is started we're able to get the session ID from RemoteWebDriver and store it's string value in the sessionId variable. You then connect to the session of the Sauce REST client (which connects to the Sauce API) and use TestWatcher to send either a passed or failed status using the Sauce REST API.

With a conditional check in each @Override statement you make sure the sauceClient commands only trigger when a Sauce session has been established.

When a test is successful the succeeded() method will fire, marking the Sauce job for the test as passed. When a test fails the failed() method will trigger, and the job will be marked as failed. When there's a failure, we'll want to know the URL to view the job on Sauce Labs so you concatenate the URL and output it to the console using the System.out.println command.

Now when you run mvn clean test -Dhost=saucelabs in terminal, then check your Sauce Labs account page. On the right, you should be able to see a status of passed with each test.

Passed Tests

Final Code

See an example of the completed code to compare.

Test Rule and WatcherSession ID and SauceREST APITest WatcherSauce Rest in pom.xml

It's easy to add in a new type of browser in your Config file, however there is a caveat; each browser has other capabilities that are possible to add along with it. In this module you will:

Video

Run Selenium Java Tests on Different Browsers on Sauce Labs

Use Browser Options Capabilities

Currently in your test, you are using MutableCapabilities, which allow you to add any capability in that you want (you can even make up and add a capability that doesn't exist, or make up your own).

A better way to set these capabilities is with sets of browser options, such as SafariOptions(), FirefoxOptions(), etc.

Each browser has created a limited set of ‘capabilities' or settings that are possible use with each browser, which we are going to start using for testing on different browsers. The different capabilities we will use are:

Update Capabilities

First, you are going to remove the new Mutablecapabilities(); from the capabilities declaration since you will be declaring browser capabilities depending on which browser you are using:

// filename BaseTest.java
// ...
  @Override
          protected void before() throws Exception {
              switch (host) {
                  case "saucelabs": {
                      String sauceUrl = "https://ondemand.us-west-1.saucelabs.com/wd/hub";
                      MutableCapabilities sauceOptions = new MutableCapabilities();
                      sauceOptions.setCapability("username", sauceUser);
                      sauceOptions.setCapability("accesskey", sauceKey);
                      sauceOptions.setCapability("name", testName);
                      MutableCapabilities capabilities; //modified

Next, add in a switch statement with four cases and a default:

// filename BaseTest.java
// ...
        MutableCapabilities capabilities;
                 switch(browserName) {
                     case : {
                         capabilities =
                         break;
                     }
                     case : {
                         capabilities =
                         break;
                     }
                     case : {
                         capabilities =
                         break;
                     }
                     case : {
                         capabilities =
                         break;
                     }
                     default: {
                         capabilities =
                         break;
                     }
                 }

Now, import the BrowserType class from Selenium:

// filename BaseTest.java
// ...
import org.openqa.selenium.remote.BrowserType;

For this code, we are going to use the Selenium BrowserType interface to determine the type of browser and use the correct set of browser capabilities.

Inside each of the switch cases, add the name of the browser type, and the options for each browser, setting the default to ChromeOptions:

// filename BaseTest.java
// ...
                MutableCapabilities capabilities;
                switch(browserName) {
                    case BrowserType.SAFARI: {
                        capabilities = new SafariOptions();
                        break;
                    }
                    case BrowserType.FIREFOX: {
                        capabilities = new FirefoxOptions();
                        break;
                    }
                    case BrowserType.IE: {
                        capabilities = new InternetExplorerOptions();
                        break;
                    }
                    case BrowserType.EDGE: {
                        capabilities = new EdgeOptions();
                        break;
                    }
                    default: {
                        capabilities = new ChromeOptions();
                        break;
                    }
                }

Now, you will need to make sure you have imported the different sets of options for each browser you have set up to test on. You should already have Chrome and Firefox options since you used those to test locally. Right underneath those, import the options for the rest of the options:

// filename BaseTest.java
// ...

import org.openqa.selenium.edge.EdgeOptions;
import org.openqa.selenium.ie.InternetExplorerOptions;
import org.openqa.selenium.safari.SafariOptions;

Run Tests in Different Environments

Now, when you replace the options using the Sauce Labs Platform Configurator to find the values for browserName, browserVersion, and platformName you should be able to run in any browser & version that is supported by Sauce Labs.

This example will run on version 90.0 of the Edge browser:

// filename: tests/Config.java
package tests;

public class Config {
    // ...
    public static final String browserName = System.getProperty("browserName", "MicrosoftEdge");
    public static final String browserVersion = System.getProperty("browserVersion", "90.0");
    public static final String platformName = System.getProperty("platformName", "macOS 10.14");
   // ...

Use Maven and the -D Flag

Beacuse we have set up variables for each of the capabilities, you can use the -D flag with your mvn clean test command to run your suite of tests in a certain operating system. For each variable you want to change, add -DvariableName="variableValue" after the mvn clean test command.

This example would run your tests on Firefox Version 86.0 on a Windows 10 machine:

mvn clean test -DbrowserName="firefox" -DbrowserVersion="86.0" -DplatformName="Windows 10"   

Using these commands can be helpful if you want to use a Continuous Integration tool (such as Jenkins of Github Actions) to schedule tests to run in several environments at once.

Sauce W3C case

Learn more about setting up tests with Jenkins in the Selenium Java Course

Final Code

See an example of the completed code to compare.

Import browser optionsSwitch statement with browser optionsModifying environment with config

Your tests are currently taking a good deal of time to run since they're executing in series (i.e., one after another).

In this lesson you will be learning how to set up Sauce Labs to run tests in parallel. This means that you can run two or more tests or classes at the same time. In this lesson you will:

Video

Running Java Tests in Parallel on Sauce Labs

Parallelization is one of the main advantages to using a platform like Sauce Labs, however you also must be careful when designing a test suite to make sure the tests can be run in parallel, and in any order, or else account for and create code that does run certain tests in order. Luckily, our test suite has been well set up to run in parallel.

To run the tests in parallel, you will be using JUnit and Maven's Surefire Plugin.

Add Configuration Options to your Maven Surefire Plugin

You should already have the maven-surefire-plugin in your pom.xml file within the <build> tags. If for some reason you don't, add it in now:

// filename: pom.xml
// ...
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.18.1</version>
    </plugin>
// ...

What you will do is add in <configuration> options that will allow you to run tests in parallel. Underneath artifactID, add in a <configuration> opening and closing tag, and within that tag add in the all <parallel> parameter.

// filename: pom.xml
// ...
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
            <parallel>all</parallel>
        </configuration>
        <version>2.18.1</version>
    </plugin>

The all value tells you to run all suites, classes, and methods in parallel. You can also choose to run one or multiple of those options.

Underneath parallel, add in parameters for the number of thread counts for methods, set unlimited thread counts to true. This will allow you to use as many threads as CPUs you have available to you, and not run multiple tests in the same thread. <threadCountMethods> sets the number of methods (but not classes or suites) to be tested at once at a limit of 30. These configurations optimize the running of large test suites.

You also don't want the output for the reporting to be created in a file, otherwise you cannot run tests in parallel, so we set <redirectTestOutputToFile> to false.

// filename: pom.xml
// ...
           <threadCountMethods>30</threadCountMethods>
           <useUnlimitedThreads>true</useUnlimitedThreads>                    
           <redirectTestOutputToFile>false</redirectTestOutputToFile>
// ...

Run Parallel Tests

Before you get started, head to Sauce Labs and look under Account > User settings and check out how many tests you (and your team) can run at once.

Sauce W3C case

Once you are sure that you are able to run tests in parallel (you should have less tests than your concurrency limit), you can run your tests. If you send more jobs than your concurrency limit, Sauce Labs will queue the excess and run them as the initial batch of jobs finish.

Run mvn clean test -Dhost=saucelabs and visit Sauce Labs while your tests are running. You should see more than one test running at the same time, and notice that your test suite as a whole runs more quickly! You can see the completed code here.

Concurrent tests running

Part 3: Randomize Your Tests

You might ask, why randomization? This is a very effective way to see if your tests are truly atomic and independent of one another. As you run more and more tests, it's important to make sure that they aren't dependent on the behavior of a different test, because conditions will not always be the same as you use tests for different cases.

In your pom.xml file, below the <redirectTestOutputToFile>, add in:

// filename: pom.xml
// ...
           <runOrder>random</runOrder>
// ...

Now run your tests using mvn clean test -Dhost=saucelabs and you should see your tests run in a different order than before.

Final Code

You can see the example code herepom.xml Final code