ScalaWebTest has two main features.
- Handling of web requests and state
- Gauges for simple tests
The short description of the two main features is followed by a getting started section suitable for beginners and a detailed description of all features.
Handling of web requests and state
ScalaWebTest's IntegrationSpec trait helps you to structure your tests and keep them boilerplate free. It extends the most important traits, to write integration tests for web applications, from ScalaTest. It provides an easy way to configure the HtmlUnit webdriver and it takes care of login, web requests and cookies. Before executing your tests, it takes care of login, if needed. Next it requests the resource, which your test wants to verify. To keep your tests boilerplate free, the URL to the tested resource is split in 3 segments. The host and projectRoot are part of the IntegrationSettings. The last part is the path itself. In most cases a test should only work with a single resource, in this case the path doesn't change throughout the test. Per default the IntegrationSpec will, before each test, take care that the current resource in the WebDriver is the one belonging to the defined path. This behavior can be changed. Therefore your test does not have to interact with the WebDriver. Instead it can directly verify the content of the current resource via webdriver.
Gauges for simple tests
The HtmlUnit webdriver and selenium are very powerful. But writing integration tests for web application is still very cumbersome and the resulting tests are often hard to read. We think the easiest way to define the expected result of a web request is using the returned data format itself. Therefore we use pseudo HTML (let's focus on HTML for the moment, but we also support JSON) to describe the expected result. Of course a simple string comparison would be to naive. Therefore we treat the HTML (technically we require XML, because it has native support in Scala), which is used to describe the result, as gauge definition (other would call it a template). If we can find all elements from the gauge definition in the verified HTML page, it is considered valid.
Add ScalaWebTest to your project
Before you get to use all the nice features, you have to introduce ScalaWebTest to your project. All you have to do is adding the core module to your test or integration test dependencies.
Add ScalaWebTest to your SBT project
You can add ScalaWebTest to your testing dependencies in the build.sbt/scala of your sbt project as follows.
We recommend to bind the ScalaWebTest dependencies to the IntegrationTest configuration, which can be referenced with "it". The IntegrationTest configuration is not active by default. Follow the guide for SBT 1.0 or SBT 0.13 to make it part of your build. Then you can add ScalaWebTest to your project as follows.
If you want to manage the most important transitive dependencies of ScalaWebTest, you might configure it as follows (replace "it" with "test", if you do not make use of the IntegrationTest configuration. Especially selenium-java and htmlunit-driver have to align with each other.
Add ScalaWebTest to your maven project
You have to add the following dependency to your maven project to use ScalaWebTest.
ScalaWebTest is also available for Scala 2.10 and 2.11, change the version after the underscore, to get a ScalaWebTest version which is binary compatible with the Scala version which you use.
In case you want to manage the most important transitive dependencies of ScalaWebTest, you can add our bill-of-materials to your project. Just add the following to your dependencyManagement.
Write your first test
To write your first test, you have to create a scala file in src/it/scala or src/test/scala, depending on your test setup. Lets assume we want to verify our homepage. Create a class named HomepageSpec, and let it extend IntegrationFlatSpec. This uses FlatSpec style from ScalaTest, other styles are available.Try it in Scastie
Write your first gauge
This test can be rewritten making use of the HtmlGauge. Writing gauges provides a detailed introduction to this feature.Try it in Scastie
Build your base traitConfigurations, style choice and possible custom extension should be shared across your project. To do so, we recommend to create a base trait, which is extended by all your tests. ScalaWebTest uses this concept for it's own integration tests as well. This base trait uses FlatSpec style, FormBasedLogin and HtmlGauge. It configures host, loginPath and projectRoot and sets a suitable loginTimeout. Our integration tests have two purposes. Not only do we use them to test our framework, but we also use it as documentation. They are a good starting point when looking for best practice on how to make the most out of ScalaWebTest.
Selecting a style
ScalaTest provides a wide variety of testing styles. ScalaWebTest supports all available styles. As we believe the FlatSpec style is easy to read for most people, we choose it for our own documentation and most of our integration tests. There is no technical reason behind it, it is simply a matter of taste. We recommend that you extend one of the Integration*Spec/Suite classes from Styles.scala in your base trait.
Configure your tests
A reasonable default configuration is provided, which can easily be adapted. If you want to change a configuration for all your tests, we recommend to change it in your base trait, otherwise best practice is to change it in the constructor of your TestSpec. Two config objects exist, one which is used during login (if login is needed) and one which is used during test execution.
Using gauges to write your integration tests, is the core idea of ScalaWebTest. We think using org.scalatest.selenium.WebBrowser.Query, i.e CssSelectorQuery, to select elements in a HTML document and then verifying it is cumbersome and resulting tests are hard to read. Instead we copy a concept from the manufacturing industry. They build gauges (or templates), which they then lay their workpiece into. If the workpiece fits in the gauge, it satisfies the requirements. Our gauges are defined in HTML (XML to be more precise, or scala.xml.NodeSeq to be exact) instead of being forked from steel. Let's have a look at a simple example to get an idea how those gauges work.
This test would be successful, when run against, when navigation.jsp contains the following HTML
We can already, see that fits() ignored html, head and body, elements before the navigation. But following the idea from manufacturing, we should strip our gauge, from everything which we don't need. Let's assume we only want to make sure our navigation contains the correct links. We don't care about classes added for design or text contained in the links. Therefore we reduce our test to.
If we don't care about the unnumbered list, we can reduce our test even further.
The document then fits our gauge as follows. Matching parts are green, parts in grey have no influence whether the document fits or not.
<!DOCTYPE html> <html> <head> </head> <body> <nav id="mainNav"> <ul class="blue_theme"> <li> <a href="/path/to/first/element">first navigation element</a> </li> <li class="active"> <a href="/path/to/second/element">second navigation element</a> </li> </li> <li> <a href="/path/to/third/element">third navigation element</a> </li> </ul> </nav> </body> </html>
If the gauge doesn't fit, we get an error. Lets change the test to the following.
This will of course not match, because the text within the first link is wrong and belongs to the second link. See the difference highlighted in the following html block.
<!DOCTYPE html> <html> <head> </head> <body> <nav id="mainNav"> <ul class="blue_theme"> <li> <a href="/path/to/first/element">
second navigation elementfirst navigation element</a> </li> <li class="active"> <a href="/path/to/second/element">second navigation element</a> </li> </li> <li> <a href="/path/to/third/element">third navigation element</a> </li> </ul> </nav> </body> </html>
This results in the following error message from ScalaWebTest
Test for classes on elements
The class attribute is special, because it is basically an unsorted set of class attributes. Usually we don't bother, if additional classes are present in our HTML and for sure we never care about the order. ScalaWebTest therefore handles the class attribute different from the rest. Per default the attribute content, has to match exactly the one you provided in your gauge, but for classes, it only asserts, that the classes which an element contains within your gauge definition, are all present on that element in the HTML document. The following gauge therefore matches all the elements shown below.
Test single elements
Often trying to fit the complete document into the defined gauge isn't the most natural and efficient thing to do. Especially when a website contains multiple elements of the same kind, such as content cards, gallery images or items of a product list, finding all those elements first, and then trying to fit each element with the gauge, is better.
When using fits or doesntFit, the same features are available, no matter if the whole document, or a single element is checked. Therefore the following chapters apply to both usecases. There are two restrictions when using gauges for single elements (built using ElementGaugeBuilder.GaugeFromElement)
- the gauge may only contain one top level element
- the top level element of the gauge has to be the same, as the one which is checked (not one of its children)
Test for containment
It is common that you don't want to verify the complete attribute value, or text of an element. You may use @contains to trigger the ContainsMatcher instead of the DefaultMatcher. Thanks to this matcher, this gauge matches the following HTML element
The containsMatcher is also available, when matching text.
Hint: always add a space after the matcher annotation. We enforce this space to improve readability of tests.
Test for regex matches
When we want to enforce textual rules on our content, we need a more powerful Matcher. The RegexMatcher is our friend. It is triggered using @regex
This gauge will fit
but it won't fit
Negating your tests
Sometimes you want to make sure that a certain content is not shown. For example you want to make sure, that no login form is shown, when a user is logged in. Or you want to make sure that a post doesn't appear before it is made public. To do so you may use doesnt fit() or not fit(). They are synonyms. Choose whatever reads better in your current context.
In case the login form is mistakenly rendered, the following error will appear.
Testing a complete process
Lets try this out using a page with protected content. On first load a form will be shown, after logging in the form disappears, instead the text sensitive information appears.
This example test was split into three smaller tests. This is technically not necessary, but has the following advantage. When only part of the tests fails, you have a better idea what went wrong. For example, if the login fails, but the login form is correctly shown, the issue has to be your login process. But if the login form isn't shown, the problem has a very different cause. In addition the verbose output from the gauge should help you finding the root cause.
Every web application framework has its own specialities, which are relevant for testing. ScalaWebTest allows to share this framework specific logic via modules. We encourage others, to create additional modules for their most used framework.
What should be shared via modules?
We think even simple things such as default login method, ports and urls might be worth sharing. Functions to create content, update accounts, or requesting a page in debug mode, could be even better candidates for a module.
As the creators of ScalaWebTest work a lot with Adobe Experience Manager, they decided to create the first module for this CMS
To use this module extend AemTweaks from the aem module in your BaseTrait. This will cause the following.
- FormBasedLogin is activated
- loginPath is set to the default AEM 6 login path
- pages are requested with wcmmode cookie set to DISABLED
Disclaimer: This trait is currently only available in scalawebtest-aem_2.11, due to dependencies on play-json which wasn't yet released for 2.12
The PageProperties trait populates the pageProperties, and if applicable componentProperties and suffixProperties fields with a JsValue representing the properties of the currentPage, component and suffix respectively. It does so by retrieving the JSON representation of the currentPage. This works by default on all CQ/AEM author instances. In addition it provides convenience methods to access the pageProperties content.
- pageProperties(name: String) - retrieve a page property by name
- jcrContent(name: String) - retrieve a property from jcr:content by name
- findByResourceType(value: String) - search through a parsys field in the pageProperties and find all component with given resourceType
- findByProperty(name: String)(value: String) - search through a parsys field in the pageProperties and find all component with given property name and value
It populates the pageProperties field with a play.api.libs.json.JsValue, which represents the properties of the currentPage. In case the url/path points to something below jcr:content, the componentProperties will be populated with the properties of the component, and the pageProperties with those of the containing page. In case the url/path contains a suffix, the suffixProperties will be populated with the properties of the page referenced in the suffix. It does so by manipulating the url field, to request the JSON representation of the currentPage from CQ/AEM. This feature is available on CQ/AEM author instances by default. The enable.json property of the org.apache.sling.servlets.get.DefaultGetServlet of your CQ/AEM instance has to be set to true. Only extend this trait in tests which need the feature, as it otherwise unnecessarily slows down your tests, due to additional requests for page properties.
JSON is currently the most important data exchange format in the web. We think writing integration tests for web-services, which return JSON, should be as easy as for websites. Therefore we created the JsonGauge, which provides functionality comparable to the HtmlGauge. All its magic is exposed via the according builder, which provides implicit conversions for JsValue, JsLookup and functions retrieving the currentPage from the webDriver. We choose to use play-json to parse the JSON response. To keep the core module clean from additional dependencies a module was created.
Only verifying a few values
To verify the value or type of a few values, we consider standard play-json sufficient. Just use Json.parse to parse the response and then use \ and \\ to navigate the json structure.
All JSON tests in the following examples use the following JSON response
Of course this way of verifying JSON responses reaches its limitations very soon. Therefore we created JsonGauge
JsonGauge - the ScalaWebTest way
Same as when verifying HTML, we think the easiest and most natural way to formulate your expectation is using the same language, as the tested response itself. With the JSON gauge you can use JSON to specify the gauge, into which the JSON response has to fit.
Often verifying the complete response isn't ideal. We recommend to parse and traverse the document using play-json and then verify if the JsLookup or JsValue fits the defined gauge.
As with the HTML gauge, controlled variance is allowed. The original response contains additional key/value pairs (isTuringAwardWinner and universities). Also the order of the name and firstName is the other way around. Nevertheless this test would succeed, when run. It would report an error, if the response would:
- contain a different value for one of the keys
- miss a key, which is contained in the gauge
- contain a different hierarchy of the key/value pairs
Behaviors are a good way to share tests. When sharing test via behaviors, verifying values is often too specific. Therefore we provide fits types of in addition to fits values of.
Now the values no longer matter. They are only used to determine the expected type. Therefore we recommend to use values, which indicate that only the type matters. Use empty strings and 0. This test would succeed. It would fail, when the response would:
- contain a value with a type different from the one expected for the given key
- miss a key, which is contained in the gauge
- contain a different hierarchy of the key/value pairs
- when an array value would contain an element with a mismatching type
Lets have a closer look at arrays. Usually we expect all array elements to match certain expectations. When using fits types of, you only have to define those expectations once. All array elements then have to fulfill them. Lets have a look at an example. Lets assume the test of the previous example would be replaced with the following
This would enforce all elements of the universities array to contain a key name with a value of type string and the keys begin and end with a value of type number. The elements might contain additional keys, such as department, but no contradictions. Sometimes this is not the behavior one is looking for. Two more options exist to verify arrays. One common scenario is, that the size of the array matters, i.e. coordinates in a GeoJson document. To test this use fits typesAndArraySizes of. In this case every array element is considered unique, you have to provide specific expectations for every element, but those expectations are allowed to differ. For example, we expect Martin to have studied and worked at three universities. For the first two an end is known. This isn't the case for the last one.
One more common situation exists with arrays. Often the array elements are not sorted. Although we expect the array to contain a specific element, we don't know it's position. For this situation containsElementFitting values/types/typesAndArraySizes of exists.
Because of its simplicity we prefer play-json over Argonaut and Circe. A good alternative to our JSON module is cornichon. Cornichon requires your tests to be a bit more specific then the ScalaWebTest ones, but it provides nice ways to do so. As cornichon uses Akka Http instead of HtmlUnit to retrieve the JSON response, it is only partially compatible with ScalaWebTest. They might coexist in the same codebase and both are based on ScalaTest, but cornichon can't make much use of the boilerplate reduction provided by our IntegrationSpec.
One of the goals for ScalaWebTest is to make it as as easy as possible, to extend it. Therefore the foreseen extension points are a part of the documentation.
Create a custom matcher
One of the core extension points for gauge testing, are the matchers. ScalaWebTest uses two types of matchers, one for text and one for attributes. You may create your own TextMatcher or AttributeMatcher by extending those traits. You can then add the your own matcher, by prepending to the list of textMatchers or attributesMatchers in the Matchers object.
Here is a full example, of how to implement your own Matcher
Create a custom login
If the available login mechanisms don't work with your web application. No worries, simply create your own implementation of the Login interface and extend it in your base trait. Ideally you then contribute it back to the project via pull request.
The FormBasedLogin provides a good example on how to implement your custom login
Create your own module
If you implement multiple web applications with the same framework, you might want to create a framework specific module. This allows to share framework specific testing code among multiple projects. You can start using it within your company/project and share it later with the rest of the community by creating a pull request. If you provide your own module to ScalaWebTest, we ask you to help maintaining it.
The IntegrationSpec is the base trait, which all testing style specific traits extend. When writing a spec you should not extend IntegrationSpec, but one of the testing style specific traits, such as IntegrationFlatSpec
The base trait extends the following traits
- Webbrowser - Selenium DSL for ScalaTest
- Suite - encapsulates a conceptual suite of tests
- BeforeAndAfterEach - provides beforeEach and afterEach test hooks
- BeforeAndAfterAll - provides beforeAll and AfterAll tests hooks
- WebClientExposingHtmlUnit - wrapper for HtmlUnitDriver, which exposes the WebClient
- IntegrationSettings - a set of fields to configure IntegrationSpec
- Eventually - provides eventually function
There is a complete set of test style specific base traits. Choose a testing style and then extend the according style specific Integration_Spec trait to create your test.
This traits extend the following traits
- _Spec i.e. FlatSpec - provides the testing style specific DSL
- IntegrationSpec - integrates ScalaTest and Selenium with additional features from ScalaWebTest
- Matchers - provides DSL for assertions using the word should
- Inspectors - provides nestable inspector methods that enable assertions to be made about collections
One way to share assertions between testing suites, is to use behavior. A behavior contains a set of assertions. You can then use X behaves like SomeBehavior within the suite. Behavior and Suite have to use the same testing style. Therefore ScalaWebTest builds its style specific base traits on the style specific behavior traits. The following traits, which are part of the Integration_Spec, are in fact inherited from the behavior trait
- _Spec i.e. FlatSpec
The Gauge provides the methods fit, fits, doesnt fit and not fit. All of them are used to verify if the current webpage matches the given gauge specification. To do so, the gauge, searches for elements, as defined by the specification, in the current webpage. Elements are found by element name and containing classes. All elements which fulfill the search criteria, are considered candidates. Candidates are then verified for correct attributes and textcontent, using Matchers, next they are verified by checking whether their children match. As soon as something doesn't match a Misfit is reported. If no candidate matches all criteria, the gauge doesn't fit. While verifying the candidates, a list of Misfits was acquired, the gauge will only report the most specific of all Misfits.
A Misfit is a container for an error message, when something didn't fit a given Matcher. It contains an error message and a relevance. The deeper the current check in the gauge specification is, the higher its relevance. In the end only the Misfits with the highest relevance will be part of the error message.
When working with gauges for testing, Matchers are used to test attributes and text content of elements. By default the following Matchers are available.
- Default Matcher: tests whether an attribute or text exactly matches a given String
- Contains Matcher: tests whether an attribute or text contains a given String
- Regex Matcher: tests whether an attribute or text matches the given Regex
All Matchers, which are available by default, can be used for attributes and text. When creating a new Matcher, one is not forced to implement for both. For each matcher type a specific trait exists. Extend AttributeMatcher or TextMatcher or both, when creating your own Matcher
When creating a custom Matcher the most important thing is to provide a detailed Some(Misfit) in case the Matcher doesn't match. In case of a match, the Matcher simply returns None.
The Login trait is very simple. It only defines what the username and password field should be called. It is used to mark specific implementations as Login and to assert that username and password are consistent over all implementations. Depending on the authentication process for which a Login trait was implemented, it might make sense to overwrite the login function from IntegrationSpec.
ScalaWebTest uses a custom web driver, more precisely a wrapper for the HtmlUnitDriver. Thanks to this wrapper, we can expose the WebClient to enable more control over the WebClient. We also rely on the HtmlUnitDriver, because it has the option to compare the order of html elements within the document. This is not available with all WebDriver implementations, but crucial for the implementation of Gauge.
The WebDriverConfigFixtures provide the option to execute a closure with a specific configuration. After the closure is executed, the WebDriver configuration will be reverted to what it was before the call of the fixture.