In today’s blog on testing tools, I wanted to focus on a tool that does something a little different than what we’ve covered before. An AI-driven tool, called Applitools Eyes that helps teams with Visual and Cross-Browser/Cross-Device testing. This is critical in this day and age of multi-device applications where we require apps to look and run perfectly across a variety of different platforms.
What is visual testing?
Before I get into the tool itself a little further though I thought it would be good to perhaps describe what visual testing is first. Visual testing is a software testing technique that focuses on verifying the graphical user interface (GUI) of an application to ensure that it appears correctly and functions as intended from a visual standpoint. Unlike traditional functional testing, which mainly examines the behavior and functionality of software components, visual testing specifically evaluates the visual aspects of an application, which include the following:
UI Appearance: Visual testing checks whether the user interface elements such as buttons, images, text, colors, fonts, layout, and spacing appear correctly on different devices, screen sizes, resolutions, and browsers.
Consistency Across Platforms: It ensures that the visual appearance remains consistent across various platforms (e.g., web browsers, mobile devices, operating systems) to provide users with a uniform experience regardless of the device they are using.
UI Responsiveness: Visual testing assesses how the UI responds to user interactions, such as mouse clicks, keyboard inputs, touch gestures, and screen resizing. It verifies that the UI elements adjust dynamically and maintain their integrity under different user actions.
Visual Regression Testing: Visual testing includes the concept of visual regression testing, which compares the current version of the UI with a baseline or reference version to detect any unintended visual changes introduced during development or code modifications. It helps prevent visual defects from being released into production.
Handling Dynamic Content: Visual testing also involves handling dynamic content and animations within the UI, ensuring that they render correctly and do not cause visual anomalies or distortions.
Cross-Browser and Cross-Device Compatibility: It verifies that the UI appears consistently across different web browsers (e.g., Chrome, Firefox, Safari, Edge) and devices (e.g., desktops, tablets, smartphones) by capturing screenshots and comparing them across various environments.
Accessibility Compliance: Visual testing may include checks for accessibility compliance, ensuring that the UI is accessible to users with disabilities and adheres to relevant accessibility standards (e.g., WCAG).
How Does it Work
Applitools Eyes can achieve this through the following uses of artificial intelligence:
Visual AI Algorithms
Applitools Eyes employs sophisticated visual AI algorithms to analyze and compare screenshots captured during testing. These algorithms can detect even subtle visual differences between baseline images and the current state of the application UI.
Intelligent Element Recognition
The tool intelligently identifies and categorizes UI elements within the application, such as buttons, text fields, images, and other components. This helps in focusing the comparison process on relevant elements and ignoring irrelevant changes, such as browser rendering discrepancies or dynamic content.
Dynamic Content Handling
Applitools Eyes' AI algorithms are trained to understand and handle dynamic content within the application UI, such as animations, pop-ups, and overlays. This ensures accurate comparison results even when elements on the page change position or appearance over time.
Layout and Structural Analysis
AI algorithms analyze the layout and structure of the UI elements to detect layout shifts, spacing inconsistencies, alignment issues, and other layout-related defects. This enables comprehensive visual validation of the application's layout across different viewport sizes and devices.
Smart Baseline Maintenance
AI-powered baseline maintenance capabilities help in automatically updating baseline images when significant changes occur in the application UI. The tool intelligently distinguishes between intentional UI updates and unintended visual defects, reducing the need for manual intervention in baseline management.
Regression Prediction
AI algorithms can predict potential regression issues based on historical testing data and patterns. This helps in proactively identifying areas of the application that are prone to visual regressions, allowing testers to focus their efforts on critical areas during testing cycles.
Its incredibly sophisticated approach to visual testing makes it easy for teams with front-end heavy applications to gain confidence in their UI’s ability to work across a wide variety of different devices and achieve compliance from a visual perspective. While, it doesn’t provide any functional coverage, which is where the majority of testing effort should remain, it does help simplify the often timeous and expensive approach to visual testing which meant companies needed to test their application against a variety of different devices and browsers through either their device farm or making use of a farm connected over the cloud.
The best part is that because it operates at a significantly faster pace it can more easily be integrated into CI/CD pipelines and executed more often for more regular and reliable results. On this latter point, you do pay for each execution, so you need to ensure it doesn’t run at all stages of pipeline execution and only just before your deployment stage so you can gain maximum effectiveness without running up unnecessary costs.
While Eyes won’t help you find those functional defects that occur on different devices, it’s those visual defects that are often the hardest to catch and require the most effort to simulate, which is what makes the tool such a time-saver for many modern development teams.
And thanks to the way it reads existing components and renders them in the appropriate frameworks and formats the devices and browsers use, its results are incredibly accurate too.
How to use Applitools Eyes
If you are interested in making use of Applitools in your testing framework, you can utilize the following steps to set it up, configure, run, and test results:
Install Applitools SDK: Start by installing the Applitools SDK for your preferred programming language and testing framework. Applitools provides SDKs for various languages, including Java, JavaScript, Python, C#, Ruby, and more. You can typically install the SDK using package managers like npm, Maven, NuGet, pip, etc.
Import Applitools Library: Once you've installed the SDK, import the Applitools library into your test scripts or application code. This allows you to access the functionalities provided by Applitools Eyes, such as capturing screenshots, performing visual validations, and managing test results.
Initialize Eyes Instance: Create an instance of the Eyes class provided by the Applitools SDK and initialize it with your API key. The API key authenticates your account and allows you to use the Applitools services for visual testing.
Open Eyes Session: Begin a visual testing session by calling the eyes.open() method and specifying the application under test, along with any desired test configuration options such as the browser type, viewport size, and other settings.
Capture Screenshots: Use the eyes.checkWindow() or eyes.checkRegion() method to capture screenshots of the application UI at different stages of your test scenarios. You can capture the entire window or specific regions of interest within the UI for visual validation.
Perform Visual Validation: After capturing screenshots, invoke the eyes.close() method to perform visual validation against a baseline image. Applitools Eyes compares the captured screenshots with the baseline and highlights any differences detected between them.
View Test Results: Once the visual validation is complete, you can view the test results and any detected differences in the Applitools dashboard. The dashboard provides detailed insights into the visual changes, including side-by-side comparisons of baseline and current images, annotations, and collaboration features for issue resolution.
Below is a list of pros and cons of using Applitools Eyes
Pros
Automated Visual Testing: Applitools Eyes automates the visual testing process, saving time and effort compared to manual inspection.
Cross-Browser and Cross-Device Testing: It allows for testing the visual appearance of applications across different browsers, devices, and screen resolutions.
Dynamic Content Handling: The tool can handle dynamic content such as animations, pop-ups, and overlays, ensuring accurate visual validation.
Intelligent AI Algorithms: Applitools Eyes employs advanced AI algorithms to accurately detect visual differences and reduce false positives.
Smart Baseline Maintenance: Baseline images are automatically updated when significant changes occur, streamlining the baseline management process.
Integration with Testing Frameworks: It seamlessly integrates with popular testing frameworks like Selenium, WebDriverIO, Cypress, and Appium, allowing for easy incorporation into existing test automation workflows.
Comprehensive Reporting: The tool generates comprehensive reports with visual evidence of bugs or discrepancies, facilitating communication and issue resolution.
Collaboration Features: Applitools Eyes provides collaboration features for team members to share test results, annotate visual differences, and collaborate on issue resolution.
Accessibility Testing: It includes features for accessibility testing, helping to ensure that the application UI complies with accessibility standards.
SaaS Solution: Applitools Eyes is offered as a Software-as-a-Service (SaaS) solution, eliminating the need for infrastructure setup and maintenance that other visual device farms may require.
CI/CD integration: Its rapid execution makes it easy to integrate directly into development pipelines for more regular feedback and regression.
Cons
Cost: While Applitools Eyes offers a free trial and various pricing plans, some users may find the cost prohibitive, especially for larger teams or enterprise-level usage.
No functional coverage: While the tool can pick up some functional errors where pages or errors render, it's not going to provide you with a wide coverage of app functionality. So teams will still need to build automated tests can check this.
Learning Curve: Users may require some time to familiarize themselves with the tool's features, APIs, and best practices for effective visual testing.
Limited Offline Testing: Since Applitools Eyes is primarily cloud-based, it may not be suitable for scenarios where testing needs to be performed offline or within restricted network environments.
Integration Complexity: While integration with popular testing frameworks is supported, configuring and setting up the integration may require some effort, especially for users unfamiliar with the respective frameworks.
When to consider choosing Applitools Eyes
You have an application that is used across many devices and browser types and needs to maintain visual fidelity throughout
You deploy frequently and need to ensure you can integrate visual testing into your CI/CD pipelines.
The cost of your app not rendering consistently is of higher value and can justify the costs of this visual regression.
Applitools has carved out a nice niche in the market with their Eyes product. They are not the only players in this AI-driven visual testing market but having been one of the early innovators, have built up an extensive array of tooling with remarkable accuracy that makes them easily the best player currently in this space. Its wide support and ease of execution further help to cement its value and even if it might be quite expensive when used widely across a large company, it can add a lot of value in terms of speed of delivery.