The Testing, Tools, and Frameworks Zone encapsulates one of the final stages of the SDLC as it ensures that your application and/or environment is ready for deployment. From walking you through the tools and frameworks tailored to your specific development needs to leveraging testing practices to evaluate and verify that your product or application does what it is required to do, this Zone covers everything you need to set yourself up for success.
Test Parameterization With JUnit 5.7: A Deep Dive Into @EnumSource
Black Box Tester in Python
Ensuring application reliability is a never-ending quest. Finite state machines (FSMs) offer a solution by modeling system behavior as states and transitions, a useful tool that can help software engineers understand software behavior and design effective test cases. This article explores the pros and cons of FSMs via simple examples. We will also make a short comparison between the usefulness and applicability of FSMs and program graphs in software testing. What Are FSMs? FSMs are a powerful tool used to model systems that exhibit distinct states and transitions between those states. They are our visual roadmaps for a system's behavior. Here's a breakdown of their core principles: An FSM is a directed graph where nodes represent states and edges represent transitions between states. Transitions are triggered by events, and actions might occur upon entering or leaving a state. Labels on transitions specify the events that trigger them and the actions that occur during the transition. FSMs are a simple and visual way to represent systems that react differently to various events. Let's explore Python code for a simple vending machine and demonstrate how an FSM aids in designing effective test cases. Python class VendingMachine: def __init__(self): self.state = "idle" self.inserted_amount = 0 self.product_selected = None def insert_coin(self, amount): if self.state == "idle": self.inserted_amount += amount print(f"Inserted ${amount}. Current amount: ${self.inserted_amount}") else: print("Machine busy, please wait.") def select_product(self, product): if self.state == "idle" and self.inserted_amount >= product.price: self.state = "product_selected" self.product_selected = product print(f"Selected {product.name}.") else: if self.state != "idle": print("Please dispense product or return coins first.") else: print(f"Insufficient funds for {product.name}.") def dispense_product(self): if self.state == "product_selected": print(f"Dispensing {self.selected_product.name}.") self.state = "idle" self.inserted_amount = 0 self.product_selected = None else: print("No product selected.") def return_coins(self): if self.state == "idle" and self.inserted_amount > 0: print(f"Returning ${self.inserted_amount}.") self.inserted_amount = 0 else: print("No coins to return.") # Example products class Product: def __init__(self, name, price): self.name = name self.price = price product1 = Product("Soda", 1.00) product2 = Product("Chips", 0.75) # Example usage vending_machine = VendingMachine() vending_machine.insert_coin(1.00) vending_machine.select_product(product1) vending_machine.dispense_product() vending_machine.insert_coin(0.50) vending_machine.select_product(product2) vending_machine.dispense_product() vending_machine.return_coins() The code simulates a basic vending machine with functionalities like coin insertion, product selection, dispensing, and coin return. Let's see how an FSM empowers us to create robust test cases. FSM Design for the Vending Machine The vending machine's FSM may have four states: Idle: The initial state where the machine awaits user input Coin insertion: State active when the user inserts coins Product selection: State active after a product is selected with sufficient funds Dispensing: State active when the product is dispensed and change (if any) is returned Transitions and Events Idle -> Coin Insertion: Triggered by the insert_coin method Coin Insertion -> Idle: Triggered if the user tries to insert coins while not in the "idle" state (error scenario) Idle -> Product Selection: Triggered by the select_product method if sufficient funds are available Product Selection -> Idle: Triggered if the user selects a product without enough funds or attempts another action while a product is selected Product Selection -> Dispensing: Triggered by the dispense_product method Dispensing -> Idle: Final state reached after dispensing the product and returning change Test Case Generation With FSM By analyzing the FSM, we can design comprehensive test cases to thoroughly test the program: 1. Valid Coin Insertion and Product Selection Insert various coin denominations (valid and invalid amounts). Select products with exact, sufficient, and insufficient funds. Verify the machine transitions to the correct states based on inserted amounts and selections. Example Test Case: Start in "Idle" state. Insert $1.00 (transition to "Coin Insertion"). Select "Soda" (transition to "Product Selection" if funds are sufficient, otherwise remain in "Idle"). Verify the message: "Selected Soda." Insert $0.25 (transition to "Coin Insertion"). Select "Chips" (transition to "Product Selection" if the total amount is sufficient; otherwise, remain in "Product Selection"). Verify the message: "Dispensing Chips." or "Insufficient funds for Chips." (depending on the previous coin insertion). Expected behavior: The machine should dispense "Chips" if the total amount is $1.25 (enough for product and change) and return the remaining $0.25. If the total amount is still insufficient, it should remain in the "Product Selection" state. 2. Edge Case Testing Insert coins while in "Product Selection" or "Dispensing" state (unexpected behavior). Try to select a product before inserting any coins. Attempt to dispense the product without selecting one. Return coins when no coins are inserted. Verify the machine handles these scenarios gracefully and provides appropriate messages or prevents invalid actions. Example Test Case: Start in "Idle" state. Insert $1.00 (transition to "Coin Insertion"). Select "Soda" (transition to "Product Selection"). Try to insert another coin (should not allow in "Product Selection"). Verify the message: "Machine busy, please wait." Expected behavior: The machine should not accept additional coins while a product is selected. 3. State Transition Testing Verify the program transitions between states correctly based on user actions (inserting coins, selecting products, dispensing, returning coins). Use the FSM as a reference to track the expected state transitions throughout different test cases. Benefits of FSMs FSMs provide a clear understanding of the expected system behavior for different events. They aid in defining and documenting requirements. By mapping the FSM, testers can efficiently design test cases that cover all possible transitions and ensure the system reacts appropriately to various scenarios. FSMs can help identify inconsistencies or missing logic in the early design stages. This prevents costly bugs later in the development process. They act as a bridge between technical and non-technical stakeholders, facilitating better communication and collaboration during testing. But let's see some examples: Clear Requirements Specification A tech startup was developing a revolutionary smart building management system. Their latest challenge was to build an app that controls a sophisticated elevator. The team, led by an enthusiastic project manager, Sofia, was facing a communication breakdown. "The engineers keep changing the app's behavior!" Sofia exclaimed during a team meeting. "One minute it prioritizes express calls, the next it services all floors. Clients are confused, and we're behind schedule." David, the lead software engineer, scratched his head. "We all understand the core functionality, but translating those requirements into code is proving tricky." Aisha, the new UI/UX designer, piped up, "Maybe we need a more visual way to represent the elevator's behavior. Something everyone can understand from a glance." Sofia pulled out a whiteboard. "What if we create an FSM for our app?" The team huddled around as Sofia sketched a diagram. The FSM depicted the elevator's different states (Idle, Moving Up, Moving Down, Door Open) and the events (button press, floor sensor activation) that triggered transitions between them. It also defined clear outputs (door opening, floor announcement) for each state. "This is amazing!" David exclaimed. "It clarifies the decision-making process for the elevator's control system." Aisha smiled. "This FSM can guide the user interface design as well. We can show users the elevator's current state and expected behavior based on their input." Over the next few days, the team refined the FSM, ensuring all user scenarios and edge cases were accounted for. They used the FSM as a reference point for coding, UI design, and even client presentations. The results were impressive. Their app functioned flawlessly, prioritizing express calls during peak hours and servicing all floors efficiently. The clear user interface, based on the FSM, kept everyone informed of the elevator's current state. "The FSM was a game-changer," Sofia declared during a successful client demo. "It provided a shared understanding of the system's behavior, leading to smooth development and a happy client." The success of the app served as a testament to the power of FSMs. By providing a clear visual representation of a system's behavior, FSMs bridge communication gaps, ensure well-defined requirements, and can lead to the development of robust and user-friendly software. Test Case Generation Another startup was working on an AI-powered security gate for restricted areas. The gate-controlled access is based on employee ID badges and clearance levels. However, the testing phase became a frustrating maze of random scenarios. "These bugs are popping up everywhere!" groaned Mike, the lead QA tester. "One minute the gate opens for a valid ID, the next it denies access for no reason." Lisa, the lead developer, frowned. "We've written tons of test cases, but these glitches keep slipping through." New to the team, Alex, a recent computer science graduate, listened intently. "Have you guys considered using an FSM?" Mike asked, "Finite State Machine? What's that?" Alex explained how an FSM could visually represent the app's behavior. It would show various states (Idle, Verifying ID, Access Granted, Access Denied) and the events (badge swipe, clearance check) triggering transitions. "By mapping the FSM," Alex continued, "we can systematically design test cases that cover all possible transitions and ensure that our app reacts appropriately in each scenario." The team decided to give it a try. Together, they sketched an FSM on a whiteboard. It detailed all possible badge swipes (valid ID, invalid ID, revoked ID) and corresponding state transitions and outputs (gate opening, access denied messages, security alerts). Based on the FSM, Mike and Alex designed comprehensive test cases. They tested valid access for different clearance levels, attempted access with invalid badges, and even simulated revoked IDs. They also included edge cases, like simultaneous badge swipes or network disruptions during the verification process. The results were remarkable. The FSM helped them identify and fix bugs they hadn't anticipated before. For instance, they discovered a logic error that caused the app to grant access even when the ID was revoked. "This FSM is a lifesaver!" Mike exclaimed. "It's like a roadmap that ensures we test every possible pathway through the system." Lisa nodded, relieved. "With these comprehensive tests, we can finally be confident about our app's reliability." The team learned a valuable lesson: FSMs aren't just theoretical tools, but powerful allies in the software testing battleground. Early Error Detection Another development team was building a VoIP app. It promised crystal-clear voice calls over the internet, but the development process had become a cacophony of frustration. "The call quality keeps dropping!" Mary, the lead developer, grimaced. "One minute the audio is clear, the next it's a mess." Jason, the stressed project manager, pinched the bridge of his nose. "We've been fixing bugs after each test run, but it feels like a game of whack-a-mole with these audio issues." Anna, the new UI/UX designer, suggested, "Maybe we need a more structured approach to visualizing how our VoIP app should behave. Something that exposes potential glitches before coding begins." Mark remembered a concept from his first-year computer science degree. "What about a Finite State Machine (FSM)?" The team gathered around the whiteboard as Mark sketched a diagram. The FSM depicted the app's various states (Idle, Initiating Call, Connected, In-Call) and the user actions (dialing, answering, hanging up) triggering transitions. It also defined expected outputs (ringing tones, voice connection, call-ended messages) for each state. "This is amazing!" Anna exclaimed. "By mapping out the flow, we can identify potential weaknesses in the logic before they cause audio problems down the line." Over the next few days, the team painstakingly detailed the FSM. They identified a crucial gap in the logic early on. The initial design didn't account for varying internet connection strengths. This could explain the erratic call quality that Mary described. With the FSM as a guide, Alex, the network engineer, refined the app's ability to adapt to different bandwidths. The app dynamically adjusted audio compression levels based on the user's internet speed. This ensured a smoother call experience even with fluctuating connections. The FSM unveiled another potential issue: the lack of a clear "call dropped" state. This could lead to confusion for users if the connection abruptly ended without any notification. Based on this insight, the team designed an informative "call ended" message triggered by unexpected connection loss. By launch day, the VoIP app performed flawlessly. The FSM helped them catch critical glitches in the early stages, preventing user frustration and potential churn. Improved Communication Another development team was building a mobile banking app. It promised cutting-edge security and user-friendly features. However, communication between the development team and stakeholders had become a financial nightmare of misunderstandings. "Marketing wants a flashy login animation," Nick, the lead developer, sighed. "But it might conflict with the two-factor authentication process." Joe, the project manager, rubbed his temples. "And the CEO keeps asking about facial recognition, but it's not in the current design." John, the intern brimming with enthusiasm, chimed in, "Have you considered using a Finite State Machine (FSM) to model our app?" John explained how an FSM could visually represent the app's flow. It would show different states (Idle, Login, Account Selection, Transaction Confirmation) with user actions (entering credentials, selecting accounts, confirming transfers) triggering transitions. "The beauty of an FSM," John continued, "is that it provides a clear and concise picture for everyone involved. Technical and non-technical stakeholders can readily understand the app's intended behavior." The team decided to give it a shot. Together, they sketched an FSM for the app, detailing every step of the user journey. This included the two-factor authentication process and its interaction with the login animation. It was now clear to marketing that a flashy animation might disrupt security protocols. The FSM became a communication bridge. Joe presented it to the CEO, who easily grasped the limitations of facial recognition integration in the current design phase. The FSM helped prioritize features and ensure everyone was on the same page. Testers also benefited immensely. The FSM served as a roadmap, guiding them through various user scenarios and potential edge cases. They could systematically test each state transition and identify inconsistencies in the app's behavior. By launch time, the app functioned flawlessly. The FSM facilitated clear communication, leading to a well-designed and secure banking app. Stakeholders were happy, the development team was relieved, and John, the hero with his FSM knowledge, became a valuable asset to the team. The team's key takeaway: FSMs are not just for internal development. They can bridge communication gaps and ensure smooth collaboration between technical and non-technical stakeholders throughout the software development lifecycle. FSMs vs. Program Graphs: A Comparison While both FSMs and program graphs are valuable tools for software testing, they differ in their scope and level of detail. To understand how both tools can be related, the following analogy may help. Assume we are exploring a city. An FSM would be like a map with labeled districts (states) and connecting roads (transitions). A program graph would be like a detailed subway map, depicting every station (code blocks), tunnels (control flow), and potential transfers (decision points). FSMs: What They Are Suitable for Testing State-driven systems: User interfaces, network protocols, and apps with a clear mapping between states and events Functional testing: Verifying system behavior based on user inputs and expected outputs in different states Regression testing: Ensuring changes haven't affected existing state transitions and system functionality FSM Weaknesses Limited scope: FSMs may struggle with complex systems that exhibit continuous behavior or have complex interactions between states. State explosion: As the system complexity increases, the number of states and transitions can grow exponentially, making the FSM cumbersome and difficult to manage. Limited error handling: FSMs don't explicitly represent error states or handling mechanisms, which might require separate testing approaches. Program Graphs: What They Are Suitable for Testing Software with complex logic: Code with loops, branches, functions, and intricate interactions between different parts of the program Integration testing: Verifying how different modules or components interact with each other Unit testing: Focusing on specific code functions and ensuring they execute as expected under various conditions Program Graphs' Weaknesses Complexity: Creating and interpreting program graphs can be challenging for testers unfamiliar with code structure and control flow. Abstract view: Program graphs offer a less intuitive representation for non-technical stakeholders compared to FSMs. State abstraction: Complex state changes might not be explicitly represented in program graphs, requiring additional effort to map them back to the system's states. Choosing the Right Tool For state-based systems with clear events and transitions, FSMs are a great starting point, offering simplicity and ease of use. For more complex systems or those with intricate control flow logic, program graphs provide a more detailed and comprehensive view, enabling thorough testing. In many cases, a combination of FSMs and program graphs might be the most effective approach. FSMs can provide a high-level overview of system behavior, and program graphs can delve deeper into specific areas of code complexity. By understanding the strengths and limitations of each approach, you can choose the best tool for your specific software testing needs. Wrapping Up FSMs are a useful tool for software development to represent a system's behavior. They excel at clearly defining requirements, ensuring all parties involved understand the expected functionality. FSMs also guide test case generation, making sure all possible scenarios are explored. Most importantly, FSMs help catch inconsistencies and missing logic early in the development phase, preventing costly bugs from appearing later. Understanding their pros and cons can help us improve our testing efforts. After all, we can use FSMs alone or in parallel with other tools like program graphs.
Unit testing has become a standard part of development. Many tools can be utilized for it in many different ways. This article demonstrates a couple of hints or, let's say, best practices working well for me. In This Article, You Will Learn How to write clean and readable unit tests with JUnit and Assert frameworks How to avoid false positive tests in some cases What to avoid when writing unit tests Don't Overuse NPE Checks We all tend to avoid NullPointerException as much as possible in the main code because it can lead to ugly consequences. I believe our main concern is not to avoid NPE in tests. Our goal is to verify the behavior of a tested component in a clean, readable, and reliable way. Bad Practice Many times in the past, I've used isNotNull assertion even when it wasn't needed, like in the example below: Java @Test public void getMessage() { assertThat(service).isNotNull(); assertThat(service.getMessage()).isEqualTo("Hello world!"); } This test produces errors like this: Plain Text java.lang.AssertionError: Expecting actual not to be null at com.github.aha.poc.junit.spring.StandardSpringTest.test(StandardSpringTest.java:19) Good Practice Even though the additional isNotNull assertion is not really harmful, it should be avoided due to the following reasons: It doesn't add any additional value. It's just more code to read and maintain. The test fails anyway when service is null and we see the real root cause of the failure. The test still fulfills its purpose. The produced error message is even better with the AssertJ assertion. See the modified test assertion below. Java @Test public void getMessage() { assertThat(service.getMessage()).isEqualTo("Hello world!"); } The modified test produces an error like this: Java java.lang.NullPointerException: Cannot invoke "com.github.aha.poc.junit.spring.HelloService.getMessage()" because "this.service" is null at com.github.aha.poc.junit.spring.StandardSpringTest.test(StandardSpringTest.java:19) Note: The example can be found in SimpleSpringTest. Assert Values and Not the Result From time to time, we write a correct test, but in a "bad" way. It means the test works exactly as intended and verifies our component, but the failure isn't providing enough information. Therefore, our goal is to assert the value and not the comparison result. Bad Practice Let's see a couple of such bad tests: Java // #1 assertThat(argument.contains("o")).isTrue(); // #2 var result = "Welcome to JDK 10"; assertThat(result instanceof String).isTrue(); // #3 assertThat("".isBlank()).isTrue(); // #4 Optional<Method> testMethod = testInfo.getTestMethod(); assertThat(testMethod.isPresent()).isTrue(); Some errors from the tests above are shown below. Plain Text #1 Expecting value to be true but was false at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) at com.github.aha.poc.junit5.params.SimpleParamTests.stringTest(SimpleParamTests.java:23) #3 Expecting value to be true but was false at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) at com.github.aha.poc.junit5.ConditionalTests.checkJdk11Feature(ConditionalTests.java:50) Good Practice The solution is quite easy with AssertJ and its fluent API. All the cases mentioned above can be easily rewritten as: Java // #1 assertThat(argument).contains("o"); // #2 assertThat(result).isInstanceOf(String.class); // #3 assertThat("").isBlank(); // #4 assertThat(testMethod).isPresent(); The very same errors as mentioned before provide more value now. Plain Text #1 Expecting actual: "Hello" to contain: "f" at com.github.aha.poc.junit5.params.SimpleParamTests.stringTest(SimpleParamTests.java:23) #3 Expecting blank but was: "a" at com.github.aha.poc.junit5.ConditionalTests.checkJdk11Feature(ConditionalTests.java:50) Note: The example can be found in SimpleParamTests. Group-Related Assertions Together The assertion chaining and a related code indentation help a lot in the test clarity and readability. Bad Practice As we write a test, we can end up with the correct, but less readable test. Let's imagine a test where we want to find countries and do these checks: Count the found countries. Assert the first entry with several values. Such tests can look like this example: Java @Test void listCountries() { List<Country> result = ...; assertThat(result).hasSize(5); var country = result.get(0); assertThat(country.getName()).isEqualTo("Spain"); assertThat(country.getCities().stream().map(City::getName)).contains("Barcelona"); } Good Practice Even though the previous test is correct, we should improve the readability a lot by grouping the related assertions together (lines 9-11). The goal here is to assert result once and write many chained assertions as needed. See the modified version below. Java @Test void listCountries() { List<Country> result = ...; assertThat(result) .hasSize(5) .singleElement() .satisfies(c -> { assertThat(c.getName()).isEqualTo("Spain"); assertThat(c.getCities().stream().map(City::getName)).contains("Barcelona"); }); } Note: The example can be found in CountryRepositoryOtherTests. Prevent False Positive Successful Test When any assertion method with the ThrowingConsumer argument is used, then the argument has to contain assertThat in the consumer as well. Otherwise, the test would pass all the time - even when the comparison fails, which means the wrong test. The test fails only when an assertion throws a RuntimeException or AssertionError exception. I guess it's clear, but it's easy to forget about it and write the wrong test. It happens to me from time to time. Bad Practice Let's imagine we have a couple of country codes and we want to verify that every code satisfies some condition. In our dummy case, we want to assert that every country code contains "a" character. As you can see, it's nonsense: we have codes in uppercase, but we aren't applying case insensitivity in the assertion. Java @Test void assertValues() throws Exception { var countryCodes = List.of("CZ", "AT", "CA"); assertThat( countryCodes ) .hasSize(3) .allSatisfy(countryCode -> countryCode.contains("a")); } Surprisingly, our test passed successfully. Good Practice As mentioned at the beginning of this section, our test can be corrected easily with additional assertThat in the consumer (line 7). The correct test should be like this: Java @Test void assertValues() throws Exception { var countryCodes = List.of("CZ", "AT", "CA"); assertThat( countryCodes ) .hasSize(3) .allSatisfy(countryCode -> assertThat( countryCode ).containsIgnoringCase("a")); } Now the test fails as expected with the correct error message. Plain Text java.lang.AssertionError: Expecting all elements of: ["CZ", "AT", "CA"] to satisfy given requirements, but these elements did not: "CZ" error: Expecting actual: "CZ" to contain: "a" (ignoring case) at com.github.aha.sat.core.clr.AppleTest.assertValues(AppleTest.java:45) Chain Assertions The last hint is not really the practice, but rather the recommendation. The AssertJ fluent API should be utilized in order to create more readable tests. Non-Chaining Assertions Let's consider listLogs test, whose purpose is to test the logging of a component. The goal here is to check: Asserted number of collected logs Assert existence of DEBUG and INFO log message Java @Test void listLogs() throws Exception { ListAppender<ILoggingEvent> logAppender = ...; assertThat( logAppender.list ).hasSize(2); assertThat( logAppender.list ).anySatisfy(logEntry -> { assertThat( logEntry.getLevel() ).isEqualTo(DEBUG); assertThat( logEntry.getFormattedMessage() ).startsWith("Initializing Apple"); }); assertThat( logAppender.list ).anySatisfy(logEntry -> { assertThat( logEntry.getLevel() ).isEqualTo(INFO); assertThat( logEntry.getFormattedMessage() ).isEqualTo("Here's Apple runner" ); }); } Chaining Assertions With the mentioned fluent API and the chaining, we can change the test this way: Java @Test void listLogs() throws Exception { ListAppender<ILoggingEvent> logAppender = ...; assertThat( logAppender.list ) .hasSize(2) .anySatisfy(logEntry -> { assertThat( logEntry.getLevel() ).isEqualTo(DEBUG); assertThat( logEntry.getFormattedMessage() ).startsWith("Initializing Apple"); }) .anySatisfy(logEntry -> { assertThat( logEntry.getLevel() ).isEqualTo(INFO); assertThat( logEntry.getFormattedMessage() ).isEqualTo("Here's Apple runner" ); }); } Note: the example can be found in AppleTest. Summary and Source Code The AssertJ framework provides a lot of help with their fluent API. In this article, several tips and hints were presented in order to produce clearer and more reliable tests. Please be aware that most of these recommendations are subjective. It depends on personal preferences and code style. The used source code can be found in my repositories: spring-advanced-training junit-poc
We don't usually think of Git as a debugging tool. Surprisingly, Git shines not just as a version control system, but also as a potent debugging ally when dealing with the tricky matter of regressions. The Essence of Debugging with Git Before we tap into the advanced aspects of git bisect, it's essential to understand its foundational premise. Git is known for tracking changes and managing code history, but the git bisect tool is a hidden gem for regression detection. Regressions are distinct from generic bugs. They signify a backward step in functionality—where something that once worked flawlessly now fails. Pinpointing the exact change causing a regression can be akin to finding a needle in a haystack, particularly in extensive codebases with long commit histories. Traditionally, developers would employ a manual, binary search strategy—checking out different versions, testing them, and narrowing down the search scope. This method, while effective, is painstakingly slow and error-prone. Git bisect automates this search, transforming what used to be a marathon into a swift sprint. Setting the Stage for Debugging Imagine you're working on a project, and recent reports indicate a newly introduced bug affecting the functionality of a feature that previously worked flawlessly. You suspect a regression but are unsure which commit introduced the issue among the hundreds made since the last stable version. Initiating Bisect Mode To start, you'll enter bisect mode in your terminal within the project's Git repository: git bisect start This command signals Git to prepare for the bisect process. Marking the Known Good Revision Next, you identify a commit where the feature functioned correctly, often a commit tagged with a release number or dated before the issue was reported. Mark this commit as "good": git bisect good a1b2c3d Here, a1b2c3d represents the hash of the known good commit. Marking the Known Bad Revision Similarly, you mark the current version or a specific commit where the bug is present as "bad": git bisect bad z9y8x7w z9y8x7w is the hash of the bad commit, typically the latest commit in the repository where the issue is observed. Bisecting To Find the Culprit Upon marking the good and bad commits, Git automatically jumps to a commit roughly in the middle of the two and waits for you to test this revision. After testing (manually or with a script), you inform Git of the result: If the issue is present: git bisect bad If the issue is not present: git bisect good Git then continues to narrow down the range, selecting a new commit to test based on your feedback. Expected Output After several iterations, Git will isolate the problematic commit, displaying a message similar to: Bisecting: 0 revisions left to test after this (roughly 3 steps) [abcdef1234567890] Commit message of the problematic commit Reset and Analysis Once the offending commit is identified, you conclude the bisect session to return your repository to its initial state: git bisect reset Notice that bisect isn't linear. Bisect doesn't scan through the revisions in a sequential manner. Based on the good and bad markers, Git automatically selects a commit approximately in the middle of the range for testing (e.g., commit #6 in the following diagram). This is where the non-linear, binary search pattern starts, as Git divides the search space in half instead of examining each commit sequentially. This means fewer revisions get scanned and the process is faster. Advanced Usage and Tips The magic of git bisect lies in its ability to automate the binary search algorithm within your repository, systematically halving the search space until the rogue commit is identified. Git bisect offers a powerful avenue for debugging, especially for identifying regressions in a complex codebase. To elevate your use of this tool, consider delving into more advanced techniques and strategies. These tips not only enhance your debugging efficiency but also provide practical solutions to common challenges encountered during the bisecting process. Script Automation for Precision and Efficiency Automating the bisect process with a script is a game-changer, significantly reducing manual effort and minimizing the risk of human error. This script should ideally perform a quick test that directly targets the regression, returning an exit code based on the test's outcome. Example Imagine you're debugging a regression where a web application's login feature breaks. You could write a script that attempts to log in using a test account and checks if the login succeeds. The script might look something like this in a simplified form: #!/bin/bash # Attempt to log in and check for success if curl -s http://yourapplication/login -d "username=test&password=test" | grep -q "Welcome"; then exit 0 # Login succeeded, mark this commit as good else exit 1 # Login failed, mark this commit as bad fi By passing this script to git bisect run, Git automatically executes it at each step of the bisect process, effectively automating the regression hunt. Handling Flaky Tests With Strategy Flaky tests, which sometimes pass and sometimes fail under the same conditions, can complicate the bisecting process. To mitigate this, your automation script can include logic to rerun tests a certain number of times or to apply more sophisticated checks to differentiate between a true regression and a flaky failure. Example Suppose you have a test that's known to be flaky. You could adjust your script to run the test multiple times, considering the commit "bad" only if the test fails consistently: #!/bin/bash # Run the flaky test three times success_count=0 for i in {1..3}; do if ./run_flaky_test.sh; then ((success_count++)) fi done # If the test succeeds twice or more, consider it a pass if [ "$success_count" -ge 2 ]; then exit 0 else exit 1 fi This approach reduces the chances that a flaky test will lead to incorrect bisect results. Skipping Commits With Care Sometimes, you'll encounter commits that cannot be tested due to reasons like broken builds or incomplete features. git bisect skip is invaluable here, allowing you to bypass these commits. However, use this command judiciously to ensure it doesn't obscure the true source of the regression. Example If you know that commits related to database migrations temporarily break the application, you can skip testing those commits. During the bisect session, when Git lands on a commit you wish to skip, you would manually issue: git bisect skip This tells Git to exclude the current commit from the search and adjust its calculations accordingly. It's essential to only skip commits when absolutely necessary, as skipping too many can interfere with the accuracy of the bisect process. These advanced strategies enhance the utility of git bisect in your debugging toolkit. By automating the regression testing process, handling flaky tests intelligently, and knowing when to skip untestable commits, you can make the most out of git bisect for efficient and accurate debugging. Remember, the goal is not just to find the commit where the regression was introduced but to do so in the most time-efficient manner possible. With these tips and examples, you're well-equipped to tackle even the most elusive regressions in your projects. Unraveling a Regression Mystery In the past, we got to use git bisect when working on a large-scale web application. After a routine update, users began reporting a critical feature failure: the application's payment gateway stopped processing transactions correctly, leading to a significant business impact. We knew the feature worked in the last release but had no idea which of the hundreds of recent commits introduced the bug. Manually testing each commit was out of the question due to time constraints and the complexity of the setup required for each test. Enter git bisect. The team started by identifying a "good" commit where the payment gateway functioned correctly and a "bad" commit where the issue was observed. We then crafted a simple test script that would simulate a transaction and check if it succeeded. By running git bisect start, followed by marking the known good and bad commits, and executing the script with git bisect run, we set off on an automated process that identified the faulty commit. Git efficiently navigated through the commits, automatically running the test script on each step. In a matter of minutes, git bisect pinpointed the culprit: a seemingly innocuous change to the transaction logging mechanism that inadvertently broke the payment processing logic. Armed with this knowledge, we reverted the problematic change, restoring the payment gateway's functionality and averting further business disruption. This experience not only resolved the immediate issue but also transformed our approach to debugging, making git bisect a go-to tool in our arsenal. Final Word The story of the payment gateway regression is just one example of how git bisect can be a lifesaver in the complex world of software development. By automating the tedious process of regression hunting, git bisect not only saves precious time but also brings a high degree of precision to the debugging process. As developers continue to navigate the challenges of maintaining and improving complex codebases, tools like git bisect underscore the importance of leveraging technology to work smarter, not harder. Whether you're dealing with a mysterious regression or simply want to refine your debugging strategies, git bisect offers a powerful, yet underappreciated, solution to swiftly and accurately identify the source of regressions. Remember, the next time you're faced with a regression, git bisect might just be the debugging partner you need to uncover the truth hidden within your commit history. Video
A sign of a good understanding of a programming language is not whether one is simply knowledgeable about the language’s functionality, but why such functionality exists. Without knowing this “why," the developer runs the risk of using functionality in situations where its use might not be ideal - or even should be avoided in its entirety! The case in point for this article is the lateinit keyword in Kotlin. Its presence in the programming language is more or less a way to resolve what would otherwise be contradictory goals for Kotlin: Maintain compatibility with existing Java code and make it easy to transcribe from Java to Kotlin. If Kotlin were too dissimilar to Java - and if the interaction between Kotlin and Java code bases were too much of a hassle - then adoption of the language might have never taken off. Prevent developers from declaring class members without explicitly declaring their value, either directly or via constructors. In Java, doing so will assign a default value, and this leaves non-primitives - which are assigned a null value - at the risk of provoking a NullPointerException if they are accessed without a value being provided beforehand. The problem here is this: what happens when it’s impossible to declare a class field’s value immediately? Take, for example, the extension model in the JUnit 5 testing framework. Extensions are a tool for creating reusable code that conducts setup and cleanup actions before and after the execution of each or all tests. Below is an example of an extension whose purpose is to clear out all designated database tables after the execution of each test via a Spring bean that serves as the database interface: Java public class DBExtension implements BeforeAllCallback, AfterEachCallback { private NamedParameterJdbcOperations jdbcOperations; @Override public void beforeAll(ExtensionContext extensionContext) { jdbcOperations = SpringExtension.getApplicationContext(extensionContext) .getBean(NamedParameterJdbcTemplate.class); clearDB(); } @Override public void afterEach(ExtensionContext extensionContext) throws Exception { clearDB(); } private void clearDB() { Stream.of("table_one", "table_two", "table_three").forEach((tableName) -> jdbcOperations.update("TRUNCATE " + tableName, new MapSqlParameterSource()) ); } } (NOTE: Yes, using the @Transactional annotation is possible for tests using Spring Boot tests that conduct database transactions, but some use cases make automated transaction roll-backs impossible; for example, when a separate thread is spawned to execute the code for the database interactions.) Given that the field jdbcOperations relies on the Spring framework loading the proper database interface bean when the application is loaded, it cannot be assigned any substantial value upon declaration. Thus, it receives an implicit default value of null until the beforeAll() function is executed. As described above, this approach is forbidden in Kotlin, so the developer has three options: Declare jdbcOperations as var, assign a garbage value to it in its declaration, then assign the “real” value to the field in beforeAll(): Kotlin class DBExtension : BeforeAllCallback, AfterEachCallback { private var jdbcOperations: NamedParameterJdbcOperations = StubJdbcOperations() override fun beforeAll(extensionContext: ExtensionContext) { jdbcOperations = SpringExtension.getApplicationContext(extensionContext) .getBean(NamedParameterJdbcOperations::class.java) clearDB() } override fun afterEach(extensionContext: ExtensionContext) { clearDB() } private fun clearDB() { listOf("table_one", "table_two", "table_three").forEach { tableName: String -> jdbcOperations.update("TRUNCATE $tableName", MapSqlParameterSource()) } } } The downside here is that there’s no check for whether the field has been assigned the “real” value, running the risk of invalid behavior when the field is accessed if the “real” value hasn’t been assigned for whatever reason. 2. Declare jdbcOperations as nullable and assign null to the field, after which the field will be assigned its “real” value in beforeAll(): Kotlin class DBExtension : BeforeAllCallback, AfterEachCallback { private var jdbcOperations: NamedParameterJdbcOperations? = null override fun beforeAll(extensionContext: ExtensionContext) { jdbcOperations = SpringExtension.getApplicationContext(extensionContext) .getBean(NamedParameterJdbcOperations::class.java) clearDB() } override fun afterEach(extensionContext: ExtensionContext) { clearDB() } private fun clearDB() { listOf("table_one", "table_two", "table_three").forEach { tableName: String -> jdbcOperations!!.update("TRUNCATE $tableName", MapSqlParameterSource()) } } } The downside here is that declaring the field as nullable is permanent; there’s no mechanism to declare a type as nullable “only” until its value has been assigned elsewhere. Thus, this approach forces the developer to force the non-nullable conversion whenever accessing the field, in this case using the double-bang (i.e. !!) operator to access the field’s update() function. 3. Utilize the lateinit keyword to postpone a value assignment to jdbcOperations until the execution of the beforeAll() function: Kotlin class DBExtension : BeforeAllCallback, AfterEachCallback { private lateinit var jdbcOperations: NamedParameterJdbcOperations override fun beforeAll(extensionContext: ExtensionContext) { jdbcOperations = SpringExtension.getApplicationContext(extensionContext) .getBean(NamedParameterJdbcOperations::class.java) clearDB() } override fun afterEach(extensionContext: ExtensionContext) { clearDB() } private fun clearDB() { listOf("table_one", "table_two", "table_three").forEach { tableName: String -> jdbcOperations.update("TRUNCATE $tableName", MapSqlParameterSource()) } } } No more worrying about silently invalid behavior or being forced to “de-nullify” the field each time it’s being accessed! The “catch” is that there’s still no compile-time mechanism for determining whether the field has been accessed before it’s been assigned a value - it’s done at run-time, as can be seen when decompiling the clearDB() function: Java private final void clearDB() { Iterable $this$forEach$iv = (Iterable)CollectionsKt.listOf(new String[]{"table_one", "table_two", "table_three"}); int $i$f$forEach = false; NamedParameterJdbcOperations var10000; String tableName; for(Iterator var3 = $this$forEach$iv.iterator(); var3.hasNext(); var10000.update("TRUNCATE " + tableName, (SqlParameterSource)(new MapSqlParameterSource()))) { Object element$iv = var3.next(); tableName = (String)element$iv; int var6 = false; var10000 = this.jdbcOperations; if (var10000 == null) { Intrinsics.throwUninitializedPropertyAccessException("jdbcOperations"); } } } Not ideal, considering what’s arguably Kotlin’s star feature (compile-time checking of variable nullability to reduce the likelihood of the “Billion-Dollar Mistake”) - but again, it’s a “least-worst” compromise to bridge the gap between Kotlin code and the Java-based code that provides no alternatives that adhere to Kotlin’s design philosophy. Use Wisely! Aside from the above-mentioned issue of conducting null checks only at run-time instead of compile-time, lateinit possesses a few more drawbacks: A field that uses lateinit cannot be an immutable val, as its value is being assigned at some point after the field’s declaration, so the field is exposed to the risk of inadvertently being modified at some point by an unwitting developer and causing logic errors. Because the field is not instantiated upon declaration, any other fields that rely on this field - be it via some function call to the field or passing it in as an argument to a constructor - cannot be instantiated upon declaration as well. This makes lateinit a bit of a “viral” feature: using it on field A forces other fields that rely on field A to use lateinit as well. Given that this mutability of lateinit fields goes against another one of Kotlin’s guiding principles - make fields and variables immutable where possible (for example, function arguments are completely immutable) to avoid logic errors by mutating a field/variable that shouldn’t have been changed - its use should be restricted to where no alternatives exist. Unfortunately, several code patterns that are prevalent in Spring Boot and Mockito - and likely elsewhere, but that’s outside the scope of this article - were built on Java’s tendency to permit uninstantiated field declarations. This is where the ease of transcribing Java code to Kotlin code becomes a double-edged sword: it’s easy to simply move the Java code over to a Kotlin file, slap the lateinit keyword on a field that hasn’t been directly instantiated in the Java code, and call it a day. Take, for instance, a test class that: Auto-wires a bean that’s been registered in the Spring Boot component ecosystem Injects a configuration value that’s been loaded in the Spring Boot environment Mocks a field’s value and then passes said mock into another field’s object Creates an argument captor for validating arguments that are passed to specified functions during the execution of one or more test cases Instantiates a mocked version of a bean that has been registered in the Spring Boot component ecosystem and passes it to a field in the test class Here is the code for all of these points put together: Kotlin @SpringBootTest @ExtendWith(MockitoExtension::class) @AutoConfigureMockMvc class FooTest { @Autowired private lateinit var mockMvc: MockMvc @Value("\${foo.value}") private lateinit var fooValue: String @Mock private lateinit var otherFooRepo: OtherFooRepo @InjectMocks private lateinit var otherFooService: OtherFooService @Captor private lateinit var timestampCaptor: ArgumentCaptor<Long> @MockBean private lateinit var fooRepo: FooRepo // Tests below } A better world is possible! Here are ways to avoid each of these constructs so that one can write “good” idiomatic Kotlin code while still retaining the use of auto wiring, object mocking, and argument capturing in the tests. Becoming “Punctual” Note: The code in these examples uses Java 17, Kotlin 1.9.21, Spring Boot 3.2.0, and Mockito 5.7.0. @Autowired/@Value Both of these constructs originate in the historic practice of having Spring Boot inject the values for the fields in question after their containing class has been initialized. This practice has since been deprecated in favor of declaring the values that are to be injected into the fields as arguments for the class’s constructor. For example, this code follows the old practice: Kotlin @Service class FooService { @Autowired private lateinit var fooRepo: FooRepo @Value("\${foo.value}") private lateinit var fooValue: String } It can be updated to this code: Kotlin @Service class FooService( private val fooRepo: FooRepo, @Value("\${foo.value}") private val fooValue: String, ) { } Note that aside from being able to use the val keyword, the @Autowired annotation can be removed from the declaration of fooRepo as well, as the Spring Boot injection mechanism is smart enough to recognize that fooRepo refers to a bean that can be instantiated and passed in automatically. Omitting the @Autowired annotation isn’t possible for testing code: test files aren't actually a part of the Spring Boot component ecosystem, and thus, won’t know by default that they need to rely on the auto-wired resource injection system - but otherwise, the pattern is the same: Kotlin @SpringBootTest @ExtendWith(MockitoExtension::class) @AutoConfigureMockMvc class FooTest( @Autowired private val mockMvc: MockMvc, @Value("\${foo.value}") private val fooValue: String, ) { @Mock private lateinit var otherFooRepo: OtherFooRepo @InjectMocks private lateinit var otherFooService: OtherFooService @Captor private lateinit var timestampCaptor: ArgumentCaptor<Long> @MockBean private lateinit var fooRepo: FooRepo // Tests below } @Mock/@InjectMocks The Mockito extension for JUnit allows a developer to declare a mock object and leave the actual mock instantiation and resetting of the mock’s behavior - as well as injecting these mocks into the dependent objects like otherFooService in the example code - to the code within MockitoExtension. Aside from the disadvantages mentioned above about being forced to use mutable objects, it poses quite a bit of “magic” around the lifecycle of the mocked objects that can be easily avoided by directly instantiating and manipulating the behavior of said objects: Kotlin @SpringBootTest @ExtendWith(MockitoExtension::class) @AutoConfigureMockMvc class FooTest( @Autowired private val mockMvc: MockMvc, @Value("\${foo.value}") private val fooValue: String, ) { private val otherFooRepo: OtherFooRepo = mock() private val otherFooService = OtherFooService(otherFooRepo) @Captor private lateinit var timestampCaptor: ArgumentCaptor<Long> @MockBean private lateinit var fooRepo: FooRepo @AfterEach fun afterEach() { reset(otherFooRepo) } // Tests below } As can be seen above, a post-execution hook is now necessary to clean up the mocked object otherFooRepo after the test execution(s), but this drawback is more than made up for by otherfooRepo and otherFooService now being immutable as well as having complete control over both objects’ lifetimes. @Captor Just as with the @Mock annotation, it’s possible to remove the @Captor annotation from the argument captor and declare its value directly in the code: Kotlin @SpringBootTest @AutoConfigureMockMvc class FooTest( @Autowired private val mockMvc: MockMvc, @Value("\${foo.value}") private val fooValue: String, ) { private val otherFooRepo: OtherFooRepo = mock() private val otherFooService = OtherFooService(otherFooRepo) private val timestampCaptor: ArgumentCaptor<Long> = ArgumentCaptor.captor() @MockBean private lateinit var fooRepo: FooRepo @AfterEach fun afterEach() { reset(otherFooRepo) } // Tests below } While there’s a downside in that there’s no mechanism in resetting the argument captor after each test (meaning that a call to getAllValues() would return artifacts from other test cases’ executions), there’s the case to be made that an argument captor could be instantiated as an object within only the test cases where it is to be used and done away with using an argument captor as a test class’s field. In any case, now that both @Mock and @Captor have been removed, it’s possible to remove the Mockito extension as well. @MockBean A caveat here: the use of mock beans in Spring Boot tests could be considered a code smell, signaling that, among other possible issues, the IO layer of the application isn’t being properly controlled for integration tests, that the test is de-facto a unit test and should be rewritten as such, etc. Furthermore, too much usage of mocked beans in different arrangements can cause test execution times to spike. Nonetheless, if it’s absolutely necessary to use mocked beans in the tests, a solution does exist for converting them into immutable objects. As it turns out, the @MockBean annotation can be used not just on field declarations, but also for class declarations as well. Furthermore, when used at the class level, it’s possible to pass in the classes that are to be declared as mock beans for the test in the value array for the annotation. This results in the mock bean now being eligible to be declared as an @Autowired bean just like any “normal” Spring Boot bean being passed to a test class: Kotlin @SpringBootTest @AutoConfigureMockMvc @MockBean(value = [FooRepo::class]) class FooTest( @Autowired private val mockMvc: MockMvc, @Value("\${foo.value}") private val fooValue: String, @Autowired private val fooRepo: FooRepo, ) { private val otherFooRepo: OtherFooRepo = mock() private val otherFooService = OtherFooService(otherFooRepo) private val timestampCaptor: ArgumentCaptor<Long> = ArgumentCaptor.captor() @AfterEach fun afterEach() { reset(fooRepo, otherFooRepo) } // Tests below } Note that like otherFooRepo, the object will have to be reset in the cleanup hook. Also, there’s no indication that fooRepo is a mocked object as it’s being passed to the constructor of the test class, so writing patterns like declaring all mocked beans in an abstract class and then passing them to specific extending test classes when needed runs the risk of “out of sight, out of mind” in that the knowledge that the bean is mocked is not inherently evident. Furthermore, better alternatives to mocking beans exist (for example, WireMock and Testcontainers) to handle mocking out the behavior of external components. Conclusion Note that each of these techniques is possible for code written in Java as well and provides the very same benefits of immutability and control of the objects’ lifecycles. What makes these recommendations even more pertinent to Kotlin is that they allow the user to align more closely with Kotlin’s design philosophy. Kotlin isn’t simply “Java with better typing:" It’s a programming language that places an emphasis on reducing common programming errors like accidentally accessing null pointers as well as items like inadvertently re-assigning objects and other pitfalls. Going beyond merely looking up the tools that are at one’s disposal in Kotlin to finding out why they’re available in the form that they exist will yield dividends of much higher productivity in the language, less risks of trying to fight against the language instead of focusing on solving the tasks at hand, and, quite possibly, making writing code in the language an even more rewarding and fun experience.
Imagine your online store’s checkout button disappears just before a major sale. Or, your company’s logo suddenly turns neon pink across the entire site. Even minor UI bugs can have disastrous consequences, leading to frustrated users, lost sales, and damaged brand reputation. Visual regression testing ensures that code changes don’t accidentally break the way your website or app looks. It focuses on catching layout issues, color problems, and other visual glitches that can disrupt the user experience. With a wide range of visual regression testing tools available, choosing the right one is crucial. This blog will help you navigate the options and find the perfect fit for your testing needs. What To Look for While Choosing Visual Regression Testing Tools When evaluating visual regression testing solutions, there are several key factors to consider: 1. Coding Skills You Need There are two types of automated testing tools available: Testing frameworks to assist developers in writing coded test scripts and No-code tools that allow anyone to generate tests using a drag-and-drop interface or a record-and-playback plugin. If your QA team lacks programmers, you’ll want to look for a no-code solution. 2. Filteration of False Positive Ideally, the tool should reliably flag visual regressions impacting user experience while intelligently filtering out acceptable UI modifications. Many open-source options are too sensitive, causing failures for any pixel-level change. Assess each tool’s capabilities in this regard. 3. Does It Require Any Add-Ons? A few visual testing tools are one-stop testing solutions that may be used to build, run, and update tests. However, they only provide visual validation; hence, they should be used with another testing solution. Other products are simply plug-ins for your existing automation framework that add visual assurance. They commonly accomplish this by inserting a snapshot into an automated test script. On the other hand, TestGrid io is a one-stop solution for automating both traditional and visual regression testing. (However, I’ll get to that later.) 4. Open-Source or Commercial There are some free, open-source tools that you’ll never have to pay for, but they usually don’t come with customer service, require programming knowledge to operate, and are more limited in what they can do. When it comes to commercial tools, several include pricing tiers that push you to upgrade to a far more expensive level once you’ve used up your beginner plan. Furthermore, specific tools bind you to long-term agreements. An ideal tool will allow you to pay for only the testing you require, with the ability to scale down or scale up as your needs change. Top Visual Regression Testing Tools 1. TestGrid TestGrid is among the most popular AI-powered end-to-end test automation solutions for high-velocity teams across the globe. It offers a built-in automated visual testing feature that enables users to assess the visual elements of mobile apps and websites. This eliminates the need to add any external SDK to projects and change the functional test case code. With TestGrid’s visual testing, you simply need to write a few lines of code, and AI takes care of the rest. As the automation tests run, you get a complete comparison of visual testing. Moreover, TestGrid ensures complete test coverage. It validates every element of the user interface, including layouts, images, buttons, etc., without requiring you to write unreliable tests. It covers complex, end-to-end scenarios, like BFSI and eCommerce flows, giving you the confidence that your mobile app or website looks as intended. 2. SikuliX SikuliX is an open-source visual regression testing tool developed in Java and is compatible with Windows, macOS, and some Linux systems. It leverages image recognition powered by OpenCV to detect GUI components. When you don’t have direct access to a GUI’s internals or an app’s/website’s source code, this tool comes in handy. You can use a mouse and keyboard to interact with the identified GUI elements. Moreover, SikuliX is equipped with OCR (Optical Character Recognition) to look for text in images. This tool does not work on mobile devices. However, it works well with emulators on a desktop or laptop. Some major use cases of SikuliX are automating Flash objects, conducting visual testing on web pages and desktop apps, and automating certain tasks on GUI. 3. Aye Spy Aye Spy is yet another underrated, open-source visual regression testing tool. The creators of this tool were inspired by BackstopJS and Wraith. They pinpointed performance as the weakness of other visual regression tools available on the market and created Aye Spy to address it. This tool provides 40 UI comparisons in just 60 seconds. You require Selenium Grid to use Aye Spy. This combination makes cross-browser testing a breeze, as Selenium Grid supports parallel testing on multiple computers. The creators suggest using Selenium’s Docker images. Further, the tool supports AWS S3 for storing snapshots. Clean documentation helps you set up and navigate the tool easily. 4. Hermione.js Hermione.js is also an open-source visual regression testing tool that combines the capabilities of integration testing. However, it is ideal only for more straightforward websites. If you have prior knowledge of Mocha and WebdriverIO, it becomes easier to use Hermione.js. It facilitates parallel testing across several browsers. You need to configure Hermione.js either using DevTools or the WebDriver Protocol that requires a Selenium grid. This tool has a user-friendly interface and offers plugins to extend its functionality. As the tool reruns failed testing events, it significantly minimizes incidental test failures. 5. Vizregress Colin Williamson created Vizregress, an open-source visual regression testing tool, as a research project. He created it to address an issue with Selenium WebDriver – it could not differentiate layouts if the CSS elements stayed the same and only visual representation was modified. Vizregress compares screenshots against an approved set of screenshots to spot visual regression in an app/website. It allows you to ignore a region on an app/website that needs to be ignored. Built on the AForge.Net Framework, this tool uses the framework’s core image utilities to compare screenshots. 6. Percy Visual Testing Percy is one of the most sophisticated visual testing tools available. You can use it to integrate, run, and review visual tests. Integration can be accomplished using test automation frameworks, continuous integration, delivery services, or directly through your application. We can begin performing the required visual tests on the apps and components following integration. The Percy tool extracts UI screenshots across many browsers and responsive widths when a visual test is started. It then compares your UI against a baseline pixel by pixel and finds any relevant visible changes. You can check the screenshots for any visual concerns once they’ve been produced. This tool’s pixel-by-pixel and responsive diffs provide excellent visual coverage. The snapshot stabilization option also helps to reduce false positives. This is a paid service, but it does provide a free trial version. 7. Applitools It’s one of the most widely used commercial frameworks for visual regression testing and automated visual testing. This tool gives you a cognitive vision driven by AI. It employs artificial intelligence to help you with visual testing and monitoring from start to finish. You can test hundreds of UI components across all platforms and configurations with Applitools without writing any code or with very little code. In addition, it supports over 40 different testing frameworks and languages. It also helps with DevOps by allowing you to easily integrate visual test cases into the CI/CD workflow. You may also use this tool to build bespoke graphic reports. Some of the world’s most well-known organizations, like Sony, SAP, MasterCard, and PayPal, use this framework. It also has a lot of great feedback from customers. Starter, Enterprise Public Cloud, and Enterprise Dedicated Cloud are the three versions of this framework offered by the manufacturer. 8. Screener.io This cloud-based automated testing solution allows you to record and run tests in real time. Test flows can be readily automated without the need for coding. It detects UI discrepancies across multiple platforms automatically. It also aids in the testing of storytelling components. Combining visual and functional testing in the same test run allows you to increase total test coverage. For example, Microsoft, Yammer, Uber, and more companies use screener.io. 9. Cross-Browser Testing Visual Testing You may easily filter your test results to determine which ones have aesthetic flaws. You can also go straight to a live test and debug or fix the visual flaws. This software includes a local connection tool to test local and development environments. Once you’ve completed a screenshot test for visual regression testing, you can schedule it on a daily, weekly, or monthly basis. Notifications of test results will also be sent out by the tool. This program contains an automated comparison engine that allows you to take automatic screenshots of the same page in different configurations. You can then choose a baseline browser and compare the highlighted layout variations. It also has sophisticated features to help you make your exam more effective. Basic authentication, Login profile, Selenium script, Screenshot delay, Send emails, Hide fixed items, and so on are some available choices. 10. Endtest This UI testing platform uses machine learning to allow codeless automated testing. It enables you to develop automated tests quickly, save them, and run them in the cloud. In addition, they provide a Chrome addon that allows you to record the tests. Generate random test data, advanced assertion, automatic backups, geolocation, live videos, screenshot comparison, and more capabilities are included in this program. Conclusion Visual regression testing is as vital as functional testing if you want a pleasant user experience. However, a range of tests can be covered when visual and functional testing is coupled. This testing can be done with various open-source and commercial tools and frameworks. We have also discussed the most valuable tools. Those visual regression testing tools can help you completely automate your testing efforts.
Test automation is essential for ensuring the quality of software products. However, test automation can be challenging to maintain, especially as software applications evolve over time. Self-healing test automation is an emerging concept in software testing that uses artificial intelligence and machine learning techniques to enable automated tests to detect and self-correct issues. This makes test automation more reliable and cost-effective and reduces the amount of time and resources required to maintain test scripts. In this article, we will discuss the benefits of self-healing test automation, how it works, and how to implement it in your organization. What Is Self/Auto-Healing Test Automation? Self-healing test automation is a new approach to test automation that uses artificial intelligence (AI) and machine learning (ML) to make test scripts more robust and adaptable. With self-healing test automation, test scripts can automatically detect and repair themselves when changes are made to the application under test, including shifting layouts and broken selectors. This makes it possible to automate tests for complex applications with frequently changing user interfaces without having to constantly maintain and update the test scripts. Why Is Self-Healing Test Automation Necessary? Test automation scripts can easily break when changes are made to the user interface. This is because test automation scripts are typically designed to interact with specific elements on the screen, such as buttons, text fields, and labels. When these elements change, the script may no longer be able to find them or interact with them correctly. This can lead to test failures and false positives, which can be time-consuming and frustrating to resolve. Also, the user interfaces are constantly evolving, with new features and bug fixes being added frequently. This means that test automation scripts need to be updated regularly to adapt to these changes. However, updating test automation scripts can be a manual and time-consuming process, which can be challenging to keep up with the pace of change. Self-healing test automation addresses this fragility of traditional test automation scripts by adapting to changes in the user interface automatically to make test scripts more robust and adaptable. Self-healing test scripts can automatically detect and repair themselves when changes are made to the application under test, which can help to reduce test maintenance costs, improve test quality, and increase test coverage. How Does Self-Healing Mechanism Work? Step 1: The self-healing mechanism gets triggered whenever “NoSuchElement” or a similar error occurs for an element mentioned in the automation script. Step 2: The algorithm analyzes the test script to identify the root cause of the error. Step 3: The algorithm uses AI-powered data analytics to identify the exact object in the test script that has changed. An object can be any interface item like a webpage, navigation button, or text box. Step 4: The algorithm updates the test script with the new identification parameters for the affected object(s). Step 5: The updated test case is re-executed to verify that the remediation has been successful. How Self-Healing Test Automation Adds Value to Your Software Delivery Process Leveraging self-healing capabilities allows test automation to adapt to changes, improving test coverage, reducing maintenance efforts, and enabling faster feedback. Saves Time and Effort Self-healing test automation can save organizations a significant amount of time and effort in software testing. Traditional test automation approaches require manual intervention to fix errors or failures that occur during test execution. This can be a time-consuming and error-prone process, especially when dealing with large and complex test suites. Self-healing test automation eliminates the need for manual intervention, allowing tests to recover automatically from failures or errors. Improves Test Coverage Self-healing test automation can help to improve test coverage by allowing testers to focus on writing new tests rather than maintaining existing tests. This is because self-healing tests can automatically adapt to changes in the software under test, which means that testers do not need to spend time updating their tests every time the software changes. As a result, testers can focus on writing new tests to cover new features and functionality. Self-healing automation can improve test coverage by 5-10% by eliminating unnecessary code, resulting in shorter delivery times and higher returns on investment. Prevents Object Flakiness Object flakiness is a common problem in test automation, especially for GUI testing. Object flakiness occurs when a test fails because it is unable to locate an object on the page. This can happen for a variety of reasons, such as changes to the UI, changes to the underlying code, or network latency. Self-healing test automation can detect and prevent object flakiness by analyzing test results and identifying patterns that indicate flaky tests. By preventing object flakiness, teams can reduce the number of false positives and negatives, improving the overall accuracy and reliability of test results. Faster Feedback Loop Self-healing test automation also enables a faster feedback loop. With traditional test automation approaches, tests are often run manually or through a continuous integration pipeline. However, with self-healing test automation, tests can be run continuously, providing immediate feedback on the quality of the application under test. This enables teams to identify and fix issues faster, improving the overall quality and reliability of the application. Conclusion In Agile methodology, applications are continuously developed and tested in short cycles. This can make it difficult to maintain test cases, as the application is constantly changing. Self-healing test automation can help to overcome this challenge by automatically updating test cases when the application under test changes.
Over the years, many articles have highlighted the importance of unit and integration tests and their benefits. They enable quick and accurate identification of errors, simplify the debugging process, support safe refactoring, and prove invaluable during code reviews. These tests can also significantly reduce development costs, help catch mistakes early, and ensure the final product aligns well with its specifications. As such, testing is often viewed as a central part of the development process. However, within the developer community, it's become clear in recent years that merely having unit and integration tests isn't enough. A growing number of blog posts and articles emphasize the need for well-structured and formatted tests. So, why is this aspect so crucial? Best Practices In short, poorly formatted tests or those exhibiting anti-patterns can significantly hamper a project's progress. It's not just my perspective. Many articles stress the significance of well-structured testsand provide best practices and insights on this topic. One element that frequently emerges as pivotal in these discussions is the naming of tests. Two articles in particular, Anatomy of a Good Java Test and Importance of Unit Testing underscore the crucial role of effective test naming. They advise against using the word "test" in test names, suggesting that appropriate naming can clearly describe the test's objective or what it intends to verify. Additionally, the article Clean Unit Testing highlights not only the naming of test methods but also the importance formaintainability of correct naming and ordering test variables. Branching out from naming assertions is another cornerstone in testing best practices. Take, for instance, the article 7 Tips for Writing Better Unit Tests in Java that highlights the advantage of using assertions over print statements. Other industry experts often emphasize limiting the number of assertions and correctly positioning them within a single test. The AAA pattern (Arrange, Act, Assert) is the perfect example of this intention: positioning assertions at the end of the test method ensures clarity and readability for other developers. Moreover, the transparency of the assertions themselves is also important. For instance, they should comewith descriptive messages. In fact, there are more suggestions to keep in mind: Appropriate usage of mocks and stubs. Avoiding "if" statements in test blocks. Focusing on a single case in each unit Making tests as isolated and automated as possible. Maintaining high test and code coverage. Testing negative scenarios and borderline cases, in addition to positive ones. Avoiding non-deterministic results and flaky tests Avoiding unit-test anti-patterns Yet, the realm of best practices is ever-evolving, and this list isn't exhaustive. New best practices continue to emerge. For example, the recent idea about the layout of tests highlights the importance of structuring both unit and integration tests within the source code. It's not just about refactoring tests anymore but also about organizing them systematically within the source code. In summation, as you can see, the community provides a variety of best practices for creating quality tests. The real question, however, is: Are these principles just theoretical, or are there practical solutions that can help us achieve such quality? Gap Identification Yes, I'm referring to static analyzers. Let's briefly examine the most widely used ones, even though there are many similar tools available. I will focus only on rules and checks that help to address at least some of the best practices discovered previously. Checkstyle Checkstyle is a development tool that helps programmers write Java code that adheres to a coding standard. In other words, Checkstyle is a static code analysis tool (linter) used in the Java world. Although Checkstyle doesn't provide features specifically tailored for tests, many of its features areapplicable to test code, just as they are to production code. It can assist with Javadoc comments, indentation, line length, cyclomatic complexity, etc. However, to the best of my knowledge, the only feature related to tests is the ability to enforce the test names convention by developing a specific checker. So, yes, before using it, you need to develop your own checker first.Thus, while Checkstyle is a general tool that focuses solely on Java code, it doesn't specifically address issues with tests. It doesn't consider specific rules related to assertion checks, identification of anti-patterns, or maintaining the layout of tests - all of which are essential to keep tests consistent and clear in line with industry requirements and best practices. PMD PMD is one more source code analyzer similar to Checkstyle. It finds common programming flaws like unused variables, empty catch blocks, unnecessary object creation, and so forth. While it supports many different languages, we are only interested in Java. PMD, compared with Checkstyle, has many more rules that check test quality, for example (but not limited to): JUnitAssertionsShouldIncludeMessage requires JUnit assertions to include a message. JUnitTestContainsTooManyAsserts checks if the JUnit or TestNG test contains too many assertion statements. JUnitTestsShouldIncludeAssert checks that JUnit tests include at least one assertion. TestClassWithoutTestCases checks that test classes have at least one testing method. UnnecessaryBooleanAssertion checks that JUnit assertions are used correctly without assertTrue(true) statements (line-hitter anti-pattern detection.) Here is a short example of test violations that PMD can find: Java public class Foo extends TestCase { public void testSomething() { // [JUnitAssertionsShouldIncludeMessage] Use the form: // assertEquals("Foo does not equals bar", "foo", "bar"); // instead assertEquals("foo", "bar"); } //[TestClassWithoutTestCases] Consider adding test methods if it is a test: public class Bar extends TestCase {} public class MyTestCase extends TestCase { // Ok public void testMyCaseWithOneAssert() { boolean myVar = false; assertFalse("should be false", myVar); } //[JUnitTestsShouldIncludeAssert] //Bad, don't have any asserts public void testSomething() { Bar b = findBar(); b.work(); } //[JUnitTestContainsTooManyAsserts]: //Bad, too many asserts (assuming max=1) public void testMyCaseWithMoreAsserts() { boolean myVar = false; assertFalse("myVar should be false", myVar); assertEquals("should equals false", false, myVar); //[UnnecessaryBooleanAssertion] Bad, serves no real purpose - remove it: assertTrue(true); } However, all these checks are designed primarily for JUnit assertions and, in some cases, for AssertJ. They don't support Hamcrest assertions, which are widely adopted in the industry. Also, while PMD can check method names, these checks are relatively simple. They focus on aspects such as method name length, avoiding special characters like underscores, and adhering to camel case naming conventions. Consequently, these checks are primarily intended for production code only and don't examine specific test name patterns. Moreover, to the best of my knowledge, PMD doesn't identify structural mistakes or verify the correct placement of methods. Thus, PMD provides a rather limited set of checks for tests. Sonar Qube SonarQube is also a widely used tool for checking code quality. SonarQube has a lot of rules similar to PMD that can be applied to tests, for example: TestCases should contain tests. Literal boolean values and nulls should not be used in assertions. Assertions should not compare an object to itself. Test assertions should include messages. Test methods should not contain too many assertions. Similar tests should be grouped in a single Parameterized test. At the time of writing this text, there are around 45 rules specifically designed for tests. As you might have noticed, SonarQube has more rules than PMD, although many of them overlap. However, to the best of my knowledge, SonarQube doesn't check Hamcrest assertions and doesn't maintain the layout of tests. It also doesn't show much concern about checking test anti-patterns. Others Actually, there are other tools available for detecting issues related to test quality. Some notable ones include: SpotBugs checks for correct usage of setUp/tearDown methods, empty test cases, andimproper use of assertions. ErrorProne examines test signatures and forbids the use of "test" in test names, identifies redundant methods without @Test and @Ignore and offers some other test-related checks. MegaLinter and Qulice primarily combine previously mentioned linters like PMD and Checkstyle. Essentially, they just bundle checks from other linters. Coverity is a proprietary tool that has numerous checks, including those for assertions and various resource leaks. However, some users argue that its features are similar to those PMD and SpotBugs. Jtest is another proprietary tool that has a comprehensive set of features. This includes checks for assertion statements, initialization methods, and more. The complete list of checks can be found here. There are numerous other tools, including Checkmarx Glossary, Klocwork, CodeSonar, among many others, that we simply can't cover in this article. In summary, tools like Checkstyle, PMD, SonarQube, and others offer numerous rules to ensure test code quality. However, noticeable gaps exist in their ability to tackle certain test-related issues. Checkstyle is primarily designed for Java production code, and its features for tests are limited. This often requires users to develop their own checkers for specific scenarios. PMD has a robust set of rules for JUnit assertions, yet it doesn't support popular frameworks like Hamcrest or method naming patterns. SonarQube provides an extensive rule set, which overlaps with PMD in many areas. However, it lacks some vital test checks, including those for Hamcrest assertions and test anti-patterns. Other tools have their own limitations, or they are proprietary. Significantly, none of the aforementioned tools focus on the proper placement and naming of test classes. Thus, even though these tools provide a foundation for test code quality, there's a notable gap in terms of aligning with industry test standards and best practices. Introducing jtcop To address the aforementioned gaps, we developed a new static analyzer called jtcop that focuseson test quality in Java projects. It is a simple Maven plugin that checks tests for common mistakes and anti-patterns. We use it in our projects, and it has helped us maintain consistent and clear tests. It also speeds up PR reviews significantly by preventing recurring comments about issues like improper test placement or naming. Although, we don't think our rules are the only good way to set up tests, so feel free to share your ideas and suggestions by submitting tickets and PRs. In the following, I'll explain how jtcop fits into the landscape of static analysis tools, which checks it utilizes, and how it can assist you in youreveryday programming. Test Names I'm sure you know there are many ways to name your test. For example, you can find various test naming conventions or even some threads that have lengthy discussions on how to do it correctly. Here is just a short summary of how you can name your tests: Pattern Example methodName_stateUnderTest_expected add_negativeNumbers_throwsException() when_condition_then_expected when_ageLessThan18_then_isUnderageIsTrue() given_precondition_when_action_then_result given_userIsAdmin_when_deleteIsCalled_then_deleteSuccess() test[methodName] testAdd() or testIsUnderage() should_expectedBehavior_when_condition should_throwException_when_negativeNumbersAreAdded() methodName_expected add_returnsSum() or isUnderage_returnsTrue() canAction canDeleteUser() or canCalculateSum( methodName_doesExpectedBehavior add_doesReturnSum() or isUnderage_returnsTrue() verbCondition (or verbResult) calculatesSum() or deletesSuccessfully() jtcopprefers the last pattern: Test names should use the present tense without a subject. For example, if you're testing a class Animal with a method eat(), the test name should be eats(). If you need to add more context, do it after the verb – for instance, eatsApplesOnly(). Test names should use camelCase. Name shouldn't use the word "test", as it is redundant. The @Test annotation is sufficient. Special characters like _ and $ are forbidden. Correct Names Incorrect Names eats() testEats() eatsApplesOnly() TestEatsApplesOnly() runsQuickly() _runsQuickly() jumpsOverFence() jumps_over_fence() drinksWater() drinks$Water() sleepsAtNight() sleepsZZZ() chewsGum() test_chewsGum() listensToMusic() listens_To_Music() walksInPark() WalksInPark() barksLoudly() barks__loudly() This style has been chosen by many developers and is widely used in numerous projects. If you prefer a different pattern for test naming, just let us know, and we'll be happy to add it to the plugin. Corresponding Production Class Now, let's imagine we have a test class named SumTest.java with the test method checksSum(). But what if the test occasionally fails? Most would attempt to locate the issue and find the original class where the problem occurred. But which class is it? The first guess would likely be Sum.java, right? Yet, you might not find it, perhaps because the production class is named something like Addition.java or Calculator.java. This mismatch in naming conventions can lead to significant confusion and longertroubleshooting times. In other words, if you have a test class named SumTest.java and the corresponding production class is Addition.java, it can be very confusing. The more appropriate name for the test class would be AdditionTest.java. Essentially, the name of the test class isn't merely a label; it serves as a pointer to the production class, helping developers pinpoint potential issues.This is where jtcop comes into play. It helps ensure that your tests are consistent with your production classes and suggests appropriate naming conventions for them, effectively addressing the problem described. If you're further interested in this issue, you can read about it here.The only exception in this case is integration tests. They are usually named like AdditionIT.java or AdditionIntegrationTest.java. However, they should be placed in a separate package, such as it, and have an appropriate suffix like IT or ITCase. Test Methods Only The next check is rather strict and is still considered an experimental feature. However, the rule itself is simple: test classes should contain methods that are only annotated with the @Test annotation. You might wonder what to do with initialization methods or common code shared among different test cases. The answer isn't straightforward and this rule is designed to guide you with it. There aren't actually many options available. I'm referring to methods such as static initialization methods, setup methods @BeforeEach and @AfterEach annotations, JUnit extensions, and Fake Objects. The approach you choose for initializing your tests will determine their quality. Static Methods The first idea that comes to mind is using static methods. Developers often use static methods to configure a common setup for several tests in the class. Here's a simple example: Java @Test void calculatesSum(){ Summator s = init(); Assertions.assertEquals( 2, sum(1, 1), "Something went wrong, because 1 + 1 != 2" ); } private static Summator init(){ Summator s = new Summator(); // Setup return s; } At first glance, it might seem like a good solution, but it does have inherent problems. When such a method is used within a single class, it's usually not a major concern, even though static methods typically lead to low cohesion and tight coupling. However, issues arise when you begin to use it across multipleclasses or try to consolidate such methods into a centralized TestUtils.java class. In this case, the approach with static methods can become problematic: It can lead to confusion for developers since TestUtils.java doesn't correspond to any class in the production code. TestUtils.java might be considered an anti-pattern. Thus, jtcop deems static methods in tests and utility classes as dangerous and prohibits them. If you attempt to run jtcop against the previous code sample, you'll receive the following warning message: Shell All methods should be annotated with @Test annotation. SetUp and TearDown Methods The next widely-used approach involves the so-called "setUp" methods. By "setUp" methods, I'm referring to those annotated with @BeforeAll, @BeforeEach, @AfterAll, or @AfterEach. An example of using these annotations is as follows: Java Summator s; @BeforeEach void setUp(){ s = new Summator(); // Setup } @Test void calculatesSum(){ Summator s=init(); Assertions.assertEquals( 2, sum(1,1), "Something went wrong, because 1 + 1 != 2" ); } This approach makes the situation even worse for many reasons. The most obvious reason, familiar to most developers, is the need to "jump" between test methods and the initialization part. Then, over time, as the codebase grows and changes and as the number of test cases in the test class increases, developers may become unaware of the setup/teardown that happens for each test and may end up with setup code that is unnecessary for certain tests, thus violating the principle of keeping tests minimal and setting up only what is needed. Next, using such methods can introduce another problem. They can lead to ashared state between tests if not managed properly. This harms test isolation, an extremely important quality of any test, which in turn can result in flaky tests. Moreover, using @BeforeAll and @AfterAll use static methods, which inherit all the disadvantages of the previous approach. Hence, jtcop doesn't allow the use of such setUp/tearDown methods. Test Extensions Now, let's examine the approach supported by jtcop. JUnit 5 offers Test Extensions that allow for the creation of custom extensions. These extensions. can be used to configure setup and teardown logic for all the tests in a class. Java @ExtendWith(SummatorExtension.class) public class SumTest { @Test void calculatesSum(Summator s) { Assertions.assertEquals( 2, s.sum(1, 1), "Something went wrong, because 1 + 1 != 2" ); } class SummatorExtension implements ParameterResolver { @Override public boolean supportsParameter(ParameterContext pctx, ExtensionContext ectx) { return pctx.getParameter().getType() == Summator.class; } @Override public Object resolveParameter( Summator s =new Summator(); // Setup return s; } Extensions offer a way to craft more modular and reusable test setups. In this scenario, we've bypassed the need for utility classes, static methods, and shared states between tests. These extensions are easily reused across a multitude of test classes and standalone unit tests. What's more, theseextensions often have insight into the current test class, method, annotationsused, and other contextual details, paving the way for versatile and reusablesetup logic. Fake Objects Another method for test configuration and setup that jtcop supports is the use of Fake objects, as recommended here. These are positioned with other production objects, yet they provide a unique"fake" behavior. By leveraging these objects, all setup can be handled directly in a test, making the code cleaner and easier to read. Java abstract class Discount { // Usually we have rather complicated // logic here for calculating a discount. abstract double multiplier(); static class Fake extends Discount { @Override double multiplier() { return 1; } } public class PriceTest { @Test void retrievesSamePrice() { Price p = new Price(100, new Discount.Fake()); Assertions.assertEquals( 100, p.total(), "Something went wrong; the price shouldn't have changed" ); } Fake objects often sit alongside production code, which is why jtcop doesn't classify them as test classes. While mixing production and test code might seem questionable, Fake objects aren't exclusively for testing; you might sometimes integrate them into your production code, too. Many projects have embraced the use of Fake objects, finding it a practical way to set up tests. Additionally, this strategy eliminates the need for using Mock frameworks with intricate initialization logic. Test Assertions jtcop also underscores the need to validate assertions in tests. Several tools out there offer similar checks. Yet, many of them focus solely on JUnit assertions or only catch high-level errors. jtcop supports both Hamcrest and JUnit assertions and adheres to stricter guidelines for assertions. To paint aclearer picture, let's dive into a few code snippets. Java @Test void calculatesSum(){ if(sum(1, 1) != 2){ throw new RuntimeException("1 + 1 != 2"); } } This code snippet lacks any assertions, meaning jtcop will warn about it. Check out the next snippet as a proper replacement, and note the use of the Hamcrest assertion. Java @Test void calculatesSum(){ assertThat( "Something went wrong, because 1 + 1 != 2", sum(1, 1), equalTo(2) ); } Pay attention to the explanatory messages in the assertion Something went wrong, because 1 + 1 != 2 from the code above. They're essential. Without such messages, it can sometimes be challenging to understand what went wrong during test execution, which can puzzle developers. For instance, consider this real example. I've simplified it for clarity: Java @Test void checksSuccessfully(){ assertThat( new Cop(new Project.Fake()).inspection(), empty() ); } Now, suppose this test fails. In that scenario, you'll receive the following exception message: Shell Expected: an empty collection but: <[Complaint$Text@548e6d58]> Not very informative, right? However, if you include an explanatory message in the assertion: Java void checksSuccessfully(){ assertThat( "Cop should not find any complaints in this case, but it has found something.", new Cop(new Project.Fake()).inspection(), empty() ); } With this inclusion, you're greeted with a far more insightful message: Shell java.lang.AssertionError: Cop should not find any complaints in this case, but it has found something. Expected: an empty collection but: <[Complaint$Text@548e6d58]> In a perfect world, we'd offer more details — specifically, some context. This sheds light on initialization values and provides developers with valuable hints. Line Hitters The last feature I'd like to spotlight is the Line Hitter anti-pattern detection. At first glance, the tests cover everything and code coverage tools confirm it with 100%, but in reality the tests merely hit the code, without doing any output analysis. What this means is that you might stumble upon a test method in a program that does not really verify anything. Take this for instance: Java @Test void calculatesSum(){ sum(1, 1); } This typically happens when a developer is more into their code coverage numbers than genuinely ensuring the robustness of the test. There are tools that can spot when assertions are missing in tests. But, as you know, developers might always find a way around: Java @Test void calculatesSum(){ sum(1,1); assertThat( "I'm just hanging around", true, is(true) ); } Yep, that’s our "Line Hitter" again, only this time, it's wearing the disguise of an assertion statement. Luckily, jtcop can detect such tests and flag them as unreliable. Setting up jtcop To get started with jtcop, simply add the plugin to your build configuration file. If you're using Maven, here's how you can do it: XML <build> <plugins> <plugin> <groupId>com.github.volodya-lombrozo</groupId> <artifactId>jtcop-maven-plugin</artifactId> <version>1.1.1</version> <executions> <execution> <goals> <goal>check</goal> </goals> </execution> </executions> </plugin> </plugins> </build> By default, the plugin operates in the verify phase, so there is no need to specify it. However, if you wish to modify it, simply add the desired phase to the execution section. Then, to run jtcop, use the mvn jtcop:checkcommand. If you stumble upon an issue, say, a test lacking a corresponding productionclass, you'll get a clear error message: Shell [ERROR] Test SumTest doesn't have corresponding production class. [ERROR] Either rename or move the test class ./SumTest.java. [ERROR] You can also ignore the rule by adding @SuppressWarnings("JTCOP.RuleAllTestsHaveProductionClass") annotation. [ERROR] Rule: RuleAllTestsHaveProductionClass. [ERROR] You can read more about the rule here: <link> Similarly, for the "Line Hitter" pattern previously mentioned: Shell [ERROR] Method 'calculatesSum' contains line hitter anti-pattern. [ERROR] Write valuable assertion for this test. [ERROR] You can also ignore the rule by adding @SuppressWarnings("JTCOP.RuleLineHitter") annotation. [ERROR] Rule: RuleLineHitter. [ERROR] You can read more about the rule here: <link> By default, jtcop will halt the build if it detects issues with your tests. If you only want to use it to highlight problems without interrupting the build, you can configure jtcop to display only warning messages by adjusting the failOnError property. XML <configuration> <failOnError>false</failOnError> </configuration> However, I highly recommend keeping the default setting to maintain high-quality tests. Experimental Features As I mentioned earlier, some features are still experimental. To try them out, just add the following configuration to your pom.xml file: XML <configuration> <experimental>true</experimental> </configuration> Once done, all experimental features will be active in your project, ensuring cleaner and more organized tests. Benefits jtcop has already helped us in several ways: Code Review: The primary issue addressed by jtcop is the frequent appearance of comments such as "place this test class here," "rename this test method," or "that's a testing anti-pattern." `jtcop` saves time and aids developers in resolving these issues before even making a PR into arepository. Onboarding: Another advantage we've observed is that well-structured and appropriately named test methods not only facilitate code understanding and maintenance but also reduce the time spent explaining or documenting code style guides. As a result, we often receive well-formatted pull requests from new team members with little to no additional guidance. Consistency: jtcop ensures our tests remain consistent across numerous projects. So, when you delve into a project that uses jtcop, it becomes significantly easier to comprehend its workings and start contributing to it. Overall, integrating `jtcop` has significantly streamlined our processes, enhancing collaboration and understanding across our development projects. Future Plans Looking ahead, we're preparing to enhance jtcop with additional rules. One of our primary focuses is to address several anti-patterns like the ones highlighted in this StackOverflow thread. Just to name a few: The Mockery: Tests that have too many mocks. Excessive Setup: Tests that demand extensive setup. Wait and See: Tests that need to pause for a specific duration before verifying if the tested code works as intended. It's worth noting that these are just a few examples; there's a broader spectrum of anti-patterns we're considering. Additionally, we've also encountered issues with projects that have many tests written in various styles. At times, it's incredibly tedious to address these issues manually. Thus, another viable avenue is developing an application that will automatically solve most of these problems. So, if you have ideas or suggestions, please don't hesitate to open an issue or submit a pull request in our repository and share your thoughts with us. We're always eager to get feedback or contributions from the community. Feel free to fork it if you want and craft your own test checkers that fit your needs, or simply use jtcop as is.
Have you ever found yourself in the position of a test engineer embedded in one of the Agile engineering teams? While you have daily interactions with peers, connecting with them on a profound level for the successful execution of job duties might be challenging. Although there is a shared goal to release features successfully, we often experience isolation, especially while others, like developers, find comfort within the team. In the realm of dispersed Agile teams with time zones adding an extra layer of complexity, the longing for a team to resonate with, connect with, and brainstorm on all test automation challenges is prevalent. In the expansive landscape of test automation, the creation of an automation guild is more than just collaboration; it stands as a testament to the resilience of SDETs working across diverse time zones and Agile teams. Through this guide, I aim to share the benefits and challenges overcome, the enrichment of test engineers or SDETs, and the establishment of a collective force dedicated to advancing excellence in testing. Breaking Silos In a world where time zones separate teams and Agile methodologies dictate the rhythm of development, test engineers face a unique challenge. Even though they are part of an Agile team with a shared goal, i.e., successful release, they must navigate independently without a clear direction or purpose. The guild, however, becomes a bridge across these temporal gaps, offering a platform for asynchronous collaboration. It not only allows them to demo their progress, accomplishments, and new utility that can be leveraged by others but also their challenges and blockers. It will surprise you to see how often those obstacles are common among other guild members. Now that they have each other, all heads come together to brainstorm and find common, effective solutions for any testing problem. Fostering Through Training and Contribution As important as regular guild meet-ups and collective commitment are, continuous learning and training initiatives are equally vital to empower test engineers to contribute effectively. From workshops on emerging testing methodologies to skill-building webinars, the guild evolves into a learning haven where members grow together, ensuring each test engineer is equipped to make a meaningful impact. It enhances members’ efficiency by reducing redundant efforts. Understanding what others are working on and what tools are available for use, such as common utilities and shared definitions, enables them to save time by avoiding duplication of efforts and contribute more effectively. This isn’t just about individual efficiency; it’s a strategic move toward collective empowerment. Grow Your Network and Your Profile Within the guild, networking is not confined to individual teams. It offers the creation of a network that spans across Agile teams, allowing Test Engineers to understand overall solutions from diverse perspectives. This isn’t just about sharing knowledge; it’s about broadening domain knowledge. Turning new members into seasoned members who can then mentor new juniors, ensuring that the guild is not just a community but a mentorship ecosystem that thrives on collective wisdom. If there’s one aspect that has been repeatedly demonstrated in the guild, it would be that challenges are not roadblocks but opportunities for innovation and collaboration. The guild stands as a testament to the fact that, even in the world of test automation, where distances and time zones pose challenges, excellence can be achieved through collective strength. Automation guild is not just about crafting code; it’s about crafting a community that advances excellence in testing, collectively and collaboratively. The future, as envisioned through the chronicles, is one where Test Engineers, regardless of time zones, work seamlessly in a guild that stands as a beacon of innovation, knowledge-sharing, and collective growth.
Unit testing is an indispensable practice for Java developers (any developer really). The primary benefit of unit testing a pipeline is the early detection of bugs, which is less expensive to fix. It not only improves the overall quality of the code but also makes future maintenance easy. Using lambdas specifically with streams in Java makes code concise and readable. Streams are excellent for filtering, mapping, sorting, and reducing operations. The elements in the sequence are better processed in a declarative and functional manner, something which lambdas exactly fit into. The anonymous functions that are written with lambdas not only facilitate a functional style of programming but also enhance the expressiveness of the code. Streams and lambdas were introduced in Java after JDK 8, and since then, Java developers have used these features frequently in their projects. The question around these components that the article tries to address is, "How can you unit test Java streams and lambdas?" Importance of Unit Testing Pipelines and Lambdas In unit testing, an individual piece of component of a software application is tested separately. This small unit of code typically is a function, method, or subroutine. The testing mechanism is automated so that they can be done repeatedly and quickly. The test cases are usually written by developers and integrated into the CI/CD pipeline in the development process. The code can be isolated and problems can be easily identified if we use lambda because the essence of it is to make the program functional, more modular, and reusable — something which makes it friendly for unit testing pipelines. Unit Testing Stream Pipelines Since stream pipelines combine with lambdas to form a single unit, it is not obvious how to effectively unit test the pieces of the pipeline. I have always followed these two guidelines to unit test those stream pipelines: If the pipeline is simple enough, it can be wrapped in a method call, and it is enough to unit test the method call. If the pipeline is more complex, pieces of the pipeline can be called from support methods, and the support methods can be unit-tested. For example, let's say we have a stream pipeline that maps all the letters of the word to uppercase and is wrapped in a java.util.function.Function<T,R> as below: Function<List<String>, List<String>> allToUpperCase = words -> words.stream().map(String::toUpperCase).collect(Collectors.toList()); Now, the unit test for this stream pipeline can be easily written as ordinary unit testing of the allToUpperCase. @Test public void testAllToUpperCase() { List<String> expected = Arrays.asList("JAVA8", "STREAMS"); List<String> result = allToUpperCase.apply(Arrays.asList("java8", "streams")); assertEquals(expected, result); } The stream above can be wrapped in a regular function, as seen below. Also, an ordinary unit test can be written against this function: public List<String> convertAllToUpperCase(List<String> words) { return words.stream().map(String::toUpperCase).collect(Collectors.toList()); } Unit Testing Lambdas Believe me — it is very likely that you will encounter complex unit testing in real-world programming. The unit testing with complex lambdas, similar to unit testing of stream pipelines, can be simplified with the following practices: Replace a lambda that needs to be tested with a method reference and an auxiliary method. Then, test the auxiliary method. For example, I have a stream pipeline that involves a somewhat complex lambda and a mapping class for the given string class name. public static Class[] mapClasses(final List<String> exceptions) { return exceptions.stream().map(className -> { try { return Class.forName(className); } catch(Exception ex) { LOGGER.error("Failed to load class for exceptionWhiteList: {}", className); } return null; }).toArray(Class[]::new); } Here, the key point to test is whether the expression for transforming a string class name to a Class object is working. As mentioned above, this can replace the lambda expression with a method reference, along with an auxiliary method that can be placed in a companion class, as shown below: public static Class[] mapClassesBetter(final List<String> exceptions) { return exceptions.stream().map(LambdaTester::mapClass).toArray(Class[]::new); } public static Class mapClass(String className) { try { return Class.forName(className); } catch(Exception ex) { LOGGER.error("Failed to load class for name: {}", className); } return null; } Now, the key element of the original lambda is that it can be tested directly: @Test public void testMapClass() throws ClassNotFoundException { assertEquals(null, mapClass("a")); assertEquals(null, mapClass("apple")); assertEquals(Object.class, mapClass("java.lang.Object")); } Conclusion Writing unit tests is one of the core parts of software development. With new features introduced after JDK 8, it has become easier to write code concisely and declaratively. However, the proper use of features like streams and lambda brings value and, of course, makes writing unit tests easier. If you have any additional guidelines for unit testing these features, don't stop yourself from sharing them in the comments. Until next time, happy coding! Learn more about the best Java unit testing frameworks. The source code for the examples presented above is available on GitHub.
The purpose of this use case is to explain how to define different RAML data types, define the business-related status code for request payload validation, define the single unit test case for multiple field validation with dynamic request payload, and how to use the parameterized test suite. With a parameterized test suite, we can write a reusable unit test case. It can help to write and test multiple scenarios based on different inputs with a single test case. This can be more useful when we are writing a test case to validate the request and response payload. RAML Data Types Definition Define different types of data types for a request payload. In the below example, I covered integer, string, pattern, enum, array, object, datetime, datetime-only, and time-only. YAML #%RAML 1.0 DataType type: object properties: employeeId: type: integer required: true minimum: 8 firstName: type: string required: true minLength: 1 maxLength: 10 pattern: ^[A-Za-z]* lastName: type: string required: true minLength: 1 maxLength: 10 pattern: ^[A-Za-z]* email: pattern: ^.+@.+\..+$ gender: enum: [male, female] default: male required: true dateOfBirh: type: date-only required: true addresses: type: array minItems: 1 items: type: object properties: isPermanent: type: boolean required: true street: type: string required: true minLength: 5 maxLength: 50 pattern: ^[A-Za-z ]* district: type: string required: true minLength: 3 maxLength: 20 pattern: ^[A-Za-z]* state: type: string required: true minLength: 5 maxLength: 15 pattern: ^[A-Za-z]* pinNumber: type: integer required: true minimum: 6 province: type: string required: true minLength: 1 maxLength: 10 pattern: ^[A-Za-z]* phoneNumber: type: string required: true minLength: 1 maxLength: 13 pattern: ^\s*|^(0|91)?[6-9][0-9]{9}$ created: type: datetime format: rfc3339 required: true createdDateTime: type: datetime-only required: true createdTime: type: time-only required: true Sample Request Payload Based on the RAML definition, prepare a valid request payload. JSON { "employeeId": 12345678, "firstName": "Ankur", "lastName": "Bhuyan", "email": "ankur.bhuyan@gmail.com", "gender": "male", "dateOfBirh": "2000-04-01", "addresses": [ { "isPermanent": true, "street": "teachers colony", "district": "Sunitpur", "state": "Assam", "pinNumber": 784507, "province": "Tezpur", "phoneNumber": "+919590951234" } ], "created": "2016-02-28T12:30:00.090Z", "createdDateTime": "2016-02-28T12:30:00", "createdTime": "12:30:00" } Configure Parameterized Test Suite for Valid Scenarios This is a valid test case scenario for which the parameterized inputs are defined. Based on the parameterization "name" defined, the test result name will be prepared (once the test case is executed). We have defined only one property with which the input payload for the test case will be prepared, which will vary based on the test scenario we will cover. XML <munit:config name="apikit-valid-test-suite.xml" > <munit:parameterizations > <munit:parameterization name="001" > <munit:parameters > <munit:parameter propertyName="caseNumber" value="001" /> </munit:parameters> </munit:parameterization> </munit:parameterizations> </munit:config> Configure Parameterized Test Suite for Invalid Scenarios This is an invalid test case scenario for which the parameterized inputs are defined. We can add as many numbers of the parameter (like caseNumber, expectedType, expectedCode) based on the use case. This is an example to show how we can define multiple parameters for a single test case. XML <munit:config name="apikit-invalid-test-suite.xml"> <munit:parameterizations > <munit:parameterization name="001" > <munit:parameters > <munit:parameter propertyName="caseNumber" value="001" /> <munit:parameter propertyName="expectedType" value="REQUIRED_KEY" /> <munit:parameter propertyName="expectedCode" value="ANK000001" /> </munit:parameters> </munit:parameterization> <munit:parameterization name="002" > <munit:parameters > <munit:parameter propertyName="caseNumber" value="002" /> <munit:parameter propertyName="expectedType" value="TYPE" /> <munit:parameter propertyName="expectedCode" value="ANK000002" /> </munit:parameters> </munit:parameterization> <!-- define all posible test parameters --> </munit:parameterizations> </munit:config> Dynamic Payload for Munit Test This is how we can write a test case with a dynamic request payload. This will help to define a number of different input payloads based on the test scenario that can be used as a parameterized input. XML <ee:transform doc:name="payload" doc:id="7c934c10-d874-4207-be27-de6c8b1a1c5a" > <ee:message > <ee:set-payload > <![CDATA[%dw 2.0 output application/json var fileBaseName = "-invalid-payload.json" var caseNumber = Mule::p("caseNumber") var fileName = caseNumber ++ fileBaseName --- readUrl("classpath://test-data/employee/payload/$(fileName)", "application/json")]]> </ee:set-payload> </ee:message> </ee:transform> Test Output We can test multiple field validation with a single munit test by using dynamic payload and parameterized input. Code Reference To find more on the above description, please follow the full code reference here.
Arnošt Havelka
Development Team Lead,
Deutsche Börse
Thomas Hansen
CTO,
AINIRO.IO
Soumyajit Basu
Senior Software QA Engineer,
Encora
Nicolas Fränkel
Head of Developer Advocacy,
Api7