Information Technology | Testing and Quality assurance » Harry Robinson - Intelligent test automation

This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine. Tools & Automation Intelligent Test Automation A model-based method for generating tests from a description of an application’s behavior by Harry Robinson Warning: The fairy tale you are about to read is a fibbut it’s short, and the moral is true. O nce upon a product cycle, there were four testers who set out on a quest to test software. started hands-on testing immediately, and found some nice bugs. The development team happily fixed these bugs, and gave Tester 1 a fresh version of the software to test. More testing, more bugs, more fixes. Tester 1 felt productive, and was happyat least for a while. After several rounds of this find-and-fix cycle, he became bored and bleary-eyed from running virtually the same tests over and over again by hand. When Tester 1 finally ran out of enthusiasmand then out of patiencethe software was declared “ready to ship.” Customers found it

too buggy and bought the competitor’s product. 24 www.stqemagazinecom Software Testing & Quality Engineering September/October 2000 ILLUSTRATIONS BY STEVE BJÖRKMAN Tester1 This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine. QUICK LOOK ■ Improving the efficiency of your automated testing through modeling ■ Overcoming the limitations of hands-on and static automation testing Tester2 started testing by hand, but soon decided it made more sense to create test scripts that would perform the keystrokes automatically. After carefully figuring out tests that would exercise useful parts of the software, Tester 2 recorded the actions in scripts. These scripts soon numbered in the hundreds At the push of a button, the scripts would spring to life and run the software through its paces. Tester 2 felt clever, and was happyat least for a while. The scripts required a lot of maintenance when the software changed. He spent weeks

arguing with developers to stop changing the software because it broke the automated tests. Eventually, the scripts required so much maintenance that there was little time left to do testing. When the software was released, customers found lots of bugs that the scripts didn’t cover. They stopped buying the product and decided to wait for version 2.0 Tester3 didn’t want to maintain hundreds of automated test scripts. She wrote a test program that went around randomly clicking and pushing buttons in the application. This “random” test program was hypnotic to watch, and it found a lot of crashing bugs. Tester 3 enjoyed uncovering such dramatic defects, and was happyat least for a while. Since the random test program could only find bugs that crashed the application, Tester 3 still had to do a lot of hands-on testing, getting bored and bleary-eyed in the process. Customers found so many functional bugs in the software when it was released that they lost trust in the company and

stopped buying its software. 25 September/October 2000 Software Testing & Quality Engineering www.stqemagazinecom This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine. Tester4 began with hands-on, exploratory testing to become familiar with the application and used the knowledge gained during the hands-on testing to create a very simple behavioral model of the application. Tester 4 then used a test program to test the application’s behavior against what the model predicted. The behavioral model was much simpler than the application under test, so it was easy to create. Since the test program knew what the application was supposed to do, it could detect when the application was doing the wrong thing. As the product cycle progressed, developers wrote new features for the application. Tester 4 quickly updated the model, and the tests continued running. The program ran day and night, constantly generating new test sequences Tester

4 was able to run the tests on a dozen machines at once and get several days of testing done in a single night. After several rounds of testing and bug fixes, Tester 4’s test generator began to find fewer bugs. Tester 4 upgraded the model to test for additional behaviors and continued testing. Tester 4 also did some hands-on testing and static automation for those parts of the application which were not yet worth modeling. When Tester 4’s software was released, there were very few bugs to be found. The customers were happy. The stockholders were happy And Tester 4 was happy. Commentaries These four scenes show some of the approaches available in software testing today. Tester 1 is a typical hands-on tester, manually running all tests from the keyboard. Hands-on testing is common throughout the industry todayit provides immediate benefits, but in the long run it is tedious for the tester and expensive for the company. “One of the saddest sights to me has always been a human at a

keyboard doing something by hand that could be automated. It’s sad but hilarious” Boris Beizer, Black-Box Testing: Techniques for Functional Testing of Software and Systems Tester 2 practices what I call “static test automation.” Static automation scripts exercise the same sequence of com- mands in the same order every time. These scripts are costly to maintain when the application changes The tests are repeatable; but since they always perform the same commands, they rarely find new bugs. “Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else’s footprints minimizes the chance of being blown up by a land mine.” James Bach, “Test Automation Snake Oil,” Windows Tech Journal, October 1996 Tester 3 operates closer to the cutting edge of automated testing. These types of “random” test programs are called dumb monkeys because they essentially bang on the keyboard

aimlessly. They come up with unusual test action sequences and find many crashing bugs, but it’s hard to 26 www.stqemagazinecom Software Testing & Quality Engineering September/October 2000 This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine. direct them to the specific parts of the application you want tested. Since they don’t know what they are doing, they miss obvious failures in the application. “Monkey testing should not be your only testing. Monkeys don’t understand your application, and in their ignorance they miss many bugs.” Noel Nyman, “Using Monkey Test Tools,” STQE, January/February 2000 Hands-on testing is a good way to start the test automation process. I call this phase “exploratory modeling” because it combines exploratory testing with the discovery of a model that can later be used to generate tests. As you begin to understand the behavior of each action, you can create rules that will help you

model and test the application. Tester 4 combines the other testers’ approaches with a type of intelligent test automation called “model-based testing.” Model-based testing doesn’t record test sequences verbatim like static test automation does, nor does it bang away at the keyboard blindly. Model-based tests use a description of the application’s behavior to determine what actions are possible and what outcome is expected. This automation generates new test sequences endlessly, adapts well to changes in the application, can be run on many machines at once, and can run day and night. “[An artist] paints with his brain, not with his hands.” Michelangelo Buonarroti ate rules that will help you model and test the application. This is the essence of model-based testing: To describe the behavior you expect in a way that can be used to generate tests. Ask yourself the following two questions for every action you are going to test: 1. When is this action possible? 2. What is

the outcome when this action is executed? For instance, suppose you have been asked to test the behavior of files in a Windows folder. In particular, you are going to test the Create, Delete, and Invert Selection actions The Moral of the Story Tester 1’s method required that his hands always be at work on the keyboard. Eventually Tester 1’s brain and hands gave out. Tester 2’s static scripts repeated keyboard actions that his hands had already performed. Tester 3’s monkeys were essentially brainless hands banging on the keyboard. Tester 4, on the other hand, supplemented the others’ techniques by: ■ thinking about the application’s behavior, Modeling the “Create” Action ■ When is Create possible? This example is kept simple by limiting the number of files in the folder to 1 File. Therefore Create is only possible in this model when there are 0 Files in the folder ■ What is the outcome when Create is executed? When you Create a new file in a folder, the number of

files in the folder increases by one. The newly created file is initially Selected, so it appears highlighted in the folder. In fact, the new file is the only Selected file in the folder, no matter how many were Selected before the Create action. ■ describing that behavior to a test generator, and Modeling the “Delete” Action ■ letting the test generator create and run test cases. By generating tests from a description of the application’s behavior, Tester 4 could perform tests that were not practical under the other test approaches. The moral of the tale: Automate your brain, not just your hands. ■ When is Delete possible? Delete is only possible in this model when there is at least 1 Selected File in the folder. ■ What is the outcome when Delete is executed? When you execute the Delete action, any Selected file disappears from the folder. Modeling the “Invert Selection” Action Use Your Brain Let’s look at an example of creating and using a behavioral model to

test a software application. Hands-on testing is a good way to start the test automation process. I call this phase “exploratory modeling” because it combines exploratory testing with the discovery of a model that can later be used to generate tests. As you begin to understand the behavior of each action, you can cre- ■ When is Invert Selection possible? Invert Selection is always possible in this model, even when there are 0 Files in the folder. ■ What is the outcome when Invert Selection is executed? When you execute the Invert Selection action, all Selected files in the folder become Unselected, and all Unselected files become Selected. When there are 0 Files in the folder, Invert Selection leaves the folder unchanged. 28 www.stqemagazinecom Software Testing & Quality Engineering September/October 2000 This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine. Creating a State Model You can now construct what is called a

“state model” of the system’s behavior, as shown in Figure 1. It incorporates all the behaviors described above. Note the way the Invert Selection action loops from the 0 Files State back to the 0 Files State. That models the way Invert Selection does nothing if there’s nothing to invert. Invert Selection Invert Selection Create Delete 0 Files Invert Selection 1 Selected File 1 Unselected File FIGURE 1 The state model Very Pretty. So What? Now that you understand how the application works, you could manInvert Selection ually test these actions and verify whether the Windows folder behaves as you expect. However, be2 1 cause your understanding is being Invert Create Selection carried around inside your head, 4 your results are limited by your time and your stamina. 5 On the other hand, if you Delete Invert could somehow communicate this Selection 3 state model directly from your brain to a computer, it could generate and execute tests on the system 0 Files 1 Selected

File 1 Unselected File for you. Fortunately, this model can be FIGURE 2 A random walk represented in a format known as a “state table” that the computer can read. Each row of the state table (see Table 1) shows the Starting State, you can choose either of these two Ending State that will result when an action is applied to actions: the application in the Starting State. ■ Invert Selection (which leaves you in the 0 Files State) Use the Computer’s Brain, Too ANNIE BISSETT Once we have put the state model into a state table that the computer can understand, what can the computer do for us? How can we exploit our information about the application’s behavior? The computer can use the state table to generate sequences of tests to run against the application. As you will see in the following examples, these test sequences can be chosen for their novelty, their efficiency of testing, or their exhaustiveness. This test generation is a Starting State powerful way to apply your

understandingand this is what model0 Files based testing is all about. 0 Files A Random Walk Through the State Model One simple way to generate test actions is to randomly select any available action from the current state of the application. For example, if you are in the 0 Files ■ Create (which leaves you in the 1 Selected File State) By choosing random actions in this way, you can generate many unusual sequences (just like Tester 3’s random “monkey test program”), and you will eventually exercise all of the actions in the model. Figure 2 shows a typical random walk. Notice that the random walk executed the same action (Invert Selection) four times in a row, but has Action Ending State Invert Selection 0 Files Create 1 Selected File 1 Selected File Invert Selection 1 Unselected File 1 Selected File Delete 0 Files 1 Unselected File Invert Selection 1 Selected File TABLE 1 A state table for behavior of files in a Windows folder 29 September/October 2000

Software Testing & Quality Engineering www.stqemagazinecom This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine. plication. If you think that bugs are more likely to occur where the ap1 plication changes state, you may want to prioritize your efforts by first testing the state-changing acInvert Create tions. Selection 2 3 A simple way to do this is to filter out from the state table any 4 actions that don’t change the state. 5 Delete Invert In Table 1, that would remove the Selection first action (Invert Selection). Running the Chinese Postman algorithm over this reduced state 0 Files 1 Selected File 1 Unselected File model generates a test sequence that covers all of the model’s FIGURE 3 A Chinese Postman walk state-changing actions in four stepsessentially removing the first step of the previous example: so far left two other actions untouched. Such is the nature of random testing. Invert Selection E N D I N G S TAT E Create

Invert Selection Invert Selection Invert Selection Invert Selection 1 Selected File 1 Unselected File 1 Selected File 1 Unselected File 1 Selected File 1. 2. 3. 4. An Efficient Walk Through the State Model: The Chinese Postman Walk Random walks are inefficient at reaching all test actions when the model is large. How can we test each of the actions in the model efficiently? This turns out to be the same problem a letter carrier faces when delivering mail. Imagine that each of the actions in the model is a street where mail must be deliveredand that each of the states in the model is an intersection where the letter carrier can change direction. Just as the letter carrier must travel down each street to deliver the mail, we must test each action in the model. And in both cases, we would like to minimize the amount of additional travel needed. A Chinese mathematician named Kwan Mei-Ko formulated an elegant solution to this problem, and it is known as the Chinese Postman algorithm in

his honor (see Figure 3). Kwan’s method generates a path through the state model that exercises every action in the model in the fewest number of steps. The test sequence listed below covers all five actions in the model in only five steps. This efficiency can be handy if you have a large application that you want to test quickly. 1. 2. 3. 4. 5. ACTION TO EXECUTE E N D I N G S TAT E Invert Selection Create Invert Selection Invert Selection Delete 0 Files 1 Selected File 1 Unselected file 1 Selected File 0 Files ACTION TO EXECUTE E N D I N G S TAT E Create Invert Selection Invert Selection Delete 1 Selected File 1 Unselected File 1 Selected File 0 Files Shortest Round Trips Back to the Starting State Suppose you want to exhaustively test every sequence that takes the Windows folder from the 0 Files State back to the 0 Files State in a certain number of steps or less? Sequences like these that constantly generate new variations would be unthinkable for Tester 2’s static

automation. It is trivial for a computer to generate a list of such paths from the state model. You can generate sequences of increasing length as long as you have computer cycles to burn, probing deeper and deeper into the model. Figure 4 shows all round trips that start at the 0 Files State and have a path length less than or equal to four steps. Path A has a length of one step: A1: Invert Selection Path B has a length of two steps: B1: Create B2: Delete Path C has a length of four steps: C1: Create C2: Invert Selection C3: Invert Selection C4: Delete Use the Computer’s Hands An Even More Efficient Walk: The StateChanging Chinese Postman Walk Some actions in a modelsuch as hitting Invert Selection with 0 Files in the folderdo not change the state of the ap- The output of each of these algorithms is a sequence of test actions to execute. What would be the best way to perform these actions? You could hand a human tester the list of actions to execute by handbut this would be slow,

tedious, and cruel. Who would want to spend their day executing lists of actions? Such repetitious work is the kind of mindnumbing scenario that caused poor Tester 1 such grief. 30 www.stqemagazinecom Software Testing & Quality Engineering September/October 2000 ANNIE BISSETT 1. 2. 3. 4. 5. ACTION TO EXECUTE This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine. Invert Selection A1 C2 C1 Invert Selection Create B1 B2 Invert Selection Delete C4 0 Files 1 Selected File C3 1 Unselected File FIGURE 4 All round trips in four steps or less Instead, you can write a simple test execution program that will read the list and then execute test code for each action in that list. For instance, in Visual Test, the code to implement the Create action is: plication is in the 1 Selected File State, the test program will report an error because the test code will fail when it finds no menu selection for Delete. Second, the model is

always aware of what state the application should be in. Knowing the expected ending state of each action means that we can create test oracle routines to check (at the conclusion of each action) that the appropriate number of files are present and selected in the folder. For instance, when the Delete action above is executed, the Ending State should have 0 Files in the folder (and of course, 0 Files Selected). Programmatic test languages usually provide functions that allow the test program to verify aspects of the application. Two useful Visual Test functions for the current model are: ■ WViewCount( ) which indicates the number of files in the folder, and WToolbarButtonClick("@1","File") Open the File menu WMenuSelect("New") Select New File WMenuSelect("Text Document") Choose Text Document Play "{Enter}" Accept the default filename ■ WViewItemSelected( ) which tells how many files in the folder are Selected. In typical

static automation, this code would be embedded in a scriptbut in a model-based test program, this snippet of code is invoked whenever the list of test actions says to perform the Create action. The test program can verify that the application is in the correct state, as shown in Table 2. The Delete action discussed above should leave the application in the 0 Files State. If WViewCount( ) returns a value other than 0, the test program oracle will report an error because the number of files in the folder is incorrect. ANNIE BISSETT Use the Computer’s Eyes How to Update Model-based Tests Automating the test actions is only half the battle. You also need an automated method of determining if the applicaRemember how Tester 2’s static test automation attempts tion is working correctly. were frustrated by application changes? Tester 4, in conThis methoda function that determines if the applitrast, was able to adapt the model-based test automation cation has behaved correctly in

response to a test action quickly to changes in the application. is called a test oracle. Some test methods, such as Tester 3’s random monkey test programs, must rely on crude test Incorporating New Actions into the Model oracles such as whether the application has crashed. Suppose your development team tells you that they have imModel-based testing gives the test program the ability to plemented the Select All action. How should you update your check for indicators of good behavior more subtle than tests for this new action? Simpleupgrade the state model “didn’t crash.” From the information in the state table, the to incorporate the Select All action, and regenerate the tests. model “knows” what actions should be available from each First, you model the Select All action by answering our state and the expected outcome of each action. For instance, two basic questions: the model says that the test program should be able to execute the Expected Return Expected Return Delete

action from the 1 Selected File Value of Value of State. The model also says that exeApplication State WViewCount( ) WViewItemSelected( ) cuting that Delete action should leave the application in the 0 Files 0 Files 0 0 State. This knowledge provides two 1 Selected File 1 1 ways to verify that the application has behaved correctly. 1 Unselected File 1 0 First, the test program can detect if an action is not available when it should be. If the Delete ac- TABLE 2 A state table showing Visual Test functions WViewCount( ) and tion is not available when the ap- WViewItemSelected( ) 31 September/October 2000 Software Testing & Quality Engineering www.stqemagazinecom This article is provided courtesy of Software Testing & Quality Engineering (STQE) magazine. 1. When is Select All possible? Select All is always possible in this model, even when there are 0 Files in the folder. Select All Delete Select All Select All 1 Selected File E N D I N G S TAT E Invert Selection

Create Invert Selection Select All Invert Selection Invert Selection Select All Delete Select All 0 Files 1 Selected File 1 Unselected File 1 Selected File 1 Unselected File 1 Selected File 1 Selected File 0 Files 0 Files ■ Models are simple to modify, so model-based tests are economical to maintain over the life of a project. ■ Models can generate innumerable test sequences tailored to your needs. ■ Models allow you to get more testing accomplished in a shorter amount of time because a test generator can create and verify test sequences around the clock on multiple machines. ■ Model-based testing can supplement other forms of testing, and can perform tests that aren’t practical under other test approaches. You and I know that software testing is no fairy tale, and that happily-ever-afters are never guaranteed. But adding modelbased intelligence to your testing is a powerful tool to help you find your way toward your own happy ending. STQE The next step would be to

determine the code that is used to invoke the Select All action whenever it occurs in the test sequence. For Visual Test it would be as follows: Harry Robinson is software test lead with the Intelligent Search Group at Microsoft. He maintains the ModelBased Testing Home Page (wwwmodel-based-testingorg), and is a long-time advocate and practitioner of modelbased testing. WToolbarButtonClick("@1","Edit") WMenuSelect("Select All") Invert Selection It can take significant effort to understand and model an application. And it can be difficult to leave the easy path of hands-on testing or static automation long enough to invest time thinking about how to test that applicationas we saw in the trials and tribulations of our fairy tale’s four testers. The rewards, however, are great: 1 2 3 Invert Selection Create 5 Select All 4 Delete Invert Selection 8 9 7 Select All ■ Model-based testing creates flexible, useful test automation from practically

the first day of development. 1 Unselected File FIGURE 5 State model including Select All ACTION TO EXECUTE In Summary Invert Selection 0 Files Figure 5 shows the new state model with the Select All action incorporated. Running the Chinese Postman algorithm on the updated model (see Figure 6) gives a nine-step test sequence using the 0 Files Starting Statethat exercises every action in the model, including the new Select All: 1. 2. 3. 4. 5. 6. 7. 8. 9. Invert Selection Create 0 Files 6 Select All 1 Selected File 1 Unselected File FIGURE 6 A Chinese Postman walk on the new state model 32 www.stqemagazinecom Software Testing & Quality Engineering September/October 2000 ANNIE BISSETT 2. What is the outcome when Select All is executed? When you execute Select All, all the files in the folder become Selected. If there are 0 Files in the folder, Select All leaves the folder unchanged. This is indicated in the illustration below, where the Select All action loops

from the 0 Files State back to the 0 Files State. Invert Selection