Information Technology | Testing and Quality assurance » Harry Robinson - How to Build Your Own Robot Army

How to Build Your Own Robot Army Harry Robinson STAR West 2006 1 Why do you need a robot army? The number of test cases you have time to execute The universe of all possible test inputs You Warning: this diagram is NOT to scale. 2 Two More Reasons for Letting Robots Test “Test suites wear out.” “Only weak bugs have a logic to them Subtle bugs have no definable pattern – they are wild cards.” Boris Beizer, Software Testing Techniques, 2nd edition 3 Robots as Monkeys “When subjected to random valid input that could be produced by using the mouse and keyboard, we crashed 21% of applications that we tested and hung an additional 24% of applications. When subjected to raw random Win32 messages, we crashed or hung all the applications that we tested.” Forrester and Miller, “An Empirical Study of the Robustness of Windows NT Applications Using Random Testing” But, could robots do much more? 4 A Quick Look at the Economics of Test Generation •

• A typical test engineer • Earns $50,000 per year • Works 40 to 50 hours per week A typical test machine • Costs $300 per year to buy and operate • Can work 100 hours per week 5 Should You Hire Additional Testers? $402,400 per year for 16640 hours of testing = $24/hour 6 A Robotic Approach $52,400 per year for 41600 hours of testing = $1.25/hr 7 How to Build Your First Robot Incremental Robot Building a.ka TSTTCPFAB The Simplest Thing That Could Possibly Find A Bug 8 Incremental Robotics, step 1 Start Clock.exe Stop 9 Incremental Robotics, step 2 Analog Digital Analog Start DblClk Clock.exe Stop DblClk 10 Incremental Robotics, step 3 Digital Stop Start Clock.exe Analog Digital DblClk Stop Analog DblClk Start Start DblClk Clock.exe Stop Stop DblClk Start 11 Clock.exe Clock.exe What Kinds of Bugs do Robots Find? Start Maximize Stop Start Minimize Stop Start Restore Stop 12 Understanding Where Robots Fit in

Your Process • The Pygmalion Problem • Of Zergs and Orcs • The Probe Droid Option • The Cyborg Possibility 13 Pygmalion 14 The Pygmalion Problem Some testers get so involved in creating robots that they forget to test: Don’t fall in love with your robot! 15 Zergs, Orcs 16 Zergs, Yes! Zergling Rush To overwhelm an opponent through the use of cheaply made units at the expense of any long-term strategy (taken from Blizzards "StarCraft"). www.urbandictionarycom 17 Orcs, No! Don’t allow yourself to think that you are doing a good job testing because you are creating and running lots of tests: Numbers do not equal effectiveness 18 A Sample ‘Zerg’ Database Query SELECT TOP 2 6o , ((-1 )%(-(-(T1.qty ))))/(-(-2 )), (2 )+(T0min lvl ), ^p: FROM jobs T0, sales T1 WHERE ( ( (T0.job id ) IS NOT NULL ) OR ((Feb 24 7014 10:47pm )= ( SELECT DISTINCT Jun 2 5147 6:17am FROM employee T2, titleauthor T3, jobs T4 WHERE ( T2.job lvl BETWEEN (3 ) AND

(((-(T4max lvl ))%((3 )-(-5 )))-(((-1 )/(T4.job id ))%((3 )%(4 )))) ) OR (EXISTS ( SELECT DISTINCT TOP 7 MIN(LTRIM(Hqz6=14I )), LOWER( MIN(T5.country )), MAX(REVERSE((LTRIM(REVERSE(T5.city ))+ LOWER(Iirl )))), MIN(T5city ) and so on [from Don Slutz’s 1998 paper “Massive Stochastic Testing of SQL”] 19 A Sample ‘Orc’ Database Query JhygttfythjkiuiRTHJRTVFDRT76trftghyuikuytr4edfghyuio;k,lojuhjugtfhytfrgtrftdnh kshiuyfizncl0eu9823865wefpodkjp09g9y8idkfud Hsdiufy7wey7y7yy7654edcvbnm,klooiujhkiuhjmk, @#$rft67uJI()OLK<::PO(*&^%$#Edscvbnm,kloi987654e3dfcvbnmkiuytrfd XDR^YTGFY*&^TFVBNMJKI&^%$$%%^&OLKLKJJYT$#@@!@WSDE#$% YJIKKKIIUJJJ O((*&JMJMKOKIUYTFFFYhjuuejfrjdujuUJmjyet357u485869ijdjr4 and so on [from rolling my knuckles across the keyboard] 20 Having millions of Orc tests may impress people but do those tests find bugs? 21 Probe Droids 22 The Probe Droid Option Lewis and Clark send out their probe droid. 23 Writing a Probe

Droid for Google Maps 24 Exercising the Controls 25 Checking Every Zip Code Zip Co de s 26 Checking Driving Directions 27 The Scenic Route? 28 Isn’t It A Long Way Home? 29 Cyborgs 30 The Cyborg Possibility Understanding the strengths and limitations of robots What is Wrong with this Picture? 31 32 33 34 Humans and Machines – Perfect Together? 35 DC Comics Is this a Routing Bug? 11 miles 1.4 miles 36 You Need Context to Judge the Routing 11 miles 1.4 miles 37 Changing the Software Lifecycle • Earlier automation means earlier stability • Earlier test involvement skews metrics • Testers become designers • No longer an economy of scarcity Me developer tester 38 Changing the Software Lifecycle • Earlier automation means earlier stability • Earlier test involvement skews metrics • Testers become designers • No longer an economy of scarcity developer tester 39 Changing the Software

Lifecycle • Earlier automation means earlier stability • Earlier test involvement skews metrics • Testers become designers • No longer an economy of scarcity developer tester 40 Changing the Software Lifecycle • Earlier automation means earlier stability • Earlier test involvement skews metrics • Testers become designers • No longer an economy of scarcity developer tester 41 Changing the Software Lifecycle • Earlier automation means earlier stability • Earlier test involvement skews metrics • Testers become designers • No longer an economy of scarcity developer tester 42 Changing the Software Lifecycle • Earlier automation means earlier stability • Earlier test involvement skews metrics • Testers become designers • No longer an economy of scarcity developer tester 43 Changing the Software Lifecycle • Earlier automation means earlier stability • Earlier test involvement skews metrics • Testers become designers • No longer

an economy of scarcity developer tester 44 How Your Robots Can Make You Look Good • Be involved early with development • Understand that you are now doing development • Name your robot • Write and present about your robot • Get testimonials from those you help • Walk your manager past the test lab 45 Meet Some of Your New Friends Randomness Oracles Balance Opportunity Time Spare Machines 46 Free Tools for Robot Construction • Watir – wtr.rubyforgeorg • Ruby – www.ruby-langorg • Python – www.pythonorg 47 Recommended Reading • Rodney Brooks, “Fast, Cheap And Out Of Control: A Robot Invasion Of The Solar System” • Doug Hoffman, “Using Oracles in Test Automation” • Cem Kaner, Walter Bond, Pat McGee, “High Volume Test Automation” • William McKeeman, “Differential Testing for Software” • Noel Nyman, “In Defense of Monkey Testing” • Don Slutz, “Massive Stochastic Testing of SQL” 48 Thank you!

www.model-based-testingorg/robot armyhtm harryr@google.com 49