Experiences of Test Automation

Dorothy Graham is a world-renowned consultant, speaker, and author with nearly forty years of experience in software testing. After nineteen years with Grove Consultants, she now concentrates on conferences and writing. She was Programme Chair for the 1993 and 2009 EuroSTAR conferences and holds the European Excellence Award in Software Testing. Mark Fewster has thirty years of software testing and automation experience. As developer and manager for a multi-platform graphical application, he designed an architecture for long-lasting test automation. With Grove Consultants since 1993, he provides training and consultancy in all aspects of software testing. Graham and Fewster coauthored the popular book Software Test Automation (Addison-Wesley, 1999).  Contributed chapter lead authors include Lisa Crispin, Henri van de Scheur, Ken Johnston, Bo Roop, John Kent, Ane Clausen, Elfriede Dustin, Alan Page, Stefan Mohacsi, Simon Mills, Jason Weden, Bryan Bakker, Antti Jääskeläinen, Christoph Mecke, Björn Boisschot, Michael Williamson, Lars Wahlberg, Jonathan Kohl, Albert Farré Benet, Seretta Gamba, Wim Demey, Ursula Friede, John Fodeh, Mike Baxter, Ross Timmerman, Ed Allen, and Harry Robinson.

• Foreword xxix Preface xxxi  Reflections on the Case Studies (by Dorothy Graham and Mark Fewster ) 1A Management Issues 2B Technical Issues 8C Conclusion 16  Chapter 1: An Agile Team’s Test Automation Journey: The First Year (by Lisa Crispin) 171.1 Background for the Case Study 181.2 Whole Team Commitment 191.3 Setting Up the Automation Strategy 201.4 Applying Acceptance Test-Driven Development (ATDD) to Test behind the GUI Using FitNesse 241.5 Use an Incremental Approach 261.6 The Right Metrics 271.7 Celebrate Successes 281.8 Incorporate Engineering Sprints 281.9 Team Success 291.10 Continuous Improvement 311.11 Conclusion 32  Chapter 2: The Ultimate Database Automation (by Henri van de Scheur) 332.1 Background for the Case Study 332.2 Software under Test 352.3 Objectives for Test Automation 362.4 Developing Our Inhouse Test Tool 372.5 Our Results 402.6 Managing Our Automated Tests 402.7 Test Suites and Types 412.8 Today’s Situation 432.9 Pitfalls Encountered and Lessons Learned (the Hard Way) 432.10 How We Applied Advice from the Test Automation Book 452.11 Conclusion 472.12 Acknowledgments 48  Chapter 3: Moving to the Cloud: The Evolution of TiP, Continuous Regression Testing in Production (by Ken Johnston and Felix Deschamps) 493.1 Background for the Case Study 503.2 Moving Our Testing into the Cloud 523.3 How We Implemented TiP 553.4 Sample of Monthly Service Review Scorecards 583.5 Exchange TiP v2–Migrating TiP to the Windows Azure Cloud 623.6 What We Learned 633.7 Conclusion 673.8 Acknowledgments 67  Chapter 4: The Automator Becomes the Automated (by Bo Roop) 694.1 Background for the Case Study: My First Job 694.2 My Great Idea . . . 724.3 A Breakthrough 744.4 Conclusion 80  Chapter 5: Autobiography of an Automator: From Mainframe to Framework Automation (by John Kent) 835.1 Background for the Case Study 845.2 A Mainframe Green-Screen Automation Project 885.3 Difference between Mainframe and Script-Based Tools 895.4 Using the New Script-Based Tools 915.5 Automating Tests for IBM Maximo 975.6 Conclusion 1025.7 Additional Reading 103  Chapter 6: Project 1: Failure!, Project 2: Success! (by Ane Clausen) 1056.1 Background for the Case Study 1056.2 Project 1: Failure! 1076.3 Project 2: Success! 1096.4 The Next Time Period: Testing for Real 1186.5 Conclusion 127  Chapter 7: Automating the Testing of Complex Government Systems (by Elfriede Dustin) 1297.1 Background for the Case Study 1297.2 Our Requirements for Automation 1317.3 Automated Test and Re-Test (ATRT), Our Automated Testing Solution–What Is It? 1327.4 Automated Testing Solution Applied 1407.5 Conclusion 142  Chapter 8: Device Simulation Framework (by Alan Page) 1438.1 Background for the Case Study 1438.2 The Birth of Device Simulation Framework (DSF) 1458.3 Building the DSF 1468.4 Automation Goals 1488.5 Case Studies 1498.6 No Silver Bullets 1538.7 Conclusion 1548.8 Acknowledgments 154  Chapter 9: Model-Based Test-Case Generation in ESA Projects (by Stefan Mohacsi and Armin Beer) 1559.1 Background for the Case Study 1559.2 Model-Based Testing and Test-Case Generation 1579.3 Our Application: ESA Multi-Mission User Services 1619.4 Experience and Lessons Learned 1689.5 Conclusion 1739.6 References 1749.7 Acknowledgments 175  Chapter 10: Ten Years On and Still Going (by Simon Mills) 17710.1 Background for the Case Study: “Before” 17710.2 Insurance Quotation Systems Tested Automatically Every Month 17910.3 What Happened Next? 19310.4 Conclusion 193  Chapter 11: A Rising Phoenix from the Ashes (by Jason Weden) 19711.1 Background for the Case Study 19711.2 The Birth of the Phoenix 19911.3 The Death of the Phoenix 20211.4 The Rebirth of the Phoenix 20311.5 The New Life of the Phoenix 20711.6 Conclusion 212  Chapter 12: Automating the Wheels of Bureaucracy (by Damon Yerg [A Pseudonym]) 21712.1 Background for the Case Study 21712.2 The Agency Automation 21912.3 From 2000 to 2008 22312.4 An Alignment of Planets 22612.5 Building Capability within Test Teams 23112.6 Future Directions: The Journey Continues 23312.7 Conclusion 235  Chapter 13: Automated Reliability Testing Using Hardware Interfaces (by Bryan Bakker) 23713.1 Background for the Case Study 23813.2 The Need for Action 23913.3 Test Automation Startup (Incremental Approach) 24013.4 Buy-In from Management 24213.5 Further Development of Test Framework 24413.6 Deployment and Improved Reporting 24813.7 Conclusion 250  Chapter 14: Model-Based GUI Testing of Android Applications (by Antti Jääskeläinen, Tommi Takala, and Mika Katara) 25314.1 Background for the Case Study 25314.2 MBT with TEMA Toolset 25614.3 Modeling Application Behavior 26114.4 Generation of Tests 26614.5 Connectivity and Adaptation 26814.6 Results 27214.7 Conclusion 27314.8 Acknowledgments 27414.9 References 274  Chapter 15: Test Automation of SAP Business Processes (by Christoph Mecke, Melanie Reinwarth, and Armin Gienger) 27715.1 Background for the Case Study 27815.2 Standards and Best Practices 28215.3 eCATT Usage Examples 28615.4 Conclusion 29215.5 Acknowledgments 293  Chapter 16: Test Automation of a SAP Implementation (by Björn Boisschot) 29516.1 Background for the Case Study 29516.2 Project Overview 29816.3 Phase 1: Proof of Concept 29916.4 Phase 2: Project Start 30716.5 Conclusion 319  Chapter 17: Choosing the Wrong Tool (by Michael Williamson) 32117.1 Background for the Case Study 32117.2 Our Preexisting Automation (or Lack Thereof) 32417.3 Decision Needed: New Tool or Major Maintenance Effort? 32617.4 Moving Forward with eggPlant 32817.5 What Did We Do after eggPlant? 33617.6 Conclusion 336  Chapter 18: Automated Tests for Marketplace Systems: Ten Years and Three Frameworks (by Lars Wahlberg) 33918.1 Background for the Case Study 34018.2 Automated Test Frameworks 34118.3 Test Roles 34418.4 Abstraction Layer 34518.5 Configuration 34818.6 Cost and ROI 34918.7 Conclusion 352  Chapter 19: There’s More to Automation Than Regression Testing: Thinking Outside the Box (by Jonathan Kohl) 35519.1 Background for the Case Study 35519.2 Two Tales of Task Automation 35719.3 Automation to Support Manual Exploratory Testing 36219.4 Automating Data Interactions 36419.5 Automation and Monitoring 36819.6 Simulating Real-World Loads by Combining Simple Tools 37019.7 Conclusion 37219.8 References 372  Chapter 20: Software for Medical Devices and Our Need for Good Software Test Automation (by Albert Farré Benet, Christian Ekiza Lujua, Helena Soldevila Grau, Manel Moreno Jáimez, Fernando Monferrer Pérez, and Celestina Bianco) 37520.1 Background for the Case Study 37620.2 Comparison of the Different Approaches to Each Project 38120.3 Project hamlet 38520.4 Project phoenix 38620.5 Project doityourself 38820.6 Project miniweb 39120.7 Test Execution 39220.8 Result Reporting 39320.9 Conclusion 396  Chapter 21: Automation through the Back Door (by Supporting Manual Testing) (by Seretta Gamba) 40121.1 Background for the Case Study 40121.2 Our Technical Solution 40321.3 Implementing Test Automation with ISS Test Station 40621.4 Implementing Test Automation 40921.5 Supporting Manual Testing 41321.6 The New Manual Test Process 41721.7 Conclusion 42221.8 References 423  Chapter 22: Test Automation as an Approach to Adding Value to Portability Testing (by Wim Demey) 42522.1 Background for the Case Study 42722.2 Portability Testing: Love or Hate It 42822.3 Combination of Both Worlds as a Solution 42822.4 Conclusion 43522.5 Acknowledgment 435  Chapter 23: Automated Testing in an Insurance Company: Feeling Our Way (by Ursula Friede) 43723.1 Background for the Case Study 43723.2 The Application 43923.3 Objectives 44023.4 The Work 44123.5 Lessons 44323.6 Conclusion 444  Chapter 24: Adventures with Test Monkeys (by John Fodeh) 44724.1 Background for the Case Study 44724.2 Limitations of Automated Regression Testing 44924.3 Test Monkeys 45124.4 Implementing Test Monkeys 45324.5 Using Test Monkeys 45424.6 Benefits and Limitations 45824.7 Conclusion 45924.8 Additional Reading 460  Chapter 25: System-of-Systems Test Automation at NATS (by Mike Baxter, Nick Flynn, Christopher Wills, and Michael Smith) 46125.1 Background for the Case Study 46125.2 Test Execution Tool Integration 46525.3 Pilot Project for the Tool 46625.4 In-Service Model 46725.5 Implementation 46725.6 Typical Script Template 47025.7 Lessons Learned 47225.8 Conclusion 474  Chapter 26: Automating Automotive Electronics Testing (by Ross Timmerman and Joseph Stewart) 47726.1 Background for the Case Study 47726.2 Objectives for Automation Project 48026.3 Brief History of the Automation Project 48026.4 Results of the Automation Project 48326.5 Conclusion 483  Chapter 27: BHAGs, Change, and Test Transformation (by Ed Allen and Brian Newman) 48527.1 Background for the Case Study 48527.2 Buy-In 48727.3 The Story of Building the Automation Framework 49127.4 Description of our Automation Framework 49327.5 The Test Environment 49727.6 Metrics 49927.7 Conclusion 501  Chapter 28: Exploratory Test Automation: An Example Ahead of Its Time (by Harry Robinson and Ann Gustafson Robinson) 50528.1 Background for the Case Study 50528.2 What’s a Trouble Manager? 50728.3 Testing a Trouble Manager Transaction 50928.4 Constructing Test Cases Programmatically 51028.5 New Ways to Think about Automated Tests 51128.6 Testing the Trouble Manager Workflow 51328.7 Test Generation in Action 51828.8 Home Stretch 52028.9 Post-Release 52128.10 Conclusion 52228.11 Acknowledgments 522  Chapter 29: Test Automation Anecdotes 52329.1 Three Grains of Rice (by Randy Rice) 52329.2 Understanding Has to Grow (by Molly Mahai) 52729.3 First Day Automated Testing (by Jonathon Lee Wright) 52829.4 Attempting to Get Automation Started (by Tessa Benzie) 53529.5 Struggling with (against) Management (by Kai Sann) 53629.6 Exploratory Test Automation: Database Record Locking (by Douglas Hoffman) 53829.7 Lessons Learned from Test Automation in an Embedded Hardware—Software Computer Environment (by Jon Hagar) 54529.8 The Contagious Clock (by Jeffrey S. Miller) 54929.9 Flexibility of the Automation System (by Mike Bartley) 55129.10 A Tale of Too Many Tools (and Not Enough Cross-Department Support) (by Adrian Smith) 55229.11 A Success with a Surprising End (by George Wilkinson) 55629.12 Cooperation Can Overcome Resource Limitations (by Michael Albrecht) 56129.13 An Automation Process for Large-Scale Success (by Michael Snyman) 56229.14 Test Automation Isn’t Always What It Seems (by Julian Harty) 567  Appendix: Tools 573  About the Case Study Authors 587About the Book Authors 605Index 607

“What you hold in your hands is a treasure trove of hard-won knowledge about what works and what doesn’t in test automation. It can save you untold hours and costs by steering you away from paths that lead nowhere and guiding you towards those that lead to success.” –Linda Hayes  “From tools to methodology, Dorothy Graham and Mark Fewster weave a compelling set of stories that provide a learning experience in automation. This comprehensive tome is the first of its kind to take the reader deep into the world of automated testing, as depicted by case studies that show the realities of what happened across a multitude of projects spanning a wide variety of industries and technology environments. By identifying similarities and repeated themes, the authors help the reader focus on the essential learning lessons and pitfalls to avoid. Read this book cover to cover for inspiration and a realization of what it takes to ultimately succeed in test automation.”–Andrew L. Pollner, President & CEO of ALP International Corporation  “Many years after their best-seller Software Test Automation, Mark Fewster and Dorothy Graham have done it again. Agile methodologies have given test automation a dominant presence in today’s testing practices. This is an excellent, highly practical book with many well-documented case studies from a wide range of perspectives. Highly recommended to all those involved, or thinking about getting involved, in test automation.”– Erik van Veenendaal, Founder of Improve Quality Services and vice-chair of TMMi Foundation  “This book is like having a testing conference in your hand, with a wealth of case studies and insights. Except that this book is much cheaper than a conference, and you don’t have to travel for it. What impressed me in particular was that it is all tied together in a concise ‘chapter zero’ that efficiently addresses the various aspects I can think of for automation success. And that is something you will not get in a conference.”–Hans Buwalda  “An exciting, well-written, and wide-ranging collection of case studies with valuable realworld experiences, tips, lessons learned, and points to remember from real automation projects. This is a very useful book for anyone who needs the evidence to show managers and colleagues what works–and what does not work–on the automation journey.”–Isabel Evans, FBCS CITP, Quality Manager, Dolphin Computer Access  “Experiences of Test Automation first describes the essence of effective automated testing. It proceeds to provide many lifetimes worth of experience in this field, from a wide variety of situations. It will help you use automated testing for the right reasons, in a way that suits your organization and project, while avoiding the various pitfalls. It is of great value to anyone involved in testing–management, testers, and automators alike.”–Martin Gijsen, Independent Test Automation Architect  “This offering by Fewster and Graham is a highly significant bridge between test automation theory and reality. Test automation framework design and implementation is an inexact science begging for a reusable set of standards that can only be derived from a growing body of precedence; this book helps to establish such precedence. Much like predecessor court cases are cited to support subsequent legal decisions in a judicial system, the diverse case studies in this book may be used for making contemporary decisions regarding engagement in, support of, and educating others on software test automation framework design and implementation.”–Dion Johnson, Software Test Consultant and Principle Adviser to the Automated Testing Institute (ATI)  “Even with my long-established ‘test automation won’t work’ stance, this book did make me pause and ponder. It opened my mind and gave me a few ‘oh, I hadn’t thought of that’ moments. I would recommend this book as an initial reference for any organization wanting to introduce test automation.”–Audrey Leng  “This book is a stunning achievement. I believe that it is one of the best books ever written in test automation. Dot and Mark’s approach presenting 28 case studies is a totally new concept including eye-catching tips, good points, and lessons learned. The case studies are coming from life experiences, successes and failures, including several aspects of automation, different environments, and a mixture of solutions. Books are ‘the’ source of wisdom, and what a good idea for using storytelling to increase our learning through triggering our memories. This book is a must for everyone who is thinking of or involved in test automation at all levels. It is truly unique in its kind.”–Mieke Gevers