Testing Computer Software

CEM KANER consults on technical and software development management issues and teaches about software testing at local universities and at several software companies. He also practices law, usually representing individual developers, small development services companies, and customers. He founded and hosts the Los Altos Workshops on Software Testing. Kaner is the senior author of Bad Software: What to Do When Software Fails (Wiley). JACK FALK consults on software quality management and software engineering management. Jack is certified in Software Quality Engineering by the American Society of Quality. He is Vice Chair of the Santa Clara Valley Software Quality Association and an active participant in the Los Altos Workshops on Software Testing. HUNG Q. NGUYEN is Founder, President, and CEO of softGear technology. He has worked in the computer software and hardware industries, holding management positions in engineering, quality assurance, testing, product development, and information technology, as well as making significant contributions as a tester and programmer. He is an ASQ-Certified Quality Engineer, and a senior member and San Francisco Section Certification Chairman of the American Society for Quality.

• Preface xiiiNotes on the book’s structure and layout xviiAcknowledgments xxiSECTION 1—FUNDAMENTALS1. An example test series 1The first cycle of testing 1The second cycle of testing 11What will happen in later cycles of testing 162. The objectives and limits of testing 17You can’t test a program completely 17The tester’s objective: Program verification? 23So, why test? 253. Test types and their place in the software development process 27Overview of the software development stages 30Planning stages 32Testing during the planning stages 33Design stages 35Testing during the design stages 39Glass box code testing is part of the coding stage 41Regression testing 49Black box testing 50Maintenance 574. Software errors 59Quality 59What is a software error? 60Categories of software errors 605. Reporting and analyzing bugs 65Write Problem Reports immediately 66Content of the Problem Report 66Characteristics of the Problem Report 74Analysis of a reproducible bug 76Tactics for analyzing a reproducible bug 79Making a bug reproducible 82SECTION 2—SPECIFIC TESTING SKILLS6. The problem tracking system 87The prime objective of a problem tracking system 90The tasks of the system 90Problem tracking overview 90The users of the tracking system 97Mechanics of the database 106Further thoughts on problem reporting 115Glossary 1217. Test case design 123Characteristics of a good test 124Equivalence classes and boundary values 125Visible state transitions 132Race conditions and other time dependencies 133Load testing 134Error guessing 135Function equivalence testing: automation, sensitivity analysis & random input 135Regression testing: checking whether a bug fix worked 139Regression testing: the standard battery of tests 140Executing the tests 1418. Testing printers (and other devices) 143Some general issues in configuration testing 144Printer testing 1469. Localization testing 169Was the base code changed? 170Work with someone fluent in the language 170Is the text independent from the code? 171Translated text expands 171Character sets 171Keyboards 172Text filters 172Loading, saving, importing, and exporting high and low ASCII 173Operating system language 173Hot keys 173Garbled in translation 173Error message identifiers 174Hyphenation rules 174Spelling rules 174Sorting rules 174Uppercase and lowercase conversion 174Underscoring rules 174Printers 175Sizes of paper 175CPU’s and video 175Rodents 175Data formats and setup options 175Rulers and measurements 176Culture-bound graphics 176Culture-bound output 176European product compatibility 176Memory availability 176Do GUIs solve the problem? 177Automated testing 17710. Testing user manuals 179Effective documentation 179The documentation tester’s objectives 180How testing documentation contributes to software reliability 181Become the technical editor 182Working with the manual through its development stages 183Online help 18811. Testing tools 189Fundamental tools 189Automated acceptance and regression tests 191Standards 197Translucent-box testing 20012. Test planning and test documentation 203The overall objective of the test plan: product or tool? 204Detailed objectives of test planning and documentation 205What types of tests to cover in test planning documents 210A strategy for developing components of test planning documents 213Components of test planning documents 217Documenting test materials 242A closing thought 253SECTION 3—MANAGING TESTING PROJECTS AND GROUPS13. Tying it together 255Software development tradeoffs 257Software development models 258Quality-related costs 264The development time line 266Product design 267Fragments coded: first functionality 274Almost alpha 275Alpha 277Pre-beta 286Beta 286User interface (UI) freeze 293Pre-final 295Final integrity testing 299Release 301Project post-mortems 30114. Legal consequences of defective software 303Breach of contract 305Torts: lawsuits involving fault 317Whistle blowing 34015. Managing a testing group 343Managing a testing group 344The role of the testing group 345A test group is not an unmixed blessing 349An alternative? Independent test agencies 350Scheduling tips 352Your staff 359Appendix: common software errors 363User interface errors 375Error handling 396Boundary-related errors 399Calculation errors 401Initial and later states 403Control flow' errors 406Errors in handling or interpreting data 416Race conditions 421Load conditions 423Hardware 427Source, version, and ID control 430Testing errors 432References 437Index 451About the Authors 480

"I enjoyed reading Testing computer software. The text contains numerous highlights Offering practical advice, authoritative figures you can cite to customers and higher management, and entertaining anecdotes to share with coworkers Although some sections need updating, I still think it is a valuable training and reference source for software testers, managers, and developers." --Diomidis Spinellis; IEEE software magazine (May /June 2001))"Deep insight and a great deal of experience is contained in this book" (Database & Network Journal, Vol 30/5 2000)