| |
Book_Software_Testing_SAMS
Page history last edited by Anonymous 1 yr ago
Book Details
Software Testing
- Ron Patton
- Sams
- August 5, 2005
Top
Introduction
- This book will introduce you to the basics of software testing, teaching you not just the fundamental technical skills but also the supporting skills necessary to become a successful software tester.
- You will learn how to immediately find problems in any computer program, how to plan an effective test approach, how to clearly report your findings, and how to tell when your software is ready for release.
About the Second Edition
What This Book Will Do for You
In this book you will learn something about nearly every aspect of software testing:
- How software testing fits into the software development process
- Basic and advanced software testing techniques
- Applying testing skills to common testing tasks
- Improving test efficiency with automation
- Planning and documenting your test effort
- Effectively reporting the problems you find
- Measuring your test effort and your product's progress
- Knowing the difference between testing and quality assurance
- Finding a job as a software tester
How This Book Is Organized
- This book is designed to lead you through the essential knowledge and skills necessary to become a good software tester.
- Software testing is not about banging on the keyboard hoping you'll eventually crash the computer.
- A great deal of science and engineering is behind it, lots of discipline and planning, and there can be lots of fun, too, as you'll soon see.
Part I: The Big Picture
The chapters in Part I lay the foundation for this book by showing you how software products are developed and how software testing fits into the overall development process.
You'll see the importance of software testing and gain an appreciation for the magnitude of the job.
- Chapter 1, "Software Testing Background," helps you understand exactly what a software bug is, how serious they can be, and why they occur.
You'll learn what your ultimate goal is as a software tester and what traits will help make you a good one. - Chapter 2, "The Software Development Process," gives you an overview of how a software product is created in the corporate world.
You'll learn what components typically go into software, what types of people contribute to it, and the different process models that can be used. - Chapter 3, "The Realities of Software Testing," brings a reality check to how software is developed.
You'll see why no matter how hard you try, software can never be perfect. You'll also learn a few fundamental terms and concepts used throughout the rest of this book.
Part II: Testing Fundamentals
The chapters in Part II teach you the fundamental approaches to software testing.
The work of testing software is divided into four basic areas, and you will see the techniques used for each one:
Part III: Applying Your Testing Skills
The chapters in Part III take the techniques that you learned in Part II and apply them to some real-world scenarios that you'll encounter as a software tester:
- Chapter 8, "Configuration Testing," teaches you how to organize and perform software testing on different hardware configurations and platforms.
- Chapter 9, "Compatibility Testing," teaches you how to test for issues with different software applications and operating systems interacting with each other.
- Chapter 10, "Foreign-Language Testing," shows you that a whole world of software is out there and
that it's important to test for the special problems that can arise when software is translated into other languages. - Chapter 11, "Usability Testing," teaches you how to apply your testing skills when checking a software application's user interface and how to assure that your software is accessible to the disabled.
- Chapter 12, "Testing the Documentation," explains how to examine the software's documentation such as help files, user manuals, even the marketing material, for bugs.
- Chapter 13, "Testing for Software Security," shows you how to find bugs that allow hackers to gain access to (supposedly) secure computer systems and data.
- Chapter 14, "Website Testing," takes everything you've learned so far and applies it to a present-day situation.
You'll see how something as simple as testing a website can encompass nearly all aspects of software testing.
Part IV: Supplementing Your Testing
The chapters in Part IV show you how to improve your test coverage and capability by leveraging both technology and people to perform your testing more efficiently and effectively:
Part V: Working with Test Documentation
The chapters in Part V cover how software testing is documented so that its plans, bugs, and results can be seen and understood by everyone on the project team:
Part VI: The Future
The chapters in Part VI explain where the future lies in software testing and set the stage for your career:
- Chapter 21, "Software Quality Assurance," teaches you the big difference between software testing and quality assurance.
You'll learn about different software industry goals such as ISO 9000 and the Capabilities Maturity Model and what it takes to achieve them. - Chapter 22, "Your Career as a Software Tester," gives you that kick in the behind to go out and be a software tester.
You'll learn what types of jobs are available and where to look for them. You'll also find many pointers to more information.
Top
Chapter 1. Software Testing Background
- In 1947, computers were big, room-sized machines operating on mechanical relays and glowing vacuum tubes.
The state of the art at the time was the Mark II, a behemoth being built at Harvard University. - Technicians were running the new computer through its paces when it suddenly stopped working.
They scrambled to figure out why and discovered, stuck between a set of relay contacts deep in the bowels of the computer, a moth. It had apparently flown into the system, attracted by the light and heat, and was zapped by the high voltage when it landed on the relay. - The computer bug was born. Well, okay, it died, but you get the point.
- Highlights of this chapter include:
- How software bugs impact our lives
- What bugs are and why they occur
- Who software testers are and what they do
Infamous Software Error Case Studies
Disney's Lion King, 1994 - 1995
- In the fall of 1994, the Disney company released its first multimedia CD-ROM game for children, The Lion King Animated Storybook.
- It turns out that Disney failed to test the software on a broad representation of the many different PC models available on the market.
- The software worked on a few systems - likely the ones that the Disney programmers used to create the game - but not on the most common systems that the general public had.
Intel Pentium Floating-Point Division Bug, 1994
- Enter the following equation into your PC's calculator: (4195835 / 3145727) * 3145727 - 4195835
- If the answer is zero, your computer is just fine.
If you get anything else, you have an old Intel Pentium CPU with a floating-point division bug - a software bug burned into a computer chip and reproduced over and over in the manufacturing process.
NASA Mars Polar Lander, 1999
- The result was catastrophic, but the reason behind it was simple. The lander was tested by multiple teams.
- One team tested the leg fold-down procedure and another the landing process from that point on.
The first team never looked to see if the touch-down bit was set it wasn't their area; the second team always reset the computer, clearing the bit, before it started its testing. Both pieces worked perfectly individually, but not when put together.
Patriot Missile Defense System, 1991
- The U.S. Patriot missile defense system was first put to use in the Gulf War as a defense for Iraqi Scud missiles.
- Although there were many news stories touting the success of the system, it did fail to defend against several missiles,
including one that killed 28 U.S. soldiers in Dhahran, Saudi Arabia. - Analysis found that a software bug was the problem.
A small timing error in the system's clock accumulated to the point that after 14 hours, the tracking system was no longer accurate. In the Dhahran attack, the system had been operating for more than 100 hours.
The Y2K (Year 2000) Bug, circa 1974
Dangerous Viewing Ahead, 2004
- On April 1, 1994, a message was posted to several Internet user groups and then quickly circulated as an email
that a virus was discovered embedded in several JPEG format pictures available on the Internet. - The warning stated that simply opening and viewing the infected pictures would install the virus on your PC.
- Variations of the warning stated that the virus could damage your monitor and that Sony Trinitron monitors were "particularly susceptible."
- Many heeded the warning, purging their systems of JPEG files.
Some system administrators even went so far as to block JPEG images from being received via email on the systems.
- Ten years later, in the fall of 2004, a proof-of-concept virus was created, proving that a JPEG picture could be loaded with a virus that would infect the system used to view it.
- Software patches were quickly made and updates distributed to prevent such a virus from spreading.
Top
What Is a Bug?
- It can be inconvenient, as when a computer game doesn't work properly, or it can be catastrophic, resulting in the loss of life.
- It can cost only pennies to fix but millions of dollars to distribute a solution.
- In the examples, above, it was obvious that the software didn't operate as intended.
- As a software tester you'll discover that most failures are hardly ever this obvious.
Most are simple, subtle failures, with many being so small that it's not always clear which ones are true failures, and which ones aren't.
Terms for Software Failures
- Depending on where you're employed as a software tester, you will use different terms to describe what happens when software fails. Here are a few:
Defect, Variance, Fault, Failure, Problem, Inconsistency, Error, Feature, Incident, Bug, Anomaly (There's also a list of unmentionable terms, but they're most often used privately among programmers.) - You might be amazed that so many names could be used to describe a software failure.
Why so many? It's all really based on the company's culture and the process the company uses to develop its software. - They also have inferred meanings by how they're used in day-to-day conversation. For example:
- fault, failure, and defect tend to imply a condition that's really severe, maybe even dangerous.
It doesn't sound right to call an incorrectly colored icon a fault. These words also tend to imply blame: "It's his fault that the software failed." - Anomaly, incident, and variance don't sound quite so negative and are often used to infer unintended operation rather than all-out failure.
"The president stated that it was a software anomaly that caused the missile to go off course." - Problem, error, and bug are probably the most generic terms used.
- It's interesting that some companies and product teams will spend hours and hours of precious development time arguing and debating which term to use.
A well-known computer company spent weeks in discussion with its engineers before deciding to rename Product Anomaly Reports (PARs) to Product Incident Reports (PIRs). Countless dollars were spent in the process of deciding which term was better. Once the decision was made, all the paperwork, software, forms, and so on had to be updated to reflect the new term. It's unknown if it made any difference to the programmer's or tester's productivity.
- So, why bring this topic up? It's important as a software tester to understand the personality behind the product development team you're working with.
- How they refer to their software problems is a tell-tale sign of how they approach their overall development process.
- Are they cautious, careful, direct, or just plain blunt?
- Although your team may choose a different name, in this book, all software problems will be called bugs.
- It doesn't matter if it's big, small, intended, unintended, or someone's feelings will be hurt because they create one.
- There's no reason to dice words. A bug's a bug's a bug.
Software Bug: A Formal Definition
- First, you need a supporting term: product specification.
- A product specification, sometimes referred to as simply a spec or product spec, is an agreement among the software development team.
- It defines the product they are creating, detailing what it will be, how it will act, what it will do, and what it won't do.
- This agreement can range in form from a simple verbal understanding, an email, or a scribble on a napkin, to a highly detailed, formalized written document.
- In Chapter 2, "The Software Development Process," you will learn more about software specifications and the development process, but for now, this definition is sufficient.
- For the purposes of this book and much of the software industry, a software bug occurs when one or more of the following five rules is true:
- The software doesn't do something that the product specification says it should do.
- The software does something that the product specification says it shouldn't do.
- The software does something that the product specification doesn't mention.
- The software doesn't do something that the product specification doesn't mention but should.
- The software is difficult to understand, hard to use, slow, or -- in the software tester's eyes -- will be viewed by the end user as just plain not right.
- To better understand each rule, try the following example of applying them to a calculator.
- The specification for a calculator probably states that it will perform correct addition, subtraction, multiplication, and division.
If you, as the tester, receive the calculator, press the + key, and nothing happens, that's a bug because of Rule #1. If you get the wrong answer, that's also a bug because of Rule #1. - The product spec might state that the calculator should never crash, lock up, or freeze.
If you pound on the keys and get the calculator to stop responding to your input, that's a bug because of Rule #2. - Suppose that you receive the calculator for testing and find that besides addition, subtraction, multiplication, and division, it also performs square roots.
Nowhere was this ever specified. An ambitious programmer just threw it in because he felt it would be a great feature. This isn't a feature - it's really a bug because of Rule #3. The software is doing something that the product specification didn't mention. This unintended operation, although maybe nice to have, will add to the test effort and will likely introduce even more bugs. - The fourth rule may read a bit strange with its double negatives, but its purpose is to catch things that were forgotten in the specification.
You start testing the calculator and discover when the battery gets weak that you no longer receive correct answers to your calculations. No one ever considered how the calculator should react in this mode. A bad assumption was made that the batteries would always be fully charged. You expected it to keep working until the batteries were completely dead, or at least notify you in some way that they were weak. Correct calculations didn't happen with weak batteries, and it wasn't specified what should happen. Rule #4 makes this a bug. - Rule #5 is the catch-all. As a tester you are the first person to really use the software.
If you weren't there, it would be the customer using the product for the first time. If you find something that you don't feel is right, for whatever reason, it's a bug. In the case of the calculator, maybe you found that the buttons were too small. Maybe the placement of the = key made it hard to use. Maybe the display was difficult to read under bright lights. All of these are bugs because of Rule #5. - Every person who uses a piece of software will have different expectations and opinions as to how it should work.
It would be impossible to write software that every user thought was perfect. As a software tester, you should keep this in mind when you apply Rule #5 to your testing. Be thorough, use your best judgment, and, most importantly, be reasonable. Your opinion counts, but, as you'll learn in later chapters, not all the bugs you find can or will be fixed
Top
Why Do Bugs Occur?
- There are several reasons specifications are the largest bug producer.
In many instances a spec simply isn't written. Other reasons may be that the spec isn't thorough enough, it's constantly changing, or it's not communicated well to the entire development team. Planning software is vitally important. If it's not done correctly, bugs will be created. - The next largest source of bugs is the design. This is where the programmers lay out their plan for the software.
Compare it to an architect creating the blueprints for a building. Bugs occur here for the same reason they occur in the specification. It's rushed, changed, or not well communicated. - Coding errors may be more familiar to you if you're a programmer.
Typically, these can be traced to the software's complexity, poor documentation (especially in code that's being updated or revised), schedule pressure, or just plain dumb mistakes. It's important to note that many bugs that appear on the surface to be programming errors can really be traced to specification and design errors. It's quite common to hear a programmer say, "Oh, so that's what it's supposed to do. If somebody had just told me that I wouldn't have written the code that way." - The other category is the catch-all for what's left. Some bugs can be blamed on false positives, conditions that were thought to be bugs but really weren't.
There may be duplicate bugs, multiple ones that resulted from the same root cause. Some bugs can also be traced to testing errors. In the end, these bugs (or what once were thought of as bugs) turn out to not be bugs at all and make up a very small percentage of all the bugs reported.
Top
The Cost of Bugs
- The costs are logarithmic that is, they increase tenfold as time increases.
- A bug found and fixed during the early stages when the specification is being written might cost next to nothing, or $1 in our example.
- The same bug, if not found until the software is coded and tested, might cost $10 to $100.
- If a customer finds it, the cost could easily be thousands or even millions of dollars.
- As an example of how this works, consider the Disney Lion King case discussed earlier.
The root cause of the problem was that the software wouldn't work on a very popular PC platform. - If, in the early specification stage, someone had researched what PCs were popular
and specified that the software needed to be designed and tested to work on those configurations, the cost of that effort would have been minimal. - If that didn't occur, a backup would have been for the software testers to collect samples of the popular PCs and verify the software on them.
They would have found the bug, but it would have been more expensive to fix because the software would have to be debugged, fixed, and retested. - The development team could have also sent out a preliminary version of the software to a small group of customers in what's called a beta test.
Those customers, chosen to represent the larger market, would have likely discovered the problem. - As it turned out, however, the bug was completely missed until many thousands of CD-ROMs were created and purchased.
Disney ended up paying for telephone customer support, product returns, replacement CD-ROMs, as well as another debug, fix, and test cycle. It's very easy to burn up your entire product's profit if serious bugs make it to the customer.
Top
What Exactly Does a Software Tester Do?
- You may run across product teams who want their testers to simply confirm that the software works, not to find bugs.
Reread the case study about the Mars Polar Lander, and you'll see why this is the wrong approach. If you're only testing things that should work and setting up your tests so they'll pass, you will miss the things that don't work. You will miss the bugs. - As a software tester you shouldn't be content at just finding bugs you should think about how to find them sooner in the development process, thus making them cheaper to fix.
The goal of a software tester is to find bugs and find them as early as possible.
- But, finding bugs, even finding them early, isn't enough.
Remember the definition of a bug. You, the software tester, are the customer's eyes, the first one to see the software. You speak for the customer and must seek perfection. The goal of a software tester is to find bugs, find them as early as possible, and make sure they get fixed. - This final definition is very important. Commit it to memory and refer back to it as you learn the testing techniques discussed throughout the rest of this book.
- It's important to note that "fixing" a bug does not necessarily imply correcting the software.
- It could mean adding a comment in the user manual or providing special training to the customers.
It could require changing the statistics that the marketing group advertises or even postponing the release of the buggy feature. You'll learn throughout this book that although you're seeking perfection and making sure that the bugs get fixed, that there are practical realities to software testing. Don't get caught in the dangerous spiral of unattainable perfection.
Top
What Makes a Good Software Tester?
- At first glance, it may appear that a software tester's job would be easier than a programmer's.
Breaking code and finding bugs must surely be easier than writing the code in the first place. - Surprisingly, it's not.
The methodical and disciplined approach to software testing that you'll learn in this book requires the same hard work and dedication that programming does. - It involves very similar skills, and although a software tester doesn't necessarily need to be a full-fledged programmer, having that knowledge is a great benefit.
- Today, most mature companies treat software testing as a technical engineering profession.
- They recognize that having trained software testers on their project teams and allowing them to apply their trade early in the development process allows them to build better quality software.
- Unfortunately, there are still a few companies that don't appreciate the challenge of software testing and the value of well-done testing effort.
In a free market society, these companies usually aren't around for long because the customers speak with their wallets and choose not to buy their buggy products. A good test organization (or the lack of one) can make or break a company.
- Here's a list of traits that most software testers should have:
- They are explorers. Software testers aren't afraid to venture into unknown situations.
They love to get a new piece of software, install it on their PC, and see what happens. - They are troubleshooters. Software testers are good at figuring out why something doesn't work. They love puzzles.
- They are relentless. Software testers keep trying.
They may see a bug that quickly vanishes or is difficult to re-create. Rather than dismiss it as a fluke, they will try every way possible to find it. - They are creative. Testing the obvious isn't sufficient for software testers. Their job is to think up creative and even off-the-wall approaches to find bugs.
- They are (mellowed) perfectionists. They strive for perfection, but they know when it becomes unattainable and they're okay with getting as close as they can.
- They exercise good judgment. Software testers need to make decisions about what they will test, how long it will take, and if the problem they're looking at is really a bug.
- They are tactful and diplomatic. Software testers are always the bearers of bad news. They have to tell the programmers that their baby is ugly.
Good software testers know how to do so tactfully and professionally and know how to work with programmers who aren't always tactful and diplomatic. - They are persuasive. Bugs that testers find won't always be viewed as severe enough to be fixed.
Testers need to be good at making their points clear, demonstrating why the bug does indeed need to be fixed, and following through on making it happen.
- A fundamental trait of software testers is that they simply like to break things.
They live to find those elusive system crashes. They take great satisfaction in laying to waste the most complex programs. They're often seen jumping up and down in glee, giving each other high-fives, and doing a little dance when they bring a system to its knees. It's the simple joys of life that matter the most.
- Lastly, if you're an expert in some non-computer field, your knowledge can be invaluable to a software team creating a new product.
- Software is being written to do just about everything today.
- Your knowledge of teaching, cooking, airplanes, carpentry, medicine, or whatever would be a tremendous help finding bugs in software for those areas.
Top
Chapter 2. The Software Development Process
- To be an effective software tester, it's important to have at least a high-level understanding of the overall process used to develop software.
- If you write small programs as a student or hobbyist, you'll find that the methods you use are much different from what big companies use to develop software.
- The creation of a new software product may involve dozens, hundreds, even thousands of team members all playing different roles and working together under tight schedules.
- The specifics of what these people do, how they interact, and how they make decisions are all part of the software development process.
- The goal of this chapter isn't to teach you everything about the software development process - that would take an entire book!
- The goal is to give you an overview of the all the pieces that go into a software product and a look at a few of the common approaches in use today.
- With this knowledge you'll have a better understanding of how best to apply the software testing skills you learn in the later chapters of this book.
- The highlights of this chapter include:
- What major components go into a software product
- What different people and skills contribute to a software product
- How software progresses from an idea to a final product
Top
Product Components
- In reality, many hidden pieces go into making that software. There are also many pieces that "come in the box" that are often taken for granted or might even be ignored.
- Although it may be easy to forget about all those parts, as a software tester, you need to be aware of them, because they're all testable pieces and can all have bugs.
What Effort Goes Into a Software Product?
- The term used in the software industry to describe a software product component that's created and passed on to someone else is deliverable.
- The easiest way to explain what all these deliverables are is to organize them into major categories.
Customer Requirements
- Software is written to fulfill some need that a person or a group of people has. Let's call them the customer.
- To properly fill that need, the product development team must find out what the customer wants.
- Some teams simply guess, but most collect detailed information in the form of surveys, feedback from previous versions of the software,
competitive product information, magazine reviews, focus groups, and numerous other methods, some formal, some not. - All this information is then studied, condensed, and interpreted to decide exactly what features the software product should have.
- A popular means to get direct feedback from potential customers of a software product is to use focus groups.
- Focus groups are often organized by independent survey companies who set up offices in shopping malls.
- The surveyors typically walk around the mall with a clipboard and ask passers-by if they want to take part in a study.
- They'll ask a few questions to qualify you such as "Do you have a PC at home? Do you use software X? How much time do you spend online?" And so on.
- If you fit their demographic, they'll invite you to return for a few hours to participate with several other people in a focus group.
- There, you'll be asked more detailed questions about computer software. You may be shown various software boxes and be asked to choose your favorite.
Or, you may discuss as a group features you'd like to see in a new product. Best of all, you get paid for your time. - Most focus groups are conducted in such a way that the software company requesting the information is kept anonymous.
But, it's usually easy to figure out who they are.
Specifications
- The result of the customer requirements studies is really just raw data.
- It doesn't describe the proposed product, it just confirms whether it should (or shouldn't) be created and what features the customers want.
- The specifications take all this information plus any unstated but mandatory requirements and truly define what the product will be, what it will do, and how it will look.
- The format of specifications varies greatly.
- Some companies - especially those developing products for the government, aerospace, financial, and medical industries - use a very rigorous process with many checks and balances.
- The result is an extremely detailed and thorough specification that's locked down, meaning that it can't change except under very extreme conditions.
- Everyone on the development team knows exactly what they are creating.
Schedules
Software Design Documents
Test Documents
What Parts Make Up a Software Product?
- It's also important to realize that when the product is ready to be boxed up and shipped out the door, it's not just the code that gets delivered.
Numerous supporting parts go along with it. Since all these parts are seen or used by the customer, they need to be tested too. - It's unfortunate, but these components are often overlooked in the testing process.
- These seem like simple things to test, but no one probably even gave them a second look before the product was okayed for release. You will.
- Later in this book you'll learn about these non-software pieces and how to properly test them.
Until then, keep this list in mind as just a sampling of what more there is to a software product than just the code: Help files, User's manual, Samples and examples, Labels and stickers, Product support info, Icons and art, Error messages, Ads and marketing material, Setup and installation, Readme file
Don't Forget to Test Error Messages
Top
Software Project Staff
Top
Software Development Lifecycle Models
- The process used to create a software product from its initial conception to its public release is known as the software development lifecycle model.
- As discussed previously, there are many different methods that can be used for developing software, and no model is necessarily the best for a particular project.
There are four frequently used models, with most others just variations of these: - Big-Bang
- Code-and-Fix
- Waterfall
- Spiral
- Each model has its advantages and disadvantages.
- As a tester, you will likely encounter them all and will need to tailor your test approach to fit the model being used for your current project.
Refer to these model descriptions as you read the rest of this book and think about how you would apply the various testing techniques you learn under each of them.
Big-Bang Model
- The big-bang model for software development follows much the same principle.
A huge amount of matter (people and money) is put together, a lot of energy is expended often violently and out comes the perfect software product…or it doesn't. - The beauty of the big-bang method is that it's simple.
There is little if any planning, scheduling, or formal development process. All the effort is spent developing the software and writing the code. - It's a process that is used if the product requirements aren't well understood and the final release date is completely flexible.
It's also important to have very flexible customers, too, because they won't know what they're getting until the very end.
- In most cases, there is little to no formal testing done under the big-bang model.
If testing does occur, it's squeezed in just before the product is released. It's a mystery why testing is sometimes inserted into this model, but it's probably to make everyone feel good that some testing was performed. - If you are called in to test a product under the big-bang model, you have both an easy and a difficult task.
Because the software is already complete, you have the perfect specification the product itself. And, because it's impossible to go back and fix things that are broken, your job is really just to report what you find so the customers can be told about the problems.
- The downside is that, in the eyes of project management, the product is ready to go, so your work is holding up delivery to the customer.
The longer you take to do your job and the more bugs you find, the more contentious the situation will become. Try to stay away from testing in this model.
Book_Software_Testing_SAMS
|
|
Tip: To turn text into a link, highlight the text, then click on a page or file from the list above.
|
|
|
|
|
Comments (0)
You don't have permission to comment on this page.