01 Testing Fundamentals
Software testing is an activity to check whether the actual results match the expected results and to ensure that the software system is Defect free. It involves execution of a software component or system component to evaluate one or more properties of interest.
Software testing also helps to identify errors, gaps or missing requirements in contrary to the actual requirements. It can be either done manually or using automated tools. Some prefer saying Software testing as a white box and Black Box Testing.
This tutorial introduces testing software to the audience and justifies its importance
Please be patient. The Video will load in some time. If you still face issue viewing video click here
Testing is important because software bugs could be expensive or even dangerous. Software bugs can potentially cause monetary and human loss, history is full of such examples.
Typically Testing is classified into three categories.
| Testing Category | Types of Testing |
|---|---|
| Functional Testing |
|
| Non-Functional Testing |
|
| Maintenance |
|
This is not the complete list as there are more than 150 types of testing types and still adding. Also, note that not all testing types are applicable to all projects but depend on nature & scope of the project.
---------------------------------------------------------------------------------------------------------
This tutorial introduces the seven basic principles of Software Testing every professional Software tester and QA professional should know.
Please be patient. The Video will load in some time.
Background
It is important that you achieve an optimum test results while conducting software testing without deviating from the goal. But how you determine that you are following right strategy for testing? For that, you need to stick to some basic testing principles. Here are the common seven testing principles that are widely practiced in the software industry.
To understand this, consider a scenario where you are moving a file from folder A to Folder B.
Think of all the possible ways you can test this.
Apart from the usual scenarios, you can also test the following conditions
Trying to move the file when it is Open
You do not have the security rights to paste the file in Folder B
Folder B is on a shared drive and storage capacity is full.
Folder B already has a file with the same name, infact the list is endless
Or suppose you have 15 input fields to test ,each having 5 possible values , the number of combinations to be tested would be 5^15
If you were to test the entire possible combinations project EXECUTION TIME & COSTS would rise exponentially. We need certain principles and strategies to optimize the testing effort
Yes!Exhaustive testing is not possible. Instead, we need the optimal amount of testing based on the risk assessment of the application.
And the million dollar question is, how do you determine this risk ?
To answer this let's do an exercise
In your opinion, Which operation is most likely to cause your Operating system to fail?
I am sure most of you would have guessed, Opening 10 different application all at the same time.
So if you were testing this Operating system, you would realize that defects are likely to be found in multi-tasking activity and need to be tested thoroughly which brings us to our next principle Defect Clustering
Defect Clustering which states that a small number of modules contain most of the defects detected. This is the application of the Pareto Principle to software testing: approximately 80% of the problems are found in 20% of the modules.
By experience, you can identify such risky modules. But this approach has its own problems
If the same tests are repeated over and over again , eventually the same test cases will no longer find new bugs.
Repetitive use of the same pesticide mix to eradicate insects during farming will over time lead to the insects developing resistance to the pesticide Thereby ineffective of pesticides on insects. The same applies to software testing. If the same set of repetitive tests are conducted, the method will be useless for discovering new defects.
To overcome this, the test cases need to be regularly reviewed & revised , adding new & different test cases to help find more defects.
Testers cannot simply depend on existing test techniques. He must look out continually to improve the existing methods to make testing more effective. But even after all this sweat & hard work in testing, you can never claim your product is bug free. To drive home this point , let's see this video of public launch of Windows 98
You think a company like MICROSOFT would not have tested their OS thoroughly & would risk their reputation just to see their OS crashing during its public launch!
Hence, testing principle states that - Testing talks about the presence of defects and don’t talk about the absence of defects. i.e. Software Testing reduces the probability of undiscovered defects remaining in the software but even if no defects are found, it is not a proof of correctness.
But what if , you work extra hard , taking all precautions & make your software product 99% bug-free. And the software does not meet the needs & requirements of the clients.
This leads us to our next principle, which states that- Absence of Error
It is possible that software which is 99% bug-free is still unusable. This can be the case if the system is tested thoroughly for the wrong requirement. Software testing is not mere finding defects, but also to check that software addresses the business needs. Absence of Error is a Fallacy i.e. Finding and fixing defects does not help if the system build is unusable and does not fulfill the user's needs & requirements.
To solve this problem , the next principle of testing states that Early Testing
Early Testing - Testing should start as early as possible in the Software Development Life Cycle. So that any defects in the requirements or design phase are captured in early stages. It is much cheaper to fix a Defect in early stages of testing. But how early one should start testing? It is recommended that you start finding the bug the moment the requirements are defined. More on this principle in a later training tutorial.
Testing is context dependent which basically means that the way you test an e-commerce site will be different from the way you test a commercial off the shelf application. All the developed software’s are not identical. You might use a different approach, methodologies, techniques and types of testing depending upon the application type. For instance testing, any POS system at a retail store will be different than testing an ATM machine.
| Principle 1 | Testing shows presence of defects |
| Principle 2 | Exhaustive testing is impossible |
| Principle 3 | Early Testing |
| Principle 4 | Defect Clustering |
| Principle 5 | Pesticide Paradox |
| Principle 6 | Testing is context dependent |
| Principle 7 | Absence of errors - fallacy |
This is so very untrue. Test Principles will help you create an effective Test Strategy and draft error catching test cases.
But learning testing principles is just like learning to drive for the first time.
Initially while you learn to drive, you pay attention to each and everything like gear shifts, speed, clutch handling, etc. But with experience, you just focus on driving the rest comes naturally. Such that you even hold conversations with other passengers in the car.
Same is true for testing principles. Experienced testers have internalized these principles to a level that they apply them even without thinking. Hence the myth that the principles are not use in practise is simply not true.
---------------------------------------------------------------------------------------------------------------------
This tutorial explains in detail the Software/System Development Life Cycle (SDLC) like the Waterfall cycle & Iterative cycle like RAID & Agile. And further, it proceeds to explain the V-Model of testing and STLC (Software Test Life Cycle).
Suppose, you are assigned a task, to develop a custom software for a client. Now, irrespective of your technical background, try and make an educated guess about the sequence of steps you will follow, to achieve the task.
The correct sequence would be.
| Different phases of Software Development Cycle |
Activities performed in each stage |
|---|---|
| Requirement Gathering stage |
|
| Design Stage | |
| Build Stage |
|
| Test Stage |
|
| Deployment stage |
|
| Maintenance stage |
|
All these levels constitute the waterfall method of software development lifecycle. As you may observe, that testing in the model starts only after implementation is done.
But if you are working in the large project, where the systems are complex, it's easy to miss out the key details in the requirements phase itself. In such cases, an entirely wrong product will be delivered to the client and you might have to start afresh with the project OR if you manage to note the requirements correctly but make serious mistakes in design and architecture of your software you will have to redesign the entire software to correct the error.
Assessments of thousands of projects have shown that defects introduced during requirements & design make up close to half of the total number of defects.
Also, the costs of fixing a defect increases across the development life cycle. The earlier in life cycle a defect is detected, the cheaper it is to fix it. As the say, "A stitch in time saves a nine."
To address this concern, the V model of testing was developed where for every phase, in the Development life cycle there is a corresponding Testing phase
Apart from V model, there are iterative development models, where development is carried in phases, with each phase adding a functionality to the software. Each phase comprises of its independent set of development and testing activities.
Good examples of Development lifecycles following iterative method are Rapid Application Development, Agile Development
SDLC is Software Development Life Cycle. It is the sequence of activities carried out by Developers to design and develop high-quality software.
Though SDLC uses the term ‘Development’, it does not involve just coding tasks done by developers but also incorporates the tasks contributed by testers and stakeholders.
In SDLC, test cases are created.
STLC is Software Testing Life Cycle. It consists of series of activities carried out by Testers methodologically to test your software product.
Though STLC uses the term “testing” it does not involve just testers, at some instances, they have to involve developers as well.
In STLC Test cases are executed.
Waterfall model is a sequential model divided into different phases of software development activity. Each stage is designed for performing specific activity during SDLC phase. Testing phase in waterfall model starts only after implementation of the system is done.
Testing is done within the SDLC.
V- model is an extension of the waterfall model. Unlike waterfall model, In V-model, there is a corresponding testing phase for each software development phase. Testing in V-model is done in parallel to SDLC stage.
Testing is done as a sub project of SDLC.
There are numerous development life cycle models. Development model selected for a project depends on the aims and goals of that project.
-------------------------------------------------------------------------------------------------------------------------------
Contrary to popular belief, Software Testing is not a just a single activity.
Software Testing Life Cycle (STLC) is defined as a sequence of activities conducted to perform Software Testing.
It consists of series of activities carried out methodologically to help certify your software product.
Diagram - Different stages in Software Test Life Cycle
Each of these stages have a definite Entry and Exit criteria; , Activities & Deliverables associated with it.
Entry Criteria:Entry Criteria gives the prerequisite items that must be completed before testing can begin.
Exit Criteria: Exit Criteria defines the items that must be completed before testing can be concluded
You have Entry and Exit Criteria for all levels in the Software Testing Life Cycle (STLC)
In an Ideal world you will not enter the next stage until the exit criteria for the previous stage is met. But practically this is not always possible. So for this tutorial, we will focus on activities and deliverables for the different stages in STLC life cycle. Lets look into them in detail.
During this phase, test team studies the requirements from a testing point of view to identify the testable requirements.
The QA team may interact with various stakeholders (Client, Business Analyst, Technical Leads, System Architects etc) to understand the requirements in detail.
Requirements could be either Functional (defining what the software must do) or Non Functional (defining system performance /security availability )
.Automation feasibility for the given testing project is also done in this stage. Activities
Deliverables
Typically , in this stage, a Senior QA manager will determine effort and cost estimates for the project and would prepare and finalize the Test Plan. In this phase, Test Strategy is also determined.
Activities
Deliverables
This phase involves creation, verification and rework of test cases & test scripts. Test data , is identified/created and is reviewed and then reworked as well. Activities
Deliverables
Test environment decides the software and hardware conditions under which a work product is tested. Test environment set-up is one of the critical aspects of testing process and can be done in parallel with Test Case Development Stage. Test team may not be involved in this activity if the customer/development team provides the test environment in which case the test team is required to do a readiness check (smoke testing) of the given environment. Activities
Deliverables
During this phase the testers will carry out the testing based on the test plans and the test cases prepared. Bugs will be reported back to the development team for correction and retesting will be performed. Activities
Deliverables
Testing team will meet , discuss and analyze testing artifacts to identify strategies that have to be implemented in future, taking lessons from the current test cycle. The idea is to remove the process bottlenecks for future test cycles and share best practices for any similar projects in future. Activities
Deliverables
Finally, summary of STLC Phases along with Entry and Exit Criteria
| STLC Stage | Entry Criteria | Activity | Exit Criteria | Deliverables |
|---|---|---|---|---|
| Requirement Analysis | Requirements Document available (both functional and non functional) Acceptance criteria defined. Application architectural document available. |
Analyse business functionality to know the business modules and module specific functionalities. Identify all transactions in the modules. Identify all the user profiles. Gather user interface/ authentication, geographic spread requirements. Identify types of tests to be performed. Gather details about testing priorities and focus. Prepare RequirementTraceability Matrix (RTM). Identify test environment details where testing is supposed to be carried out. Automation feasibility analysis (if required). |
Signed off RTM Test automation feasibility report signed off by the client |
RTM Automation feasibility report (if applicable) |
| Test Planning | Requirements Documents Requirement Traceability matrix. Test automation feasibility document. |
Analyze various testing approaches available Finalize on the best suited approach Preparation of test plan/strategy document for various types of testing Test tool selection Test effort estimation Resource planning and determining roles and responsibilities. |
Approved test plan/strategy document. Effort estimation document signed off. |
Test plan/strategy document. Effort estimation document. |
| Test case development | Requirements Documents RTM and test plan Automation analysis report |
Create test cases, test design, automation scripts (where applicable) Review and baseline test cases and scripts Create test data |
Reviewed and signed test Cases/scripts Reviewed and signed test data |
Test cases/scripts Test data |
| Test Environment setup | System Design and architecture documents are available Environment set-up plan is available |
Understand the required architecture, environment set-up Prepare hardware and software development requirement list Finalize connectivity requirements Prepare environment setup checklist Setup test Environment and test data Perform smoke test on the build Accept/reject the build depending on smoke test result |
Environment setup is working as per the plan and checklist Test data setup is complete Smoke test is successful |
Environment ready with test data set up Smoke Test Results. |
| Test Execution | Baselined RTM, Test Plan , Test case/scripts are available Test environment is ready Test data set up is done Unit/Integration test report for the build to be tested is available |
Execute tests as per plan Document test results, and log defects for failed cases Update test plans/test cases, if necessary Map defects to test cases in RTM Retest the defect fixes Regression Testing of application Track the defects to closure |
All tests planned are executed Defects logged and tracked to closure |
Completed RTM with execution status Test cases updated with results Defect reports |
| Test Cycle closure | Testing has been completed Test results are available Defect logs are available |
Evaluate cycle completion criteria based on - Time, Test coverage , Cost , Software Quality , Critical Business Objectives Prepare test metrics based on the above parameters. Document the learning out of the project Prepare Test closure report Qualitative and quantitative reporting of quality of the work product to the customer. Test result analysis to find out the defect distribution by type and severity |
------------------------------------------------------------------------------------------------------------------------