What is regression testing?
Regression testing, which is also known as repeated testing, is the process to ensure all the old functionalities still correctly works with the new changes. In other words, handling regression testing is to test an already tested application to find defects as a result of the changes. This is a usual step of any software development process and done by testing specialists. Testers do regression testing by re-executing the tests against the modified application to evaluate whether the revised code breaks anything which was working earlier. The only reason regression testing might not work is that changing or adding new code to a program can easily introduce errors into code that is not intended to be changed.
Regression testing is required in the following scenarios:
- When the code is modified according to the change of need in the requirement
- When new functionality is added
- While fixing defects
- While fixing performance issues
To explore how automated regression testing is used and how it is efficiently handled in real-life, here are your resource.
Why should regression testing be automated?
Let’s consider an example and how to perform regression testing in that situation:
At the beginning of a project, assuming we have six developers and two testers. The Agile model is set up every two weeks for a Sprint. Everything started.
In the first Sprint, we start with the basic features (e.g., about ten functions), the testers begin designing testing scenarios for testing (e.g. about 100 scenarios). The very first Sprint receives a good rating from customers.
In the second sprint, the developers continue to create new features – about 10, and testers also do things like in the first Sprint – with 100 new scripts plus 100 old scenarios that need to be retested. Well, only 200 scripts, everything is still in control.
In the next Sprint, the developers need to make eight new features and update the two old features due to new customer requirements. At this point, the two testers not only have to design test scripts for the 8 new features but also have to test and update 200 old scenarios. The whole implementation is about 300 scenarios. Did you feel there is something wrong?
Over the next few Sprints, the three developers still meet the number of features and the changing requirements, but with two testers, the number of scripts to create and update are much more. Tiredness begins to spread. The lack of time and the risk of miss bug is higher and higher. Too many problems arise.
Therefore, when regression testing is automated, it enables testers to check into a variety of changes and unusual cases in the production environment. Not all regressions are caused by new features or the consequences of routine bug fixes; database updates or new browser versions can also cause them. Regression can also be an issue with efficiency and speed. Automating those cases which are stable and repeatable allows manual testers to spend more time testing various environments and for merging more complicated cases at a higher level.
And what’s more, regression analysis is the key to success. It needs to deal with intelligence rather than hard work.
- Highest Return: Execute tests that contribute to high coverage of the requirements, then any others…
- Quickly Lower Risk: Execute tests for the most critical requirements, then any others…
- Practically Safe: Execute tests for all the critical requirements, then any others…
Especially since often ~20% of the test cases are covering ~80% of the business value
How to improve automated regression testing?
Two automated regression testing approaches are most used: data-driven testing and keyword-driven testing.
Data-driven testing is a framework where test input and output values are read from data files (data pools, ODBC sources, CSV files, Excel files, DAO objects, ADO objects, and such) and are loaded into variables in captured or manually coded scripts. In this approach, variables are used for both input values and output verification values. Navigation through the program, reading of the data files, and logging of test status and information are all coded in the test script. Let’s consider a simple example of a data-driven script for setting up Admin accounts in a system:
|
1 2 3 4 5 6 7 8 9 10 11 |
private void Auto_ID_400_test() { Common.InitializeWebDocument(); DataInit.Init(); BasePage.NavigateToHomePage(); HomePage.NavigateToSettings(); SettingBasePage.NavigateToBase(); Base.AdminSection.AddNewAdmin(Data.Read("NewAdmin")); } |
|
1 2 3 4 5 6 7 8 9 10 |
public static void AddNewAdmin(DataTable dataTable) { Report.Log(ReportLevel.Info, "Add new Admin:"); OpenAddnewAdminPopUp(); FillNewAdminData(dataTable); ConfirmAdmin(Read(dataTable.GetValue"NewAdminName")); } |
| NewAdminName | NewAdminPassword | NewAdminPasswordRepeat | AdminRole | AdminTenant | ExpectedResult |
| base_admin | admin123 | admin123 | Basic tenant administrator | tenantA | Created |
| dasklfkj@#@!# | admin333 | admin333 | Basic tenant administrator | tenantF | Invalid |
| invalid_admin | admin444 | admin444 | Basic tenant administrator | tenantA | Invalid |
| base_admin1 | admin555 | admin555 | Basic tenant administrator | tenantB | Created |
| base_admin8 | admin888 | admin888 | Basic tenant administrator | tenantB | Invalid |
| g&32LKLG | admin777 | admin777 | Basic tenant administrator | tenantB | Invalid |
| base_admin2 | admin456 | admin456 | Basic tenant administrator | tenantC | Created |
| base_admin9 | admin773 | admin773 | Basic tenant administrator | tenantE | Created |
Keyword-driven testing is a technique that separates much of the programming work from the actual test steps so that the test steps can be developed earlier and can often be maintained with only minor updates, even when the application or testing needs change significantly. The keyword-driven testing methodology divides test creation into two stages: Planning Stage, Implementation Stage.
Example: Test case for login: Navigate to the login page, type a user ID, type the password, Verify login.
Then, automated testing engineers develop scripts for the keywords:
| Actions | Keyword |
| Navigate to the login page | NavigateTo |
| Type a user ID | TypeUserID |
| Type the password | TypePass |
| Verify login | VerifyLogin |
This strategy does not require programming skills, so it much easier for non-technical people to contribute to the regression testing process. By way of explanation, it is more suitable for the projects catering to a large audience and requiring a broader focus than the data-driven testing.
Distinguish Re-testing and Regression Testing
| Re-testing | Regression Testing |
| Re-testing is carried out to confirm the test cases that failed in the final execution are passing after the defects are fixed | Regression Testing is carried out to verify whether a recent program or code change has not adversely affected existing features |
| The Defect fixes do Re-testing | The purpose of Regression Testing is that new code changes should not have any side effects to existing functionalities |
| Defect verification is the part of re-testing | Defect verification is not the part of Regression Testing |
| Priority of re-testing is higher than regression testing, so it is carried out before regression testing | Based on the project and availability of resources, Regression Testing can be carried out parallel with Re-testing |
| You cannot automate the test cases for Retesting | You can do automation for regression testing; Manual Testing could be expensive and time-consuming |
| Re-testing is a planned testing | Regression testing is known as a generic testing |
| Retesting is done only for failed test cases | Regression testing is done for passed test cases |
| Re-testing makes sure that the original fault has been corrected | Regression testing checks for unexpected side-effects |
| Re-testing executes a defect with the same data and the same environment with different inputs with a new build | Regression testing is only done when there is any modification or changes become mandatory in an existing project |
| Test cases for retesting cannot be obtained before start testing. | Test cases for regression testing can be obtained from the functional specification, user tutorials and manuals, and defect reports in regards to corrected problems |
Learn more:
- What is Regression Testing? Test Cases, Tools & Examples (by Guru99)
- Different Types of Test Design Techniques You Need to Know
- All Things You Need To Know About Exhaustive Testing