GAMP -Automated Test Execution and Computerized Test Management Tools

As I've promised on my last post – Good Automated Manufacturing Practice (GAMP) Test Execution – part 1 I will now continue to elaborate on the execution of a test using automated (computerized) test tools. These tools should be conducted in accordance with the Test Plan or Strategy. This should require that all automated tools used to test GxP systems have been subjected to assessment and appropriate validation prior to their use, in order to establish s a high degree of assurance that they will perform as intended, and return accurate and secure results.

Before discussing the benefits and use of automated test tools, it is useful to describe the difference between a computerized test management tool and automated testing.

A computerized test management tool facilitates the task of Test Case and Test Script authoring, review and approval (pre and post execution), test execution, and test deviation management. This is accomplished using workflow enabled processes, electronic test artifact management (i.e., Test Cases, Test Scripts, Deviation Reports), and possibly electronic signatures.

However, even with computerized test management, the actual test execution is performed by the tester, even when this task is aided by the presentation of on-screen test scripts and electronic capture of test evidence. The tester must still manually follow the test steps, provide inputs to the system under test, and record the test evidence.

With automated test execution, the computerized test tool also executes the actual test and records the test evidence. This can significantly reduce the test execution time. Because most automated test execution is based upon initial manual test execution (where the tool often 'records' the manual actions of the tester), automated testing is most often used when executing tests during a second test cycle and is well sited to regression testing.

Benefits of Computerized Test Management Tools and Automated Test Execution

When planned properly, automating the test activities can bring considerable benefits within the project life cycle. However, if poorly planned, computer based tools and test automation may be difficult, complex, and time consuming task, with little or no return on investment.

Computerized test management tools can significantly reduce the amount of paper used during testing and provide helpful test management support. This includes the ability to report on the test activities status and facilitate test activities using workflow. In most large testing projects, the use of such tool can reduce testing timescales.

Not every aspect of a computerized system can be automatically tested. A key factor in the successful use of such tools is the early identification of the types of tests that should be automated, and the types of tests to be executed manually.

While computerized test management tools and automated testing can both provide benefits, they should be considered separately. Some project may benefit from the use of computerized test management tools, but may derive little benefits using automated test tools.

Before deciding on the usefulness of computerized test management and automated test tools, the following points should be considered:

Will the tools be used in a single project or more broadly within the organization? Typically in a large project or organization, a wide use is necessary to provide return on investment.

How many test cycles are typically executed, and how much regression testing is required? This will help determine the need for automated testing.

Will the tool be used by a dedicated test team, or in all projects? This has an impact on on-going administration and training requirements.

The use of computerized test management and automated test tools need to be regarded as a SW development activity that has its own life cycle. These tools will have their own implementation and validation life cycle. This will place additional requirements on the test team, particularly in terms of the skills required by test script developers.

The use of these tools is only effective if supported by adequate processes and if the staff has the necessary skills to use the tools effectively. The usability of a test tool is significantly enhanced if it is application-interdependent (i.e., used to test multiple application).

Easy maintenance of a test suite is crucial for a successful implementation of an automated test tool.

Automated test tools which include a version and configuration management can significantly improve the effectiveness and efficiency of retesting.

Features of Computerized Test Management and Automated Test Tools

Different test management and automated test tools are used In the IT industry, and the architecture of the application under test often determines the most suited tool. Most tools are purchased as configurable off-the-shelf SW packages, and the test team then configures the package locally to meet the particular usage required.

Before acquiring computerized test management tools or automated test tools, it is important that an organization understands and defines the requirements and form of use.

Test Management Tools

Test management tools may be classified into four general types:

General computerized test management tools

Developer-oriented tools

Functional test tools

Load and performance related tools

General Computerized Test Management Tools

Manage testing across the project.

Facilitates authoring, review and approval of Test Cases and Test Scripts, often using workflow and electronic signatures.

Allow developers, testers and project managers to track tests being run on various applications.

Trace tests back to their original requirements.

Summaries the defects found during the testing process.

The test management tools help to determine whether changes need to be made to the automated test tools themselves, as a result of changes to the applications being tested. However, they do not provide the facility to automate test execution.

Other tools do provide automated test execution (often integrated into the test management tools). The type of automated test execution tool depends to a large extent on the type of SW application under test and the type of test being executed. Three such tools are presented below

Developer-Oriented Tools

Are used in unit, component and module testing to address issues such as memory leaks or other early performance problems.

Are used to test specific pieces of the developer's application code independent of other units of code within the SW application.

May incorporate capture-replay utilities, which run sample tests against working versions of a program, capturing the activity it generates. During the playback phase, developers can see whether or not they are generating the results they expected.

Are particularly useful in testing large applications where different developers are working on different parts of the application.

Test that the code being developed is acting as expected.

Often contain keyword or table driven execution engines, which allow the development and execution of repeatable tests.

Often have scripting capabilities, allowing testers to modify their scripts to test additional items.

Can often be linked back to other packages providing Requirements Capture and Tracking capabilities, thus facilitating comprehensive requirements traceability.

Are particularly useful to test what happens to the code when multiple users access the application simultaneously, or transactions are required to be submitted, or responses are achieved in tight deadlines that manual testing could not reproduce accurately or consistently.

Are particularly useful for critical web based applications, because it is often difficult to predict the volatility of load change. Tools developed for this type of work can often drill down to lower levels of the code to find out where the bottlenecks are, and what is causing any delays.

These tools can also be valuable when testing the application scalability.

Functional Test Tool

Load and Performance Related Tools

Use of Computerized Test Management and Automated Test Tools

As stated above, Computerized Test Management Tools and Automated Test Tools can prove advantageous on large projects or within an organization. However, the use of such tools will require:

Use of testing procedures (SOPs or Work Instructions) for the activities supported by the tools, such as:

Test Case / Test Script authoring

Test Case / Test Script review and approval

Test execution

Test result review and approval

Test defect reporting

Test defect categorization

Test defect disposition

Test defect closure

Other activities, such as test planning and test status reporting, are less critical from a regulatory perspective and will not require formal procedures.

Users of the test tools that are properly trained in the use of the procedures.

Administration of the tool, including:

Configuration of the tool

Creation of test projects

Administration of users

Support and maintenance activities (backup and restore, capacity planning and performance monitoring, etc.)

Smartlogic ,when implementing test tools, initially defines the test processes to be supported by the tool(s). These test processes should comply with the testing good practices defined by GAMP, and the specific test policies and processes of the owning organization, which should define the requirements for using this tool(s).

All users of the test tools should be trained, not only in the use of the tools, but also in good testing practices.

When using test tools, clear user roles and responsibilities should be defined for each test process. Appropriate technical or procedural access restrictions should ensure that only authorized users are allowed to fulfill a defined role, and that the independence of the test process can be verified and enforced.

When used, computerized test management tools and automated test execution tools should provide the same level of test data integrity as an equivalent paper based process.

All these conditions and requirements can be facilitated through the use of features such as:

Role based authorities and user restrictions

Use of workflow to enforce test processes

Use of electronic or digital signatures

Secure, computer generated audit trails

Verification of Computerized Test Management and Automated Test Tools

The need to assure the security, integrity and availability of test data requires the appropriate selection and verification of the automated test tools (refer to GAMP 4, Appendix M4, section 4 for further details). Because such tools usually have only an indirect impact on product quality, they are considered low risk priority and do not require a detailed a lengthy validation process.

The verification of a computerized test management or automated test tools should focus on demonstrating that the tools are fit for purpose as far as critical requirements are concerned. The critical requirements that should form the basis of the tool verification are:

Security of the test data, facilitated by role based authorities and user restrictions, the use of electronic or digital signatures, and the use of secure, computer generated audit trails. Where the tools do not provide these features, the verification of the tool(s) should focus on establishing such security using logical or physical security controls, procedural controls and paper based audit trails.

Ability of the test tool to enforce defined test processes using workflow or equivalent controls. Where this is not possible, the verification of the tool(s) should focus on the ability of the tool to provide audit trail evidence of the test processes.

The scope of the verification activities may be scaled based upon risk, and should take into account the track record of the supplier and tools in the life sciences.

Test data managed within test tools would not usually be determined to be an electronic record within the scope of predicate rules, and the use of electronic or digital signatures would not usually be determined within the scope of predicate rules, so that the requirements of regulations such as 21 CFR Part 11 would not usually apply.

However, there may be cases where the SW or application tested using such tools is of greater regulatory significance, for example, where the SW or application is defined as a medical device or part of it. In these cases, a more formal validation of the test may be required:

The test record may also be determined to be an electronic record (i.e., an acceptance record under 21 CFR Part 820 Subpart H – Acceptance Activities).

Any associated signatures may be determined to be electronic or digital signatures (i.e., under 21 CFR Part 820.80).

The test management tools may need to comply with the requirements of 21 CFR Part 11, and other regulator guidance in electronic records and digital signatures