ABSTRACT - System testing involves testing for non-functional as well as functional requirements, non-functional requirements being the ones relating to attributes like performance, usability, availability, etc. Though functionality testing is well planned in most projects, the same can not be said about non-functional testing. The test team needs to explicitly focus on non-functional testing during each phase of the testing lifecycle.
This paper discusses as to how focus can be brought on the non-functional attributes during various phases of software test life cycle namely: test planning, test design, test execution and test reporting.
摘要：
系统测试不仅包含了对功能性需求的测试，还包含了对非功能性需求的测试，而非功能性需求会涉及到一些诸如性能、可用性、可行性等属性。尽管在大多数项目中，能够对功能性测试进行良好规划，但是对非功能性测试却很难这么做。因此，测试团队需要在测试生命周期的每一个阶段都非常重视非功能性测试。
这篇论文研讨了在测试生命周期的几个不同阶段——测试计划、测试设计、测试执行、测试报告——对非功能性测试需要重点关心什么。
KEY WORDS - Testing, test planning, requirements, performance, usability.
关键词： 测试、测试计划、需求、性能、可用性
1. Introduction
1. 简介
Product quality means more than meeting the functional requirements of the product as stated by the customer. It is beneficial to the end user, the customer organization and the development organization to identify and improve the non functional requirements of the product like performance, usability, reusability, availability etc. The product design and implementation should ensure that the product meets the specified non-functional requirements. It is aspects like poor performance, usability etc which cause system re-design after delivery. In MIEL, as part of the product quality improvement initiative, projects are getting to focus explicitly on non-functional attributes during the requirements phase. These non-functional requirements which are explicitly captured in the requirements book, need to be verified/validated as well.
Some non-functional requirements like evolvability can be verified through analysis methods like SAAM[2], while some others like usability and performance can be validated by testing. In this paper we are exploring the role of system testing in validating the non-functional requirements. We outline activities & strategies relating to non-functional attributes that can be done in various phases of software test life cycle: test planning, test design, test execution etc. Performance and usability testing in particular are covered in greater detail.
In section 2, we discuss the software test lifecycle, and in section 3 the focus is on non-functional testing related activities on the different phases of the software test lifecycle. In section 4 we touch upon few key issues relating to non-functional testing. Sections 5 and 6 discuss testing for performance and usability respectively.
产品质量意味着不止满足客户提出的功能性需求。这才有利于最终用户、客户组织以及开发组织确定并改进诸如性能、可用性、重用性、可行性等非功能性需求。产品的设计和实现应该务必使之能满足特定的非功能性需求。很明显，性能低、可用性低等情况往往会导致系统在提交之后需要重新设计。在MIEL中，作为产品质量实现的初始阶段的一部分，项目在需求阶段变的十分重视非功能性属性。这些写入需求说明书的非功能性需求同样也需要被验证和确定。
一些像可演化性这样的非功能性需求能够通过分析模型来验证，例如SAAM。而另一些像可用性以及性能这样的需求则能够用测试来确定。在这篇论文中，我们将探究系统测试在确定非功能性需求的过程中所扮演的角色。我们为非功能性属性勾画一系列的活动和策略，这些活动和策略在不同的软件测试生命周期（测试计划、测试设计、测试执行、测试报告）中可以被实施。其中，对性能和可用性测试会讨论的更加详细。
在第二部分，我们将讨论软件测试生命周期，而在第三部分，重点是在软件测试生命周期的不同阶段与非功能性测试相关的活动。在第四部分，我们会接触到一些非功能性测试的关键点，而在第五和第六部分，将分别讨论对性能和可用性的测试。
2. Software Test life Cycle
2. 软件测试生命周期
Figure – 1 depicts the generic software development / maintenance life cycle followed in MIEL. Non-functional requirements need to be identified during the requirements phase[4]. The captured non-functional requirements are analyzed and some of them which are testable and are very crucial from customer perspective become part of the requirements book and others are input to the design as design goals.
Figure-1 also identifies the different phases of software test life cycle. Please see table (1) for the different phases and their inputs, outputs and activities. Test case development and test software development can be parallel activities.
Non-functional requirements are fundamentally hard to verify and. Some requirements like usability and maintainability tend to be very subjective and therefore are not readily testable. In such cases it helps to identify objective attributes which quantify these aspects of the system. In other cases like stress and performance, simulation in the test lab is the challenge. This requires that extreme care should be taken in every phase of STLC to ensure that provisions are made to test for the identified non-functional attributes.
|
PHASE |
INPUTS |
PROCESS |
OUTPUTS |
| Test Planning | ReqB, SPMP, SQAP, SCMP | Identify scope, strategy, effort, resources etc.. | SSTP |
| Test RequirementSpecification | ReqB, SSTP, Operation profile | Identify test case, test software and setup requirements.Define categories of test to be performed – performance, conformance, stress etc.. | TRS |
| Test Case Development | TRS, ReqB/ PSRS | Identify test cases and sequencing. |
TCD, TSC |
| Test Software Development | TRS, ICD, TCD | Design, code and test the test software. Prepare user manuals | Test software code and documentation |
| Test Execution | TCD, TSC | Execute the test cases, document the results, defects and observations | Test Log Reports |
| Test Reporting | Test Logs | Analyze the test logs. | TPR, STSR |
|
Table 1: Phases of software test life cycle | |||
|
Category |
Examples |
Test Requirements |
Test Procedure |
| Performance | · Traffic capacity · Transaction / data throughput · UI response time · Call setup time · Memory Utilization |
· Instrumentation of the code to help in measurements (time) · Special tools to load the SUT and measure time, throughput. |
· Identify the dependencies – configuration, external and system state · Identify the values the dependencies can take and decide on the different combinations on which testing will be done. · Execute tests and record measurements · Co-relate parameters and dependencies by plotting charts |
| Usability | · Installation · Routine Procedures · Documentation |
· UI Subjects matching user profiles | · Identify critical procedures, sections of UI etc.. for which usability is of concern · Identify the subjects · Test with subjects – observe/videotape · Analyze |
| Reliability | Multiple rounds of system testing | · Collect the failure and execution time data from multiple rounds of testing. · Use a suitable software reliability model (e.g.: Goel and Okumoto,1979) to predict reliability | |
| Security | Firewalls, Networks | Can not prove system is secure. Establish to sufficient degree of confidence that it is secure enough. | Use of tiger teams (professional hackers!!!) |
| Configuration | Hardware and software configurations Redundancy means more configurations | · Identify the combinations · Select important (more used or more susceptible) ones · Do a selected subset of test cases · Degradation and restoration testing (transition between configurations) | |
| Volume | · No. of subscribers · Predefined load profile over time |
Automation required | |
| Stress | BHCA | Tool to generate the stress condition | Real loads can be used for Volume and stress testing – they are the best and worst kind of load to use for testing. |
| Recovery | Automatic restart Switchover | Simulate the condition to initiate recovery.Confirm resumption of processing without transaction compromise. | |
|
Table 2: Categories of non-functional testing | |||
延伸阅读
文章来源于领测软件测试网 https://www.ltesting.net/
