Building the Test Environment

1
Building the TestEnvironment
Skill Category 2
2

• Skill Category 2
• Building the Test Environment ....................
  .......................................... 133
• Management Support ...............................
  ..............................................
  133
• Management Tone ..................................
  ................................... 134
• Integrity and Ethical Values .....................
  ................................................
  135
• Commitment to Competence .........................
  ............................................ 137
• Managements Philosophy and Operating Style
  ..................................................
  ........... 137
• Organizational Structure .........................
  ............................................. 138
• Test Work Processes ..............................
  ................................................
  140
• The Importance of Work Processes
  ..................................................
  .................... 140
• Responsibility for Building Work Processes
  ..................................................
  .................. 142
• Developing Work Processes ........................
  ..............................................
  149
• Testers Workbench ...............................
  ....................................... 150
• Analysis and Improvement of the Test Process
  ..................................................
  ............ 153
• Test Tools .......................................
  ....................................... 167
• Tool Development and Acquisition
  ..................................................
  .................... 167
• Tool Usage .......................................
  ............................... 179
• Testers Competency ...............................
  ...............................................
  180

3
The test Environment is comprised of the
following conditions within the organization

• policies
• procedures
• culture
• attitudes
• rewards
• test processes
• test tools
• methods for developing and improving test
  processes
• managements support of software testing
• test labs developed for the purpose of testing
  software
• multiple operating environments
• assuring the test environment fairly represents
  the production environment to enable realistic
  testing to occur

4
Management Support Good for testers.

• Following will be discussed
• Management tone Management sets the tone by
  providing testers the resources and management
  time needed to do their job effectively.
• Test process alignment Management creates a test
  function that is aligned with the business needs
  of the organization.

5
Testers need to know 3 things about the control
environment

• The control environment is established by the
  highest levels of management and works downward
  through the organization.
• The test function cannot create the
  organizations control environment, but can
  influence how that environment is implemented
  within the test function.
• The control environment will influence the way in
  which testers perform the work which may be
  ethical or unethical.

6
Integrity and Ethical Values

• Excerpt
• Integrity is a prerequisite for ethical
  behavior in all aspects of an enterprises
  activities. A strong corporate ethical climate at
  all levels is vital to the well-being of the
  corporation, all of its constituencies, and the
  public at large.

7
Incentives and Temptations

• Individuals may engage in dishonest, illegal
  or unethical acts simply because their
  organizations give them strong incentives or
  temptations to do so.

8

• Incentives cited for engaging in fraudulent
  unethical behavior
• Pressure to meet unrealistic performance targets,
  particularly for short-term results
• High performance-dependent rewards.
• Temptations for employees to engage in
  improper acts
• Nonexistent or ineffective controls, such as poor
  segregation of duties in sensitive areas that
  offer temptations to steal or to conceal poor
  performance.
• High decentralization that leaves top management
  unaware of actions taken at lower organizational
  levels and thereby reduces the chances of getting
  caught.
• A weak internal audit function that does not have
  the ability to detect and report improper
  behavior.
• Penalties for improper behavior that are
  insignificant or unpublicized and thus lose their
  value as deterrents.

9
Providing and Communicating Moral Guidance

• The most effective way of transmitting a message
  of ethical behavior throughout the organization
  is by example.

10
Commitment to Competence

• Competence should reflect the knowledge and
  skills needed to accomplish tasks that define the
  individuals job.
• Management needs to specify the competence levels
  for particular jobs and to translate those levels
  into requisite knowledge and skills.

11
Managements Philosophy and Operating Style

• Managements philosophy and operating style
  affect the way testing is managed, including the
  kinds of business risks accepted.
• Informal An informally managed company may
  control operations largely by face-to-face
  contact with key managers.
• Formal A more formally managed exception
  reports.
• Other elements of managements philosophy and
  operating style include
• attitudes toward financial reporting
• conservative or aggressive selection from
  available alternative accounting principles
• conscientiousness and conservatism with which
  accounting estimates are developed
• attitudes toward data processing and accounting
  functions and personnel.

12
Organizational Structure

• An entitys organizational structure provides the
  framework within which its activities for
  achieving entity-wide objectives are planned,
  executed, controlled and monitored.
• Activities that relate to the value chain
• inbound (requirements) activities
• operations or production
• outbound (software)
• deployment and maintenance
• Support Functions
• administration,
• human resources
• Technology development

13
Assignment of Authority and Responsibility

• Management has an important function with the
  assignment of authority and responsibility for
  operating activities, and establishment of
  reporting relationships and authorization
  protocols.
• Management function deals with
• policies describing appropriate business
  practices
• knowledge and experience of key personnel
• resources provided for carrying out duties

14
Human Resource Policies and Practices

• Messages to employees
• Standards for hiring the most qualified
  individuals, with emphasis on educational
  background, prior work experience, past
  accomplishments and evidence of integrity and
  ethical behavior, demonstrate an organizations
  commitment to competent and trustworthy people.
• Recruiting practices that include formal,
  in-depth employment interviews and informative
  and insightful presentations on the
  organizations history, culture and operating
  style send a message that the organization is
  committed to its people.
• Training policies that communicate prospective
  roles and responsibilities and include practices
  such as training schools and seminars, simulated
  case studies and role-play exercises, illustrate
  expected levels of performance and behavior.
• Rotation of personnel and promotions driven by
  periodic performance appraisals demonstrate the
  entitys commitment to the advancement of
  qualified personnel to higher levels of
  responsibility.
• Competitive compensation programs that include
  bonus incentives serve to motivate and reinforce
  outstanding performance. Disciplinary actions
  send a message that violations of expected
  behavior will not be tolerated.

15
Test Work Processes

• Work processes are the policies, standards and
  procedures in a quality IT environment. Test work
  processes are those work processes specific to
  the testing activity.
• Policy Managerial desires and intents concerning
  either process (intended objectives) or products
  (desired attributes).
• Standards The measure used to evaluate products
  and identify nonconformance. The basis upon which
  adherence to policies is measured.
• Procedure The step-by-step method that ensures
  that standards are met.

16
The Importance of Work Processes

• It is important to a quality IT environment to
  establish, adhere to, and maintain work processes
  in the testing activity. It is also critical that
  the work processes represent sound policies,
  standards and procedures.

17
Major purposes and advantages to having work
processes

• Improves communication
• Enables knowledge transfer
• Improves productivity
• Assists with mastering new technology
• Reduces defects and cost

18
Responsibility for Building Work Processes

• IT management is responsible for issuing IT
  policy. Policies define direction and by
  definition are general rules or principles. It is
  the standards, which will add the specificity
  needed for implementation of policies.
• Policies are needed in
• Building Systems
• Testing Systems
• Maintaining Systems
• Operating Systems
• Quality of Systems
• Security of Systems
• Allocation of Resources
• Planning for Systems
• Training Personnel

19
Responsibility for Standards and Procedures

• The workers who use the procedures and are
  required to comply with the standards should be
  responsible for the development of those
  standards and procedures. Management sets the
  direction and the workers define that direction.
  This division permits each to do what they are
• best qualified to do.

20
The key concepts of a process engineering program
are

• Management provides an organizational structure
  for the workers to develop their own standards
  and procedures.
• The program is driven by management policies.
• Absolute compliance to standards and procedures
  is required.
• A mechanism is provided for the continual
  maintenance of standards and procedures to make
  them more effective.

21
Test Process Selection

• Selecting, developing, and acquiring work
  processes is an overall IT organization
• responsibility. Software testers need to both
• understand how the activity operates AND
  participate when test processes, and related
  processes, are selected and put into practice.

22
Test Process Selection (contd)

• IT groups should develop a plan for
  implementing and operating a process engineering
• program. This would require a policy for
  standards, and a charter or job description for
  the
• function. These need to be customized for
  each organization. The specific components that
• need to be addressed include
• Building a Process Engineering Organization
• Developing a Standard and Procedure for Standards
• Planning for Standards
• Writing, Storing, and Retrieving Standards and
  Procedures
• Enforcing Standards

23
Building a Process Engineering Organization

• The structure that is put in place to develop
  and update policies, standards, and procedures
  must involve both staff and management. The
  process engineering program should be directed by
  management, but implemented by staff.

24
Building a Process Engineering Organization
(contd)

• Some guidelines on establishing an
  organizational structure for process engineering
  are
• Establish a Process Engineering Committee
  compromised of the most senior IT managers
  possible.
• Represent all IT organizational areas on the
  Process Engineering Committee.
• Appoint an individual as the Process Engineering
  Manager. (Note that this can be a part-time
  assignment.)
• Appoint Ad Hoc Committees (i.e., special task
  forces) to develop individual standards and
  procedures.
• Let the Standards Ad Hoc Committees develop the
  technical standard.

25
Recommended Standards Organizational Structure

• The recommended organizational structure is
  comprised of the following three components
• Process Engineering Manager
• A full-time or part-time individual responsible
  for managing the standards program.
• Process Engineering Committee
• A policy-setting board, which runs the standards
  program.
• Ad Hoc Committee
• Small groups which develop single standards at a
  time.

26
Organizational chart for process engineering
27
Role of Process Engineering Manager

• Promote the concept of process engineering.
• Be the driving force behind processes.
• Administer the standards program defined by the
  Process Engineering Committee.
• Be a resource to the Process Engineering
  Committee and Ad Hoc Committees.
• Ensure involved parties are adequately trained.

28
Role of the Process Engineering Committee

• Accept topics for processes.
• Set priority for implementation of processes.
• Obtain the resources necessary to develop the
  process.
• Approve or reject developed processes.

29
Role of the Ad Hoc Committee

• Gain representatives from all involved areas.
• Ensure that the committee has between three and
  eight members in size.
• Create the standard and procedure.
• Coordinate reviews of the standard with involved
  parties.
• Periodically review and update the standards and
  procedures previously developed by
• the Ad Hoc Committee.

30
Selecting Process Engineering Committee Members

• The makeup of the Process Engineering
  Committee is important. Some guidelines in
  selecting
• members are
• Select the highest-level manager who will accept
  the position.
• Assign individuals who are supportive and
  enthusiastic over standards.
• Make long-term assignments to the Process
  Engineering Committee.
• Select individuals who are respected by their
  peers and subordinates.
• Ensure the appropriate areas of interest are
  involved.

31
Selecting Process Engineering Committee Members
(contd)

• Some guidelines to ensure the appropriate
  areas of interest are included in developing
• standards are
• Is every organizational function within IT
  involved?
• Are the activities that interface to the IT
  function involved for example, key users and
  customers?
• Are activities having a vested interest involved,
  such as auditors?
• Are all the IT businesses represented such as
  the project leaders, systems programmers, data
  library, security, help desk, and so forth?

32
Developing Work Processes

• Prior to creating any processes, the Process
  Engineering Committee must develop a standard and
  procedure for developing standards and
  procedures. The standards and procedures
  developed by the standards program are products.
  Those products must be standardized just as
• the other products produced in an IT function.

33
Defining the Attributes of a Standard for a
Standard

• The standard for a standard and a standard for
  a procedure, define the format, style, and
• attributes of a document called a standard and
  a document called a procedure. This standard
• for a standard and standard for a procedure
  become the prototype that will be used by the Ad
• Hoc Committees in developing standards and
  accompanying procedure(s).
• This standard should define the following
• A testing policy
• A test workbench to support the policy (i.e.,
  standards and processes)

34
Establishing a Testing Policy

• A testing policy is managements objective for
  testing. It is the objective to be accomplished.
• Good testing does not just happen, it must be
  planned, and a testing policy should be the
  cornerstone of that plan.

Sample Testing Policy
35
Testers Workbench

• The testers workbench shows that input products
  drive the workbench, which uses procedures and
  standards to produce output products.

36
Professional Test Standards

• Test professionals should be familiar with the
  standards published by organizations
• such as the International Standards Organization
  (ISO), U.S. Department of Commerce
• (National Institute of Standards and Technology),
  and IEEE. In addition, testers should know
• when these standards apply, and how to obtain the
  latest versions.

37
Professional Test Standards (contd)

• IEEE publishes a body of standards that address
  software engineering and maintenance. A portion
  of these standards addresses software testing and
  test reporting. Listed below are the current
  standards that apply to the test process.
• 829-1998 IEEE Standard for Software Test
  Documentation
• 830-1998 IEEE Recommended Practice for Software
  Requirements
• Specifications
• 1008-1987 (R1993) IEEE Standard for Software Unit
  Testing (ANSI)
• 1012-1998 IEEE Standard for Software Verification
  and Validation
• 1012a-1998 IEEE Standard for Software
  Verification and Validation Supplement to
  1012-1998 Content Map to IEEE 12207.1
• 1028-1997 IEEE Standard for Software Reviews
• Other professional organizations have similar
  standards. The CSTE candidate should have an
  overall familiarity with the purpose and contents
  of these standards.

38
Analysis and Improvement of the Test Process

• Test Process Analysis Test quality control is
  performed during the execution of the process.
  Analysis is performed after the test process is
  completed.
• Effectiveness and efficiency of test processes
• The test objectives are applicable, reasonable,
  adequate, feasible, and affordable
• Testers do not have the needed competencies to
  meet the test objectives
• The test program meets the test objectives
• The correct test program is being applied to the
  project
• The test methodology is used correctly
• The task work products are adequate to meet the
  test objectives
• Analysis of the results of testing to determine
  the adequacy of testing
• Adequate, not excessive, testing is performed

39
Continuous Improvement

• A testing process assessment is one excellent way
  to determine the status of your current test
  process.
• Process assessment is applicable in the following
  circumstances
• Understanding the state of processes for
  improvement.
• Determining the suitability of processes for a
  particular requirement or class of requirements.
• Determining the suitability of another
  organizations processes for a particular
  contract or class of contracts.
• The framework for process assessment
• Encourages self-assessment.
• Takes into account the context in which the
  assessed process operates.
• Produces a set of process ratings (a process
  profile) rather than a pass or fail result.
• Addresses the adequacy of the management of the
  assessed processes through generic practices.
• Is appropriate across all application categories
  and sizes of organization.

40
Test Process Improvement Model

• A model for test process improvement has these
  eight steps
• Examine the Organizations Needs and Business
  Goals
• Conduct Assessment
• Initiate Process Improvement
• Analyze Assessment Output and Derive Action Plan
• Implement Improvements
• Confirm Improvements
• Sustain Improvement Gains
• Monitor Performance

41
Test Process Alignment

• In establishing the test environment management
  must assure that the mission/goals of the test
  function are aligned to the mission/goals of the
  organization.

42
Adapting the Test Process to Different Software
Development Methodologies

• Understanding the software development process
  used in his/her organization is important in
  conducting software testing for the following
  three reasons
• Understanding the developmental timetable and
  deliverables.
• Integrating software
• Understanding how software is developed

43
Six major categories of developmental
methodologies

• Waterfall
• D-shaped
• Prototype
• Spiral
• RAD
• Incremental

44
Testers testing software developed by a specific
software development methodology need to

• Understand the methodology.
• Understand the deliverables produced when using
  that methodology which is needed for test
  purposes.
• Identify compatible and incompatible test
  activities associated with the developmental
  methodology.
• Customize the software test methodology to
  effectively test the software based on the
  specific developmental methodology used to build
  the software.

45
Test Tools

• It is difficult to perform testing economically
  without the aid of automated tools.
• The most important aspect of software testing
  tools is the process used to acquire those tools.
  Most of this component of the environment will
  focus on acquiring or developing testing tools.

46
Tool Development and Acquisition

• Difficulties surrounding the introduction of
  tools can arise in three areas
• Organizational obstacles
• Problems arising from the tools
• Obstacles in the computer environment
• The activities associated with the introduction
  of tools should include these activities
• Identifying the goals to be met by the tool (or
  by the technique supported by the tool), and
  assigning responsibility for the activities
  required to meet these goals.
• Approving a detailed tool acquisition plan that
  defines the resource requirements for procurement
  and in-house activities.
• Approving the procurement of tools and training,
  if this is not explicit in the approval of the
  acquisition plan.
• Determining, after some period of tool use,
  whether the goals have been met.

47
Tool Development and Acquisition (contd)

• Test management is assigned responsibility for
• Identifying tool objectives.
• Approving the acquisition plan (it may also
  require approval by funding management).
• Defining selection criteria.
• Making the final selection of the tool or the
  source.
• The Test Manager is responsible for the following
  activities. Note that a selection group for the
  more important test tools may assist the Tool
  Manager.
• Identifying candidate tools.
• Applying the selection criteria (in informal
  procurement) or preparing the RFP (in formal
  procurement).
• Preparing a ranked list of tools or sources.
• Conducting any detailed evaluations or conference
  room pilots.

48
Tool Development and Acquisition (contd)

• This distribution of responsibilities reduces the
  chances of selecting a tool that
• Does not meet the recognized needs of the
  organization
• Is difficult to use
• Requires excessive computer resources
• Lacks adequate documentation

49
Event Sequencing

• The following event sequence addresses only the
  introduction of existing tools.
• The recommended event sequence allows for two
  procurement methods for bids
• 1. Informal procurement (by purchase order)
• 2. Formal procurement (by a request)

50
Event Sequence Steps

• Goals
• Tool Objectives
• Acquisition Activities for Informal Procurement
• Acquisition Plan
• Selection Criteria
• Identify Candidate Tools
• User Review of Candidates
• Score Candidates
• Select Tool
• Acquisition Activities for Formal Procurement
• Acquisition Plan
• Technical Requirements Document
• User Review of Requirements
• RFP Generation
• Solicitation of Proposals
• B6 Technical Evaluation should be Consistent
• Source Selection
• Procure Tool
• Evaluation Plan

51
Tool Usage

• The most commonly used tools can be grouped
  into these eight areas
• Automated Regression Testing Tools
• Defect Management Tools
• Performance/Load Testing Tools
• Manual Tools
• Traceability Tools
• Code Coverage
• Test Case Management Tools
• Common tools that are applicable to testing

52
Tool Usage (contd)

• Most testing organizations agree that if the
  following three guidelines are adhered to tool
  usage will be more effective and efficient.
• Guideline 1 Testers should not be permitted to
  use tools for which they have not received formal
  training.
• Guideline 2 The use of test tools should be
  incorporated into test processes so that the use
  of tools is mandatory, not optional.
• Guideline 3 Testers should have access to an
  individual in their organization, or the
  organization that developed the tool, to answer
  questions or provide guidance on using the tool.

53
Testers Competency

• Test competency is a direct responsibility of the
  individual and the organization that employs that
  individual.
• Software testing organizations should develop a
  roadmap for testers to pursue to advance their
  competencies. The roadmap will have these two
  paths
• Skill Sets
• Performance Skills

54
This figure is typical of how a software testing
organization may measure an individual
testerscompetency.