The fact that every load test is different doesn't mean that we cannot be systematic in our approach. The checklist below, that we use as we prepare our plan for a Load Test Project, gives a succinct overview of how we would expect a Load Test to progress, from Project Initiation to Closedown.
PROJECT INITIATION
Internal Setup
Setup project folders
Setup project media
Setup project log
Store copy of proposal in project area
Store copy of check list in project area
Customer Logistics
Completion date for benchmark agreed
Customer contact for information agreed
Benchmark type agreed
Components to stress agreed
Server
Client
Network
Scope for tuning agreed
Server hardware
Client hardware
Network
Server OS
Client OS
DBMS
Customer attendance at benchmark test/s identified
Benchmark disclaimers and constraints identified and agreed
Project deliverables agreed (report and possibly presentation)
Confirm contractual obligations
Place copy in project file
Configuration Supplier Logistics
Supplier manager contact identified and agreed
Benchmark site identified
Site access times and contact identified
Equipment availability times documented and agreed (which days, what times)
Supplier technical resource (who and availability) documented and agreed
Hardware setup and tuning requirements communicated and agreed
Agree availability of hardware and software documentation
Operating system setup and tuning requirements communicated and agreed
DBMS setup and tuning requirements communicated and agreed
Application Supplier Logistics
Supplier manager contact identified and agreed
Supplier technical resource (who and availability) documented and agreed
Procedure for application errors documented and agreed
Agree availability of application documentation
DBMS setup and tuning requirements communicated and agreed
Date and time of application installation and setup documented and agreed
Ensure number of users for software licenses communicated and agreed
The Preliminary Risk Assessment
Clarity of Objectives
Technological
PATH Port needed
Product development needed
Limited availability of machine environment
Weaknesses in the target operating environments
Human Resource
New people
External people
Availability
Parallel projects
Low quality bodies
Financial
Fixed price
Low margin
Customer's financial standing
Slow payment
Legal
Penalty clauses
Loose contracts
3rd Party Exposure
Impossibility of passing on external delay costs
Dependence on delivery dates
Weaknesses in 3rd party organisation staffing, etc.
Client
Service provider
Hardware supplier
et al.
Long Term Guarantees (Need suitable disclaimers)
Log all relevant information in project log
SETUP WITH CUSTOMER
Workload Setup (one per system and scenario)
Scenario identified, documented and agreed
Time period for benchmark (duration, ie. peak hour) documented and agreed
Transactions to test documented and agreed
Numbers of each transaction to test documented and agreed
Number of logged on users documented and agreed
Understanding of transactions attained
Constraints on data to enter into transactions documented and agreed
Log relevant information in project log
Data Setup
Entity sizes documented and agreed
Data shape documented and agreed
Log relevant information in project log *
Risk Re-Assessment
Has anything changed? If so, revisit
Are we tracking the plan?
SETUP WITH SUPPLIERS (one per supplier)
Configuration Supplier Logistics
Benchmark hardware and software configuration documented and agreed
Server
Network
Client
Diagnostic software available identified (with example formats if required)
Ensure number of users for software licenses communicated and agreed
Log relevant information in project log
Application Supplier Logistics
Benchmark application configuration documented and agreed
Application tuning requirements communicated and agreed
Diagnostic software available identified (with example formats if required)
Risk Re-Assessment
Has anything changed? If so, revisit
Are we tracking the plan?
INTERNAL SETUP
Project tasks identified and documented
Task resources identified and documented
Task timings (duration, when done) identified and documented
Tasks achievable at office identified and documented
Task schedule produced and sent to customer and suppliers
Benchmark information reviewed, documented and sent to customer and suppliers
Relevant previous benchmark documentation, inputs, intermediate outputs and media reviewed and copied if necessary
Log relevant information in project log
Risk Re-Assessment
Has anything changed? If so, revisit
Are we tracking the plan?
BENCHMARK PREPARATION
Configuration Tests
Server
Ensure all agreed hardware components are installed
Ensure OS and file system installed correctly (agreed versions, settings and tuning)
Ensure product documentation available
Ensure backup and restore hardware and media are available
Ensure diagnostic software installed and available
Ensure license limits correct
Ensure necessary users are set up
Document actual server configuration
Log relevant information in project log
Client
Ensure agreed number of clients available
Ensure all agreed hardware components are installed
Ensure product documentation available
Ensure OS and network software installed correctly (agreed versions, settings and tuning)
Ensure diagnostic software installed and available
Document actual client configuration
Log relevant information in project log
Network
Ensure all agreed hardware components are installed
Ensure software installed correctly (agreed versions, settings and tuning)
Ensure product documentation available
Ensure relevant network addresses are recorded on client and server
Ensure diagnostic software installed and available
Ensure access possible from client to server
Document actual network configuration
Log relevant information in project log
Database
Ensure agreed components are installed
Ensure product documentation available
Ensure installed correctly (agreed versions, settings and tuning)
Ensure files are striped or positioned correctly
Ensure diagnostic software installed and available
Ensure license limits correct
Ensure necessary database users are set up
Document actual database configuration
Ensure network access set up is correct
Ensure that database can be put in sql trace mode successfully
Log relevant information in project log
Application Tests
Ensure agreed components are installed
Ensure installed correctly (agreed versions, settings and tuning)
Execute each required transaction to ensure working
Document actual application configuration
Log relevant information in project log
PATH Tests (If not using E2 Systems hardware)
Ensure PATH software runs on provided hardware and OS
Establish the appropriate awk (awk, nawk, mawk or gawk)
Set up a PATH profile
Create simple PATH script with event
Check the script can be rerun; check the recorded timing
Determine if PATH needs to be run by root or administrator
Switch on process accounting
Compare output of acctcomm or lastcomm (Ultrix) with hisps; recompile hisps if necessary
Check availability and CPU costs of:
sar (and sadc)
iostat
vmstat
netstat
sysmon/perfmon
tcpdump (or other packet filters, eg. snoop, windump)
Wireshark
Process Explorer (and Sysinternals Suite)
Debuggers
System Monitor
Identify application target terminal type if applicable
Create PATH script using application
Adjust terminfo definition so that it is compatible if applicable
Check Service Level parameters for fdreport
Check on autopop version if data generation required
Document PATH configuration
Backup PATH environment to magnetic or optical media
Log relevant information in project log
For Client-Server or 3 Tier Emulation
Examine the client/server protocol
Identify encryption issues
If we have an applicable PATH driver, compile it up
If we are going to use the application's API, and we have a PATH driver
Compile the PATH drivers
If SQL Client required
Use tabdiff backup and restore options for initial sanity check
Use badsort and schema_cmp for more ambitious tests
Otherwise, Reverse engineer the protocol, and enhance PATH software accordingly. HIGH RISK.
Pre-Test Tuning (if allowed)
Review all components (server, network, client, DBMS, application) and component documentation to identify significant tuning opportunities
Document tuning opportunities
Agree new tuning with customer and suppliers
Apply new tuning to system
Adjust documentation for new tuning settings
Log relevant information in project log
Risk Re-Assessment
Has anything changed? If so, revisit
Are we tracking the plan?
BENCHMARK BUILD
Database Build
Verify database space is sufficient
Construct database load scripts
Verify scripts against available database size and shape documentation
Run database build scripts
Backup database and store in project file
Backup database build scripts and store in project file
Construct database size and shape verification reports
Run database size and shape verification reports
Review size and shape reports to verify correct against requirements
Backup size and shape programs and store in project file
Log relevant information in project log
Script Build
Server/RTE (Driver Machine(s))
Construct benchmark scripts
Combine into user threads
Test run threads
Document script and thread contents and numbers
Verify thread contents and numbers against workload requirements
Backup scripts and store in project file
Log relevant information in project log
Client
Construct GUI Test scripts for transactions if applicable
Combine into client threads
Adjust data in threads
Test run threads
Document script and thread contents and numbers
Verify thread contents and numbers against workload requirements
Backup scripts and store in project file
Log relevant information in project log
Risk Re-Assessment
Has anything changed? If so, revisit
Are we tracking the plan?
BENCHMARK ORIENTATION RUNS
Orientation Tests
Setup OS diagnostic tools
Setup DBMS diagnostics
Setup network diagnostics
Setup client diagnostics
Run server benchmark threads
Run client benchmark threads
Run individual server benchmark scripts if relevant (interactive tuning identification)
Run individual client benchmark scripts if relevant (interactive tuning identification)
Document diagnostic information (when run, characteristic, etc.)
Backup diagnostic information and store in project file
Log relevant information in project log
Orientation Analysis and Tuning (if allowed)
Run PATH information extract programs against diagnostic information
Review diagnostic information and PATH extract information
Identify and document server CPU problems and possible tuning solutions
Identify and document server memory problems and possible solutions
Identify and document server disk sub-system problems and possible tuning solutions ..
Identify and document server OS problems and possible tuning solutions
Identify and document network component problems and possible tuning solutions
Identify and document database setup problems and possible tuning solutions
Identify and document database index problems and possible tuning solutions
Identify and document application problems and possible tuning solutions
Identify and document application data navigation problems and possible solutions
Identify and document client CPU problems and possible tuning solutions
Identify and document client memory problems and possible tuning solutions
Identify and document client OS problems and possible tuning solutions
Group tuning solutions into definite and possible lists
Discuss, agree and document tuning solutions to apply to benchmark
Apply agreed tuning solutions to the benchmark configuration
Update documentation with new applied tuning solutions
Log relevant information in project log
Risk Re-Assessment
Has anything changed? If so, revisit
Are we tracking the plan?
BENCHMARK EXECUTION
Formal Tests
Setup OS diagnostic tools
Setup DBMS diagnostics
Setup network diagnostics
Setup client diagnostics
Run server benchmark threads
Run client benchmark threads
Document diagnostic information (when run, characteristics, etc.)
Backup diagnostic information and store in project file
Log relevant information in project log
Initial Analysis
Run PATH information extract programs against diagnostic information
Review diagnostic information and PATH extract information
Assess server CPU utilisation
Assess server memory utilisation
Assess server disk IO sub-system utilisation
Assess network utilisation
Assess client CPU utilisation
Assess client memory utilisation
Assess DBMS efficiency
Assess Application Server efficiency
Assess Web Server efficiency
Identify whether workload throughput achieved
Identify any rogue transactions / programs - assess reasons for performance
