|
PATH is a proven set of tools to aid the automation of multi-user testing and the
building of test environments. The tools address
- Script capture
- Script engineering
- Database builds and expansions
- Multi-Test synchronisation over WANs
- SQL investigation
- Result Analysis
..... in addition to the generic script drivers which drive the widest range of protocols.
The Basic Idea.
An automated test tool has to reproduce the traffic between the user and the system under test, in a manner that is application-specific and timely. It must record meaningful response times, and readily detect errors. Scripts executed for each virtual test user must vary the data submitted in order to exercise
any back end database in a realistic way.
In our early days we developed and sold a commercial software product for automated testing.
Our design objectives were to minimise the resources required by our user emulator to execute a test, and to avoid the need for a practitioner to have advanced programming skills.
Our software met its performance objectives, and achieved modest commercial success. We sold the Product to three customers, including the U.K. Ministry of Defence.
However, within 18 months our Product was shelf-ware everywhere we had sold it.
As indeed, we suspect, are our competitors' products!
However, we believe that our subsequent success as Consultants using our own software validates our minimalist approach.
- We consider test scripts to be Document Templates rather than Programs. In our model, Test Generation is more akin to Mail-merge than anything else.
- Our drivers run on multiple platforms thus sychronised tests distributed over a heterogeneous network with heterogeneous hardware are readily accommodated.
- Our minimised driver resource usage means that at least for initial testing
the drivers can be accommodated on the System Under Test (SUT) without invalidating results.
In keeping with our minimalist philosophy, our preferred approach is to create our script templates from network traffic (no matter what protocol) as it leaves the client PC, and to drive the scripts with a driver that effectively injects and processes network traffic.
This approach assures us that we are capturing ALL the effects that the PC is having on its environment, and if it is practicable it allows us to drive thousands of concurrent simulated users with an obsolete laptop.
Over the years we have reverse engineered dozens of protocols this way. Apart from the trivial case of HTTP, protocols that are still commercially significant include Java RMI, ORACLE Forms Services, and the Documentum and Objective Enterprise Document Management Systems.
Second choice is use of a low-level client API. Most of our "Thick Client" tests follow this approach; the API might be ORACLE OCI, Sybase CT-Lib, or something Microsoft-proprietary. It may mean that the number of simulated clients that can be driven from a given piece of hardware is reduced by a factor of 10.
Last choice is driving using mouseclicks and key-strokes. Software that runs on PC's often assumes that all the PC resources are at its disposal all the time, so 'busy wait' behaviour abounds. As an aside, such assumptions can make Citrix implementations interesting.
Script capture and script driving do not need to take the same approach. For ORACLE OCI, we capture from the network, and drive using the API; for Citrix we capture mouse-clicks and keystrokes, but again drive with an API.
PCs need to be tested and their contribution to end-user response understood but this is usually best handled independently of the testing of the multi-user components (e.g. WAN, LAN, load-balancers, middle tier/application servers, web servers, database servers).
Our minimalist approach extends to our Software Licencing Charges.
We don't make any.
We feel that our competitors' packages cost in license terms much more than is necessary for an entire testing project and thus without the need for multiple testing projects cannot be the most cost-effective approach.
The Benefits of PATH
PATH has a 17 year track record and has been used with the widest range of technologies.
PATH offers a flexible and versatile approach thus reducing costs and time scales.
The test drivers can be run from a mixture of operating systems including all flavours of UNIX and Windows.
PATH can be used to intelligently drive any protocol and can thus be used to simulate data feeds and other service bi-directional interfaces in additional to any form of terminal/PC activity. For instance, IP Phones accessed via Computer/Telephony Interfaces.
PATH script drivers can be distributed around a global network, tests run, results collected - all without leaving the central test control point.
PATH and E2 Systems represent the ultimate in Cost-effective Performance Testing. But don't just take our word for it. Have a look at the Web Version of our Workbench (we also have Green Screen and Thick Client versions) here, and see for yourself how you can lop a zero off the cost of your Volume Tests.
|
