Testing and Monitoring platforms provide a broad range of user simulation methods such as protocol, headless, and real browser-based. In this post, I will outline the main aspects of those, followed by a comparison matrix which you could use for choosing an appropriate simulation approach.
Protocol-level Simulation
Protocol level or HTTP-based testing was very popular in the early days of our digital age. With the rise of rich web client technology, the proved simulation approach has been more and more outdated.
A typical HTTP-based test driver executes service requests and parses responses. Modern web 2.0 applications consist of many client-side scripts, which are totally ignored and not measured, in this type of test execution. In worst-case, complex use cases cannot be simulated on protocol level due to a shortage of client side generated ids.
Complex protocol based use cases can be difficult to implement. A performance engineer needs to deal with cookies, session ids, and other dynamic parameters. Depending on the used technology of your system under test, some web form names often change once a new version have been deployed, which will cause the HTTP-based script to fail.
// Sample SilkPerformer protocol level script transaction TMain var hContext: number; begin WebPageUrl(“http://lab3/st/”, “Greetings”); WebPageStoreContext(hContext); WebPageLink(“Join the experience!”, ” – New Visitor”); WebPageSubmit(“Continue”, CONTINUE001, “Main menu”); WebPageLink(“Products”, “ShopIt – Products”); WebPageLink(NULL, “ShopIt – Product”, 3); WebPageSubmit(“Search”, SEARCH001, ” – Search”, 0, NULL, hContext); end TMain; dclform CONTINUE001: “name” := “jack”, “New-Name-Button” := “Continue”; SEARCH001: “search” := “boot”;
After all, protocol level scripts are perfect for service based tests in continuous integration environments, uptime monitoring or also for stress tests.
Headless Browser Simulation
With the rise of web 2.0 technologies, the testing and monitoring business was faced with serious challenges. Rich browser applications could no longer be tested or simulated on the protocol level due to the missing client side logic during script replay. Therefore, several headless browsers have been introduced such as HtmlUnit, PhantomJS or SlimerJS. They are often built on top of WebKit, the engine behind Chrome and Safari.
Headless Browsers have all the advantages of real Browsers, and they run faster without the heavy GUI. Much test automation, monitoring, and performance testing platforms are using headless browsers as they allow realistic user simulation.
Some tool providers have built their headless browser engines and ran in maintenance pitfalls. They have to keep the pace with new browser versions. It’s highly recommended using free available headless browser kit because there is a broad community which works on improvements.
// Sample phantomjs script “use strict”; var page = require(‘webpage’).create(), server = ‘http://posttestserver.com/post.php?dump’, data = ‘universe=expanding&answer=42’; page.open(server, ‘post’, data, function (status) { if (status !== ‘success’) { console.log(‘Unable to post!’); } else { console.log(page.content); } phantom.exit(); });
Finally, headless browsers are ideal for test automation, performance testing, and SLA monitoring.
Real Browser Simulation
Web2.0 applications are full of JavaScript, Flash, Ajax and CSS. Without a full browser, it’s not possible to measure the actual end-to-end response times of the whole web page. Real Browser Monitoring allows you to verify the site’s functionality and performance as perceived by the end-user.
A typical real browser monitoring solution collects loading times of images, javascript, CSS and more. Often they provide waterfall charts, which visualize the load time of those components.
One of the biggest disadvantages is their footprint. A single real browser script requires much more CPU and memory resources on the load injection or monitoring machine. Therefore, real user based simulation is not recommended for stress test scenarios.
// sample SilkPerformer real browser-based script transaction TMain begin BrowserStart(BROWSER_MODE_DEFAULT, 800, 600); // navigate to the login site BrowserNavigate(“http://demo.com/TestSite/LoginForm.html”); // set the authentication for the secure site BrowserSetText(“//INPUT[@name=’user’]”, “BasicAuthUser”); BrowserSetPassword(“//INPUT[@name=’pwd’]”, Decrypt3DES(“Ax7/X9sk1kIfHlbAZ434Pq4=”)); // submit the form BrowserClick(“//INPUT[@value=’Submit Query’]”, BUTTON_Left); end TMain;
After all, real browser simulation is useful for acceptance testing or SLA monitoring. Don’t use it for performance or stress testing because the resource footprint is too high.
Comparison Matrix
Obviously, there are good reasons for protocol, headless or real browser-based user simulation. The table below provides some guidance to choose the appropriate approach. Criteria HTTPHeadless BrowserReal BrowserRealistic User simulation
NoYes, but with some limitationsYesEasy script creation
No, depends on application complexityYesYesRobust script replayYesYes, sometimes tricky tweaks requiredYesEasy Script maintainability
NoYes, debugging could be trickyYesMulti-Browser Support
YesYes, but with some limitationsYesLow footprint on load injection machineYesYesNoGood for continuous integration
YesNoNoGood for performance tests
YesYesNoGood for
stress tests
YesNoNoGood for uptime monitoring
YesYesNoGood for SLA monitoring
NoYesYes
Keep doing the good work and share your experience with me.
Comments