Instart Logic Safari Test Automation: Navigating Through the Jungle

1
SAFARI TEST AUTOMATION NAVIGATING THROUGH THE
JUNGLE
BY KARAN KUMAR AND JAMES CHUONG
2
At Instart Logic, our focus is on delivering high
performance, engaging web experiences to users
across devices with our Software-Defined
Application Delivery (SDAD) approach, with a
particular emphasis on optimizing for mobile
devices. To this end, it is important for us to
be able to test and quantify the performance
improvements in application delivery that our
service brings to the various web platforms.
3
With iOS being a dominant platform for
e-commerce it is critical for us to optimize for
this platform. In this post wed like to focus on
the particular challenges of Apples Safari
browser on desktop and iOS, and our approach to
automate testing of Safari and Mobile Safari,
thus making it possible to find both functional
and performance bottlenecks with websites on this
key platform.
SAFARI CHALLENGES
Unlike other browsers, Safari does not offer much
to automate the browser itself. To this effect it
is important for us to identify issues listed in
my colleague Rajaram Gaunkers recent blog post
Holiday Wish List for Browser Makers
Requirements for an Open Web in an automated and
scalable way. In terms of performance, Apple
recently added navigation timings as part of
Safari 8 for OS X and iOS8, but unfortunately
these were removed in iOS 8.1.1. This makes it
difficult to reliably measure performance on
Safari. In addition, these issues generally tend
to be non-trivial and can affect a websites
Quality of Experience, both in terms of
functionality and performance.
4
To this effect, we have developed a way to
automate testing of Safari and Mobile Safari,
enabling us to find both functional and
performance bottlenecks with websites on the
Safari platform. Now more on our approach and
testing methodology.
OUR APPROACH ARCHITECTURE
Our automated testing setup is currently a simple
client-server model. The client starts a test and
sends it to the server to run. In Selenium
Grid terms, the client would be the user, the
server is the hub, and currently the hub is its
own node.
5
We selected Selenium as the core driver for our
automation testing, since it is the leading
standard on several platforms, with many language
bindings. Selenium provides the Safari driver
with a JavaScript extension that allows it to
control the Safari browser. It also provides
the JSON Wire Protocol, which defines how
implementations of the web drivers should
communicate with browsers. Thus, all web drivers
will have the same API and can easily drive any
browser with the same code. One such
implementation is Appium, the open source project
that uses Selenium bindings to drive both Android
and iOS applications, on real and simulated
devices. In fact, as Selenium deprecated its own
iPhone driver, it recommended that users use
Appium or iOS-driver as a replacement. On the
Appium side, there is an additional application
being used  ios-webkit-debug-proxy by
Google. We use Appium, which is a node.js server
to get commands from Selenium. The node would
need to be able to start and manage the Appium
server to ensure that it is running and ready to
drive the mobile device. Apple has JavaScript
running through Instruments, a tool within XCode.
Appium has Ruby and Java bindings to translate
the Selenium API into Webkits Remote Debugging
Protocol, and thus allows Selenium to automate
iOS. Furthermore, the ios-webkit-debug-proxy is
used to translate Webkits Remote Debugging
Protocol into Apples iOS Webkit Debugging
Protocol. In our case, we are starting a test
remotely with our test configuration. The server
receives this test configuration, applies the
settings and runs the test. Using Selenium, it
starts up the browser and routes web traffic
through it. For mobile, the service starts up
Appium and ios-webkit-debug-proxy, hooks onto
Appium and starts Safari. Then, using Selenium to
drive Safari, the server navigates to the given
URLs.
6
MEASURING PERFORMANCE
Selenium itself can only perform very basic
functional tests by driving the browser, without
any way to measure performance. So in addition to
Selenium, we use BrowserMob Proxy, which creates
a local proxy that can route browser traffic. The
proxy allows us to shape the network so that we
can test under different network conditions.
Furthermore, the proxy allows us to capture the
network traffic into HTTP Archive Report (HAR)
files so that we can see the time it takes to
load elements on the page. It can be controlled
via its RESTful APIs. Now that we are driving
the mobile device, we just need to route the
traffic from the device through the machine.
Then, we can measure performance by routing that
traffic through Browsermob. The traffic is
received by the service, and recorded by
Browsermob Proxy. The recorded data is written
out as an HTTP Archive Report (HAR), and can be
sent to a reporting software for details and
aggregation.
7
Heres a visual depiction of the automated
testing (https//youtu.be/NoY_PGxroxg)
8
CONCLUSION
We hope youve found our take-aways on how to
launch both functional and performance tests on
Safari Desktop and Safari on iOS helpful. This
has given us at Instart Logic a way to conduct a
large battery of tests and enable us to deliver
the best possible Quality of Experience on Safari
with our SDAD platform. If you have thoughts on
this and other possible approaches, we encourage
you to share your comments with the community
here.
9
Visit our Blog for more information