Skip to main content

A Holistic Approach To Mobile Testing

Most of us probably have the same "mental image" of what it means to do mobile testing. We envision a person sitting at their workstation holding a mobile phone in their hands and pecking out the sequence of steps needed to exercise some function of a mobile application or service. On their desk we see a whole bunch of different handsets laid out in a row. When you finish all the test cases on one handset, you set it down, pick up the next handset and repeat.

A more advanced variation of this approach is where you "rent time" to access a particular handset in a hosted service model. It is still testing using a real handset, you are just accessing it remotely. Testing with real handsets allows you to verify that what is shown on the LCD of the device "looks cosmetically correct" for each step of the test case. The downside of this approach is that you have limited access to any type of diagnostic information other than what is shown on the LCD of the real handset. If your question is "Why is the page slow to load?", testing with a real handset does not provide detailed root cause analysis information needed to solve this problem.

For testing mobile internet content, a second approach is based on a "virtual device" or emulator. In this approach, the handset under test is mostly based in software. Why would you want to test using this approach? One of the biggest reasons is that a virtual device approach lets you see what is happening "behind" the LCD. Since the virtual device is in control of its own software stack, you can collect important testing information on "each component" of a content page. For example, how long does it take for each image on the page to load? Does the page contain URL redirects that slow down the loading process? Are all the images returned in a size that is compatible with the target device? What is the full URL of a broken link on the page? These questions can all be easily answered using a virtual device testing approach.


Each approach offers answers to certain types of testing questions. Using both techniques offers a holistic approach to mobile internet testing.

For example if the testing goal is to validate that the LCD "looks cosmetically correct", then the right approach is to test with the real device. Any type of virtual device could never mimic the exact behavior of the real handset.

On the other hand, what if you are testing with a real device and you receive the dreaded "Page Cannot Be Displayed" error? If you are testing with a real device, you can not quickly determine the root cause, but with a virtual device you could access the markup code of the returned page, as well as the headers of the content request. For these types of errors the virtual device is superior in finding the root cause of the problem.

There is also the issue of economics. To test with a real device you need physical access to that device. If you have multiple testers, and-or multiple groups doing testing in different geographies, then providing access to physical devices becomes problematic. You could "rent time" on remote devices, but the costs of this activity used for every single test case could quickly add up to huge amounts.

For certain types of test cases where "cosmetic accuracy" is not the goal of the test, a much cheaper testing model based on a virtual device can be used. Many types of content compatibility tests such as verifying image sizes, or identifying broken links can be accurately and quickly performed using the virtual device model. In most cases virtual device testing tools are downloaded and installed on your local PC. There is no hourly rental fee.

Tb3Lastly, the virtual device approach solves the "device scarcity" problem. What if you need to test on a device that has not been released? What if you need to test on a device that is brand new in the market place and because of demand, few real devices are available for purchase? These types of problems are also solved using the virtual device approach. The virtual device approach allows you to quickly define and configure new device "profiles". The speed of deployment is much faster than waiting for a scarce physical real device. Multiple testers can also share virtual device definitions so that scalability is achieved within the testing group.

Both approaches should be used in concert to create a testing methodology that is thorough, practical, and economical. In the same way that a mechanic or carpenter uses different tools for different purposes, so should the mobile testing group use a collection of tools in order to produce the highest quality result.

Comments

Popular posts from this blog

Apple vs Windows Font Rendering ::: Who gets the headache?

The Apple philosophy of showing screen fonts in a way consistent with printing is opposite of the Windows way of displaying screen fonts for readability. This issue has been discussed at length in the past, but because of the recent surge in popularity of Apple laptops and desktops, I wonder if this issue will resurface as more ex-Windows users switch to the Mac and find out that the fonts might look very different to them. I am a "dual mode" user. I have a XP machine I have to use at work, and at home I have a Mac. Since I was a Windows users for a long time before I started using the Mac, my brain is "wired" to think that the Windows way of showing the fonts looks best. My XP machine is a Thinkpad X60. The Thinkpads have always had really great laptop displays that are really clear and easy to read. Because of this, I run Windows with no font smoothing. To me it looks "pixel perfect" and I can stare at the screen all day with absolutely no eye strain. W

Living with a Greyhound

We have a retired racing Greyhound. His dog name is "Stormy", his racing name was "Village Luigi". Even though many Greyhounds are now being adopted, many people still haven't seen them up close. When I have Stormy with me, I get a lot of questions about what it is like to have a Greyhound living in your house. I will attempt to answer some of those questions. Of course there will always be exceptions depending on the individual animal, but these are my experiences. First Contact: Almost all Greyhounds that people have in their private homes came from a racing program. Racing is an industry, and the dogs literally drove it. You don't see ads in the local newspaper saying "Greyhound puppies for sale". Greyhounds are pure breed animals and the breeding process is tightly controlled . The ones that people adopt are animals that have left the active racing program. You might get a dog that raced dozens of times, or you might get a dog that only raced

The "Hurt" is over.........somewhat

It has been a while since I've been able to look at this blog. Mostly because of the posts regarding my pets. "Stormy" my first greyhound dying, then when "Sizzle" my second greyhound died, I couldn't really look at my old posts regarding my dogs, and I couldn't bring myself to delete them. So, I stopped looking. Shortly after Sizzle passed away, my other best friend in the world "Buddy" (the Coatimundi) also passed away from old age. He was 11. I still can't watch any dog/pet related movies, TV shows, etc. In our house we still have "Nilla" (the Papillon), she is also 10+, but doing fine. Anyway, time to move forward.