Blog Archives

SOFTWARE TEST AUTOMATION – THE WHYs, WHATs, WHENs and HOWs?

Let me cut the crap and jump over to the actual topic. That’s the least I can do for you J

Why should I think of automating the tests?

The product development has reached a milestone and yet to be enriched with more features. We wish to make sure that the new changes don’t break the existing features. Yes, we can always test them manually during each release. However, there are two major demerits.

Let us assume the releases to the test team are located far apart. During the time gap the product might be updated with many changes. In an agile environment we would like to verify often or when the changes made to the product, that the changes have not broken the existing features than waiting for the next test cycle. It would be impossible to verify often all test cases manually. So a right approach will be to automate the tests for the existing features and run them and report the deviations every release. Verify only those test cases manually which are difficult to automate and we do not wish to automate.

Let us assume the test cycles are not far apart. Over a period the manual test efforts will keep on growing as the more features Test Automation - TestInsaneadded. If we have to keep the test cycle short and unchanged then we would need more manual test hours. While we plan everything in terms of cost and resources (a human is treated as a resource), we forget that it’s living humans who are testing the nonliving product. The tester who has to repeat the same tests every test cycle starts to feel monotony and may not test the product with same keen and attentive state of mind thereby failing to make sure the quality of test. A hard truth! That’s why many complain software testing is boring and non-challenging as there is less time available for him/her to acquire new skills and use them to test the product with fresh mind-set. The solution is to automate them and manually test only the areas which are not covered by the automation tests. The test engineer can now explore new ways to test the earlier features at the same time focusing more on testing the new features introduced.

What if we intend to repeat each test several times to test for the reliability? A tester will die of boredom or lack of sleep. But machines can run all day and they can do repetitive task uninterrupted at better speed than a human can. So we have hired machines to work longer and faster to do repetitive and boring tasks.

The take-away from automating the tests are,

  • Save the cost of delivering stable product by reducing the manual efforts in relatively quick time compared to manual tests.
  • Make testing less boring and monotonous for our engineers so that they can focus on acquiring new skills to test the product better.
  • Consistency in test quality as there is no human intervention (Remember if the automation tests are not written rightly then the purpose of the automation may not be met).
  • Immediate feedback of the changes to the developers in CI environment so that they can fix the changes immediately than waiting for the next test cycle.
  • Improved speed in achieving the better quality and reduced test cycle duration.

 

What and When to automate?

It’s not practical to automate everything and it can be done. But, it is very challenging to deliver the best test results which a skillful tester can deliver as one can think of better permutations and combinations of actions and creative test ideas which may not be automated for valid reasons. I am not saying we cannot. There are products which have more test code than the product code and that’s a sign of good product. But, as mentioned before we can cover major use cases of all important features which make sure the greater stability of the product with good volume of test data (of differing combinations and permutations).

  • We should automate test cases which are difficult to be tested by the human as we are limited by speed and ability to generate huge test data and test for all the cases in limited time.

 

  • We should not automate the test cases which add little value and the more cost in automating. Say, CAPTCHA string recognition in automating login feature. Rather I would disable the CAPTCHA in the development server and automate the login feature (unless the CAPTCHA generation is purposefully weakened in test environment and their order is made known to enable the automation in development environment). Otherwise, I would need a module which would do CAPTCHA image decoding, which is an overhead mostly. Say, the charts are generated and the numbers are to be verified against the chart values. We would need to do chart image decoding. The easier option is to do image comparison to expected image if we know the test data input. Imagine testing a photo organizing app which organizes photos belonging to a person as album. How do we automate the test case to verify if the photo belonging to a person are grouped as an album? Unless we have predefined data and we know the expected result to compare against, the test case cannot be automated for any test data which is dynamically generated. However, it still helps to verify that the feature is not fully broken by checking it to work with the small test data we have (and we know the expected results).

 

  • If certain features of the product are expected to behave consistently when the same tests are performed repeatedly for long hours, several times as required by the product requirements, the automation is the only way forward. An example would be channel change in Television to work without failure for several times which is typically high.

 

  • We should automate when most of the feature implementation is complete, requirements almost frozen and stable. Otherwise, we end up maintaining the automation suite often than expected.

 

How to build a good automation framework?

  • Carefully identify test cases keeping automation as the end goal to assert the stability of the application under test. The test cases should cover most important use cases with a goal to uncover bugs and assure the features work across releases meeting least required quality.

 

  • The automation scripts and test data separation is a must. This demands the need to write generic code which enables testing for wide range of test data and test cases without a need to change or add the code or for a developer to get involved. Ability to change or edit the test data should be easier and adaptable to the changes in the minor changes or enhancements to the product features.

 

  • Avoiding dependency between test cases as much as possible unless the test case and test data get too big. Think of an application which allows the user to navigate to other pages/screen and use them only if they complete registration form. Let’s assume there are 5 features to automate and registration being one of them. Let’s assume we have coded and automated 5 test cases where first test case executed is registration. All other tests cases are executed if the first test case which is registration pass. If the registration fails for some reason rarely (unless failing always) then all other test cases are skipped or fail. I do not like that. So I would make registration as one of the step to be executed in every test case. Even if the registration fails in one of the test case, rest of the test cases executed successfully.

 

  • Choosing an automation software and the binding (programming language) which has immense support for other plugins which are needed and enhance the automation framework. The tool selected should be stable, not too difficult to learn, has good technical documentation and support, the fixes for serious bugs are delivered quick enough and lastly should have the features to automate almost everything that we have and we will have in the product to be automated. If the tool supports multiple bindings I would prefer a binding which has wider support for third party plugins to support the features which are most needed in an automation framework (Say, test results reporting, test case management, and spreadsheet read/write plugins such as JUNIT, TestNg and apache POI etc. TestNg is supported in java, python does not support it). If there are multiple bindings which are equally good, I shall prefer the one in which I am more good at and comfortable than learning the new programming language. But, there have been organizations who prefer a binding(programming language) which is not having support for plugins as much as other bindings for a reason that they are mostly comfortable with that binding and do not intend to migrate to new language which would demand time to learn and code proficiently. Change is always a difficult option!!

 

  • Implementing test cases which are easy to scale and maintain. If I have to give an example, think of a login page in a web application which needs to be automated. We write tests which automate entering the username and password text fields and click on “submit” button. The script works as expected now. Tomorrow the username and password text fields order are exchanged. Unless the developer has though over this and has coded the scripts with locator which are robust enough to rightly locate the username and password fields irrespective of the order is bound to fail.

 

  • Implementing test cases with readability and maintainability as one of the necessity. The automation scripts should reuse the code. So always write helper functions common across all products and specific to a product and reuse them to reduce code clutter and to enable readability and modularity. Say an application has number of pages where we need to enter the mailing address having fields first name, last name, building number, street name, city, state and pin code. Instead of repeating the same lines of code for test cases for different pages which prompt for mail address, I will add a helper function which accepts all fields as arguments and edits the fields. Now each test case need to reuse this function with the right data passed. The result is scripts are more readable. Imagine there is a bug or a new field has been added to “mail address” block say, landmark. All we need to do is extend the helper function. So easy to maintain.

 

  • Apply and enforce coding guidelines so that it’s easier to understand and modify the scripts over a long time. Even if the original developer of the script has quit the organization or during original developers absence the other developer does not find difficulty in understanding the implementation to fix or to change existing scripts to address new requirements.

 

  • Ability to generate a test report which would give enough information about why the test case failed and other useful statistics (without a need of human presence as the automation suite is run). This demands ability to embed extra intelligence which would attempt to diagnose to zero-in on the cause for failure and other supporting details.

 

  • It should enable test case management, so that we can choose the test cases to skip in easier way instead of getting into at the code level.

 

  • Ability to support continuous integration and test result reporting by mediums such as email or SMS (as an SOS for test case failure) to the stakeholders to get a quick feedback and act so. Triggering automation run needs to be time driven (say every day at 10:15PM) and event driven (some code changes are committed to code repository or new build generated in the build server triggered by code change commits).

 

  • The automation framework should enable necessary level of customization by means of configuration files which can either be edited directly or by using a GUI tool than getting into the code and other technical aspects. The other aspect is to apply configuration changes without having to rebuild the automation framework unless it hits the performance of the automation in action or it’s highly difficult and value added is not worth.

 

  • Speed of the test execution is crucial when we intend to test large number of test cases with even larger amount of test data. Should avoid fixed delays except rare cases and design should also be taken care at function level for speed. It would be great if the automation tool supports running the tests in parallel on multiple machines with different environment/platform to speed up the tests and to verify cross platform compatibility.

 

  • Provide test coverage report at test case and code level (at code level it’s called code coverage).

 

  • Ideally should be test platform/environment agnostic (OS, browser, machine architecture, time zone etc.) as far as possible. This needs intelligently designing the framework.

 

These are purely my opinions about a good automation test framework based on my experience and learning in software automation and software development in general. A constructive feedback or questions or discussion are welcome. Good day!

ABOUT SANDEEP TUPPAD (VP, Mobile Apps Testing and Automation)

SandeepTuppad - VP, Mobile Apps TestingSandeep Tuppad has a decade of experience in software development (Drivers & Application Software) for Consumer Electronic Devices such as Digital Television and other display devices. He has prior experience of implementing automation framework and utilities aiding software testing. At TestInsane, he is responsible for exploring and implementing test automation framework, utilities, gathering knowledge and mentoring the team to enable “quality mobile apps testing”. He loves to travel and pursue outdoor adventure sports.

 

Try our web and mobile test automation services – Buzz me at sandeep (at) testinsane (dot) com.

TestInsane - Web and Mobile Test Automation

TestInsane – Web and Mobile Test Automation

Sandeep Tuppad’s View & Experience On Mobile Application Security Testing

The smart phones and the availability of the connectivity has simplified many things in our lives. Whether it’s about navigating the streets or connecting with our friends and family or mobile banking or paying bills or booking movie tickets online. There is absolutely no limit. However it has also made the smart phone users and the service providers vulnerable to the security threats. It could be threat to finance or private information. So smart phone users and app developers need to be smarter to minimize the risks. However, most smart phone users are not tech savvy and even if they are they may be blissfully unaware. In such a scenario the onus is on the app developer and app security testers to ensure security. Why do I say minimize? Because there are few smartest people out there who can hack into almost any software/hardware. The mobile devices which capture

Data Theft - Web App Security Testing

Data Theft – Mobile App Security Testing

most of the market are android and iOS. Other smaller players are Windows and Black Berry. Each platform is vulnerable, some more and others less compared to other. Open source platform android is more vulnerable compared to iOS as android app and OS internals are available to be understood. I also believe the app framework of android is naturally more vulnerable for exploitation by hacker.

There are two players who can be impacted by the security of the app.
1) App “User” –
The user of the app running it on a mobile device or tablet.
2) App “Service Provider” or “Business” –
The app provider such as WhatsApp, Bank providing online services to users, online Food ordering app etc. These can be Business or non-paid service providers running the servers to serve the customers using their apps.

There are two other players who need to ensure the security of the app.
3) App “Developer” – The one who develops the app for a service provider or business
4) App “Security Tester” – The one who tests the app for its intended use and vulnerabilities. Often hired by the service provider.

Security from App Users perspective:
The Apps in which the user credentials and private information is stored on device or on the server which an unintended person can get access by some means. It can be user’s login credentials, credit card/debit card details for an online shopping or user’s private conversation as in case of a social networking app or any other user specific details at more technical level which can be used by imposter commit a fraud or harm. These are just a few examples and not limited to that.

Security from App Service Provider or Business perspective:
When the app user himself/herself attempts to exploit the vulnerabilities which can harm the app service provider or business. Imagine the app is to make online shopping. The reward points are granted based on users buying habits. What if the app user is able to modify the reward points by exploiting the app’s vulnerability? This would cause loss to business. What if an app user is able to get access and modify to other users data which he/she not intended to access. This can cause great damage to the service provider in terms of credibility.

There is also a possibility wherein the app developer himself/herself has added malicious code to eavesdrop and steal sensitive data. It could be your contacts list, messages, location details, call log or anything private and confidential.

In my experience following are some of the many areas where app could be vulnerable if not implemented with security in place.

Area What’s the risk? Incidents I have come across?
Device Logs It’s possible that during the development stages of the app various sensitive details which are directly or indirectly a risk may be printed to console or to some log files. But developer may forget to eliminate and they slip into production app. Credit card details being logged.
Rest API requests and responses logged.
Network traffic interception and tampering HTTP requests are unencrypted and can be intercepted and tampered by an intruder. With a session data stolen one can launch range of attacks. This is easily possible in public or untrusted hot spots.If the user himself/herself is intercepting the traffic whether HTTP or HTTPS (which can be unencrypted by the user if he/she has the malicious certificate installed on device) and the rest api calls are not implemented with security in mind then he/she can cause damage to business directly or indirectly by hacking into other user’s data.If the apps are caching the HTTP requests locally it can be a threat.
Insecure data storage in files/databases The app may store lot of sensitive details related to user or business on the device in files and databases. If these files can be accessed and the file contents are unencrypted it can be a risk. Private conversations in social networking app, which can be hidden in UI with password but can be read through access to database. We can modify the conversations too.
Also Found out the Lock Pattern password which is stored as an array of characters.
The C2DM registration ID which can be stolen to impose as the real user by the hacker.
Setting in file which enables rewards to a user. It can set again and again locally to get discounts.
Lack of Binary Protections and Reverse engineering If the apps sensitive code blocks are not obfuscated then they can be reverse engineered. This would give an unintended access to app’s sensitive logic which when understood can lead to possible exploitations. It can also be sensitive data being leaked not just the app logic.If the attacker is smartest he/she can modify the code with malicious logic, recompile, install and run. Think of an online shopping app which is loaded with the currency. Imagine attacker modifying the code block which verifies if the user has sufficient balance to make a purchase. The verification can be bypassed thereby allowing to shop irrespectively. Email login credentials found in reverse engineered code (SMTP Details).Critical code logics and sensitive strings discovered
App execution control flow manipulation Using GDB the app is run in debug mode. One can modify the variables and the control flow to exploit bunch of possible vulnerabilities.
Authentication and OTP codes If the authentication code or user verification codes are short and there is no limit to retries or sufficient incorrect authentication timeout before next retry then password can be hacked with automated test. In simple words its brute force attack.
Intra and Inter app data communication If the apps are placing the sensitive data in clipboard for later retrieval or to be passed to another app or within app it’s a threat. A malicious app can read the clipboard data and dump it to a file to be retrieved by the attacker remotely or locally when has the access.The data is passed between apps and within different entities within apps as in case of android app as intents. It’s possible to intercept the intent data using a malicious app installed on device if security measures not taken care while implementing an app.

Whether it’s about intercepting and tampering the network traffic or access the files and database or to launch brute force attacks or to reverse engineer and add malicious code to app or to intercept the inter app communication data, capture device logs, we need tools and utilities either running outside the device or on the device. Some of the tools are sophisticated ones and coming with our own such tool may be tough task. There are other tools which are not very complex to implement but I would rather use them if available to save my time and efforts. But those tools do have limitations in terms of the attacks I can carry out for a specific area for vulnerability. But tools limitations should not limit a security tester. She/he should go beyond the tools and perform the attacks manually with other means. If possible implement own tool which may be generic or has to be customized for every app to be tested for a specific area for vulnerability. But this would need the tester to educate himself/herself with the internals of the app framework and OS architecture. Having the knowledge of app development one can think of the possible security holes which may slip out into app. The mobile app framework (android or iOS) itself could be having limitations in terms of level of security in an area. So a real security tester should go beyond being just a tester and should be able to write or understand source code. Unless I know coding how can I exploit the app by reverse engineering? I have been developer most part of my life and at TestInsane, I am Developer in Test and Tester who brings in a different value to testing as a Developer. The security testing thrills me. Why? Because it’s challenging and feels like treasure hunt. My experience of being a developer has truly helped me in this never ending endeavor.

But the mobile devices come with user permissions which won’t let us access the protected file systems, access device shell and execute privileged commands or install new software (Only the App Store is source in case of iOS). These operations are only available to super user (also called root user).So we can’t install tools and utilities on the device meant for the security testing. But can we gain super user access? Yes we can. It’s called rooting the device. On android it’s easier. But on iOS it depends on release version. Apples iOS is not open source, so it’s the hackers who discover the iOS security loop holes to gain root access. Once rooted any third party apps and binaries can be installed and run. However rooting comes with a risk of bricking (leaving your device un-operational which is as good as brick). But successful rooting is equally rewarding considering what you can achieve as a user or a security tester. But from a developer point of view I would say the app (which needs to be implemented with high level security in place) should be capable of discovering and taking a counter measure to mitigate the risk of being exploited if device rooted ϑ

Security testing is a specialized skill beyond holding tester or developer tag.  It needs time and hunger to keep learning forever. Think of a fighter who has all lethal weapons and knowledge of the weapons. But that alone not enough to win the battle. He needs know how to fight using them (and without them) while continually exploring, acquiring or building and adding new weapons to his repertoire. So tools are like weapons, they help you fight better but they are not everything.

ABOUT SANDEEP TUPPAD

SandeepTuppad - VP, Mobile Apps Testing

SandeepTuppad – VP, Mobile Apps Testing

Sandeep Tuppad has a decade of experience in software development (Drivers & Application Software) for Consumer Electronic Devices such as Digital Television and other display devices. He has prior experience of implementing automation framework and utilities aiding software testing in the past. At TestInsane, he is responsible for exploring and implementing test automation framework, utilities, gathering knowledge and mentoring the team to enable “quality mobile apps testing”. He loves to travel and pursue outdoor adventure sports.

Follow him on LinkedIn at https://www.linkedin.com/profile/view?id=145478507

Our Insight On Mobile Test Automation Tools Appium, MonkeyTalk & MonkeyRunner

Mobile Automation Testing

Mobile Test Automation

There are many automation tools out there in the market. Both open source and commercial. We made an attempt to try some well-known open source tools to find out if they are useful to adapt for mobile app testing. We have documented their features and the limitations as we tried them to test some of the android apps from Google Play store. We are yet to discover their value on iOS if they are supporting iOS. Calabash is one more open source tool which we would be trying sooner. Going forward we will try out commercial tools if we feel they are worth giving a try. We will share our thoughts as we do it.  Please feel free to give a constructive feedback.

Monkey Runner by Android:

  • There are three classes in package com.android.monkeyrunner: MonkeyDevice, MonkeyImage and MonkeyRunner.
  • The MonkeyDevice, represents a device or an emulator. The class includes methods to perform various UI and user actions on device. The MonkeyImage represents a screen shot image of the device or an emulator. MonkeRunner: This includes methods to establish connection between device and monkey runner programs and managing the execution of the monkey runner program. Individually they support set of operations as explained below to enable automation.
  • The methods supported are drag gesture, touch, key press, type equivalent to consecutive key presses, wake up the screen, reboot device, launch an activity, install a package, uninstall a package, broadcast intents as if it’s generated by app, Execute a shell command, Take screen shot which is a monkey image. Read the device property variables such as current package and class of activity, device name, screen height, screen width, capture the screen shot, write it file or compare with the other image etc.
  • The touch and drag methods need the co-ordinates on the screen to be specified, which limits the developing the scripts which are scalable and can easily maintained. However there are plugins developed by third party but they seem to be in primitive stage.
  • Provide methods to enable interactive execution of the script like prompt for inputting the data, prompt list of choices.
  • Supports only Jython as scripting language.
  • Supports only android apps and doesn’t have support for UI locators. UI interactions are co-ordinate based. All these limitations make monkey runner not a good choice for automation.

Monkey Talk by Gorilla Logic:

  • Supports android and iOS platforms including corresponding emulator/simulator.
  • Does instrumentation of the app (The instrumentation of apk distorts the layout).The agent on the device establishes connection between app and the off device tool.
  • Supports test case Record feature.
  • Supports exporting the recorded actions scripts .mt files into java script .js code.
  • Supports ant interface to run the .mt scripts enabling CI support.
  • Supports data driven testing by executing the parameterized editing of the .mt scripts with default values. The data is part of a CSV file and specified as source for the script.
  • Supports verification commands by checking the value of the component matches the argument by absolute value, by wildcard, by regex expression and by image
  • Supports think time and time out in milli seconds as the delay for each command as well as global think time and time out if not specified for any command
  • Supports command to wait for an element optionally with a time out (default 10s). Also has waitfornot which is inverse of waitfor
  • We can call .mt scripts within .mt scripts using the monkey talk language.
  • Can edit test suites .mts which invoke multiple tests which are test scripts. Before invoking they call a setup script and post execution a tear down script with optional CSV file as input or other arguments. This is a GUI based editing. The tests are in line with the JUNIT tests and the results are pare part of the junit panel and an xml file.
  • Supports JAVA API’s to write the tests as pure java standard junit tests with the support of monkeytalk-java-all-in-one.jar in its class path. Can integrate with junit runners such as eclipse, ant and maven to clean, compile, run and report the results of the junit test code. The reports are generated as xml/html files.
  • Supports elements such as text input, text label, spinner, list, menu, radio group, check box, radio button
  • Supports gestures such as tap, move, drag and swipe (Note the gestures recording didn’t work when tested with android app, not very reliable I would say).can capture screenshots.
  • Customer Support lacks big time at least for open source community
  • When instrumented the layout is screwed by the monkey talk. To verify view the layout elements and their mapping in layout using the uiatomatorviewer tool with app running on and without instrumentation.
  • Cannot launch the app. Only upon launching the agent establishes connection.
  • Agent is not very reliable in terms of establishing connection when the app in background is brought to foreground i.e. resumed.
  • Cannot test for scenarios where the app under test launches another app which is not instrumented. Cannot test with the devices pre-installed apps.
  • Does not support key presses.
  • No mention of pinch and zoom or multi touch gesture support
  • Can invoke native shell commands in the scripts.
  • The support isn’t good for the queries on forum. The product bug fixing is slow.
  • Has good support for non-critical and good to have features such as CI, scripting and recording tests.  But some of the core features to enable testing such as gestures support don’t seem be stable with record and playback. It does too much instrumentation.
  • Does not support Windows mobile apps

Appium by Sauce Labs:

  • Appium is a server and the client implements Mobile JSON wire protocol. The clients can implemented as webdriver compatible binding for Java, Python, Ruby, Perl, C#.
  • Supports android and iOS platforms including android emulator and iOS simulator. Supports native, hybrid and mobile web apps.
  • Implements subset of selenium based API’s and then the custom API’s compatible to both android and iOS. However there are some platform specific as well as emulator/simulator only API’s.
  • The iOS app’s can be automated with appium only on Mac-OS X. However the appium for android can used on windows and Linux. Check for the corresponding versions of the tools and software for compatibility.
  • It looks like java binding supports most number of the API’s. I tried python bindings to try the automation for some of the apps but java has more methods. There are other scripting language support such as ruby, C#, python, java script etc.
  • It has support for vast number of actions. single and multi-touch gestures like tap, flick, pinch, zoom, move, drag and drop, swipe, long press by element id or co-ordinates, shake, Has key press support, can start activities, install an app, remove an app, launch an app, close an app, lock screen, check for locked screen, hide keyboard, reset app (stop and launch?) check if an app already installed, open notifications, force the app to back ground.
  • Support various class of locators by name and by accessibility id, locators specific to android (uiautomator) or iOS (uiautomation).
  • Supports Selendroid with limited features, for Android 2.3 and above for native or hybrid apps, above android 4.2(API level 17) uiatomator support is available. However some of the features such as zoom/pinch are available starting API level 18.
  • Supports pull and push file interfaces.
  • We can invoke shell commands as well in client program. At least I could do it in python for android.
  • Web apps are supported with safari on iOS and browser on android.
  • In case of hybrid app’s there is an interface to switch the context to any of the web view and native view specifying the context id. Can identify multiple contexts
  • Works across apps without instrumentation.
  • Has fairly good support with its discussion forums (involves mostly appium users and sauce labs support engineers) and documentation on-line.
  • The product is being improved with new features and being stabilized at good speed.
  • Does not support record test case feature.
  • Does not support Windows mobile apps.

 

What features make a good mobile automation tool?

1) Support for most well-known mobile OS platforms such as android, iOS and Windows. Other one being blackberry.
2) Support for most famous versions of the OS’s (Android, iOS and Windows are most important).
3) Supports record and playback feature.
4) Ability to export the recorded actions as standard test scripts (preferably junit tests) and able to be edited by the engineer. Ex: testing login feature with data sources.
5) Supports invoking all kinds of UI objects (or views or controls) and their discovery by various search criterion (by name, by type, by class name, by id etc.).
6) Supports all kinds of gestures such as swipe, pinch, zoom, flick, shake, scroll and patterns.
7) Does not need any instrumentation of the apk, thereby testing the app as it is.
8) Does not need to provide apk source as in case of testing the pre-installed apps on mobile.
9) Ability to record or write the test scripts for one platform (OS and its version) and run the same scripts on other platform with minimal changes.
10) Provides supports for the continuous integration for agile development and testing.
11) Supports verify (or assert) during test and generate test report at the end of each test.
12) Supports native, hybrid and web apps.
13) This is not really a feature automation tool should support. But, it’s good to have. The sauce labs mobile cloud is compatible with the appium test scripts. Xmarin cloud supports calabash for android and iOS. Calabash is an open source automation framework supporting ruby binding. If the customer needs us to test with various devices to address device fragmentation and we do not have those devices in our organization then we need to have an account for mobile cloud. With appium or calabash we know which cloud to choose if it has those devices which we are targeting to test on.
14) Support object based discovery of UI elements rather than image based to easily maintain and scale test scripts.
15) Support key presses and have support for things such as forcing the app to background and bringing it back to foreground, install app, remove app, launch app, terminate app, wake up the screen, lock and unlock screen etc.
16) Supports various well known programming/scripting languages such as java, python. C#, ruby etc. to write automated scripts for various test cases.