Automated Calabash Testing on Marshmallow

Recently I upgraded an Android app to targetMarshmallow (API 23, OS v 6.0).  I made all the necessary changes to our source code, and the app seemed to be working as expected.  However, I was pretty bummed to see that almost half of our automated Calabash tests failed for Marshmallow!  Upon closer inspection I saw that all failures were due to one thing: the Permissions Dialog popup.

This app is using android.permission.ACCESS_FINE_LOCATION, and with Marshmallow’s new permission management scheme, requires users to explicitly accept the permission.  Right before your code attempts to access fine location, user will see a popup asking if they would like to allow or deny your application’s access to this permission.  Our automated calabash UI tests failed because the dialog was covering elements they were trying to access, and the tests could not proceed.

I needed to amend our Calabash Ruby code to handle the dialogs for Marshmallow permissions, while still handling the older permissions model.

Initially I wanted to add code that would:

1. Check if the Android OS is <= 23
2. If it is, press the “Allow” button for location

Unfortunately, Calabash is a UI focused tester, so it is just touching view elements on the screen.  That means it doesn’t have system level knowledge, like what OS the app is running on.  Since I wouldn’t be able to know for sure what OS I was on, I decided I would just always check for the dialog; if it was there, press “Allow, if it wasn’t there I could just move on to the next step of the test.

I moved on to finding the view elements of the permissions dialog.  I did this by installing the app via the Ruby console:

calabash-android console <path to apk>
reinstall_apps
start_test_server_in_background

Then I navigated through the app until the permissions dialog popped up. Finally, I ran a calabash query on every view element it could see:

query('*')

I ran the query on everything because I wasn’t sure how Calabash would “see” the dialog, and I didn’t want to miss it because I had queried on the wrong view element.

I got a huge list of elements back from Calabash, but frustratingly nothing related to the permissions dialog. This was when I learned that Calabash actually can’t “see” the permissions dialog at all because it is an Android system dialog, which Calabash is not authorized to see.

So obviously I had been thinking about testing this like an Android developer that has access to everything. 😉 I needed to start thinking more like a regular user.

My new strategy was to set the location permission to accepted via the app’s settings, before launching any calabash tests.  I would have to figure out how to do this programmatically, because calabash reinstalls the app several times during our test suite, and the permissions would have to be reset after each install.

After some searching, I saw a Calabash user suggest that you might be able to set permissions in your Ruby Calabash script via adb shell, but they hadn’t tried it out to see if it could work.  This sounded promising! The command I wanted to run via the ruby script was:

adb shell pm grant com.kiodev.myapp android.permission.ACCESS_FINE_LOCATION

Here is the breakdown of this command:

• adb = Android Debug Bridge; how we talk to connected android devices via command line
• shell = opens up a Unix shell so we can chat with our device
• pm = Package Manager; Allows us to get/set info about specific apps on the device
• grant <pkg name> <permission> = Give app <pkg name> official permission for <permission> so dialog boxes don’t pop up and re-ask the user about the permission during <pkg name> runtime.

I found the ‘Before’ block in our code that executes before each install, so now I just needed to hone in on the correct syntax to kick off the command.  I tried several incantations, but there was always something off.  Finally I remembered the wonderful fact that Calabash is open source!  I definitely could find a method in their code that kicked off an adb shell, and use that as an example.

In our Before block we call a calabash method clear_app_data so I started there.  It didn’t take me long before I found the method in the Ruby doc:

# File 'lib/calabash-android/operations.rb', line 571

def clear_app_data
  cmd = "#{adb_command} shell am instrument #{package_name(@test_server_path)}/sh.calaba.instrumentationbackend.ClearAppData"
  raise "Could not clear data" unless system(cmd)
end

Using the Calabash source code as my guide, I was able to pin point this as the code I wanted in my own Ruby script:

  cmd = "adb shell pm grant com.kiodev.myapp android.permission.ACCESS_FINE_LOCATION"
  raise "Could not set permission for ACCESS_FINE_LOCATION" unless system(cmd)

Now we can run through the same set of automated tests on all Android devices. Yay!

EDIT: Nov. 29, 2015

One of the awesome QA devs I work with brought to my attention the release of calabash 0.5.15.  In this release, all permissions are now granted for your app on Marshmallow devices.  Nice!  I took a look at the source code for this chaange, and it turns out you can check device API via calabash after all. If you want more fine-tuned control around specific permissions, check out the suggestions from Dan Bennu in this thread.