Android Accessibility Service: The Unexplored Goldmine

Accessibility service is a long-running privileged service/special feature of the Android framework designed to communicate to the user or perform actions on behalf of user/application installed on an Android device.

Examples include converting text to speech, providing haptic feedback when a user is hovering on something, performing touch/swipe gestures.
In this article, we’ll primarily focus on how to perform touch & swipe gestures on-screen on behalf of the user.

Basic steps are to declare accessibility service in AndroidManifest & creating a new Class extending AccessibilityService.

Here, creating separate resource XML for accessibility service metadata is mandatory to avoid exceptions from PackageParser.java.

canPerformGestures is meta-tag that let’s service to perform gestures on behalf of the user. This meta also can be granted at runtime.

Your accessibility service class should extend AccessibilityService. Primary functions should you override if required are onServiceConnected(), onAccessibilityEvent() & onInterrupt().
In case one need to add any flags or eventTypes at runtime then it can be done in onServiceConnected() via,

serviceInfo.eventTypes & serviceInfo.flags

All privileged actions must be done from the class which extends AccessibilityService or via the instance of accessibility service. For our example, we will be injecting touch & swipe gestures from this class.

GestureDescription is the first step to do so and it describes Gestures that are made up of one or more strokes. Strokes consist of Path, start time & duration. To create strokes another class Accessibility provides is StrokeDescription. Gestures are immutable once built and will be dispatched to the specified display.
Dimensions we specify throughout are in screen pixels & Time is in milliseconds.

Here’s the example of creating GestureDescriptions for simulating tap & swipe gesture.

For the click, X & Y coordinates on screen where touch needs to get performed. To make it a swipe gesture we just need another coordinates for where to end the gesture & duration for it.
Function buildSwipe() deliberately does simple horizontal/vertical swipe, just to demonstrate the capability of the service, it can be extended further by building Path as per your very own needs.

Once we have the GestureDescription, all we need is the instance of accessibility service to dispatch the gesture.

dispatchGesture takes GestureDescription as an argument, for which created buildClick() & buildSwipe() functions above. Second & third arguments are GesureResultCallback & Handler respectively, both of ’em are optional.

Once these are in place, one can create APIs out of these to make things easier for end-user or to automate things, possibilities are endless for this.

Gesture support on accessibility service was introduced from Android API level 24. Classes GestureDescription, StrokeDescription & function dispatchGesture all are available only from Android N onwards.

Starting Accessibility Service

To start the service, a user must navigate to Android settings > Accessibility > Your Accessibility Service > Click on Use Service.
These steps are very important as there is no way to start accessibility service programmatically. This is done very deliberately to avoid any android app taking over on entire device.

starting your accessibility service

We can ease a few clicks for the user by directly landing user to the Accessibility settings screen. This can be done simply by launching an intent from your activity.

val intent = Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS)
context.startActivity(intent)

Restriction for starting service programmatically remains the same for all other capabilities regardless if your service wants to perform gestures or not.

As soon as a user starts your service onServiceConnected() function is called.
To check if your service is started & running or not we need to fetch all enabled accessibility services on device and loop through it to match our service.

Much more can be done here other than just dispatching gestures for the user.
Like draw over other apps, accessibility service also allows putting content on the screen, above all. Doing so also is pretty simple.

As soon as the user navigates to Accessibility settings and starts the service, the layout will be visible on the screen.

Unlike drawing over other apps, which is just limited to manipulation within the package, here we have control across the device.
We can put some buttons which can perform a series of actions which may take require user ages to perform.
Another function provided by Accessibility service is performGlobalAction() which takes an integer as an argument.
This allows to perform an action that can be performed at any moment regardless of the current application or user location in that application. For example going back, going home, opening recent, etc.

public static final int GLOBAL_ACTION_BACK = 1;
public static final int GLOBAL_ACTION_HOME = 2;
public static final int GLOBAL_ACTION_RECENTS = 3;
public static final int GLOBAL_ACTION_NOTIFICATIONS = 4;
public static final int GLOBAL_ACTION_QUICK_SETTINGS = 5;
public static final int GLOBAL_ACTION_POWER_DIALOG = 6;
public static final int GLOBAL_ACTION_TOGGLE_SPLIT_SCREEN = 7;
public static final int GLOBAL_ACTION_LOCK_SCREEN = 8;
public static final int GLOBAL_ACTION_TAKE_SCREENSHOT = 9;

One of the few catches that make using Accessibility service a difficult is not being able to start the service programmatically. This, however, is obvious to not have a loophole in System. Still, privileges this service have are unmatched.

Another thing to keep in mind is, accessibility service needs to be restarted on each app update even if service was started by the user. This behaviour is the same even in case there was no change in accessibility service as part of App Update. This also is to prevent apps from misusing the privilege by exploiting service on an app update.
To avoid this, we can make accessibility service a separate module and install as additional apk in the user’s device when required. This will ultimately a new app without any activity or UI but only Accessibility service. This will avoid the android system from stopping service when your actual app is updated.

I have not covered each every function of what Accessibility service is capable of but it is worth to explore what all is offered on official docs.

I hope this article will be helpful in some way!
Would love to hear your comments/suggestions/corrections/questions!