Skip to main content

Evaluating performance on Android devices

Performance refers to a device’s ability to swiftly accomplish a task that matters to the user. When talking about performance users might think of speedy app launches, high benchmarks scores, smooth gaming experience, or that innate (but hard to describe) feeling of the device not holding you back during usage.

The absolute limit of a device’s performance is capped by its hardware. However in reality a device’s performance is usually held back by its software. As an analogy, a car may be swift but its speed is limited by traffic laws and the amount of grip between its tires and the road.

For smartphones, we can broadly evaluate performance in the following aspects:

  • Speed
  • Responsiveness
  • Smoothness

Speed#

Does it take forever to get past the Temple Run loading screen?

Users tend to be more familiar with this. Speed is the measure of throughput (i.e. how much work can be done in a span of time). A faster CPU can process tasks quicker, a faster GPU can render frames in less time, and faster storage can provide a higher rate of data transfer.

Several ways to evaluate speed include the following:

  • App launch times
  • Fingerprint unlock times
  • Web page load times
  • Time it takes to capture a photo on your camera
  • Benchmark scores (Antutu, GeekBench, etc)

All things being equal, the speed of a device increases with higher CPU/GPU/other hardware frequency. OEMs often implement CPU and GPU frequency boosting during intensive workloads, such as scrolls, flings and app launches. Several OEMs also tune their cpufreq governors (e.g. Interactive governor) to ramp up frequencies quicker on heavier loads.

Smoothness#

Is the Temple Run gameplay non-stuttery?

Does it stutter a lot when I fling through Google Calendar?

Smoothness is the measure of the ability to meet tight deadlines and provide a buttery and non-stuttery experience. Devices that exemplify excellent smoothness include the Google Pixel 2 and Razer phone. Smoothness is affected by a phone’s display hardware and frame rendering performance.

The majority of Android smartphone displays can handle a refresh rate of 60Hz, which means that the display hardware is capable of producing 60 frames every second (fps). Several phones, such as the Razer phone boast a maximum refresh rate of 120Hz, which give them the potential to provide a smoother visual experience than 60Hz displays.

Frame rendering performance is critical as well, and relates to the fps that the device can render. It is meaningless to have a high refresh rate if the device’s fps cannot keep up. Stutters and janks can be seen when the device fails to produce the same fps as its display refresh rate. As an example, a phone that renders at 30fps on a 120Hz display will likely appear more stuttery than another phone that achieves 60fps on a 60Hz display. To appear smooth a device with a 60Hz display has to achieve 60 fps, which means it needs to render and display one frame within the deadline of 16.7ms (1/60 seconds). On the other hand, the Razer phone has to render and display a frame within 8.3ms (1/120 seconds) on its 120Hz display. I tend to be more interested in frame rendering performance than display refresh rate, because the former is something that can be improved via software.

It is harder to qualitatively evaluate smoothness than speed and responsiveness because users have different sensitivity to micro-stutters and janks. A more objective and quantitative method of checking whether your device appears smooth is to enable Android’s Profile GPU Rendering feature1 2. In Fig 1, the horizontal green line represents the 16.7ms deadline (for 60Hz displays). The height of the bar corresponds to the total time required to render each frame. A bar that is higher than the green line means that a frame failed to render within the 16.7ms deadline, and hence a jank occured.

Fig 1. OnePlus 6 Settings app

Fig 1. OnePlus 6 Settings app


Note: Profile GPU Rendering does not account for touchscreen latency, which is hardware-specific. Special measuring equipment3 is required to provide a more accurate gauge of overall smoothness.

Janks are usually caused by inefficient task scheduling, delays in I/O, or very heavy processing. Some janks are inevitable, such as those caused during scrolling through a web page that is loading a lot of content over a slow internet connection, or launching a resource-intensive app. Sometimes janks are the result of an inefficiently-coded app. What we can try to do is minimise the time it takes to render each frame for common cases, such as scrolling through lists, viewing animations and gaming (i.e. achieve a lower bar height in Profile GPU Rendering). This can be accomplished with a combination of frequency boosting and improvements to task scheduling.

Let us imagine the frame rendering process as a babysitter bringing a baby up an elevator [Fig 2] to the highest floor of a skyscraper (don't ask me why) within a deadline of 30 seconds. The speed of the elevator is analogous to the CPU frequency. A speedier elevator (frequency boosting) will reduce the time it takes for the babysitter to reach his destination and increase his chances of meeting the deadline. However, what if the elevator is crowded with other people trying to reach other levels? As fast as the elevator’s speed may be, it has to make many stops to drop off the other passengers. The babysitter might fail to reach the top floor even at maximum elevator speed! Perhaps it may be quicker overall for the babysitter to take a slower elevator that no one else is using. Likewise, scheduling a rendering task on a CPU with fewer tasks could improve frame rendering time due to less competition between all the tasks for that CPU core’s resources.

Fig 2. Crowded vs non-crowded elevator

Fig 2. Crowded vs non-crowded elevator


Although your device may rock a flagship-level SOC, it can still encounter janks if task scheduling is poor or there is insufficient boosting. This is why the smoothness of two devices with the same SOCs can sometimes feel so different! For real-life examples of stutters, you may observe how stuttery the Google Maps experience was on the Honor View 10 and Huawei P20 Pro [Fig 3] until a later software update that fixed the issue.

Fig 3. Honor View 10 and Huawei P20 Pro Google Maps (jankfest)

Fig 3. Honor View 10 and Huawei P20 Pro Google Maps (jankfest)


A device that performs smoothly under certain tasks may stutter a lot on another. OnePlus 6 on stock kernel performs smoothly during scrolling through the Settings app [Fig 1]. However it stutters endlessly [Vid 1] when I try to scroll through a webpage with a lot of text even at maximum frequencies. This goes to show that a device running at maximum CPU frequencies can still stutter if scheduling is poor! The good news is that such stutters can usually be fixed by improvements to software/kernel, as Huawei has demonstrated.

Vid 1. OnePlus 6 scrolling janks in Github


Besides evaluating smoothness by checking the Profile GPU Rendering graph in various apps, a standardised way of testing is to run a consistent workload such as those found in UIBench [Fig 4].

Fig 4. UIBench Jitter and Rendering/Bitmap Upload tests

Fig 4. UIBench Jitter and Rendering/Bitmap Upload tests


Responsiveness#

When you click on an app’s icon in your launcher, does a window immediately pop up or does the system hang for a moment before the window appears?

Does a transition from one view in the app to another view occur instantaneously or freeze for a moment?

How long does it take for the Share UI in an app to fully appear when I click the Share button?4

Responsiveness is the measure of how quickly a task can be started and provide immediate visual feedback to the user. It is affected by both smoothness and speed. For instance, when launching an app the system has to schedule the task quickly and give it priority over the other tasks. It should also ramp up CPU frequencies to rapidly load the app’s resources so that it can create and display the app’s initial window in as little time as possible after the user taps the app’s icon. The whole process should have minimal delays and feel instantaneous to the user. To improve responsiveness apps should be coded well to minimise delays, and the operating system should implement good boosting and task scheduling mechanisms to meet the needs of the specific workload.

To conclude, in this post I covered the ways that performance can be understood and evaluated. I have also presented a surface-level understanding of how performance can be improved. In future posts I will cover in greater detail and with more analogies how better performance can be achieved.


  1. https://developer.android.com/studio/profile/inspect-gpu-rendering
  2. Open Settings app. Enable and open Developer options. Change Profile GPU Rendering to "On screen as bars". You can also toggle it via the QS tile after enabling Developer options.
  3. https://github.com/google/walt/
  4. https://www.reddit.com/r/Android/comments/8h7ymm/google_please_fix_androids_slow_bloated_share_ui/