OUP user menu

Assessing the Quality and Usability of Smartphone Apps for Pain Self-Management

Charmian Reynoldson BSc, Catherine Stones PhD, Matthew Allsop PhD, Peter Gardner PhD, Michael I. Bennett MD, S. José Closs PhD, Rick Jones MD, Peter Knapp PhD
DOI: http://dx.doi.org/10.1111/pme.12327 898-909 First published online: 1 June 2014


Objective To evaluate smartphone apps intended for self-management of pain using quality assessment criteria and usability testing with prospective users.

Design 1) Survey and content analysis of available apps; and 2) individual usability study of two apps.

Setting University of Leeds, United Kingdom.

Participants Forty-one participants (aged 19–59 years) with experience of chronic or recurrent pain episodes.

Methods We undertook a survey, content analysis, and quality appraisal of all currently available mobile phone apps for self-management of pain. Two apps were then selected and assessed with usability testing.

Results Twelve apps met the inclusion criteria. The quality assessment revealed wide variation in their clinical content, interface design, and usability to support self-management of pain. Very little user or clinician involvement was identified in the development of the apps. From the usability testing, participants stated a preference for an interface design employing a lighter color scheme and particular text font. Although very few participants were aware of pain-reporting apps prior to participation, many would consider use in the future.

Conclusions Variation in app quality and a lack of user and clinician engagement in development were found across the pain apps in this research. Usability testing identified a range of user preferences. Although useful information was obtained, it would be beneficial to involve users earlier in the process of development, as well as establishing ways to merge end user requirements with evidence-based content, to provide high-quality and usable apps for self-management of pain.

  • Smartphone
  • Technology
  • App
  • Pain
  • Self-Management
  • Usability


Digital and electronic technologies are used in a variety of ways in health care, including inpatient assessment and monitoring, social support, patient education, and research [1], with mobile technology forming an important component of this growing trend. However, there is a growing divide between how national health services, and service providers interact with patients and those patients' lives that are furnished with technology [2]. For example, in 2011 in the United Kingdom, 92% of the population (and 70% of those aged 65 or over) owned a mobile phone, with 45% possessing a smartphone [3]. Similar rates are seen in other high-income countries.

The use of smartphones and their apps (downloadable programs designed to run on the smartphone operating system [4]) in health care has been demonstrated in a research context, for example behavioral change interventions [5] and the assessment of psychosis [6]. While the development and issuing of such apps has occurred in research, there is currently no available guidance for manufacturers generating health care apps for the wider market, although in the United States draft guidance is currently out for consultation [7]. It is important that technologies for health care should be accessible by all potential users, including older people, those with low literacy, and those with disability. However, mobile phone technology for health is often displayed on sophisticated devices with relatively complicated user interfaces and requiring high levels of manual dexterity and visual acuity [8].

Apps for use in chronic pain are a case in point. While pain self-management has been outlined as an excellent setting for new product development, early attempts were relatively primitive [4]. Recent UK data identified the prevalence of chronic widespread pain in up to 17.5% of the population [9], with the rate rising to 48% when including any specific-site chronic pain [10]. The opportunity exists to fuse research (such as effective interventions to manage chronic pain [11]) and pervasive modes of technology to deliver improved patient care. However, the lack of guidance on how to develop apps that are both suitable for users and aligned with evidence might be one factor inhibiting their potential. Health care benefits could include improved communication between professionals, and between professionals and patients—possibly leading to enhanced patient care and reduced costs [12].

This article provides two new perspectives on mobile phone apps in pain management. Firstly, it examines mobile phone apps available in the United Kingdom, which have been designed for self-management of pain. It builds on the results of a 2011 survey of apps [4] by developing assessment criteria and using them to assess the quality of available apps. The criteria that might be used to assess the quality of apps are variable and a matter of debate. For example, a recent survey of apps available for the self-management of asthma [13] used criteria that the researchers had adapted from criteria used to evaluate the quality of Websites. Secondly, it evaluates two of the found apps by conducting usability testing with potential users. The usability testing explored which types of information the users are able to input and store, and how easily, and asked them to evaluate both apps. We did not aim to evaluate how these devices complement or replace current clinical assessment and management of chronic pain.


The research involved two studies comprising: 1) a survey, content analysis, and quality appraisal of mobile phone apps used to record episodes of pain; and 2) usability testing of two apps with people with experience of chronic pain.

Approval to conduct the research was obtained from the Research Ethics Committee of the Institute of Psychological Sciences, University of Leeds.

The Survey


The study was a survey of currently available smartphone apps.



A search was conducted in June 2012 on the two largest smartphone operating systems: iOS (developed by Apple Inc., Cupertino, CA, USA) and Android (a Linux-based system currently owned by Google, Mountain View, CA, USA). iOS apps were searched using iTunes (Apple Inc.), and Android apps were searched on Google Play, a digital application distribution platform. Both iTunes and Google Play were searched using the keywords: “pain” (used alone and also in combination with) “track,” “log,” “diary,” “record,” “manage,” “management,” “scale,” “severity,” and “journal.”

Entry Criteria

For inclusion in the survey, apps had to:

  • Have as their primary function allowing people to record pain episodes;

  • Allow records to be saved for future reference;

  • Allow users to record pain of any type or location;

  • Have as the intended user the person with pain, rather than a health care professional; and

  • Be available in the English language.

Apps were excluded if:

  • Their intended use was limited to specific forms of pain or pain from specific conditions, such as arthritis, inflammatory bowel disease, or migraine.

Quality Assessment Criteria

Quality assessment criteria were developed for the study as there are currently no widely accepted criteria. The criteria drew on five components of an app, three of which were scored and summated to enable comparisons to be drawn (clinical content, interface design, and ease of use), and two of which were described (product description and development team). See Table for details.

  • Product description—assessed the app's specifications and, when available, its download data;

  • Development team—the team that created the app, including whether there was input from experts in health or pain;

  • Clinical content—whether an app captured details of a patient's pain, using the SOCRATES (Site, Onset, Character, Radiation, Associated symptoms, Timing, Exacerbating and relieving factors, and Severity) mnemonic [14],which is frequently used by clinicians;

  • Interface design—assessed using an adapted version of heuristics for user interface design [15] that concentrates on logical and thematic use of colors and fonts, and ease of reading and legibility;

  • Ease of use was assessed using heuristics for evaluation in mobile computing, designed for identifying usability flaws [16]. Ease of use focuses on the navigation, content, accessibility, security, and interactivity of an app.

View this table:
Table 1

The quality criteria assessment

Scoring was undertaken for clinical/content, design, and usability criteria only.
Items judged Y/N were scored as follows:
1 point = yes, the application meets the criterion.
0 points = no, the application does not meet the criterion.
Items below marked with an asterisk (*) were scored as follows:
3 points = 100% of the application meets the criterion.
2 points = 50% or more of the application meets the criterion.
1 point = less than 50% of the application meets the criterion.
0 points = the application does not meet the criterion at all.
A maximum of 72 points were available. When comparing apps, if a question was not applicable, the number of points that could be obtained from the question was deducted, for calculating the percentage of criteria that were met.
A. Description
  1. Name
  2. Operating system + version
  3. Price
  4. Presence of free “lite” version (if applicable)
  5. Download size
  6. Date of last update
  7. Number of downloads (if available)
  8. Number of ratings (if available)
  9. Average user rating (if available)
  10. Overall Purpose (as claimed by market description)
  11. Main method of recording pain
  12. Data input
    a. How is data inputted? (pain scale, calendar, body image, free text, and list)
    b. Is there a results section?
    c. How are results presented?
    d. Is there an email function?
  13. Extra resources
    a. Does it provide additional information about pain causes, symptoms, and/or treatment?
    b. What information does it provide?
  14. Any other features?
B. Developer
  1. Developer
  2. Individual or team?
  3. What is their profession/background?
  4. Have they worked with clinicians/pain experts if these are not already part of the team?
  5. What other applications have they created?
C. Clinical content
A good application for recording pain should include the ability to input the following details where appropriate: (Y/N)
  1. Site—Where is the pain?
  2. Onset
    a. What date did the pain occur?
    b. What time did the pain occur?
  3. Character—What is the pain like? What patterns does it appear in?
  4. Radiation—Does it move anywhere else?
  5. Associated symptoms—Is there anything else with it (e.g., nausea, blurred vision, and numbness)?
  6. Timing—How long was the duration of the pain episode?
  7. Exacerbating or relieving factors
    a. Does anything make it worse?
    b. Does anything make it better?
  8. Environment—Place in which the pain occurred (e.g., home, work, and outdoors).
  9. Severity—A rating scale (e.g., numerical—from 1 to 10, with 10 being worst pain)
  10. Medication or treatment—A separate treatment section allowing users to input details of medication taken or treatment undergone.
    a. Does it include such a section?
    b. What medication was taken/treatment undergone?
    c. When did it happen?
    d. How much was taken? (dose of medication; details of treatment.)
    e. How did it affect the pain? Did it make it better or worse, or have no effect?
  11. Notes—Ability to add extra notes about the episode (e.g., if something was important but not covered by the questions).
  12. Can options be modified (e.g., options can be added or removed to make application more personal/relevant)? (Y/N)
  13. Does the application remain true to the claims made by the developer in the market description?
  14. Results
    a. Is there a results section? (Y/N)
    b. Can results be filtered? (Y/N)
    c. Does it generate reports? (Y/N)
D. Design
  1. Are the colors used logically related (i.e., complementary colors or colors that fit to a particular theme, such as shades of one particular color)?*
  2. Is there a consistent color theme? (Y/N)
  3. Are the color choices visually accessible (i.e., easy to read with normal or corrected vision)?*
  4. Is the design audience appropriate?
    a. Information should be presented in a way appropriate for the target audience.*
    b. Standard text size should be readable for normal/corrected vision without adjustment for those who do not know how to adjust screen.*
  5. Are the font styles easily legible?*
  6. Is there a consistent font theme (i.e., one or two fonts used in a logical manner, not lots of different fonts used inconsistently)? (Y/N)
E. Ease of use
  1. Navigation
    a. All pages should be able to be reached in as few clicks as possible—from the main page of the subsection (e.g., a new record section, or results section), it should not take more than three navigational clicks to reach a page. (Y/N)
    b. Content should be structurally separate from navigational elements.*
    c. Clickable items should stylistically indicate that they are clickable.*
    d. Navigation should be logical and intuitive with signs obvious and not obscured.*
    e. There should be a call to action on every page—no dead ends. (Y/N)
  2. Content
    a. Details and content should be presented in a natural and logical order.*
    b. Details or content suit the purpose of the application and are appropriate for target audience.*
    c. There should be an “about” page identifying the author and application details. (Y/N)
    d. Resources for content not written by author should be credited. (Y/N)
    e. Resources for content not written by author should be reliable—from a credible source, backed up with evidence. (Y/N)
    f. Application should be regularly updated to keep up with changes in pain literature/guidelines—the last update should be within the last 6 months from the date of review. (Y/N)
  3. Accessibility
    a. Is the application available in other languages? (Y/N)
    b. Can text size be altered, or is it possible to zoom into pages? (Y/N)
    c. Applications should provide an easy way of inputting data, reducing or avoiding need for user to use both hands.*
    d. Does the application provide a help section/user guide? (Y/N)
    e. Does the application provide a way of contacting the developers for support (e.g., an email address/contact details, link to email application, forum, or link to Website)? (Y/N)
    f. Can the application remember details to avoid having to input the same details repeatedly? (Y/N)
  4. Security
    a. User's data should be kept private and safe—can data be encrypted in the event of loss or malfunction of system? (Y/N)
    b. Can information be backed up/restored in case of loss/malfunction of device? (Y/N)
  5. Interactivity and connectivity
    a. Can application connect to other devices (e.g., via Bluetooth)? (Y/N)
    b. Can information be sent via email or other alternative? (Y/N)
Quality Assessment Procedure

Android apps were viewed on an HTC Desire S smartphone with a 3.7 inch touch screen display, using Android software (version 2.3.5, Google) and HTC Sense technology (version 3.0, HTC Corporation, New Taipei City, Taiwan). Apple apps were viewed on a 4th generation iPod Touch with a 3.5 inch touch screen display and Apple iOS software (version 5.1, Apple Inc.). The specification required for displaying and inputting data into the apps (multitouch capable 3.5″ 960 × 640 “Retina” display) are equivalent across devices.

If an app had both a “pro” version (full version of an app available for a fee, with no limits on function, no advertisements, and full technical support) and a “lite” version (free to download version, which may have limited functions and/or be supported by advertisements), the full “pro” version was selected to view the app in its entirety.

The quality criteria were applied by one researcher (C. R.). The apps were assessed in the measures listed in Table . Information on the product description and the development team was captured as the app was downloaded. The clinical content, interface design, and usability were assessed after download, using the criteria checklist (see Table ).

View this table:
Table 2

Measures used in the usability study

Items 1–10 system usability scale (adapted); each item offers the following responses: strongly disagree (scored 1), disagree (2), neither agree nor disagree (3), agree (4), and strongly agree (5).
  1. I think that I would like to use this app frequently.
  2. I found the app unnecessarily complex.
  3. I thought the app was easy to use.
  4. I think that I would need the support of a technical person to be able to use this app.
  5. I found the various functions on this app were well integrated.
  6. I thought there was too much inconsistency in this app.
  7. I would imagine that most people would learn to use this app very quickly.
  8. I found the app very cumbersome to use.
  9. I felt very confident using the app.
10. I needed to learn a lot of things before I could get going with this app.
11. Do you have any overall comments about the usability of this app?
Items 12–18 design questionnaire; each item used a five-point Likert response format with wording on the poles (e.g., for item 12 “very unattractive” scored 1 and “very attractive” scored 5).
12. How attractive did you find the use of colors in the app?
13. How attractive did you find the fonts used in the app?
14. How pleasant did you find the layout of the app?
15. How professional did you feel the app was to look at?
16. To what extent do you feel the design suited the purpose of the app?
17. How pleasant did you find the look of the app overall?
18. Overall, how much did you like the design of the app?
19. Do you have any overall comments on the design of this app?
Items 20–23 used a five-point Likert response format with wording on the poles (e.g., for item 21 “not useful at all” scored 1 and “very useful” scored 5).
20. How detailed did you find the content of the app?
21. How useful did you find the “results” section?
22. How useful did you find the “reports” the app generated?
23. How sufficient did you feel the content of the app was in enabling you to record your pain?
24. Do you have any overall comments about the content of this app?
Items 25–28 scored “yes” or “no” with free text comments below.
25. Is there anything you would add to this app to make it more useful in recording your pain?
26. Is there anything in this app you felt was unnecessary or could be removed?
27. Is there anything you particularly liked about this app?
28. Is there anything you particularly disliked about this app?
Items 29–30 on future use (scored “yes” or “no”)
29. Would you download this app?
30. Would you recommend this app to a friend?
31. Do you have any final comments about this app?

Data Analysis

Mean (and standard deviation [SD]) app quality scores were calculated for each platform type.

The Usability Study


The study used a within-subjects design to compare the two apps, using counter balancing to vary the order of testing to account for potential effects of practice or fatigue.


Participants were staff or students at the University of Leeds. They were recruited opportunistically via publicity posters and group emails and were offered the incentive of inclusion in a prize draw.

Eligibility criteria for participants were that they: 1) should own a smartphone (such as an Apple iPhone, Android device, or Blackberry) and/or understand how to use one; and 2) should have some experience, past or present, of long-term pain or recurrent pain episodes.

The study included 41 participants aged between 19 and 59 years (mean 35.0; SD 12.0). Most participants (31; 76%) were female. Eight participants were undergraduate students, three were friends or relatives of staff at the University, and the remaining 30 participants were university staff. All participants reported having normal or corrected-to-normal vision. Informed consent was obtained from each participant prior to testing.

Participants reported their pain experiences most commonly occurring exclusively in “legs or feet” (20%) or “joints” (20%). Pain in multiple locations was reported by 24% participants. The most commonly reported cause of pain was “previous injury,” in 22% participants.

Tested Apps

Two apps were selected from those included in the survey, one of each of the two most common types of app (a pain diary and a pain scale). They were selected as the most frequently downloaded examples of each type and provided useful exemplars of the range of apps available at the time. Download data are not available on Apple products, and so the choice was limited to Android apps. The tested pain diary was Manage My Pain and the pain scale was Pain Scale (see Figure ).

Figure 1

Screenshots of the pain recording pages of Manage My Pain and Pain Scale. Participants used these pages to record their pain episode during the study.


  1. Time taken to complete a data entry task on the app (see Procedure below).

  2. Participants' self-rated confidence with smartphones (“How confident do you feel using a smartphone?”, using a five-point rating scale from “not confident at all”—scored 1—to “very confident”—scored 5).

  3. The system usability scale (SUS) [17]. It is a 10-item scale that captures ratings of electronic devices or systems (in this case, an app), including respondent assessments of future use, complexity, ease of use, and perceived usefulness of the display of results. Participants could also add additional comments.

  4. Design questionnaire. This seven-item measure was developed for the study and included questions with Likert scale responses. Each question was scored on a five-point scale, with high scores indicating more positive ratings. Participants also responded to six open-ended questions, to provide more detailed evaluation of design aspects of the app.

  5. App use questionnaire, which comprised two questions on participants' likely future use of the two apps and gathered the impressions of apps they had held before the study (see Table for all measures).


After assessment of eligibility, participants attended an individual 30-minute session, held in a quiet, self-contained testing room at the University of Leeds. After providing demographic data, they rated their self-confidence in app use. Participants were then given a standardized demonstration of how to use the first of the two apps (the order of testing was counter-balanced).

Participants were then asked to recall two recent occurrences of their pain and to use these as two scenarios, for use with both apps. Participants were asked to think about the pain memories and use them to record both episodes into the first app. They were asked to consider the first scenario as typical of the average level of pain they regularly experience and the second scenario to reflect the worst pain they have experienced.

A researcher (C. R.) recorded the time taken to complete data entry for the two scenarios. After completing data entry on the first app, participants completed the SUS measure.

After a short break, participants repeated the data entry of the two scenarios using the second app, which was again timed, and they rated it on the SUS. Participants then completed the design and app use questionnaires in relation to both apps.

Data Analysis

For each app we calculated mean (and SD) SUS and questionnaire scores, and for each scenario we calculated mean (and SD) timings. Apps and scenarios were compared using the Student's t-test and analysis of variance (anova). We calculated correlation statistics to measure the association of SUS scores and timings, and confidence scores and timings.


The Survey

In total, 41 apps were identified. Twenty-nine were excluded because they were specific to a type or site of pain (such as migraine). Of the 12 included apps, six were available exclusively to the Apple market, and five exclusively to Android. One app, Pain Care, was available to both markets and was evaluated using the Android platform. Table outlines the included apps, indicating the operating systems on which they are used.

View this table:
Table 3

Apps included for quality assessment

App NameAppleAndroid
Manage My Pain
Pain Log
Pain Scale
Pain Tracker Pro
Pain Tracker (Pain Log) 2
Pain Care
Chronic Pain Tracker (iPain)
My Pain Diary
Pain Diary
Pain Monitor Lite
Pain Tracker
Track and Share

Product Description

Two types of app were identified (pain diaries [N = 9] and pain scales [N = 3]). Pain diary apps allow users to input details of pain episodes and record them in a manner similar to written pain diaries. Pain scale apps are limited to allowing users to input the severity of pain, usually on a visual analog scale (VAS). Where a pain diary app also contained a VAS, its classification remained as a pain diary.


Three apps were free to download. The cost to download the full version of the remaining nine apps varied from £0.63 to £5.49 (median £1.49). The median price of the Apple apps was £1.87 while the Android apps had a median cost of £2.24. Six of the nine pay-to-use apps also had “lite” versions that were available free to download.

Updates and Maintenance

Six of the 12 apps had been updated within the previous 6 months. Two apps, one for Apple and one for Android, had not been updated since 2010, and no information was available to indicate when the app had first been produced.

Downloads and Ratings

No information on downloads was available for Apple apps. Android provide information in download ranges (e.g., 500+, 1,000+, and 10,000+). The number of downloads for pain apps on Android ranged from 10+ to 1,000+ for full versions. Download information for the “lite” versions ranged up to 10,000+. Customer ratings were available on both platforms (using a scale from 0.0 to 5.0), but ratings were not always available. For those apps with rating information, they had a mean rating of 2.6 for apps on Android and 3.6 for apps on Apple.

Data Input

A variety of ways to input data was evident, ranging from selecting words from stored lists to indicating pain location on an interactive picture. All but one of the apps contained a list, allowing users to select an answer from a number of stored options. Nine apps featured a free text option, allowing users to create options using their own words. Nine apps featured a VAS to record pain severity. The scales were most often presented in a numbered format and accompanied by representations of faces (see Figure for an example) and/or corresponding text.

Figure 2

Screenshot taken from Pain Monitor Lite (available for Apple iOS). Example of a visual analog pain severity scale with accompanying face-rating scale.

Three apps included an interactive body map (see example in Figure ), allowing users to select an anatomical area, which would then select and log the relevant location name.

Figure 3

Example of an interactive body map. This image is a screenshot of the interactive map with body painter feature taken from Chronic Pain Tracker (available for Apple iOS).

Results Display on the Apps

Five apps included a list of the record summaries provided by the user, usually with the ability to click on it and view it in more detail. An interactive calendar was included in four apps; this allows users to view summaries of results, or to click on a date and view the information available for that day. Eleven of the 12 apps featured a graphical method of showing details of the pain episodes, including bar and pie charts, and line graphs of pain severity over time.

All the apps could be used to send reports or results to any email address. The connection of Bluetooth devices for sending reports, or for transferring data for storage, was possible with four of the apps.

Additional Features

One app (Pain Care) included a page that automatically linked to recent journal articles on pain research; it also allowed users to record details of their health care team and contained a “nearby” feature to identify the closest location for emergency care. A reminder function was available in two apps, enabling users to set an alarm to remind them to make a record. Three apps allowed users to input supplementary information, such as mood, activity, sleep patterns, blood pressure, weight, height, and body mass index.

App Development Team

Half of the apps were developed in a team, with the remainder developed by individuals. Background information for developers could be obtained only for five of the 12 apps. These included software or computer engineers and developers, creative designers, health marketing experts, doctors (or other clinicians), and public health specialists. Email correspondence with developers revealed that three of the five apps were developed by individuals whose backgrounds were as software or computer engineers, but who had personal or family experience of chronic pain.

All but one of the 12 apps was part of a range provided by developers, and eight of these developers had produced at least one other health care app. Among these, five had produced apps unrelated to health care, such as games, business apps, and teaching aids. No information was provided relating to user involvement in the development or product testing of the apps.

Quality Assessment

Clinical Content

The lowest score achieved in this section was 22.2% for Pain Tracker Pro. This app was marketed as a pain diary but only allowed users to track symptoms by choosing an option from a modifiable list. The list included symptoms not relevant to some types of pain, such as “bright lights,” “damp weather,” “exertion,” and “smoking”. The app only scored points for the presence of date and time input, exacerbating factors and the ability to modify options. The highest score was obtained by Pain Diary, which met 95.4% of the quality assessment criteria (see Table ).

View this table:
Table 4

Ranking of pain apps in order of score obtained for each section of the quality assessment and overall


Mean (and SD) clinical content quality assessment scores were calculated by platform, app type, and overall. The mean score for Apple apps was 75.0 (SD = 19.9) and for Android 51.2 (SD = 26.1). Pain diary apps had a mean rating of 66.5 (SD = 30.6), and the pain scale apps were 55.6 (SD = 29.4). The overall mean clinical content score achieved by the apps was 62.9 (SD = 26.0).

Interface Design

The lowest scoring app (Pain Log) scored 58.8, mainly due to inconsistent use of color and fonts. Different color schemes were used on each page, and some of the font colors made text difficult to read. Four apps scored 100 (Pain Care, Pain Diary, Track and Share, and Pain Tracker Pro), satisfying all design criteria. The mean scores by platform were 94.1 (SD = 6.8) for Apple and 83.3 (SD = 18.0) for Android. Scoring by app type was the pain diaries at 91.2 (SD = 6.4) and pain scale apps at 82.3 (SD = 17.6). The mean design score for all apps was 88.2 (SD = 14.0).

Ease of Use Assessment

A number of concerns were identified with accessibility. Only six apps provided user guides, and no apps supported modifications to text size. The lowest app score was 51.6 (Pain Log) and the highest 90.9 (My Pain Diary). The mean scores for platform were 81.4 (SD = 7.0) for Apple and 68.4 (SD = 13.8) for Android. The mean for pain diary apps was 75.9 (SD = 12.8) and for pain scale apps 73.5 (SD = 11.8). The mean usability score achieved by all apps was 74.9 (SD = 12.6).

Overall Quality Assessment Scores

Overall quality assessment scores for the apps ranged from 50.8 (Pain Log) to 91.4 (Pain Diary). Apple apps scored the higher of the two operating systems, with a mean total of 83.0 (SD = 8.7) compared with the score of the Android apps (68.6 [SD = 15.2]). Pain diary apps scored consistently higher than pain scale apps (mean 77.3 [SD = 13.4] to 73.3 [SD = 15.7]). The mean overall quality assessment score across all apps was 75.7 (SD = 14.2).

Usability Testing Phase

Participants' smartphone confidence ratings (which could be scored 1–5) ranged from 3 to 5 (mean = 4.2; SD = 0.8).

Usability task timings: time taken to complete a pain report ranged from 1′22″ to 6′09″ for Manage My Pain (mean = 3′01″; SD = 1′14″) for the first entry and from 0′44″ to 4′05″ (mean = 2′00″; SD = 0′49″) for the second entry. The second Manage My Pain entry of participants was on average 34.4% quicker (95% confidence interval 26.7% to 42.1%).

For the Pain Scale app, times ranged from 0′02″ to 0′59″ (mean = 0′19″; SD = 0′12″) for the first entry and from 0′02″ to 0′37″ (mean = 0′14″; SD = 0′08″) for the second entry. Participants' second entry for Pain Scale was 23.6% quicker (95% confidence interval 9.6% to 37.5%).

When participants' times for two entries for each app were combined, the entry for Pain Scale was considerably quicker (mean 89.0% faster; 95% confidence interval 77.5–99.5%).

The systems differed greatly in the requirements made of participants. Manage My Pain, a pain diary, provided a number of screens for users to document aspects of pain. The Pain Scale app had more limited functions, requiring plotting of a pain rating on a scale presented in the app.

A 3 × 4 repeated measures anova was conducted to compare the effects of confidence ratings (of smartphone use) on the time taken to use the apps. Mauchley's test was statistically significant (P < 0.001), indicating violation of sphericity, and so a Greenhouse–Geisser correction was made. Results show a significant main effect of time: F(1.50, 56.88) = 219.64 (P < 0.001), showing that participants were statistically significantly quicker to make a pain report on the second app they tested. However there was no effect of self-rated smartphone confidence on time taken, nor a significant interaction between time and confidence: F(2.99, 56.88) = 2.66, P = 0.57.

SUS Scores

The SUS scores were calculated for both apps. Scores for Manage My Pain ranged from 57.5 to 100, with Pain Scale scores ranging from 60 to 100.

A paired samples t-test was conducted to compare the mean SUS scores of the two apps. There was no statistically significant difference in SUS scores between Manage My Pain (M = 78.7, SD = 10.4) and Pain Scale (M = 81.6, SD = 10.3); t(11) = −1.821, P = 0.076.

Paired samples t-tests were conducted to compare the mean scores for participants' responses to questions relating to color, font and app layout. There was a significant difference in the color ratings between Manage My Pain (M = 4.0, SD = 0.8) and Pain Scale (M = 3.5, SD = 1.1); t(40) = 2.930, P = 0.006. See Figure for examples of screen design for Manage My Pain and Pain Scale. Manage My Pain uses a lighter set of colors, incorporating light blue and white.

There was a statistically significant difference in font ratings between Manage My Pain (M = 3.8, SD = 0.9) and Pain Scale (M = 3.4, SD = 0.8); t(40) = 2.512, P = 0.016. The fonts used in Manage My Pain (which were ornate and nonstandard) were rated as more attractive.

There was no statistical difference in layout ratings for Manage My Pain (M = 3.9, SD = 0.8) and Pain Scale (M = 3.7, SD = 1.0); t(40) = 1.506, P = 0.140.

There was a significant difference in the scores for usefulness of results sections in the apps between Manage My Pain (M = 4.0, SD = 0.9) and Pain Scale (M = 3.6, SD = 1.1); t(40) = 2.206, P = 0.033.

As part of the results section, inquiring about the usefulness of options to mail results from the apps, a significant difference was found between Manage My Pain (M = 3.9, SD = 0.9) and Pain Scale (M = 2.9, SD = 1.3); t(40) = 4.109, P < 0.001.

Evaluation Questionnaire

Only six participants were aware of smartphone apps for reporting pain prior to the study. Almost all participants (38/41) said that they would consider using a pain app in future. Of the three who said they would not, all had been unaware of them prior to the study.

Thirty-four (of 41) participants said that they would consider downloading Manage My Pain, and 40 said that they would recommend it to a friend. For Pain Scale, 22 participants said that they would consider downloading the app, and 26 would recommend it to a friend.

When outlining the cost of the apps, after revealing the cost of each (£2.99 for Manage My Pain and £0.63 for Pain Scale), 28 participants felt that Manage My Pain was the right price for its content, 11 thought it was more expensive than expected, and the remainder thought that it was cheaper than expected. Thirty-two participants thought that Pain Scale was the right price, three that it was cheaper than expected, and six that it was more expensive.

When indicating overall preference between the two apps, 85% (35/41) of participants preferred Manage My Pain, four preferred Pain Scale, and two had no preference.


The survey and quality assessment of apps revealed wide variation in the clinical content, interface design, and usability of apps to support self-management of different types of pain, available on either Apple or Android platforms. All apps were either in a pain diary format or pain scale, and pain diary apps had higher overall quality assessment scores. Most of the apps were not free to download, although cost did not determine quality. Very little involvement of health professionals or people with chronic pain was mentioned in the development of apps. From the usability testing of two apps with prospective users, it was clear that meaningful data were obtained that could be used by developers to refine the content and presentation before release.

Asking participants to evaluate more than one app, using a counter-balanced pair-wise design, was useful in establishing preferences both for design and usefulness. It was notable that participants' pain reports were quicker for the second app they tested, suggesting substantial practice effects, as is the case with many devices. However, given that nine of the 12 apps offered either free downloads or free “lite” versions, it is important that the first use of the app is intuitive; otherwise, users may not attempt use again, given no monetary investment. There was a notable large variation among participants in the time taken to input data into the apps; this aspect was not the focus of this study, but may be worthy of further research to examine the consequences of individual preferences and cognitions.

The quality assessment criteria were compiled from several existing measures of interface design. Although designed to be relevant to the selected problem, neither the reliability nor the external validity of the criteria have been verified. In addition, only one researcher applied the criteria to review the selected apps, and this limitation may impact on the reliability of the findings. We based our clinical content criteria on SOCRATES because this is commonly taught and used in clinical practice. Although some apps were designed to rate pain severity only, a number of apps covered aspects such as site, onset, periodicity, and exacerbating factors. Our approach therefore allowed a standardized judgment to be made of each app against the most important elements of clinical pain assessment.

Established methods were adopted for usability testing with participants. Participants were mostly of high education status, which limits the external validity of the study. It would be useful to include a more diverse and representative sample in future research, and also a more homogenous clinical sample. Similarly, the usability study was limited to two apps (a pain diary and a pain scale), and the use of larger, more varied samples in future research would be beneficial. The stationary user interface evaluation approach to usability testing was helpful to obtain the general preferences of participants. Supplementing this method with field trials could be a next step to better understanding how people use pain apps as part of normal routine [18].

This research demonstrates that assessment criteria can be developed and used to identify differences in the quality of pain apps. Currently, apps can be developed and marketed without regulation or guidance, which is a particular concern for apps used in health settings, such as pain, where users may be highly motivated to find an intervention providing a benefit [4]. Developing a better understanding of quality criteria is important to raise public awareness of the quality of health-promotion apps; such an approach is predicted to stimulate rather than stifle innovation in the apps market [19]. Content that has validity and that is evidence based is an important aspect of the quality and value of any health care app, but so, too, is the initial and continued engagement of users. As has been highlighted in medical device development and health care technology more generally, involving users in design leads to products with better content and interface, and also increases their functionality, usability, and quality [20]. Currently, the market in health care apps shows a marked inverse relationship between download rate and their adherence to evidence-based guidelines [19]; that is, those that are downloaded most tend to be those less likely to follow guidelines. Understanding the cause of such consumer behavior will help the development of apps that appeal to users but also contain evidence-based content.

User involvement must also integrate health professionals as well as those suffering with pain. There was an apparent lack of health professional (or health service) engagement in the development of the pain apps; this is concerning particularly in relation to an included feature that allows users to email updates to their general medical practitioner (GP). GPs' willingness to use new consultation media is conditional on the media being clinically appropriate and there being available medico-legal and technical support [21]. The apps identified in this research seem somewhat detached from this aspect and at odds with growing evidence for the need for the integration of patient and professional information sources across care pathways [22]. This is a matter for concern and enhanced further by the variety and number of apps available and the fast-moving, even transient nature of app provision. However, the level of standardization and integration that might be expected of apps will depend on the health care setting, for example see Huckvale et al. [13] in relation to asthma self-management. Integration may be less important in market-driven health systems with a diversity of providers where consumer preference could predominate than in state or insurance-based systems where institutional factors such as standardized record keeping is considered more important.

While there is currently a push for the use of technologies, such as mobile phones, in health care [2] and their clinical and cost-effectiveness are starting to be subjected to evaluation [23], app developers need to be mindful of how best to support existing health care practice. Handheld devices and apps offer great potential for improving the communication between health professionals and patients, and also for impacting on self-management of long-term conditions (including when provider feedback is not a feature), but their potential may not be realized unless their design and development takes account of usability and users' preferences.


The study found 12 smartphone apps intended for self-management of pain, which were of varying quality. Most evident was an apparent lack of user, clinician, or health service engagement in their development. Involving users earlier in the development of health apps is advised, as well as establishing ways to merge user requirements with evidence-based content, to provide high-quality and usable apps for the self-management of pain.


We wish to thank three anonymous reviewers for their helpful comments on a previous version of this article.


  • Disclosure: The authors have no conflicts of interest or disclosures.


View Abstract