OUP user menu

KnowPain-50: A Tool for Assessing Physician Pain Management Education

John M. Harris Jr MD, MBA, John V. Fulginiti PhD, Paul R. Gordon MD, MP, Thomas E. Elliott MD, Bennet E. Davis MD, Charles Chabal MD, Randa M. Kutob MD, MPH
DOI: http://dx.doi.org/10.1111/j.1526-4637.2007.00398.x 542-554 First published online: 1 July 2008


Background. Despite a need for better physician pain management education, there are no widely accepted assessment or outcome measures to support this work.

Objective. Create a self-assessment tool to measure physician educational needs and the effectiveness of chronic pain educational programs.

Design. We used expert consensus to draft a 142-item survey that covered essential areas of chronic pain management. We tested the survey in 106 physicians, including 22 pain management experts and used predefined psychometric criteria to eliminate 70 items. We then eliminated 22 remaining items that did not correlate with the management of a standardized chronic pain patient by 27 academic physicians. We evaluated internal consistency using Cronbach's alpha.

Results. The final 50-item survey assessed physician knowledge, attitudes, and beliefs in: 1) initial pain assessment; 2) defining goals and expectations; 3) development of a treatment plan; 4) implementation of a treatment plan; 5) reassessment and management of longitudinal care; and 6) management of environmental issues. The survey demonstrated good internal consistency in all physician populations studied (α = 0.77−0.85). Average scores in 84 “pilot” physician users of a CME Website (135.8–138.5) were significantly lower (P< 0.01) than scores in 27 academic physicians (150.0), or 22 pain experts (177.5).

Conclusions. This survey, the KnowPain-50, has good psychometric properties, correlates with clinical behaviors, and appears to distinguish between physicians with different levels of pain management expertise. It may be a useful measure of the effectiveness of physician pain management education programs.

  • Education
  • Continuing
  • Outcome assessment
  • Pain Training Programs


Since Marks and Sachar documented the undertreatment of medical inpatients in 1973 [1], surveys have continued to show that practicing physicians feel they lack knowledge and comfort in using opioids and in many other aspects of treating chronic pain, particularly chronic nonmalignant (noncancer) pain [2–5]. Other studies have documented various and sometimes conflicting assessments of physicians' knowledge and skill in pain treatment [6–8]. An unfortunate but common thread in this work is that the surveys, tests, and questionnaires used by these researchers to assess physician knowledge, attitudes, and beliefs (KAB) in pain management have been locally developed, using whatever processes appealed to the investigators. They have not been standardized nor have they been used by others. We could find only one instance where an evaluation tool was itself evaluated. In this case, the questionnaire used by Weissman to evaluate the effectiveness of the “Role Model” program for teaching pain management skills to health professionals was found by Janjan to have acceptable levels of internal consistency and reliability over time, but the researchers acknowledged that validity concerns were not addressed [9]. Without standardized tools for measuring physician pain management KAB, it is virtually impossible to compare findings from one setting to another or to measure and compare the effectiveness of different pain management education programs.

We have previously used standard educational and psychological testing techniques to develop a self-administered survey to assess physician readiness to manage another difficult clinical problem, intimate partner violence (IPV). This survey, which is in the public domain, has good psychometric properties, appears to measure KAB and self-reported behaviors that are important to the management of IPV, correlates with independently measured office practices, and is responsive to educational interventions. It has been used to measure the effectiveness of an IPV educational program [10,11]. We sought to use a similar strategy to devise a practical, self-administered physician survey that could assess physician KAB important in managing chronic nonmalignant pain.

We also wished to extend previous work by incorporating two novel approaches to the development of self-administered physician surveys that may enhance their usefulness. These included the use of clinical data obtained via unannounced standardized patients (SPs) to support instrument refinement and a Likert-type scoring methodology for dichotomous true/false items to provide an increase in scoring discrimination. Our goal was to provide a practical and standardized evaluation tool that would allow persons with an interest in improving pain management education to calibrate learner needs and quantify the outcomes of their educational programs.

We acknowledge that the technologies required for educational needs assessment for practicing physicians are, at best, underdeveloped [12]. This report describes a set of processes for refining and testing a pain management KAB survey. Although further information on additional experiences with the survey is described in a companion article [13], the real benefit of this work will be the extent to which those with an interest in pain management education can use and improve upon the survey and the methods with which it was developed.


Survey Item Development

We began by convening an outside panel of seven experts in pain treatment, education, and policy (Elliott, Davis, Chabal, Kutob, and those noted in the Acknowledgements). We used consensus-building procedures to prepare a master list of proposed survey items that would measure physician KAB important to the successful management of chronic nonmalignant pain. Our expert group based their work on the currently accepted construct that chronic pain is a bio-psychosocial disorder, which requires a multimodal approach to assessment and management [14,15]. They determined that a useful tool should measure the types of attitudes and beliefs that Weissman described as being important to the management of cancer pain [16].

Item Refinement

After the survey items were developed, we used a predefined psychometrically based technical plan that specified acceptable item difficulty levels and discrimination indices for determining which survey items would be retained, based on physician response data. Once this process began, no survey items were modified or added to the list.

The initial psychometric standards we used to remove survey items included: 1) an item difficulty index of >0.60; 2) a negative point-biserial correlation; and 3) an item response variance of 0. The difficulty index measures the proportion of persons answering the question correctly. During this phase of the work, Likert-type items were scored “correct” only if the extreme value was selected, thus increasing the item difficulty. The point-biserial correlation measures the relationship between the response to an individual item and the overall score on the survey. A negative point-biserial correlation means, for example, that an item answered incorrectly was positively correlated with a high or “good” survey score or vice versa. An item response variance of 0 means that all respondents answered the item in the same way (i.e., all correct or all incorrect). As we tested the draft survey in a group of pain experts, we further modified these criteria to also remove items with very high (>0.90) or very low (<0.10) difficulty indices in this population and items that were scored lower by experts than by average physicians. We used Cronbach's alpha to measure the internal consistency of survey items. Cronbach's alpha ranges from 0 to 1.0, with values greater than 0.7 generally indicating that survey or test items move consistently and appear to be measuring the same hypothetical construct or latent variable [17].

After the item list had been reduced as much as possible based on psychometric criteria, we performed an additional clinical evaluation to further refine the survey and enhance its validity and usefulness. This stage involved the correlation of individual survey items with overall pain management scores obtained via visits from unannounced SPs. Survey items that did not correlate with SP-based pain management scores during this evaluation were removed from the draft survey to create the final survey instrument. Although SPs have been used to calibrate computer-based clinical vignettes [18], to our knowledge, SPs have not previously been used to refine or validate written KAB surveys.

The SP refinement protocol we developed for this study began with standard procedures used by the University of Arizona to develop SP-based evaluations. Content experts (Chabal, Davis) prepared live scenarios for the initial evaluation and management of two common pain management situations, chronic headache and chronic low back pain. An author with experience in the use of SPs for student evaluation (Gordon) videotaped the live scenarios and developed checklists of appropriate physician behaviors, which were reviewed by the content experts [19,20]. The SP checklists included history-taking, physical examination, assessment, treatment, and patient education actions that a knowledgeable primary care physician should perform when evaluating a new patient with one or the other of these chronic pain syndromes. The checklists were not the same for the two conditions, but emphasized major elements, such as “asks about medications” and “evaluates function” in the specific clinical context. The checklists provided several ways of demonstrating appropriate actions. Higher checklist scores indicated better pain management practices, as outlined by the content experts. Six experienced SPs were trained in one of the two scenarios and scheduled to see study physicians as unannounced patients.

Study physicians first completed a draft version of the survey and were then scheduled to see one unannounced SP in their clinic within 2 months. The SP visit was not recorded by audio or video technology or otherwise distinguishable from a regular new patient visit. SP clinic assignments were managed by a study coordinator from the Department of Family Medicine. The type of SP patient seen by the physician was not random; rather the visit was determined by physician and SP scheduling constraints. After the encounter, the SP completed the appropriate checklist and returned it and any prescriptions or lab test slips to the study coordinator. During this final stage in the item refinement process, we eliminated remaining survey items that were not positively correlated with overall SP checklist scores (r ≤ 0).

Study Populations

This project was determined to be exempt from the Federal Policy for the Protection of Human Subjects by Argus IRB. We performed the initial psychometric studies in 2003–2004 in two groups of 53 and 31 physician subscribers to a commercial CME Website (Pilot 1 and Pilot 2 groups). All physician subscribers to the CME Website (1,200 total physician subscribers in November, 2003) were offered a payment of $90 of “store credit” at the Website, which could only be used toward the purchase of CME programs, for completing the survey online. To be eligible, physicians had to be in the active practice of adult-care medicine. No CME credit was offered for completing the survey. We have used this approach in other similar work [10] and have previously found the Website subscriber base to be representative of U.S. physicians in general [21].

We next studied the survey's performance in a group of 22 pain experts in 2004. These individuals were referred by our pain management co-authors (Chabal, Davis, Elliott) and were in the active practice of pain medicine or were participating in pain management fellowship programs. These physicians were paid $100 for completing an online version of the survey.

Finally, we compared results on the survey with findings from an unannounced SP with chronic pain in a group of 27 general internal medicine and family medicine physicians from the University of Arizona College of Medicine in 2005. These academic physicians were either faculty (23) or senior residents (4). They were paid $50 to complete a paper-based draft version of the pain competence survey. Their offices were paid $100 to partially compensate for the cost of managing the unannounced chronic pain SP.

Analytic Techniques

The analytic techniques we used to develop and refine the survey are described above. We compared mean survey scores between different physician groups using oneway analysis of variance.


The expert-led consensus process resulted in a draft survey containing 142 items covering six key pain management activities (number of questions, percent of total): 1) initial pain assessment (36, 25%); 2) defining treatment goals and expectations (18, 13%); 3) development of a treatment plan (44, 31%); 4) implementation of a treatment plan (13, 9%); 5) reassessment and management of longitudinal care (20, 14%); and 6) management of environmental issues (11, 8%).

Psychometric Refinement

The major characteristics of the survey during the subsequent four-stage refinement process are outlined in Figure 1. During the initial (Pilot 1) stage, the test group of 53 physician CME Website subscribers had an average age of 48.7 years and 66% were men. Most (70%) practiced family or internal medicine. Based on data from this group, 17 items did not meet our predetermined criteria and were removed. At the next (Pilot 2) stage, we tested the 125-item survey in a different group of 31 physician subscribers to the CME Website. The average age of this group was 49.5 years and 77% were men. Seventy-four percent practiced family or internal medicine. An additional 13 survey items did not meet our test specifications, either because of failing to meet difficulty acceptance thresholds or lack of discrimination between better and poorer performing examinees (i.e., negative point-biserial correlations).

Figure 1

Flow of Survey Items through Item Refinement Process. Figure shows flow and number of remaining items following each stage of the survey refinement process. First three stages (Pilot 1 → Pain Experts) based on psychometric criteria as outlined in text. Final refinement stage based on correlation between individual survey responses and overall scores on unannounced standardized pain patient checklists. See text for details. *Cronbach's alpha based on number of items and particular study group.

During the third stage, we examined the revised 112-item survey using data from a group of 22 physicians with expertise in pain management. At this point, we used a hierarchical item removal process that included data from the Pilot 1 and Pilot 2 groups. We first removed survey items that demonstrated negative point-biserial correlations in the 22-physician expert group. We then removed items showing zero variance from all cases (i.e., responses from all examinees were identical.) These types of items offered no information that differentiated between levels of expertise. Next, review of the difficulty index and point-biserial correlations in the data from the pilot and expert samples allowed us to remove other items (i.e., those displaying a difficulty index greater than 0.60 and those with negative point-biserial correlations, the criteria used during early pilot studies.)

Comparing item performance between expert and pilot responses, we also found items that were more difficult for experts than typical physicians, indicating item response foils (or point descriptors) that were drawing experts away from correct responses. As expertise level should be associated with successful responses to survey items, we also removed items with this unanticipated result. Finally, we sought any remaining items with extreme difficulty indices among experts. Such items provided less variability and contributed less information to the survey results. Subsequent to all previous removal strategies, we located and deleted items that experts almost always answered correctly (item difficulty > 0.90) but which few of the general sample did. We also located and removed items that almost none of the experts answered correctly (item difficulty < 0.10), indicating extreme item difficulty for any respondent. Following this phase of the item analysis, we had a survey instrument of 72 items, which showed strong internal consistency (alpha = 0.94) in the combined data set of 106 physicians.

Clinical Refinement

During the final stage of the refinement process, we administered the revised 72-item draft survey to a group of academic physicians and correlated individual item results with SP checklist scores, as described above. Twelve physicians evaluated an unannounced headache SP and 15 physicians evaluated a back pain SP. The initial results showed a moderate correlation between the 72-item survey and overall SP scores (r = 0.33, P =0.09). There were 22 survey items whose correlation with overall SP checklist score was ≤0 and these items were removed from the survey, leaving a 50-item final survey with a stronger overall SP score correlation (r = 0.40, P = 0.04).

Final Survey Features

The distribution of items in the final 50-item survey was: 1) initial pain assessment (13, 26%); 2) defining treatment goals and expectations (10, 20%); 3) development of a treatment plan (16, 32%); 4) implementation of a treatment plan (3, 6%); 5) reassessment and management of longitudinal care (1, 2%); and 6) management of environmental issues (7, 14%). We have named this survey the KnowPain-50.

Our goal was to develop a survey tool that would be sensitive to changes in expertise and competence; therefore, we used a final scoring scale that provided 5 points for a correct answer to one of the five categorical questions (e.g., “By far the most common adverse side effect of opioid therapy is:”) and a range of 0-5 points for one of the 45 six-category Likert-type questions. For example, the statement “Elderly patients cannot tolerate medications such as opioids for pain” has a correct answer of “Strongly Disagree.” If a user selected this answer, they received five points. If they selected “Disagree” they received four points. If they selected “Somewhat Agree” they received two points and so on to zero points for the sixth and most incorrect response. See the Appendix for a copy of the KnowPain-50. There was no further weighting of scores, thus this scoring protocol yielded a possible scoring range of 0–250 for the KnowPain-50.

We reanalyzed data from all study populations using the final KnowPain-50 items and the final scoring protocol. These data are presented in Table 1. As can be seen, there was a statistically significant difference in average scores between physician populations, which moved in the expected direction, that is scores increased from physicians in general, to academic physicians and physicians receiving education, to pain experts. The difference in average scores between the two pilot groups and the academic physicians was 13 points (9%) and the difference between the academic physicians and the pain experts was 27.5 points (18%). The internal consistency (alpha) of the KnowPain-50 was high in all populations studied (0.77–0.85). This finding also indicates that the six management activities we used to develop the survey items were highly intercorrelated in the final version.

View this table:
Table 1

KnowPain-50 Performance Characteristics.

Study GroupStatistic
Mean Survey ScoreMedian Survey ScoreStandard Error of MeanStandard DeviationMinimum ScoreMaximum ScoreItem Consistency (alpha)
Pilot 1 (Internet MDs; N = 53)135.8*1342.719.4 961840.81
Pilot 2 (Internet MDs; N = 31)138.5*1413.821.1 991720.85
Pain experts (N = 22)177.5*1764.018.81332060.84
Academic physicians (N = 27)150.0*1492.915.11181820.77
  • * Differences among groups significant (F = 27.526, P < 0.01), oneway anova, with pairwise post hoc analyses revealing the following pattern among groups: Pain Experts > Academic MDs > Internet MDs 1 = Internet MDs 2.

  • Cronbach's alpha based on final 50-item survey.


To our knowledge, the KnowPain-50 is the most thoroughly studied physician pain management KAB survey tool that is available for general use. It is internally consistent, it correlates with clinical data, and it clearly distinguishes pain medicine experts from other physicians. In a companion article, we demonstrate that the KnowPain-50 is responsive to educational interventions [13]. Despite these findings, we are reluctant to describe the KnowPain-50 as a “valid” measure of physician pain management knowledge, expertise, competence, or skill. We accept that “validity is a unitary concept” reflecting “… the degree to which all of the accumulated evidence supports the intended interpretation of test scores for the intended purposes”[22]. Accordingly, the KnowPain-50 cannot be born with validity; it must acquire it over time, through the experience of others.

As noted, the development process we used incorporated features that would benefit from further study. One of these is the KnowPain-50 scoring system, whereby potentially dichotomous statements such as, “A placebo can be used to determine if pain is real” are scored on a 1-6 “strongly agree” to “strongly disagree” Likert-type scale rather than a “true/false” scale. On a practical basis, this system opens up the scoring range, thus potentially providing more discrimination. On a theoretical basis, it may also be more sensitive to educational interventions. A true/false question cannot distinguish between the person who is quite sure of the answer and one who is much less sure or guessing outright. A desirable effect of an educational intervention may be to increase the student's confidence in his or her knowledge or belief. Such a change can be detected by a Likert-type measure, but not by a dichotomous true/false measure. Finally, this approach is similar to confidence-based scoring (assessment), which considers the correctness of the answer and the respondent's confidence in the answer [23]. Confidence-based scoring has not been widely used in medical education, but one study has shown that it can better predict performance than conventional scoring [24].

Another feature of our development process was the use of data obtained from unannounced SPs to refine the KnowPain-50. SPs and structured clinical examinations have been used in educational settings to enhance learning and evaluate student performance [25] and in clinical settings to calibrate computer-based clinical vignettes [18], but, to our knowledge, they have not been previously used to refine physician KAB surveys. We believe that KAB surveys that are developed using SP data will better correlate with important clinical practices than will surveys that have not had such clinical refinement, but this remains to be shown empirically.

There are practical limitations of the KnowPain-50 that are inherent to all such tools. At best, this survey reflects the current opinions of a single group of experts on the optimal approach to pain management. Others will have to review the survey and make their own decisions about the relevance of this group's judgements on the KAB that physicians caring for chronic pain patients should exhibit. Additionally, while the expert contributors sought to develop items that would be durable, certain of the KnowPain-50 items, for example, a question about the user's knowledge of selective COX-2 inhibitors, may be subject to changes in medical practice. By making the survey an open-source document, we recognize these limitations and encourage others to refine it and reevaluate it in light of their own experience.

A potential limitation of the survey is that the KnowPain-50 relies on self-reported data. However, patient surveys depend on self-reported data and are commonly used in health services research and in clinical care; thus there is no a priori reason to dismiss a survey simply because it depends on self-reported data. As the KnowPain-50 is intended to be a formative assessment tool, rather than a high-stakes test, we believe that such self-reported data can provide useful and reliable information on educational needs and program outcomes.

Another potential limitation is that, although correlated with clinical data, the KnowPain-50 does not directly measure clinical endpoints, only physician KAB. These data are considered relatively low level educational outcomes in many evaluation schema, such as those developed by Kirkpatrick (level 2 of 4) [26] and Moore (level 3 of 6) [27]. However, Kirkpatrick states that “Learning can be defined as the extent to which participants change attitudes, improve knowledge, and/or increase skill as a result of attending the program”[28]. Moore notes that “competence” or possessing “the knowledge, skills, and attitudes to perform as expected” is the real result of learning and that “… the ultimate goal for a CME activity is to improve the competence of physicians to manage patients”[27]. In this context, standardized KAB surveys, such as the KnowPain-50, may have a role to play because they are specifically designed to reflect KAB transmitted via education. Thus, well-designed KAB surveys may be more sensitive indicators of educational outcomes than supposedly “higher” measures, such as patient well-being, which may be affected by numerous factors beyond physician education.


We thank three unnamed reviewers of Pain Medicine for their constructive comments on the manuscript. We also thank additional members of the expert consultant team who helped develop early versions of the survey and define the essential skills necessary for assessing competence in managing chronic nonmalignant pain:

Marc Hoffing, MD—COO Desert Medical Group, Palm Springs, CA;

Stuart Levine, MD—Chief Medical Officer, Prospect Medical Holdings, Culver City, CA;

Aaron Gilson, PhD—Pain & Policy Study Group, University of WI, Madison, WI.

The development of the KnowPain-50 and the research study were supported by Small Business Innovation and Research (SBIR) grants, R43-NS045361 and R44-NS045361 from the National Institute of Neurological Disorders and Stroke. The sponsor had no role in the design and conduct of the study, the collection, management, analysis, and interpretation of the data, or in the preparation, review, or approval of the manuscript. The Principle Investigator (JMH) had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.


Appendix KnowPain-50

1. A 33-year-old woman complains of pain “all over” with pain intensity ratings ranging from 4 to 8 on the 0-10 scale, fatigue, forgetfulness, poor sleep, headaches, and dizziness. This symptom complex is most consistent with which of the following?

□ Fibromyalgia syndrome*

□ Chronic fatigue syndrome

□ Chronic myofascial pain syndrome

□ Depression

□ Pain disorder with psychological factors

□ Don't know

2. Which of the (one) following statements is true regarding selective COX-2 inhibitors?

□ They cost twice as much as non-selective NSAIDs

□ Gastroduodenal injury risk is similar to non-selective NSAIDs

□ There is no increased risk for acute myocardial infarction or congestive heart failure

□ They are no more effective as an analgesic than non-selective NSAIDs*

□ Low dose daily aspirin for cardiovascular prophylaxis is not needed when selective COX-2 inhibitors are used

□ Don't know

3. Anticonvulsants and analgesic antidepressants obtain about a 50% response rate (pain intensity reduction in half of patients treated) in neuropathic pain. Which of the following drug classes obtains similar results?

□ Benzodiazepines


□ COX-2 inhibitors

□ Opioids*

□ Phenothiazines

□ Don't know

4. Which of the following therapies for fibromyalgia syndrome has been shown to yield the most consistent improvement?

□ Massage

□ Trigger point injections

□ Acupuncture

□ Aerobic exercise*

□ Transcutaneous electrical nerve stimulation (TENS)

□ Don't know

5. By far the most common adverse side effect of opioid therapy is:

□ Constipation*

□ Nausea and vomiting

□ Sedation and cognitive dysfunction

□ Respiratory depression

□ Don't know

View this table:


View Abstract