- Open Access
Significant discrepancies exist between clinician assessment and patient self-assessment of functional capacity by validated scoring tools during preoperative evaluation
Perioperative Medicinevolume 5, Article number: 18 (2016)
Preoperative assessment of functional capacity is necessary to direct decisions regarding cardiac evaluation and may help identify patients at high risk for perioperative complications. Patient self-triage regarding functional capacity could be useful for discerning which patients benefit from a clinician evaluation at a Preoperative Evaluation Center prior to the day of surgery. We evaluated the feasibility of preoperative, patient self-triage regarding functional capacity.
Patients were recruited immediately prior to their preoperative evaluation. Study participants completed electronic versions of the Duke Activity Status Index (DASI) and the Patient-Reported Outcomes Measurement System (PROMIS)–Short Form 12a–Physical Function. DASI and PROMIS questionnaire responses were scored and evaluated for correlation with clinician assessments of functional capacity. Correlation was analyzed around the dichotomous outcome of <4 metabolic equivalents of task (METs) or ≥4 METs. Patients also evaluated the usability of the questionnaires.
After IRB approval, 204 patients were enrolled and completed both DASI and PROMIS questionnaires. Clinicians assessed functional capacity at <4 METs for 109 patients (53.4 %) compared to 18 (8.8 %) patient self-assessments <4 METs as estimated by DASI. These results represent a significant discrepancy between assessments (Fisher’s exact, two-tailed P value <0.0001). The standard T-score of PROMIS estimates of functional capacity correlated with DASI estimates (R 2 0.76). The mean and standard deviation for PROMIS T-scores were 43.3 and 9.86, respectively (mean 50.0; SD 10.0 for the general population).
Of the 203 patients who completed the entire study survey, 192 (94.6 %) stated that they did not require assistance from another person, and 187 (94 %) responded either “agree” or “strongly agree” to the DASI questionnaire being “easy to understand” and “easy to complete;” 186 (93 %) and 188 (94 %), respectively, responded similarly to the PROMIS questionnaire.
While both electronic questionnaires were easy to understand and complete for most study participants, there was a significant discrepancy between clinician assessments and patient self-assessments of functional capacity. Further study is needed to determine if either patient self-triage by means of activity questionnaires or clinician evaluation is valid and reliable in the preoperative setting.
Valid and reliable assessment of functional capacity is an important component of the preoperative evaluation. Patient functional capacity directs decisions about preoperative cardiac evaluation and is useful for risk stratification prior to surgery (Fleisher et al. 2014). Poor performance on formal exercise tolerance testing reliably correlates to increased risk for perioperative complications in several different patient populations and treatment settings (Snowden et al. 2010; Wilson et al. 2010). However, because of expense and practical considerations, exercise tolerance testing is not routinely performed prior to non-cardiac surgery in the USA. Functional capacity is commonly assessed through obtaining the patient’s history regarding their ability to perform certain physical activities. Clinician-elicited stair-climbing ability has been shown to correlate to perioperative cardiac events and other complications (Reilly et al. 1999), and categorical metabolic equivalents of task (METs) estimates, as determined through clinician history of physical capabilities, have been shown in a univariate analysis to be predictive of perioperative cardiac outcomes (Wiklund et al. 2001).
Activity questionnaires, such as the Duke Activity Status Index (DASI) (Hlatky et al. 1989) and the Patient-Reported Outcomes Measurement System (PROMIS)–Short Form 12a–Physical Function (www.nihpromis.org), are available to guide clinicians when estimating METs in the preoperative assessment of functional capacity (Fleisher et al. 2014). Patient-completed versions of the DASI questionnaire have been shown to correlate moderately well with physiologic measures of functional capacity or exercise tolerance in several clinical settings (Dunagan et al. 2013; Shaw et al. 2006; Struthers et al. 2008), and patient reported exercise capacity has been shown to be predictive of survival in vascular surgery patients (Boult et al. 2015). Patient self-assessment of functional capacity by means of electronic questionnaires would allow METs estimates to be known prior to in-person, preoperative evaluations, enabling preoperative triage of patients based on estimated functional capacity, a core component of preoperative evaluation (Fleisher et al. 2014).
In this study, we sought to evaluate the feasibility patient self-triage regarding functional capacity by investigating the correlation between clinician assessments and patient self-assessment of functional capacity, as assisted by electronic, patient-completed DASI and PROMIS questionnaires. In addition, we analyzed patient survey data regarding the usability of these two validated activity questionnaires.
Population and enrollment
The study was approved by the Vanderbilt University Institutional Review Board. All patients, age 18 years or older, who were scheduled for elective surgery at our institution and seen in the Preoperative Evaluation Center (PEC) prior to their surgery were eligible for enrollment. Patients undergoing moderate to high-risk surgeries or who have moderate to high-risk comorbidities are referred to our PEC by their surgeon. A member of the study team recruited patients immediately prior to the preoperative evaluation and obtained written informed consent. Baseline data was not available for power analysis prior to initiating enrollment; therefore, a convenience sample of patients was recruited during the month of March 2015.
Prior to initiation of the clinician encounter, participants were asked to independently complete electronic questionnaires on a tablet computer (iPad, Apple Inc.; Cupertino, CA). The DASI and the PROMIS questionnaires, in addition to questions to assess the comparative usability of these two formal activity questionnaires (see Additional files 1, 2, and 3), were administered using the research electronic data capture system (Harris et al. 2009). The study administrator was not present with the patients as they completed the questionnaire. Patients were asked to complete the questionnaire without assistance but were permitted help from an accompanying family member, friend, or care provider if necessary.
Clinician evaluation and functional capacity assessment
Following completion of the study questionnaire, each patient underwent preoperative clinical evaluation. The clinicians performed and documented the evaluation in accordance with the standard practice for all preoperative consultations in the PEC at our institution. Routine documentation of PEC evaluations includes estimating functional capacity in our electronic medical record as one of five categories: excellent (>7 METs); very good (5–7 METs); good (4 METs); fair (2–3 METs); and poor (1–2 METs). As a reference tool, clinicians are provided with a list of physical activities and the METs associated with those activities as described in the 2007 American College of Cardiology/American Heart Association Guidelines on Perioperative Cardiac Evaluation and Care for Noncardiac Surgery (Fleisher et al. 2007). Additionally, clinicians have structured documentation for reasons for physical limitations, including angina, dyspnea, claudication, and fatigue, as well as the ability to provide free text descriptions of other reasons for physical function limitations. The clinicians performing the preoperative assessments were blinded to the patient responses on the DASI and PROMIS forms.
Questionnaire scoring and data elements
At the conclusion of patient enrollment, the DASI and PROMIS questionnaire elements were scored. The DASI questionnaire was scored according to the published methodology to estimate functional capacity in terms of METs (Hlatky et al. 1989). Individual DASI questions carry different weight, and the questionnaire can be scored to produce a METs estimate from 2.74 to 9.89 METs. A raw score for the PROMIS questionnaire is generated from the responses to the five-point Likert options. Raw scores range from 6 to 60 and correlated to a standard T-score. The correlation of raw scores to standard T-scores is developed from population statistics of functional capacity.
Responses to the questions regarding the clarity and usability of the DASI and PROMIS electronic questionnaires were directly analyzed for comparative usability of these two formal activity questionnaires.
From the electronic medical record, we retrieved documented clinician estimates of functional capacity, as well as American Society of Anesthesiologists (ASA) physical status. The previously described categorical estimates of functional capacity were then compared to the results from patient-completed DASI and PROMIS electronic questionnaires. We also searched the medical record for study participants who had completed exercise tolerance testing or exercise stress testing.
To determine the correlation between clinician assessments and patient self-assessments of functional capacity, DASI METs estimates were compared to clinician categorical assessments of functional capacity using the dichotomous categories of ≥4 METs and <4 METs. Statistical correlation was analyzed using a two-tailed, Fisher’s exact test. This METs threshold was chosen, as it is a branch point in the algorithm for preoperative evaluation of coronary artery disease, as described by the 2014 American College of Cardiology/American Heart Association Guideline on Perioperative Cardiovascular Evaluation and Management of Patients Undergoing Noncardiac Surgery (Fleisher et al. 2014). The T-scores from the PROMIS questionnaire results were compared to the DASI METs estimates using linear regression analysis.
After IRB approval, 211 patients consented for participation; 204 patients were eligible for inclusion in the final analysis of functional capacity assessments. Reasons for exclusion from the final analysis include failure to complete the survey (six patients) and absence of a documented clinician estimate of functional capacity (one patient). Of the six patients who did not complete the survey, two were due to clinician interruption, two were due to participant refusal to answer specific survey questions, one was due to inability to understand the questions, and one was due to accidental closure of the electronic survey application. One patient completed both the DASI and PROMIS components of the survey and then accidentally closed the electronic survey application prior to completion of the final usability field; thus, only 203 participants are included in the final usability analysis.
Of the 204 patients included in the final analysis of functional capacity assessments, the mean age was 56.8 (standard deviation 15.3); 32.2 % of the participants were classified as ASA I/II, 67.8 % ASA III/IV (Table 1).
Functional capacity estimates
Clinicians assessed functional capacity at <4 METs for 109 patients (53.4 %), while only 18 patients (8.8 %) assessed their functional capacity at <4 METs, as calculated by their responses on the DASI. These results represent a significant discrepancy between assessments around the clinically relevant point of 4 METs (Fisher’s exact, two-tailed P value <0.0001). Graphical relationship between categorical clinician functional capacity assessments and DASI patient self-assessments is displayed in Figs. 1 and 2. The standard T-score of PROMIS estimates of functional capacity correlated linearly (R 2 0.76) with DASI estimates of functional capacity (see Fig. 3). The mean and standard deviation for PROMIS T-scores were 43.3 and 9.86, respectively (mean 50.0; SD 10.0 for the general population). No patients enrolled in our study had documentation of exercise testing; thus, comparison of clinician and patient assessments of functional capacity to physiologic measures of functional capacity was not possible.
Of the 203 included in the usability analysis, 192 (94.6 %) stated they did not require assistance from another person. One hundred eighty-seven (94 %) responded either “agree” or “strongly agree” to the DASI questionnaire being “easy to understand” and “easy to complete,” and 186 (93 %) and 188 (94 %), respectively, responded similarly for the PROMIS questionnaire. See Table 2 for further usability data.
The assessment of functional capacity is an integral component of the preoperative evaluation. We sought to determine whether patient self-assessment of functional capacity using electronic activity questionnaires is feasible and valid in the setting of preoperative evaluations. In our patient population, while both electronic questionnaires were easy to understand and complete for most study participants, there was a significant discrepancy between clinician assessments and patient assessments from formal valid questionnaires, particularly around the dichotomous result of whether or not a patient can achieve four or more METs of physical work.
METs assessment discrepancy
The significant discrepancy between clinician and patient assessments of functional capacity in our patient population highlights that a clinician-elicited history regarding physical capabilities may be very different than physical capabilities that are purely patient-reported. This distinction was evident even in the initial validation of the DASI (Hlatky et al. 1989). The DASI was developed in two phases, a development phase in which an interviewer asked the subjects questions regarding physical function, and a validation phase in which the subjects independently completed the initial version of the DASI. In both phases, subjects underwent exercise tolerance testing after either the interview or completion of the DASI questionnaire. Both phases showed statistically significant correlation of “patient-reported” functional capacity to physiologic measures of functional capacity; however, the correlation was better in the development phase (Spearman’s correlation 0.81) than in the validation phase (Spearman’s correlation 0.58) (Hlatky et al. 1989). It may be that the personal interaction between interviewer and interviewee increases the accuracy of patient-reported physical capabilities. McGlade et al. (2001) demonstrated that patient-completed DASI scores correlated with a next of kin’s DASI assessment of functional capacity; however, patients slightly overestimated their capabilities as compared to their next of kin (McGlade et al. 2001). In our study, participants were often accompanied by a spouse, friend, child, or other care provider, who may have added to the interaction through verbal or non-verbal communication with the clinician, which could increase the accuracy of clinician functional capacity assessments.
Reliability of patient-reported health information
Several studies have evaluated the ability of patients to identify clinical risk factors in the setting of chronic diseases and demonstrated that patient-completed surveys regarding diagnosed medical conditions are probably at least as reliable as the medical record (Tisnado et al. 2006; Okura et al. 2004). It has also been shown that Revised Cardiac Risk Index (Lee et al. 1999) scores calculated from patient survey data correlate well with scores calculated from clinician documentation of risk factors (Manaktala et al. 2013). However, these studies did not evaluate the reliability of patient-reported functional capacity, which may be different than other types of patient-reported health information.
Patients may have an increased tendency to over-estimate their physical capabilities when completing an activity questionnaire. Dunagan et al. (2013) showed slight over-estimation of METs level by the DASI questionnaire when compared to exercise tolerance testing in the setting of cardiac rehabilitation, although this over-estimation did not represent a significant difference in their sample (Dunagan et al. 2013). Formal activity questionnaires are traditionally validated by having participants complete the questionnaire and then subsequently asking patients to undergo a physiologic measure of functional capacity, such as exercise tolerance testing (Hlatky et al. 1989; Struthers et al. 2008). In the absence of being required to demonstrate physical capabilities, as was the case in our study, participants may have an increased tendency to exaggerate physical capabilities. McGlade et al. (2001) found that of 68 patients who answered affirmatively to DASI question number four regarding stair climbing, 13 patients were unable to demonstrate the ability to climb a flight of stairs (McGlade et al. 2001). The authors concluded that rather than asking patients if they are capable of a physical task, it may be more useful to ask them to demonstrate their ability to do so (McGlade et al. 2001).
It is also possible that clinician underestimation of functional capacity contributed to the discrepancy demonstrated in our study. Anecdotally, the authors have observed that clinicians often use one or two history questions in the assessment of a patient’s functional capacity. Thus, a negative response to a single question may lead a clinician to underestimate a patient’s functional capacity when a patient is capable of achieving four METs of physical work during activities not addressed by the clinician.
Additionally, while the clinician functional capacity tool employed at our PEC includes phrases to describe physical activity that are very similar or equivalent to the language used in DASI questions, the categorical METs assessments assigned to a particular group of activities does not necessarily correspond to the DASI scoring formula. Thus, it is possible for clinician and patient-completed DASI METs estimates to be discrepant, even if patients described their physical capabilities to clinicians in complete concordance with how they responded to DASI questions.
While there is poor correlation between clinician and questionnaire estimates of functional capacity, the majority of our study participants found the electronic questionnaires easy to understand and easy to complete. Patients were able to complete these surveys on a touch-screen, tablet computer independently and in a timely manner. Our patient-entered responses to the usability of the electronic questionnaires suggest that this modality would be acceptable for application in patient-self triage if self-assessment of functional capacity using a formal activity questionnaire was validated as an accurate method of determining functional capacity in the preoperative setting.
As there were no physiologic measures of functional capacity obtained on patients enrolled in our study, we cannot comment on the validity of either clinician assessments or patient assessments of functional capacity relative to performance on physiologic measures of functional capacity, such as cardiopulmonary exercise testing. Without physiologic measures of functional capacity, we cannot determine whether the discrepancy between clinician and questionnaire estimates of functional capacity represents patient over-estimation or clinician underestimation of functional capacity. Similarly, without data regarding the surgical outcomes of the study participants, we cannot determine whether clinician or patient assessments of functional capacity are more clinically relevant for predicting surgical complications or major adverse cardiac events, nor can we determine the relative clinical significance patient responses to particular survey questions, such as stair-climbing ability. Whether or not patients can accurately report their functional capacity or clinicians accurately assess functional capacity, both assessments may still be useful in screening for high-risk patients (Reilly et al. 1999; Wiklund et al. 2001; Boult et al. 2015).
Directions for future study
Electronic versions of formal activity questionnaires have theoretical potential to enable patient self-triage regarding functional capacity. As it is unclear which patients may benefit from a clinician evaluation at a PEC prior to the day of surgery, implementation of patient self-assessments using such tools could improve the efficiency and quality of preoperative evaluations while reducing costs by limiting unnecessary evaluations and testing in patients who have good functional capacity. However, further work is needed to determine whether patients accurately assess their own functional capacity in the preoperative setting using electronic versions of formal activity questionnaires. Future study is also needed to determine whether self-assessment of functional capacity by means of electronic questionnaires can predict perioperative complications or major adverse cardiac events. Finally, since functional capacity is commonly assessed by clinicians, more study is needed to determine whether clinicians accurately and reliably assess a patient’s functional capacity through routine history questions.
Our patient population found electronic versions of both the DASI and PROMIS activity questionnaires easy to understand and complete, suggesting a potential for application of similar tools for patient self-triage prior to preoperative evaluations. However, we found a significant discrepancy between clinician and patient self-assessment of functional capacity. Before preoperative, patient self-triage regarding functional capacity is implemented, more study is needed to determine whether patients accurately assess their own functional capacity using activity questionnaires. Additionally, our results highlight the importance of the distinction between clinician-elicited and patient-reported functional capacity, as the two may not be equivalent.
ASA, American Society of Anesthesiologists; DASI, Duke Activity Status Index; METs, metabolic equivalents of task; PEC, Preoperative Evaluation Center; PROMIS, Patient-Reported Outcomes Measurement Information System
Boult M et al. Self-reported fitness of American Society of Anesthesiologists class 3 patients undergoing endovascular aortic aneurism repair predicts patient survival. J Vasc Surg. 2015;62:299–303.
Dunagan J, Adams J, Cheng D, Barton S, Bigej-Cerqua J, Mims L, Molden J, Anderson V. Development and evaluation of a treadmill-based exercise tolerance test in cardiac rehabilitation. Proceedings (Baylor Univ Med Center). 2013;26:247.
Fleisher LA, Beckman JA, Brown KA, Calkins H, Chaikof EL, Fleischmann KE, Freeman WK, Froehlich JB, Kasper EK, Kersten JR, Riegel B, Robb JF, Smith SC Jr, Jacobs AK, Adams CD, Anderson JL, Antman EM, Buller CE, Creager MA, Ettinger SM, Faxon DP, Fuster V, Halperin JL, Hiratzka LF, Hunt SA, Lytle BW, Nishimura R, Ornato JP, Page RL, Riegel B, Tarkington LG, Yancy CW. ACC/AHA 2007 guidelines on perioperative cardiovascular evaluation and care for noncardiac surgery: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to Revise the 2002 Guidelines on Perioperative Cardiovascular Evaluation for Noncardiac Surgery) developed in collaboration with the American Society of Echocardiography, American Society of Nuclear Cardiology, Heart Rhythm Society, Society of Cardiovascular Anesthesiologists, Society for Cardiovascular Angiography and Interventions, Society for Vascular Medicine and Biology, and Society for Vascular Surgery. J Am Coll Cardiol. 2007;17:1707–32.
Fleisher LA, Fleischmann KE, Auerbach AD, Barnason SA, Beckman JA, Bozkurt B, Davila-Roman VG, Gerhard-Herman MD, Holly TA, Kane GC, Marine JE, Nelson MT, Spencer CC, Thompson A, Ting HH, Uretsky BF, Wijeysundera DN. 2014 ACC/AHA guideline on perioperative cardiovascular evaluation and management of patients undergoing noncardiac surgery: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J of the Am Col of Cardiol. 2014;64:e77–137.
Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–81.
Hlatky MA, Boineau RE, Higginbotham MB, Lee KL, Mark DB, Califf RM, Cobb FR, Pryor DB. A brief self-administered questionnaire to determine functional capacity (the Duke Activity Status Index). Am J of Cardiol. 1989;64:651–4.
Lee TH, Marcantonio ER, Mangione CM, Thomas EJ, Polanczyk CA, Cook EF, Sugarbaker DJ, Donaldson MC, Poss R, Ho KKL, Ludwig LE, Pedan A, Goldman L. Derivation and prospective validation of a simple index for prediction of cardiac risk of major noncardiac surgery. Circulation. 1999;100:1043–9.
Manaktala S et al. Validation of pre-operative patient self-assessment of cardiac risk for non-cardiac surgery: foundations for decision support. AMIA Annu Symp Proc. 2013;2013:931–8.
McGlade DP et al. The use of a questionnaire and simple exercise test in the preoperative assessment of vascular surgery patients. Anaesth Intensive Care. 2001;29:520–6.
Okura Y et al. Agreement between self-report questionnaires and medical record data was substantial for diabetes, hypertension, myocardial infarction and stroke but not for heart failure. J Clin Epidemiol. 2004;57:1096–103.
Patient Reported Outcomes Measurement Information System SF 12a. National Institutes of Health. www.nihpromis.org. Accessed January 27, 2015.
Reilly DF, McNeely MJ, Doerner D, Greenberg DL, Staiger TO, Geist MJ, Vedovatti PA, Coffey JE, Mora MW, Johnson TR, Guray ED, Van Norman GA, Fihn SD. Self-reported exercise tolerance and the risk of serious perioperative complications. Arch of Intern Med. 1999;159:2185–92.
Shaw LJ, Olson MB, Kip K, Kelsey SF, Johnson BD, Mark DB, Reis SE, Mankad S, Rogers WJ, Pohost GM, Arant CB, Wessel TR, Chaitman BR, Sopko G, Handberg E, Pepine CJ, Merz CNB. The value of estimated functional capacity in estimating outcome: results from the NHBLI-Sponsored Women’s Ischemia Syndrome Evaluation (WISE) study. J Am Col Cardiol. 2006;47:s36–43.
Snowden CP, Prentis JM, Anderson HL, Roberts DR, Randles D, Renton M, Manas DM. Submaximal cardiopulmonary exercise testing predicts complications and hospital length of stay in patients undergoing major elective surgery. Annals of Surg. 2010;251:535–41.
Struthers R, Erasmus P, Holmes K, Warman P, Collingwood A, Sneyd JR. Assessing fitness for surgery: a comparison of questionnaire, incremental shuttle walk, and cardiopulmonary exercise testing in general surgical patients. Br J Anaesth. 2008;101:774–80.
Tisnado DM et al. What is the concordance between the medical record and patient self-report as data sources for ambulatory care? Med Care. 2006;44:132–40.
Wiklund RA, Stein HD, Rosenbaum SH. Activities of daily living and cardiovascular complications following elective, noncardiac surgery. Yale J Bio Med. 2001;74:75.
Wilson RJT, Davies S, Yates D, Redman J, Stone M. Impaired functional capacity is associated with all-cause mortality after major elective intra-abdominal surgery. Brit J of Anaesth. 2010;105:297–303.
We would like to thank Martha Tanner for her assistance in final preparation and formatting of the manuscript.
Dr. Wanderer is supported by the Foundation for Anesthesia Education and Research (FAER) and the Anesthesia Quality Institute (AQI), Mentored Research Training Grant in Health Services Research (MRTG-HSR).
JS helped has seen the original study data, reviewed the analysis of the data, conduct the study, enrolled participants, analyzed the data, and wrote the manuscript. JW has seen the original study data, reviewed the analysis of the data, designed the study, advised in the data analysis, edited the manuscript, and is the author responsible for archiving the study files. MM has seen the original study data, reviewed the analysis of the data, participated in study design, data analysis, and edited the manuscript. All authors read and approved the manuscript.
Dr. McEvoy receives funding for research from the GE Foundation and Edwards Life Sciences. This project did not have funding or involvement from either of these entities. Neither entity is referenced in the manuscript.
Dr. Wanderer is supported by the Foundation for Anesthesia Education and Research (FAER) and the Anesthesia Quality Institute (AQI), Mentored Research Training Grant in Health Services Research (MRTG-HSR).
The authors declare that they have no competing interests.
Usability Survey. This file represents the survey completed by participants to assess the usability of the two formal activity questionnaires employed in this study for patient self-triage regarding functional capacity. (DOCX 73 kb)
Patient-Reported Outcomes Measurement System (PROMIS)–Short Form 12a–Physical Function. This file is a text version of the PROMIS–Short Form 12a–Physical Function formal activity questionnaire. (PDF 46 kb)
Duke Activity Status Index (DASI). This file is a text version of the Duke Activity Status Index (DASI) formal activity questionnaire. (PDF 43 kb)