Title: A Scoping Review of Interrater Reliability for Lumbar Somatic Dysfunction
Authors: Tianfu Shang, OMS I; Phillip G. Munoz, OMS III; Angie K. Maxson, OMS IV; Kristen P.W. Gavin, OMS IV; Crystal Martin, DO
Introduction
Osteopathic physicians (DO), physical therapists (PT), and chiropractors learn multiple modalities for evaluating lumbar somatic dysfunctions (SD). The skills used are outlined by the Educational Council on Osteopathic Principles (ECOP) in A Teaching Guide for Osteopathic Manipulative Medicine. Therefore, validity and reliability of these techniques in clinical practice is crucial for consistent instruction, evaluation, diagnosis, and treatment. This scoping review assesses interrater reliability (IRR) of palpatory, and visual assessment techniques as outlined by ECOP for the evaluation of lumbar SD.
Methods
A literature search was conducted through PubMed, the Journal of Osteopathic Medicine (JOM) /Journal of American Osteopathic Association, OSTMED.DR, and Cochrane Review. PubMed queries included terms from the ECOP relating to SD, and (interrater[tiab] OR interobserver[tiab] OR interexaminer[tiab] OR intertester[tiab]) AND (humans[filter]) NOT (“diagnostic imaging”[mesh] OR radiography[mesh]). JOM, OSTMED.DR, and Cochrane Review queries were interrater, interobserver, interexaminer, and intertester. Each abstract was screened independently by two reviewers with a third reviewer as tiebreaker. Articles were reviewed and included based on publication in an English speaking, peer reviewed journal that included numerical statistics on IRR between healthcare professionals using palpatory and visual techniques in the evaluation of lumbar SD.
Results
The search produced 167 unique articles (8 Cochran, 7 JOM, 14 OSTMED.DR, and 138 PubMed), of which 30 articles met our inclusion criteria. Many studies evaluated multiple regions with only 6 studies focusing exclusively on the lumbar spinal segments. Distribution of examiner’s discipline is as follows: 8 PTs, 8 chiropractors, 6 DOs, 5 interdisciplinary, and 2 unspecified. Examiner’s training level ranged from students to practitioners. The number of examiners per study ranged from 2 to 28 with an average of 4.3. The number of subjects evaluated ranged from 17 to 161 with an average of 53.9. Results were reported using Cohen’s kappa, percent agreement, or interclass correlation (ICC) by 23 studies. Reported kappa ranged from poor to excellent (-0.36 < k < 1.00), though many studies concluded IRR for most items were poor to moderate (0.20 < k < 0.41).
Discussion
This scoping review has revealed that the available data for the IRR of many palpatory and visual techniques for evaluating lumbar SD has significant variation in methodology, sample size, examiner training, modality tested, and reporting of results. Furthermore, the wide variability in reported results leaves room for more targeted studies to investigate the clinical validity and reliability of many techniques, special tests, and maneuvers as outlined by ECOP.
Thank you for this presentation, Student Dr. Shang. I have been assigned as one of the judges for this project. Do you have any ideas for how DOs could increase standardization and objectivity in assessing lumbar dysfunction?
Thank you for your interest and attention, Dr. Briggs-Early. Stemming from this scoping review I believe there are a few methods to increase standardization of future studies which would allow for better assessment of IRR. First, there could be a standardization of the criterion for subject inclusion and exclusion. Many studies were more narrowly focused on specific subsets of subjects which limited the demographics evaluated and the size of study. Second, standardization to using Cohen’s Kappa to evaluate IRR between each set of two raters would allow for better comparison between studies. Lastly, there was considerable variation in blindedness of the raters, subjects, and presence of a training period. These are some of the few steps that could be implemented in future studies to improve objectivity in assessing IRR for palpatory techniques. Thank you for your inquiry.
Great, thanks so much for your presentation and thoughtful response to my question!
Thank you for your presentation, being a physical therapist, I think it would be great to work with DO’s on projects such as these to further standardize these practices. How would you suggest that either discipline approach the other to work on standardization of protocols?
Thank you for your interest and attention, Dr. Walser. One challenge faced when analyzing the reported data between studies was the variation in nomenclature and description of palpatory technique. Standardization in these areas would allow for better comparative analysis between disciplines. There is also variation among professions on the preference of statistical methodology. Standardization around the more granular Cohen’s Kappa between two raters for IRR assessment would allow for better comparison of similar techniques. Lastly, more interdisciplinary teams of investigators could allow for more collaboration and consensus building on shared methodology between our two closely related fields. Thank you for your inquiry.
I’m one of the judges assigned to this project and I’d like to thank you, Student Dr. Shang, for sharing your presentation. You mentioned that the examiners ranged from the student to practitioner level and come from different disciplines. Were there differences between interrater reliability across levels/disciplines?
Thank you for your attention and interest, Dr. Smith. Our initial analysis did not examine whether a relationship existed between different disciplines or level of training. This is partially due to the high variation in the methodology of studies which would muddy the analysis. It is also partially due to studies self equalizing training levels through either prepatory sessions, reporting IRR within training level, or reported IRR as an average of the group of raters. This would be an area for further evaluation which none of the articles that met inclusion criteria examined as a primary goal. Thank you for your inquiry.