United States

Patient-reported outcomes (PROs) are increasingly recognized as a valuable data source that captures a patient’s experience of their disease, yet routine clinical PRO collection is still uncommon. We sought to understand the barriers and facilitators to sustainable PRO collection in an outpatient musculoskeletal practice. We performed a formal root-cause analysis and separated identified issues into three categories: technical, process, and human. We developed a multipronged intervention to target root causes. Qualitative feedback from patients and staff suggested that these changes resulted in more streamlined workflows and increased satisfaction.

AMA Research Challenge 2023

Optimizing patient-reported outcome collection in a musculoskeletal outpatient clinic setting

poster

Welcome to the **largest national, multi-specialty research** event featuring research from nearly 700 medical students, residents, fellows, and international medical graduates.

Use the buttons below or navigation menu to access the event.
 


[![](https://assets.underline.io/uploads/markdown_image/1/image/3e33a8c754cc49610b30950617d813e7.png)](https://underline.io/events/434/sessions?searchGroup=all)
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/551404d429c4e274f9a54cbdd7081918.png)](https://underline.io/events/434/posters?searchGroup=all)
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/677d92f566b880cebd36905fc738fc10.png)](https://underline.io/events/434/lounge)
 
[![](https://assets.underline.io/uploads/markdown_image/1/image/c7392833c7261c3874c5b43fa2f4df19.png)](https://underline.io/events/434/help-desk)
 

#### Event Details

The 2023 American Medical Association Research Challenge includes a poster symposium hall and a semifinals hall with the top scored research. **Log in at any time throughout the event to:**

* **View posters and presentations** on a variety of topics and specialties
* **Score posters in the semifinals** to help decide the top five finalists (AMA member exclusive opportunity)
* **Connect with participants** and provide feedback to help advance their research
* **Attend education sessions** with physician experts on key research topics

At the conclusion of this event, the AMA will announce the top five finalists who will advance to the AMA Research Challenge finals, present their research to an elite panel of judges, and compete to win a $10,000 grand prize sponsored by Laurel Road. 

### Grand Prize Sponsor

AMA would like to thank Laurel Road for sponsoring a $10,000 grand prize for the winner of the AMA Research Challenge. Laurel Road is not involved in the judging or scoring process to select the winner. The winner will be announced on February 4, 2024 during the AMA Research Challenge finals. Payment of the grand prize will be directly from Laurel Road. See the AMA Research Challenge Official Rules for terms and conditions.

![](https://assets.underline.io/uploads/markdown_image/1/image/611e331d4a0eba0920f240cfc04b4d77.png)

**There is no registration fee for this event! **

Register to explore research in a variety of topics. AMA members get exclusive access to score posters in the semifinals.

**Please use this link [here](https://www.ama-assn.org/about/events/ama-research-challenge) to register for free.**

Registration Is Required

Join the AMA Research Challenge poster symposium and semifinals to explore top research posters, score posters in the semifinals (an AMA member exclusive opportunity) and attend education sessions with experienced physicians to learn tips to advance your research and career. 

Background

The Resource-Based Relative Value Scale (RBRVS) is utilized by the Centers for Medicare & Medicaid Services (CMS) and private payers alike to determine fee for service physician reimbursement. RVUs incorporate clinical and nonclinical resources, expertise required, and physician effort. Studies have demonstrated durable differences in RVUs for lumbar fusion instrumentation, but there has been no study on RVU reimbursement in cervical spine fusion. This study aims to compare mean work RVUs (wRVUs), mean operative time (OpTime), and wRVUs/min in single and multi-level anterior (ACDF) and posterior (PCDF) cervical decompression and fusions performed between 2011 and 2020.
Methods

The American College of Surgeons National Surgical Quality Improvement Program (NSQIP) was queried for Current Procedural Terminology (CPT) codes. CPT code 22551 reflects a primary ACDF level and subsequent 22552 codes correspond to additional levels while 22600 reflects a primary PCDF level and subsequent 22614 codes correspond to additional levels. The cohort was stratified into 5 groups: single-level, 2-level, 3-level, 4-level, and 5+ level anterior and posterior cervical fusions. Mean operative times, mean wRVUs, and wRVU/min were calculated and compared by the heteroscedastic student’s t-test.
Results



100,997 patients who underwent cervical fusion met the criteria for inclusion in this study in the following cohorts, with the vast majority receiving 1 and 2 level fusions. In single-, 3-level, 4- level, and 5+ level fusion, the anterior approach demonstrated significantly lower mean wRVU (p<0.001). Interestingly, an anterior approach demonstrated higher mean RVU for a 2-level fusion (67.76 ± 14.72 vs 65.98 ± 23.02, p<0.0001) over the study period. In single-, 2-, 3-, and 4-level fusion the anterior approach had significantly lower OpTime (all p<0.001). The anterior
approach also demonstrated significantly higher wRVU/min in single- and 2-levels (p<0.001) but lower wRVU/min in 3- and 4-level fusions ( p<0.001). No differences were seen in OpTime or wRVU/min between anterior and posterior approach for 5+ level fusions (p>0.05).
Conclusion

For single through 4 level fusions, anterior approach demonstrated lower OpTime and higher wRVU/min for single and 2-level fusions. As the number of fusion levels increased to 3- and 4- levels, wRVU/min was higher for posterior approaches. At 5+ levels, these returns became indistinguishable. These results may guide individual practices as spine surgeons aim to optimize surgical quality and outcomes as health services reimbursement shifts toward value based bundled reimbursements.

Comparison of Relative Value Unit (RVU) reimbursement anterior or posterior cervical spinal fusion

Introduction/purpose: 
Increased access to the internet and subsequent utilization of online health education materials by patients have established a demand for formal appraisal of such resources. The aim of this study was to evaluate the quality, credibility, and readability of web-based information related to turbinoplasty.

Methods: 
Search terms including “turbinoplasty nasal surgery”, “turbinate reduction”, “turbinate surgery”, and “nasal conchae surgery” were used to perform a search on Google.com, Bing.com, and Yahoo.com. Within each search term used, the top 30 results from Google.com were assessed, while the top 10 and top 5 results were assessed from Bing.com and Yahoo.com, respectively. The webpages were separated by authorship classification into six distinct groups: academic, commercial, governmental agency, health information platform, professional society, or other. To assess web page readability, the Flesch-Kincaid grade level, Gunning-Fog Index, SMOG Index, and Coleman-Liau Index were utilized. Webpage quality was computed through the DISCERN Instrument (DISCERN), the Journal of the American Medical Association benchmark criteria (JAMA), Novel Turbinoplasty Index (NTI) and presence of code certification by Health on The Net (HON).

Results: 
A final count of 71 unduplicated text-based web pages were appraised. Information quality was measured by an average DISCERN score of 47.4±7.40, indicating “fair” quality according to the scoring bands. The average readability was 9.7±1.57, notably higher than both grade level readability recommendations from either the American Medical Association (AMA) or National Institutes of Health (NIH). Of all web pages, only 11 (15.49%) proficiently met all four listed JAMA criteria. Significant correlations between web page classification and average DISCERN (p = 0. 0042) as well as JAMA score (p < 0.001) were discovered. The web pages that had HON code certification revealed significantly higher quality metrics with higher DISCERN scores (p < 0.001), JAMA scores (p < 0.001) and NTI scores (p = 0.038).

Conclusion:
Online health information associated with turbinoplasty has a considerably “fair” quality, while the average readability is several grade levels above the current AMA and NIH recommendations. Healthcare professionals should be encouraged to instruct their patients on proper search techniques for high-quality health resources on the web, as well as provide these materials in a physical written format when appropriate. We also recommend that health-based web pages amend their published information in a manner which can more effectively reach their intended consumers without sacrificing quality.

Quality and Readability Analysis of Turbinoplasty on the Web

Objective
To develop an evaluation clinic for toddlers suspected of having autism spectrum disorder (ASD) that improves the efficiency of the evaluation process and care coordination for those diagnosed with ASD. 

Purpose
While most children in the U.S. (85%) identified with ASD had documented caregiver concerns about their development by 3 years of age, less than half received an evaluation before their 3rd birthday.1 In addition, after receiving a diagnosis of ASD, many families struggle to connect with the recommended services and caregiver understanding of common features of ASD can be limited. MIND Institute staff recognized the need to develop a streamlined process to evaluate young children referred for ASD and assist families with navigating services. 

Methods
Children ages 15 to 42 months at time of referral for concern of ASD were triaged to the clinic. A licensed clinical social worker (LCSW) completed a telephone intake with families prior to the evaluation to explain the nature of the clinic and complete a structured interview inquiring about developmental milestones and behaviors concerning for ASD. The in-person evaluation consisted of an Autism Diagnostic Observation Schedule 2nd Edition, and the Bayley Scales of Infant and Toddler Development 4th Edition (BSID-4) cognitive subtest or the Differential Ability Scales 2nd Edition (DAS-II). The BSID-4 and DAS-II were not completed if either developmental assessment was recently completed and documented in the electronic health record. If the patient was diagnosed with ASD, caregivers met with a LCSW to discuss next steps in service coordination and were referred to the UC Davis MIND Institute Caregiver Coaching and Family Navigator programs.

Results
Since the clinic started in August 2022, 29 children ages 19 to 49 months (23 M, 6 F) have received an ASD evaluation. One appointment was rescheduled, and none were canceled. The average time from referral to evaluation was 5 months. Twenty-five children (86%) were diagnosed with ASD; 16 (64%) of these families were referred to the Family Navigator program. Twenty-one of the 29 families were insurance eligible for the Caregiver Coaching program; of those, 4 (19%) have been enrolled and 12 (57%) are pending enrollment. 

Conclusion 
Preliminary data suggests that structured intakes completed by a LCSW prior to and service coordination after an initial ASD evaluation can improve clinic show rates and engagement in services. 


A Clinic Model for Evaluation and Care Coordination for Toddlers Suspected of Having Autism

Background: To assess school-aged children’s wellbeing and to characterize changes in school-related needs from 2021 to 2022.
Methods: We surveyed a nationally representative sample of parents of children enrolled in grades K-12 in the 2021-2022 academic year in June 2021 and again one year later (June 2022). Parents completed the Strengths and Difficulties Questionnaire (SDQ) and reported their child’s general health and needs for school-related services. Weighted regressions examined associations among wellbeing, needs, and socio-demographic characteristics.
Results: 1,504 participants completed baseline and 1,199 completed follow up surveys. The percentage of children with deficits in total difficulties, hyperactivity, and prosocial behavior decreased from baseline to follow up, but a high percentage of children continued to have peer problems. Parents reported receiving support from their school for 45.3% of needs during the 2021-2022 school-year but 72.5% of parents continued to have at least one school-related need. Highest priority needs were academic enrichment, socialization, physical activity, tutoring, and coping with stress.
Implications for School Health Policy, Practice, and Equity: While many children with school-related needs received school support, a large percentage continue to have unmet needs and experience peer-related problems.
Conclusions: Continued investment in school-based services is necessary to support child health.

Changes in School-Age Children’s Wellbeing and School-Related Needs during the COVID-19 Pandemic

Background: The Adverse Childhood Experiences (ACEs) questionnaire is a widely used screener for pediatric trauma exposure and general household dysfunction. On a population level, ACEs are cumulative risk factors for a range of mental health and physical conditions. ACEs research has been largely based on self-reported ACEs exposures and health conditions, anthropometric measures, and biomarkers. In contrast, data mined directly from the electronic health record (EHR) allows for more objective analysis and reduces potential harms and limitations of self-report. The objectives of this pilot were to generate and test an algorithm for deriving Proxy ACEs values from existing EHR data, and to study the population-level association between derived Proxy ACEs, medical morbidity, and healthcare utilization. 

Methods: Proxy ACE scores were derived from structured EHR data including diagnoses, chief complaints, and social and family history sections. Medical morbidity was operationalized using the total number of medical diagnoses and prescribed medications, while healthcare utilization was represented by the number of emergency room visits, number of hospitalizations, and total hospital length of stay. We generated a dataset of 142,719 patients <18 years old who received care at Yale, spot-checked the derivation algorithm, and iterated modifications to obtain their final Proxy ACE scores. 

Results: Proxy ACE scores were positively correlated with all medical morbidity and healthcare utilization outcome variables at a significance level of P<0.01, with the strongest associations for number of medical diagnoses (r=0.30), number of prescribed medications (r=0.19), and number of emergency room visits (r=0.17). After accounting for outliers using Cook’s d and applying hierarchical multiple regression, Proxy ACE scores predicted an additional 9.2% of the variability in number of medical diagnoses after controlling for age, sex, race, and ethnicity (β=.31). 

Conclusion: Our findings are proof of concept that Proxy ACEs scores can be meaningfully derived from the EHR. Future directions include incorporating natural language processing methodology to account for unstructured data stored in the EHR and creating and validating an individual-level multivariable predictive model using our Proxy ACEs dataset.

Harnessing electronic health record data to examine the association between adverse childhood experiences, medical morbidity, and healthcare utilization

Background: Depression is a public health concern affecting the well-being and development of adolescents. Despite evidence-based guidelines promoting routine depression screening, many youths remain unidentified and untreated. Quality improvement (QI) initiatives may enhance depression screening practices. This retrospective chart review assesses a QI initiative's impact on depression screening rates among 12 to 18-year-old patients at four pediatric clinics in urban Nebraska from 2018 to 2022. The study evaluates trends in positive depression screens and the effectiveness of targeted interventions to increase adolescent depression screening. 
Methods: Electronic health record data was used to calculate adolescent depression screening rates and the percentage of positive depression screens. The screening rate equaled the total number of completed adolescent depression screens divided by the total number of well child exams for 12 through 18-year-old patients. The percentage of positive depression screens was the percentage of all adolescents who were screened and were positive on the Patient Health Questionnaire-4 (PHQ-4). The QI initiative comprised a “Lunch and Learn" session to raise provider awareness about the importance of depression screening. Additional interventions involved creating supportive materials for managing positive screenings, integrating depression screening into well-child exam rooming, and posting educational materials in the clinic's computer system for easy access by patients and providers. The impact of the initiative on depression screening rates was analyzed annually over five years. 
Results: The data revealed variations in depression screening rates and stability in the percentage of positive depression screens over the five-year study period. In 2018, the screening rate at well- child exams was 25.9%, with 8.1% of patients positive for depression. In 2019, the screening rate decreased to 21.6%, while the percentage of positive screenings increased to 13.1%. Following the initiation and continuation of the QI initiative in 2019, screening rates notably increased, reaching 37.2% with 11.7% positive patients in 2020. Screening rates further improved to 48.4% and 88.6% in 2021 and 2022, respectively, while the percentage of positive screenings remained relatively stable at 11.2% and 10.1%. 
Conclusion: The QI initiative positively impacted depression screening rates among adolescent patients at the four pediatric clinics, empowering healthcare providers to recognize and support adolescents facing depression. Continuous screening practice improvement efforts are crucial for early identification and intervention. The initiative offers a promising approach to enhance adolescent mental health care. 


Impact of Targeted Interventions on Depression Screening Rates among Adolescent Patients: A Retrospective Chart Review

Background
Pediatric hypertension is underrecognized, affecting 4% of adolescents in the USA with an additional 10% having elevated BP. Disparities exist among racial/ethnic groups. The global aim of this initiative is to improve recognition of elevated BP and hypertension and eliminate disparities in adolescents at the three NYU Langone Hospital Long Island primary care ambulatory sites.

Methods
Data between 2019-2021 was analyzed to determine prevalence and distribution of elevated BP and hypertension in patients >13 years by sex, race/ethnicity, and preferred language. Continuous variables were compared using Wilcoxon rank-sum test, and Chi-square/Fisher’s exact tests utilized for categorical variables. SAS v9.4 was used with statistical significance if p-value <0.05.

An interprofessional QI team was chartered to standardize BP measurement and improve recognition/management of adolescent hypertension. QI measures included percentage of appropriate documentation of elevated BP and hypertension in the care plan, timely primary care follow-up, and appropriate nephrology referral. Random sampling of 10-15 adolescents/month per site with elevated BP were reviewed. Data for 6 months was utilized for baseline QI measures. Our SMART aim is to increase recognition and appropriate follow-up by 25% from baseline across all sites in twelve months. Key driver diagram was constructed, and change ideas implemented using PDSA cycles. Interventions included multi-modal education and site visits to standardize BP measurement. BP reference and management algorithm handouts were distributed. Data were analyzed quarterly and plotted using run charts.

Results
Retrospective data from 2019-2021 (n=6158) showed 32% prevalence of elevated BP and hypertension in our cohort, 66.3% of which were male. Race/ethnicity distribution of adolescents with elevated BP and hypertension were 37.1% Black, 31.9% Hispanic, 31.7% White, and 27.7% Asian. Distribution by preferred language revealed 33.1% Spanish and 32.1% English (remainder other).

Appropriate documentation of elevated BP and hypertension in the care plan improved by 4.5 % (from 2% to 6.5%). Inter-site comparison revealed site-site variations, with the largest improvement of 11% at our Patient-Centered Medical Home. Twelve-month baseline for timely primary care follow-up and appropriate nephrology referral were 7% and 13%, respectively. 

Conclusion
The prevalence of elevated BP was higher in our cohort, potentially due to variations in BP measurement, and was targeted by our interventions. We achieved our SMART aim of increased recognition and documentation of adolescents with elevated BP and hypertension through standardization of BP measurement and provider education. Our next step is to target timely primary care follow-up and appropriate nephrology referral. An additional goal is to identify and reduce disparities across race, ethnicity, and preferred language for recognition and management of adolescent hypertension.

Measurement of Ambulatory Blood Pressure (BP) Standardization (MAPS) in Adolescents: A Quality Improvement (QI) Initiative

Background: Predicting the discharge destination, otherwise known as discharge disposition, can offer several advantages for the inpatient rehabilitation patient, including treatment and discharge planning. Machine learning (ML) models have demonstrated a strong predictive capacity for discharge disposition, particularly for patients with specific diagnoses such as traumatic brain injuries, or in specific settings such as acute neurologic care, or among specific populations. 



Objective: Explore various ML models’ ability to predict inpatient rehabilitation discharge disposition to home versus non-home using patient encounter data. 



Methods: Patient data included demographics, diagnoses, lab values, length of stay, and insurance status from all inpatient rehabilitation encounters at a large midwestern health system occurring between 2015 and 2022 (N=4919). We compared two commonly used ML approaches: a random forest (RF) classifier and a decision tree (DT) classifier. Two bagging classifiers, which aggregate the results of multiple RF or DT classifiers to improve predictions, were also tested. Because the imbalance in number between home (N=3687) and non-home discharges (N=1232) can impair predictions, balancing techniques were used prior to training the ML classifiers. The ML classifiers were then optimized with grid search and cross validation. Performance of the ML classifiers was determined by the receiver operating characteristic area under the curve (AUC) – a measure of the ability to discriminate the true discharge disposition, accuracy (computed using the Jaccard Index), and macro f1-score (indicative of classification ability across classes). 



Results: 

A bagging classifier using RF showed highest predictive performance in terms of AUC (0.68) and accuracy (0.75), however it struggled to predict non-home discharges (macro f1-score = 0.54). A bagging DT classifier improved prediction of non-home discharges (macro f1-score = 0.59), however AUC (0.65) and accuracy (0.69) were lower. Generally, bagging classifiers combining the predictions from multiple RF or DT classifiers improved predictive performance compared to single RF or DT classifiers. 



Conclusions: As artificial intelligence and ML continue to grow in healthcare applications, this study showcased the effectiveness of ML approaches to predict discharge disposition from inpatient rehabilitation. Our findings complement previous studies in other diagnoses and settings, due to our diversity of available variables and differences in our patient population and treatment paradigms. Of note, utilization of bagging classifiers proved valuable in enhancing the performance of the tested base classifiers. Our results may help guide optimization of ML approaches to predict discharge disposition, in hopes of improving patients' hospital courses and post-hospital planning and treatment.

Evaluation of Machine Learning Approaches for Predicting Inpatient Rehabilitation Patient Discharge Disposition

Background
Post Traumatic Stress Disorder (PTSD) is a disorder characterized by exposure to some traumatic event leaving the individual with a variety of symptoms that can result in debilitating disability. Treatment has historically been centered around Cognitive Processing Therapy (CPT) and literature has shown a significant risk of suicide in those left untreated. Thus, the importance of appropriate healthcare utilization is important for addressing morbidity and mortality of PTSD. Determining appropriate utilization is not well recorded in the literature though some studies show increased utilization with increased symptom severity. The aim of
this study is to replicate these findings and determine if there exists any range of symptom severity that over or under-utilizes the healthcare system.

Methods
Data was collected by Rosen et al. 2012 through Project VALOR, a longitudinal registry of PTSD in combat exposed veterans of Afghanistan and Iraqi military theaters. PTSD checklist (PCL) scores and data regarding number of healthcare utilizations for PTSD treatment year to year were recorded over a five-year period. Utilization of healthcare services for PTSD were stratified by this study into categories and compared against PCL scores using an ordinal logistical regression model to predict probability of being placed in a higher utilization category. Quadratic analysis was conducted on the PCL scores to determine linear relationship of findings.

Results
Ordinal logistical regression of mean centered PCL value was 1.042 (p=0.000). Quadratic effect of ordinal logistical regression of mean centered PCL2 was determined to be 0.99 (p=0.061).

Conclusion
Self-reported PCL scores are positively predictive of healthcare utilization. For every 1 increase in PCL score, there is a predicted 4.29% increased chance of being placed into the next highest utilization category (p=0.000). In addition, the relationship between PCL score and healthcare utilization was determined to be log-linear due to quadratic effect being found not significant 0.99 (p=0.061). This indicates no PCL score is correlated with increased probability of increased or decreased healthcare utilization. This likely represents healthcare services are appropriately being administered based on symptom severity and no group is over or under-utilizing resources. Ultimately, this study replicates previous literature that increased severity is associated with increased healthcare utilization while also quantifying the predictive value of increasing PCL scores. Currently the healthcare system is correctly identifying and capturing individuals who would in theory benefit from increased healthcare utilization. Additional research into capture strength would be beneficial in determining impact of hospital system capture rate of at risk patients.

Increasing PTSD Symptom Severity Predicts Increasing Healthcare Utilization Among OEF/OIF Veterans.

Obstructive sleep apnea (OSA) affects millions of U.S. Veterans and is linked to severe illnesses such as stroke and heart failure. However, understanding how OSA interacts with common comorbidities and the additive risk of OSA in developing severe illnesses is lacking. Large language models (LLMs) are promising tools for clinical natural language processing (NLP). These models are pre-trained on vast amounts of text data and learn to generate coherent and contextually appropriate text by predicting tokens in a sequence. This study explores the quality of LLM embeddings in clustering disease phenotypes related to OSA and its comorbidities such as atrial fibrillation, coronary artery disease, heart failure, hypertension, stroke, and type 2 diabetes mellitus. By identifying sub-cohorts of patients with similar diagnoses, procedures, medications, and adherence to continuous positive airway pressure (CPAP) treatment, we aim to score LLM embeddings by how well pre-trained LLMs discriminate OSA-related phenotypes. Document-level embeddings for discharge reports in the MIMIC-IV database were computed using NERSC's Perlmutter supercomputer. Twelve clinical LLMs of different sizes and pre-training schemes were evaluated including BioBART, BioBERT, BioGPT, BioMegatron, Gatortron, and RadBERT. To visualize the LLMs embedding, UMAP was used to enable plotting of embedded documents in 2D. Each set was clustered using K-Means with increasing values for K, and the purity of the clusters was measured using the Shannon entropy of cluster assignments. Finally, we test the performance of quantized document-level data for supervised phenotype classification.
Qualitatively, we examined the LLM embeddings along UMAP plots and reviewed K-Means cluster assignment by reading sampled text. Gatrotron_medium, BioGPT_large, and BioBERT_base were the best models at separating OSA from its comorbidities. Quantitatively, the clusters were measured on their Shannon entropy factor in which the lower value correlated to greater purity of clusters and vice versa. Gatortron_medium, Gatortron_base, and BioBART_base consistently scored with low entropy across comorbidities. The developed method selected the Gatortron_medium LLM as the best pre-trained model for OSA-related phenotypes both qualitatively and quantitatively. The method also has potential for assessing bias in LLMs by measuring how embeddings group patients according to social determinants of health such as gender and race. In the context of the Department of Veterans Affairs (VA) healthcare, there is a need for early OSA detection and prevention. Leveraging LLMs for feature discovery and risk factor identification, this research can provide valuable insights for allocating CPAP treatment effectively by grouping patients with similar health histories and OSA outcomes.

Downloads

Next from AMA Research Challenge 2023

Comparison of Relative Value Unit (RVU) reimbursement anterior or posterior cervical spinal fusion