Working…
COVID-19 is an emerging, rapidly evolving situation.
Get the latest public health information from CDC: https://www.coronavirus.gov.

Get the latest research information from NIH: https://www.nih.gov/coronavirus.
ClinicalTrials.gov
ClinicalTrials.gov Menu

Local Participatory Systems Dynamics to Increase Reach of Evidence Based Addiction and Mental Health Care

The safety and scientific validity of this study is the responsibility of the study sponsor and investigators. Listing a study does not mean it has been evaluated by the U.S. Federal Government. Read our disclaimer for details.
 
ClinicalTrials.gov Identifier: NCT04356274
Recruitment Status : Enrolling by invitation
First Posted : April 22, 2020
Last Update Posted : April 22, 2020
Sponsor:
Information provided by (Responsible Party):
Lindsey Zimmerman, Palo Alto Veterans Institute for Research

Study Description
Go to 
Top of Page Study Description Study Design Arms and Interventions Outcome Measures Eligibility Criteria Contacts and Locations More Information
Brief Summary:
The most common reasons Veterans seek VA addiction and mental health care is for help with opioid and alcohol misuse, depression and PTSD. Research evidence has established highly effective treatments that prevent relapse, overdose and suicide, but even with policy mandates, performance metrics, and electronic health records to fix the problem, these treatments may only reach 3-28% of patients. This study tests participatory business engineering methods (Participatory System Dynamics) that engage patients, providers and policy makers against the status quo approaches, such as data review, and will determine if participatory system dynamics works, why it works, and whether it can be applied in many health care settings to guarantee patient access to the highest quality care and better meet the addiction and mental health needs of Veterans and the U.S. population.

Condition or disease Intervention/treatment Phase
PTSD Depression Alcohol Use Disorder Opioid Use Disorder Other: Participatory System Dynamics (PSD) Other: Audit and Feedback (AF) Not Applicable

Detailed Description:
The broad aim is to empower all healthcare stakeholders to provide the highest quality care to all patients. The specific aims address the complexities and tradeoffs associated with implementing evidence-based practices (EBPs) in outpatient addiction and mental health systems. There is scientific consensus about the best evidence-based psychotherapies and pharmacotherapies (EBPs) to meet the needs of patients with opioid and alcohol use disorder, PTSD and depression. However, EBP coordination over time, within and across multidisciplinary teams of providers, is complex and constantly changing. Veterans Health Administration (VA) policy mandates, national training programs, and incentivized quality measures, have been insufficient for reaching more than 3 to 28% of patients with the highest quality treatments. In fact, limited EBP reach is common in health systems and the field of implementation science seeks to address it. One routine strategy is data auditing with provider feedback (audit-and-feedback; AF), however, the impact is highly variable. As an alternative, participatory system dynamics (PSD) has been used to explain causes of complex problems in business management for 60 years. Partnering with frontline staff using PSD to determine how EBP reach emerges from local resources and constraints, and is determined by system dynamics, such as delays and feedback. The dynamics of EBP reach were formally specified in differential equation models, and tested against VA data drawn from a national VA SQL database. Using existing enterprise data to tailor model parameters to each care team. PSD models were made accessible via a 'Modeling to Learn' interface and training, during which teams safely evaluated local change scenarios via simulation to find the highest yield options for meeting Veterans' needs. PSD learning simulations produce immediate, real-time feedback to the teams who coordinate care, improving day-to-day decisions and long-term improvement plans. The study design is a two-arm, 24-site (12 sites/arm) cluster randomized trial to test the effectiveness of PSD simulation as compared to more standard team AF data review. The hypothesis is that PSD will be superior to AF for improving EBP initiation and dose (Aim 1). The investigators will test the PSD theory of change that the effect of PSD on improved EBP reach is explained by improvement in team systems thinking (Aim 2). To confirm the potential for widespread usefulness of PSD, by also testing the generalizability of PSD causal dynamics across PSD and AF arms (Aim 3). This study has the potential to inform a new paradigm, by determining what works to improve health system quality defined as EBP reach, why it works, and under what conditions. If PSD is effective, study activities will address a national priority to improve Veterans' addiction and mental health care to prevent chronic symptoms, relapse, suicide and overdose. Findings from the proposed tests of effectiveness, causality, and generality, could also catalyze future applications to make a significant public health impact across the continuum of healthcare.
Study Design
Go to 
Top of Page Study Description Study Design Arms and Interventions Outcome Measures Eligibility Criteria Contacts and Locations More Information

Layout table for study information
Study Type : Interventional  (Clinical Trial)
Estimated Enrollment : 720 participants
Allocation: Randomized
Intervention Model: Parallel Assignment
Intervention Model Description:

Participatory System Dynamics: Participatory system dynamics is a facilitated health care quality improvement or evidence-based practice implementation strategy that includes frontline addiction and mental health staff running simulations of clinic improvement strategies to find the best approaches for improving the reach of evidence-based psychotherapy and evidence-based pharmacotherapy.

Audit and Feedback: Audit and feedback is a health care quality improvement or evidence-based practice implementation strategy that includes frontline addiction and mental health staff reviewing data to find the best approaches for improving the reach of evidence-based psychotherapy and evidence-based pharmacotherapy.

Anticipate that 720 frontline providers will participate across both arms of this trial. There will be no interaction with current patients for the purposes of research. No new data will be collected beyond data generated during routine care.
Masking: None (Open Label)
Primary Purpose: Health Services Research
Official Title: Participatory System Dynamics vs Audit and Feedback: A Cluster Randomized Trial of Mechanisms of Implementation Change to Expand Reach of Evidence-based Addiction and Mental Health Care
Actual Study Start Date : February 1, 2019
Estimated Primary Completion Date : August 31, 2024
Estimated Study Completion Date : August 31, 2024

Resource links provided by the National Library of Medicine


Arms and Interventions
Go to 
Top of Page Study Description Study Design Arms and Interventions Outcome Measures Eligibility Criteria Contacts and Locations More Information

Arm Intervention/treatment
Experimental: Participatory System Dynamics (PSD)
12 clinics assigned to PSD
 Other: Participatory System Dynamics (PSD)
Participatory system dynamics is a facilitated health care quality improvement or evidence-based practice implementation strategy that includes frontline addiction and mental health staff running simulations of clinic improvement strategies to find the best approaches for improving the reach of evidence-based psychotherapy and evidence-based pharmacotherapy.
Experimental: Audit and Feedback (AF)
12 clinics assigned to AF
 Other: Audit and Feedback (AF)
Audit and feedback is a health care quality improvement or evidence-based practice implementation strategy that includes frontline addiction and mental health staff reviewing clinical care team data to find the best approaches for improving the reach of evidence-based psychotherapy and evidence-based pharmacotherapy.


Outcome Measures
Go to 
Top of Page Study Description Study Design Arms and Interventions Outcome Measures Eligibility Criteria Contacts and Locations More Information

Primary Outcome Measures :
  1. Proportion of patients diagnosed with alcohol use disorder, depression, opioid use disorder, or PTSD who meet evidence-based psychotherapy and pharmacotherapy initiation and course measures divided by total number of patients with these diagnoses [ Time Frame: Pre-/Post- 12-month period average of evidence-based practice reach (24 months total observation) ]
    Initiation of an evidence-based practice is indicated by an evidence-based psychotherapy template or evidence-based pharmacotherapy prescription after intake. Adequate course is based on receiving an adequate number of evidence-based psychotherapy sessions to be a "completer" (typically 8 sessions) or enough refills for a guideline-recommended adequate trial of each medication (varies by medication). Data is gathered based on electronic health record data from the VA Corporate Data Warehouse (CDW).

  2. Proportion of completed evidence-based practice templates during sessions with a relevant CPT code [ Time Frame: Pre-/Post- 12-month period average of evidence-based practice reach (24 months total observation) ]
    We will study 5 evidence-based psychotherapies: 3 for depression (Cognitive Behavior Therapy (CBT-D), Acceptance and Commitment Therapy (ACT), and Interpersonal Psychotherapy (IPT)) and 2 for PTSD (Prolonged Exposure (PE) and Cognitive Processing Therapy (CPT)). Data is gathered based on electronic health record data from the Corporate Data Warehouse (CDW).

  3. Proportion of combination of prescriptions placed with the VA pharmacy and sessions with a relevant CPT code [ Time Frame: Pre-/Post- 12-month period average of evidence-based practice reach (24 months total observation) ]
    We will study 8 evidence-based pharmacotherapies: 2 for depression (84 and 180 days therapeutic continuity at new antidepressant start), 2 for Opioid Use Disorder (OUD) (methadone and buprenorphine), and 4 for Alcohol Use Disorder (AUD) (Acamprosate, Disulfiram, Naltrexone, and Topiramate). Data is gathered based on electronic health record data from the Corporate Data Warehouse (CDW).


Secondary Outcome Measures :
  1. Degree of acceptability of intervention assessed by the Acceptability of Intervention Measure (AIM) [followed by its scale information in the Description] [ Time Frame: At 6 months ]

    Assesses degree of differences in team perceptions of PSD and AF on a survey with 4 items.

    Scale: 1-5, in 1 point increments (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree)


  2. Degree of appropriateness of intervention assessed by the Intervention Appropriateness Measure (IAM) [followed by its scale information in the Description] [ Time Frame: At 6 months ]

    Assesses degree of for differences in team perceptions of PSD and AF on a survey with 4 items.

    Scale: 1-5, in 1 point increments (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree)


  3. Degree of feasibility of intervention assessed by the Feasibility of Intervention Measure (FIM) [followed by its scale information in the Description] [ Time Frame: At 6 months ]

    Assesses degree of differences in team perceptions of PSD and AF on a survey with 4 items.

    Scale: 1-5, in 1 point increments (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree)


  4. Patient Aligned Care team Burnout Measure (PACT) [followed by its scale information in the Description] [ Time Frame: At baseline and 6 months ]

    Quality of work satisfaction and burnout in a 4-item descriptive survey with measures from VA team-based primary care that tracks 1) years of experience with the team, 2) working on more than one team, 3) turnover/change in team staff, 4) team overwork, and the single-item 5) self-reported burnout (sensitivity 83.2% and specificity 87.4%)

    (Question 1) Answered in # of years (Question 2-3) Yes or No (Question 4-5) Scale: 1-5, in 1 point increments (1 = Never, 2 = Rarely, 3 = Sometimes, 4 = Very Often, 5 = Always)


  5. Learning Organization Survey (LOS-27) [followed by its scale information in the Description] [ Time Frame: At baseline and 6 months ]

    Psychological safety in the workplace using a 27-item survey developed out of the learning organization tradition and demonstrated to have good psychometric properties during VA validation which assesses 7 clinic context factors: a) supportive learning environment (including psychological safety), b) leadership that reinforces learning, c) experimentation, d) training, e) knowledge acquisition, f) time for reflection, and g) performance monitoring

    (Questions 1-23) Scale: 0-4, in 1 point increments (0 = Never, 4 = Always) (Questions 24-27) Scale: 0 to 7, in 1 point increments (0 = Highly Inaccurate, 7 = Highly Accurate)


  6. Team Decision Making Questionnaire (TDMQ) [followed by its scale information in the Description] [ Time Frame: At 6 months ]

    Team dynamics in the workplace using a four factor scale survey validated to assess the impact of a team intervention on team decision-making, support learning and development of quality services

    Scale: 1-7, in 1 point increments incl N/A (N/A = Not Applicable, 1 = Not at all, 2 = To a very small extent, 3 = To a small extent, 4 = To a moderate extent, 5 = To a great extent, 6 = To a very great extent, 7 = To a vast extent)


  7. Systems Thinking Scale (STS) [followed by its scale information in the Description] [ Time Frame: At baseline and 6 months ]

    Use of systems thinking in the work place and the ability to recognize, understand, and synthesize interactions and interdependencies, including how actions and components can reinforce or counteract each other.

    Scale: 1-5, in 1 point increments (1 = Never, 2 = Seldom, 3 = Some of the time, 4 = Often, 5 = Most of the time)


  8. Systems Thinking Codebook and Session Observations [followed by its scale information in the Description] [ Time Frame: Over 6 months ]

    Observation of systems thinking in language/explanations, and performance demonstrating system thinking skills (competence) measured on four constructs: Complex, Feedback, Behavior, Time

    Scale: Level 1-4 in 1 point increments (1 = Construct is demonstrated at most simple level, 4 = Construct is fully demonstrated at most complex level)


  9. Facilitator Fidelity to Intervention Guides and Theory of Change [ Time Frame: Over 6 months ]
    Review fidelity with qualitative checks against AF/PSD facilitator scripts for session learning objectives, 'key idea' and 'definitions,' including tracking the proportion of AF/PSD session activities (in minutes) on these components.


Eligibility Criteria
Go to 
Top of Page Study Description Study Design Arms and Interventions Outcome Measures Eligibility Criteria Contacts and Locations More Information

Information from the National Library of Medicine

Choosing to participate in a study is an important personal decision. Talk with your doctor and family members or friends about deciding to join a study. To learn more about this study, you or your doctor may contact the study research staff using the contacts provided below. For general information, Learn About Clinical Studies.


Layout table for eligibility information
Ages Eligible for Study:   18 Years and older   (Adult, Older Adult)
Sexes Eligible for Study:   All
Accepts Healthy Volunteers:   Yes
Criteria

Inclusion Criteria Clinics:

  • VA divisions and community-based outpatient clinics (CBOCs) or 'clinics' from regional VA health systems
  • Must be below the overall VA quality median (as assessed by the Strategic Analytics for Improvement and Learning or SAIL), which includes 3 of 8 SAIL measures associated with four evidence-based psychotherapies and three evidence-based pharmacotherapies for depression, PTSD, and opioid use disorder.

Exclusion Criteria Clinics:

  • clinics with less than 12 months of data in 2018
  • clinics already involved in Office of Veterans Access to Care (OVACS) quality improvement program at baseline.
  • clinics where the VA Cerner electronic health record (EHR) implementation rollout will occur during the project period (Veterans Integrated Services Networks (VISNs) 20, 21 ,22, and 7)
  • clinics who serve less than 122 unique patients each month on average
  • clinics without an onsite multidisciplinary team of mental health or addiction service providers (minimum required: 1 psychiatrist, 1 psychologist, 1 social worker onsite)
Contacts and Locations
Go to 
Top of Page Study Description Study Design Arms and Interventions Outcome Measures Eligibility Criteria Contacts and Locations More Information

Information from the National Library of Medicine

To learn more about this study, you or your doctor may contact the study research staff using the contact information provided by the sponsor.

Please refer to this study by its ClinicalTrials.gov identifier (NCT number): NCT04356274


Locations
Layout table for location information
United States, California
VA Palo Alto Health Care System
Palo Alto, California, United States, 94304
Sponsors and Collaborators
Palo Alto Veterans Institute for Research
Investigators
Layout table for investigator information
Principal Investigator: Lindsey E Zimmerman, PhD National Center for PTSD, Dissemination & Training Division
  Study Documents (Full-Text)

Documents provided by Lindsey Zimmerman, Palo Alto Veterans Institute for Research:
Study Protocol and Statistical Analysis Plan  [PDF] November 30, 2019

More Information
Go to 
Top of Page Study Description Study Design Arms and Interventions Outcome Measures Eligibility Criteria Contacts and Locations More Information
Publications:
Ruzek JI, Karlin BE, & Zeiss AM. Implementation of Evidence-Based Psychological Treatments in the Veterans Health Administration. In: McHugh RK, Barlow DH, eds. Dissemination of evidence-based psychological treatments. New York, NY: Oxford University Press. , 2012.
Department of Defense, & Department of Veterans Affairs. The management of MDD Working Group. VA/DOD clinical practice guideline for management of major depressive disorder (MDD). , 2009.
Department of Veterans Affairs, & Department of Defense. VA/DoD Clinical practice guideline for the management of post-traumatic stress. , 2010.
Department of Veterans Affairs, & Department of Defense. VA/DoD Clinical practice guideline for the management of substance use disorders. , 2009.
Flottorp SA, Jamtvedt G, Gibis B, et al. Using audit and feedback to health professionals to improve the quality and safety of health care. World Health Organ. 2010.
Diehl E, & Sterman JD. Effects of feedback complexity on dynamic decision making. Organ Behav Hum Decis Process. 1995;62:198-215.
Sterman JD. Modeling managerial behavior: Misperceptions of feedback in a dynamic decision making experiment. Manag Sci. 1989;35:321-339.
Cronin MA, Gonzalez C, & Sterman JD. Why don't well-educated adults understand accumulation? A challenge to researchers, educators, and citizens. Organ Behav Hum Decis Process. 2009;108:116-130.
Sterman JD. Learning in and about complex systems. Syst Dyn Rev. 1994;10:291-330.
Andersen DF, Vennix JA, Richardson GP, et al. Group model building: Problem structing, policy simulation and decision support. J Oper Res Soc. 2007;:691-694.
Vennix JAM. Group model building: facilitating team learning using system dynamics. Chichester ; New York: J. Wiley, 1996: 1-297.
Rouwette EAJA, Vennix JAM, & Mullekom T van. Group model building effectiveness: A review of assessment studies. Syst Dyn Rev. 2002;18:5-45.
Bendoly E. System dynamics understanding in projects: Information sharing, psychological safety, and performance effects. Prod Oper Manag. 2014;23:1352-1369.
Sterman JD. Does formal system dynamics training improve people's understanding of accumulation? Syst Dyn Rev. 2010;26:316-334.
Simon HA. Bounded rationality and organizational learning. Organ Sci. 1991;2:125-134.
Rahmandad H, Repenning N, & Sterman J. Effects of feedback delay on learning. Syst Dyn Rev. 2009;25:309-338.
Oliva R. Model calibration as a testing strategy for system dynamics models. Eur J Oper Res. 2003;151:552-568.
Barlas Y. Formal aspects of model validity and validation in system dynamics. Syst Dyn Rev. 1996;12:183-210.
Forrester, J.W. The model versus a modeling process. Syst Dyn Rev. 1985;:133-134.
Edmondson A. Psychological safety and learning behavior in work teams. Adm Sci Q. 1999;44:350-383.
Campbell MJ, & Walters SJ. How to Design, Analyise and Report Cluster Randomised Trials in Medicine and Health Related Research. : Wiley, 2014: 1-264.
Forrester JW. System dynamics, systems thinking, and soft OR. Syst Dyn Rev. 1994;10:245-256.
Forrester JW. Some basic concepts in system dynamics. Sloan Sch Manag Mass Inst Technol Camb MA. 2009.
Hong G. Ratio of mediator probability weighting for estimating natural direct and indirect effects. In: Proceedings of the American Statistical Association, Biometrics Section. : American Statistical Association Alexandria, VA, 2010: 2401-2415.
Hong G, Deutsch J, & Hill HD. Ratio-of-Mediator-Probability Weighting for Causal Mediation Analysis in the Presence of Treatment-by-Mediator Interaction. J Educ Behav Stat. 2015;40:307-340.
Qin X, & Hong G. Causal mediation analysis in multi-site trials: An application of ratio-of-mediatorprobability weighting to the Head Start Impact Study. JSM Proc Soc Stat Sect. 2014;:912-926.
Homer JB. Partial-model testing as a validation tool for system dynamics (1983). Syst Dyn Rev. 2012;28:281-294.
Rahmandad H, & Sterman JD. Reporting guidelines for simulation-based research in social sciences. Syst Dyn Rev. 2012;28:396-411.
Sweeney LB, & Sterman JD. Bathtub dynamics: Initial results of a systems thinking inventory. Syst Dyn Rev. 2000;16:249-286.
Centers for Medicare & Medicaid Services Alliance to Modernize Healthcare (CAMH). Independent Assessment of the Health Care Delivery Systems and Management Processes of the Department of Veterans Affairs (Volume 1: Integrated Report).
Department of Veterans Affairs. Uniform mental health services in VA medical centers and clinics. Washington DC: Veterans Health Administration, 2008.
Brownson RC, Colditz GA, & Proctor EK. Dissemination and Implementation Research in Health: Translating Science to Practice. Oxford, UK: Oxford, 2012.
Tabak RG, Khoong EC, Chambers D, et al. Models in dissemination and implementation research: useful tools in public health services and systems research. Front Public Health Serv Syst Res. 2013;2:8.
Damschroder LJ, & Lowery JC. Efficient Synthesis: Using Qualitative Comparative Analysis (QCA) and the CFIR across Diverse Studies. 2015.
Baumgartner M, & Thiem A. Often Trusted but Never (Properly) Tested: Evaluating Qualitative Comparative Analysis. Sociol Methods Res. 2017;:4912411770148.
Hug S. Qualitative Comparative Analysis: How Inductive Use and Measurement Error Lead to Problematic Inference. Polit Anal. 2013;21:252-265.
Krogslund C, Choi DD, & Poertner M. Fuzzy Sets on Shaky Ground: Parameter Sensitivity and Confirmation Bias in fsQCA. Polit Anal. 2015;23:21-41.
Braumoeller B. QCAfalsePositive: Tests for Type I Error in Qualitative Comparative Analysis (QCA). 2015.
Senge P. The Fifth Discipline. : Currency/Doubleday, 1990.
Richardson GP. Concept models in group model building: G. P. Richardson: Concept Models in Group Model Building. Syst Dyn Rev. 2013;29:42-55.
Richardson GP. Reflections on the foundations of system dynamics: Foundations of System Dynamics. Syst Dyn Rev. 2011;27:219-243.
Garcia HA, McGeary CA, Finley EP, et al. Burnout among psychiatrists in the Veterans Health Administration. Burn Res. 2015;2:108-114.
Garcia HA, Kelley LP, Rentz TO, et al. Pretreatment predictors of dropout from cognitive behavioral therapy for PTSD in Iraq and Afghanistan war veterans. Psychol Serv. 2011;8:1-11.
Senge, P.M. The fifth discipline: The art and practice of the learning organization. : Broadway Business, 2006.
Meadows DH. Thinking in systems: A primer. : Chelsea Green Publishing, 2012.
Senge PM, ed. The dance of change: the challenges of sustaining momentum in learning organizations, 1st ed. New York: Currency/Doubleday, 1999: 1-596.
Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. : McGraw-Hill Education, 2000: 1-1008.
Rouwette EA, & Vennix JA. System dynamics and organizational interventions. Syst Res Behav Sci. 2006;23:451-466.
Morecroft J, & Sherman J. Modeling for learning organizations. Portland OR: Productivity Press, 1994.
Argyris C, & Schön DA. Organizational learning: A theory of action perspective. Reis. 1997;:345-348.
Forrester JW. Industrial Dynamics. Cambridge, MA: MIT Press, 1961.
Simon HA. Models of Man: Social and Rational-Mathematical Essays on Rational Human Behavior in a Social Setting. Oxford, UK: Wiley, 1957.
Huz S, Andersen DF, Richardson GP, et al. A framework for evaluating systems thinking interventions: an experimental approach to mental health system change. Syst Dyn Rev. 1997;13:149-169.
Moore SM, Dolansky MA, Singh M, et al. The Systems Thinking Scale. 2010.
Hovmand PS. Community Based System Dynamics. New York, NY: Springer New York, 2014.
Minkler M, & Wallerstein N. Community-Based Participatory Research for Health: From Process to Outcomes. : John Wiley & Sons, 2011: 1-758.
Senge P, & Forrester JW. Tests for building confidence in system dynamics models. 1980;14:209-228.
Rahmandad H, & Sterman JD. Reporting guidelines for simulation-based research in social sciences: Reporting Guidelines for Simulation-Based Research. Syst Dyn Rev. 2012;28:396-411.
Hovmand PS, Andersen DF, Rouwette E, et al. Group model-building "scripts" as a collaborative planning tool: Scripts as a collaborative planning tool. Syst Res Behav Sci. 2012;29:179-193.
Andersen D, & Richardson G. Scripts for group model building. Syst Dyn Rev. 1997;13:107-129.
Hovmand, Peter, S., et al. Scriptapedia 4.0.6. 2013.
Gallaher E, Steensma DP, Chrisope TR, et al. Individualized Medicine and Biophysical System Dynamics: An Example from Clinical Practice in End Stage Renal Disease. 2011.
Homer J. Levels of evidence in system dynamics modeling. Syst Dyn Rev. 2014;30:75-80.
Oliva R, & Sterman JD. Death spirals and virtuous cycles. In: Handbook of Service Science. : Springer, 2010: 321-358.
Snow G. Package "blockrand." 2015.
Rosseel Y, Oberski D, Byrnes J, et al. Package "lavaan" Version 0.5-23.1097. 2017.
Shavelson RJ, & Webb NM. Generalizability theory: A primer. Newbury Park, CA: Sage., 1991.
Wolak M. Package "ICC" Version 2.3.0. 2015.
Bates D, Maechler M, Bolker B, et al. Package "lme4" Version 1.1-13. 2017.
Rotondi MA. Package "CRTSize" Version 1.0. 2015.
Rotondi MA, & Donner A. Sample Size Estimation in Cluster Randomized Educational Trials: An Empirical Bayes Approach. J Educ Behav Stat. 2009;34:229-237.
Ventana Systems Inc. Vensim@ Version 6.3. 2014.
Hyndman, R.J., & Khandakar, Y. Automatic time series forecasting: The forecast package for R. J Stat Softw. 2008;26.
Hyndman R, O'Hara-Wild Mi, Bergmeir C, et al. Package "forecast" Version 8.0. 2017.
Ogrinc GS, & Headrick L. Fundamentals of health care improvement: A guide to improving your patients' care. : Joint Commission Resources, 2008.

Layout table for additonal information
Responsible Party: Lindsey Zimmerman, Principal Investigator, Palo Alto Veterans Institute for Research
ClinicalTrials.gov Identifier: NCT04356274    
Other Study ID Numbers: ZIM_0002
First Posted: April 22, 2020    Key Record Dates
Last Update Posted: April 22, 2020
Last Verified: April 2020
Individual Participant Data (IPD) Sharing Statement:
Plan to Share IPD: No

Layout table for additional information
Studies a U.S. FDA-regulated Drug Product: No
Studies a U.S. FDA-regulated Device Product: No
Keywords provided by Lindsey Zimmerman, Palo Alto Veterans Institute for Research:
implementation science
quality improvement
participatory system dynamics
audit and feedback
 evidence-based psychotherapy
evidence-based pharmacotherapy
addiction
mental health/behavioral health
Additional relevant MeSH terms:
Layout table for MeSH terms
Disease
Alcoholism
Behavior, Addictive
Pathologic Processes
Compulsive Behavior
 Impulsive Behavior
Alcohol-Related Disorders
Substance-Related Disorders
Chemically-Induced Disorders
Mental Disorders