Information sources and literature search

We used three main strategies to identify outcome studies of intervention programs for women with substance abuse issues and their children: online bibliographic database searches, checking printed sources, and requests to researchers (cf., [22]). First, we searched relevant bibliographic databases (PsycINFO, MedLine, PubMed, Web of Science, EMBASE, Proquest Dissertations, Sociological Abstracts, and CINAHL) for studies published in English from 1990 to May 2011, using a subject heading and keyword search for the terms "substance abuse (or substance use or addict* or alcohol*) and intervention (or treatment or therapeutic or rehab*) and women (or mother) and child (or infant) and mental health and prenatal (or parent*), singly and in combination. Secondly, we examined reference lists of retrieved articles for potentially relevant documents. In addition, we manually searched relevant journals in the area (Addiction, Addictive Behaviours, International Journal of the Addictions, Journal of Drug Issues, Journal of Psychoactive Drugs, Journal of Substance Abuse, Journal of Substance Abuse Treatment, Journal of Substance Use, and Substance Use and Misuse). Documents that appeared to be relevant on the basis of titles or abstracts were retrieved. Finally, we searched for grey literature (technical reports, clinical trials registry, unpublished data). All researchers identified through these searches, as well as researchers presenting at relevant conferences identified using Google and Cross Currents (Upcoming Events), were contacted by email to request any relevant published or unpublished data. Of the 200 researchers identified and emailed, 48% responded and 28 additional studies were identified.

Eligibility criteria and study selection

Data extraction

Upon completion of the literature search, we developed a codebook based on theoretical treatment models, literature review, and data availability. We collected data on dependent variable characteristics (type of outcome measure, type of data), and outcome statistics (e.g., F value, p value) and coded study context (author, document date, type of document, country), methodology (sample size, attrition, study design), participant characteristics (age, marital status, education, employment, income, substance abuse history, previous substance abuse treatment, mental and physical health, involvement with the legal system), child characteristics (age, custody, involvement with child protection services, positive toxicology at birth), and treatment program characteristics (population served, planned length of treatment, intensity of treatment, location, services). Project staff and investigators pilot tested the codebook, which we revised based on consensus before formally coding the studies. In a coding policy manual, we recorded variables that were added or deleted and decisions regarding clarification of specific variables.

A trained research assistant (AS) coded each study and met frequently with the principal investigator (KM) during the development of the codebook and early stage of coding. Both AS and KM coded 20% of studies. We calculated Cohen's Kappa and percent agreement for all variables. There was 100% agreement for identification of dependent variables and, for client and program variables, 94% mean agreement for continuous variables and a Kappa of 0.97 for categorical variables. We resolved discrepancies by consensus.

There were considerable missing data (especially on client characteristics and program services) and limited quantitative data on outcomes (e.g., standard deviations, sample sizes). In an attempt to obtain missing data, we contacted 89 researchers up to three times each. Our attempts to contact researchers occurred throughout the coding process up until data analyses were completed. In total, 79% responded, with 37% providing some additional data (Additional file 1).

Study quality

To assess the quality of randomized trials, we used the Jadad Scale [23], which is widely used in the medical literature. On the Jadad Scale, studies are rated on a scale from 0 to 5, with the highest possible score (5) given for those with descriptions of the randomization process, an appropriate method of randomization, double-blinding, an appropriate method of double-blinding, and withdrawal and dropouts. To assess the quality of non-randomized studies, we used the Newcastle-Ottawa Scale (NOS; [24]). On the NOS, studies are rated on a scale from 0 to 9 on the basis of three main issues: study group selection, group comparability, and outcome ascertainment. NOS content validity and inter-rater reliability have been established and further evaluation is being conducted [24]. A trained research assistant (AS) and Master's student (JL) coded study quality. Inter-rater reliability, based on 16% (19) of the 121 eligible studies, was high, Kappa = 0.81. We resolved discrepancies by consensus.

Calculating effect sizes

In order to facilitate our summary and comparison of studies, effect sizes were calculated, where possible. We transformed results from each study to the standardized mean difference (Cohen's d) and used conventional definitions of effect size (d), i.e., small = 0.20 or less (i.e., one fifth or less of one standard deviation difference); medium = 0.50; large = 0.80 (i.e., four fifths or more of one standard deviation difference) [25].

Program characteristics as moderators

We reviewed studies that examined factors that may have moderated the effect of treatment on outcomes. Specifically, we examined parenting effect sizes of studies in relation to program characteristics (e.g., residential or not, types of program services provided, targeted substance, whether or not children reside), as these potential moderating factors have been examined in previous studies [26]. Also, we reviewed studies comparing two types of integrated programs to examine which specific integrated program characteristics are associated with better parenting outcomes than others.

Study selection

In total, 329 studies were retrieved and coded for eligibility. Using the eligibility criteria, we excluded 207 and considered 122 studies eligible for inclusion in the larger systematic review. Two randomized trials were not included in our review because their samples included men [27, 28] and two other randomized trials were not included because they did not include addiction treatment [29, 30]. Based on a random sample of 20% of the studies, inter-rater reliability for eligibility coding was high, Kappa = 0.81. We resolved discrepancies by consensus. We estimated the completeness of the search using the capture re-capture method [31]. Based on this method, the estimated number of missing articles is eight (95% confidence interval [CI]: 2, 24), which suggests a 90% capture rate (i.e., the identified studies cover 90% of the search horizon). This reasonably high capture rate suggests that we retrieved a sufficient number of studies to avoid bias in the results of the systematic review. Of the 122 eligible studies, 91 studies did not have quantitative data on parenting outcomes. Of the 31 studies with parenting data, 24 were cohort studies and 3 were quasi-experimental studies [32–34]. Therefore, for the present review, we included four randomized trials [35–39]. See Figure 1 for a flow diagram.

Study characteristics

Studies varied in terms of assessment times and parenting measures. Parenting outcomes were assessed at varying time points (e.g., at prenatal intake, intake, 12 months postpartum, discharge, 6 weeks after discharge, 6 months after discharge). Three studies included measures of parenting skills (Parent-child Relationship Inventory, Parental Acceptance Rejection Questionnaire, Working Model of the Child Interview, Parent Development Interview, Nursing Child Assessment Satellite Training) and one study examined parenting capacity (child protection services involvement).

One study [35] involved pregnant women and the other three studies involved mothers with children (an average of 2 or 3). The average age was 29-36 years. Most women had experienced trauma, had mental health problems, and were unemployed, single mothers. Client race varied among studies, reflecting the geographic setting. Authors provided little specific information on the children, who were of a wide age range (infants to adolescents). Programs were 3-12 months and had a high dropout rate.

Study quality

Jadad Scale scores for two of the randomized trials were 3, which is a moderate score [36, 37]. Both studies were described as randomized but were not double blind as participants were aware of the treatment allocation. They both provided descriptions of an appropriate method of randomization and withdrawal and dropouts. The Jadad Scale score for one other randomized trial was 2, which is a low score [38, 39]. The study was described as randomized and provided a description of withdrawal and dropouts but was not double blind and did not provide a description of an appropriate method of randomization. The Jadad Scale score for another randomized trial was 1, which is a low score [35]. The study was described as randomized but was not double blind and did not provide descriptions of an appropriate method of randomization and withdrawal and dropouts.

Are integrated programs more effective than addiction treatment-as-usual in improving parenting outcomes?

As part of the Washington State MOMS project, Huber [35] randomly assigned pregnant women with substance abuse issues to an integrated residential program, an integrated outpatient program, or a standard outpatient program. The integrated programs included prenatal care, maternal health care, parenting education and support, and children's services. Huber [35] found no group differences in the percentage of clients involved with child protection services. Child protection services involvement appeared to increase for all groups from intake in the prenatal period to one year postpartum, but did not report p values.

Luthar and Suchman [36] randomly assigned mothers (of children under 16 years old) at three methadone clinics in New Haven, Connecticut, to standard treatment or standard treatment plus a relational psychotherapy mothers' group. Standard treatment included addiction counseling, pharmacological intervention (methodone), case management to assist with basic needs such as housing, welfare benefits, and legal aid. Maltreatment risk was assessed by maternal report on the Parental Acceptance Rejections Questionnaire and parenting skills were assessed using the Parent-child Relationship Inventory. At the end of the 6-month treatment, mothers in integrated treatment had significantly more improved affective interaction scores than mothers in standard treatment and there was a trend toward more decreased maltreatment risk scores and more improvement in parenting satisfaction scores for mothers in integrated treatment than for mothers in standard treatment. At 6-month follow-up, group differences were not significant but there was a trend toward more decreased maltreatment risk scores and more improvement in affective interaction scores for mothers in integrated treatment than for mothers in standard treatment. Limitations of this study include the small sample size, unknown variability in the standard treatment, changes to the integrated program over the course of the study, and dosage differences between the two groups.

With another sample, Luthar et al. [37] randomly assigned mothers (of children under 16 years old) at three methadone clinics in New Haven, Connecticut, to standard treatment plus recovery training or standard treatment plus a relational psychotherapy mothers' group. Maltreatment risk was assessed by maternal report on the Parental Acceptance Rejections Questionnaire and parenting skills were assessed using the Parent-child Relationship Inventory. At the end of the 6-month treatment and at 6-month follow-up, mothers in integrated treatment had more decreased maltreatment risk scores and more improvements in affective interaction and parenting satisfaction, but group differences were not significant.

Are some integrated program characteristics associated with better parenting outcomes than others?

Examination of parenting effect sizes (where available) among the 31 studies with parenting outcome data suggested that residential programs appeared to have larger effects than outpatient programs and programs with a maternal mental health service appeared to have larger effects than programs that did not offer a maternal mental health service. Only two cohort studies and one randomized trial specifically examined factors associated with parenting outcomes. Kern et al. [40] examined correlations between changes in various domains of parenting stress over the course of treatment and reduction in depressive symptoms. Findings indicated that reduction in depressive symptoms was significantly correlated with improvements in parenting competence, isolation, attachment, and role restriction. Knight and Wallace [41] found that when children resided in the treatment facility, mothers were five times more likely to have custody of their children at the end of treatment.

In a study comparing two integrated programs, Suchman and colleagues [38, 39] randomly assigned mothers (of children under 3 years old) in outpatient substance abuse treatment to the Mothers and Toddlers Program (MTP; an attachment-based parenting intervention) or the Parent Education Program (PE; case management and child guidance pamphlets). Quality of mental representations of parenting was assessed using the Working Model of the Child Interview, caregiving behavior was assessed using the Nursing Child Assessment Satellite Training, and maternal reflective functioning was assessed using the Parent Development Interview. As can be seen in Table 1, d s ranged from -0.22 to 0.70 and most indicated greater improvements in scores for attachment-based parenting intervention than parent education, but this advantage was typically small. At the end of the 3-month treatment, mothers in the MPT had significantly more improved scores for caregiving behavior and reflective functioning and a trend for more improved sensitivity score than mothers in the PE group. At 6-week follow-up, there were no significant group differences in improvements in scores.

Implications

The findings of this systematic review are consistent with those reported in previous reviews of substance abuse treatment for women [20, 21], meta-analyses of integrated programs showing their positive impact on maternal mental health and birth outcomes [42, 43], qualitative studies in which women stated that integrated programs helped them gain insight into intergenerational influences on parenting, how to strengthen emotional bonds with children, and use positive discipline techniques [44], studies of parent interventions with parents (mothers and fathers) in methadone maintenance treatment [27, 28], mothers in drug court [29], and other at-risk populations [16]. Results from this systematic review are important given the risks for poor outcomes in children of women with substance abuse issues [45]. The findings suggest that the risks to parenting could be minimized with intervention, which could have long-term impact. For example, integrated programs may improve parenting, which has been shown to reduce the risk of child maltreatment [18]. Even though the advantage of integrated programs over addiction treatment-as-usual may be small, it could have a potentially large impact on the associated financial and human burden in this vulnerable population (e.g., it may reduce the need for foster care placement, child treatment, psychiatric admissions, crime, etc.).

Limitations

There were a number of challenges encountered in completing this systematic review that highlight current limitations in research on integrated treatment programs. First, among the 122 studies examining outcomes of integrated programs there were only 31 with data on parenting outcomes, despite the fact that improving parenting is often a stated goal of integrated programs. Among the studies reporting parenting outcome data, few were comparison group studies. While not included in the present review, 24 cohort studies assessing parenting outcomes were identified in the literature search. This type of study design provides information about parenting outcomes for women in integrated treatment but does not provide a comparison group that enables one to determine if these outcomes are significantly better than those for women who participated in other types of treatment. Despite the limited number of studies included in the systematic review, we are confident that the search was not biased. We used several approaches to mitigate potential bias, including our attempts to identify grey literature by searching databases that include unpublished studies and contacting researchers for unpublished data as well as our use of the capture re-capture method to estimate the completeness of the literature search (identified studies covered 90% of the search horizon, suggesting that a sufficient number of studies were retrieved to avoid bias).

A second limitation of the present systematic review is that study quality was not high, as is typical of the substance abuse treatment field generally [26]. Studies included in the systematic review were of low to moderate quality, although it was unclear if the scores reflected study design per se or the reporting of study quality elements.

A third limitation is that studies had small samples and relatively few parenting outcome measures. The randomized trials comparing integrated programs to addiction treatment-as-usual did not involve observational measures of parenting, which may be more objective and valid than self-report measures. Also, these studies did not involve measures of some important areas of maternal functioning that can be impacted by substance abuse, such as maternal responsiveness, sensitivity, and reflective functioning, nor did they involve longitudinal follow-up on parenting or an assessment of cost effectiveness.

Fourthly, missing data limited our review. Information on research methods and data needed to calculate effect sizes precluded meta-analysis and hampered attempts at assessing study quality. Often program, client, and study characteristics that might moderate treatment outcomes was not available. Moderator analyses can have important practice implications, however, specific recommendations (e.g., regarding specific intervention strategies or specific subpopulations to target such as mothers of children in or out of foster care or mothers of younger or older children) await further research with better reporting to allow meta-analysis of variables that impact outcomes (c.f., [26]).

Recommendations for future research

More high quality studies comparing integrated programs to addiction treatment-as-usual are needed, especially studies of programs that target parent-child interaction with mothers of young children examining a variety of parenting outcomes. A multi-site study could address statistical limitations inherent to small samples. The most rigorous design would be a randomized trial, but this may be challenging in the world of real-life service provision. Further, the examination of moderators is critical, given the variability in clients served and services offered. Also, examination of moderators would help identify effective components of intervention and ultimately examine what works best for whom under what circumstances. As some effects may not be immediately evident, follow-up for at least two years (or, ideally, longer) would be advisable. Linear regression, generalized estimating equations, or linear growth curve modelling could be used to analyze parenting outcomes with group and other variables as predictors (e.g., maternal and child characteristics, program components), as well as the impact of mediators and moderators over time [46]. The propensity score method could be used to address the potential problem of baseline differences between groups [47]. Evaluation of parenting outcomes could be improved through the use of observational measures, with videotaped observations coded by research assistants blind to group status. Ensuring the availability of essential information to assess study quality and describe studies in future reviews could be accomplished by improvements in the editorial review process and creation of a registry of funded studies that would require submission of standard information (such as the Cochrane Collaboration on health care intervention), as has been recommended previously (e.g., [26]).