Measurement issues in longitudinal studies of mental health problems in children with neurodevelopmental disorders

Abstract

Purpose

To develop and test an approach for assessing the risk of bias in four measurement-related domains key to the study of mental health problem trajectories in children with neurodevelopmental disorders (NDD): (1) conceptual overlap between mental health problems and NDD diagnostic criteria, (2) over-reliance on a single informant, (3) unwarranted omission of the child’s perspective, and (4) the use of instruments not designed for or adapted to the population.

Methods

Building upon a previous systematic review, this study established supplementary criteria for assessing the risk of bias domains. Following this, the criteria were applied to measures used in 49 longitudinal studies of mental health problems in children with NDD.

Results

The general risk of bias across domains was rated as high in 57.1% of the 49 included studies. The highest risk of bias was seen in domain four (rated as high in 87.8% of studies) and the lowest in domain three (24.5%).

Conclusions

The risk of bias items enhance our understanding of the quality of the evidence about mental health problem trajectories in children with NDD. The methodological quality of future research can be increased by selecting conceptually clear scales developed for the population - preferably in the form of cognitively accessible self-report scales - and adopting a multi-informant approach.

Introduction

Neurodevelopmental disorders (NDD) such as intellectual disability (ID), autism spectrum disorder (ASD), and cerebral palsy (CP) have repeatedly been linked to heightened levels of mental health problems and mental disorders across childhood [1,2,3,4,5,6]. However, the measurement of longitudinal trajectories of mental health problems in children with NDD is associated with specific methodological challenges, relating to the interplay among the longitudinal design, study group characteristics, and the standard methods for measuring mental health problems in children. In a recent systematic review of longitudinal mental health problem trajectories in children with NDD [7], we observed that the instruments and checklists used to assess the risk of bias [8] were inadequate in addressing some of these challenges. Specifically, four domains related to aspects of outcome measurement stood out: (1) conceptual overlap between mental health problems and NDD diagnostic criteria, (2) over-reliance on a single informant, (3) unwarranted omission of the child’s perspective, and (4) the use of instruments not designed for or adapted to the population. The assessment of these domains, in our review, required more detailed data to be extracted from the included studies, and additional consideration of risks of bias. The present study aimed to develop an approach for evaluating the risk of bias in the four domains and to investigate the extent to which the four domains influence the validity of the findings about longitudinal trajectories of mental health problems in children with NDD.

The first of these specific risks concerns the potential conceptual overlap between mental health problems and NDD constructs [9]. It stems from the fact that mental illness, mental health problems, and mental disorders may overlap, depending on how they are defined [9]. In the Diagnostic and Statistical Manual of Mental Disorders (5th ed.; DSM-5 [10]), a mental disorder is defined as a syndrome “characterized by clinically significant disturbance in an individual’s cognition, emotion regulation, or behavior that reflects a dysfunction in the psychological, biological, or development processes underlying mental functioning” (p. 20). According to this definition, mental disorders encompass conditions that are typically regarded as mental illnesses (i.e., anxiety disorders and major depressive disorder), NDD (i.e., disorders typically manifested early in development, characterised by developmental deficits that produce impairments of personal, social, academic, or occupational functioning [10]), and other diagnoses. Mental health problems are commonly regarded as conceptually similar to mental illness, but as a broader construct, also covering milder problems and distress not meeting the criteria for a mental disorder [9]. In the present review, both internalising – (e.g., depression and anxiety) and externalising problems (e.g., aggressive behaviour) were considered as part of the mental health problem umbrella. Although this meant further widening of the mental health problems concept, it was necessary to enable comparisons of different aspects of emotional and behavioural problems. We also adopted a broad understanding of NDD by including childhood sensory and motor disorders, brain injuries acquired in childhood, and other diagnoses associated with sensory, motor, and mental impairments, in addition to the diagnoses listed as NDD in the DSM-5 [10] or the ICD-11 [11]. Many diagnoses involving such impairments that are not listed as NDD in the DSM-5 [10] arguably share important characteristics with those that are. For example, CP [12] and childhood hearing loss [13] are also characterised by developmental deficits that may produce impairments in different aspects of functioning. Moreover, conditions such as spina bifida [14] and CP [15] have previously been described as NDDs, with the latter showing genetic overlap with other NDDs, including intellectual disability and autism.

Importantly, an overlap between the mental health problem studied and the NDD is not problematic per se, but it may be, depending on the study’s aim and the interpretation of results. There is substantial symptomatic overlap across the mental disorders listed in the DSM-5 [16]. For example, attention deficit hyperactivity disorder (ADHD) and major depressive disorder both involve symptoms regarding concentration difficulties [10], and any instrument measuring the full symptomatology of any of these constructs will inevitably tap into the other construct. Further, overlap may occur when an instrument is applied to measure a mental health construct containing items that are identically worded, or similar to, criteria used to diagnose the population studied, and when results – despite this overlap – are reported as a separate mental health construct. For example, the Strengths and Difficulties Questionnaire (SDQ; [17]), assesses emotional and behavioural problems in four specific scales (emotional symptoms, conduct problems, hyperactivity, and peer relationship problems). Notably, the SDQ can be used to screen for specific NDD, such as ASD and ADHD (e.g., [18, 19]), and to measure emotional and behavioural problems more broadly in children with different NDD (e.g., [20,21,22]). The specific scales can be combined to form the broad-band scales of internalising (emotional symptoms and peer relationships problems) and externalising problems (conduct problems and hyperactivity) and a total difficulties scale (the sum of all four). Each specific scale consists of five items, and in the case of the peer relationship problems scale, several items are either closely related to or overlapping diagnostic criteria for ASD (e.g., “Rather solitary, tends to play alone”). Whether this overlap is problematic or not is related to which scales are reported and how the results are interpreted. For example, when the score of the peer relationships problems is reported as an indication of ASD the overlap is necessary (e.g., [18]) and when subscales with overlap are purposely omitted (e.g., [23, 24]) the risk for bias due to overlap is avoided. However, when the broad-band scales or the total scale are reported as an indication of a mental health problem construct there is a risk that disability-related difficulties are confused with mental health problems. This could lead to inflated scores which could contribute to incorrect conclusions about differences between groups.

The second domain covered in the review concerns bias arising from an over-reliance on a single type of informant in reporting the mental health problems in focus. Evidence shows that correlations between different types of informants (e.g., parents, teachers, children) can be low to modest depending on the combination [25,26,27]. Low inter-informant correlation does not necessarily mean that one informant is right and the other one is wrong or implies measurement error [28]. More likely, it relates to systematic differences among informants, such as the contexts in which they observe behaviour [29]. For example, a child may display hyperactivity in school but not at home. Relying on a single informant might provide an incomplete picture of mental health problems. A multi-informant approach is often advised to reduce bias when studying child mental health issues [26, 30]. However, the number and types of informants needed to provide a valid representation may depend on factors like the child’s age, as suggested by the larger discrepancy between informants seen in older as compared to younger children [25].

The third risk of bias domain, which can be considered a special case of the second domain, involves the exclusion of the child’s perspective, and the use of a parent as a sole informant. Exclusion of the child’s perspective might lead to bias through the depression-distortion hypothesis, i.e., the tendency for depressed mothers to exaggerate descriptions of child problems [31]. This risk is highlighted separately in this review because some aspects of mental health problems are inherently subjective. This subjectivity could be an explanation for the larger informant discrepancies observed in internalising problems, such as anxiety and depression, as compared to externalising problems [25,26,27]. Furthermore, there is a strong ethical argument for including the child’s perspective whenever possible. According to the United Nations Convention on the Rights of the Child [32], every child has a right to be heard in matters that concern the child. However, it is not realistic to expect all children to be able to self-report mental health problems. For children with NDD, the disability itself may be associated with problems with self-reporting, for example, difficulties with interpreting questions, retrieving information, and generating responses, related to underlying cognitive processes such as long-term memory, working memory, and judgment [33, 34]. Some children with NDD, such as those with severe-profound ID, by definition, have a level of impairment in cognitive and communicative functions [10], which makes the use of self-rating questionnaires improbable [35]. Similarly, young children may not have developed the necessary level of cognitive functioning to self-rate on mental health problems, regardless of NDD or not (see for example [36, 37]). However, determining the specific age and cognitive level at which children’s self-report reaches acceptable validity is challenging. For example, Varni, Limbers, and Burwinkle [38] demonstrated that typically developing children as young as five years may be able to make valid reports on their health-related quality of life using the Pediatric Quality of Life Inventory™ (PedsQL™). However, a later analysis of PedsQL™ data showed insufficient psychometric properties for many children between the ages of five and seven years [39]. For children with mild-moderate ID, some evidence indicates that self-rating of mental health problems may be feasible from 11 years, using standard self-rating instruments such as the SDQ [40] or the Youth Self-Report [41], with some adaptations made to the procedure (i.e., questions administered as an interview [40, 41], allowing item content to be explained [41]). Any effort to identify a specific and universal age and cognitive functioning threshold for child self-ratings is likely to fail since validity is also influenced by material and procedural factors [42]. Still, if a child’s self-report is not sought, when possible, an important perspective on the mental health problem is missing. Importantly, this is not the same as saying that there is no merit to parent reports, but rather that parent and child reports are not interchangeable.

The fourth and final risk of bias domain concerns the appropriateness of instruments used to measure mental health problems in the population. Many scales, such as the SDQ and the Child Behavior Checklist (CBCL; [43, 44]), were originally developed for typically developing children. Using these scales could be problematic if manifestations of mental health problems differ between typically developing children and those with NDD. For example, it has been argued that the number and type of symptoms for some psychiatric disorders need to be adapted for use with people with ID [45]. This would imply that questions in diagnostic interviews and screening questionnaires need to be phrased differently. Additionally, all questionnaires and interview procedures, presume some level of cognitive and communicative functioning in respondents. A cognitively accessible design reduces cognitive demands and supports cognitive processes to enable respondents to interpret and respond to assessment items as intended [46]. For example, in many scales, the respondent is to consider a time frame of several weeks or months when rating items. A valid response to the items in such scales presumes a comparatively high level of episodic memory functioning in the respondent, which should pose a bigger challenge to children with impairments in memory functions, such as those with ID [47], than children without memory impairments. Self-assessment could be made a feasible option for a larger proportion of children with NDD through the development of more accessible instruments. This could be achieved by adapting well-established scales to the needs of children with NDD (e.g., [48]) or developing new scales suitable for the target group (e.g., [49]). Of course, not all scales need to be changed to be valid for use in specific groups of children with NDD, as demonstrated by the tentative evidence concerning the use of SDQ and YSL in children with mild-moderate ID [40, 41]. But in samples where cognitive impairments vary or are unknown, higher cognitive accessibility should increase the likelihood of valid responses. This is also true for parent-rated versions of scales. In ID [50], for example, there is a strong genetic component, indicating that cognitive support needs are likely to be expected in many parents of children with ID as well as the children themselves.

In summary, the four domains (see Table 1) involve known methodological challenges that researchers and clinicians attempting to track the longitudinal change in mental health problems in children with NDD need to consider and manage. Common risk of bias appraisal tools like the checklists used in the Critical Appraisal Skills Programme [8] do not provide detailed instructions for assessing bias in these specific yet important domains. Given that many other design features need to be considered when assessing the risk of bias, there is a possibility that these NDD-specific questions may be overlooked. Hence, the purpose of this study was (1) to develop an approach to assessing the risk of bias associated with the four identified domains and (2) to assess the risk of these biases in a recently conducted systematic review of longitudinal trajectories of mental health problems in children with NDD [7]. The research questions for this study were:

• 1 Is the risk for overlap between the outcome measures and the criteria used to define the study group (i.e., the NDD) dealt with satisfactorily?

• 2 To what extent has a multi-informant approach been taken to capture variation in mental health problem expression in different contexts?

• 3 (a) Is the child’s perspective represented in the assessment of the mental health problem? (b) When not, is there a reasonable basis for excluding child self-assessment?

• 4 Are the instruments and procedures designed to be cognitively accessible, or have they been adapted in some way to the specific needs of the study group?

Table 1 Four domains presenting challenges in measuring longitudinal changes in mental health problems in children with neurodevelopmental disorders (NDD)

The present study builds on findings from an earlier systematic review of longitudinal mental health problem trajectories in children with NDD [7]. That review identified indications of methodological issues that were not adequately captured by the standard risk of bias tool employed. To better understand the scope and nature of these issues, additional data extraction from the included studies was required. By addressing these four key questions, this study sought to contribute to a better understanding of the methodological weaknesses and strengths of the field of mental health problem trajectories in children with NDD. The findings will be summarised and used to critically evaluate how the field has dealt with the challenges posed.

The study protocol for the systematic review of mental health problem trajectories in children with NDD [7] was registered in PROSPERO (142,412). Some aspects of the design, e.g., search strategy and eligibility criteria, are summarised or appended as supplementary material in the current review, but a more comprehensive description can be found in Danielsson et al. [7]. Taken together, the reviews adhere to the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines [51].

Search strategy

Searches were performed in PsycINFO, ERIC, Web of Science, PubMed, and CINAHL in September 2019 and June 2021 with combinations of words (i.e., synonyms, examples, or MeSH-terms) representing the constructs “mental health”, “disability”, “longitudinal”, and “child”. The searches resulted in 94,662 records, which were reduced to 72,582 with duplicates removed. Another 8,599 were identified by going through the reference lists of relevant reviews. The records were then screened in a three-stage process based on title, abstract, and full-length texts. Due to the large number of identified records, 22 reviewers were involved in the process. An overview of the flow of records through the study is provided in Supplementary Fig. 1, and the detailed eligibility criteria are outlined in Supplementary Table 1.

Selection criteria

Studies were included based on the eligibility criteria found in Supplementary Table 1. In short, longitudinal studies of mental health problems (defined broadly) in children under 19 years of age with NDD were included. Studies with at least three waves of data collection (with two or more years between the first and last wave) were included. Studies, where the mental health problems of interest were the explicit target of an intervention, were excluded, along with papers written in any language other than English. No limitations regarding publication year were applied.

Data extraction process

For the original review [7], the extraction of relevant information was done independently by random pairs of reviewers (n = 22) and synthesised by a third reviewer. In cases where the original reviewers disagreed, the third did an independent extraction of data and made a final decision based on all three sources. For the current study, key variables relating to measurement constructs/concepts, respondents, and cognitive accessibility were added, and the additional data for these variables were extracted by one investigator (M.I.). In ambiguous cases, another investigator (H.D. or C.I.) was consulted and a consensus decision was made. Information about diagnostic criteria and instruments was retrieved from relevant diagnostic manuals like the DSM-5 and instrument manuals. Most of the data reported were extracted from the included studies for the current study. However, some of the study characteristics (e.g., study populations’ mean age at the first data point), are included in the present study for context, even though they have been reported previously [7].

Risk of bias assessment protocol

The general risk of bias assessment of the included studies has been previously reported by Danielsson et al. [7]. For the current study, the first author (M.I.), developed a supplementary tool to assess the risk of bias in four domains addressing specific challenges related to the measurement of longitudinal trajectories of mental health problems in children with NDD. Specific assessment criteria (questions) were formulated for each domain to guide the evaluation of the risk of bias. The risk of bias for each domain was rated on a three-point scale based on the responses to the underlying questions (Table 2 provides a detailed description of the assessment for each domain): “Low” (indicating minimal risk), “High” (indicating elevated risk), and “Unclear” (reflecting uncertainties in the bias assessment, either due to mixed findings or insufficient information). The overall risk of bias, across the four domains, was also assessed for each study according to the following principles: (1) if any level of risk of bias (“low”, “unclear”, or “high”) was assigned more frequently than any of the others in the four domains then the overall risk of bias was rated at that level, (2) if “high” and “low” was assigned two times each the overall risk of bias was set to “unclear”, and (3) if two domains were considered to have a “high” or “low” risk of bias and the other two an “unclear”, the overall risk of bias was described as “unclear”. It is important to underscore that this supplement is not intended as a standalone instrument to assess the total risk of bias for a study, but as a complement to standard instruments, for example, the Cochrane Collaboration’s tool [52] or the Critical Appraisal Skills Programme checklist for cohort studies [8].

Table 2 A supplementary risk of bias tool for studies assessing mental health problems in children with NDD

Data analysis

Analyses were carried out in R [53] with RStudio [54]. The R packages papaja [55] and robvis [56] were used to compile the manuscript and to make the risk of bias figures. The data extracted from the included studies and a reproducible version of the manuscript is available at https://osf.io/hjrqc/.

Results

A total of 49 original studies were identified through the screening and eligibility process and included in the current review (see Table 3 for an overview of the included studies). Of these, at least 18 reported data from participants that, based on the name of the project or resemblances in participant characteristics, were likely to also have been participants in one or more of the other included studies. Disregarding this overlap, this review includes data from 9,446 participating children. The participants’ mean age ranged from 0.51 to 12.30 years at the first wave of data collection and 4.50 to 23.20 years at the last. The mean number of data collection waves was 5 (range 3 to 17) and the mean total length of follow-up was 5.62 years (range 2.00 to 16.74 years). A total of 148 scale scores were identified when counting the scales reported on the most general level in each included study, i.e., specific subscales were only counted in the absence of a reported total scale score or broad-band scale score. The scores were derived from 34 different instruments.

Table 3 Characteristics of the included studies

Risk of bias domain 1: Conceptual overlap

There was some level of overlap between one or more items and diagnostic criteria in 33.8% of the identified outcomes. Table 4 demonstrates examples of overlaps in two of the included studies and how the risk of bias was assessed. An overview of all studies with at least one overlap (44.9% of studies) is displayed in Supplementary Table 2. In studies where an overlap was identified, 77.3% contained at least one outcome with an overlap that was neither addressed nor discussed. The most frequently occurring instrument-diagnosis combination with a conceptual overlap was SDQ and ASD (n = 3). ADHD was the diagnosis where diagnostic criteria most commonly overlapped with at least one item in a reported mental health problem outcome (32.0% of the identified overlaps when including both pure ADHD groups and groups with ADHD and co-occurring diagnoses), followed by ASD (24.0%), and developmental disabilities (16.0%). ADHD, ASD, and developmental disabilities accounted for a larger proportion of the studies with overlap (72.0%) than would have been expected by the size of their combined share of the study groups in the included studies (30.4%).

Table 4 Examples of conceptual overlap between mental health problem outcomes and diagnostic criteria from two of the included studies

Risk of bias domain 2: Multi-informant approach

The risk of bias due to the lack of relevant perspectives on the mental health problems outcome was rated as high in 79.6% of the included studies (see Table 5 for examples of how the risk of bias was assessed in this domain and Supplementary Table 3 for an overview of all the included studies). Information about the mental health problem of interest was collected from multiple informants in 12.2% of the studies and relying on one informant was deemed justifiable, due to the young age of the population, in 8.2%.

Table 5 Informants recruited in four of the included studies and an assessment of the appropriateness of the informant recruitment approach based on the age of the participants

Risk of bias domain 3: Omission of the child’s perspective

The risk of bias due to a lack of the child’s perspective on the child’s mental health problems was rated as high in 24.5% of studies (see Table 6 for examples of how the risk of bias was assessed in this domain and Supplementary Table 4 for an overview of all the included studies). The child’s perspective was missing in 87.8% of the studies, and of these, child self-rating was deemed theoretically feasible in 53.5% based on a combination of participant age and reported level of intellectual functioning.

Table 6 Assessment of the feasibility of including the child’s perspective on the mental health problems outcomes in four of the included studies

Risk of bias domain 4: The use of instruments designed for or adapted to children with NDD

Only 8.8% of the different instruments applied were originally designed for use in children with NDD: the Aberrant Behavior Checklist, the Repetitive Behavior Scale-Revised, and the Scale for Emotional Development-Revised. No study reported that adaptations had been made to any instrument to make them more accessible or in other ways suitable for children with NDD (see Table 7 for examples of how the risk of bias was assessed in this domain and Supplementary Table 5 for an overview of all the included studies).

Table 7 Scales measuring mental health problems across four of the included studies and their suitability for use in the studied populations

Overall risk of bias in the four domains

Overall bias across domains was rated as high in 57.1%, unclear in 28.6%, and low in 14.3% of the 49 included studies (see Fig. 1). The domain with the highest proportion of studies rated as having a high risk was the fourth domain, i.e., bias due to the use of instruments not developed for or adapted to children with NDD (high risk of bias in 87.8% of the studies) and the third domain had the fewest risks, i.e., the unwarranted omission of the child’s perspective (high risk of bias in 24.5% of the included studies). All but one [96] of the included studies had a high risk of bias in one or more of the four domains (Supplementary Fig. 2 displays the risk of bias at the individual study level).

Fig. 1

The overall risk of bias for each of the four domains: D1, overlap between mental health problem outcomes and characteristics of the study group; D2, insufficient informants; D3, unwarranted omission of the child’s perspective; and D4, use of instruments not designed for or adapted to the study group (red reflects “high” risk of bias, yellow “unclear”, green “low”, and blue “no information”)

Discussion

The purpose of the present study was to develop and test an approach for assessing the risk of bias in four domains that are of particular importance in longitudinal studies of mental health problems in children with NDD and to assess how common these problems are in the field. Most notably, the results showed that some degree of bias related to these measurement domains was present in almost all of the included studies. Of these four domains, the most frequent concern was the use of instruments not designed for, or adapted to, children with NDD, followed by the restricted number of informants and perspectives represented in the mental health problem data. The risk of bias due to a lack of the child’s perspective and/or overlap between outcomes and the diagnostic criteria used to define the study group was lower compared to the other two domains but was still a substantial issue for the field as a whole.

The results show that conceptual overlap, i.e., mental health problems not being clearly distinguished from NDD diagnostic criteria, is a common concern in studies with populations with symptom-based diagnoses, such as ADHD and ASD, and to a lesser extent in those where diagnoses are based on etiology, such as pediatric traumatic brain injury or Fragile-X. In terms of outcomes, the broad-band internalising and externalising, or total scale scores of SDQ and the CBCL family of scales are often involved in cases with overlap. These scales were developed to screen broadly for problematic levels of emotional and behavioural difficulties in typically developing children and young people and, as such, do not differentiate between symptoms relating to NDD and other disorders or emotional problems on the broad-band levels. There are however subscales, such as the emotional problems scale of SDQ, where the risk for overlap is much lower than the internalising broad-band scale, which also encompasses the peer problems subscale with items very closely related to some NDDs.

The conceptual overlap is, when present, often not discussed and/or adjusted for in the analyses or through other design elements. When addressed, the approach to deal with it spans from mentioning the overlap as a limitation in the discussion [95] to clearly stating that scales with substantial overlap should be interpreted as NDD-related difficulties rather than additional mental health problems [63] and running analyses with and without items with an overlap to get a picture of their influence on the results [90]. The consequence of the conceptual overlap in the field is two-fold. Firstly, it could mean that the levels of mental health problems (as something separated from difficulties relating to NDD) are exaggerated in some groups of children with NDD. Secondly, it makes it hard to tell if a longitudinal change in the measured outcomes reflects changes in NDD-related difficulties or a separate mental health problem. For example, the natural course of NDD-specific difficulties, such as the tendency of a decreasing rate of hyperactivity over time in childhood ADHD [78, 106], risks distorting a mental health problems trajectory if the scale used includes items related to hyperactivity.

A clear majority of studies reported data from only one informant, most frequently a parent, despite recurrent recommendations to apply a multi-informant approach when assessing mental health in children [26, 30]. However, there were exceptions, such as Lahey et al. [78], in which three perspectives (child, teacher, and parent) on the mental health problems being investigated were combined. A restricted number of perspectives in a single study could be less of a problem if the field as a whole had a reasonable distribution of different perspectives. However, as seen in the results, there is an over-reliance on parents in the field today. Direct observations in the children’s natural contexts by researchers were not applied in any of the studies. This reliance on parent-reported data risks under-reporting of behaviours more typically displayed in other contexts than at home, such as problems between peers. Another risk is that parent ratings may be influenced by the parent’s mental health status [31]. This could be especially problematic for parents of children with NDD since they often report symptoms of depression, poor sleep quality, and stress [107]. A specific challenge in studies with a longitudinal design is that the most valid combination of methods and informants changes over time as the child develops. As argued by Rosema et al. [88], parents of younger children are likely to have more knowledge about their child’s mental health problems than parents of adolescents. One possible solution to this dilemma, demonstrated by Lahey et al. [78], is to add, rather than replace, informants as the child grows older.

The child’s perspective was missing in about a quarter of studies where child self-rating was deemed theoretically possible based on the participant’s age and level of intellectual functioning. Since some aspects of mental health problems are intrinsically covert (subjective), and difficult to measure without having the child describe their mental health (as pointed out by Woodruff-Borden et al. [103]), omitting the child’s perspective risks leading to a skewed picture with an overemphasis on overt behaviours. In the long run, this could lead to a self-fulfilling prophecy, where externalising problems are more often included as outcomes than internalising problems based on results from earlier studies. If the unwarranted omission of the child’s perspective is unevenly distributed between studies involving children with NDD and typical development, it follows that it could be difficult to disentangle real differences in profiles of emotional and behavioural differences between the groups from artifacts stemming from the methodological differences.

No examples of self-rating instruments specifically adapted to or designed to be cognitively accessible were identified in the current review. Very few of the scales were explicitly developed for use in the NDD population. Some instruments were claimed to have adequate psychometric properties when used in children with NDD but it was beyond the scope of the current review to go through all evidence on the psychometric properties of the included scales when used in the NDD population and evaluate the validity of such claims. Still, the use of instruments not developed for the population targeted in a study can lead to problems conceptually and practically. For example, mental health problems may have atypical presentations in children with NDD [45, 108], which means that questions may need to be phrased differently than with typically developing children. Further, when self-report is sought, scales need to be carefully designed to optimise cognitive accessibility.

Limitations

The validity and generalisability of the results of the present review are influenced by a few limitations that need to be discussed. First, even though the definitions and operationalisations applied were well-grounded, it should be noted that there may be reasons for considering other specific thresholds which would have led to slightly different outcomes. For example, a rather conservative definition of mental health problem-NDD overlap was applied, in that only explicit overlaps between diagnostic criteria and items in scales were considered. However, many etiology-based diagnoses are also closely linked to specific behavioural profiles, e.g., Fragile-X with ID and ASD [109]. Widening the definition of overlap to include difficulties typically associated with a disability would have resulted in more overlap being identified. At the same time, such a definition would have led to difficulties in drawing a clear line between NDD-related difficulties and common co-existing difficulties. Second, the present review did not quantify the extent of conceptual overlap in each of the included studies and therefore does not give a full picture of the risk of bias in that domain. Some of the studies used scales with hundreds of items whereof only a few overlapped diagnostic criteria, while other scales were much shorter and had more items with overlap. Future research will have to further investigate the exact extent of the problem. Third, the most frequently used risk of bias level across the four domains was used when calculating the overall risk of bias across domains, rather than generalising the highest risk of bias seen in a specific domain to the overall level. The reason for choosing this approach was that it allowed important variability to be exposed overall: generalising from the highest-rated item would have risked all included articles deemed to have the same (high) level of bias overall. Finally, the systematic search for evidence on which this study is based was conducted in 2021, hence more recent publications may have addressed these potential risks of bias more fully. However, the main aim of the current study was to develop and try out an approach for assessing these specific risks of bias rather than summarising the most recent evidence in the field of longitudinal mental health problem trajectories in children with NDD.

Clinical implications and future research

The overarching recommendation emanating from the results is that methods for collecting information on mental health problems in children with NDD could be chosen with more consideration than appears to have been done to date to avoid a partly skewed picture of mental health problems in both clinical and research settings. Several steps need to be taken to reduce the risk of bias in future studies. When selecting which scale(s) to use, it is important to:

• 1 Choose conceptually clear scales and subscales.

• 2 Prioritise self-report over parent-report, especially for internalising problems. When considering self-report, it is important to acknowledge that factors other than child-related factors determine whether it is feasible or not. The cognitive accessibility of scales is equally important and should be considered along with validity when deciding between scales.

• 3 As with typically developing children, a multi-informant approach is recommended, especially but not only for older children who spend a large part of the day at school, with peers, or in other contexts without parents. In longitudinal studies where the first data collecting point takes place in early childhood, it is advised that self-report, teacher-report, and/or direct observations are added as the child grows older. Future studies need to investigate the barriers to applying a multi-informant approach in longitudinal studies of mental health problems in children with NDD.

• 4 When no valid and accessible scale exists for a specific construct, a primary focus ought to be to develop one or adapt a scale originally developed for typically developing children to fit the needs of children with NDD. In research, more attention needs to be directed toward the challenge of developing and validating cognitively accessible self-report scales and procedures to assess mental health problems.

Finally, it is recommended that the four domains addressed in the current review should be considered whenever assessing the risk of bias in studies of mental health problems, and related constructs, in children with NDD in future systematic reviews. Without consideration of these additional potential risks of bias in this type of study, we may overestimate the quality of the evidence available.

Conclusions

The present study aimed to develop an approach for critically reviewing four measurement-related domains in studies investigating longitudinal trajectories of mental health problems in children with NDD, as well as to assess the risk of bias in these domains within the literature. All but one of the included studies had a high level of risk of bias in one or more domains, most commonly (1) the use of instruments not designed for or adapted to children with NDD, and in descending order, (2) an insufficient number of informants and perspectives represented in the mental health problem data, (3) overlap between the mental health problem outcomes and the diagnostic criteria used to define the study group, and (4) a lack of the child’s perspective. Taken together, these risks of bias could lead to a skewed picture of the mental health problems of children with NDD, through processes leading to both over- (e.g., conceptual overlap) and underestimation (e.g., use of instruments not developed for children with NDD). To minimise these problems in future research and clinical contexts, it is advised that instruments and procedures are chosen following a few guiding principles. Researchers and clinicians should seek to include multiple perspectives on the mental health issue of interest, use scales without conceptual overlap, preferably developed for children with NDD, and wherever possible in the form of cognitively accessible self-report scales. If no such scales exist, the development and validation of new scales should be a priority.