A paper, “Unreliable evoked responses in autism,” has appeared in the journal Neuron describing findings of consistency differences in brain responses in autistic people versus non-autistic people. When I read the paper, I had several questions, and rather than use my voice to address them (blahblahblah), I turned to several experts in the autism field--all with different perspectives--and asked them my questions, instead. Their emailed responses were so thoughtful that rather of trying to parcel out the best bits into a blog post on the paper, I asked their permission to use their comments as a roundtable discussion on the findings of this paper.
First, the abstract from the paper
'Autism has been described as a disorder of general neural processing, but the particular processing characteristics that might be abnormal in autism have mostly remained obscure. Here, we present evidence of one such characteristic: poor evoked response reliability. We compared cortical response amplitude and reliability (consistency across trials) in visual, auditory, and somatosensory cortices of high-functioning individuals with autism and controls. Mean response amplitudes were statistically indistinguishable across groups, yet trial-by-trial response reliability was significantly weaker in autism, yielding smaller signal-to-noise ratios in all sensory systems. Response reliability differences were evident only in evoked cortical responses and not in ongoing resting-state activity. These findings reveal that abnormally unreliable cortical responses, even to elementary nonsocial sensory stimuli, may represent a fundamental physiological alteration of neural processing in autism. The results motivate a critical expansion of autism research to determine whether (and how) basic neural processing properties such as reliability, plasticity, and adaptation/ habituation are altered in autism.'
The four participants in this impromptu roundtable are Jon Brock, Deborah Budding, Elizabeth Milne, and Uta Frith. Dr. Brock is a research fellow in the Department of Cognitive Science at Macquarie University and writes finely and well about autism research at his blog “Cracking the Enigma.” Dr. Budding is a neuropsychologist and supervising faculty at Harbor-UCLA Medical Center’s Neuropsychology training program, in addition to being in private practice. She also happens to be co-author of Subcortical Structures and Cognition, which she says “some people view as a seminal book in the field(s) of neuropsychology/psychiatry and others view as an excellent sleep aide.” Dr. Milne is a psychologist at the University of Sheffield, with a focus on development. She researches differences in perception in autistic compared to non-autistic people. Finally, Dr. Frith is a developmental psychologist at the Institute of Cognitive Neuroscience at University College London and highly regarded as an autism expert, having focused on the condition in her research since the 1960s. ETA: I neglected to note that I make the fifth scientist in the roundtable (developmental biologist) as the unplanned moderator. My perspective is that of a scientist and woman who, had the diagnosis been known in my childhood, would likely have been diagnosed with Asperger's, something I've described in greater detail elsewhere. My obsession with autism research (had you noticed that?) and my understanding of autism do not arise only from my experience as the parent of an autistic son but from my own experiences, as well.
My questions upon reading the paper are below; I’ve added in my rationale for asking them in brackets:
1. How valid do you find the use of fMRI for delineating groups or within-group/cross-test variability? What factors might influence changes on fMRI? [I asked this because I’ve seen several discussions of how we need to regard fMRI studies with greater suspicion, in part for reasons recognized here.]
2. This analysis relies on mathematical analyses (correlations between signal-to-noise ratio, for example, and IQ or within-group reliability from test to test). What do you think of relying on these distilled parameters for inferences in this study? [I asked this question because the analyses in the study seemed to have an “after-the-fact” flavor to them, possibly driven by the initial result of no differences in means between the groups. Also, I’ve seen discussions of how measures of IQ might not be that reliable in the autistic population.]
3. Figure 5 in the attached paper seems to represent the crux of their findings. Would you be able to comment on your impressions of these data? [Several things about this figure, which I did not obtain permission to reproduce here, had me wondering about the data, based on the distributions it showed. The discussion reveals more about that.]
4. These analyses were done with a group of 14 autistic people (four women) and 14 control participants. Can you comment on your impression of the sample size, particularly in light of the statistical analyses they performed? [I asked this question because the sample size seemed rather small for the multiple analyses the study authors did, particularly given the data I saw in Figure 5.]
Not all of the commentaries below directly address each of the questions, but together they form an interesting set of perspectives from four scientific experts in developmental neuropsychology but who come at the field from different areas. Brock investigates cognitive and language issues related to autism. Milne’s work relates directly the focus of this study, which is autism and sensory perception. Budding is a clinical neuropsychologist, and Frith has had a more global approach to autism that has included both cognitive links between brain and behavior and using understanding from research to improving the everyday lives of autistic people.
I’m presenting the commentary in alphabetical order.
(Regarding question 1), I can’t really comment on the technical side as I’m not an fMRI person. So I’ll just give you some more general comments.
First of all, it’s a really interesting and I think potentially important study. When I read the abstract, I immediately thought of half a dozen confounds and was fairly sceptical, but the authors addressed pretty much all of them. The only other thought I had was whether there might be an effect of attention to the stimuli. The experiment involved ignoring the stimuli while doing another task. Perhaps the autistic participants paid more or less attention to the stimuli. It would be interesting to know what happens to brain response variability in these paradigms if people are explicitly told to attend to the stimuli.
The main issue I have is with the authors’ claim that they have identified a “fundamental neural characteristic of autism.” It might be true, but they can’t say that at this stage. Looking at the scatterplots in Figure 5 (upper row), it’s clear that there are perhaps two to four individuals in the group of 14 autistic people who are driving the group differences. Everyone else is within the normal range of the control group. That doesn’t make it an uninteresting finding –- we should expect heterogeneity -- but I wouldn’t then call it a “fundamental characteristic.”
Given this variability, it would be really interesting to know if the individuals who had atypical brain responses also had correspondingly atypical sensory perception.
As the authors acknowledge, we currently have no idea whether this is specific to autism or not. Given that most findings in genetics and neuroimaging tend not to be specific to autism, my guess would be that the same would apply here. Again, that’s not a criticism of the actual study, but it’s another a reason not to talk about a “fundamental neural characteristic.”
I'll start by saying I'm not really competent to comment on the methodology of this study, as I don't do fMRI research. The questions/concerns I had are similar to yours. Ultimately, I am a clinician and not a researcher. I agree that this is a small sample size and they really should emphasize the limits this places on data interpretation. Apparently, to them, "subcortical" just means the lateral and medial geniculate nucleus?
My main concern about this and other studies like it is that I think they are often asking the wrong questions, sort of like losing a dime on Maple Street but looking for it on Main Street because the light is better. I would anticipate that autistic folk would be more inconsistent, so this research doesn't surprise me, whatever its limitations. The question is why.
Studies that want to explain neurodevelopmental issues without considering subcortical contributions, particularly the cerebellum, are just missing tremendous important information.
The study is presented "to motivate ..expansion of ...research etc ...[end of Abstract]." This is very proper, and I am glad these respected scientists do not oversell their study. It is interesting that they are pointing to some potential difference in neural level information processing that is not limited to social stimuli. But it is not clear how this translates to cognitive level information processing. This for me would be an essential link to pursue.
(Regarding the validity of fMRI), I believe the methods are valid, but studies always need replication.
(About the analyses), it is quite a tour de force. But I think using IQ as a covariate is a very good idea. Using signal to noise ratio is a very respectable measure in information processing. From a theoretical point of view, I would expect IQ and neural noise (whatever that may be) to have something to do with each other (but what?) rather than autism and neural noise.
(About Figure 5), I am more interested in the IQ correlation, which would confirm my suspicion above, namely that more 'neural noise' might be something to do with lower IQ.
(Regarding the group sizes), these are small numbers. However, larger numbers are not justified in such a preliminary study [EJW: This paper appears in Neuron as a “Case Study,” which means information from a limited number of cases, even as small as one.]
(About using IQ), I am quite a fan of IQ as a measure of information processing capacity. However, in my view, there are specific modular mechanisms over and above basic information processing. And this is where it's at! But this is highly contentious.
You are correct that this work is closely related to my own; I published a previous study that also found increased variability in evoked brain activity in individuals with ASD (autism spectrum disorder) compared with neurotypical controls; however, I used EEG rather than fMRI to record brain activity.
I'm afraid that I don't have any experience of using fMRI so I'm not in a position to comment on the technical aspects of the paper and can't therefore really answer your first point. My general impression of the work, however, is that it is a positive addition to the field of neuroimaging in ASD. It shows scientific convergence in the sense that it supports my previous finding of increased trial-to-trial variability in brain-imaging, and the fact that this has now been reported in using both EEG and fMRI suggests that this is likely to be a reliable finding. The paper also takes this area forward as it measures variability across three different modalities (auditory, visual, and somatosensory).
In general, this sample size is commensurate with many of the neuroimaging studies of ASD and is considered sufficient when comparing two groups of participants. From the data presented, it appears as though a low signal-to-noise ratio is associated with lower IQ and increased ASD severity. Only one of these relationships is statistically significant though, so these relationships should be considered tentative at this stage.
For me, the crux of the paper is Figure 2, i.e., the finding that there is greater trial–trial variability in response amplitude in individuals with ASD. This suggests that neural activity in those with ASD is globally disrupted, something which a number of researchers have been suggesting for a while, but (that) very few studies have found direct evidence of.
I want to thank everyone who took the time to comment on this paper in response to my questions. The study interested me because it looks at the brain responses of actual living, autistic people as they engage in an activity of living. And it’s been just as interesting to read these different perspectives from researchers and a clinician who focus on autism.