Don’t ever assume autism researchers know what they’re doing.

This is my post for Blogging Against Disablism Day 2016.  Like many of my posts, although it focuses on one specific situation (autism research) it applies to a much more broad set of circumstances if you look at it closely.  I was originally planning on doing something much more ambitious for BADD, but reality (and the reality of being a disabled person) got in the way of that.  So here is a different post that was fortunately written long in advance.  I hope it can serve as a resource for a lot of people, because I’ve been asked about this test a lot, and I usually only manage to describe part of it.  This is the first time to my knowledge that anyone has said all of these things at the same time in one place.  Apologies for the lack of sourcing, but I’m a blogger, not an academic, and if I had tried to do that I would never have been able to write it.  My inertia already prevented me doing what I wanted to do today, so I’m not going to let it prevent me from posting this.  So, with no further introductory babbling on my part:

Reading The What In The What?

The state of autism research is pretty uniformly terrible.  There is good research out there, but most research is bad.  The bad research includes some of the most popular research used to support the most popular theories about how autistic people think, and why.  Psychiatric research in general has low standards for quality control, and autism research seems to have lower standards than most.  If you can come up with a theory, it doesn’t matter how hare-brained it is.  If you have the skills to get it published, someone will publish it, whether your theory holds water or not.

The research is often so bad that I can see through it.  I have no formal training in research methods or techniques. And I have no formal training in autism.  Yet I’ve been able to see through the holes in autism research so well that I have been sought out by researchers to critique their own research ideas and suggest better avenues for research and techniques to use.

One research tool that has angered and frustrated me for years, is Simon Baron-Cohen’s “Reading The Mind in the Eyes” test.  It is most often used to show that autistic people lack empathy and the ability to figure out what other people are thinking and feeling based on body language and facial expressions.  Unfortunately, the test has very little real validity:  It measures all kinds of things it claims it doesn’t measure, and fails to measure the things it claims to measure.  And very few people understand this.  They take Simon Baron-Cohen at his word, especially because he’s a Big Name Autism Expert.  But he became a Big Name Autism Expert with research that for the most part is just as bad as this test

So for a long time, I’ve wanted to tell people what I thought was wrong with the test. The way I experience it, I feel all the wrong things at once. That makes it extremely difficult to communicate.  I feel the urgency of how wrong it is and how that wrongness is affecting autistic people around the world on a daily basis.  And I get frustrated that I can’t communicate ever single problem I see in the test. I’ve done it before, but it’s always exhausting and overloading, and I never feel like I’ve gotten all the information out there.  That’s why I am making this post.

Understand that when I bring up a potential problem with the test, I’m not saying that I absolutely know that it is a problem.  I just know that it is a potential problem, and that most of the official writing about the test does not address any of these problems.   There are possible solutions — other ways to test for the same things — but they would have to be done carefully.  And probably not by Simon Baron-Cohen, because he’s proven over time that he’s more interested in creating research that appears to support his theories, than he is in finding out what is actually happening with autistic people.

So what is the test?

The Reading the Mind in the Eyes test works like this:

A sample question from the Reading the Mind in the Eyes test.

 

 

You look at a black-and-white photograph cropped to only include a person’s eyes.  There are four words for different emotions.  You pick which emotion you think most closely matches the emotion conveyed by the picture of the eyes.  You do this with a large selection of eye pictures.  Then they score you by comparing your answers to the answers given by nonautistic people.  If your answers match the most common answer given by nonautistic people, you are scored as being correct.  If you answers do not match the most common answer given by nonautistic people, you are scored as being incorrect.  Then your scores are compared to a particular set of norms, to see whether you “pass” the test overall, or not.

I have a lot of problems with the design and administration of this test.  As I said, some of them might prove to be real and some of them might prove to be unreal.  But I can’t imagine that the test is so good that it overcomes every problem that I see with it.  So I’m going to discuss each element that I find wrong with this test.

These are still photographs.

Like many other elements of this test, this is different from how emotions occur in real life.  When we observe someone having an emotion, we don’t observe a still photograph.

The photographs are in black and white.

Unless we have achromotopsia, we don’t observe the world in black and white.  When someone is having an emotion in front of us, they do not turn into a black and white photograph.

The photographs are cropped to include only one part of the body.

We don’t know all the different ways that autistic people might read emotion.  We certainly don’t know it well enough to crop out every other indicator of emotion in order to focus on the eyes, and then make inferences about people’s understanding of the mental and emotional states of others, based only on one body part.  Since we don’t even know whether autistic people routinely pick up on body language, let alone how, it’s irresponsible to crop the image to one body part and assume that this is how we will tell whether and how autistic people pick up on body language or facial expression in general.

The photographs feature a part of the body that autistic people are notorious for having trouble looking at, and therefore avoiding, in real life.

Lack of eye contact is a very common autistic trait.  So common it has sometimes ended up described in the diagnostic criteria.  If autistic people avoid looking at people’s eyes, then it stands to reason that it’s possible we have no practice reading people’s facial expression through the eyes alone.  So why crop an image to the one part of the body we tend to look at the least, and then make broad inferences about our ability to read body language based on our ability to understand what’s going on with people’s eyes.

Different autistic people’s descriptions of why they avoid eye contact differ.  And some autistic people make eye contact, some even make more of it than nonautistic people do.  But many autistic people describe making eye contact as involving intense anxiety, physical or emotional pain, sensory overload, terror, or a sense of being flooded with the other person’s emotions.  Any of these things could prevent an autistic person from making a lot of eye contact, or from looking at people’s eyes in general, even as pictures on paper.  If there’s something that a person has spent their entire life not looking at, then obviously they’re not going to have skills that most people get from looking at eyes.

There’s a tendency in autism research to view autistic people’s abilities in ways that other disabled people’s abilities are not so frequently viewed.  Blind people can’t usually read facial expressions.  Deaf people can’t always hear tone of voice.  But people don’t assume that blind or deaf people lack theory of mind because there are elements of nonverbal communication they don’t understand due to sensory impairments.  But that’s exactly what is done to autistic people over and over — there are things that we just plain can’t perceive for various physical reasons, and then those are used as evidence that we don’t even know other people have minds at all.

And, of course, we don’t even fully know that autistic people don’t perceive those things.  Quite often, autistic people’s abilities are compared to nonautistic people’s abilities, and not taken as possibly having ability patterns of our own that are quite different:  Many autistic people report being able to read body language but not the same body language that we’re expected to be able to read.  Many autistic people report being able to read other autistic people’s body language — or some other autistic people’s body language — but not nonautistic people’s body language.  When nonautistic people can’t read autistic body language — which is usually — they’re not told they lack social skills, they’re told it’s because there’s no body language there to read, or because we lack social skills.  But when we can’t read theirs, we’re told we lack social skills.  But that’s a completely separate problem in and of itself.

The photographs are created from actors.

An actor playing an expression is not the same thing as a person spontaneously experiencing an emotion.  Acted expressions — and understanding them —  relies as much or more on popularly understood stage conventions, as it does on understanding what emotions actually look like in real life.  So this test is essentially a test of your ability to read stage conventions In a particular context, and to do so similarly to the way nonautistic people would.  If autistic people pick up on unconscious body movements that actors would not be able to change (because they’re used to only changing movements that nonautistic people perceive, and leaving alone movements not perceived by most people), then this test would not test things accurately at all.  Like all these situations, we don’t know if that’s the case but we definitely don’t know that it isn’t the case.

The correct answer in the photographs doesn’t rely on what emotions the actors said they were playing.

The correct answer was arrived at by asking large groups of nonautistic people what the person in the photograph was feeling. The most common answer became the correct answer.  So there’s a chance that the emotion the actor was playing, is totally different from the emotion that everyone says they have.  And this is not addressed at all.

The test assumes that the most common answer given by nonautistic people is the right one.

This assumption basically makes the test a circular argument:  Nonautistic people are better at reading emotions than autistic people, because we have created a test where the correct answers were created by looking at the answers of nonautistic people, thereby ensuring that nonautistic people are far more likely to get the right answer.  And then when autistic people give different answers than nonautistic people, it’s used to prove that we don’t understand what’s going on in the photographs.  Can you see how completely circular the reasoning is there?  The way the test is created, anticipates its own solution.  If autistic people’s reading of body language differs at all from nonautistic people’s, then we must be wrong.

So what if the most common answer given by nonautistic people is wrong?  Nothing about the test even allows for this possibility.  it’s created wit the assumption that nonautistic people are right, and that the only way for autistic people to be right is for our answers to resemble the answers of nonautistic people.

The test sometimes seems to rely more on how people feel looking at the eyes, than on how the person with the eyes must feel.

I chose the particular photo from the test for a reason.  The image is of a woman’s painted eyes, looking stereotypically sexy.  The correct answer is supposed to be “desire”.  It’s not clear to me at all that this has anything to do with what the woman in the photograph is supposed to be feeling.  It seems to me that it has much more to do with how the person looking at the photograph, feels in looking at a seductive pair of eyes.  I don’t actually know what the woman in the photograph is portraying for real, but I have a strong feeling that it’s not actually desire.  Desire is what people feel looking at her.  And if this is happening in this photograph, it’s possible that it has happened in others as well.  That the feeling evoked by the eyes is seen as the same as the feeling of the person with the eyes.  Another photo of a very similar pair of eyes, by the way, gives the correct answer as “fantasizing” or something like that, and again I can’t help but suspect that’s the way the person felt looking at her, rather than what she felt when the picture was taken.

The test requires a person to process nonverbal and verbal information simultaneously, or very close together.

Many autistic people have trouble processing information quickly, and have trouble multitasking.  This can take many different forms.  But it’s fairly common for autistic people to be in a situation where we can either process nonverbal or verbal information, but not both at once.

Some of us develop stable adaptations where we always process nonverbal information but never verbal information, or where we always process verbal information but never nonverbal information.  Others of us switch around so that sometimes we process verbal information, and other times we process nonverbal information.  And some of us have a baseline state where we process either verbal or nonverbal information more readily, but can switch to processing the other in certain situations.  Some of us do process both in some situations, but in those situations process both of them mediocre to badly instead of one well and the other not at all, like we’ve spread out our information processing too thin so our abilities in both areas suffer compared to what they could be.  For autistic people who switch back and forth between verbal and nonverbal information processing, we can’t necessarily make the switch at will, nor does the switch always happen in an instant.  Sometimes it takes a good deal of time.

All of this means that a test that requires a person to understand nonverbal information while picking a word out and applying that word to the nonverbal information, is possibly putting an autistic person in a position where they are not able to show what they truly understand.

Because it’s quite possible that when left to our own devices, some of us process the nonverbal information accurately.  Possibly even more accurately than nonautistic people.  But then we are unable to make the switch to processing and understanding words while applying those words to the nonverbal information. For any of a large number of reasons.  So even if we would otherwise make accurate observations, the test may be setting us up to fail.

This test relies on the ability to apply words to observations.

This sounds like my last complaint about the test, and I’ve already described how processing differences could create problems here.  But there’s one other problem.

Autistic people who can apply words to observations at any time — well or badly — well enough to take this test, may be very different from autistic people who are unable to ever apply words to observations (or unable to do so reliably enough to take the test).  There may be some overlap between the two groups, but the two groups may also contain people whose abilities differ enough to be worth noting.

Much research into autistic cognition relies on the abilities of autistic people who can use or understand language.  It’s assumed that autistic people who can’t use or understand language well enough to take part in the research, are the same, just with more severe autistic traits.  So if a verbal autistic person has some ability to get correct answers but not as well as nonautistic people do, it’s just assumed that a nonverbal autistic person would be even worse at this than a verbal autistic person.

There are, however, a number of autistic people whose stated experiences of the world completely belie these assumptions.  I happen to fall into that group of people, myself.  This is people who have spent any period of our lives, with a severe enough difficulty with processing verbal information, that at that time we would be unable to take this test.  Those of us who have spent long enough in such a cognitive state, often report that our brains prioritize nonverbal information over verbal information.  At those times, we are not just a verbal autistic person with more severe social awareness problems.  We are a completely different kind of autistic person, with a completely different cognitive profile, including nonverbal social awareness.

Now obviously you can’t get into the head of someone who has always had and will always have so much trouble processing verbal information, that they will never be able to describe their experience.  But there are autistic people who can now ocmmunicate using words at least some of the time, but have spent enough of our lives with significant enough verbal processing problems, that we can give descriptions of what our cognition was like in the past, even if it has changed now.

Some of us describe being essentially the same as most verbal autistic people, but with additional difficulties (in a wide variety of areas) that most verbal autistic people don’t have.  But others of us describe — and sometimes are observed by others to have — abilities that are completely different in their type and pattern than anything described by the average verbal autistic person.  And some of us describe the ability to pick up nonverbal information from others, at least as well as nonautistic people do, if not better.  And there is no opportunity within this test, to test people who are in a state where their nonverbal cognitive abilities are excellent but their verbal skills are too poor to take the test.

It also doesn’t take into account the possibility of people whose ability to read nonverbal cues is excellent, whose ability to use language in general is also excellent, but whose abilility to apply language to these specific situations is not so good for any of a variety of reasons.  One of which I’ll address in my next point.

This test relies in particular on the ability to apply words to emotional states.

Even autistic people who are highly verbal can have difficulty applying language to emotional states, in themselves or in other people.  This test does not take into account the possibility that a person might be good with language, good with reading body language from nonverbal cues, but bad at applying words to emotions in particular, so they would fail the test even if they knew the right answers.  Because knowing what an emotion is, and knowing that the emotion is called ‘happy’, are two different things.  There may be autistic people looking at someone’s eyes, seeing happiness in those eyes, but not knowing that the word happiness applies to the feeling they are observing.  There’s an assumption that understanding something, and understanding the word for something, are the same thing.  They’re not.  Not for everyone, anyway.

This test gives no possibility to give a different and possibly more accurate answer.

It’s a multiple choice test.  What if all of the choices are actually wrong, and something else is actually right?  Not only is this not considered possible. (because it’s just assumed that the majority of nonautistic people are going to be right about something like this).  But there’s no mechanism to deal with it if it happens.  If the autistic person understands the body language better than the nonautistic people did, and is even able to accurately apply a word to the emotion they believe they are seeing, but the word is not in the multiple choices?  Out of luck.  Maybe an autistic person can read what that woman in the photo is actually feeling, rather than ‘desire’.  Maybe they know a word for it.  Maybe the word is ‘anticipation’ or something — that’s a completely random guess on my part, not intended to be accurate.  But if ‘anticipation’ is not on the list, then even if it’s exactly how the woman is feeling, there’s no way to pursue that as the right answer and get credit for understanding her emotions.

This test assumes that autistic people’s abilities to read body language would work the same as nonautistic people who read body language.

There’s a bunch of assumptions that go into this.

One is that autistic people and nonautistic people would, if processing body language, do it in the same way.  So testing autistic and nonautistic people on the same extremely specific situation, is supposed to give overall information, rather than testing of the specific abilities necessary to get the ‘right’ answer in this extremely specific context.  So this test is seen as testing autistic people’s ability to read body language in general and assign mental and emotional states to other people (and even to know that other people have emotional states).  When it’s actually testing autistic people’s ability to come up with the same verbal answers as most nonautistic people, to a question asked while viewing heavily cropped images of actors deliberately portraying emotional states showing a still black and white photograph of their eyes.

If autistic people have any difference in how — and in what context — we process emotional information from other people in a nonverbal fashion, this test does not allow for contexts that might differ considerably.  Just as an example, what if autistic people understood nonverbal information primarily through watching hand movements, whereas nonautistic people understood nonverbal information through the whole body, or through the eyes in particular?  This test doesn’t allow for that possibility, given that it’s still photographs of eyes.

A closely related assumption that I see happen all the time in autism research, is that the only abilities that exist are the abilities we know about in nonautistic people.  So if there are ways to understand nonverbal communication, they’re assumed that the only possible ways to understand it are ways that nonautistic people understand it.  It’s assumed to be impossible, that an autistic person could be just as good at understanding emotions from nonverbal cues as a nonautistic perosn is, or even better — but that the way we do it is so different that the test doesn’t test for it.

This is one reason that such an artificial limiting of the context creates problems.  You’re assuming that if we had this ability, it would show up in our ability to pick out the same words as nonautistic people, to describe the emotions experienced by an actor deliberately portraying the emotion in a still heavily-cropped black and white photograph of their eyes.  Maybe some of us can read elements of body language that nonautistic people can’t read.  To figure that out, you would need to have as much of the original context of the body language retained as possible.

Ideally, you would test autistic people’s responses to a person experiencing a genuine emotion, in the same room as the autistic person.  That way, the autistic person would be getting all the sensory information possible, rather than getting sensory information that is limited or distorted to include only the parts that nonautistic people find the most relevant.  And if we were observing something like the tightness and sweatiness of someone’s skin combined with subtle unconscious body movements nonautistic people don’t even pick up on?  Where nonautistic people were not relying on that information at all?  Then we would still get that information and be able to give accurate answers.  Whereas it’s possible that the test as currently conceived, shuts out the information autistic people would need to get the right answers.  And then acts like we would never have gotten the answers right to begin with.

Given that it’s well-known at this point that autistic people’s perceptual experiences are significantly different than nonautistic people’s — and from each other, often enough — then there’s no excuse for limiting the context so much that you’re potentially shutting out the information sources that autistic people actually use.  Like, this test might be useful for isolating what differences exist between what autistic people rely on to process nonverbal cues, and what nonautistic people rely on to process nonverbal cues. But it’s not useful for evaluating autistic people’s overall ability to evaluate nonverbal cues. But that’s exactly the kind of broad conclusions leapt to by the author of this test and people who use it in their studies.

This test assumes that when autistic and nonautistic people’s answers differ, the nonautistic person is automatically correct and the autistic person is automatically wrong.

This is similar to something I said earlier but it’s worth repeating:  This test contains within the setup, the outcome.  Because it’s normal entirely on the responses of nonautistic people, who are presumed to be right about these things.  If autistic people’s answers differ from those of nonautistic people, we’re always assumed to be the ones in the wrong.  Always.  When this may not be the case.

I know an autistic woman who had to guess the emotions of live actors as part of her diagnostic process.  She scored terribly on the test they gave her. That basically involved looking at live actors and ascribing emotions ot them.  The problem was, she was reading actual emotional cues, not stage conventions at least some of the time.  So her answers were different — and actually more accurate — than the answers nonautistic people assumed should happen.

I’ve done the same thing in much less formal contexts — been totally unable to read the emotion a person is trying to deliberately project for others, but been better than average able to read the emotion underneath, the one that caused unconscious body movements the person was not controlling.  I’ve had this verified by the actual people having the emotions, and I think the woman who was tested talked to one of the actors afterwards and found out she was in fact describing his real emotions as well.  So there are autistic people who read unconscious emotional cues better than we read deliberate attempts to manipulate other people’s emotional awareness — but the deliberate manipulation is what nonautistic people are reading, and they’re therefore assumed to be right.

This test compares nonautistic people’s ability to read other nonautistic people’s body language, to autistic people’s ability to read nonautistic people’s body language.

There’s a possibility that people read body language more accurately for people more similar to them, either neurologically or culturally.  If this is true, then autistic people would be better at reading the body language of other autistic people — and in particular, people with similar types of autism to our own.  When I say similar types of autism, I don’t mean the official definitions of autism versus Asperger’s, or functioning levels.  I mean deeper levels of similarity and difference in how we think, process information, move, and respond to the world around us.

So if that is true, then this test isn’t a fair comparison.  A fairer comparison would be nonautistic people’s ability to read other nonautistic people, compared to autistic people’s ability to read other autistic people.  And even better — although impossible right now, because we simply don’t have good subtyping systems for autism enough to be used in research in the ways I’m thinking of — would be to have autistic people’s ability to read autistic people who are the most similar to ourselves in terms of sensory, cognitive, emotional, and motor experiences.

I was once invited to dinner with a group of people, some of whom were autistic and some of whom were nonautistic people (parents and research professionals).  I accurately described an autistic man’s level of cognitive and sensory overload, and asked people to help him become less overloaded in particular ways.  The nonautistic people in the room were stunned at my ability to (verified by him) describe his emotional, sensory, and cognitive state in so much detail and accuracy.  They had seen nothing at all wrong.  I had sensed it all over him from a variety of nonverbal information he was blaring out like a beacon as far as I could tell.

But the way that things like this normally go?  When autistic people can’t read nonautistic people, it’s because of a global social skills deficit of some kind in autistic people.  When nonautistic people can’t read autistic people, it’s either because nothing is there to read (we’re just assumed not to be giving off nonverbal cues because the cues we give off aren’t always the same as nonautistic people’s), or because autistic people have a global social skills deficit of some kind.  So either way, a difficulty reading someone unlike yourself becomes entirely the fault of autistic people’s supposed social skills deficits.  Even though it’ she exact same problem going in both directions: A difficulty reading people whose experience of the world fundamentally differs from your own, which may be a nearly universal social skills deficit in both autistic and nonautistic people.

Generally, professionals in the autism field don’t even consider the idea that some autistic people might be as good at reading autistic people similar to themselves, as nonautistic people are at reading other nonautistic people similar to themselves.  This possibility doesn’t even exist.  So even though nonautistic people have huge problems reading autistic people, and are not blamed for that, autistic people’s ability to read people in general is tested based on reading people who are very different from us.  This would be like judging all of nonautistic people’s social awareness and social skills based on their ability to interact easily with autistic people.  Very few nonautistic people can intuitively understand how to understand or interact with autistic people, it’s one reason that autistic people are considered inherently mysterious and confusing, or to lack body language altogether.

I actually have had a bizarre experience that has happened many times.  Nonautistic people have been trying to say they know what’s going on with me, when they don’t.  An autistic person walks into the room and instantly tells them everything that’s going on with me: My feelings, my thoughts, my sensory experiences, my abilities, everything, in great detail.  And almost always, the response the autistic person gets?  “That’s impossible.  Mel isn’t using body language, so you can’t possibly be reading it.”  Nonautistic people are so used to their perceptions being all there is to reality, that some of them simply can’t believe it when autistic people see things about each other that nonautistic people can’t see at all.  I’ve at times said “But this person is completely right about me!  What she’s saying is accurate!”  And still they would talk over both of us, saying that it was impossible for the other autistic person to have figured out what I was feeling, thinking, and experiencing.

It’s not just an either-or thing though.  I am good at sensing unconscious body movements and bad at sensing the meaning of consciously applied deliberate fronts that people put on.  I may pick up on the front, but I won’t always understand what’s going on in that front.  This is with most people in general.  But the more similar a person is to me — and this can be autism-related or in other areas as well, both innate traits and life experiences — the easier it is for me to read them.  A person very similar to me stands out as if they are the only three-dimensional person in a world of flat empty figures.  I can sense every layer of what they are experiencing, both the real and the false, the confusion and the reality, just an astounding level of detail.  And this is far from unheard-of among autistic people, but none of it is measured by this test, which I o quite poorly on despite explicit verification from other people like me that I read them shockingly well in a level of detail well beyond what is covered by a test like this.

Whatever the reality ends up being, it’s bound to be very complicated, and the way this test is set up is not a fair demonstration of our abilities.  Not fair as in, what we’re being tested on is not an equal task to what the nonautistic people this is normed on, are tested on.

The experiences centered on in this test are the experiences that are important to nonautistic people.

This is a variation of the last thing I just described, but it bears saying outright:  This deals specifically with experiences that nonautistic people find important.  Now, it may be that autistic and nonautistic people find at least some of the same experiences important to understand in ourselves and in other people.  But it may also be that there are experiences specific to autistic people, that we are better at picking up in ourselves and other people, because they are so important to us.  And those are not tested on a test like this — artificially elevating nonautistic people’s scores and possibly lowering autistic people’s scores as well, by elevating their priorities over our own.

One example of a thing that’s important to autistic people is overload.  It’s not just one thing.  It can come from sensory input, thoughts, feelings, movements, effort, all kinds of things.  And it can take many forms, and have elaborate nuances to it.  Most nonautistic people can’t pick up on overload at all without being taught how, let alone pick up on any nuances of overload, shutdown, and our responses to overload.  But an autistic person looking at an overloaded autistic person may be able to see all those things in great detail, as well as emotional experiences related to them.

Because language has mostly been created by nonautistic people, there’s not necessarily an easy way to show the things that autistic people are better at perceiving and prioritizing than nonautistic people are.  Because there are often literally no words for the experiences until we invent them, and therefore there’s no standardized words that we grow up hearing the way we hear ‘happy’.  This affects our ability to communicate what we do perceive in other people’s body language, even things that we are quite good at perceiving.

The test doesn’t show where the problem comes from, if it’s a problem of naming emotions.

As just described, autistic people often grow up not hearing good descriptions of emotional and other experiences that are very important to how we experience the world.  But worse, we often grow up getting bad information.  People who can’t read us well, tell us we are feeling things that we are not actually feeling, leading us to associate the wrong words with the wrong emotions, which affects our ability to apply those words either to ourselves or to other people.  This is not taken into account when people ascribe our results on tests like this to purely innate social skills deficits.  The effect of our environment on our ability to learn these things — hell, the effect of nonautistic people’s difficulty reading us on our ability to learn these things — might as well not exist as far as most people are concerned.  Including most researchers.

Not only does the test not show whether this is the case.  But discussions of our results on the test rarely take things like this into account as possibilities.  From what I have seen, good research is supposed to consider all of the possible reasons for a particular result.  But autism research rarely does, it merely gives the explanation favored by the researcher.  If it talks about other possibilities at all, it talks about them in the most brief and cursory way possible, and doesn’t delve into issues like this one at all.

If a proper research paper were written on this test as it is, it would include all of the problems I’ve described and am about to describe, and then describe ways that further research could take these problems into account and work out what’s really going on here.  But it doesn’t, the papers on this stuff pretty much never do.  They’re supposed to.  They just don’t.  Partly for reasons I think are about laziness (autism researchers are given so much license to do bad, lazy research that it’s possible to become incredibly lazy and never even notice the degree of laziness you’ve developed), but partly because the researchers simply haven’t thought of many of this because they’re so used to coming at things from their perspective alone.

This test assumes that all autistic people are essentially alike, and does not account for significant differences between different groups of autistic people.

It’s possible — in fact I find it likely — that there are many different cognitive profiles for autistic people when it comes to processing nonverbal emotional information from other human beings.  The way this test is dealt with, makes it seem as if there’s really only essentially one type.

So there may be:

• Autistic people who can’t process nonverbal social information at all.
• Autistic people who process nonverbal social information roughly the same as nonautistic people do.
• Autistic people who process nonverbal social information roughly the same as nonautistic people do, but not as well or as efficiently.
• Autistic people who process nonverbal social information more accurately than nonautistic people do, but in a completely different manner.
• Autistic people who process nonverbal social information about as accurately as nonautistic people do, but in a completely different manner.

And those are just some of the possibilities.  In addition, each of the last two groups?  There may be multiple cognitive and perceptual profiles within each of those two groups.  Such that there may be dozens of different ways that an autistic person processes nonverbal social information, that nonautistic people don’t experience at all.  in addition, there are of course autistic people who are in more than one of the above categories, or somewhere between the above categories, or who switch back and forth between multiple categories.

The way this test is handled does not account for an extremely diverse set of abilities in autistic people.  And that limits its ability to report on what we actually experience, by averaging out everything until you get a sort of ‘average autistic profile’ rather than a profile that reflects the diversity of autistic experiences of the world.

This would be sort of like if you took autistic people who were great at nonverbal things and bad at verbal things and autistic people who were great at verbal things and bad at nonverbal things, and averaged out all their abilities until you got a picture of people who were average in both verbal and nonverbal things.  That’s an oversimplified example but you get the point.  My suspicion from actually knowing autistic people, is that there’s a pretty diverse set of abilities to read body language and ways that we read it and that this test is not allowing for any of that diversity to be showcased both because of the limitations of the testing material and because of the ways the results are interpreted after the data comes in.

Potential Effects of These Assumptions on Future Research

Assumptions about the limitations of autistic people carry over into research in ways that shocked me when I first discovered them.  I once had a conversation with a researcher I really respected, someone who was trying to do right by autistic people and understand our emotional responses and the like.  I asked her the following question:  “I know you’ve talked to a lot of parents of autistic children.  How many of them overall told you that their autistic child is the first person to pick up on emotional tension in their household?”

I’ll never forget her response.  It went something like this:

“Oh wow.  Actually?  I think every single parent told me that.  But until now, I had instantly put it out of my mind.  Because researchers like Simon Baron-Cohen said that autistic people were mindblind, unable to attribute emotional states to others, or unable to read those emotional states in others.  So any time anyone told me anything that contradicted what the experts were saying about this, I just forgot about what I was being told, assumed the parents were just mistaken, things like that.”

She was a very intelligent researcher who was trying her best to get past her biases about the abilities of autistic people.  And it took that question to start putting two and two together.

By the way, I did end up describing to her, in detail, an experiment she could do that would test the abilities of autistic people to read certain emotions in other human beings, without requiring the autistic person to ever have to use or understand language throughout any of the experiment.  It made use of technology and techniques already available to the research lab in question.  And she told me that it was a very good research design.  I hope that she will one day use it, because I want to see if I’m right in what I guess about this situation.  If I am right, follow up research could be then done to isolation what precisely is going on when autistic people accurately interpret social and emotional information from others, and what subgroups of autistic people may exist cognitively and perceptually, both among people who can do this, and people who can’t.

This is just one example of bad research design.

Pretty much all of autism research is littered with the kinds of holes left by things like the Reading the Mind in the Eyes Test.  Much other psychiatric research is just as bad, but there does seem something particularly sloppy about a lot of autism research.  And the things said by autistic people much like myself, are finally being proven right, slowly, by researchers who take a much more accurate approach to the information.

One thing that autistic people have complained about for decades now is the Sally-Anne test.  It’s supposed to test people’s theory of mind — the ability to know that other people have mental states that differ from our own, to attribute those mental states to others, things like that.  It does so by testing whether a person is aware — in a fantasy scenario often acted out by dolls or puppets or the like — that a doll during a test is deceived by the actions of another doll.

The test as normally given, uses some of the most complex sentence structures in the English language.  This is something that I had noticed before.  When autistic people pass the test, which we often do, we are then given more and more nested sentence structures of that type until we can’t do the test anymore.  Those more complex sentence structures are supposed to test ‘second order theory of mind’ and ‘third order theory of mind’ and the like — but really they’re linguistic nightmares with questions like “What do you think that Eric thinks that Sally thinks about what Anne thinks?”  Such questions trip me up at the best of times, even when I understand perfectly well the underlying concepts.

So what some researchers did, was they did some studies of certain groups of children.

There’s a condition called Specific Language Impairment.  Children with Specific Language Impairment by definition cannot also be autistic.  They’re mutually exclusive categories.  Children with SLI have the language problems similar to many autistic people, but not any of the other problems of autistic people.

So they did a test where they had children with SLI try to complete the Sally-Anne test.  They did just as badly as autistic people with the same level of language impairment.   If this were an autism-related problem rather than a language-related problem, that couldn’t possibly happen, because children with SLI are not autistic.

Then, they did a test where autistic children were tested on false beliefs, but in a context that did not require language processing.  The autistic children did as well as — and sometimes better than — nonautistic people did on the same test.

Until these tests were done, everyone assumed that autistic people’s bad performance on the standard Sally-Anne test was because we lacked some element of Theory of Mind — or lacked the whole thing.  Rather than that we did badly because we had language problems and the test requires some pretty high-level language processing.

And another thing that angered me, was the way that they seemed to deliberately set up tests to trip us up in this manner.  Like everyone knew all along, that there were autistic people who passed the Sally-Anne test just fine.  In fact, a very large minority of autistic people.  Not just a tiny number.  And yet they still said that the test results of the people who performed badly, reflected on the Theory of Mind abilities of all autistic people.   Even though there were people passing the test just fine.

But they didn’t stop there.  This is the part that pisses me off the most.  When autistic people did okay on a Sally-Anne test, we were simply given more and more linguistically difficult tests, until one of them finally tripped us up with its ridiculous number of nested sentence structures to process.  So “What do you think Sally thinks” became “What do you think Sally thinks about what Eric thinks?” Which in turn became “What do you think Sally thinks about what Eric things about what Maria thinks?”  Just keep nesting more and more levels of linguistic hell, until the autistic person is tripped up and fails the test.  Then, at that point, say that they lack a different ‘order’ of theory of mind.  You start saying “Well, some autistic people have first-order theory of mind but lack second-order or third-order theory of mind.”

That pisses me off because it was like they were deliberately setting us up to fail so that they could prove we really had the problem that the test showed we didn’t have after all.  So a test shows we have theory of mind. Devise a harder test until autistic people can’t do the test anymore.  Then claim that whatever the harder test is that finally made them fail, was. actually a more sophisticated test of the same thing you were testing them on in the first place.  So “They may have passed a simple theory of mind test, but more sophisticated theory of mind tests showed they still had problems with theory of mind after all.”  In this scenario, it is nearly impossible for an autistic person to win, even if their language problems are relatively subtle most of the time.

How to dismiss everything autistic people have to say about ourselves…

The results of these tests and more, have been used to dismiss the observations of actual autistic people about the workings of our own minds, for a very long time.  It’s true, of course, that everyone can be deceived about how their mind works, and this is one reason that cognitive research exists in the first place.  It exposes things in our cognitive blind spots about ourselves.  But even so, most research assumes that the average person knows at least something about their own mind.

So most psychological research assumes that people are basically accurate in reporting their experiences, but may have certain biases or blind spots that prevent them from understanding some aspects of those experiences.  Psychological research into autism assumes that autistic people can’t understand or report anything of value about ourselves unless it dovetails with what they already believe about us, and that our having insight into ourselves is the exception rather than the rule.  So most people are thought to be basically accurate with some blind spots, autistic people are thought to be all blind spot and little to no accuracy.

And this includes accuracy about things where literally the only person who can say how they experience something is themselves.  Like, if you think in pictures (which some autistic people and many nonautistic people also do), you know that because you see the pictures inside your mind.  Nobody else can come along and say “you’re just deluded, you don’t actually see pictures in your head”.  Because that’s a purely subjective experience, that only the person reporting it can know for sure whether it’s real, and how often it’s real, and in what manner it is real.  But many researchers act like autistic people can’t even know those things about ourselves.  We’re thought to have no capacity for any insight into uur inner lives and experiences.  Even though, when lots and lots of autistic people report a particular experience (spontaneously, without being pressured or coerced into believing it in some way), pretty much always that experience is eventually shown to be accurate, by scientific research.

Another thing that happens is that random things we can’t do get described as having to do with particular theories, when this makes no sense as an assumption.  For instance, our ability to read body language is constantly equated with mindblindness and lack of theory of mind.  Blind people can’t read body language, but people don’t say they’re constitutionally incapable of understanding that other people have subjective experiences of the world.  When autistic people can’t read body language, however, it is always cited as support for the idea that we are constitutionally incapable of understanding that other people have subjective experience of the world.  Even though our reasons for being unable to read body language may be as sensory-based as blind people’s are.   Similarly, language processing problems were for a long time — and it continues to this day, despite research showing why it’s a bad assumption to make — used as the core piece of evidence that autistic people lacked Theory of Mind.

So it’s sort of like, they already decided we lacked Theory of Mind, and then no matter what thing we could be shown to do badly at, that thing was written up as if the thing we did badly at, we did badly at because we lacked Theory of Mind.  So any and every ability we could be shown — rightly or wrongly — to be bad at, was a sign we had poor to nonexistent Theory of Mind.  And entire theories of what autism is, have arisen with those studies at the base of them.  And even though those study results have been seriously called into question by modern research, the Theory of Mind theory continues to be trotted out as if it is meaningful.  Even though it’s built with no foundation whatsoever.  It’s like we’ve done the research that removes the foundation, and yet we keep building on that foundation as if it’s still there.

And again understand — I have no training in research design and no training in autism.  And yet I can see all these holes in autism research, holes that actual well-regarded researchers such as Morton Gernsbacher have specifically and publicly praised me for noticing.  Imagine if I did have the training to understand research design much more intricately than I do.  Imagine what holes I’d be able to point out then, and what research I’d be able to design.

And I’m just one person.  There are tons of autistic people who are capable of similar insights into how messed up the current research is.  Autistic people are not automatically capable of seeing these holes.

Some of us buy into everything we’re told by Autism Experts about being autistic.  Researchers have specifically told me they picked me to work with because I pretty close to never do that these days (I certainly used to do it to a larger extent than I do now).  Whereas a lot of the autistic people they knew, filtered their experiences so heavily through things like Mindblindness and the Extreme Male Brain theory and the like, that the researchers were afraid they weren’t getting accurate information out of them, just regurgitated information.  There’s a related problem where autistic people will hear something from another autistic person, and assume that they have to be similar to that other autistic person.  So if Temple Grandin says autistic people are picture thinkers, suddenly you’ve got all these autistic people saying they’re picture thinkers because Temple Grandin is one.  Even if they’re actually not.

And there are good reasons autistic people do this, some of which are related to general human traits, some of which are related specifically to autistic traits, some of which are related to traits about how autistic people are treated throughout our lives.  So some autistic people may be doing it for reasons that relate to the thing where the more often you read a statement, the more true it feels, something that is true of most human beings whether autistic or not.  Some autistic people may be doing it for reasons that have to do with common autistic traits such as echolalia, poor spontaneous expressive language, language comprehension problems, and difficulty with certain aspects of sensory processing.  Some autistic people may be doing it because of how we have been treated all our lives for being autistic — if you grow up with nobody ever listening to you, being told always that you are wrong and can have no insight into yourself, being (deliberately or otherwise) ridiculed or punished whenever you communicate something of your experience accurately?  You may well start repeating what you’ve been taught to believe about yourself, rather than what you actually experience.

So there are lots of good reasons that autistic people can have trouble reporting our actual experiences around researchers.  But most of us can do it well at least some of the time.  And when we do so in large enough numbers, we usually turn out to be shown to be right.

I also worry about minorities within the broader group of autistic people.  Because I have a lot of traits that are relatively unusual for an autistic person who can use language.  Because those traits tend to make people less likely to develop language.  Not make it impossible, just make it harder.  So a larger number of autistic people without good enough language skills to describe their experiences, exist, than exist people like me who can at least some of the time describe our own experiences.  So anyway — at least within the realm of autistic people who can talk about our experiences, I often find myself an extreme minority.  An extreme minority both in my areas of greatest weakness and in my areas of greatest strength.  People like me get left out of the research results because research takes all autistic people in averaged-out form.  When you average out a huge group of people’s experiences, outliers like me get drowned in the greater sea of data, and you miss out on the great diversity that exists within a group of people like autistic people.

For too long, autistic people have been told who we are, instead of allowed to tell the world who we are.  Test like the Reading the Mind in the Eyes test tend to fill me with an instant mini-explosion of rage.  Because I can see each and every thing I just described to you.  Only I see it all at once.  Like just this ocean of wrongness that washes over me.  And I can see all these things wrong, and feel that the are wrong, and understand that they are wrong.  But until relatively recently, it was so hard to tell people why they were wrong, that most of the time I’d just splutter incoherently or tell them one or two things without giving them the whole picture.

This post is designed to give as broad a picture as I can of what’s wrong with that test, and what the consquences are of it being so wrong.  Please refer back to it as often as you have to, if you’re truly interested in figuring out what’s potentially wrong with both this and other tests used in autism research.  I may not be an academic writer, and I may not have credentials, but I’ve been told by people who are, that I do have insight they need and don’t have.  So do lots of autistic people who lack an education in this field.

If you use ideas from this in research or elsewhere, please give me credit.  Autistic people are often not given credit for our ideas when people mine them for research, so this is more important than usual.

Edited with BlogPad Pro