Wednesday, March 17, 2010

One voice among thirty - why one person's vote matters

It's been a week since I announced that I was joining the review boards at Autism Speaks. Many of you supported me, but a few of you doubted my ability to exert much influence as the lone Aspergian on the board.

In particular, people have suggested that one person’s vote can’t make any difference to thirty scientists who share a review board with one misfit (me.) The implication is, the thirty scientists vote as they wish, and my lone vote is ignored. That perception is totally wrong, and I’d like to take a moment to show you why . . .

To understand the significance of one person’s vote, please join me as we review and rate some promising (sample) medical research. I’m describing the general process of review on the Autism Speaks boards, but the process is essentially the same for the NIH and other government agencies, and many big private foundations.

The process starts as research proposals pour into the agency. The first step is to ensure applicants met all the ground rules to qualify for the funds at hand. Agency staff does that work. This first screening may weed out half the proposals. The rejected proposals go back to the researchers with various notes and suggestions. Many researchers clean up their applications and come back again so this is really an iterative process.

Next, the proposals that made it through the initial screening reach the review board – the place I serve. Proposals are dealt out to members of the board for a first ranking. Much of the time, three reviewers read each proposal. They may be assigned randomly, or they may be dealt out by expertise. However they are allocated, if there are 30 of us on the board, and there are 100 proposals to deal with, we will each be assigned ten.

We rate the proposals we are given in several areas, like the impact on the community, how likely the work is to succeed, and whether it’s truly new research or a rehash of something already covered. Each area is scored from 1-5, or perhaps 1-7. So a proposal that I (or any of us) rated 4,4,5,5,3 in each of five areas would have a composite score of 4.2

The three initial reviewer scores are combined for a total score, and the proposals are ranked based on this first pass. At that point, staffers take the funds available for allotment and they see how far down into the ranks the money goes. For example, if we have twenty million dollars to distribute, that might be enough to fund the top third of the applications.

Given that, the agency takes all the proposals in the top third, plus a cut of the next tier, for final review. That’s where we all discuss them, and we all vote. And that’s where any one voice can really matter a lot. I’ll give you an example. Let’s say a piece of research involves social skills training, and most of the scientists gave it a 3 for importance. But I feel that it’s a really important proposal, based on my life experience, so I speak up. By doing so, I cause people around the room to rethink the proposal’s importance, and a number of people move their score from 3 to 4 or even 5. The result: that proposal’s average score rises, which moves it from “not good enough to fund” into the “recommended for funding” category.

The same thing can happen in reverse. If I said a particular proposal was irrelevant, its score might drop, and it would sink in the ranking allowing other more important work to be funded.

Any of us have the power to move ranking in this way. You might say our influence is directly proportional to how much we believe and our oratory skill.

The idea that I would be ignored or bullied – as some have suggested – is simply not realistic. Any of you who’ve met me in person know I’m six foot three and loud, so I’m hard to ignore. And these are gentle people. As Marlon Brando suggested in The Godfather, they are receptive to reasonable arguments.

The system I described varies a little bit from one agency to another but I think you can get the general idea. In the first round, the proposals are graded by teams of three, and the size of the entire board really does not matter. In the second round, a persuasive speaker can move a room of fifty people as effectively as he can move five people. So once again, the size of the board is not really indicative of any one person’s importance.

I guess being a reviewer is really what you make of it. If you’re meek as a lamb your vote won’t change anything. But if you are passionate, and articulate, you can really accomplish a lot.

And that’s what I intend to do, every time I vote.

4 comments:

jess said...

'But if you are passionate and articulate you can accomplish a lot.'

You've already proven that to be true, John - long before putting it into practice in the board room. I don't ever doubt your ability to make yourself heard.

Thank you for standing strong and making a difference.

BlogStalker said...

I'd be surprised if anyone doubts you personally. It's the organization that has a lot of repair work to do in order to make me a believer.

Silk

John said...

I hope that they have made a mistake in their selection of you (or genuinely want your input).

Also, you mentioned a few areas you are interested in... Naturally, TMS research that you've been activily involved in, as well as community funding. Could you comment on what other research areas you may push for greater funding, as well as areas that AutismSpeaks may currently fund that you feel are of lesser importance?

According to their site, you are on two boards - the Science Advisory Board and the Treatment Advisory board. There are also a Tissue Advisory Board, Innovative Treatment Advisory Board, AGRE Steering Committee, ATN Steering Committee, ATP Steering Committee, ATP Exectutive Committee, and ACTN Steering Committee. The AutismSpeaks website does not provide much explaination of the functions and differences of each of these boards. Could you comment?

autismspeaks said...

Thank you for looking at the Autism Speaks website to obtain a better understanding of the different ways in which the organization receives input and advice from both academic and stakeholder partners. The website you are referring to which lists the boards and committees is here: http://www.autismspeaks.org/science/overview/advisory_boards/index.php

Essentially there are two "types" of oversight bodies. An advisory board and an advisory committee. The advisory boards, together with outside experts in targeted fields, provide structured input on the merits of applications for Autism Speaks resources. This includes funding support, and in the case of the Tissue Advisory Board, advice on the best way to distribute previous post-mortem brain tissue for research.

Complementary to these advisory boards are advisory committees which provide input and expertise specific to one of the special initiatives or clinical programs which Autism Speaks supports. Most often this has to do with potential new activities, priorities, leadership and organization within that initiative or program. rather than guidance about Autism Speaks investment of resources. These include AGRE, or Autism Genetic Resource Exchange, ATN, or Autism Treatment Network, ATP or Autism Tissue Program and ACTN or Autism Clinical Trials Network. Autism Speaks also recruits scientific and community input on an ad-hoc basis to advise on other issues on more of a short-term basis when needed.

The purposes of these differing advisory boards and committees is to not only provide a wide range of expertise and insight to advise the organization, but to also provide more targeted input in specific scientific areas that could benefit from particular insights. These needs may differ from program to program. Autism Speaks tries to include, when possible, a parent representative or stakeholder in each of these committees and advisory boards.

We hope this is helpful to you.