Saturday, February 13, 2010

Experimental Policy

Sorry for the thin posting recently--just very, very busy.

This is not a new idea, but it is one that I think receives much too little discussion.  Everyone's got their juggernaut policy--their unstoppable idea that they think will make everything work right, or at least a lot better that it currently works.  Sometimes ideas are put in place and sometimes they're not.  Sometimes good ideas get corrupted along the way and morph into bad ideas.  But in the end, it's almost always painfully difficult to evaluate the efficacy of policies good and bad because no one takes care to test the idea before adopting it fully.   If not evaluated experimentally, there are inevitable claims that good policies were actually bad ones, that bad ones were actually good ones, but since the evidence is mucked up by all kinds of the other things that changed at or around the time the policy changed, it's really hard to know.

So what I'd like to advocate for here is an experimental approach to policy.  Before implementing an untested idea, policies should be implemented on a trial basis and evaluated experimentally.  The standard complaint with experimental approaches to policy are that they are unethical.  I disagree.  Or at least I believe they can be structured in ways that could make them both politically palatable and as ethical as any clinical trial of health treatments.  Believe you me, we wouldn't have modern medicine if it weren't for experimental trials.  So what's so unethical about saving a gazillion lives, among many other health benefits?   If ethical concerns are your knee jerk reaction, hear me out and let me know how you feel in the comments.

How should we go about structuring policy experiments?  Well, as anyone who's studied basic experimental design will tell you, the key is randomization.  We need to make sure some participants experience the benefits (or costs) of new policy while others do not, but which group an individual, family, or firm is assigned must be random to ensure that differences in achievement are due to the policy not some other unobserved factor.  When economic spillovers are possible, we need to make sure benefits or costs from one group don't spill over into another--this could be particularly difficult in some policy contexts.

A few concrete ideas may help to motivate the general idea.

One example is education.  Some say we should give up on publicly administered schools and simply issue vouchers for private schools.  I believe there are a number of experimental voucher programs currently underway.  These programs obtain a list of willing participants, which is generally much longer than the available slots, and vouchers are randomly assigned to some portion of participants.  To participate, one must agree to being tracked regardless of whether or not a voucher is received.  Educational performance of those receiving vouchers and those not receiving vouchers can then be tracked over time.  I'm no expert in this area but from what I've read, evidence on the efficacy of vouchers hasn't been especially encouraging.


Michelle Rhee, the District of Columbia's still-new Chancellor, seems to have shaken things up education wise.  But given the way she's gone about implementing her new policies, I think it's going to be extremely difficult to tell whether she has or will actually improve things or not.  If scores go up  some will say it was due to gentrification or other factors. If scores go down Rhee will say it's because the best students left, or something else got in the way.  In the end, we'll just never know.


There are, of course, zillions of ideas for how to improve education.  Teacher training appears to be effective, but good evidence is limited.  Some point to class size.  Some say it's critical to "get the right teachers," but I haven't seen clear evidence on what constitutes a good teacher, how to detect one, or how to go about recruiting them.  But whatever the idea, it surely could be tested in experimental schools that have clearly documented and implemented strategies.  For the experimental design to be valid, assignment of students to schools would have to be random.  This could be done in a way similar to the voucher program:  Parents could voluntarily sign up for a program that would randomly assign their child to one of several experimental schools.  Some would be randomly selected to stay in the status quo.  But to participate, parents must agree to keeping their child in the program to which he/she is randomly assigned, and to having their child tracked.  Since overall participation would be voluntary, I see no ethical problem, despite the random assignment.

This is, after all, how they do experimental trials for health treatments.  It works.  It saves lives.

What about health care?  This would be a toughie given the problem of "adverse selection".  Depending on how different plans were structured, some would choose the plan that was best for their particular circumstances, invalidating any evaluation of the policies.   But suppose the Republicans put together a plan and the Democrats but together a plan and then the people could voluntarily sign up to be randomly assigned to one of the two plans.  One third of the voluntary participants would be randomly assigned to the Democratic plan and one third to the Republican plan, and remaining third would be stuck with the status quo, along with all non-participants.  We could then track health outcomes, costs and other outcomes under each plan and the status quo.  Do this right and make it all very transparent and we might actually see some real information on the 6 o'clock news!

And then there is agricultural policy.  That's supposed to be the topic of this blog, right?

Well, two areas where I've got a few ideas about how policy might be objectively improved pertain to the Federal Crop Insurance Program and the Conservation Reserve Program (CRP).  These programs may be small potatoes compared education and health policy, but I still think its a big deal.  In short, I think we could develop better ways of assigning premiums to farmers enrolling in crop insurance such that they would align more closely to their true risks.  I think this would increase participation rates even if subsidies were lowered a bit.  For CRP, I think signups could be structured in a way that would be easier, more transparent, less costly to the government, and would generate more environmental benefits.  The details of how these things could be done are a little beyond the scope of this post.  Another time maybe....

But surely other academics at Iowa State, UC Davis, UC Berkeley and other places have their own crazy ideas.   So here's what I propose to RMA and FSA  (that's USDA's Risk Management Agency and Farm Service Agency): To improve management and implementation of your various programs, put out a call for proposed changes to the various land grant universities.  Take the top 3-5 proposals as judged by a peer review and test them experimentally using either a voluntary scheme or by randomly selecting counties or jurisdictions.  Make sure the objectives are clear and make the whole process and outcomes transparent to everyone. If the experimental competition were done right, I think we could see amazing gains in efficiency.

Update:  Besides, wouldn't you just love a politician who admitted he didn't have a clue what worked?  And then said he'd get proposals from the best apolitical think tanks and just run an experiment?  Imagine the humility!

9 comments:

  1. Do you really think Iowa State, UC Davis, UC Berkeley and other similar places are "apolitical"? They are all highly political. Throw enough money around and they will even prove global warming if you want them to.

    ReplyDelete
  2. Anonymous:

    Thanks for your comment.

    The scholars I know at Iowa State, UC Davis and UC Berkeley appear to have very different political leanings and prejudices. Moreover, I doubt these would influence the proposals they would have for the agricultural policies I'm talking about here. I sincerely believe their proposals would be serious, well researched, and extremely thoughtful. Surely they could accept proposals from others as well.

    The crux is getting RMA and FSA to be clear about their objectives so that outcomes from the experiments could be objectively and transparently compared.

    Today university professors get private consulting contracts from RMA and all the critical policy ideas and are discussed behind the scenes. Don't you think this would be an improvement?

    ReplyDelete
  3. Anonymous:

    You realize the denialist movement is led by the folks who said smoking doesn't cause lung cancer?

    I want to keep the comments as liberal as possible but my patience for climate-change denailism is wearing thin.

    We here at GG&G are Bayesians. That is, we let the evidence update our prior beliefs. We do not cling religiously to our prior beliefs in the face of overwhelming evidence to the contrary.

    ReplyDelete
  4. This comment has been removed by a blog administrator.

    ReplyDelete
  5. Michael,

    Doesn't this post assume that there is one objectively "best" policy? You acknowledge in discussing health care that there's an adverse selection problem because individuals would pick the plan that is best for them. I counter that this is a good thing - we want individuals to use their local knowledge to choose a policy that would do them the most good! (see anything by Hayek on this topic)

    Similarly with teachers - anything I've read suggests there's no objective way to measure what makes someone a great teacher. Perseverance and effort end up being the determining factors, note how Teach for America looks for leadership well before GPA or any other measurements.

    Even in terms of government, democracy works really well on a small scale but it becomes much harder to implement as a country gets larger. For a large country a truly benevolent dictator would likely have the most effective record in an experiment, but this doesn't mean dictatorship is objectively best.

    I'm not so well-versed in ag policy and your conclusion there looks good to me. But in other areas, I think your assumption that an objectively best policy exists needs to be explicitly stated and considered. (I don't mean for this to be a harsh comment, I did enjoy your post haha)

    ReplyDelete
  6. Yes Pete, objectives matter a lot! I've seen testimony before congress where experts rail against the congress people for not being clear enough about their objectives.

    But don't confuse the issues here: Given a clear objective, or even ambigous or competing objectives, we can best discern the impacts of policies through experimentation.

    Adverse selection *is* a huge problem if you're trying to discern cause and effect. If the proposed mechanism is to have individuals select among contracts optimally, then that has to be done after the randomization.

    People may disagree on objectives and what should be done. But I think that's usually a small issue relative to differences of opinion about what a policy will actually do.

    In education I think there needs to be more effort to define clear outcomes. Standardized testing has serious limitations, but it's better than nothing. A bigger problem is that often they don't track the performance of individual teachers very well, by looking at improvement of individual students and how much that improvement sticks around over time. Classroom/school/district averages obscure a lot, needlessly.

    "Objective" for ag. policy is equally difficult. But again, here I think I know what policies have aimed to do mainly from what I've seen them do in the past. So I'm not ascribing my values here; I'm infering objectives from earlier policies, and I'm thinking they can do what they appear to be trying to do a whole lot better.

    Of course, it would be nice if FSA and RMA made their objectives clearer in the first place. That's a political challenge that is a little beyond me.

    ReplyDelete
  7. Micheal, I'm very supportive of your approach, but I'm surprised you don't reference the enormous literature in "adaptive management" that embraces randomized trials but goes even further.

    ReplyDelete
  8. its really very nice and informative posting thanks for sharing this with us..

    ReplyDelete
  9. One question about objectives (with a little cynicism added)...will this hypothetical call for experimental policies include the long list of interest groups who must be appeased? And in the order in which they are ranked? My feeling toward many of our policies in the US is that they are so diluted by the time they actually are passed because of the constant juggling between major stakeholders. Sometimes I wonder if voters even care...

    ReplyDelete

Renewable energy not as costly as some think

The other day Marshall and Sol took on Bjorn Lomborg for ignoring the benefits of curbing greenhouse gas emissions.  Indeed.  But Bjorn, am...