Experimental Policy

Sorry for the thin posting recently--just very, very busy.

This is not a new idea, but it is one that I think receives much too little discussion.  Everyone's got their juggernaut policy--their unstoppable idea that they think will make everything work right, or at least a lot better that it currently works.  Sometimes ideas are put in place and sometimes they're not.  Sometimes good ideas get corrupted along the way and morph into bad ideas.  But in the end, it's almost always painfully difficult to evaluate the efficacy of policies good and bad because no one takes care to test the idea before adopting it fully.   If not evaluated experimentally, there are inevitable claims that good policies were actually bad ones, that bad ones were actually good ones, but since the evidence is mucked up by all kinds of the other things that changed at or around the time the policy changed, it's really hard to know.

So what I'd like to advocate for here is an experimental approach to policy.  Before implementing an untested idea, policies should be implemented on a trial basis and evaluated experimentally.  The standard complaint with experimental approaches to policy are that they are unethical.  I disagree.  Or at least I believe they can be structured in ways that could make them both politically palatable and as ethical as any clinical trial of health treatments.  Believe you me, we wouldn't have modern medicine if it weren't for experimental trials.  So what's so unethical about saving a gazillion lives, among many other health benefits?   If ethical concerns are your knee jerk reaction, hear me out and let me know how you feel in the comments.

How should we go about structuring policy experiments?  Well, as anyone who's studied basic experimental design will tell you, the key is randomization.  We need to make sure some participants experience the benefits (or costs) of new policy while others do not, but which group an individual, family, or firm is assigned must be random to ensure that differences in achievement are due to the policy not some other unobserved factor.  When economic spillovers are possible, we need to make sure benefits or costs from one group don't spill over into another--this could be particularly difficult in some policy contexts.

A few concrete ideas may help to motivate the general idea.

One example is education.  Some say we should give up on publicly administered schools and simply issue vouchers for private schools.  I believe there are a number of experimental voucher programs currently underway.  These programs obtain a list of willing participants, which is generally much longer than the available slots, and vouchers are randomly assigned to some portion of participants.  To participate, one must agree to being tracked regardless of whether or not a voucher is received.  Educational performance of those receiving vouchers and those not receiving vouchers can then be tracked over time.  I'm no expert in this area but from what I've read, evidence on the efficacy of vouchers hasn't been especially encouraging.


Michelle Rhee, the District of Columbia's still-new Chancellor, seems to have shaken things up education wise.  But given the way she's gone about implementing her new policies, I think it's going to be extremely difficult to tell whether she has or will actually improve things or not.  If scores go up  some will say it was due to gentrification or other factors. If scores go down Rhee will say it's because the best students left, or something else got in the way.  In the end, we'll just never know.


There are, of course, zillions of ideas for how to improve education.  Teacher training appears to be effective, but good evidence is limited.  Some point to class size.  Some say it's critical to "get the right teachers," but I haven't seen clear evidence on what constitutes a good teacher, how to detect one, or how to go about recruiting them.  But whatever the idea, it surely could be tested in experimental schools that have clearly documented and implemented strategies.  For the experimental design to be valid, assignment of students to schools would have to be random.  This could be done in a way similar to the voucher program:  Parents could voluntarily sign up for a program that would randomly assign their child to one of several experimental schools.  Some would be randomly selected to stay in the status quo.  But to participate, parents must agree to keeping their child in the program to which he/she is randomly assigned, and to having their child tracked.  Since overall participation would be voluntary, I see no ethical problem, despite the random assignment.

This is, after all, how they do experimental trials for health treatments.  It works.  It saves lives.

What about health care?  This would be a toughie given the problem of "adverse selection".  Depending on how different plans were structured, some would choose the plan that was best for their particular circumstances, invalidating any evaluation of the policies.   But suppose the Republicans put together a plan and the Democrats but together a plan and then the people could voluntarily sign up to be randomly assigned to one of the two plans.  One third of the voluntary participants would be randomly assigned to the Democratic plan and one third to the Republican plan, and remaining third would be stuck with the status quo, along with all non-participants.  We could then track health outcomes, costs and other outcomes under each plan and the status quo.  Do this right and make it all very transparent and we might actually see some real information on the 6 o'clock news!

And then there is agricultural policy.  That's supposed to be the topic of this blog, right?

Well, two areas where I've got a few ideas about how policy might be objectively improved pertain to the Federal Crop Insurance Program and the Conservation Reserve Program (CRP).  These programs may be small potatoes compared education and health policy, but I still think its a big deal.  In short, I think we could develop better ways of assigning premiums to farmers enrolling in crop insurance such that they would align more closely to their true risks.  I think this would increase participation rates even if subsidies were lowered a bit.  For CRP, I think signups could be structured in a way that would be easier, more transparent, less costly to the government, and would generate more environmental benefits.  The details of how these things could be done are a little beyond the scope of this post.  Another time maybe....

But surely other academics at Iowa State, UC Davis, UC Berkeley and other places have their own crazy ideas.   So here's what I propose to RMA and FSA  (that's USDA's Risk Management Agency and Farm Service Agency): To improve management and implementation of your various programs, put out a call for proposed changes to the various land grant universities.  Take the top 3-5 proposals as judged by a peer review and test them experimentally using either a voluntary scheme or by randomly selecting counties or jurisdictions.  Make sure the objectives are clear and make the whole process and outcomes transparent to everyone. If the experimental competition were done right, I think we could see amazing gains in efficiency.

Update:  Besides, wouldn't you just love a politician who admitted he didn't have a clue what worked?  And then said he'd get proposals from the best apolitical think tanks and just run an experiment?  Imagine the humility!