This "puzzle" was so articulated in a series of very influential papers by Angus Deaton and Guy Laroque. They reasoned that commodity prices should be autocorrelated to some extent due to the buffering effects of storage. In plentiful times some production is saved; in bad years, inventories are drawn down to supplement production. Consumption is therefore smoother than production and prices are less volatile that they would be without storage, but are autocorrelated even though production shocks can appear random (like from the weather). The puzzle came about from the formal modeling of this process and some sophisticated techniques they used to fit that model to observed commodity prices. That model indicated a certain amount of autocorrelation but nowhere near as much as we observe in reality.
So, all this might sound very technical, and it is, but it is pretty fundamental for understanding why we occasionally have huge commodity price spikes like the one in 2008, which can cause starvation, malnutrition, and civil conflict in developing countries. When staple food gets too expensive the world turns very ugly.
A new paper by Carlo Cafiero, Eugenio and Juan Bobenrieth, and Brian Wright (one of my professors at Berkeley) in the Journal of Econometrics (sorry, only a gated version) have a pretty clear solution to this puzzle. It turns out that Deaton and Laroque's puzzling findings stemmed mainly from imperfect calibration of their model that came from using approximation methods that were too coarse. I'd guess computers simply weren't fast enough back then to make super fine approximations. Cafiero et al. simply improved the resolution of the approximation, obtained better estimates of the few key parameters, and can now model commodity prices that look very much like the real world.
The crux, it seems, is that demand for commodities is extremely inelastic. Thus, when inventories get very low, prices spike very high. This creates a strong incentive to keep a lot of inventories in speculation for the day when prices do spike. So we rarely get stock outs but when we do, or get close, prices can go very high. That's pretty much what happened in 2008. It may have been precipitated somewhat by the ethanol boom that increased grain demand more than expected. In the initial years after the boom, inventories were probably too low for the new demand regime, which made the market particularly susceptible to inventory draw downs and a huge price spike.
To my mind, this is one of the best examples of a dynamic rational expectations economic model that actually works.
Update: Below is a picture of the solution and how a poor approximation missed what was critical. The essential problem involves finding a function that relates price (p) to the available supply (z). Available supply includes inventories carried over from the last period plus current production, which can vary randomly. The 'kink' in the line is where storage reaches zero and the market consumes everything that's available. So the upper part of the price function is just the consumption demand curve. When prices are below that line, the amount stored is the horizontal difference between the price function and the dashed line that follows from the straight line above the kink. So, this is the essential picture that tells when its optimal to sell high and buy low when it comes to commodities. The kink is critical--that's where prices can really start to spike more in relation to small shocks. The red line is Deaton and Laroque's approximation, which doesn't quite fit to correct (more finely callibrated) line. It's especially far off near the kink, which is key. The incorrectly smooth price function suggested prices less autocorrelated that we observe. But it was all just approximation error, not necessarily a fundamental flaw in the theory.
