Joe Francis is doing more replications.
He finds that Galor and Özak’s claims on “the agricultural origins of time preference” <https://www.aeaweb.org/articles?id=10.1257/aer.20150020> attain statistical significance only because their “area weights” variable is wired into the analysis exactly backward: instead of downweighting small areas that people would find it easy to move away from, they downweight large areas where people are more likely to stick.
You have a prior—a strong, theoretically motivated, Malthusian-selection prior that patience got culturally bred into populations where the agronomic return to waiting was high. You run the specification. It comes out significant and correctly signed. And here is the trap that catches all of us who are even passable Bayesians: you do not audit results that confirm what you already believe. You audit the surprises. Confirmation is where your guard is down. Oded and Ömer divided where they should have multiplied, the number came out the color they expected, and nobody re-derived the weight variable because nobody had a reason to. It happens. It has very nearly happened to me.
Thus: “The estimator gives the largest weight to the smallest regions. A tiny 34 km² Swiss region receives 170,000 times the weight of the largest region. The influence of small, urban regions where internal migration is most severe is thereby maximized, making the problem that area weights were supposed to address worse…”
And once you wire things in the right way, the coefficient loses statistical significance completely and loses its sign: it is not the case that higher potential crop yields and shorter growth cycles that rewarded delayed gratificatication are associated with the development of a more patient human cultural matrix, but rather the reverse.
The broken academic incentive structure is Francis’s deeper target: even a demonstrable error won’t trigger retraction or correction, because the “top five” journals essentially never do so.
<https://poorrichworld.blog/p/running-out-of-patience-with-economics> <https://poorrichworld.blog>
When doing post-publication peer review, it is easy to become pessimistic about the whole human enterprise of knowledge production. It is easy to run out of patience due to the errors one finds. Furthermore, even if I find a fault in one of the articles published in the “top five” economics journals, I know that it will not be retracted or even corrected.
The article “The Agricultural Origins of Time Preference” (2016) by Oded Galor and Ömer Özak is a case in point. Published in the American Economic Review, the article presents a grand narrative of persistence. Galor and Özak hypothesize that “geographical variations in the natural return to agricultural investment generated a persistent effect on the distribution of time preference across societies” (p. 3065). Higher potential crop yields and shorter growth cycles, they say, rewarded delayed gratification, so a culture of long-term orientation was selected for over generations and persists in populations today.
The article has garnered over 1,000 citations on Google Scholar, but it is in no way, shape, or form robust.
The evidence for this narrative comes from a simple correlation between individuals in the World Values Survey and historical FAO data on potential crop yields and growth cycles in the places of their ancestral origins. Macro-historical regressions across countries are, however, famously vulnerable to confounding factors such as geography, modern national institutions, or contemporary culture, which can easily drive the correlation.
Galor and Özak therefore run a version of their regressions with country fixed effects. Individuals are compared within each country, and national institutions drop out. In the published article that specification is statistically significant: historical crop yields and patience, across 1,356 regions.
But the problem is that people have moved. The agricultural characteristics of a person’s current location might not then reflect the environment that supposedly shaped their ancestors’ time preference.
They claim to deal with this by using area weights. As they put it, “observations are weighted by the scale of each region, mitigating the effect of internal migration” (p. 3097). Larger regions, on this logic, are less susceptible to internal migration crossing their borders, so assigning them higher weights should reduce this measurement error.
Nonetheless, their code then does the opposite.
For weights, it uses a variable called “invwvsarea.” When that variable is reverse engineered, it turns out that it is in fact area weights inverted. Why they did this, is not clear, but it has the effect of aggravating the issue that they claim to be addressing. By weighting by the inverse of the area, the estimator gives the largest weight to the smallest regions. A tiny 34 km² Swiss region receives 170,000 times the weight of the largest region. The influence of small, urban regions where internal migration is most severe is thereby maximized, making the problem that area weights were supposed to address worse.
When the actual, non-inverted area weights are used instead, the coefficient for crop yield drops from 0.032 (p=0.010) to an insignificant −0.033 (p=0.39), as shown in Table 1. Indeed, all their main results essentially disappear. And the same happens when no weights are used at all: the coefficient falls to 0.001 (p=0.85), indistinguishable from zero. Their results are artifacts of doing the opposite of what is described in the text.
Yet this probably does not matter. Even though the article is not robust, it will not be retracted, or even corrected. That will not be possible because the “top five” economics journals do not do such things. As the Economist recently observed, we know that things are much better in economics than in the rest of the humanities and the social sciences because the “five leading journals have seen just four withdrawals in their combined 570-year history.” Their peer review is so robust that it would be impossible for an article like this to be published in the first place. Consequently, those area weights cannot really have been inverted.
The article examined in this post is:
Galor, Oded, and Ömer Özak. “
The Agricultural Origins of Time Preference.”American Economic Review106, no. 10 (2016): 3064–3103.Data and replication archive.
The replication package for this post is available here.
<https://poorrichworld.blog/p/running-out-of-patience-with-economics> <https://poorrichworld.blog>
- Brad DeLong here*: I part company with Joe is the counsel of despair—the bleak “it will never be retracted, so none of this matters.” I don’t buy it. Retraction is the wrong metric. The right metric is what the profession
It is true, to highlight what Joe writes:
When doing post-publication peer review, it is easy to become pessimistic about the whole human enterprise of knowledge production. It is easy to run out of patience due to the errors one finds. Furthermore, even if I find a fault in one of the articles published in the “top five” economics journals, I know that it will not be retracted or even corrected…. [Oded Galor and Ömer Özak’s] results are artifacts of doing the opposite of what is described in the text. Yet this probably does not matter. Even though the article is not robust, it will not be retracted, or even corrected. That will not be possible because the “top five” economics journals do not do such things…
Nevertheless, I think Joe is wrong to despair here. Yes, it looks like Oded and Ömer divided where they should have multiplied—it happens—and did not catch it because the results were as they expected and, as good Bayesians, you don’t spend that much time checking expected results—that also happens. But things that do not replicate are weeded out, albeit much more slowly than they should be. And people do have a very healthy skepticism about results that are close to the edge of statistical or economic significance, or that cannot be shown to rely on a strong solid correlation.
It is not just computational error. It is also that, say, you have ten yes-no decisions to make in running any empirical analysis that could go either way. What you should do is to choose the way that makes most sense to you, and run along several tracks where things are debateable. But there is the temptation to choose the track that gets you closer to the result you think is sensible. Succumb to that, and you wind up with the strongest of 1024 possible results. Alternatively, get yourself checked by somebody hostile for ideological reasons, and they can come up with the weakest of 1024 possible results as a way of dismissing your analysis.
Stepping back, the persistence literature keeps reaching for a lever—settler mortality, ancestral crop yields, ruggedness, caloric suitability—that will let a cross-sectional correlation masquerade as a causal parameter, without a structural model that says how the past reaches into the present.
It turned me into a Heckmanite.
I do not believe you have earned the right to say “causal” unless you can write down the structural model and the story about why the identifying variation is even remotely exogenous. “Deep roots” regressions almost never do. The area weight was a half-hearted patch over the wound that people can and do move to opportunity. Inverting it tore the patch off and rubbed salt in.
But things that do not replicate do get downgraded; they stop anchoring dissertations; the second generation of citations turns sour. It is a shame it takes a decade and a Joe Francis rather than an afternoon and a journal’s own referees. We should be able to do much better.
Which is why we need preregistration and replication packages. And that is why Joe Francis undertaking this mission is a very good thing for the profession as a whole.
##public-reason
##crosspost
##joes-subheadline-a-strange-case-of-inverted-area-weights-but-i-mostly-want-to-praise-joe-francis-for-spending-so-much-of-his-time-on-post-publication-peer-review
#joe-francis-running-out-of-patience-with-economics
#joe-francis
#running-out-of-patience-with-economics
#post-publication-review
#agricultural-origins
#replication-crisis
#garden-of-forking-paths
#persistence-studies
#heckmanite
#causal-inference
#preregistration
#replication-packages
#economic-history
#confirmation-bias
