Measurement validity and deceptive economic statistics in autocracies

Use validation checks to develop meaningful outcome indicators

How can we develop valid indicators for intangible concepts, like the quality of democracy, the credibility of an election, or the fairness of a judicial system? A seminal paper on outcome measurement describes the main types of evidence for measurement validity, such as content validity and convergent/discriminant validity. Content validity refers to measuring all elements of a concept and only elements of that concept. For example, an indicator of universal suffrage for adults would lack content validity if it only measured male suffrage or if it included suffrage for children. Convergent/discriminant validity refers to statistical relationships between the indicator and other measures. For example, a measure of an individual’s commitment to democracy should correlate with their belief in elections but not with their love of football. This research describes a process that development practitioners can use to validate outcome indicators.

Satellite imagery suggests autocracies lie about their economic growth

Researchers have long sought an independent measure of economic growth. Administrative statistics often miss the informal sector and can suffer from inaccurate government reporting, among other limitations. After multiple researchers showed how satellite imagery can accurately measure economic activity, other researchers have used satellite imagery to investigate the veracity of official economic growth numbers. A recent paper compared reported GDP growth to the GDP growth predicted by satellite imagery and found that autocracies overstate their annual GDP growth by 35%. The pattern suggests strategic manipulation by autocrats: reported GDP in autocracies seems most inflated (a) for the government spending portion of GDP, (b) when their true growth is lower than the world average, and (c) when economic growth will not make them ineligible for foreign aid. This research shows how accurate measurement is necessary for us to make accurate inferences about the world.

Do you have a study we should share for a future Facty Friday? Send an email to drg.el@usaid.gov!