RCT estimators can be misleading with a wrong theory - an example
Mar. 2023
Here’s a fictional example where an RCT would provide a nice, unbiased effect – and yet could lead to wrong conclusions. The story takes place in a generic airport. First-class passengers board the airplane and arrange their bags.
2nd-class passengers enter next and take the rest of the overhead compartment. Finally, 3rd-class is invited into the airplane. They try squeezing their bags, but there is little room left; some luggage must be checked. Stuff falls on people. Babies cry. Departure is now late.
All passengers are unhappy. Some report to the airline that “3rd-class people are causing delays” and quickly point to the social roots of such untidy behavior. They are proud of their diagnosis and happy to contribute to a better world, a world with punctual flights.
The airline manager, a fan of TED talks who always likes it when people tweet “correlation is not causation,” dismisses the observational evidence and orders a randomized trial: during the next month, half of the flights will not have 3rd-class passengers.
The manager also decides that everything else, including the number of 1st, 2nd, and 3rd-class seats, should be kept constant on all flights. “Balanced covariates,” he whispers to himself.
The results are in, and a team of statisticians concludes: flights with 3rd-class passengers are more likely to depart late, lots of stars on the table. Armed with causal powers, the manager concedes that the 3rd class people were indeed the root of the problem. Right?
Also published here.