Lord’s Paradox with Three Statisticians

AERA Annual Meeting · April 2006

Lord’s Paradox with Three Statisticians — AERA Annual Meeting, April 2006.
Published

April 5, 2006

Event
Conference
Slides
16
Format
Converted from Beamer

This deck is an image-based copy of the original slides. Builds and animations are flattened; the slide text is reproduced below for search and screen readers.

Slide text
  1. Lord’s Paradox with Three Statisticians Damian W. Betebenner Lynch School of Education Boston College American Educational Research Association Annual Meeting San Francisco, April 9th, 2006 D. W. Betebenner Lord’s Paradox with Three Statisticians
  2. Introduction History of Lord’s Paradox Analysis Lord’s Paradox Solution Discussion Lord’s Paradox In an article entitled A Paradox in the Interpretation of Group Comparisons (1967), Lord introduced a deceptively simple conundrum that defied attempts at resolution for nearly 15 years: A large university is interested in investigating the effects on the students of the diet provided in the university dining halls and any sex differences in these effects. Various types of data are gathered. In particular, the weight of each student at the time of his arrival in September and his weight the following June are recorded [Lord, 1967, p. 304]. D. W. Betebenner Lord’s Paradox with Three Statisticians
  3. Introduction History of Lord’s Paradox Analysis Lord’s Paradox Solution Discussion Lord’s Paradox The paradox arises from two seeminly conflicting analyses of the data performed by two statisticians: Statistician 1 Performs a gain score analysis for the two sexes and shows zero average gain for both. Statistician 2 Performs an ANCOVA analysis and demonstrates a significant difference between the adjusted June weights of the two sexes. The paradox: One analysis indicates that the diet produces a differential effect for the two sexes while the other analysis suggests that it does not. D. W. Betebenner Lord’s Paradox with Three Statisticians
  4. Introduction History of Lord’s Paradox Analysis Lord’s Paradox Solution Discussion Lord’s Paradox Redux Lord (1973) extended the paradox to situations involving two groups and two treatments: . . . consider the problem of evaluating federally funded special education programs. A group of disadvantaged children are pretested in September, then enrolled in a special program, and finally posttested in June. A control group of children are similarly pretested and posttested but not enrolled in the special program. Since the most disadvantaged children are selected for the special program, the control group will typically have higher pretest scores than the disadvantaged group [Lord, 1973, p. 235]. D. W. Betebenner Lord’s Paradox with Three Statisticians
  5. Introduction History of Lord’s Paradox Analysis Lord’s Paradox Solution Discussion Lord’s Paradox Solution An elegant solution to the paradox was provided by Holland and Rubin (1983) in a Festschrift devoted to Lord. The paradox lies with the incompatible causal inferences one wishes to draw from the data: differential effectiveness versus no-differential effectiveness. In both forms of Lord’s Paradox, a causal conclusion rests on an assumption concerning an unobservable control group. Holland and Rubin derive necessary conditions for the control group in order that the causal conclusions hold. Since the control group is unobservable, these conditions must be tested based upon knowledge of the given situation. D. W. Betebenner Lord’s Paradox with Three Statisticians
  6. Introduction History of Lord’s Paradox Analysis Lord’s Paradox Solution Discussion Rubin’s Causal Model Causality [Holland, 1986] with regard to effect t, relative to effect c, is conceived as the difference in outcome on unit u as measured by Y : Yt (u) − Yc (u) It is impossible to simultaneously observe and measure t and c applied to u. Scientists work around this difficulty by exploiting homogeneity and invariance assumptions with regard to the observational units. Statisticians, for example, consider average causal effect by utilizing random assignment of units: Effect = (Yt − Yc ) = (Yt ) − (Yc ) D. W. Betebenner Lord’s Paradox with Three Statisticians
  7. Introduction Analysis Discussion Using Teachers as the treatment The scenarios utilized within Lord’s paradox are not unlike current situations in which one wishes to determine the quality of a given teacher: Students, assigned to one of four teachers, are administered a pre-test at the beginning of the school year and a post-test at the end of the school year. Three statisticians, attempting to determine teacher efficacy, perform three types analyses: gain score analysis, ANCOVA, and mixed-effects analysis using using pre-test and post-test scores. To investigate this scenario, “toy" data is produced for the four teachers. D. W. Betebenner Lord’s Paradox with Three Statisticians
  8. Introduction Analysis Discussion Rubin’s Causal Model Assuming one could assign students at random to both teachers and “control", the average causal effects associated with each teacher, T , is given by: ∆i = (Yt − Yc |T = i ) To compare teachers i1 and i2 , one computes the difference of the average causal effects: = (Yt |T = i1 ) − (Yc |T = i2 ) − (Yt |T = i1 ) − (Yc |T = i2 ) = (Yt |T = i1 ) − (Yt |T = i2 ) − (Yc |T = i1 ) − (Yc |T = i2 ) D. W. Betebenner Lord’s Paradox with Three Statisticians
  9. Introduction Analysis Discussion Assumptions Estimating differential teacher effect requires one to make certain assumptions concerning what would happen to the same children under “control" conditions. The three statisticians make the following assumptions concerning their control groups: Statistician 1 The gain score analysis assumes the unobserved control group’s post-test score would equal their pre-test score, Yc = a. In some situations [Wainer, 1991] this is a justifiable assumption. Statistician 2 The ANCOVA analysis assumes the post-test score is a linear function of the pre-test score, Yc = a + bX . Statistician 3 The Mixed-effects analysis (random intercepts/fixed slopes) has analogous assumption to ANCOVA. The difference being the BLUPs of the residual effects. D. W. Betebenner Lord’s Paradox with Three Statisticians
  10. Introduction Analysis Discussion Lord’s Paradox Illustration Pretest/Posttest data for four teachers Posttest Pretest D. W. Betebenner Lord’s Paradox with Three Statisticians
  11. Introduction Analysis Discussion Lord’s Paradox Illustration Pretest/Posttest data for four teachers Pretest=Posttest Posttest Pretest D. W. Betebenner Lord’s Paradox with Three Statisticians
  12. Introduction Analysis Discussion Lord’s Paradox Illustration Pretest/Posttest data for four teachers Pretest=Posttest Posttest Pretest D. W. Betebenner Lord’s Paradox with Three Statisticians
  13. Introduction Analysis Discussion Lord’s Paradox Illustration Pretest/Posttest data for four teachers Pretest=Posttest Posttest Pretest D. W. Betebenner Lord’s Paradox with Three Statisticians
  14. Introduction Analysis Discussion Comparison’s of three different effects Gain Scores 4 ANCOVA Mixed−Effects Effect Teacher D. W. Betebenner Lord’s Paradox with Three Statisticians
  15. Introduction Analysis Discussion Summary Different analyses can lead to irreconcilable conclusions regarding the efficacy of teachers. These differences arise from the causal claims made about the teachers under consideration. Following Rubin’s Causal Model, reconciling these differences requires one to make arguments concerning the behavior of unobservable control groups. Mixed-effects analyses, like gain score and ANCOVA analyses, have accompanying assumptions that must be fulfilled in order that causal claims hold. D. W. Betebenner Lord’s Paradox with Three Statisticians
  16. Introduction Analysis Discussion References Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, Holland, P. W. and Rubin, D. B. (1983). On Lord’s paradox. In Wainer, H. and Messick, S., editors, Principles of Modern Psychological Measurement, pages 3–25. Lawrence Erlbaum Associates, Hillsdale, New Jersey. Lord, F. M. (1967). A paradox in the interpretation of group comparisons. Psychological Bulletin, 68(5):304–305. Lord, F. M. (1973). Lord’s paradox. In Anderson, S. B., Ball, S., and Murphy, R. T., editors, Encyclopedia of Educational Evaluation, pages 232–236. Jossey-Bass, San Francisco. Wainer, H. (1991). Adjusting for differential base rates: Lord’s paradox again. Psychological Bulletin, 109(1):147–151. D. W. Betebenner Lord’s Paradox with Three Statisticians