Determining the causal effect of special education is a critical topic when making educational policy that focuses on student achievement. However, current special education research is facing challenges from persistent selection bias and complex confounding. Bayesian Additive Regression Trees (BART) is employed in this study to provide a flexible estimation of the academic performance. Targeted Maximum Likelihood Estimation (TMLE) is also integrated into the BART model, supporting doubly robust estimation of the special education effect. This study extracted survey data from the Early Childhood Longitudinal Study, Kindergarten Class (ECLS-K), to estimate the causal impact of special education status on students’ combined mathematics and reading achievement scores. The analysis results of the BART-TMLE model show that children receiving special education services demonstrated approximately 9 points lower scores on average for combined math and reading scores, even adjusting for a considerable number of covariates, compared to their peers who did not receive these services. The estimated negative treatment effect persists after controlling for observed covariates that are closely correlated to the combined test score. The negative effect likely reflects unobserved factors, such as the underlying severity of learning disabilities, parent involvement and other potential traits, which are actual factors that determine the placement of special education status, rather than indicating the ineffectiveness of special education service. The achievement gap in academic performance reflects the current observable status of special education. The estimated effect could be improved by future research incorporating educational domain knowledge, allowing the model to be constructed more accurately.
Blackorby, J., Wagner, M., Cameto, R., Davies, E., Levine, P., Newman, L., Marder, C., Cardoso, D. and Garza, N. (2005) Engagement, Academics, Social Adjustment, and Independence: The Achievements of Elementary and Middle School Students with Disabilities. National Center for Special Education Research. https://www.seels.net/designdocs/engagement/All_SEELS_outcomes_10-04-05.pdf
Wagner, M., Newman, L., Cameto, R., Garza, N. and Levine, P. (2005) After High School: A First Look at the Postschool Experiences of Youth with Disabilities. National Center for Special Education Research. https://files.eric.ed.gov/fulltext/ED494935.pdf
Hanushek, E.A., Kain, J.F. and Rivkin, S.G. (2002) Inferring Program Effects for Special Populations: Does Special Education Raise Achievement for Students with Disabilities? Review of Economics and Statistics , 84, 584-599. https://doi.org/10.1162/003465302760556431
Shifrer, D., Muller, C. and Callahan, R. (2011) Disproportionality and Learning Disabilities: Parsing Apart Race, Socioeconomic Status, and Language. Journal of Learning Disabilities , 44, 246-257. https://doi.org/10.1177/0022219410374236
Feng, Y. (2024) Assessing the Effectiveness of Special Education Services on Fifth Grade Math Scores: Using Traditional and Machine Learning Methods with ECLS-K Data. Applied and Computational Engineering , 45, 7-16. https://doi.org/10.54254/2755-2721/45/20241019
Setoguchi, S., Schneeweiss, S., Brookhart, M.A., Glynn, R.J. and Cook, E.F. (2008) Evaluating Uses of Data Mining Techniques in Propensity Score Estimation: A Simulation Study. Pharmacoepidemiology and Drug Safety , 17, 546-555. https://doi.org/10.1002/pds.1555
Rosenbaum, P.R. and Rubin, D.B. (1983) The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika , 70, 41-55. https://doi.org/10.1093/biomet/70.1.41
Stuart, E.A. (2010) Matching Methods for Causal Inference: A Review and a Look Forward. Statistical Science , 25, 1-21. https://doi.org/10.1214/09-sts313
Austin, P.C. (2011) An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies. Multivariate Behavioral Research , 46, 399-424. https://doi.org/10.1080/00273171.2011.568786
Hill, J.L. (2011) Bayesian Nonparametric Modeling for Causal Inference. Journal of Computational and Graphical Statistics , 20, 217-240. https://doi.org/10.1198/jcgs.2010.08162
Tourangeau, K., Nord, C., Lê, T., Pollack, J.M. and Atkins-Burnett, S. (2006) Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K), Com-Bined User’s Manual for the ECLS-K Fifth-Grade Data Files and Electronic Code-Books (NCES Publication No. 2006-032). U.S. Department of Education, National Center for Education Statistics. https://nces.ed.gov/pubs2006/2006032_2.pdf
Imai, K. and Ratkovic, M. (2014) Covariate Balancing Propensity Score. Journal of the Royal Statistical Society Series B : Statistical Methodology , 76, 243-263. https://doi.org/10.1111/rssb.12027
Morgan, P.L., Frisco, M.L., Farkas, G. and Hibel, J. (2010) A Propensity Score Matching Analysis of the Effects of Special Education Services. The Journal of Special Education , 43, 236-254. https://doi.org/10.1177/0022466908323007
Imberman, S.A. (2014) The Effect of Special Education Services on Student Achievement. Journal o f Public Economics , 118, 98-110. https://doi.org/10.2139/ssrn.1031693
Sullivan, A.L. and Field, S. (2013) Does Special Education Improve Preschoolers’ Academic Skills? CASTL Research Brief. Curry School of Education, University of Virginia. https://files.eric.ed.gov/fulltext/ED544029.pdf
Chipman, H.A., George, E.I. and McCulloch, R.E. (2010) BART: Bayesian Additive Regression Trees. The Annals of Applied Statistics , 4, 266-298. https://doi.org/10.1214/09-aoas285
Tourangeau, K., Nord, C., Lê, T., Sorongon, A.G. and Najarian, M. (2009) Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K), Com-Bined User’s Manual for the ECLS-K Eighth-Grade and K-8 Full Sample Data Files and Electronic Codebooks (NCES Publication No. 2009-004). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. https://nces.ed.gov/ecls/data/eclsk_k8_manual_part1.pdf
Kapelner, A. and Bleich, J. (2016) BartMachine: Machine Learning with Bayesian Additive Regression Trees. Journal of Statistical Software , 70, 1-40. https://doi.org/10.18637/jss.v070.i04
Dorie, V., Hill, J., Shalit, U., Scott, M. and Cervone, D. (2019) Automated versus Do-It-Yourself Methods for Causal Inference: Lessons Learned from a Data Analysis Competition. Statistical Science , 34, 43-68. https://doi.org/10.1214/18-sts667
Green, D.P. and Kern, H.L. (2012) Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees. Public Opinion Quarterly , 76, 491-511. https://doi.org/10.1093/poq/nfs036
Friedman, J.H. (2001) Greedy Function Approximation: A Gradient Boosting Machine. The Annals of Statistics , 29, 1189-1232. https://doi.org/10.1214/aos/1013203451
Breiman, L. (2001) Random Forests. Machine Learning , 45, 5-32. https://doi.org/10.1023/a:1010933404324
Chipman, H.A., George, E.I. and McCulloch, R.E. (1998) Bayesian CART Model Search. Journal of the American Statistical Association , 93, 935-948. https://doi.org/10.1080/01621459.1998.10473750
Tan, Y.V. and Roy, J. (2019) Bayesian Additive Regression Trees and the General BART Model. Statistics in Medicine , 38, 5048-5069. https://doi.org/10.1002/sim.8347
George, E.I. and McCulloch, R.E. (1993) Variable Selection via Gibbs Sampling. Journal of the American Statistical Association , 88, 881-889. https://doi.org/10.1080/01621459.1993.10476353
Rubin, D.B. (1974) Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. Journal of Educational Psychology , 66, 688-701. https://doi.org/10.1037/h0037350
Hahn, P.R., Murray, J.S. and Carvalho, C.M. (2017) Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects. SSRN Electronic Jo urnal . https://doi.org/10.2139/ssrn.3048177
Pearl, J. (2009) Causality. 2nd Edition, Cambridge University Press. https://doi.org/10.1017/cbo9780511803161
Linero, A.R. (2018) Bayesian Regression Trees for High-Dimensional Prediction and Variable Selection. Journal of the American Statistical Association , 113, 626-636. https://doi.org/10.1080/01621459.2016.1264957
Ročková, V. and van der Pas, S. (2020) Posterior Concentration for Bayesian Regression Trees and Forests. The Annals of Statistics , 48, 2108-2131. https://doi.org/10.1214/19-aos1879
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., et al. (2018) Double/Debiased Machine Learning for Treatment and Structural Parameters. The Econometrics Journal , 21, C1-C68. https://doi.org/10.1111/ectj.12097
Tibshirani, R. (1996) Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society Series B : Statistical Methodology , 58, 267-288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x
Zou, H. and Hastie, T. (2005) Regularization and Variable Selection via the Elastic Net. Journal of the Royal Statistical Society Series B : Statistical Methodology , 67, 301-320. https://doi.org/10.1111/j.1467-9868.2005.00503.x
Belloni, A., Chernozhukov, V. and Hansen, C. (2014) Inference on Treatment Effects after Selection among High-Dimensional Controls. The Review of Economic Studies , 81, 608-650. https://doi.org/10.1093/restud/rdt044
Wasserman, L. and Roeder, K. (2009) High-Dimensional Variable Selection. The Annals of Statistics , 37, 2178-2201. https://doi.org/10.1214/08-aos646
Fithian, W., Sun, D. and Taylor, J. (2014) Optimal Inference after Model Selection. arXiv:1410.2597. https://doi.org/10.48550/arXiv.1410.2597
Rinaldo, A., Wasserman, L. and G’Sell, M. (2019) Bootstrapping and Sample Splitting for High-Dimensional, Assumption-Lean Inference. The Annals of Statistics , 47, 3438-3469. https://doi.org/10.1214/18-aos1784
Berk, R., Brown, L., Buja, A., Zhang, K. and Zhao, L. (2013) Valid Post-Selection Inference. The Annals of Statistics , 41, 802-837. https://doi.org/10.1214/12-aos1077
Gruber, S. and van der Laan, M.J. (2012) tmle: An R Package for Targeted Maximum Likelihood Estimation. Journal of Statistical Software , 51, 1-35. https://doi.org/10.18637/jss.v051.i13
Van der Laan, M.J. and Rose, S. (2011) Targeted Learning: Causal Inference for Observational and Experimental Data (pp. 89-135, 165-210). Springer. https://doi.org/10.1007/978-1-4419-9782-1
Porter, K.E., Gruber, S., van der Laan, M.J. and Sekhon, J.S. (2011) The Relative Performance of Targeted Maximum Likelihood Estimators. The International Journal of Biostatistics , 7, 1-34. https://doi.org/10.2202/1557-4679.1308
Schuler, M.S. and Rose, S. (2017) Targeted Maximum Likelihood Estimation for Causal Inference in Observational Studies. American Journal of Epidemiology , 185, 65-73. https://doi.org/10.1093/aje/kww165
Zheng, W. and van der Laan, M.J. (2011) Cross-Validated Targeted Minimum-Loss-Based Estimation. In: van der Laan, M.J. and Rose, S., Eds., Targeted Learning , Springer, 459-474. https://doi.org/10.1007/978-1-4419-9782-1_27
Imbens, G.W. and Rubin, D.B. (2015) Causal Inference for Statistics, Social, and Biomedical Sciences. Cambridge University Press. https://doi.org/10.1017/cbo9781139025751
Hahn, J. (1998) On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects. Econometrica , 66, 315.-331 https://doi.org/10.2307/2998560
Hirano, K., Imbens, G.W. and Ridder, G. (2003) Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score. Econometrica , 71, 1161-1189. https://doi.org/10.1111/1468-0262.00442
Belloni, A., Chernozhukov, V., Chetverikov, D., Hansen, C. and Kato, K. (2017) High-Dimensional Econometrics and Regularized GMM. arXiv:1806.01888. https://doi.org/10.48550/arXiv.1806.01888
Athey, S. and Imbens, G.W. (2019) Machine Learning Methods That Economists Should Know about. Annual Review of Economics , 11, 685-725. https://doi.org/10.1146/annurev-economics-080217-053433
Wainwright, M.J. (2019) High-Dimensional Statistics. Cambridge University Press. https://doi.org/10.1017/9781108627771
Kurz, A., Elliott, S.N., Wehby, J.H. and Smithson, J.L. (2010) Alignment of the Intended, Planned, and Enacted Curriculum in General and Special Education and Its Relation to Student Achievement. The Journal of Special Education , 44, 131-145. https://doi.org/10.1177/0022466909341196
Burke, M.A. and Sass, T.R. (2013) Classroom Peer Effects and Student Achievement. Journal of Labor Economics , 31, 51-82. https://doi.org/10.1086/666653