Research ArticleOpen AccessGoogle Scholar indexed
Outlier Detection Based on Robust Mahalanobis Distance and Its Application
School of Economic, Jinan University, Guangzhou, China
School of Economic, Jinan University, Guangzhou, China
School of Economic, Jinan University, Guangzhou, China
Department of Anatomy, Division of Basic Medicine, YongZhou Vocational Technical College, Yongzhou, China
- 1 School of Economic, Jinan University, Guangzhou, China
- 2 School of Economic, Jinan University, Guangzhou, China
- 3 School of Economic, Jinan University, Guangzhou, China
- 4 Department of Anatomy, Division of Basic Medicine, YongZhou Vocational Technical College, Yongzhou, China
Open Journal of Statistics·Volume 09 (2019)·Pages 15–26·Published 18 January 2019·DOI10.4236/ojs.2019.91002
Copy link · social · email
Abstract
Classical Mahalanobis distance is used as a method of detecting outliers, and is affected by outliers. Some robust Mahalanobis distance is proposed via the fast MCD estimator. However, the bias of the MCD estimator increases significantly as the dimension increases. In this paper, we propose the improved Mahalanobis distance based on a more robust Rocke estimator under high-dimensional data. The results of numerical simulation and empirical analysis show that our proposed method can better detect the outliers in the data than the above two methods when there are outliers in the data and the dimensions of data are very high.
KeywordsMCD EstimatorRocke EstimatorOutlierMahalanobis Distance
- Tukey, J.W. (1977) Exploratory Data Analysis (Vol. 2).
- Drineas, P., Mahoney, M.W. and Muthukrishnan, S. (2006) Sampling Algorithms for l 2 Regression and Applications. Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithm, Society for Industrial and Applied Mathematics, Philadelphia, PA, 1127-1136. https://doi.org/10.1145/1109557.1109682
- Drineas, P., Magdon-Ismail, M., Mahoney, M.W. and Woodruff, D.P. (2012) Fast Approximation of Matrix Coherence and Statistical Leverage. Journal of Machine Learning Research, 13, 3475-3506.
- Drineas, P., Mahoney, M.W., Muthukrishnan, S. and Sarlós, T. (2011) Faster Least Squares Approximation. Numerischemathematik, 117, 219-249. https://doi.org/10.1007/s00211-010-0331-6
- Mahalanobis, P.C. (1936) On the Generalized Distance in Statistics. National Institute of Science of India, 2, 49-55.
- Holland, J.H. (1975) Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence.
- Yan, Z., Dai, X.W. and Tian, M.Z. (2016) Diagnosis of Outlier Based on Sampling of Lever Large Dataset. Mathematical Statistics and Management, 35, 794-802.
- Shi, D.D. Jia, R.Y. and Huang, Y.T. (2009) Improvement of Outlier Detection Algorithm in High Dimension Based on Genetic Algorithm. Journal of Computer Technology and Development, 19, 141-143.
- Stahel, W.A. (1981) Robusteschatzungen: Infinitesimaleoptimalitat und schatzungen von kovarianzmatrizen. Doctoral Dissertation, ETH, Zurich.
- Donoho, D.L. (1982) Breakdown Properties of Multivariate Location Estimators. Technical Report, Harvard University, Boston.
- Rousseeuw, P.J. (1985) Multivariate Estimation with High Breakdown Point. Mathematical Statistics and Applications, 8, 37. https://doi.org/10.1007/978-94-009-5438-0_20
- Grübel, R. (1988) A Minimal Characterization of the Covariance Matrix. Metrika, 35, 49-52. https://doi.org/10.1007/BF02613285
- Rousseeuw, P. and Yohai, V. (1984) Robust Regression by Means of S-Estimators. In: Franke, W.H.J. and Martin, D., Eds., Robust and Nonlinear Time Series Analysis, Springer, New York, 256-272. https://doi.org/10.1007/978-1-4615-7821-5_15
- Rousseeuw, P.J. and Driessen, K.V. (1999) A Fast Algorithm for the Minimum Covariance determinant Estimator. Technometrics, 41, 212-223. https://doi.org/10.1080/00401706.1999.10485670