Welcome back to our series of articles sponsored by Intel \u2013 \u201cAsk a Data Scientist.\u201d Once a week you\u2019ll see reader submitted questions of varying levels of technical detail answered by a practicing data scientist \u2013 sometimes by me and other times by an Intel data scientist. Think of this new insideBIGDATA feature as a valuable resource for you to get up to speed in this flourishing area of technology. If you have a data science question you\u2019d like answered, please just enter a comment below, or send an e-mail to me at: daniel@insidehpc.com. This week\u2019s question is from a reader who asks for an explanation of data leakage.<\/p>\n
Q: What is data leakage?<\/strong><\/p>\n A: <\/strong>As a data scientist, you should always be aware of circumstances that may cause your machine learning algorithms to over-represent their generalization error as this may render them useless in the solution of real-world problems. One such potential problem is called data leakage<\/em> \u2013 when the data you are using to train a machine learning algorithm happens to have the information you are trying to predict. It is undesirable on many levels such as a source of poor generalization and over-estimation of expected performance. Data leakage often occurs subtly and inadvertently and may result in overfitting. A leading text in the field called data leakage as one of the top ten machine learning mistakes.[1]<\/p>\n Data leakage can manifest in many ways including:<\/p>\n In general, data leakage comes from two sources in a machine learning algorithm \u2013 the feature variables, and the training set. A trivial example of data leakage would be a model that uses the response variable itself as a predictor, thus concluding for example that \u201cit is sunny on sunny days.\u201d<\/p>\n As a more concrete example, consider the use of a \u201ccustomer service rep name\u201d feature variable in a SaaS company churn prediction algorithm. Using the name of the rep who interviewed a customer when they churned might seem innocent enough until you find out that a specific rep was assigned to take over customer accounts where customers had already indicated they intended to churn. In this case, the resulting algorithm would be highly predictive of whether the customer had churned but would be useless for making predictions on new customers. This is an extreme example – many more instances of data leakage occur in subtle and hard-to-detect ways. There are war stories of algorithms with data leakage running in production systems for years before the bugs in the data creation or training scripts were detected.<\/p>\n Identifying data leakage beforehand and correcting for it is an important part of improving the definition of a machine learning problem. Many forms of leakage are subtle and are best detected by trying to extract features and train state-of-the-art algorithms on the problem. Here are several strategies to find and eliminate data leakage:<\/p>\n Once data leakage has been identified, the next step is to figure out how fix it (or even if you want to try). For some problems, living with data leakage without attempting to fix it could be acceptable. But if you decide to fix the leakage, care must be taken not to make matters worse. Usually, when there is one leaking feature variable, there are others. Removing the obvious leaks that are detected may exacerbate the effect of undetected ones, and engaging in feature modification in an attempt to plug obvious leaks, could create others. The idea is to try to figure out the legitimacy of specific observations and\/or feature variables and work to plug the leak and hopefully seal it completely. Rectifying data leakage is an active field of research that will likely yield effective results in the near future.<\/p>\n [1] Nisbet, R., Elder, J. and Miner, G. 2009. Handbook of Statistical Analysis and Data Mining Applications<\/em>. Academic Press.<\/p>\n If you have a question you\u2019d like answered, please just enter a comment below, or send an e-mail to me at: daniel@insidehpc.com.<\/p>\n Data Scientist:<\/strong> Daniel D. Gutierrez \u2013 Managing Editor, insideBIGDATA<\/p>\n","protected":false},"excerpt":{"rendered":" Welcome back to our series of articles sponsored by Intel \u2013 \u201cAsk a Data Scientist.\u201d Once a week you\u2019ll see reader submitted questions of varying levels of technical detail answered by a practicing data scientist \u2013 sometimes by me and other times by an Intel data scientist. This week\u2019s question is from a reader who asks for an explanation of data leakage.<\/p>\n","protected":false},"author":37,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[239,87,180,210,67,56,1],"tags":[95],"acf":[],"yoast_head":"\n\n
\n