When business stakeholders ask data experts seemingly simple questions, they often expect a quick and straightforward answer. On the surface, it seems like a piece of cake. But in the messy reality of data, what appears to be a simple question can quickly turn into a multi-layered onion of a problem - each layer revealing increasing complexity and ambiguity. Let’s take a practical example from the insurance industry. A sales executive asks:
Categories
Recent Posts: Data
Simple Questions, Hard Answers
Data Leakage in Machine Learning
Recently, I read a thread on Twitter about several Machine Learning papers that contained severe cases of data leakage. The authors of the papers seemed unaware of this phenomenon and therefore trained models that performed exceptionally well. Unfortunately, this was mainly due to data leakage. Not many beginners are aware of this problem and in my opinion, not many courses emphasize this issue early enough. Therefore, I would like to tell you all the things you need to know about data leakage and some ways to prevent it in this post.
Where to Eat in Munich?
I recently moved to a new city - Munich! I live in a very calm area, but soon realized that the neighborhood is not really the best when it comes to eating outside. So, I decided to try to analyze review data from the web to find out which area is most compelling for me and other foodies. I scraped online reviews, cleaned the data and then visualized it on a map, showing the average rating of restaurants in different areas in Munich.
k-Nearest Neighbors
k-Nearest Neighbors, or k-NN as I am going to call it from now on, is one of the easiest algorithms to solve classification tasks. It can be used for regression problems as well, but I am going to focus on the more common use case of classification in this post. In a nutshell, k-NN will assign a new data point to the class that the majority of its k neighbors in the training set belongs to. Let’s use another coffee-related example to see how that works.
Detect Forged Banknotes with a Logistic Regression
Counterfeit money is a serious problem for both individuals and businesses. Counterfeiters constantly find new ways and techniques to produce fake banknotes, that are essentially indistinguishable from real money. At least for the human eye! Identifying forged banknotes is a typical example of a binary classification task in Machine Learning. If we have enough data of both real and forged banknotes, we can use this data to train a model that can classify new banknotes as either real or fake.