Data Science

What is Data Imputation, and How Can You Use it to Handle Missing Data?

Written by John Terra
|
Updated on August 14, 2024

This probably comes as no surprise to you, but data isn’t perfect. It’s normal, even expected that there will be instances of inaccurate or missing information among the countless terabytes of data generated each day. Today, we’re discussing the latter.

This article discusses data imputation, including its definition, importance, techniques, use cases, and more. We’ll also share a data science bootcamp that can help you gain practical skills to open up new career opportunities.

Let’s start with the question, “What is data imputation?”

So, What is Data Imputation?

Data imputation is a data processing technique that retains most of the data set’s data and information by replacing missing data with substituted values. This process is preferable to outright discarding incomplete data samples.

Also Read: Technology at Work: Data Science in Finance

What Kinds of Missing Data Does Imputation Address?

Missing at Random (MAR)

Concerning MAR, absent data isn’t random and can be explained by other observed dataset variables. For instance, people working night shifts may be less likely to respond to a survey than one conducted during daytime hours. Their lack of response relates to work schedules, an observed variable.

Missing Completely at Random (MCAR)

MCAR situations occur when the reason for the value’s absence is completely random and unrelated to other variables in the datasets. For instance, a survey respondent inadvertently skips a question, which results in a missing value.

‍Missing Not at Random (MNAR)

MNAR occurs when the absence of data directly relates to the value itself, even after accounting for other variables. For instance, in the field of mental health research, people with more severe symptoms are less likely to complete assessments because of their condition. In this example, missing data directly relates to the severity of unobserved symptoms, not the assessment itself.

Why is Data Imputation Important?

Data imputation is a critical step in data preprocessing for several key reasons, including:

It avoids bias. Eliminating cases with missing data may lead to biased results, especially if missing values aren’t random. Imputation addresses this problem by retaining those critical data points and strategically filling in the missing values.
It completes the analysis. Missing data often leads to incomplete datasets, which may compromise the reliability and validity of statistical analyses, especially in smaller datasets. Imputation helps retain the original sample size, allowing users to perform accurate analysis and obtain actionable results.
‍It facilitates the use of machine learning models. Many machine learning algorithms rely on complete datasets to learn patterns effectively and make accurate predictions. Imputation ensures that every variable in the dataset has a value, facilitating the algorithms’ practical application.
It encourages compliance with data standards. Research and industry standards typically need datasets to meet particular thresholds regarding missing values. These may specify specific imputation techniques or acceptable missing data levels. Thus, imputation helps analysts and researchers adhere to the standards, ensuring the datasets are suitable for broader use and comparisons.
It reduces the need for more data collection. Although imputation doesn’t resolve the root causes of missing data, it can help replace the dataset’s missing values. This process reduces the need to recollect data and the associated costs.

Also Read: Five Outstanding Data Visualization Examples for Marketing

Data Imputation Techniques

Plenty of different techniques are used in data imputation. For example:

Average or linear interpolation. This technique calculates between the previous and subsequent accessible values, substituting the missing value. This process resembles previous/next value imputation but only works with numerical data. It’s essential to sort the data in advance accurately.
Fixed value. This universal technique replaces null data with a fixed value and can apply to all data types. For example, interviewers can impute a null value in a survey by using “not answered.”
K nearest neighbors. This technique’s objective is finding the k nearest examples where the value in the relevant feature is present and then substituting it with the feature value that most frequently occurs in the group.
Maximum or minimum value. The range’s minimum or maximum can be used as a replacement cost for a missing value if the user knows the data must fit within a given range and if they are aware from the data collection process that the measurement instrument ceases recording and the message extends further than one within such boundaries. For example, if a financial exchange’s price cap has been reached and the exchange procedure has stopped, missing prices can be substituted using the exchange boundary’s minimum value.
Next or previous value. There are specific imputation techniques for time-series data or ordered data. These techniques consider the data set’s sorted structure, where closer values are more comparable than further ones. When imputing incomplete data in a time series, the next or previous value within the time series is usually substituted for the missing value. This strategy works for both nominal and numerical values.
Missing value prediction. This technique uses a machine learning model to determine the ultimate imputation value for the “x” characteristic based on other features. This popular model is trained with the values in the remaining columns, plus the rows in feature “x” that lack values are employed as the training set. Depending on the feature type, we can employ any regression or classification model in this situation.
Most frequent value. This popular technique uses the most frequent value in the column to replace missing values. It is effective for both numerical and features.
Mean, median, rounded mean. Median, mean, or rounded mean are imputation techniques for numerical features. The method replaces the null values with median, mean, or rounded mean values determined for that feature across the entire dataset. Using the median rather than the mean is best if the dataset has many outliers.

Multiple Imputation Explained

Single imputation handles an unknown missing value like a true value by substituting a single value. Consequently, a single imputation overlooks uncertainty and almost always understates variation. However, multiple imputations can solve this issue, as they can account for uncertainties within and between imputations.

Multiple data imputation approaches generate “n” suggestions for each missing value. Each of these “n” values gets a plausible value, and “n” fresh datasets are produced as if a straightforward imputation had occurred in each data set.

In this way, a single table column generates “n” new data sets, each examined using particular techniques. Subsequently, these analyses are combined to produce or consolidate the data set’s results.

Here are the basic multiple imputation steps:

Create a collection of “n” values to be imputed for each attribute in the data set record that lacks a value.
Perform a statistical analysis on every data set using one of the “n” replacement ideas generated in the previous item.
Create a set of results by combining the findings of the different analyses.

Also Read: Data Science Bootcamps vs. Traditional Degrees: Which Learning Path to Choose?

Challenges in Data Imputation

Data imputation has its obstacles, including:

Data bias and distortion. Imputation can introduce bias and distort the data if not implemented carefully. Imputed values may not accurately represent the missing true values, which may lead to incorrect analysis and interpretations. Using appropriate imputation methods to minimize bias and maintain data integrity is vital.‍
Difficulties in evaluating imputation quality. There’s no guaranteed foolproof way to verify imputed value accuracy. Imputation models function by assuming particular relationships between variables to estimate missing values. However, if the imputation models don’t correctly capture the underlying relationships, the imputed values may be inaccurate and misleading.‍
Types of missing data. The imputation method changes significantly depending on the type of missing data (e.g., MCAR, MAR, or MNAR). However, it can be challenging to determine the missing data type, as it often depends on assumptions that might not always be valid.‍
There are high computational demands. Methods such as multiple imputation or complex statistical modeling can be computationally expensive, notably for larger datasets. Data scientists often require technical and domain expertise to balance accuracy and computational resource efficiency.
Working with heterogeneous data has limited reliability. Finding an imputation method that works seamlessly can be complicated if a dataset contains a mixture of numerical, categorical (non-numerical), and other data types.

What is Data Imputation, and What Are its Use Cases?

Many different sectors use data imputation methods to address missing data issues. Here are some ways these methods are used in real-world situations:

Finance. Financial Institutions might have incomplete records thanks to unrecorded transactions or system errors. Techniques like regression imputation can help estimate missing financial figures, a vital component of accurate financial reporting and practical risk assessment.‍
Healthcare. Patient data generated in clinical trials may have missing values due to a lack of responses or dropout. Multiple imputation is typically employed here because it can generate a series of possible imputations, ensuring the analysis results will be based on a full intended sample size.‍
Image processing. Computer vision images may have missing pixels because of transmission errors or sensor defects. In these cases, matrix completion techniques reconstruct missing portions based on surrounding pixel values and patterns.‍
Sensor data. Sensors in many Internet of Things (IoT) applications may intermittently fail to record data at random intervals. For example, a temperature sensor in a smart home might fail to record data for a few hours. Maybe the cat played with the sensor. Interpolation techniques help estimate missing values based on available readings from surrounding periods.

Also Read: Data Scientist vs. Machine Learning Engineer

Ready to Learn More About Data Science?

Data science is an exciting field, and if you want to learn more about it, consider a post graduate program in data science. This 44-week online bootcamp teaches data science concepts and provides practical experience working with tools like SQL, Python, Tableau for data visualization, machine learning, etc. You will also gain exposure to generative AI tools like ChatGPT, DALL-E, Midjourney, and more.

Glassdoor.com shows data scientists can make a yearly average salary of $112,874. Consider this course if you need skills to help you navigate our data-driven job market.