How to Find and Count Missing Values in a Real Dataset (The Right Way)

Learn how to detect, quantify, and analyze missing values in real datasets using Python and Pandas with practical, production-ready techniques.


Why Missing Data Matters More Than You Think

In real-world datasets, missing values aren’t rare—they’re structural. 

Whether you're ingesting data from APIs, CSVs, or databases, null values can distort analytics, break pipelines, and corrupt machine learning models if not handled correctly.

In Pandas, missing values are typically represented as NaN, None, or NaT. Your job is not just to find them—but to quantify and understand their distribution.


World Bank Dataset: https://data.worldbank.org/indicator/NY.GDP.MKTP.CD


Step 1: Load and Inspect the Dataset

import pandas as pd

df = pd.read_csv("your_dataset.csv")

df.head()
df.info()

df.info() is your first diagnostic tool—it quickly shows non-null counts per column.










Step 2: Find Missing Values (Boolean Mask)

df.isnull()

This returns a DataFrame of True (missing) and False (present). It’s useful for filtering but not for summarization.



Step 3: Count Missing Values Per Column

df.isnull().sum()

This is the most practical command. It gives you a column-wise count of missing values—critical for feature selection and cleaning decisions.



Step 4: Calculate Missing Value Percentages

Raw counts don’t tell the full story. Normalize them:

missing_percent = df.isnull().sum() / len(df) * 100
print(missing_percent)

This helps you identify columns that are:

  • Safe to keep (<5% missing)

  • Require imputation (5–30%)

  • Candidates for removal (>30%)



Step 5: Total Missing Values in the Dataset

df.isnull().sum().sum()

This gives you a global view of data quality across the entire dataset.


Step 6: You can Visualize Missing Data (Optional but Powerful)

import seaborn as sns
import matplotlib.pyplot as plt

sns.heatmap(df.isnull(), cbar=False)
plt.show()

A heatmap reveals patterns—missingness isn’t always random.


Step 7: Filter Rows with Missing Values

df[df.isnull().any(axis=1)]

This isolates problematic rows for deeper inspection or targeted cleaning.


Strategic Insight: Missingness is a Signal

In advanced analytics, missing data can indicate:

  • System failures (e.g., logging gaps)

  • User behavior (e.g., optional fields skipped)

  • Data pipeline issues

Before dropping or imputing, ask: Why is this data missing?


Final Takeaway

Handling missing values is not a cleanup task—it’s a data quality audit step. 

In production-grade pipelines (especially on AWS using tools like Glue or Athena), missing value analysis should be automated and monitored.

If you treat missing data as noise, you’ll miss the signal.



Advance Your Career With 16 Python Projects in Data & ML — All for $288.

Comments

Post a Comment

Popular posts from this blog

How to Filter Rows Using Boolean Indexing in Pandas (Afrobarometer Kenya Dataset)

Image
Learn how to filter rows in Pandas using boolean indexing with real Afrobarometer Kenya survey data. Build clean, ML-ready features with practical examples. Why Boolean Indexing Matters in Feature Engineering In Pandas, boolean indexing is how you select the exact slices of data that become features .  In Module 04 of the course, you’ll no longer exploring—you’ll be deciding what the model sees. In this case, with a dataset like Afrobarometer Kenya Round 10 , filtering is how you: Remove invalid survey responses Isolate specific demographic groups Prepare clean subsets for encoding Load and Inspect the Dataset This assumes that you’ve already converted the .sav file and loaded it: import pandas as pd df = pd.read_csv("afrobarometer_kenya_r10.csv") df.head() 1. Filter Valid Survey Responses Survey datasets contain codes like -1 , 8 , 9 for missing or “Don’t know”. df = df[df["trust_president"] >= 0] This keeps only valid numeric responses for trust in the pres...
Read more

How to Decide Whether to Drop or Fill Missing Value

Image
Learn how to decide whether to drop or fill missing values in datasets using practical, production-ready strategies. Understand trade-offs, bias risks, and data engineering best practices. Missing values are not just a data cleaning nuisance—they’re a signal . The decision to drop or fill them has direct consequences on model performance, statistical validity, and downstream business decisions. If you treat missing data casually, you risk introducing bias, noise, or false confidence into your analysis. This guide gives you a structured, decision-oriented framework you can use in real-world data workflows. 1. First Principle: Understand Why Data Is Missing Before deciding what to do , you need to understand why it’s missing . In data science, this is formalized into three categories: 1. Missing Completely at Random (MCAR) Missingness has no relationship to any variable. Example: A sensor randomly fails. Implication: Safe to drop rows without introducing bias. 2. Missing at Random (MAR...
Read more

How to create your first line chart with World Bank Kenya GDP data

Image
Learn how to pull Kenya GDP data directly from the World Bank API and create your first line chart in Python using pandas and matplotlib—clean, fast, and reproducible. 1. Install Required Libraries !pip install pandas matplotlib wbdata 2. Import Dependencies import pandas as pd import matplotlib.pyplot as plt import wbdata import datetime 3. Define the Data You Need World Bank uses indicators . For GDP (current US$), the indicator is: NY.GDP.MKTP.CD Set the country to Kenya ( KEN ) and define a time range: indicator = {'NY.GDP.MKTP.CD': 'gdp'} data_date = (datetime.datetime(2000, 1, 1), datetime.datetime(2023, 1, 1)) 4. Fetch Data from World Bank df = wbdata.get_dataframe(indicator, country='KEN', data_date=data_date) 5. Clean and Prepare Data df = df.reset_index() df['date'] = pd.to_datetime(df['date']) df = df.sort_values('date') print (df) Explanation df = df.reset_index() → Converts the index into a normal column and resets ro...
Read more