Data Cleaning and Preparation with Python and pandas
Data Cleaning and Preparation is an intermediate-level course of 5 lessons (about 3 h 5 min), with 15 exercises and 15 quiz questions. Prerequisites: Data Analysis with pandas and NumPy. Exercises run in the browser, with nothing to install.
- Python
- Intermediate
- 5 lessons
- 3 h 5 min
- 15 exercises
- Exercises run in the browser
The course in brief
This course teaches you how to turn raw, imperfect data into a reliable dataset, ready for analysis or modeling. You diagnose quality, handle missing values, fix French types and formats, remove duplicates and inconsistencies, then automate everything in a validated pipeline. The examples use realistic exports: CRM, orders, HR, sensors.
What you will be able to do
- Produce a quality report for a dataset (missing values, duplicates, types, impossible values)
- Choose and apply a suitable strategy for missing values, including by group
- Convert numbers and dates in French formats in a robust, controlled way
- Standardize text values and remove duplicates while keeping the right version
- Detect and handle outliers with the IQR and business rules
- Build a reproducible cleaning pipeline with pipe and validation assertions
Syllabus: 5 lessons
- Measure missing values, duplicates and distinct values
- Spot incorrect types and suspicious values
- Produce a summary quality report before any cleaning
Quality diagnosisDetecting missing valuesDetecting duplicates
- 02
Handling Missing Values
- Declare missing-value codes at read time or afterward
- Choose between dropping and imputing depending on the context
- Impute with a constant, a statistic, a per-group statistic or an interpolation
- 03
Fixing Types and Formats
- Convert text to numbers robustly with to_numeric
- Convert dates in various formats with to_datetime
- Choose suitable types: nullable integers, booleans, categories
- 04
Duplicates, Inconsistencies and Outliers
- Standardize text values written in several ways
- Remove duplicates by choosing the row to keep
- Detect and handle outliers with the interquartile range rule
- 05
Restructuring Data and Automating Cleaning
- Switch from a wide format to a long format and back
- Rename and reorganize columns cleanly
- Assemble a reproducible cleaning pipeline validated by assertions
Final project
Cleaning Pipeline for a Multi-Source CRM Export
A small company is merging the customer files from its CRM, its online shop and a trade show (data provided as CSV text in the code, in French formats). Build a pandas pipeline that produces a single, reliable customer table: initial quality report, declaration of missing-value codes, conversion of amounts and dates, standardization of cities and emails, deduplication keeping the most recent and most complete record, justified handling of outliers, then final validation with assertions.
Deliverables
- A script or notebook organized into functions chained with pipe
- The quality report before and after cleaning, with the number of rows modified or removed at each step
- A final validation function (email uniqueness, value ranges, no missing values on required fields)
- A short document listing each cleaning decision and its justification
Assessment criteria
- The pipeline runs end to end and can be replayed without modification on a new export
- Type conversions are controlled (number of failures measured)
- Duplicates are detected after standardization and the row kept is justified
- The handling of missing values and outliers is appropriate and documented
- The raw data is never modified and the final validation passes
Frequently asked questions
Why is data cleaning so important?
An analysis or a model is only as good as the data that feeds it. Duplicates, badly handled missing values or misread dates produce wrong figures without any error message. Cleaning makes results reliable and defensible.
Should missing values be removed or replaced?
It depends on the context: you remove a row if the missing value makes it unusable and few rows are affected, and you impute (median, median by group, interpolation) if the row is still useful. The course gives guidelines and insists on documenting each choice.
Does the course cover French file formats?
Yes: semicolon separators, decimal commas, the euro symbol, non-breaking spaces in thousands and day/month/year dates are handled explicitly.
What level do I need for this course?
You need to know how to work with a pandas DataFrame: reading, selection, filters, groupby. These concepts are covered by the Data Analysis with pandas and NumPy course (PY3).
This course, told for your job
Say what you want to be able to do: “Data Cleaning and Preparation” takes its place in your path, with examples from your field. The concepts stay those of the course.