sample project · synthetic data
data cleaning and preparation
a deliberately messy order export turned into a clean, analysis-ready table, with a log of every single change.
- Python
- pandas
- SQL
- 318raw rows in
- 273clean rows out
- 18duplicates removed
- 27rows set aside for review
before and after
the same orders, before and after cleaning
the first six rows of the raw export, and the same orders after cleaning. look at the dates, the line names and the weights. SO-1081 is missing after cleaning because its quantity is not usable, so it was set aside for review.
before: raw export
| Order ID | Order Date | Customer | Line | Material | Weight | Quantity | Status |
|---|---|---|---|---|---|---|---|
| SO-1129 | 28 Jun 2026 | Bluebird Components | line b | ALUMINIUM | 18,3 kg | 321 | shipped |
| SO-1253 | 2026-05-27 | Harbor Metals | LINE-C | steel | 15,71 | 756 | open |
| SO-1221 | 2026-06-27 | northwind works | LINE-A | aluminium | 2,47 | 879 | shipped |
| SO-1007 | 27.05.2026 | Kestrel Tooling | Line A | Aluminium | (empty) | 731 | cancelled |
| SO-1097 | 13.03.2026 | harbor metals | Line C | brass | 10.9 kg | 323 | open |
| SO-1081 | 27.05.2026 | NORTHWIND WORKS | LINE-A | ALUMINIUM | 18,10 | -164 | shipped |
after: clean table
| Order ID | Order Date | Customer | Line | Material | Quantity | Status | Weight_kg |
|---|---|---|---|---|---|---|---|
| SO-1007 | 2026-05-27 | Kestrel Tooling | Line A | aluminium | 731 | cancelled | nan |
| SO-1097 | 2026-03-13 | Harbor Metals | Line C | brass | 323 | open | 10.9 |
| SO-1129 | 2026-06-28 | Bluebird Components | Line B | aluminium | 321 | shipped | 18.3 |
| SO-1221 | 2026-06-27 | Northwind Works | Line A | aluminium | 879 | shipped | 2.47 |
| SO-1253 | 2026-05-27 | Harbor Metals | Line C | steel | 756 | open | 15.71 |
what every step changed
| step | rows affected |
|---|---|
| Duplicate rows removed | 18 |
| Values with stray spaces trimmed | 174 |
| Customer / material names standardised | 256 |
| Line labels unified (Line A / line a / LINE-A / A) | 185 |
| Dates converted to one format (YYYY-MM-DD) | 203 |
| Weights converted to kg (g and decimal commas) | 204 |
| Weights missing or not a number (left empty, flagged) | 26 |
| Rows rejected: quantity missing, negative or implausible | 27 |
all data in this project is synthetic: random numbers with a fixed seed, no real company or person. a row can be counted in more than one step.
from question to result
how the project went
the challenge
a raw order export had duplicate rows, three date formats, inconsistent names (Line A, line a, LINE-A, A), weights in kg or g with decimal commas, missing values and impossible quantities. nothing could be summed or compared reliably.
the approach
I cleaned the file with a script in six clear steps. every step is logged with the number of rows it changed, and rows that cannot be trusted are set aside in a separate file instead of silently disappearing.
the result
318 raw rows became 273 clean, analysis-ready rows. 18 duplicates were removed and 27 rows with unusable quantities were set aside for review.
the data and the code
everything is in the project folder
you can open every file, rerun the scripts and get the same result. nothing is hidden behind a screenshot.
- README.mdwhat the project is and how to run it992 B
- data/raw_orders_export.csvthe messy raw export22 KB
- src/clean.pythe cleaning script, step by step4 KB
- output/clean_orders.csvthe clean, analysis-ready table18 KB
- output/rejected_rows.csvrows set aside for review2 KB
- output/cleaning_log.csvthe log of what every step changed399 B
run it yourself
pip install pandas
python src/generate_data.py
python src/clean.pyfree consultation
got a messy export nobody trusts?
message me on whatsapp and tell me about your data. the first consultation is free.
chat with me on whatsapp
not ready to chat? send me an email instead.