Published / updated29 Aug 2026

Files, CSV and JSON

Before you start

You need: exceptions (Article 05).

Time: about 45 minutes, plus the practice.

Learning objective

Read and write files without corrupting data through wrong encoding, wrong paths or unclosed handles.

Topics

• Opening files and modes
• Encoding
• pathlib
• Reading and writing text
• CSV
• JSON
• Writing safely
• Diagnosing file errors

Opening files

with open("students.csv", "r", encoding="utf-8") as file:
    content = file.read()

Mode	Does
"r"	Read — default, fails if missing
"w"	Write — truncates an existing file
"a"	Append
"x"	Create — fails if it exists
"rb" / "wb"	Binary
"r+"	Read and write

"w" destroys the file's contents the moment it opens, before you write anything. Opening the wrong path in "w" mode has lost real data.

"x" is the safe create: it raises FileExistsError rather than overwriting.

Always use with. Without it, an exception between open and close leaks the handle, and on Windows the file stays locked:

# Wrong
file = open("students.csv")
data = file.read()      # an exception here leaks the handle
file.close()

Encoding

# Wrong — uses the platform default
with open("students.csv") as file:
    ...

# Right
with open("students.csv", encoding="utf-8") as file:
    ...

Always specify encoding. The default is locale-dependent: UTF-8 on Linux and macOS, and historically cp1252 on Windows. The same code then reads the same file differently on two machines.

For a school system holding names in Telugu, Hindi or Tamil, the wrong encoding produces UnicodeDecodeError or silently mangled characters — and once written wrongly, the original is gone.

# A file written by Excel on Windows often carries a BOM
with open("students.csv", encoding="utf-8-sig") as file:
    ...

utf-8-sig strips the byte-order mark. Without it the first column name reads RollNumber and every lookup on it fails — a genuinely confusing bug, because the name looks right when printed.

with open("legacy.csv", encoding="utf-8", errors="replace") as file:
    ...

errors="replace" substitutes `` for undecodable bytes rather than raising. Use it to inspect a broken file, never to import one — it destroys data silently.

pathlib

from pathlib import Path

data_dir = Path("data")
students_file = data_dir / "students.csv"

students_file.exists()
students_file.is_file()
students_file.suffix           # ".csv"
students_file.stem             # "students"
students_file.name             # "students.csv"
students_file.parent           # Path("data")

data_dir.mkdir(parents=True, exist_ok=True)

for path in data_dir.glob("*.csv"):
    print(path.name)

for path in data_dir.rglob("*.csv"):    # recursive
    print(path)

/ joins paths correctly on every platform. String concatenation with "/" or "\\" breaks on one of them.

content = students_file.read_text(encoding="utf-8")
students_file.write_text(content, encoding="utf-8")

For small files, read_text and write_text handle opening and closing.

Relative paths

# Depends on the working directory — breaks when run from elsewhere
path = Path("data/students.csv")

# Relative to this source file — stable
BASE_DIR = Path(__file__).resolve().parent
path = BASE_DIR / "data" / "students.csv"

A relative path resolves against the current working directory, not the script's location. Running python src/main.py from the project root and from inside src gives two different paths — and the second reports FileNotFoundError for a file that clearly exists.

Path(__file__).resolve().parent is the fix, and it belongs in any script that reads data files.

Reading text

with open(path, encoding="utf-8") as file:
    content = file.read()              # whole file as one string

with open(path, encoding="utf-8") as file:
    lines = file.readlines()           # list of lines, newlines included

with open(path, encoding="utf-8") as file:
    for line in file:                  # one line at a time — memory-efficient
        print(line.rstrip("\n"))

Iterating the file object streams it. read() on a 2 GB log loads 2 GB into memory; the loop uses a buffer.

rstrip("\n") removes the trailing newline. Plain .strip() also removes leading whitespace, which matters when indentation is significant.

Writing text

with open(path, "w", encoding="utf-8") as file:
    file.write("Roll number,Name\n")
    file.write("NCA-2024-0012,Ravi Kumar\n")

with open(path, "w", encoding="utf-8") as file:
    file.writelines(f"{s.roll_number},{s.name}\n" for s in students)

with open(path, "a", encoding="utf-8") as file:
    file.write("NCA-2024-0044,Sneha Patel\n")

write does not add a newline. writelines does not either — despite the name, it writes the strings as given.

CSV

Never parse CSV by splitting on commas.

# Broken by any name containing a comma
for line in file:
    parts = line.strip().split(",")

A parent named Reddy, Vijay produces one extra field, every subsequent column shifts, and the import silently writes wrong data.

import csv

with open(path, newline="", encoding="utf-8") as file:
    reader = csv.DictReader(file)

    for row in reader:
        print(row["RollNumber"], row["Name"])

newline="" is required. Without it, the csv module and Python's newline translation both act, producing blank rows between records on Windows.

DictReader uses the header row for keys, so a reordered column does not break the code — unlike index access.

with open(path, newline="", encoding="utf-8") as file:
    reader = csv.reader(file)
    header = next(reader)

    for row in reader:
        print(row[0], row[1])

csv.reader gives lists. next(reader) consumes the header — omitting it processes the header row as data.

Writing

with open(path, "w", newline="", encoding="utf-8") as file:
    writer = csv.DictWriter(file, fieldnames=["RollNumber", "Name", "ClassName", "Marks"])
    writer.writeheader()

    for student in students:
        writer.writerow({
            "RollNumber": student.roll_number,
            "Name": student.name,
            "ClassName": student.class_name,
            "Marks": student.marks
        })

The writer quotes and escapes correctly — a name containing a comma, a quote or a newline round-trips.

writer = csv.DictWriter(file, fieldnames=fields, extrasaction="ignore")

extrasaction="ignore" drops keys not in fieldnames; the default raises, which is usually what you want.

Robust import

from dataclasses import dataclass

@dataclass
class ImportResult:
    students: list[Student]
    errors: list[str]


def import_students(path: Path) -> ImportResult:
    students: list[Student] = []
    errors: list[str] = []

    with open(path, newline="", encoding="utf-8-sig") as file:
        reader = csv.DictReader(file)

        required = {"RollNumber", "Name", "ClassName"}
        missing = required - set(reader.fieldnames or [])

        if missing:
            raise ValueError(f"Missing required columns: {', '.join(sorted(missing))}")

        for line_number, row in enumerate(reader, start=2):
            try:
                students.append(Student(
                    roll_number=row["RollNumber"].strip(),
                    name=row["Name"].strip(),
                    class_name=row["ClassName"].strip(),
                    marks=int(row["Marks"]) if row.get("Marks", "").strip() else None
                ))
            except (ValueError, KeyError) as error:
                errors.append(f"Line {line_number}: {error}")

    return ImportResult(students=students, errors=errors)

Four things this gets right:

• Header validation up front, with a message naming every missing column.
• start=2 so reported line numbers match what the user sees in Excel — row 1 is the header.
• One bad row does not stop the import. Errors are collected and reported together.
• A blank numeric field becomes None, not 0. The absent-versus-zero distinction again.

Reporting all errors at once matters: fixing them one run at a time is why bulk imports take an afternoon.

JSON

import json

student = {
    "name": "Ravi Kumar",
    "rollNumber": "NCA-2024-0012",
    "marks": 87,
    "isActive": True,
    "address": None
}

text = json.dumps(student)
text = json.dumps(student, indent=2)
text = json.dumps(student, indent=2, ensure_ascii=False)

parsed = json.loads(text)

ensure_ascii=False keeps non-Latin characters readable. The default escapes them to \uXXXX, so a Telugu name becomes unreadable in the file — valid JSON, and impossible to review by eye.

with open(path, "w", encoding="utf-8") as file:
    json.dump(students, file, indent=2, ensure_ascii=False)

with open(path, encoding="utf-8") as file:
    students = json.load(file)

dump/load work with a file; dumps/loads with a string. The s is for "string".

What JSON cannot hold

json.dumps({"date": datetime.now()})       # TypeError
json.dumps({"amount": Decimal("50.00")})   # TypeError
json.dumps({"ids": {1, 2, 3}})             # TypeError — sets

def json_default(value):
    if isinstance(value, (datetime, date)):
        return value.isoformat()

    if isinstance(value, Decimal):
        return str(value)

    raise TypeError(f"Cannot serialise {type(value).__name__}")

json.dumps(payment, default=json_default)

Serialise Decimal as a string, not a float. float(Decimal("0.1")) reintroduces the binary rounding error the Decimal existed to avoid.

Dates round-trip as strings — json.loads gives you "2024-06-15", not a date. Convert explicitly:

def parse_student(raw: dict) -> Student:
    return Student(
        name=raw["name"],
        roll_number=raw["rollNumber"],
        date_of_birth=date.fromisoformat(raw["dateOfBirth"])
    )

Parsing safely

def load_config(path: Path) -> dict:
    try:
        with open(path, encoding="utf-8") as file:
            return json.load(file)
    except FileNotFoundError:
        return {}
    except json.JSONDecodeError as error:
        raise ValueError(f"{path} is not valid JSON: line {error.lineno}, {error.msg}") from error

JSONDecodeError carries lineno, colno and msg. Reporting the line number turns "invalid JSON" into something actionable.

API responses

import requests

response = requests.get("https://api.nexcoding.in/api/students", timeout=10)
response.raise_for_status()

data = response.json()

raise_for_status() raises on 4xx and 5xx — without it, an error page is parsed as JSON and fails confusingly. timeout is not optional: a request with no timeout can hang indefinitely.

A parsed response is untrusted data. data["items"][0]["name"] raises KeyError or IndexError when the shape differs. Validate what you depend on, or use .get() with defaults.

Writing safely

import os
import tempfile

def write_atomic(path: Path, content: str) -> None:
    """Write to a temporary file, then replace the target atomically."""
    directory = path.parent
    directory.mkdir(parents=True, exist_ok=True)

    handle, temporary = tempfile.mkstemp(dir=directory, suffix=".tmp")

    try:
        with os.fdopen(handle, "w", encoding="utf-8") as file:
            file.write(content)
            file.flush()
            os.fsync(file.fileno())

        os.replace(temporary, path)
    except BaseException:
        Path(temporary).unlink(missing_ok=True)
        raise

A crash halfway through a plain "w" write leaves a truncated file — worse than no file, because the original is gone and the corruption is not obvious.

Writing to a temporary file in the same directory and then calling os.replace is atomic on both POSIX and Windows: the target is either the old content or the new, never a mixture.

Same directory matters — os.replace across filesystems is not atomic.

import shutil

shutil.copy2(path, path.with_suffix(".csv.bak"))

Take a backup before overwriting anything a user provided.

Diagnosing file errors

Error	Cause
FileNotFoundError	Wrong path, or a relative path resolved from an unexpected directory
PermissionError	No access, or the file is open in Excel on Windows
IsADirectoryError	The path is a directory
UnicodeDecodeError	Wrong encoding
First column name is odd	A BOM — use utf-8-sig
Blank rows between records	Missing newline=""
json.JSONDecodeError	Malformed JSON — read lineno
Data silently wrong after import	Split on commas instead of using csv

print(Path.cwd())                      # where relative paths resolve from
print(path.resolve())                  # the absolute path being used
print(path.exists(), path.is_file())

Print Path.cwd() first. Most FileNotFoundError reports are a working-directory problem, not a missing file.

with open(path, "rb") as file:
    print(file.read(200))              # look at the raw bytes

Reading the first bytes shows a BOM (b'\xef\xbb\xbf') or the wrong encoding immediately, where the text-mode error only says the decode failed.

Errors you will hit

Message	Cause	Fix
FileNotFoundError	Wrong path, or wrong working directory	Use an absolute path, or pathlib
UnicodeDecodeError	Wrong encoding	encoding="utf-8"
PermissionError	File open elsewhere, or a directory	Close it; check the path
json.decoder.JSONDecodeError	Not valid JSON — often an HTML error page	Print the first 100 characters
CSV row has the wrong number of fields	Commas inside values	Use the csv module, not split(",")
File is empty after writing	Never closed	Use with open(...)

Always open files with with. It closes the file even when something raises, which a manual close() does not.

Common mistakes

• No encoding= on open
• utf-8 where utf-8-sig is needed, leaving a BOM in the first column name
• Opening in "w" and destroying the file
• No with, leaking the handle
• Splitting CSV lines on commas
• No newline="" with the csv module
• Forgetting next(reader) to skip the header
• Relative paths that break when run from another directory
• String concatenation to build paths
• read() on a large file
• One bad CSV row aborting the whole import
• Decimal serialised as a float
• ensure_ascii=True mangling non-Latin names
• No raise_for_status() or timeout on an HTTP call
• Non-atomic writes to important files

Practice

The course exercise is process a CSV file.

1. Write a CSV with csv.DictWriter, including a name containing a comma. Read it back with DictReader and confirm it round-trips.
2. Read the same file by splitting on commas. Confirm the corruption.
3. Omit newline="" on Windows and confirm the blank rows.
4. Save a CSV from Excel and read it with utf-8, then utf-8-sig. Print the first field name from each with repr.
5. Write a Telugu or Hindi name with no encoding, then with encoding="utf-8". Compare.
6. Build the robust importer with header validation, per-row error collection and start=2. Feed it a file with three bad rows and confirm all three are reported.
7. Leave a numeric field blank and confirm it becomes None, not 0.
8. Open a file in "w" mode with the wrong path and confirm the target is emptied. Then use "x".
9. Run a script from two different directories with a relative path. Confirm one FileNotFoundError, then fix it with Path(__file__).
10. Read a 100 MB file with read() and by iterating. Compare memory usage.
11. Serialise a datetime and a Decimal with json.dumps. Record the TypeError, then add a default handler.
12. Serialise a Decimal as a float and confirm the precision loss.
13. Write JSON with and without ensure_ascii=False for a non-Latin name. Open both in a text editor.
14. Parse malformed JSON and report the line number from JSONDecodeError.
15. Implement write_atomic. Kill the process mid-write with a plain "w" and confirm the truncated file, then repeat with the atomic version.
16. Call an API with requests, omitting raise_for_status(). Point it at a 404 and record what .json() does.

Exercises 2, 4 and 15 correspond to three real data-loss incidents.

You can now

• Read and write text, CSV and JSON safely
• Always use with open(...)
• Specify encoding explicitly
• Use the csv module rather than splitting on commas
• Report a bad row by number instead of failing the import

Review questions

1. Why must encoding always be specified?
2. What does newline="" prevent, and when does it matter?
3. Why is splitting a CSV line on commas wrong?
4. Why write to a temporary file and then replace the target?

Next: Modules and virtual environments