Python Multiprocessing – True Parallelism for CPU-Bound Tasks

The Process Class

multiprocessing.Process spawns a new OS process.

from multiprocessing import Process
import os

def worker(name):
    print(f"Worker {name} | PID: {os.getpid()}")

if __name__ == "__main__":  # Required on Windows!
    processes = []
    for i in range(3):
        p = Process(target=worker, args=(i,))
        processes.append(p)
        p.start()

    for p in processes:
        p.join()

Pool – Parallel Map Operations

Pool distributes work across multiple CPU cores automatically.

from multiprocessing import Pool
import time

def square(n):
    time.sleep(0.1)   # Simulate work
    return n * n

if __name__ == "__main__":
    numbers = list(range(8))

    # Sequential: 0.8s
    # With 4 workers: ~0.2s
    with Pool(processes=4) as pool:
        results = pool.map(square, numbers)

    print(results)

ProcessPoolExecutor – Modern API

concurrent.futures.ProcessPoolExecutor is the high-level interface for multiprocessing.

from concurrent.futures import ProcessPoolExecutor

def heavy_computation(n):
    return sum(i**2 for i in range(n))

if __name__ == "__main__":
    inputs = [1_000_000, 2_000_000, 3_000_000]

    with ProcessPoolExecutor(max_workers=3) as exe:
        results = list(exe.map(heavy_computation, inputs))

    print(results)

Multiprocessing vs Threading

Choose the right tool for the job.

# I/O bound (waiting for network, disk, DB)
# → Use threading or asyncio
# → Threads share memory, low overhead

# CPU bound (calculation, image processing)
# → Use multiprocessing
# → Each process gets full CPU core
# → Bypasses the GIL

# Rule of thumb:
import os
CPU_COUNT = os.cpu_count()
print(f"Available cores: {CPU_COUNT}")

multiprocessing: Real Parallelism by Dodging the GIL

Because the GIL lets only one thread run Python bytecode at a time, threads can't speed up CPU-heavy work. multiprocessing sidesteps it by launching separate processes — each with its own interpreter, its own GIL, and its own memory — so they truly run on multiple cores.

from multiprocessing import Pool

def heavy(n):
    return sum(i*i for i in range(n))

if __name__ == "__main__":          # ← REQUIRED guard on Windows/macOS spawn
    with Pool(4) as pool:           # 4 worker processes
        results = pool.map(heavy, [10_000_000] * 4)   # runs in parallel

The costs: data passed to workers is pickled (serialized) and copied, so sending huge objects is slow — and anything you send must be picklable (no lambdas, no open files). Because memory isn't shared, use Queue, Pipe, or Manager to communicate. And always wrap the launch in if __name__ == "__main__": — without it, spawn-based platforms re-import your module and fork-bomb. Rule: CPU-bound → processes; I/O-bound → threads/async.

Interview Questions on Multiprocessing

• What is multiprocessing and how does it differ from multithreading?
• Why does multiprocessing bypass the GIL?
• When should you use processes instead of threads?
• What is a process pool?
• What is the difference between Pool and ProcessPoolExecutor?
• What are the downsides of multiprocessing (overhead, IPC)?
• How do processes share data?

Related Topics