Advertisement
📡 Real-World

Python REST API Consumption – requests & httpx

Consuming REST APIs is daily Python work. requests makes HTTP simple; httpx adds async. Learn auth, pagination, retries, and error handling.

⏱️ 20 min read🎯 Real-World📅 Updated 2026

Basic HTTP Requests

Python
import requests

resp = requests.get("https://api.github.com/users/python")
resp.raise_for_status()
data = resp.json()
print(data["public_repos"])

new_post = {"title": "Hello", "body": "World", "userId": 1}
resp = requests.post(
    "https://jsonplaceholder.typicode.com/posts",
    json=new_post,
    timeout=10
)
print(resp.status_code, resp.json())

Authentication

Python
# Bearer token
headers = {"Authorization": "Bearer YOUR_TOKEN"}
resp = requests.get("https://api.example.com/data", headers=headers)

# Reuse session (faster, shares cookies)
session = requests.Session()
session.headers.update({"Authorization": f"Bearer {token}"})
resp = session.get("https://api.example.com/profile")

Pagination

Python
def fetch_all(url, headers=None):
    results, page = [], 1
    while True:
        resp = requests.get(url, params={"page": page, "per_page": 100}, headers=headers)
        resp.raise_for_status()
        data = resp.json()
        if not data:
            break
        results.extend(data)
        if "next" not in resp.links:
            break
        page += 1
    return results

Retries & Error Handling

Python
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retry = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503])
session.mount("https://", HTTPAdapter(max_retries=retry))

try:
    resp = session.get("https://api.example.com/data", timeout=30)
    resp.raise_for_status()
    return resp.json()
except requests.exceptions.Timeout:
    print("Timed out")
except requests.exceptions.HTTPError as e:
    print(f"HTTP {e.response.status_code}")
Tip: Use httpx instead of requests for async code — nearly identical API but supports async/await.

Consuming REST APIs with requests

The requests library is the standard way to call other APIs from Python. The two habits that separate toy scripts from robust code: checking the status, and handling failures.

import requests

resp = requests.get("https://api.example.com/users/5",
                    headers={"Authorization": f"Bearer {token}"},
                    timeout=10)          # ALWAYS set a timeout
resp.raise_for_status()                  # raise on 4xx/5xx instead of ignoring
data = resp.json()                       # parse JSON body

# POST JSON
requests.post(url, json={"name": "Ann"})  # json= sets the header + encodes
HabitWhy
timeout=a hung server won't freeze your program forever
raise_for_status()turn HTTP errors into exceptions you handle
resp.json()decode the response body

Two things beginners skip and regret: (1) always pass a timeout — without it, a slow or dead server can hang your program indefinitely. (2) A response with status 404 or 500 is still a "successful" call as far as requests is concerned; call resp.raise_for_status() (or check resp.ok) so errors surface. Efficiency: for many calls to the same host, use a requests.Session() to reuse the connection. Security: put tokens/keys in environment variables, never hard-code them, and wrap calls in try/except requests.RequestException for network failures.

🏋️ Practical Exercise

Consume a REST API with requests:

  1. Make a GET request to a public API and parse the JSON response.
  2. Check response.status_code and call raise_for_status().
  3. Send a POST request with a JSON body.
  4. Add an authorization header with a token.

🔥 Challenge Exercise

Write a client for a paginated REST API: fetch all pages by following the pagination (page number or “next” link), handle authentication via a header, and retry failed requests with exponential backoff for transient errors (timeouts, 5xx). Aggregate the results and save them to a file. Bonus: respect rate limits by reading the relevant response headers and pausing when needed.

📋 Summary

  • REST APIs expose resources over HTTP using standard methods and status codes.
  • The requests library is the de-facto way to call APIs synchronously.
  • Check status codes and use raise_for_status() to catch errors early.
  • Authentication is typically a token or key sent in a header.
  • Pagination requires looping until all pages are fetched.
  • Robust clients add retries with backoff and respect rate limits.

Interview Questions on REST APIs

  • What is a REST API?
  • What are the main HTTP methods and what does each do?
  • What do common HTTP status code ranges (2xx, 4xx, 5xx) mean?
  • How do you handle authentication when calling an API?
  • How do you deal with pagination?
  • How should a client handle transient failures and rate limits?
  • What is the difference between requests and an async HTTP client?
📄 JSON 🌐 API Development 🕸️ Web Scraping ⚡ Async ⚠️ Exception Handling

FAQ

What is REST? +

REST is an architectural style for web APIs where resources are identified by URLs and manipulated with standard HTTP methods (GET, POST, PUT, PATCH, DELETE). Responses are usually JSON, and the server is stateless between requests.

How do I handle a paginated API? +

Request the first page, then keep fetching subsequent pages — using a page/offset parameter or following a “next” URL in the response — until there are no more results. Accumulate the items as you go.

How should I handle failed requests? +

Distinguish client errors (4xx — usually your fault, don’t retry) from transient ones (timeouts, 5xx — safe to retry). For transient failures, retry a few times with exponential backoff, and respect any rate-limit headers the API returns.

When should I use an async HTTP client instead of requests? +

When you need to make many concurrent requests, an async client like httpx or aiohttp overlaps the waiting and is far faster. For simple, sequential calls, requests is simpler and perfectly fine.

Advertisement