How do I make HTTP requests concurrently in Python?

Question

I need to fetch data for 500 product IDs from an API. Doing it one by one with `requests` takes ~4 minutes. How do I speed it up without getting rate-limited?

Accepted Answer

Use async I/O with `httpx` and cap concurrency with a semaphore:

```python
import asyncio, httpx

async def fetch_all(ids, max_concurrency=10):
    sem = asyncio.Semaphore(max_concurrency)
    async with httpx.AsyncClient(timeout=10) as client:
        async def one(pid):
            async with sem:
                r = await client.get(f"https://api.example.com/products/{pid}")
                r.raise_for_status()
                return r.json()
        return await asyncio.gather(*(one(i) for i in ids), return_exceptions=True)

results = asyncio.run(fetch_all(range(500)))
```

- `max_concurrency` keeps you under the API's rate limit. Check its docs and add retry with backoff on `429`s (e.g. `tenacity`).
- `return_exceptions=True` means one failure doesn't cancel everything; filter the errors out afterwards.

If you'd rather stay synchronous, `concurrent.futures.ThreadPoolExecutor(max_workers=10)` with `requests` works well too for I/O-bound work.

1 Answer