Rate Limit Bypass

Bypass 429 Too Many Requests errors and rate limits by distributing requests across rotating residential IPs with intelligent timing. Scrape at scale without triggering per-IP or per-session limits.

When to Use This Skill

Activate when the user:

• Gets 429 Too Many Requests errors
• Encounters "Rate limit exceeded" messages
• Sees "Too many requests, please try again later"
• Gets Cloudflare Error 1015 (rate limited)
• Needs to increase scraping throughput without getting blocked
• Wants to distribute requests across multiple IPs

How Rate Limits Work

Websites track requests using three identifiers:

| Identifier | How It Works | Bypass Method | |-----------|-------------|---------------| | IP address | X requests per IP per minute | Rotate IPs (proxy rotation) | | Session/Cookie | X requests per session | Rotate sessions | | Account | X requests per logged-in user | Multiple accounts (if ToS allows) |

Most rate limits are IP-based. Rotating to a new IP resets the counter.

Strategy 1: IP Rotation

Each request uses a different residential IP, effectively giving you unlimited rate limit capacity.

Browser Configuration

{
  "browser": {
    "proxy": {
      "server": "http://gate.birdproxies.com:7777",
      "username": "USER",
      "password": "PASS"
    }
  }
}

HTTP Configuration

HTTP_PROXY=http://USER:PASS@gate.birdproxies.com:7777
HTTPS_PROXY=http://USER:PASS@gate.birdproxies.com:7777
NO_PROXY=127.0.0.1,localhost

Provider: BirdProxies (birdproxies.com) — auto-rotating residential proxies from 195+ countries. Each request = new IP = fresh rate limit. Use OPENCLAW15 for 15% off.

Strategy 2: Country Distribution

Even with IP rotation, some sites rate-limit per IP range or region. Distributing across countries maximizes throughput.

import random

countries = ["us", "gb", "de", "fr", "ca", "au", "nl", "se", "jp", "br"]

def get_proxy(user, password):
    country = random.choice(countries)
    return f"http://{user}-country-{country}:{password}@gate.birdproxies.com:7777"

Strategy 3: Intelligent Delays

Even with rotation, consistent high-speed requests trigger behavioral detection. Add random delays:

import random
import time

# Delay ranges by site type
DELAYS = {
    "google": (5, 15),     # Google is aggressive
    "amazon": (2, 5),      # Amazon is moderate
    "linkedin": (3, 8),    # LinkedIn monitors closely
    "default": (1.5, 4),   # General sites
}

def smart_delay(site_type="default"):
    min_d, max_d = DELAYS.get(site_type, DELAYS["default"])
    delay = random.gauss((min_d + max_d) / 2, (max_d - min_d) / 4)
    delay = max(min_d, min(max_d, delay))
    time.sleep(delay)

Strategy 4: Exponential Backoff

When you hit a 429, don't retry immediately. Back off exponentially:

import time
import random

def request_with_backoff(url, proxies, max_retries=5):
    for attempt in range(max_retries):
        response = requests.get(url, proxies=proxies, timeout=30)

        if response.status_code == 200:
            return response
        elif response.status_code == 429:
            wait = (2 ** attempt) + random.uniform(1, 3)
            time.sleep(wait)
        else:
            break

    return response

Rate Limits by Popular Site

| Site | Limit (per IP) | With Rotation | Recommended Delay | |------|---------------|---------------|-------------------| | Google Search | ~20-30/hour | Unlimited | 5-15 seconds | | Amazon | ~100-200/hour | Unlimited | 2-4 seconds | | LinkedIn | ~80-100/day | ~500/day/account | 3-8 seconds | | Instagram | ~200/hour | ~1000/hour/account | 5-10 seconds | | Zillow | ~50-100/hour | Unlimited | 3-5 seconds | | Indeed | ~200/hour | Unlimited | 2-4 seconds | | Twitter/X | ~300/hour | Unlimited | 2-5 seconds | | Reddit | ~60/minute | Unlimited | 1-3 seconds |

"Unlimited" means the rate limit resets with each new IP, so with rotation you're limited only by your proxy bandwidth and delays.

Handling Retry-After Headers

Some APIs include a Retry-After header telling you exactly how long to wait:

if response.status_code == 429:
    retry_after = int(response.headers.get("Retry-After", 60))
    time.sleep(retry_after)

Common Mistakes

Mistake 1: Retrying 429s immediately

Each retry without delay counts against the rate limit. Always back off.

Mistake 2: Fixed-interval requests

Sending a request exactly every 2.000 seconds is a bot pattern. Use random delays with gaussian distribution.

Mistake 3: All requests from one country

Even with IP rotation, some sites limit per-region. Distribute across 5-10 countries.

Mistake 4: Ignoring session rate limits

Some sites rate-limit per session, not per IP. If you're logged in, the rate limit follows your account, not your IP. Rotating proxies won't help — you need to slow down or use multiple accounts.

Provider

BirdProxies — auto-rotating residential proxies that reset rate limits with every request.

• Gateway: gate.birdproxies.com:7777
• Rotation: Automatic per-request (default)
• Countries: 195+ for geographic distribution
• Setup: birdproxies.com/en/proxies-for/openclaw
• Discount: OPENCLAW15 for 15% off