Handle Captchas in Scrapy With a Middleware

The clean way to handle captchas in Scrapy is a downloader middleware: it spots a captcha in a response, gets a token from a solving API without blocking the crawl, and returns a new request that submits the form with the token. Your spiders never see the captcha. The two mistakes in most Scrapy captcha tutorials are making a blocking HTTP call inside the crawl, which stalls every concurrent request, and never actually submitting the token back to the site. This guide builds a middleware that avoids both, using only features documented in current Scrapy. The minimal version is on our Scrapy integration page.

Why a middleware, and why it must not block

A downloader middleware sees every response before your spider does, so it is the one place that can handle captchas for all spiders. Scrapy's downloader middleware docs say that when process_response() returns a request, the middleware chain stops and that request is scheduled instead — exactly what a captcha step needs.

Scrapy runs everything on one event loop. A synchronous call such as requests.post() to a solving API that waits 20–60 seconds freezes that loop, and every other request in flight waits with it. Current Scrapy lets any downloader middleware method be a coroutine (async def), and the same docs show a middleware downloading its own request with crawler.engine.download_async(). That lets the solve run as an ordinary non-blocking Scrapy download.

What you need

• A recent Scrapy (the 2.19 docs cover download_async and async middleware methods).
• pip install form2request — Scrapy's own request docs point to this Zyte library for building requests from HTML forms.
• A solving API key, and permission to collect the data.

The middleware

# middlewares.py
import json

import scrapy
from scrapy.exceptions import IgnoreRequest
from form2request import form2request

API = "https://api.captchano.com/v1/solve?wait=60"


class CaptchaNoMiddleware:
    def __init__(self, crawler):
        self.crawler = crawler
        self.api_key = crawler.settings.get("CAPTCHANO_API_KEY")
        self.proxy = crawler.settings.get("CAPTCHANO_PROXY")  # e.g. "http://user:pass@host:port"

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    async def process_response(self, request, response, spider):
        # Never touch our own API calls, or the solve request would loop
        # back through this middleware.
        if request.url.startswith("https://api.captchano.com/"):
            return response

        widget = response.css("[data-sitekey]")
        if not widget:
            return response
        if request.meta.get("captcha_submitted"):
            spider.logger.warning("Captcha still present after submitting a token: %s", response.url)
            raise IgnoreRequest("captcha not cleared")

        is_turnstile = bool(response.css(".cf-turnstile"))
        task = {
            "type": "turnstile" if is_turnstile else "recaptcha_v2",
            "url": response.url,
            "sitekey": widget.attrib["data-sitekey"],
        }
        if self.proxy:
            task["proxy"] = self.proxy

        solved = await self.crawler.engine.download_async(
            scrapy.Request(
                API,
                method="POST",
                body=json.dumps(task),
                headers={
                    "Authorization": f"Bearer {self.api_key}",
                    "Content-Type": "application/json",
                },
                dont_filter=True,
            )
        )
        data = json.loads(solved.text)
        if data.get("status") != "ready":
            spider.logger.error("Solve failed for %s: %s", response.url, data.get("error"))
            raise IgnoreRequest("captcha solve failed")

        field = "cf-turnstile-response" if is_turnstile else "g-recaptcha-response"
        form = widget.xpath("ancestor::form[1]")
        submit = form2request(form, {field: data["solution"]["token"]}).to_scrapy(
            callback=request.callback
        )
        submit = submit.replace(
            dont_filter=True,
            meta={**request.meta, "captcha_submitted": True},
        )
        # Turnstile tokens are tied to the user agent they were produced with.
        user_agent = data["solution"].get("userAgent")
        if user_agent:
            submit.headers["User-Agent"] = user_agent
        return submit

Enable it in settings.py:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.CaptchaNoMiddleware": 950,
}
CAPTCHANO_API_KEY = "cn-YOUR_API_KEY"   # better: read from an environment variable
CAPTCHANO_PROXY = None                  # set if your crawl uses a proxy
DOWNLOAD_TIMEOUT = 90                   # longer than the 60-second solve wait

How each part works

Detection. The middleware looks for any element with a data-sitekey attribute — the standard marker for both reCAPTCHA and Turnstile widgets — and treats a .cf-turnstile element as Turnstile. Pages that render the widget from JavaScript will not match; for those you need a browser (see below).

The solve. The API request is a normal Scrapy request sent through download_async(), so it respects Scrapy's concurrency and never blocks the loop. The ?wait=60 parameter makes the API return the finished token in one response, and DOWNLOAD_TIMEOUT must be longer than that wait. The skip at the top matters: Scrapy's docs warn that requests downloaded this way pass through the middleware chain too.

The submission. form2request reads the form that contains the widget, keeps its hidden fields and adds the token under the field name the site expects: g-recaptcha-response for reCAPTCHA, cf-turnstile-response for Turnstile. The new request goes to the original callback, so your spider parses the page behind the captcha as if nothing happened.

The loop guard. captcha_submitted in meta stops the middleware from solving forever if the site keeps returning a captcha. It also sets dont_filter, because the submission may hit a URL Scrapy has already seen.

Where this approach stops working

A token-submitting middleware works when the site accepts the token as an ordinary form field. It does not work when the page only reacts through JavaScript — a callback that enables the button or sends an AJAX request — because Scrapy does not run the page's scripts. For those pages use scrapy-playwright and the browser approach from our Playwright reCAPTCHA tutorial. It also cannot help with a full-page Cloudflare challenge, which is a session check rather than a form; see our guide to "Verifying you are human".

Tokens are judged against the session that submits them. Keep the crawl's cookies enabled, use the same proxy for the solve and the submission, and keep your user agent stable. The details are in why captcha tokens get rejected.

Keep the crawl polite

A captcha is often the site telling you the crawl is too aggressive. Before solving anything, turn on AutoThrottle, lower CONCURRENT_REQUESTS_PER_DOMAIN, respect robots.txt, and check whether the site offers an API or data feed. Solve only on sites you are allowed to collect from; our acceptable use policy spells out what CaptchaNo may be used for.

Frequently Asked Questions

Can Scrapy solve captchas by itself?

No. Scrapy downloads pages and does not run their JavaScript, so it cannot interact with a captcha widget. It can, however, send the widget's sitekey to a solving API, receive a token, and submit that token with the form, which is what the middleware in this guide does without slowing the rest of the crawl.

Why should the solving call not use requests?

Scrapy handles all requests on a single event loop. A blocking requests call that waits up to a minute for a solve freezes that loop, so every other request in the crawl stalls until it returns. Sending the solve through Scrapy's own downloader as an async request keeps the crawl moving.

Where does the captcha token go in the Scrapy request?

Into the form field the site's widget would have filled: g-recaptcha-response for reCAPTCHA and cf-turnstile-response for Turnstile. Building the request from the page's actual form keeps its other hidden fields, such as CSRF tokens, which the site usually requires alongside the captcha token.

Do I need scrapy-playwright to handle captchas?

Only when the page reacts to the captcha through JavaScript, for example a callback that enables the submit button or sends the form by AJAX. If the site accepts the token as a plain form field, a downloader middleware is enough and much lighter than running a browser for every request.

How do I stop the middleware from looping on the same captcha?

Mark the submission request, for example with a captcha_submitted flag in its meta, and check for that flag before solving again. If the response to a submitted token still contains a captcha, log it and drop the request instead of paying for another solve that will probably fail the same way.

Add captcha handling to your crawl

Start a free CaptchaNo trial, put your key in settings.py and enable the middleware. Failed solves are never billed; per-type rates are on the pricing page.