In partnership with
Understanding how they fit together as a layered architecture, and which tool belongs at which layer for which job, is what separates a functional data pipeline from one that gets blocked, misses dynamic content, or pays per-request for something you can run locally for free.
SnackOnAI Engineering | Senior AI Systems Researcher | Technical Deep Dive | June 26, 2026
The commercial web scraping industry exists because most teams treat each tool as a standalone product: they reach for a managed API when they need clean data, pay per page, and never build the underlying infrastructure. The tools in this issue are what those APIs are built on. Firecrawl runs on Playwright. Zyte's anti-bot evasion is the same TLS fingerprinting curl-impersonate performs for free. The proxy rotation and queue management enterprise teams pay thousands for is exactly what Crawlee ships as open source.
The insight this newsletter delivers: these tools do not compete with each other. They compose. A production AI data pipeline runs four to five of these in layers, each doing the job it was designed for, with the output of each feeding the next. The architectural mistake most teams make is reaching for one tool and trying to make it do everything.
Scope: all ten repos, their specific architectural role in the web data stack, how they compose into a working pipeline, and which tool to reach for in which situation. Not covered: commercial managed services built on top of these, CAPTCHA-solving integrations, or legal analysis of scraping specific targets.
What They Actually Do: The Layered Stack
These ten tools map cleanly onto four distinct layers. Understanding the layers is more important than any individual tool.
Focus on the four-layer separation. Every single tool a commercial scraping vendor charges for operates at one of these layers. The reason enterprise scraping costs what it does is not that these problems are hard. It is that most teams do not know which layer their problem lives at.
The Code, Annotated
Snippet One: The Firecrawl vs Crawl4AI Decision, in Code
# The two most-starred LLM-native crawlers: when to use each
# This is NOT a benchmark. It is an architectural decision guide.
# The choice is infrastructure ownership vs. managed reliability.
# ─── CRAWL4AI: self-hosted, free, full control ────────────────────────────────
# Use when: you need zero per-page cost, data sovereignty, local LLM extraction,
# or want to run at scale without a monthly bill.
# ← Apache 2.0, Playwright under the hood, 68k+ stars
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.extraction_strategy import LLMExtractionStrategy
async def crawl_for_ai_pipeline(url: str) -> str:
browser_config = BrowserConfig(
verbose=False,
headless=True,
)
run_config = CrawlerRunConfig(
# ← BM25 content filter: keeps only sections relevant to your query
# This is what reduces 50,000 tokens of raw HTML to 3,000 relevant tokens
word_count_threshold=10,
excluded_tags=["form", "header", "footer", "nav"],
exclude_external_links=True,
process_iframes=True,
remove_overlay_elements=True,
cache_mode=CacheMode.BYPASS,
)
async with AsyncWebCrawler(config=browser_config) as crawler:
result = await crawler.arun(url=url, config=run_config)
# Returns: result.markdown (clean, LLM-ready), result.media, result.links
return result.markdown
# ─── FIRECRAWL: managed, API-keyed, operationally trivial ───────────────────
# Use when: you want zero DevOps, need geographic proxy coverage, or are
# building across multiple languages (not just Python).
# ← AGPL-3.0, 5B+ requests served, $83/mo for 100k pages
from firecrawl import Firecrawl
def crawl_with_firecrawl(url: str) -> str:
app = Firecrawl(api_key="fc-YOUR_API_KEY")
result = app.scrape(url)
# Returns: result.markdown (same output format as Crawl4AI)
# ← The API abstracts all Playwright/proxy/retry management
# ← You pay $0.00083 per page at the Starter tier
return result.markdown
# ─── THE REAL COST COMPARISON ───────────────────────────────────────────────
# Crawl4AI "free": server ($50-100/mo AWS) + proxy service ($50-200/mo)
# + your engineering time for maintenance
# Firecrawl $83/mo: 100k pages, no infra, no maintenance
# Break-even: ~500k pages/mo, or if you have strict data sovereignty requirements
# ← THIS is the decision: not which tool is "better", but which cost structure fits
The BM25 content filter in Crawl4AI is the feature most teams miss. Feeding 50,000 tokens of raw page HTML into an LLM is expensive and noisy. Crawl4AI's BM25 filter ranks page sections by relevance to your query and keeps only the top-scoring chunks before the LLM ever sees the content. At scale, this is not a quality-of-life feature; it is a cost-reduction mechanism.
Snippet Two: The Transport Layer, Where the Paid APIs Are Secretly Built
# curl-impersonate: the lowest-level tool in the stack
# This is what Firecrawl, Scrapling, and most anti-bot evasion APIs use under the hood
# ← Most developers have never heard of JA3/JA4. This is why that matters.
import subprocess
import json
def fetch_with_browser_fingerprint(url: str, impersonate: str = "chrome124") -> str:
"""
curl-impersonate sends HTTP requests with a TLS fingerprint that exactly
matches a specific browser version. Anti-bot systems check the TLS handshake
(the JA3/JA4 fingerprint) to detect non-browser clients.
A regular requests.get() has a Python/urllib JA3 fingerprint.
A requests.get() with fake headers still has a Python/urllib JA3 fingerprint.
curl-impersonate CHANGES THE FINGERPRINT at the TLS layer.
← THIS is the trick: anti-bot detection is not checking your User-Agent header.
It is checking your TLS fingerprint, HTTP/2 settings, ALPN order, cipher
suite order, and timing patterns. None of those are in the HTTP header.
curl-impersonate replicates the exact binary TLS handshake of Chrome/Firefox.
"""
result = subprocess.run(
[
"curl-chrome124", # ← drop-in curl replacement, Chrome 124 fingerprint
"--silent",
"--location", # follow redirects
"--compressed", # Accept-Encoding: gzip,deflate,br (like a real browser)
url
],
capture_output=True,
text=True,
)
return result.stdout
# ─── WHERE THIS FITS IN THE STACK ────────────────────────────────────────────
# Scrapling uses curl-impersonate or curl_cffi internally for its "stealth" mode
# Crawlee's proxy rotation is meaningless without fingerprint spoofing
# Firecrawl's anti-bot handling uses Playwright (which handles fingerprinting
# via browser rendering), but for non-JS sites, the same TLS layer matters
# ─── WHEN TO USE EACH APPROACH ───────────────────────────────────────────────
# Site blocks Python requests: curl-impersonate → probably unblocks it
# Site blocks curl-impersonate: curl_cffi (same concept, newer JA4 support)
# Site blocks curl_cffi: full browser rendering (Playwright/Crawlee)
# Site blocks automated browsers: browser-use (human-like interaction patterns)
# App has no website at all: scrcpy (Android mirror + automation)
# ← The escalation ladder is: HTTP → fingerprinted HTTP → headless browser →
# human-driven browser → mobile automation. Never skip a layer.
# ─── AUTOSCRAPER: the pattern-learning shortcut ───────────────────────────────
from autoscraper import AutoScraper
scraper = AutoScraper()
# ← THIS is the trick: show one example of what you want
# autoscraper reverse-engineers the DOM pattern that produces that element
# and applies it to the rest of the site automatically
url = "https://news.ycombinator.com"
wanted_items = ["Show HN: I built X in Y"] # ← one real example from the page
results = scraper.build(url, wanted_items)
# Output: all titles matching the same DOM pattern across the page
# No CSS selectors written. No XPath maintained. Updates automatically if structure changes.
# ─── SCRAPY VS CRAWLEE: WHEN SCALE MEANS DIFFERENT THINGS ────────────────────
# Scrapy: best for millions of pages, complex extract logic, downstream ML pipelines
# Crawlee: best for proxy rotation, retries, fingerprint spoofing at crawl-framework level
# ← Scrapy handles what to crawl and what to do with data
# ← Crawlee handles how to stay unblocked while crawling
# These compose: Scrapy for orchestration, Crawlee/curl-impersonate for transport layer
The escalation ladder at the end of the second snippet is the most important decision framework in this issue. The mistake is reaching for a full headless browser (expensive, detectable) when fingerprinted HTTP would have worked. The other mistake is using fingerprinted HTTP when the site renders its content entirely in JavaScript. Know which layer your target lives at before choosing your tool.
It In Action: End-to-End Pipeline for an AI Research Agent
Task: Build a data pipeline that feeds an AI research agent with clean, structured content from competitor websites, financial news sources, and a mobile-only fintech app that has no public website.
Step 1: Transport layer decisions
# Decision tree: what does each source require?
sources = {
"competitor_docs.com": {
"js_rendering": False, # static HTML, Playwright overkill
"anti_bot": True, # Cloudflare
"tool": "curl-impersonate",
"cost": "$0",
},
"financial_news_site.com": {
"js_rendering": True, # React SPA
"anti_bot": False,
"tool": "crawl4ai", # async Python, Playwright, free
"cost": "$0 (self-hosted)",
},
"api_gated_platform.com": {
"js_rendering": True,
"anti_bot": True,
"login_required": True, # ← static crawlers fail entirely here
"tool": "browser-use", # AI agent drives real browser, logs in
"cost": "LLM API calls for agent reasoning",
},
"mobile_only_fintech_app": {
"has_website": False, # ← no web presence at all
"tool": "scrcpy", # Android mirror + automation
"cost": "$0",
},
}
Step 2: Crawl4AI extracts the financial news source
async def extract_financial_news(urls: list[str]) -> list[dict]:
async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
results = []
for url in urls:
result = await crawler.arun(
url=url,
config=CrawlerRunConfig(
word_count_threshold=50,
excluded_tags=["nav", "footer", "aside", "advertisement"],
cache_mode=CacheMode.USE_CACHE,
),
)
results.append({
"url": url,
"content": result.markdown,
# Average output: 800-1500 tokens of clean article text
# vs 8,000-15,000 tokens of raw HTML
# Token reduction: ~90% before LLM ever sees the content
})
return results
# Timing: ~2-4 seconds per page (Playwright overhead)
# Cost: $0 API fees, ~$0.002 compute per page at reasonable server size
# vs Firecrawl equivalent: $0.00083 per page at Starter tier
# Break-even crossover: ~130k pages/month (before accounting for engineering time)
Step 3: browser-use handles the auth-gated platform
from browser_use import Agent
from langchain_openai import ChatOpenAI
agent = Agent(
task="""
Log into platform.com with credentials from environment.
Navigate to the competitor analysis dashboard.
Extract the last 30 days of market share data as a table.
""",
llm=ChatOpenAI(model="gpt-4o"),
# ← browser-use drives a real Chrome instance
# The AI agent sees the page, decides what to click, fills in the login,
# navigates to the right dashboard, and extracts structured data
# No selector maintenance. No site map. No prior knowledge of the DOM.
)
result = await agent.run()
# Output: structured data from a page a Scrapy spider could never reach
# Cost: ~$0.05-0.20 in LLM API calls per navigation sequence
# Time: 30-120 seconds per task (human-pace interaction)
Step 4: MarkItDown converts legacy PDF reports
from markitdown import MarkItDown
md = MarkItDown()
result = md.convert("quarterly_report_2026.pdf")
print(result.text_content)
# Output: clean Markdown, preserving headers, tables, and text structure
# Input: 47-page PDF with mixed text and embedded tables
# Processing: ~3 seconds local, no API key, no upload to external service
# ← The format-conversion step that gates all the document scraping pipelines
# that treat PDFs as opaque blobs
Full pipeline economics (per month, 50,000 pages):
curl-impersonate (15k static pages): $0 (self-hosted)
Crawl4AI (25k dynamic pages): $0 + ~$120/mo server
browser-use (1,000 auth flows): ~$100-200 LLM API costs
scrcpy (mobile data, 10k sessions): $0 (device + developer time)
MarkItDown (5,000 PDFs): $0
Total: ~$220-320/mo vs ~$500-1,500/mo for equivalent managed services
Why This Stack Works, and What It Trades Away
The layered architecture is correct because each tool was built with a specific failure mode in mind. Scrapy was built for millions of static pages. Crawlee was built to stay unblocked at Scrapy scale. curl-impersonate was built to defeat TLS fingerprinting that proxy rotation cannot fix. browser-use was built for the cases where even a full headless browser fails because human-like interaction is required. scrcpy was built for the cases where there is no website at all. None of these tools is trying to do the others' job. They compose because they are not competing.
The autoscraper pattern-learning approach is the correct choice when you have a clearly structured target and no interest in maintaining selectors over time. The tradeoff is robustness under site redesigns: autoscraper relearns patterns, but if the underlying page structure changes fundamentally rather than incrementally, the learned model breaks and needs to be rebuilt with new examples.
Scrapling's adaptive layout detection is the correct approach for long-running monitors that need to survive site redesigns without manual updates. The tradeoff is that stealth focus adds complexity to the pipeline, and its niche (anti-detection) overlaps with what curl-impersonate provides at a lower level.
What this stack trades away:
Every self-hosted tool in this list costs engineering time that a managed API does not. curl-impersonate requires staying current with browser fingerprint updates. Crawl4AI requires managing Playwright infrastructure, including browser process lifecycle management, memory limits, and crash recovery. browser-use costs real LLM API calls for every page navigation, making it expensive at scale. The decision to self-host is the decision to spend engineering time instead of API fees, and that trade is only correct when volume justifies it.
Proxy infrastructure is absent from all ten of these tools by default. Scrapling and Crawlee provide proxy integration hooks but not proxy services. High-volume crawling of protected targets requires a residential proxy provider, which is a separate cost and service regardless of which open-source tool you use.
Technical Moats
The mobile layer (scrcpy) has no open-source competition. For data that exists only in a native mobile app, the choices are: reverse-engineer the app's API (legal and technical risk), automate the Android device via ADB (what scrcpy enables), or pay a managed mobile proxy service. scrcpy's 130k stars represent a community that figured out the ADB path. There is no equivalent browser automation ecosystem for mobile-only data that is both open-source and maintained at this quality level.
curl-impersonate's TLS fingerprinting approach is the unsexy foundation that makes everything else work. Anti-bot vendors (Cloudflare, Akamai, F5 Shape) have moved detection from HTTP header analysis to TLS handshake analysis. Any crawler that does not address the TLS fingerprint layer will fail on these protections regardless of how sophisticated its request queuing or proxy rotation is. curl-impersonate solves the problem at the correct layer. Most developers have never heard of JA3 or JA4. The ones who have are running it in production.
MarkItDown's institutional backing matters. Microsoft published MarkItDown under MIT because they use it internally in Azure AI services for document understanding. It handles Office formats (DOCX, XLSX, PPTX), PDFs, images, and HTML. An alternative maintained by a single developer without Microsoft's investment in Office format parsing would not have this breadth. The format-conversion step is where most open-source pipelines break: either they use a library that only handles one format, or they pay a managed document AI service. MarkItDown does both office and web, maintained by the company that owns the file formats.
Insights
Insight One: The star counts on these repos are a misleading signal for production suitability. browser-use has 95k stars and was built in a year by two researchers. Scrapy has been in production for 18 years and crawls hundreds of millions of pages. These are not comparable. Scrapy's lower relative star count reflects that it predates the GitHub star economy. browser-use's star count reflects that it solved an exciting new problem (AI agent browser control) at the exact moment the AI agent ecosystem needed it. For any data pipeline making >100k requests per month, Scrapy's battle-tested middleware architecture and error handling is more production-relevant than anything under 3 years old, regardless of star counts. Stars measure excitement. Production history measures reliability.
Insight Two: The $16 frustration that spawned Crawl4AI, documented in the founder's own README, is the most accurate description of the scraping market's failure mode: companies were charging per-page for a wrapper around Playwright that anyone can run locally. The same critique applies to every layer of the paid scraping stack. Proxy rotation services charge $200-500/month for infrastructure that Crawlee ships free. TLS fingerprinting APIs charge per-request for what curl-impersonate does locally for free. Document conversion APIs charge per-page for what MarkItDown does offline for free. The commercial scraping industry is a collection of infrastructure-as-a-service margins stacked on top of open-source tooling. These ten repos are that tooling, unpackaged.
Surprising Takeaway
The browser-use repo from two ETH Zurich researchers, MIT license, 95k stars in approximately one year, is the tool that changes the fundamental assumption of web scraping: that you have to understand a site's structure to extract data from it. Every other tool in this list, from Scrapy's XPath selectors to autoscraper's learned patterns to Firecrawl's FIRE-1 agent, requires some prior knowledge of the target site: its URL structure, its HTML patterns, its authentication flow. browser-use's AI-driven approach lets an agent navigate sites it has never seen, log into accounts with credentials, interact with dynamic elements, and extract data without a human ever mapping the site. The research lineage is the same ETH Zurich computational vision and NLP work that has consistently produced tools that make automation accessible without configuration. The implication for AI data pipelines: the maintenance burden of keeping scrapers updated as sites redesign is the cost that browser-use eliminates, at the cost of LLM API fees per session. For high-value, low-frequency extractions from auth-gated sources, that tradeoff is almost always correct.
TL;DR For Engineers
The ten repos map to four layers: HTTP transport (curl-impersonate, MarkItDown), crawl orchestration and scale (Scrapy, Crawlee, autoscraper, Scrapling), browser automation and agents (browser-use, scrcpy), and AI-native crawlers (Firecrawl, Crawl4AI). Compose them; do not substitute one for another.
The escalation ladder: try fingerprinted HTTP first (curl-impersonate or curl_cffi). If the site requires JavaScript rendering, move to Crawl4AI or Crawlee. If auth or human interaction is required, use browser-use. If the target is mobile-only with no website, use scrcpy.
Firecrawl vs Crawl4AI is an infrastructure ownership decision, not a quality decision. Both output clean LLM-ready Markdown. Firecrawl is managed at $83/month for 100k pages. Crawl4AI is free plus server and proxy costs. Break-even is approximately 130k pages per month.
curl-impersonate is the unsexy foundation that makes everything else work. Anti-bot detection has moved from HTTP header analysis to TLS handshake fingerprinting (JA3/JA4). Any crawler that does not address this layer will fail on Cloudflare, Akamai, and F5 Shape regardless of its other features.
browser-use (95k stars, MIT, ETH Zurich) is the one tool that eliminates site-structure knowledge as a prerequisite. The cost is LLM API calls per session ($0.05-0.20 each). For auth-gated, high-value, low-frequency extractions, that tradeoff is almost always worth taking.
The Paid Scraping Industry Is a Margin Stack on Open Source
Every commercial scraping product at any price point is one or more of these ten repos, packaged with infrastructure, a support contract, and SLA guarantees. The SLAs are real value. The support is real value. The proxy network is real value for teams that need geographic diversity without managing it. But the underlying technical capability, the thing that actually touches the web and returns data, is open source and always has been.
The decision for any engineering team is not "open source vs managed." It is "which layers of the stack do I want to own?" Managed services give you back engineering time at the cost of per-request economics. Self-hosted gives you economics at the cost of operational ownership. The ten tools in this issue let you make that decision at each layer independently, which is the correct level of granularity.
References
firecrawl/firecrawl, AGPL-3.0, 130k+ stars
unclecode/crawl4ai, Apache 2.0, 68k+ stars
browser-use/browser-use, MIT, 95k stars
apify/crawlee, Apache 2.0
scrapy/scrapy, BSD, since 2008
microsoft/markitdown, MIT
D4Vinci/Scrapling, MIT
Genymobile/scrcpy, Apache 2.0, 130k+ stars
Ten open-source repos covering the complete four-layer web data stack, from HTTP transport (curl-impersonate for TLS fingerprint impersonation, MarkItDown for document-to-Markdown conversion) through crawl orchestration at scale (Scrapy for industrial-strength spiders, Crawlee for proxy rotation and browser fingerprint spoofing, autoscraper for pattern-learning extraction, Scrapling for stealth and adaptive layout detection) through browser automation and mobile access (browser-use for AI agent-driven real browser interaction at 95k stars MIT, scrcpy for Android device mirroring at 130k+ stars Apache 2.0) to AI-native LLM-ready crawlers (Firecrawl at 130k+ stars AGPL-3.0 with managed API, Crawl4AI at 68k+ stars Apache 2.0 self-hosted with BM25 content filtering). The commercial scraping industry is a managed-infrastructure layer on top of exactly these tools. The architectural decision is not which single tool to use but which layers of this stack to own versus pay for, and the break-even point for self-hosting versus managed services falls at approximately 130k pages per month for the most common use case.
Sponsored Ad
If you enjoy practical AI insights, check out SnackOnAI and support the newsletter by subscribing, sharing, and exploring our sponsored ad — it helps us keep building and delivering value 🚀
The ones showing up in LLMs convert 3× better than Google
They optimized for LLMs, not just Google.
FAQs. Comparison pages. Transparent pricing. LinkedIn presence. These aren't vanity plays. They're what gets you cited in ChatGPT, Gemini, and Claude when your buyers are researching, your investors are looking, and your future hires are deciding where to work.
Download the free AEO Playbook for Startups from HubSpot and get the exact checklist. Five minutes to read.