SearXNG + Crawl4AI: Why we fired SerpAPI

The 847 Euro Bill That Forced Us to Rethink

In March 2024, we paid a total of 847 Euros for SerpAPI and Firecrawl. For search queries and structured crawling. Two services we had integrated into four n8n workflows to gather research material for content briefings and market analyses. The bill arrived, we nodded, paid, and then someone asked: "Why aren't we self-hosting this, actually?"

Three weeks later, SearXNG and Crawl4AI were running on our own infrastructure. Both sourcing flows, one for topic research, one for competitor monitoring, now pull data from our own stack. The monthly bill for these two services: zero. The realization: Self-hosting isn't always the answer, but if you make the same API call 40,000 times a month, it quickly becomes economical.

Architecture: Two Tools, One Goal

Our setup consists of two components that work sequentially. SearXNG is a meta-search engine that aggregates results from Google, Bing, DuckDuckGo, and others, without us needing separate API keys for each provider. We run an instance on a dedicated server that is only accessible for internal workflows. The search queries come from n8n workflows that have either manual triggers or scheduled runs.

Crawl4AI handles the second step: structured scraping of the URLs provided by SearXNG. The Library extracts text, metadata, and, if desired, structured data via LLM-powered parsing. We mainly use the Markdown export because it can be directly fed into our briefing templates. Crawl4AI runs in a Python container, which we access via HTTP API from n8n.

The four workflows built on this stack cover different use cases: topic research for editorial plans, competitor analysis (which keywords do competitors rank for?), lead enrichment (public information on target companies), and trend monitoring (what is being discussed in specialist media?). Each workflow first calls SearXNG, filters the results by relevance, and sends the top 10 URLs to Crawl4AI. The extracted content lands in a database from which other systems process it further.

Why not just Google Custom Search?

Google Custom Search API would have been cheaper than SerpAPI, 5 dollars per 1,000 queries instead of 50. But: It only returns 10 results per request, and you can't filter by date or language as granularly as we need. SearXNG aggregates multiple sources, gives us 50+ results per query, and runs entirely on our infrastructure. No rate limit, no vendor lock-in, no sudden API deprecations.

Decision One: SearXNG instead of Vendor-APIs

We didn't drop SerpAPI for ideological reasons. The API was stable, the results good, the documentation solid. But: We were running between 35,000 and 42,000 search requests per month. With SerpAPI, that costs between 400 and 600 Euros, depending on the plan. SearXNG costs us the server rental (80 Euros/month for a dedicated machine that also hosts other internal services) plus a few hours of setup time.

The break-even was reached after six weeks. Since then, we've been saving a three-digit amount every month. But the real gain is control: We can run as many queries as we want, without anyone raising the rate limits or changing prices. And we don't log anything externally, which is a real argument for GDPR-sensitive research (e.g., when we collect market data on behalf of a client).

The Catch: Maintenance and Uptime

SearXNG is not SaaS. If the instance is down, the pipeline stops. In the first two months, we experienced three times that Google or Bing temporarily blocked our server's IP because the request pattern looked suspicious. Solution: We now rotate across multiple outgoing IPs and have built-in rate limiting. Nobody would have taken that burden from us with SerpAPI.

Decision Two: Crawl4AI instead of Firecrawl

Firecrawl was our previous scraper service. Easy to integrate, reliable, but expensive: 0.01 to 0.03 Euros per scraped page, depending on the plan. With 15,000 to 20,000 pages per month, that adds up. Crawl4AI is open source, runs locally, and only costs us computing time. The Library is written in Python, uses Playwright for JavaScript rendering, and offers LLM integration for structured parsing.

We deployed Crawl4AI as a FastAPI service, which our n8n workflows access via HTTP. A request contains the URL and a few parameters (timeout, whether JavaScript should be rendered, what output format). The response is either Markdown or JSON with extracted fields. For most use cases, Markdown export is sufficient; if we need structured data (e.g., prices, contact details), we send the Markdown output to an LLM, which extracts the fields.

What we underestimated: Error Handling

Firecrawl handled errors elegantly. Timeout? You get a partial result. Bot detection? The service automatically rotates through different user agents and proxies. Crawl4AI doesn't do that. If a page doesn't load, you get an empty string or an HTTP 500. We had to build retry logic, timeout handling, and fallback strategies ourselves. That cost two days and was annoying, but now we have it.

What went wrong: The first production run

We ran the system in test mode for three weeks before putting it into production. Everything looked good. Then we started the first production run, 8,000 URLs in four hours, and after two hours, the Crawl4AI container was dead. Out of memory. We had forgotten that Playwright starts a browser process for each URL, and with 50 parallel requests, that quickly eats up 16 GB of RAM.

The solution was simple: Concurrency limit to 10, and we spun up a second container to distribute the load. But it was a classic self-hosting moment: With a SaaS service, that would have been their problem. For us, it was ours. We fixed it, but it cost us two hours of downtime and a bunch of Slack messages.

We don't have numbers yet (and that's okay)

We are not yet tracking systematic metrics for this pipeline. No latency dashboards, no success rates, no cost-per-query analyses. The system has been running for eight weeks, and so far, we are focusing on ensuring it runs stably at all. Quantitative evaluations will come later, probably Q3, when we have enough data to draw meaningful comparisons.

Qualitatively, we can say: The result quality of SearXNG is comparable to SerpAPI, sometimes even better because we aggregate multiple sources. Crawl4AI delivers cleaner Markdown than Firecrawl, but we have to catch more error cases ourselves. Bottom line: It works, it saves money, and we have control. That's enough for us for now.

Where we ended up

Self-hosting is not an end in itself. If you run 500 search queries a month, use SerpAPI. If you run 50,000, build it yourself. At Blck Alpaca, we are in a situation where we use research pipelines intensively, for ourselves and for client projects. In that case, the effort pays off.

The most important insight: You exchange monthly API costs for one-time engineering effort and ongoing maintenance. If you don't have a team that can maintain servers, or if your usage fluctuates, stick with SaaS. If you have consistently high volumes and GDPR compliance is an argument, then switching is worthwhile.

We would do it again. But next time, we would build in concurrency limits and better error handling from the start. And we would start collecting metrics earlier, not because we need them immediately, but because it's tedious to retrofit them later.

Last updated: August 2026

Blck Alpaca is a Vienna-based AI marketing automation agency specializing in data-driven marketing, custom AI agents, and enterprise workflow automation for businesses in the DACH region.