ProfounddBot

If you found this page from your server logs, you're in the right place.

Who we are

Profoundd is an independent search engine focused on source transparency. We crawl a curated list of news, medical, legal, and government sites to build our own index, rather than relying on third-party search APIs.

User-Agent

Our crawler identifies itself as:

ProfounddBot/1.0 (+https://profoundd.com/bot)

Rate limits

  • Default: 1 request every 2 seconds per domain
  • Honors Crawl-delay in robots.txt (will use your value if specified)
  • Concurrent workers: We crawl multiple domains in parallel, but only one request at a time per domain
  • Timeout: 15 seconds per request

robots.txt

We respect robots.txt. If you want to block us, add this to your /robots.txt:

User-agent: ProfounddBot
Disallow: /

Or restrict specific paths:

User-agent: ProfounddBot
Disallow: /private/
Crawl-delay: 5

What we crawl

  • Articles, news stories, opinion pieces, and editorial content
  • Public pages only — we never attempt to bypass paywalls or login walls
  • We skip: CSS/JS/images/PDFs, tag pages, search results, login/checkout pages

What we don't do

  • We do not hammer your server — rate limits are strict
  • We do not crawl behind login walls
  • We do not republish full articles — we store content for search indexing, but display results link back to your site
  • We do not sell your content or pass it to third parties

Request removal or contact

If you want to be excluded, prefer a different crawl rate, or have any questions:

Email: [email protected]

We will honor removal requests within 48 hours.

Why indexing matters

Search engines that rely entirely on Google or Bing inherit whatever those indexes decide to surface. By crawling directly, Profoundd can offer readers access to sources the big platforms deprioritize or exclude. Linking to your content helps your audience find you.