Browser Rendering Launches /crawl Endpoint for Automated Website Crawling
By
jeffpalmer
Properly proved. Has structure, has flavour, has a point.
Summary
Browser Rendering has launched a new /crawl endpoint in open beta that allows developers to crawl entire websites with a single API call. The service automatically discovers pages from a starting URL, renders them in a headless browser, and returns content in multiple formats including HTML, Markdown, and structured JSON. The crawler respects robots.txt and AI Crawl Control by default, making it compliant with website rules and reducing the likelihood of being blocked.
Key quotes
· 3 pulledYou can now crawl an entire website with a single API call using Browser Rendering's new /crawl endpoint, available in open beta.
Submit a starting URL, and pages are automatically discovered, rendered in a headless browser, and returned in multiple formats, including HTML, Markdown, and structured JSON.
The endpoint is a signed-agent that respects robots.txt and AI Crawl Control by default, making it easy for developers to comply with website rules, and making it less likely for crawlers to be blocked.
You might also wanna read
Browser Use: Tool for Making Websites Accessible to AI Agents
Browser Use is a tool that makes websites accessible for AI agents by extracting all interactive elements, allowing agents to focus on their
Product Hunt·3mo ago
Firecrawl: Web Data API for AI Agents and Developers
Firecrawl is a web data API designed for AI agents and developers that converts URLs into LLM-ready markdown or structured data with a singl
Product Hunt·3d ago
Tool to Check Website Crawlability by ChatGPT and Other AI Models
This article describes a tool called "Check if your website can get crawled by ChatGPT" that helps website owners test whether their sites c
Product Hunt·10mo ago
Browserbase Launches Director AI for Web Automation Using Natural Language
Browserbase introduces Director, a new AI-powered tool that allows users to automate web tasks using natural language prompts. The tool enab
Product Hunt·7mo ago
Browser Cash: Browser Automation Platform for AI Agents and Web Scraping
Browser Cash is a browser automation platform designed for AI agents, web scraping, and internet intelligence. The platform allows developer
Product Hunt·6mo ago
Scrapeless: Browser-Tool Infrastructure for AI Agents and Automation
Scrapeless is an infrastructure that powers AI agents, automation workflows, and data extraction with browser-tool technology.
Product Hunt·11mo ago