AI-Powered Web Crawler

SmartCrawler uses Claude AI to intelligently select and analyze web pages based on your specific objectives. Automatically discover sitemaps, select relevant URLs, and get detailed content analysis.

Get Started View on GitHub

See SmartCrawler in Action

Watch how SmartCrawler intelligently selects and analyzes web content

Powerful Features

🤖

AI-Powered URL Selection

Uses Claude AI to intelligently select relevant URLs from sitemaps based on your specific objectives.

🗺️

Automatic Sitemap Discovery

Finds and parses XML sitemaps across multiple domains automatically.

📄

Smart Content Analysis

AI-powered analysis of scraped content for objective-specific insights and structured data extraction.

🌐

Multi-Domain Support

Crawl multiple websites in a single session with intelligent domain handling.

⚡

Dynamic Content Loading

Scrolls through pages to capture JavaScript-rendered content that traditional crawlers miss.

📊

Structured Output

Results saved in JSON format with structured entities for further analysis and processing.

Quick Installation

📦 Pre-built Binaries

Download ready-to-use binaries for your platform:

Windows: smart-crawler-windows-x64.zip
macOS: smart-crawler-macos-x64.tar.gz
Linux: smart-crawler-linux-x64.tar.gz

Download from Releases

🔧 Package Installers

Install using your system's package manager:

                
                  # Ubuntu/Debian

                  sudo dpkg -i smart-crawler.deb

                  # RHEL/CentOS/Fedora

                  sudo rpm -i smart-crawler.rpm

                  # Windows MSI installer available

⚙️ Build from Source

For developers who want the latest features:

                
                  git clone https://github.com/brainless/SmartCrawler.git

                  cd SmartCrawler

                  cargo build --release

Getting Started

Set up Claude API Key

export ANTHROPIC_API_KEY="your-api-key-here"

Start WebDriver

geckodriver --port 4444

Run SmartCrawler

                  smart-crawler --objective "Find pricing information" --domains "example.com" --max-urls 5
                

Real-World Examples

E-commerce Price Research

Research competitor pricing across multiple stores

                
                  smart-crawler \

                    --objective "Find product pricing, discounts, and shipping costs" \

                    --domains "shop1.com,shop2.com,competitor.com" \

                    --max-urls 15 \

                    --output pricing-research.json

Company Information Gathering

Extract contact information and team details

                
                  smart-crawler \

                    -o "Find company contact information, team members, and office locations" \

                    -d "company.com" \

                    -m 8 --delay 2000 -v

Technical Documentation Search

Find API docs and developer resources

                
                  smart-crawler \

                    --objective "Find API documentation, integration guides, and developer resources" \

                    --domains "docs.example.com,api.service.com" \

                    --max-urls 20 --output api-docs.json