scrapling
Adaptive web scraping framework with anti-bot bypass and spider crawling.
Best for data extractionWorks with GitHub
⌘source
- author
@LeoYeAI- language
- Python
★2kstars
306forks
May 6, 2026
✦overview.md
Key Features
- ·Adaptive web scraping framework
- ·Anti-bot bypass support
- ·Spider crawling multi-page
- ·HTTP and browser automation
- ·Stealth/dynamic mode via Playwright
Use Cases
- →Extract data from blogs, news sites, or documentation
- →Research topics from multiple websites
- →Crawl site structures with Spider
- →Reverse engineer APIs from web pages
Best for
- ✓data extraction
- ✓research projects
FAQs
name
scrapling
description
Adaptive web scraping framework with anti-bot bypass and spider crawling.
version:1.0.8
metadata:{"openclaw":{"emoji":"🕷️","requires":{"bins":["python3"]},"tags":["web-scraping","crawling","research","automation"]}}
Scrapling - Adaptive Web Scraping
"Effortless web scraping for the modern web."
Credits
Core Library
- Repository: https://github.com/D4Vinci/Scrapling
- Author: D4Vinci (Karim Shoair)
- License: BSD-3-Clause
- Documentation: https://scrapling.readthedocs.io
API Reverse Engineering Methodology
- GitHub: https://github.com/paoloanzn/free-solscan-api
- X Post: https://x.com/paoloanzn/status/2026361234032046319
- Author: @paoloanzn
- Insight: "Web scraping is 80% reverse engineering"
Installation
# Core library (parser only)
pip install scrapling
# With fetchers (HTTP + browser automation) - RECOMMENDED
pip install "scrapling[fetchers]"
scrapling install
# With shell (CLI tools) - RECOMMENDED
pip install "scrapling[shell]"
# With AI (MCP server) - OPTIONAL
pip install "scrapling[ai]"
# Everything
pip install "scrapling[all]"
# Browser for stealth/dynamic mode
playwright install chromium
# For Cloudflare bypass (advanced)
pip install cloudscraper
Agent Instructions
When to Use Scrapling
Use Scrapling when:
- Research topics from websites
- Extract data from blogs, news sites, docs
$install
1-click copynpx skills add LeoYeAI/openclaw-master-skills --skill scraplingSafety assessment
★
Clarity score
How clear and easy to understand the SKILL.md instructions are, rated from 1 to 5.
3/ 5
goodMostly clear, but there are still a few confusing or poorly structured parts.
◎
Actionability score
How directly an agent can act on the SKILL.md instructions, rated from 1 to 5.
3/ 5
mediumPartially actionable with several concrete steps, but still missing important details.