web-crawling
Use when scraping web pages, automating browser interactions, crawling sites for content, or extracting data from rendered JavaScript pages
Best for Content extraction from JS-…Works with GitHubLow risk
⌘source
- author
@jeremylongshore- language
- Python
★2.1kstars
281forks
May 1, 2026
✦overview.md
Key Features
- ·Multi-tool support (Playwright, Puppeteer, Scrapy, Crawlee, Katana)
- ·Handles JavaScript-rendered pages
- ·Anti-bot protection workarounds with Crawlee
- ·Screenshots and PDF generation
- ·URL discovery and endpoint reconnaissance
- ·Large-scale structured crawling pipelines
Use Cases
- →Scraping product listings from an e-commerce site that renders content with JavaScript
- →Automating login and form submission to extract data behind authentication
- →Crawling a website to discover all endpoints and URLs for security testing
- →Taking screenshots of web pages for visual regression testing
Best for
- ✓Content extraction from JS-heavy sites
- ✓Security reconnaissance
- ✓Large-scale data collection
Not ideal for
- !Simple static page scraping (use wget/curl)
- !High-frequency real-time scraping without rotation setup
FAQs
name
web-crawling
description
Use when scraping web pages, automating browser interactions, crawling sites for content, or extracting data from rendered JavaScript pages
license
MIT
allowed-tools:["Bash(node*)","Bash(python3*)","Bash(scrapy*)","Bash(katana*)","Read","Write"]
version:1.0.0
author:ykotik
Web Crawling
When to Use
- Scraping content from web pages (especially JavaScript-rendered)
- Automating browser interactions (login, form submission, navigation)
- Crawling a site to discover and extract structured data
- Taking screenshots or generating PDFs of web pages
- Discovering URLs and endpoints on a target site
- Extracting data from sites with anti-bot protection
Tools
| Tool | Purpose | When to choose |
|---|---|---|
| Playwright | Cross-browser automation (Chromium, Firefox, WebKit) | JS-rendered pages, screenshots, complex interactions |
| Puppeteer | Chrome/Firefox automation via DevTools Protocol | Chrome-specific automation, familiar API |
| Scrapy | Production-grade Python crawling framework | Large-scale structured crawling with pipelines |
| Crawlee | Node.js crawling with anti-blocking built in | Sites with bot detection, proxy rotation needed |
| Katana | Fast Go-based URL discovery crawler | Endpoint discovery, reconnaissance, speed |
Patterns
Playwright: Scrape a page (inline Node.js script)
node -e "
const { chromium } = require('playwright');
(async () => {
$install
1-click copynpx skills add jeremylongshore/claude-code-plugins-plus-skills --skill web-crawlingSafety assessment
★
Clarity score
How clear and easy to understand the SKILL.md instructions are, rated from 1 to 5.
4/ 5
very goodClear and well structured, with only minor parts that might need a second read.
◎
Actionability score
How directly an agent can act on the SKILL.md instructions, rated from 1 to 5.
4/ 5
highMostly actionable with clear steps; only a few small gaps remain.