Skip to content

SHINOBI

Silent Web Scraper — Anti-Blocking Download System

Shinobi is a stealth web scraper with a single web interface. It downloads entire sites — HTML, CSS, JS, images, PDFs — while evading detection through multiple anti-blocking layers. Built with Rust (Axum) + Angular 19, part of the XWA ecosystem.

Rust (Axum)Angular 19WebSocketHeadless ChromiumPython Extractor

Overview

Shinobi is an XWA submodule focused on silent web scraping with anti-blocking. It features recursive crawling, JavaScript rendering via headless Chromium, asset download with file type filtering, same-domain scoping, real-time SSE progress, and a downloaded file browser in the web UI.

Capabilities

Recursive BFS crawling with configurable depth, JavaScript rendering for SPAs, asset download (HTML, CSS, JS, images, PDFs, archives, media, fonts), file type filtering, same-domain scoping, real-time SSE progress streaming, and a downloaded file browser. Two operation modes:Fast Test (pure Rust, zero external deps) and Deep Research(Python sidecar for structured data extraction, NLP analysis, and custom CSS selectors).

Anti-Blocking System

User-agent rotation (15 real browser UAs), header randomization, configurable request delay with random jitter, exponential backoff with retry on failures, HTTP 429/503 detection, and HTTP/HTTPS/SOCKS5 proxy rotation.

Quick Start

$./shinobi.sh --fast
$./shinobi.sh --deep
$./shinobi.sh --docker

Links & Resources

Explore the source code, documentation, and development roadmap.