This is an item in your portfolio. It can be have images or nice text. If you name the file .md, it will be parsed as markdown. If you name the file .html, it will be parsed as HTML.

Full-Stack Web Crawler: A Rust and React Project

Project Overview

I developed this modern web crawler application that combines the power of Rust’s performance with React’s interactive UI capabilities. The project demonstrates my ability to create full-stack applications with robust backends and intuitive frontends while integrating modern web technologies.

Technology Stack

Backend

• Rust: A systems programming language focused on safety, speed, and concurrency
• Axum: A modern, ergonomic web framework built on top of Tokio
• Tokio: An asynchronous runtime for the Rust programming language
• Reqwest: HTTP client for making web requests
• Scraper: HTML parsing library for extracting links and content

Frontend

• React: JavaScript library for building user interfaces
• TypeScript: Strongly typed programming language that builds on JavaScript
• Material UI: React component library implementing Google’s Material Design
• Axios: Promise-based HTTP client for making API requests

Architecture

The project follows a clear separation of concerns with a RESTful architecture:

├── Backend (Rust)
│   ├── Web Server (Axum)
│   ├── Crawler Engine
│   ├── URL Processing
│   └── Data Management
│
└── Frontend (React)
    ├── User Interface (Material UI)
    ├── State Management
    ├── API Integration
    └── Real-time Feedback

The system is designed with these key architectural principles:

• Asynchronous Processing: Using Rust’s async/await for non-blocking operations
• Concurrent Crawling: Processing multiple pages simultaneously
• Cross-Origin Communication: Implementing CORS for secure frontend-backend interaction
• Type Safety: Leveraging Rust and TypeScript’s strong type systems

Key Features & Implementation Details

1. Asynchronous Web Crawling Engine

I implemented a highly efficient web crawler in Rust that can process multiple pages concurrently. The crawling algorithm uses breadth-first search to explore links, with configurable depth limits to control the exploration. The engine is built using Tokio’s asynchronous runtime, allowing it to handle multiple HTTP requests efficiently without blocking.

// Sample code snippet showing asynchronous crawling
pub async fn crawl(config: &Config) -> Result<Vec<Page>, Box<dyn std::error::Error + Send + Sync>> {
    let visited_urls: Arc<Mutex<HashSet<String>>> = Arc::new(Mutex::new(HashSet::new()));
    let url_queue = Arc::new(Mutex::new(VecDeque::from(vec![(config.start_url.clone(), 0)])));
    // ...concurrent processing logic
}

2. RESTful API

The backend exposes a clean RESTful API built with the Axum framework. I designed the API with proper error handling, status codes, and JSON responses. The API includes endpoints for:

• GET /status: Health check and version information
• POST /crawl: Main endpoint for initiating crawl operations with configurable parameters

3. Interactive UI with Real-time Feedback

The frontend provides a modern, responsive interface built with React and Material UI. I implemented real-time feedback mechanisms to show:

• Server Connection Status: Visual indicators of backend connectivity
• Crawl Progress: Loading states during crawl operations
• Error Handling: User-friendly error messages and validation
• Results Display: Clean presentation of crawled pages with color-coded status indicators

4. Intelligent Error Handling

I implemented comprehensive error handling at both frontend and backend levels:

• URL Validation: Client-side and server-side validation of input URLs
• Network Error Recovery: Graceful handling of connection issues
• Structured Error Responses: Consistent error format for API responses
• User-Friendly Messaging: Transforming technical errors into understandable messages

Development Challenges & Solutions

Challenge 1: Async Trait Compatibility in Rust

One of the most challenging aspects was implementing asynchronous handlers in Axum that work with Rust’s trait system. I encountered issues with the Send and Sync bounds required for multithreaded contexts.

Solution: I restructured the code to use concrete response types and moved HTML processing to avoid crossing await points with non-Send types. This required a deep understanding of Rust’s ownership model and trait system.

Challenge 2: Cross-Origin Communication

Setting up secure communication between the React frontend and Rust backend required careful configuration of CORS policies.

Solution: I implemented a CORS layer in the Axum server that allows specific origins, methods, and headers, ensuring secure cross-origin requests while maintaining application security.

Challenge 3: Handling Non-Deterministic Web Content

The unpredictable nature of web content made creating a robust crawler challenging.

Solution: I implemented defensive parsing with fallbacks for missing data, proper error handling for network issues, and graceful degradation when encountering malformed HTML.

Project Outcomes & Skills Demonstrated

This project demonstrates my proficiency in:

1. Full-Stack Development: Creating end-to-end solutions with modern technologies
2. Systems Programming: Using Rust for high-performance concurrent applications
3. Frontend Architecture: Building responsive, user-friendly interfaces with React
4. API Design: Creating clean, RESTful APIs with proper error handling
5. Asynchronous Programming: Managing concurrent operations efficiently
6. Type-Safe Development: Leveraging strong type systems in both Rust and TypeScript
7. Problem Solving: Overcoming complex technical challenges in a multi-language environment

Future Enhancements

I have several planned improvements for this project:

1. Visualization Enhancements: Adding network graphs to visualize crawled page relationships
2. Performance Optimization: Implementing connection pooling and request throttling
3. Content Analysis: Integrating text analysis features to extract keywords and sentiment
4. Persistent Storage: Adding a database for long-term storage of crawl results
5. Authentication: Adding user accounts to save and compare different crawl sessions

Running the Project

If you’d like to run this project locally:

1. Clone the repository:

   git clone https://github.com/zazabap/web_crawler.git
   cd web_crawler
   

2. Start the backend:

   cargo run
   

3. Launch the frontend:

   cd frontend
   npm install
   npm run dev
   

4. Open your browser and navigate to http://localhost:5173

Reflection

Building this web crawler allowed me to combine my interest in web technologies with systems programming. The project pushed me to deepen my understanding of Rust’s asynchronous programming model and React’s component architecture.

I particularly enjoyed the challenge of integrating these two different technology stacks and creating a cohesive application that leverages the strengths of both. The performance capabilities of Rust paired with the interactive experience of React resulted in a powerful yet user-friendly web crawler that can efficiently explore and analyze web content.

This project is open-source and available on GitHub.

[Shiwen An] - [2024-11]