WebisMultimodal Data Extraction Framework

An efficient end-to-end pipeline covering web data crawling, multimodal data cleaning, and thematic knowledge base construction

Introduction

Install

Efficient Pipeline Framework

The entire Webis framework integrates functions ranging from web data crawling and multimodal cleaning to thematic knowledge base construction, serving as a complete data processing pipeline

Multimodal Data Extraction

The framework integrates multiple open-source tools including PyPDF to batch process various types of source files, supporting the unified parsing of formats such as PDF, DOCX, and TXT

Self-developed Web Processing Tool

The self-developed web processing tool Webis_HTML adopts an innovative three-tier architecture, enabling lightweight and efficient web page cleaning