🕸️ Cross Graph - Knowledge Graph Construction & Cross-Document QA Generation

Cross-Graph is an end-to-end system powered by Large Language Models (LLM) designed to construct knowledge graphs from unstructured text (PDF/TXT/MD) and generate high-quality cross-document multi-hop reasoning questions based on graph paths.

🌟 Key Features

1. Bilingual UI Support: Seamlessly switch between Chinese and English interfaces.
2. Automated Schema Design: AI-assisted generation of entity and relation schemas based on domain descriptions.
3. Data Preprocessing: Batch convert PDF, TXT, and MD files into JSONL format ready for extraction.
4. Knowledge Extraction: Parallelized information extraction using LangExtract with OpenAI-compatible LLMs (GPT-4o, DeepSeek, etc.).
5. Graph Storage: Automatic import of extracted knowledge into Neo4j graph database.
6. Rare Node Analysis: Identify long-tail/rare nodes to discover deep, non-obvious knowledge connections.
7. Random Walk Sampling: Generate reasoning paths by performing cross-document random walks starting from rare nodes.
8. Complex QA Generation: Produce high-difficulty multi-hop question-answer pairs based on graph paths.

📂 Project Structure

cross_graph/
├── src/
│   ├── core/                 # Core Logic Modules
│   │   ├── data_processor.py    # Data Preprocessing (Folder -> JSONL)
│   │   ├── extractor.py         # Knowledge Extraction (LLM-based)
│   │   ├── graph_importer.py    # Neo4j Import
│   │   ├── rare_node_analyzer.py # Rare Node Analysis
│   │   ├── random_walker.py     # Random Walk Path Generation
│   │   └── question_generator.py # QA Pair Generation
│   └── ui/                   # User Interface
│       └── app.py               # Gradio Main Application
├── examples/                 # Example Data and Configs
├── docs/                     # Documentation
├── requirements.txt          # Dependencies
├── README.md                 # English Documentation
└── README_ZH.md              # Chinese Documentation

🚀 Quick Start

1. Prerequisites

• Python 3.8+
• Neo4j Database: Ensure Neo4j is installed and running (Community or Enterprise).

2. Installation

Install the required Python packages:

pip install -r requirements.txt

3. Launch the Application

Run the following command in the project root directory:

# Default launch (http://127.0.0.1:7860)
python -m src.ui.app

# Custom host and port
python -m src.ui.app --host 0.0.0.0 --port 8000

# Create a public share link (e.g. *.gradio.live)
python -m src.ui.app --share

Once started, access the web interface at the displayed URL.

📖 User Guide

Tab 1: Schema Design

Define the structure of your Knowledge Graph.

1. Domain Setup: Enter the target domain (e.g., "Finance", "Medicine").
2. AI Auto-Gen: Configure your LLM (API Key, Base URL) and click "🚀 AI Auto-Generate Config" to automatically create entities and relations.
3. Manual Refinement: Use the UI to add, edit, or delete entities and relations.
4. Examples: Provide few-shot examples to improve extraction accuracy.
5. Save Config: Click "💾 Save Config" to save extraction_config.json.

Tab 2: Data Processing & Extraction

Process raw files and extract knowledge.

1. Data Preprocessing:
   • Input: Folder path containing source files (PDF/TXT/MD).
   • Action: Click "🔄 Execute Preprocessing".
   • Output: A .jsonl file.
2. Knowledge Extraction:
   • Input: The generated .jsonl file.
   • Config: Set LLM parameters (API Key, Model, etc.) and Advanced Settings (Concurrency, Batch Size).
   • Action: Click "▶️ Start Extraction".
   • Output: A .jsonl file containing extracted entities and relations.

Tab 3: Import & Analysis

Import data into Neo4j and analyze graph structure.

1. Import to Neo4j:
   • Enter Neo4j URI, User, and Password.
   • Select the extraction output file.
   • Click "⬆️ Execute Import".
2. Rare Node Analysis:
   • Click "🔍 Analyze Rare Nodes" to find nodes with low degree centrality.
   • Adjust "Top Percent" to control the rarity threshold.
3. Random Walk:
   • Generate paths starting from rare nodes to uncover hidden connections.
   • Adjust "Max Path Length" and "Cross-document Weight".

Tab 4: QA Generation

Generate high-quality Q&A pairs for training or evaluation.

1. Configuration:
   • Select the path file generated in Tab 3.
   • Configure LLM settings for question generation.
2. Generation:
   • Click "✨ Start Question Generation".
   • The system will use the paths to create multi-hop reasoning questions.
   • Results are saved to a JSON file.

🛠️ Core Modules

• extractor.py: Handles parallel knowledge extraction using langchain and custom prompts.
• graph_importer.py: Manages Cypher query generation and batch execution for Neo4j.
• random_walker.py: Implements biased random walks that prefer cross-document edges.
• app.py: The Gradio-based frontend orchestration layer.