X-LLM

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

🚀 X-LLM aligns multiple frozen single-modal encoders and a frozen large language model (ChatGLM) through X2L interfaces, where "X" denotes the multi-modalities (image, speech, videos) and "L" denotes languages. By treating each modality as a "foreign language", X-LLM builds a Multimodal LLM with impressive multimodal chat abilities — sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images and instructions.

X-LLM is a general multimodal LLM framework that lets you incorporate diverse modalities into LLMs.

The X-LLM framework

X-LLM connects multiple pre-trained, frozen single-modal encoders (e.g., the ViT-g visual encoder, CIF audio encoder) with the frozen large language model ChatGLM through X2L interfaces, following a three-stage training procedure:

• Stage 1 — Converting Multimodal Information. Each X2L interface is trained separately to align its output with the respective single-modal encoder; only the X2L interfaces are updated.
• Stage 2 — Aligning X2L Representations with the LLM. Each single-modal encoder is aligned with the LLM independently through its X2L interface; only the X2L interfaces are updated.
• Stage 3 — Integrating Multiple Modalities. All single-modal encoders are jointly aligned with the LLM through the X2L interfaces; only the adapters in the X2L interfaces are updated.

Beyond multimodal chat, X-LLM also conducts quantitative studies on using the LLM for ASR and multimodal ASR, aiming to promote the era of LLM-based speech recognition.

---

Table of Contents

• Release
• Install
• Dataset
• Training
• Evaluation
• Performance
• Examples
• Acknowledgement
• Citation

---

Release

• [5/6] Code will be released as soon as possible! Stay tuned. ⭐

---

Install

1. Create a conda environment

   conda create -n lavis python=3.8
   conda activate lavis

2. Build from source

   git clone https://github.com/phellonchen/X-LLM.git
   cd X-LLM
   pip install -e .

---

Dataset

Please see README_DATA.md for details.

---

Training

Please see README_TRAIN_EVAL.md for details.

---

Evaluation

Please see README_TRAIN_EVAL.md for details.

---

Performance

We construct an evaluation set of 30 unseen images, each paired with three instruction types — conversation, detailed description, and complex reasoning — yielding 90 language-image instructions. We test X-LLM and GPT-4 on these instructions and use ChatGPT to rate each response from 1 to 10, reporting the summed score and relative score per type.

Overall, X-LLM achieves an 84.5% relative score compared with GPT-4, demonstrating the effectiveness of the proposed method in multimodal settings.

---

Examples

Visual input example — The Forbidden City

Visual input example — Honor of Kings

---

Acknowledgement

• ChatGLM — The codebase we built upon, and our base model ChatGLM-6B, with its amazing Chinese language capabilities!
• BLIP-2 — The architecture of X-LLM follows BLIP-2. Be sure to check out this great open-source work if you haven't already!

---

Citation

If you find X-LLM useful for your research and applications, please cite it using this BibTeX:

@article{chen2023x,
  title   = {X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages},
  author  = {Chen, Feilong and Han, Minglun and Zhao, Haozhi and Zhang, Qingyang and Shi, Jing and Xu, Shuang and Xu, Bo},
  journal = {arXiv preprint arXiv:2305.04160},
  year    = {2023}
}

---

⭐ If you find this repository helpful, please consider giving it a star.