How to Train Heavy Multimodal Models Without a Zoo of Scripts

If you've ever tried to fine-tune a fresh multimodal model like Qwen2.5-VL or run a pretrain of a diffusion-based video generator, you probably remember that feeling of despair. In one repository, training crashes due to PyTorch and FlashAttention version conflicts. In another, the author wrote a custom data pipeline that loads 500 GB of images directly into RAM. In a third one, FSDP freezes solid on the second optimization step.

Each new architecture brings its own training script, its own memory-saving hacks, and its own workarounds for parallelism. The team from the EvolvingLMMs-Lab lab decided to gather all this scattered experience in one place. That's how the lmms-engine repository came to be.

It's a modular engine for scalable multimodal model training. It handles the grunt work of distributed training, sequence packing, and low-level kernel optimizations, leaving you with just a config and data.

What's Inside and Who It's For

The framework was developed primarily for researchers and ML engineers working with heavy multimodal networks. While regular text LLMs can be comfortably fine-tuned with tools like Axolotl or LLaMA-Factory, images, audio, and video are a different story.

Multimodal inputs create massive context lengths. A single high-resolution frame or a couple seconds of video easily expands into tens of thousands of visual tokens. At that point, standard Data Parallel quickly hits GPU memory limits.

The engine solves this with a combination of modern parallelism techniques and GPU-level optimizations. The list of supported models is impressive:

• Vision-Language models: Qwen2.5-VL, Qwen3-VL, Qwen3-VL MoE, LLaVA-OneVision
• Multimodal MoE with audio and video support: Qwen2.5-Omni, Qwen3-Omni MoE, Aero
• Generative and diffusion architectures: WanVideo (1.3B and 14B parameters), SiT (Scalable Interpolant Transformers), dLLM
• Universal image understanding and generation systems: BAGEL

Four Engineering Solutions Under the Hood

The project creators were clearly inspired by the idea of minimalism: the code is written compactly, without unnecessary abstract magic, in pure PyTorch with integrations for cutting-edge libraries.

1. FSDP2 and Ulysses Sequence Parallel

For distributing weights, the engine relies on the fresh implementation of Fully Sharded Data Parallel v2 (FSDP2) based on DTensor. Unlike the old FSDP, the second version composes much more cleanly with other types of parallelism.

When 10,000+ visual tokens fly into the context, GPU memory runs out instantly. That's where Ulysses Sequence Parallel (USP) kicks in. It splits the token sequence across several GPUs within a single node. In the config, this is set with a single line:

trainer_args:
  sp_ulysses_degree: 2

2. Sequence Packing Without Unnecessary Padding

A classic headache in multimodal training is the varying sizes of images and texts in a single batch. If you pad short samples with padding tokens, the GPU spends up to half the time on useless zero computations.

The authors implemented first-fit bin packing in combination with FlashAttention use_rmpad. Data is packed into dense long sequences with no idle time. According to the authors' benchmarks, on Qwen2.5-VL fine-tuning, the compute efficiency metric (MFU) jumps from 20-25% to an impressive 35-40%.

dataset_config:
  packing: true
  packing_strategy: first_fit
  packing_length: 32000

trainer_args:
  use_rmpad: true
  use_liger_kernel: true

3. Muon Optimizer and Triton Kernels from Liger

Instead of the familiar AdamW, the project offers Muon. This optimizer applies Newton-Schulz orthogonalization via Triton kernels to 2D weight matrices. It converges faster than AdamW and requires less memory.

At the same time, the engine can swap standard model layers on the fly with fused kernels from LinkedIn's Liger Kernel library. CrossEntropy, RMSNorm, RoPE, and SwiGLU are fused into single operations, cutting around 30% of peak VRAM consumption without accuracy loss.

4. Streaming Load of Terabyte Datasets

Loading massive arrays of videos and images into memory before an epoch starts is impossible. The data pipeline in lmms-engine is built on top of IterableDataset. Data is read in a stream from Arrow, JSONL, or Parquet formats, so training starts immediately without waiting for terabyte-sized files to be indexed.

How Launching and Extension Work

The project installation is done properly through the uv package manager, although the authors also provide a ready-made Docker image with pre-installed CUDA, FlashAttention, and dependencies.

git clone https://github.com/EvolvingLMMs-Lab/lmms-engine.git
cd lmms-engine
uv pip install -e ".[all]"
uv pip install flash-attn --no-build-isolation
uv pip install liger-kernel

Training starts via standard torchrun:

torchrun --nproc_per_node=8 --nnodes=1 --node_rank=0 \
  --master_addr=127.0.0.1 --master_port=12355 \
  -m lmms_engine.launch.cli config_yaml=examples/qwen3_vl/example_config.yaml

If you need to add your own specific data format or custom feature processor, you won't have to rewrite the trainer internals. The code uses a factory pattern with registration via decorators:

from lmms_engine.datasets import register_dataset, BaseDataset

@register_dataset("my_custom_dataset")
class MyCustomDataset(BaseDataset):
    def __init__(self, config):
        super().__init__(config)

    def __getitem__(self, idx):
        # Ваша логика чтения картинки или видео
        return item

Where the Project Shines Best

The engine was developed for specific heavy scenarios:

1. Fine-tuning vision-language models (Qwen2.5-VL, Qwen3-VL) on long documents, book scans, and interface screenshots.
2. Training sparse Mixture-of-Experts architectures with expert distribution across GPUs (Expert Parallelism).
3. Experiments with diffusion for video generation based on WanVideo or SiT models.
4. Pretraining lightweight recurrent and linear attention models (FLA / DGN).

Is It Worth Trying

If your task is to quickly tweak LoRA on a text Llama, lmms-engine might seem overkill. But if you've hit a memory ceiling when training multimodal networks, are tired of manually tying FSDP2 with long video sequences, or want to squeeze maximum FLOPS out of your existing clusters, this engine will save you weeks of coding.

The easiest way to start is with the ready-made scripts in the examples/ folder: they contain battle-tested configurations for most modern architectures.