Video-LLaVA: Harden training pipeline, fix model embedding, and overhaul docs

[rodosingh]

Summary

• Harden Video-LLaVA training and inference pipeline — validate file existence before loading image/video data, handle corrupted video frames with fallback, fix text-only sample initialization, and tune DeepSpeed config (ZeRO stage 1, lower loss scale power) to mitigate NaN during fine-tuning
• Refactor model embedding logic — rewrite image token position tracking in llava_arch.py to correctly handle multi-image inputs and fix prompt_len computation; add longest common contiguous subsequence utility for PDrop salient token matching
• Overhaul README — add prerequisites (transformers patch, model consolidation), expand evaluation and training docs for LLaVA-1.5, 1.6, Video-LLaVA, and Qwen2.5-VL with full parametrized examples and token budget presets
• Cleanup — add missing dependencies (nltk, mpi4py, openai, en-core-web-sm, huggingface_hub[hf_xet]), remove dead visionzip setup block, and delete 28 deprecated pdrop scripts superseded by scripts/llava/

Test plan

• Video-LLaVA fine-tuning completes without NaN loss on a representative dataset
• Inference handles videos with corrupted/missing frames without crashing
• Text-only samples in mixed datasets are processed correctly
• All eval scripts under scripts/llava/ and scripts/videollava/ still work after deprecated script removal
• pip install -e . succeeds with the updated setup.py dependencies

Made with Cursor