Skip to content

README.md

Latest commit

 

History

History
166 lines (133 loc) · 6.01 KB

README.md

File metadata and controls

166 lines (133 loc) · 6.01 KB

Evaluations

We evaluate the Gelato-30B-A3B model on the ScreenSpot-Pro and OS-World-G benchmarks. We store our benchmark data at data. To download the corresponding images, you should do so from the official ScreenSpot-Pro and OS-World-G dataset pages.

Note on OS-World-G evaluation: The benchmark includes 54 refusal samples that can be included via --include-refusal. Gelato-30B-A3B supports refusal behavior through the gelato-refusal prompt mode. Official accuracy is calculated over all 564 samples (510 standard + 54 refusal).

Requirements

The following Python packages are required to run the evaluation code:

  • transformers==4.57.0
  • torch==2.8.0
  • pillow==11.3.0
  • qwen-vl-utils==0.0.14

Gelato-30B

Weights: mlfoundations/Gelato-30B-A3B

Eval Logs: Screenspot-Pro, OS-World-G and OS-World-G (Refined)

We run our evals on 4 40GB A100 GPUs with a maximum of 10 megapixel frames. We use huggingface's model parallelism to run the evaluation on 4 GPUs with a batch size of 1.

Run the following command to evaluate the Gelato-30B-A3B model on the ScreenSpot-Pro benchmark.

python evaluation/run_grounding_eval.py \
    --json-file eval-grounding-data/screenspot-pro-eval.json \
    --images-dir path/to/screenspot-pro-images \
    --model path/to/Gelato-30B-A3B \
    --model-type qwen3 \
    --distributed-mode sharded \
    --batch-size 1 \
    --max-pixels 10000000 \
    --prompt-mode gelato \
    --output-dir repro-results

Run the following command to evaluate the Gelato-30B-A3B model on the OS-World-G benchmark.

# Without refusal
python evaluation/run_grounding_eval.py \
    --json-file eval-grounding-data/osworld-g-eval.json \
    --images-dir path/to/osworld-g-images \
    --model path/to/Gelato-30B-A3B \
    --model-type qwen3 \
    --distributed-mode sharded \
    --batch-size 1 \
    --max-pixels 10000000 \
    --prompt-mode gelato \
    --output-dir repro-results

# With refusal
python evaluation/run_grounding_eval.py \
    --json-file eval-grounding-data/osworld-g-eval.json \
    --images-dir path/to/osworld-g-images \
    --model path/to/Gelato-30B-A3B \
    --model-type qwen3 \
    --distributed-mode sharded \
    --batch-size 1 \
    --max-pixels 10000000 \
    --prompt-mode gelato-refusal \
    --include-refusal \
    --output-dir repro-results

UI-TARS-1.5-7B + Gelato Baseline

Weights: mlfoundations/Gelato-UI-TARS-1.5-7B

Eval Logs: Screenspot-Pro, OS-World-G and OS-World-G (Refined)

We run our evals on 4 40GB A100 GPUs with a maximum of 4 megapixel frames. We run the evaluation with a global batch size of 4.

Run the following command to evaluate the Gelato-UI-TARS-1.5-7B model on the ScreenSpot-Pro benchmark.

python evaluation/run_grounding_eval.py \
    --json-file eval-grounding-data/screenspot-pro-eval.json \
    --images-dir path/to/screenspot-pro-images \
    --model path/to/Gelato-UI-TARS-1.5-7B \
    --model-type qwen2.5 \
    --distributed-mode dp \
    --num-gpus 4 \
    --batch-size 1 \
    --max-pixels 4000000 \
    --pixel-space-output \
    --prompt-mode gelato \
    --output-dir repro-results

Run the following command to evaluate the Gelato-UI-TARS-1.5-7B model on the OS-World-G benchmark.

python evaluation/run_grounding_eval.py \
    --json-file eval-grounding-data/osworld-g-eval.json \
    --images-dir path/to/osworld-g-images \
    --model path/to/Gelato-UI-TARS-1.5-7B \
    --model-type qwen2.5 \
    --distributed-mode dp \
    --num-gpus 4 \
    --batch-size 1 \
    --max-pixels 4000000 \
    --pixel-space-output \
    --prompt-mode gelato \
    --output-dir repro-results

Reproduced results for baseline models

We run our evaluation code on GTA1-7B-2507 as a baseline model to verify the correctness of our evaluation pipeline.

Run the following command to evaluate the GTA1-7B-2507 model on the ScreenSpot-Pro benchmark.

python evaluation/run_grounding_eval.py \
    --json-file eval-grounding-data/screenspot-pro-eval.json \
    --images-dir path/to/screenspot-pro-images \
    --model path/to/GTA1-7B-2507 \
    --model-type qwen2.5 \
    --distributed-mode dp \
    --num-gpus 4 \
    --batch-size 1 \
    --max-pixels 4000000 \
    --pixel-space-output \
    --prompt-mode gta1 \
    --output-dir repro-results

Run the following command to evaluate the GTA1-7B-2507 model on the OS-World-G benchmark.

python evaluation/run_grounding_eval.py \
    --json-file eval-grounding-data/osworld-g-eval.json \
    --images-dir path/to/osworld-g-images \
    --model path/to/GTA1-7B-2507 \
    --model-type qwen2.5 \
    --distributed-mode dp \
    --num-gpus 4 \
    --batch-size 1 \
    --max-pixels 4000000 \
    --pixel-space-output \
    --prompt-mode gta1 \
    --output-dir repro-results

ScreenSpot-Pro

Model Accuracy Logs
GTA1-7B-2507 50.1% Official (reported)
GTA1-7B-2507 (reproduced) 49.72% View logs

OS-World-G

Model Accuracy Logs
GTA1-7B-2507 55.1% Official (reported)
GTA1-7B-2507 (reproduced) 56.02% View logs

OS-World-G (Refined)

Model Accuracy Logs
GTA1-7B-2507 67.7% Official (reported)
GTA1-7B-2507 (reproduced) 66.31% View logs