GPQA eval score of 7B's official checkpoint is much lower than paper

[luppx]

Hi, thanks for your release, I'm evaluating the official checkpoints of Light-R1-7B-DS. The evaluation scores of AIME24 and AIME25 are similar to your paper, but GPQA is much lower than paper. The scores are shown in table below. Could you tell me where the problem is? Thx.

Model	AIME24	AIME25	GPQA
DeepSeek-R1-Distill-Qwen-7B	55.21	40.94	35.73 (49.1 reported in your paper)
Light-R1-7B-DS	56.98	45.63	25.36 (49.4 reported in your paper)

Here's my evaluation process, following Light-R1 Evaluation Usage :

1. Create environment

# Installing Python 3.10 Environment.
conda create -n deepscaler python=3.10 -y
conda activate deepscaler

cd deepscaler
pip install -e ./verl
pip install -e .

2. Running evaluation script

./scripts/eval/eval_model.sh --model [CHECKPOINT_PATH] --datasets aime aime25 gpqa --output-dir [OUTPUT_DIR]

[ElementQi]

Same GPQA low score here.

Model Path	Dataset	Pass@1	Pass@16	Cons@16	Cutoff Ratio	Mean Response Tokens	Run Hours
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B	gpqa.parquet	0.3573	0.6465	0.5278	0.0316	8375.92	1.637169458
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B	aime25.parquet	0.3958	0.6667	0.5833	0.0813	14264.06	0.487655292
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B	aime.parquet	0.5667	0.8667	0.7667	0.0708	12795.72	0.461128198

There is a part of my running log:

{'actor': {'fsdp_config': {'fsdp_size': -1,
                           'grad_offload': False,
                           'optimizer_offload': False,
                           'param_offload': False,
                           'wrap_policy': {'min_num_params': 0}},
           'optim': {'lr': 1e-06,
                     'lr_warmup_steps_ratio': 0.0,
                     'min_lr_ratio': None,
                     'total_training_steps': -1,
                     'warmup_style': 'constant'},
           'strategy': 'fsdp',
           'ulysses_sequence_parallel_size': 1},
 'data': {'batch_size': 4096,
          'data_source_key': 'data_source',
          'n_samples': 16,
          'output_path': './results/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B/new_n16/aime25.json',
          'path': './processed_data/aime25.parquet',
          'prompt_key': 'prompt',
          'response_key': 'responses',
          'reward_model_key': 'reward_model',
          'skip_format_reward': True},
 'model': {'external_lib': None,
           'path': 'deepseek-ai/DeepSeek-R1-Distill-Qwen-7B'},
 'rollout': {'do_sample': True,
             'dtype': 'bfloat16',
             'enable_chunked_prefill': True,
             'enforce_eager': True,
             'free_cache_engine': True,
             'gpu_memory_utilization': 0.95,
             'ignore_eos': False,
             'load_format': 'dummy_dtensor',
             'log_prob_micro_batch_size': 8,
             'max_num_batched_tokens': 8192,
             'max_num_seqs': 1024,
             'micro_batch_size': 256,
             'n': 1,
             'n_val': 1,
             'name': 'vllm',
             'prompt_length': 1536,
             'response_length': 32768,
             'temperature': 0.6,
             'tensor_model_parallel_size': 2,
             'top_k': -1,
             'top_p': 0.95},
 'trainer': {'n_gpus_per_node': 4, 'nnodes': 1}}

[luppx]

Same GPQA low score here.

Model Path Dataset Pass@1 Pass@16 Cons@16 Cutoff Ratio Mean Response Tokens Run Hours
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B gpqa.parquet 0.3573 0.6465 0.5278 0.0316 8375.92 1.637169458
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B aime25.parquet 0.3958 0.6667 0.5833 0.0813 14264.06 0.487655292
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B aime.parquet 0.5667 0.8667 0.7667 0.0708 12795.72 0.461128198
There is a part of my running log:

{'actor': {'fsdp_config': {'fsdp_size': -1,
                           'grad_offload': False,
                           'optimizer_offload': False,
                           'param_offload': False,
                           'wrap_policy': {'min_num_params': 0}},
           'optim': {'lr': 1e-06,
                     'lr_warmup_steps_ratio': 0.0,
                     'min_lr_ratio': None,
                     'total_training_steps': -1,
                     'warmup_style': 'constant'},
           'strategy': 'fsdp',
           'ulysses_sequence_parallel_size': 1},
 'data': {'batch_size': 4096,
          'data_source_key': 'data_source',
          'n_samples': 16,
          'output_path': './results/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B/new_n16/aime25.json',
          'path': './processed_data/aime25.parquet',
          'prompt_key': 'prompt',
          'response_key': 'responses',
          'reward_model_key': 'reward_model',
          'skip_format_reward': True},
 'model': {'external_lib': None,
           'path': 'deepseek-ai/DeepSeek-R1-Distill-Qwen-7B'},
 'rollout': {'do_sample': True,
             'dtype': 'bfloat16',
             'enable_chunked_prefill': True,
             'enforce_eager': True,
             'free_cache_engine': True,
             'gpu_memory_utilization': 0.95,
             'ignore_eos': False,
             'load_format': 'dummy_dtensor',
             'log_prob_micro_batch_size': 8,
             'max_num_batched_tokens': 8192,
             'max_num_seqs': 1024,
             'micro_batch_size': 256,
             'n': 1,
             'n_val': 1,
             'name': 'vllm',
             'prompt_length': 1536,
             'response_length': 32768,
             'temperature': 0.6,
             'tensor_model_parallel_size': 2,
             'top_k': -1,
             'top_p': 0.95},
 'trainer': {'n_gpus_per_node': 4, 'nnodes': 1}}

@ElementQi Hi, have you evaluated Light-R1-7B-DS in GPQA? Is the score also much lower than paper?

[ElementQi]

Same GPQA low score here.
Model Path Dataset Pass@1 Pass@16 Cons@16 Cutoff Ratio Mean Response Tokens Run Hours
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B gpqa.parquet 0.3573 0.6465 0.5278 0.0316 8375.92 1.637169458
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B aime25.parquet 0.3958 0.6667 0.5833 0.0813 14264.06 0.487655292
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B aime.parquet 0.5667 0.8667 0.7667 0.0708 12795.72 0.461128198
There is a part of my running log:

{'actor': {'fsdp_config': {'fsdp_size': -1,
                           'grad_offload': False,
                           'optimizer_offload': False,
                           'param_offload': False,
                           'wrap_policy': {'min_num_params': 0}},
           'optim': {'lr': 1e-06,
                     'lr_warmup_steps_ratio': 0.0,
                     'min_lr_ratio': None,
                     'total_training_steps': -1,
                     'warmup_style': 'constant'},
           'strategy': 'fsdp',
           'ulysses_sequence_parallel_size': 1},
 'data': {'batch_size': 4096,
          'data_source_key': 'data_source',
          'n_samples': 16,
          'output_path': './results/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B/new_n16/aime25.json',
          'path': './processed_data/aime25.parquet',
          'prompt_key': 'prompt',
          'response_key': 'responses',
          'reward_model_key': 'reward_model',
          'skip_format_reward': True},
 'model': {'external_lib': None,
           'path': 'deepseek-ai/DeepSeek-R1-Distill-Qwen-7B'},
 'rollout': {'do_sample': True,
             'dtype': 'bfloat16',
             'enable_chunked_prefill': True,
             'enforce_eager': True,
             'free_cache_engine': True,
             'gpu_memory_utilization': 0.95,
             'ignore_eos': False,
             'load_format': 'dummy_dtensor',
             'log_prob_micro_batch_size': 8,
             'max_num_batched_tokens': 8192,
             'max_num_seqs': 1024,
             'micro_batch_size': 256,
             'n': 1,
             'n_val': 1,
             'name': 'vllm',
             'prompt_length': 1536,
             'response_length': 32768,
             'temperature': 0.6,
             'tensor_model_parallel_size': 2,
             'top_k': -1,
             'top_p': 0.95},
 'trainer': {'n_gpus_per_node': 4, 'nnodes': 1}}

@ElementQi Hi, have you evaluated Light-R1-7B-DS in GPQA? Is the score also much lower than paper?

Yes. You can check the table below.

Model Path	Dataset	Pass@1	Pass@16	Cons@16	Cutoff Ratio	Mean Response Tokens	Run Hours
qihoo360/Light-R1-7B-DS	gpqa.parquet	0.2629	0.4949	0.4200	0.0161	9022.12	0.871510791
qihoo360/Light-R1-7B-DS	aime.parquet	0.5583	0.8333	0.8000	0.0438	13100.23	0.300890926
qihoo360/Light-R1-7B-DS	aime25.parquet	0.4521	0.7333	0.6000	0.0354	13902.84	0.614596047

[luppx]

Same GPQA low score here.
Model Path Dataset Pass@1 Pass@16 Cons@16 Cutoff Ratio Mean Response Tokens Run Hours
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B gpqa.parquet 0.3573 0.6465 0.5278 0.0316 8375.92 1.637169458
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B aime25.parquet 0.3958 0.6667 0.5833 0.0813 14264.06 0.487655292
deepseek-ai/DeepSeek-R1-Distill-Qwen-7B aime.parquet 0.5667 0.8667 0.7667 0.0708 12795.72 0.461128198
There is a part of my running log:

{'actor': {'fsdp_config': {'fsdp_size': -1,
                           'grad_offload': False,
                           'optimizer_offload': False,
                           'param_offload': False,
                           'wrap_policy': {'min_num_params': 0}},
           'optim': {'lr': 1e-06,
                     'lr_warmup_steps_ratio': 0.0,
                     'min_lr_ratio': None,
                     'total_training_steps': -1,
                     'warmup_style': 'constant'},
           'strategy': 'fsdp',
           'ulysses_sequence_parallel_size': 1},
 'data': {'batch_size': 4096,
          'data_source_key': 'data_source',
          'n_samples': 16,
          'output_path': './results/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B/new_n16/aime25.json',
          'path': './processed_data/aime25.parquet',
          'prompt_key': 'prompt',
          'response_key': 'responses',
          'reward_model_key': 'reward_model',
          'skip_format_reward': True},
 'model': {'external_lib': None,
           'path': 'deepseek-ai/DeepSeek-R1-Distill-Qwen-7B'},
 'rollout': {'do_sample': True,
             'dtype': 'bfloat16',
             'enable_chunked_prefill': True,
             'enforce_eager': True,
             'free_cache_engine': True,
             'gpu_memory_utilization': 0.95,
             'ignore_eos': False,
             'load_format': 'dummy_dtensor',
             'log_prob_micro_batch_size': 8,
             'max_num_batched_tokens': 8192,
             'max_num_seqs': 1024,
             'micro_batch_size': 256,
             'n': 1,
             'n_val': 1,
             'name': 'vllm',
             'prompt_length': 1536,
             'response_length': 32768,
             'temperature': 0.6,
             'tensor_model_parallel_size': 2,
             'top_k': -1,
             'top_p': 0.95},
 'trainer': {'n_gpus_per_node': 4, 'nnodes': 1}}

@ElementQi Hi, have you evaluated Light-R1-7B-DS in GPQA? Is the score also much lower than paper?

Yes. You can check the table below.

Model Path Dataset Pass@1 Pass@16 Cons@16 Cutoff Ratio Mean Response Tokens Run Hours
qihoo360/Light-R1-7B-DS gpqa.parquet 0.2629 0.4949 0.4200 0.0161 9022.12 0.871510791
qihoo360/Light-R1-7B-DS aime.parquet 0.5583 0.8333 0.8000 0.0438 13100.23 0.300890926
qihoo360/Light-R1-7B-DS aime25.parquet 0.4521 0.7333 0.6000 0.0354 13902.84 0.614596047

Well, your eval result is similar to mine, thanks.

[ElementQi]

Hi, @luppx. Note that the test procedure for the above two tables was created a long time ago (About 3 months ago?). Have you been using the latest version of LightR1?

I’m not entirely confident, but I found something relevant (not look into the details, but likely fixed in this PR?: #38). Even with this fix, is the GPQA result still incorrect?

[luppx]

Hi, @luppx. Note that the test procedure for the above two tables was created a long time ago (About 3 months ago?). Have you been using the latest version of LightR1?

I’m not entirely confident, but I found something relevant (not look into the details, but likely fixed in this PR?: #38). Even with this fix, is the GPQA result still incorrect?

Yes, I used the latest release code of LightR1