Speed Benchmark¶

We report the speed performance of bfloat16 models and quantized models (including FP8, GPTQ, AWQ) of the Qwen3 series. Specifically, we report the inference speed (tokens/s) as well as memory footprint (GB) under different context lengths.

Environments¶

Hugging Face Transformers¶

Hardware:
- NVIDIA H20 96GB
Software for Non-AutoAWQ:
- PyTorch 2.6.0
- Flash Attention 2.7.4
- Transformers 4.51.3
- GPTQModel 2.2.0+cu128torch2.6
Software for AutoAWQ:
- PyTorch 2.6.0+cu124
- Transformers 4.51.3
- AutoAWQ 0.2.9
- AutoAWQ_kernels 0.0.9

SGLang¶

Hardware:
- NVIDIA H20 96GB
Software:
- PyTorch 2.6.0+cu124
- Transformers 4.51.3
- SGLang 0.4.6.post1
- SGL-kernel 0.1.0
- vLLM 0.7.2 (Required by SGLang for AWQ quantization)

Notes¶

Inference Speed (tokens/s) is calculated as:

\[\text{Speed} = \frac{\text{tokens}_{\text{prompt}} + \text{tokens}_{\text{generation}}}{\text{time}}\]
We use a batch size of 1 and the minimum number of GPUs possible for evaluation.
We test the speed and memory usage when generating 2048 tokens, with input lengths of 1, 6144, 14336, 30720, 63488, and 129024 tokens.
For SGLang:
- Memory usage is not reported because SGLang pre-allocates all GPU memory.
  By default, we set mem_fraction_static=0.85.
- We configure context_length=140000 and enable enable_mixed_chunk=True.
- For AWQ quantization, we use the awq_marlin backend.
- We set skip_tokenizer_init=True and perform generation using input_ids instead of raw text prompts.
FP8 Performance in Transformers: The inference speed of Transformers in FP8 mode is currently not optimal and requires further optimization.
GPTQ-INT4 Performance in SGLang: The performance of GPTQ-INT4 in SGLang also needs improvement, and we are actively working with the team to enhance it.

Results¶

Qwen3-0.6B (SGLang)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)
Qwen3-0.6B	1	BF16	1	414.17
		FP8	1	458.03
		GPTQ-Int8	1	344.92
	6144	BF16	1	1426.46
		FP8	1	1572.95
		GPTQ-Int8	1	1234.29
	14336	BF16	1	2478.02
		FP8	1	2689.08
		GPTQ-Int8	1	2198.82
	30720	BF16	1	3577.42
		FP8	1	3819.86
		GPTQ-Int8	1	3342.06

Qwen3-0.6B (Transformers)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)	GPU Memory(MB)
Qwen3-0.6B	1	BF16	1	58.57	1394
		FP8	1	24.60	1217
		GPTQ-Int8	1	26.56	986
	6144	BF16	1	154.82	2066
		FP8	1	73.96	1943
		GPTQ-Int8	1	93.84	1658
	14336	BF16	1	168.48	2963
		FP8	1	104.99	2839
		GPTQ-Int8	1	219.61	2554
	30720	BF16	1	175.93	4755
		FP8	1	132.78	4632
		GPTQ-Int8	1	345.71	4347

Qwen3-1.7B (SGLang)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)
Qwen3-1.7B	1	BF16	1	227.80
		FP8	1	333.90
		GPTQ-Int8	1	257.40
	6144	BF16	1	838.28
		FP8	1	1198.20
		GPTQ-Int8	1	945.91
	14336	BF16	1	1525.71
		FP8	1	2095.61
		GPTQ-Int8	1	1707.63
	30720	BF16	1	2439.03
		FP8	1	3165.32
		GPTQ-Int8	1	2706.16

Qwen3-1.7B (Transformers)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)	GPU Memory(MB)
Qwen3-1.7B	1	BF16	1	59.83	3412
		FP8	1	23.83	2726
		GPTQ-Int8	1	28.06	2229
	6144	BF16	1	238.53	4213
		FP8	1	90.87	3462
		GPTQ-Int8	1	110.82	2901
	14336	BF16	1	352.59	5109
		FP8	1	153.37	4359
		GPTQ-Int8	1	222.78	3798
	30720	BF16	1	418.13	6902
		FP8	1	235.61	6151
		GPTQ-Int8	1	386.85	5590

Qwen3-4B (SGLang)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)
Qwen3-4B	1	BF16	1	133.13
		FP8	1	200.61
		AWQ-INT4	1	199.71
	6144	BF16	1	466.19
		FP8	1	662.26
		AWQ-INT4	1	640.07
	14336	BF16	1	789.25
		FP8	1	1066.23
		AWQ-INT4	1	1006.23
	30720	BF16	1	1165.75
		FP8	1	1467.71
		AWQ-INT4	1	1358.84
	63488	BF16	1	1423.98
		FP8	1	1660.67
		AWQ-INT4	1	1513.97
	129042	BF16	1	1371.04
		FP8	1	1497.27
		AWQ-INT4	1	1375.71

Qwen3-4B (Transformers)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)	GPU Memory(MB)
Qwen3-4B	1	BF16	1	45.94	7973
		FP8	1	17.33	5281
		AWQ-INT4	1	51.57	2915
	6144	BF16	1	159.95	8860
		FP8	1	60.55	6144
		AWQ-INT4	1	183.04	3881
	14336	BF16	1	195.31	10012
		FP8	1	96.81	7297
		AWQ-INT4	1	265.22	5151
	30720	BF16	1	217.97	12317
		FP8	1	138.84	9611
		AWQ-INT4	1	481.69	7742

Qwen3-8B (SGLang)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)
Qwen3-8B	1	BF16	1	81.73
		FP8	1	150.25
		AWQ-INT4	1	144.11
	6144	BF16	1	296.25
		FP8	1	516.64
		AWQ-INT4	1	477.89
	14336	BF16	1	524.70
		FP8	1	859.92
		AWQ-INT4	1	770.44
	30720	BF16	1	832.67
		FP8	1	1242.24
		AWQ-INT4	1	1075.91
	63488	BF16	1	1112.78
		FP8	1	1476.46
		AWQ-INT4	1	1254.91
	129042	BF16	1	1173.32
		FP8	1	1393.21
		AWQ-INT4	1	1198.06

Qwen3-8B (Transformers)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)	GPU Memory(MB)
Qwen3-8B	1	BF16	1	45.32	15947
		FP8	1	15.46	9323
		AWQ-INT4	1	51.33	6177
	6144	BF16	1	146.12	16811
		FP8	1	55.07	10187
		AWQ-INT4	1	163.23	7113
	14336	BF16	1	183.29	17963
		FP8	1	89.64	11340
		AWQ-INT4	1	242.97	8409
	30720	BF16	1	208.98	20267
		FP8	1	130.93	13644
		AWQ-INT4	1	438.62	11001

Qwen3-14B (SGLang)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)
Qwen3-14B	1	BF16	1	47.10
		FP8	1	97.11
		AWQ-INT4	1	96.49
	6144	BF16	1	174.85
		FP8	1	342.95
		AWQ-INT4	1	321.62
	14336	BF16	1	317.56
		FP8	1	587.33
		AWQ-INT4	1	525.74
	30720	BF16	1	525.80
		FP8	1	880.72
		AWQ-INT4	1	744.74
	63488	BF16	1	742.36
		FP8	1	1089.04
		AWQ-INT4	1	884.06
	129042	BF16	1	826.15
		FP8	1	1049.64
		AWQ-INT4	1	857.56

Qwen3-14B (Transformers)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)	GPU Memory (MB)
Qwen3-14B	1	BF16	1	40.66	28402
		FP8	1	13.02	16012
		AWQ-INT4	1	44.67	9962
	6144	BF16	1	108.52	29495
		FP8	1	44.86	16972
		AWQ-INT4	1	128.08	11020
	14336	BF16	1	136.36	30775
		FP8	1	71.96	18253
		AWQ-INT4	1	220.62	12438
	30720	BF16	1	155.38	33336
		FP8	1	102.63	20813
		AWQ-INT4	1	363.25	15323

Qwen3-32B (SGLang)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)	Note
Qwen3-32B	1	BF16	1	20.72
		FP8	1	46.17
		AWQ-INT4	1	47.67
	6144	BF16	1	77.82
		FP8	1	165.71
		AWQ-INT4	1	159.99
	14336	BF16	1	143.08
		FP8	1	287.60
		AWQ-INT4	1	260.44
	30720	BF16	1	240.75
		FP8	1	436.59
		AWQ-INT4	1	366.84
	63488	BF16	1	342.96
		FP8	1	532.18
		AWQ-INT4	1	425.23
	129042	BF16	2	711.40	TP=2
		FP8	1	491.45
		AWQ-INT4	1	395.96

Qwen3-32B (Transformers)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)	GPU Memory (MB)
Qwen3-32B	1	BF16	1	26.24	62751
		FP8	1	7.37	33379
		AWQ-INT4	1	41.8	19109
	6144	BF16	1	51.41	64583
		FP8	1	23.57	34915
		AWQ-INT4	1	68.71	20795
	14336	BF16	1	62.41	66632
		FP8	1	36.30	36963
		AWQ-INT4	1	107.02	23105
	30720	BF16	1	69.16	70728
		FP8	1	49.44	41060
		AWQ-INT4	1	188.11	27718

Qwen3-30B-A3B (SGLang)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)	Note
Qwen3-30B-A3B	1	BF16	1	137.18
		FP8	1	155.55
		GPTQ-INT4	1	31.29	GPTQ-Marlin
	6144	BF16	1	490.10
		FP8	1	551.34
		GPTQ-INT4	1	120.13	GPTQ-Marlin
	14336	BF16	1	849.62
		FP8	1	945.13
		GPTQ-INT4	1	227.27	GPTQ-Marlin
	30720	BF16	1	1283.94
		FP8	1	1405.91
		GPTQ-INT4	1	404.45	GPTQ-Marlin
	63488	BF16	1	1538.79
		FP8	1	1647.89
		GPTQ-INT4	1	617.09	GPTQ-Marlin
	129042	BF16	1	1385.65
		FP8	1	1442.14
		GPTQ-INT4	1	704.82	GPTQ-Marlin

Qwen3-30B-A3B (Transformers)¶

Model	Input length	Quantization	GPU Num	Speed (tokens/s)	GPU Memory (MB)	Notes
Qwen3-30B-A3B	1	BF16	1	1.89	58462
		FP8	1	0.44	30296
		GPTQ-INT4	-	-	-	MoE Kernel Unsupported
	6144	BF16	1	7.45	59037
		FP8	1	1.77	30872
		GPTQ-INT4	-	-	-	MoE Kernel Unsupported
	14336	BF16	1	14.47	59806
		FP8	1	3.5	31641
		GPTQ-INT4	-	-	-	MoE Kernel Unsupported
	30720	BF16	1	27.03	61342
		FP8	1	6.86	33177
		GPTQ-INT4	-	-	-	MoE Kernel Unsupported

Qwen3-235B-A22B (SGLang)¶

Model	Input Length	Quantization	GPU Num	Speed (tokens/s)	Note
Qwen3-235B-A22B	1	BF16	8	74.50	TP=8
		FP8	4	71.65	TP=4
		GPTQ-INT4	4	14.69	TP=4 GPTQ-Marlin
	6144	BF16	8	289.03	TP=8
		FP8	4	275.16	TP=4
		GPTQ-INT4	4	56.97	TP=4 GPTQ-Marlin
	14336	BF16	8	546.73	TP=8
		FP8	4	514.23	TP=4
		GPTQ-INT4	4	109.13	TP=4 GPTQ-Marlin
	30720	BF16	8	979.41	TP=8
		FP8	4	887.90	TP=4
		GPTQ-INT4	4	198.99	TP=4 GPTQ-Marlin
	63488	BF16	8	1493.91	TP=8
		FP8	4	1269.34	TP=4
		GPTQ-INT4	4	422.77	TP=4 GPTQ-Marlin
	129042	BF16	8	1639.54	TP=8
		FP8	4	1319.66	TP=4
		GPTQ-INT4	4	552.28	TP=4 GPTQ-Marlin