Expend compatibility check for all quantized MoE models

[JustinTong0323]

Motivation

From the perspective of user friendliness, it is essential to perform the check prior to the occurrence of the actual error. Additionally, we must ensure that the error message is rational.

This update introduces a new method, check_quantized_moe_compatibility, to validate the configuration of quantized Mixture of Experts (MoE) models. It ensures that the tensor parallel size and intermediate size adhere to the required divisibility conditions, improving error handling and model configuration validation.

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Work with maintainers to merge your PR. See the PR Merge Process

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!