[Bug] 4-bit KV cache results in very low performance while 8-bit KV cache is almost lossless

[fzyzcjy]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

Hi thanks for the library! I am trying to run qwen2.5-math-7B on 4090. It seems the 8-bit kv cache runs quite well without losing performance (on my internal math dataset), while 4bit makes it give 0% accuracy.

Reproduction

(see above)

Environment

docker

Error traceback

No response