Why set_lora_device doesn't work

[West2022]

When I load serverl loras with set_lora_device(), the code like this:

for key in lora_list:
    weight_name = key + ".safetensors"
    pipe.load_lora_weights(lora_path, weight_name=weight_name, adapter_name=key, local_files_only=True)
    adapters = pipe.get_list_adapters()
    print(adapters)
    pipe.set_lora_device([key], torch.device('cpu'))

but the GPU memory continues to grow，cames from 20G to 25G, this function doesn't work

env:
V100 32G
diffusers 0.32.0.dev0
torch 2.0.1+cu118
peft 0.12.0