ipex.optimize bug with torch.no_grad()

[AdityaKulshrestha]

Describe the bug

Hi team,

I am trying to optimize whisper large-v3 model for inference using IPEX. I am using the standard optimize API. But I faced decent amount of time battling the error:

This is the code:

import time
import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor, pipeline
from datasets import load_dataset
import intel_extension_for_pytorch as ipex


device = "cpu"
torch_dtype =  torch.bfloat16

model_id = "openai/whisper-large-v3"

model = WhisperForConditionalGeneration.from_pretrained(
    model_id,)
model.to(device)

processor = WhisperProcessor.from_pretrained(model_id)

dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
sample = dataset[0]["audio"]

model = ipex.optimize(model, dtype=torch_dtype)
model.eval()
st = time.perf_counter()
inputs_features = processor(sample.get_all_samples().data.numpy()[0], sample_rate=16000, return_tensors="pt").input_features
with torch.cpu.amp.autocast() and torch.no_grad():
    predicted_ids = model.generate(inputs_features)


print("Total Time to generate: ", time.perf_counter() - st)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]  

print(transcription)

Apparently the simple fix to the issue was to swap the order of no_grad and autocast. I am wondering why that is the case, also whether it is a bug or something which is expected.

Fix:

with torch.cpu.amp.autocast() and torch.no_grad(): -> with torch.no_grad() and torch.cpu.amp.autocast()

Alternate, use

with torch.no_grad(), torch.cpu.amp.autocast():

Versions

Facing issues with the script.

Attaching the torch version:

intel-extension-for-pytorch 2.7.0
torch                       2.7.0+cpu
torchcodec                  0.5
torchvision                 0.22.0+cpu