I am trying to train YOLOX algorithm on gaudi2. I am getting above error at this operation. grid = torch.stack((xv, yv), 2).view(1, 1, hsize, wsize, 2).type(dtype) How can I solve it? When posting a technical issue, please describe the issue; be as descriptive as possible, you can include things …

We support DDP for scaling in pytorch not horovod. Horovod is supported only on tensorflow. See here and here .

Thanks for posting. Can you try: grid = torch.stack((xv, yv), 2).view(1, 1, hsize, wsize, 2).type(torch.FloatTensor)

[image] Sayantan_S: torch.FloatTensor Thanks. That helped to get pass previous error. now getting following error. [1,0]: File “/workspace/yolox/utils/boxes.py”, line 102, in bboxes_iou [1,0]: en = (tl < br).type(torch.FloatTensor).prod(dim=2) [1,0]: │ │ │ └ <cla…

Can you point me to which code you are using? We have a yolox enabled here . Also, what is the SW version you have? Thanks

@Sayantan_S I am trying to integrate horovod into Yolox code and analyzed that when I change optimizer code from self.optimizer = self.exp.get_optimizer(self.args.batch_size, self.args.hpu) to self.optimizer = self.exp.get_optimizer(self.args.batch_size, self.args.hpu) …

Also I am using 8 gaudi2 and vault.habana.ai/gaudi-docker/1.10.0/ubuntu20.04/habanalabs/pytorch-installer-2.0.1:latest docker image

@Sayantan_S Below I have referenced line where error is happening. https://github.com/HabanaAI/Model-References/blob/3e4f2339d5240d2a83412a8dfb5898307049f08d/PyTorch/computer_vision/detection/yolox/yolox/utils/boxes.py#LL105C1-L105C46

No. This error comes when I try to train using horovod instead of pytorch distributed module. Integrating horovod in original_yolox and running it on A100 working fine, but getting mentioned error when I run with gaudi2. post mentioned by you is using code yolox_gaudi2 .

ValueError: invalid type: 'torch.hpu.FloatTensor'

Training

Sayantan_S June 1, 2023, 5:31pm 8

From this post, it seems you have been able to run the yolox model?

Topic		Replies	Views
Gaudi1 HPU doesn't support long? PyTorch pytorch	11	384	April 4, 2024
Gaudi2 PyTorch Container - Device acquire failed System Setup pytorch	1	1381	February 22, 2023
Model.to device faile: "RuntimeError: synStatus=8 [Device not found] Device acquire failed." Training models , pytorch	3	702	March 13, 2024
Gaudi2 slower compared to A100 Training	10	709	June 7, 2023
Trainer killed/Segfault PyTorch	6	700	September 1, 2023

ValueError: invalid type: 'torch.hpu.FloatTensor'

Related topics