# hl\_qual resnet50 training failed when load resnet50DecoderConfig.ini running on Gaudi2D

**URL:** https://forum.habana.ai/t/hl-qual-resnet50-training-failed-when-load-resnet50decoderconfig-ini-running-on-gaudi2d/1359
**Category:** Training
**Created:** [November 12, 2024, 6:14pm UTC](https://forum.habana.ai/t/hl-qual-resnet50-training-failed-when-load-resnet50decoderconfig-ini-running-on-gaudi2d/1359 "2024-11-12T18:14:11Z")
**Posts on this page:** 1
**Page:** 1

<div class="post-metadata">

### Author: ![RongLei-intel](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.habana.ai/ronglei-intel/32/260_2.png) [@RongLei-intel](https://forum.habana.ai/u/RongLei-intel)
#### Post date: [November 12, 2024, 6:14pm UTC](https://forum.habana.ai/t/hl-qual-resnet50-training-failed-when-load-resnet50decoderconfig-ini-running-on-gaudi2d/1359/1 "2024-11-12T18:14:11Z")

</div>

I tried to run hl\_qual resnet50 training. All the preparation works are done, and I run the following cmd:

./hl\_qual -gaudi2 -c all -rmod parallel -trainingApp -epoch 3 -train\_list train\_list.txt

After 20s, it stopped and throw this error. Anyone can help with that?  
HL-SMI Version: hl-1.18.0-fw-53.1.1.1  
Driver Version: 1.18.0-ee698fb  
################################## hl\_qual #########################################  
HL-QUAL VERSION: 524-b50bd67  
Failed (WIFSIGNALED): Process 1912190 (index 5, bus ID: 0000:5d:00.0) was terminated by signal 11.  
device 5 core dump produced  
Failed (WIFSIGNALED): Process 1912189 (index 4, bus ID: 0000:4b:00.0) was terminated by signal 11.  
device 4 core dump produced  
Failed (WIFSIGNALED): Process 1912188 (index 3, bus ID: 0000:bb:00.0) was terminated by signal 11.  
device 3 core dump produced  
Failed (WIFSIGNALED): Process 1912192 (index 7, bus ID: 0000:3c:00.0) was terminated by signal 11.  
device 7 core dump produced  
Failed (WIFSIGNALED): Process 1912185 (index 0, bus ID: 0000:ca:00.0) was terminated by signal 11.  
device 0 core dump produced  
Failed (WIFSIGNALED): Process 1912186 (index 1, bus ID: 0000:db:00.0) was terminated by signal 11.  
device 1 core dump produced  
Failed (WIFSIGNALED): Process 1912191 (index 6, bus ID: 0000:18:00.0) was terminated by signal 11.  
device 6 core dump produced  
Failed (WIFSIGNALED): Process 1912187 (index 2, bus ID: 0000:9a:00.0) was terminated by signal 11.  
device 2 core dump produced  
All processes finished successfully.  
################################## hl\_qual command line #########################################  
./hl\_qual -gaudi2 -c all -rmod parallel -trainingApp -epoch 3 -train\_list train\_list.txt  
################################## Test run report #########################################  
hl\_qual - Plugin report START, bus ID: 0000:ca:00.0  
hl\_qual - dev number bus ID: 0000:ca:00.0  
hl\_qual - board type: HL-225D  
hl\_qual - board serial: AN40008959  
hl\_qual - PCI link speed (gen): 16GT/s  
hl\_qual - PCI link width: x16  
hl\_qual - HBM size: 96 GB  
[19:13:34.156] Start running plugin  
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini  
hl\_qual - Plugin report END:

hl\_qual - Plugin report START, bus ID: 0000:db:00.0  
hl\_qual - dev number bus ID: 0000:db:00.0  
hl\_qual - board type: HL-225D  
hl\_qual - board serial: AN40008942  
hl\_qual - PCI link speed (gen): 16GT/s  
hl\_qual - PCI link width: x16  
hl\_qual - HBM size: 96 GB  
[19:13:34.249] Start running plugin  
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini  
hl\_qual - Plugin report END:

hl\_qual - Plugin report START, bus ID: 0000:9a:00.0  
hl\_qual - dev number bus ID: 0000:9a:00.0  
hl\_qual - board type: HL-225D  
hl\_qual - board serial: AN40008950  
hl\_qual - PCI link speed (gen): 16GT/s  
hl\_qual - PCI link width: x16  
hl\_qual - HBM size: 96 GB  
[19:13:34.158] Start running plugin  
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini  
hl\_qual - Plugin report END:

hl\_qual - Plugin report START, bus ID: 0000:bb:00.0  
hl\_qual - dev number bus ID: 0000:bb:00.0  
hl\_qual - board type: HL-225D  
hl\_qual - board serial: AN34020846  
hl\_qual - PCI link speed (gen): 16GT/s  
hl\_qual - PCI link width: x16  
hl\_qual - HBM size: 96 GB  
[19:13:34.160] Start running plugin  
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini  
hl\_qual - Plugin report END:

hl\_qual - Plugin report START, bus ID: 0000:4b:00.0  
hl\_qual - dev number bus ID: 0000:4b:00.0  
hl\_qual - board type: HL-225D  
hl\_qual - board serial: AN40008953  
hl\_qual - PCI link speed (gen): 16GT/s  
hl\_qual - PCI link width: x16  
hl\_qual - HBM size: 96 GB  
[19:13:34.147] Start running plugin  
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini  
hl\_qual - Plugin report END:

hl\_qual - Plugin report START, bus ID: 0000:5d:00.0  
hl\_qual - dev number bus ID: 0000:5d:00.0  
hl\_qual - board type: HL-225D  
hl\_qual - board serial: AN40008879  
hl\_qual - PCI link speed (gen): 16GT/s  
hl\_qual - PCI link width: x16  
hl\_qual - HBM size: 96 GB  
[19:13:34.147] Start running plugin  
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini  
hl\_qual - Plugin report END:

hl\_qual - Plugin report START, bus ID: 0000:18:00.0  
hl\_qual - dev number bus ID: 0000:18:00.0  
hl\_qual - board type: HL-225D  
hl\_qual - board serial: AN34021241  
hl\_qual - PCI link speed (gen): 16GT/s  
hl\_qual - PCI link width: x16  
hl\_qual - HBM size: 96 GB  
[19:13:34.147] Start running plugin  
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini  
hl\_qual - Plugin report END:

hl\_qual - Plugin report START, bus ID: 0000:3c:00.0  
hl\_qual - dev number bus ID: 0000:3c:00.0  
hl\_qual - board type: HL-225D  
hl\_qual - board serial: AN40008881  
hl\_qual - PCI link speed (gen): 16GT/s  
hl\_qual - PCI link width: x16  
hl\_qual - HBM size: 96 GB  
[19:13:34.147] Start running plugin  
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini  
hl\_qual - Plugin report END:

################################## Device measurment report #######################################

0000:ca:00.0 0000:db:00.0 0000:9a:00.0 0000:bb:00.0 0000:4b:00.0 0000:5d:00.0 0000:18:00.0 0000:3c:00.0  
Min Temp: 0 0 0 0 0 0 0 0  
Max Temp: 0 0 0 0 0 0 0 0  
min Clock: 0 0 0 0 0 0 0 0  
max Clock: 0 0 0 0 0 0 0 0  
min Power\_54V: 0 0 0 0 0 0 0 0  
max Power\_54V: 0 0 0 0 0 0 0 0  
Sram Serr count: 0 0 0 0 0 0 0 0  
Sram Derr count: 0 0 0 0 0 0 0 0  
Dram Serr count: 0 0 0 0 0 0 0 0  
Dram Derr count: 0 0 0 0 0 0 0 0  
################################## Test result summary #######################################  
hl\_qual - device with busID: 0000:ca:00.0 result: FAILED  
hl\_qual - device with busID: 0000:db:00.0 result: FAILED  
hl\_qual - device with busID: 0000:9a:00.0 result: FAILED  
hl\_qual - device with busID: 0000:bb:00.0 result: FAILED  
hl\_qual - device with busID: 0000:4b:00.0 result: FAILED  
hl\_qual - device with busID: 0000:5d:00.0 result: FAILED  
hl\_qual - device with busID: 0000:18:00.0 result: FAILED  
hl\_qual - device with busID: 0000:3c:00.0 result: FAILED  
################################## hl qual report #######################################  
FAILED