I tried to run hl_qual resnet50 training. All the preparation works are done, and I run the following cmd:
./hl_qual -gaudi2 -c all -rmod parallel -trainingApp -epoch 3 -train_list train_list.txt
After 20s, it stopped and throw this error. Anyone can help with that?
HL-SMI Version: hl-1.18.0-fw-53.1.1.1
Driver Version: 1.18.0-ee698fb
################################## hl_qual #########################################
HL-QUAL VERSION: 524-b50bd67
Failed (WIFSIGNALED): Process 1912190 (index 5, bus ID: 0000:5d:00.0) was terminated by signal 11.
device 5 core dump produced
Failed (WIFSIGNALED): Process 1912189 (index 4, bus ID: 0000:4b:00.0) was terminated by signal 11.
device 4 core dump produced
Failed (WIFSIGNALED): Process 1912188 (index 3, bus ID: 0000:bb:00.0) was terminated by signal 11.
device 3 core dump produced
Failed (WIFSIGNALED): Process 1912192 (index 7, bus ID: 0000:3c:00.0) was terminated by signal 11.
device 7 core dump produced
Failed (WIFSIGNALED): Process 1912185 (index 0, bus ID: 0000:ca:00.0) was terminated by signal 11.
device 0 core dump produced
Failed (WIFSIGNALED): Process 1912186 (index 1, bus ID: 0000:db:00.0) was terminated by signal 11.
device 1 core dump produced
Failed (WIFSIGNALED): Process 1912191 (index 6, bus ID: 0000:18:00.0) was terminated by signal 11.
device 6 core dump produced
Failed (WIFSIGNALED): Process 1912187 (index 2, bus ID: 0000:9a:00.0) was terminated by signal 11.
device 2 core dump produced
All processes finished successfully.
################################## hl_qual command line #########################################
./hl_qual -gaudi2 -c all -rmod parallel -trainingApp -epoch 3 -train_list train_list.txt
################################## Test run report #########################################
hl_qual - Plugin report START, bus ID: 0000:ca:00.0
hl_qual - dev number bus ID: 0000:ca:00.0
hl_qual - board type: HL-225D
hl_qual - board serial: AN40008959
hl_qual - PCI link speed (gen): 16GT/s
hl_qual - PCI link width: x16
hl_qual - HBM size: 96 GB
[19:13:34.156] Start running plugin
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini
hl_qual - Plugin report END:
hl_qual - Plugin report START, bus ID: 0000:db:00.0
hl_qual - dev number bus ID: 0000:db:00.0
hl_qual - board type: HL-225D
hl_qual - board serial: AN40008942
hl_qual - PCI link speed (gen): 16GT/s
hl_qual - PCI link width: x16
hl_qual - HBM size: 96 GB
[19:13:34.249] Start running plugin
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini
hl_qual - Plugin report END:
hl_qual - Plugin report START, bus ID: 0000:9a:00.0
hl_qual - dev number bus ID: 0000:9a:00.0
hl_qual - board type: HL-225D
hl_qual - board serial: AN40008950
hl_qual - PCI link speed (gen): 16GT/s
hl_qual - PCI link width: x16
hl_qual - HBM size: 96 GB
[19:13:34.158] Start running plugin
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini
hl_qual - Plugin report END:
hl_qual - Plugin report START, bus ID: 0000:bb:00.0
hl_qual - dev number bus ID: 0000:bb:00.0
hl_qual - board type: HL-225D
hl_qual - board serial: AN34020846
hl_qual - PCI link speed (gen): 16GT/s
hl_qual - PCI link width: x16
hl_qual - HBM size: 96 GB
[19:13:34.160] Start running plugin
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini
hl_qual - Plugin report END:
hl_qual - Plugin report START, bus ID: 0000:4b:00.0
hl_qual - dev number bus ID: 0000:4b:00.0
hl_qual - board type: HL-225D
hl_qual - board serial: AN40008953
hl_qual - PCI link speed (gen): 16GT/s
hl_qual - PCI link width: x16
hl_qual - HBM size: 96 GB
[19:13:34.147] Start running plugin
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini
hl_qual - Plugin report END:
hl_qual - Plugin report START, bus ID: 0000:5d:00.0
hl_qual - dev number bus ID: 0000:5d:00.0
hl_qual - board type: HL-225D
hl_qual - board serial: AN40008879
hl_qual - PCI link speed (gen): 16GT/s
hl_qual - PCI link width: x16
hl_qual - HBM size: 96 GB
[19:13:34.147] Start running plugin
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini
hl_qual - Plugin report END:
hl_qual - Plugin report START, bus ID: 0000:18:00.0
hl_qual - dev number bus ID: 0000:18:00.0
hl_qual - board type: HL-225D
hl_qual - board serial: AN34021241
hl_qual - PCI link speed (gen): 16GT/s
hl_qual - PCI link width: x16
hl_qual - HBM size: 96 GB
[19:13:34.147] Start running plugin
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini
hl_qual - Plugin report END:
hl_qual - Plugin report START, bus ID: 0000:3c:00.0
hl_qual - dev number bus ID: 0000:3c:00.0
hl_qual - board type: HL-225D
hl_qual - board serial: AN40008881
hl_qual - PCI link speed (gen): 16GT/s
hl_qual - PCI link width: x16
hl_qual - HBM size: 96 GB
[19:13:34.147] Start running plugin
Dataloader init throw an exception : Failed to read decoder configuration file: ./resnet50DecoderConfig.ini
hl_qual - Plugin report END:
################################## Device measurment report #######################################
0000:ca:00.0 0000:db:00.0 0000:9a:00.0 0000:bb:00.0 0000:4b:00.0 0000:5d:00.0 0000:18:00.0 0000:3c:00.0
Min Temp: 0 0 0 0 0 0 0 0
Max Temp: 0 0 0 0 0 0 0 0
min Clock: 0 0 0 0 0 0 0 0
max Clock: 0 0 0 0 0 0 0 0
min Power_54V: 0 0 0 0 0 0 0 0
max Power_54V: 0 0 0 0 0 0 0 0
Sram Serr count: 0 0 0 0 0 0 0 0
Sram Derr count: 0 0 0 0 0 0 0 0
Dram Serr count: 0 0 0 0 0 0 0 0
Dram Derr count: 0 0 0 0 0 0 0 0
################################## Test result summary #######################################
hl_qual - device with busID: 0000:ca:00.0 result: FAILED
hl_qual - device with busID: 0000:db:00.0 result: FAILED
hl_qual - device with busID: 0000:9a:00.0 result: FAILED
hl_qual - device with busID: 0000:bb:00.0 result: FAILED
hl_qual - device with busID: 0000:4b:00.0 result: FAILED
hl_qual - device with busID: 0000:5d:00.0 result: FAILED
hl_qual - device with busID: 0000:18:00.0 result: FAILED
hl_qual - device with busID: 0000:3c:00.0 result: FAILED
################################## hl qual report #######################################
FAILED