Graph compile failed error when running txt2image.py from Habana Model-References repo

Inference
1

Environment: AWS DL1, Ubuntu 22.04 (bare metal driver install), Python 3.10.12, SynapseAI 1.12.1

Running in habanalabs-venv on the host OS (no container)

Followed instructions from Readme.md

$ python3 scripts/txt2img.py --prompt “a professional photograph of an astronaut riding a horse” --ckpt v2-1_768-ema-pruned.ckpt --config configs/stable-diffusion/v2-inference-v.yaml --H 768 --W 768 --n_samples 1 --n_iter 3 --use_hpu_graph

Seed set to 42
Loading model from v2-1_768-ema-pruned.ckpt
Global Step: 110000
LatentDiffusion: Running in v-prediction mode
DiffusionWrapper has 865.91 M params.
making attention of type ‘vanilla’ with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type ‘vanilla’ with 512 in_channels
============================= HABANA PT BRIDGE CONFIGURATION ===========================
PT_HPU_LAZY_MODE = 1
PT_RECIPE_CACHE_PATH =
PT_CACHE_FOLDER_DELETE = 0
PT_HPU_RECIPE_CACHE_CONFIG =
PT_HPU_MAX_COMPOUND_OP_SIZE = 9223372036854775807
PT_HPU_LAZY_ACC_PAR_MODE = 1
PT_HPU_ENABLE_REFINE_DYNAMIC_SHAPES = 0
---------------------------: System Configuration :---------------------------
Num CPU Cores : 96
CPU RAM : 784282744 KB
Data shape for DDIM sampling is (1, 4, 96, 96), eta 0.0
Compiling HPU graph encode_with_transformer
Traceback (most recent call last):
File “/home/ubuntu/habanalabs-venv/Model-References/PyTorch/generative_models/stable-diffusion-v-2-1/scripts/txt2img.py”, line 360, in
main(opt)
File “/home/ubuntu/habanalabs-venv/Model-References/PyTorch/generative_models/stable-diffusion-v-2-1/scripts/txt2img.py”, line 300, in main
c_in = runner.run(model.cond_stage_model.encode_with_transformer, tokens)
File “/home/ubuntu/habanalabs-venv/Model-References/PyTorch/generative_models/stable-diffusion-v-2-1/scripts/txt2img.py”, line 222, in run
graph.capture_begin()
File “/home/ubuntu/habanalabs-venv/lib/python3.10/site-packages/habana_frameworks/torch/hpu/graphs.py”, line 34, in capture_begin
_hpu_C.capture_begin(self.hpu_graph, dry_run)
RuntimeError: Graph compile failed. synStatus=synStatus 26 [Generice failure].

Issue also logged on Github:

2

Could you please let me know which stable-diffusion you are using. There are 3 here:

Probably one of stable-diffusion-v-2-1 or stable-diffusion-finetuning ?

3

Given your command line, I assume this one:
https://github.com/HabanaAI/Model-References/tree/master/PyTorch/generative_models/stable-diffusion-v-2-1 ?

4

I am able to run this on Gaudi2 on 1.13-463 docker (1.13.0 branch of model-references), with 1.13 firmware (as shown by hl-smi)

on 1.12.1docker if i checkout out 1.12.1branch on model references I can run it as well.

I see that if I run with model-references on branch=1.12.1, and docker =1.13/firmware=1.13, it errors out. Can you please confirm if your model-references, firmware and docker are all on the same version?