# Problem to train the local dataset using Llama2 Fine-Tuning with Low-Rank Adaptations (LoRA) on Intel® Gaudi®2 AI Accelerator
**URL:** https://forum.habana.ai/t/problem-to-train-the-local-dataset-using-llama2-fine-tuning-with-low-rank-adaptations-lora-on-intel-gaudi-2-ai-accelerator/1249
**Category:** Training
**Tags:** models, advisory, tensorflow
**Created:** [July 21, 2024, 3:27am UTC](https://forum.habana.ai/t/problem-to-train-the-local-dataset-using-llama2-fine-tuning-with-low-rank-adaptations-lora-on-intel-gaudi-2-ai-accelerator/1249 "2024-07-21T03:27:24Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Afzal](https://forum.habana.ai/u/Afzal)
#### Post date: [July 21, 2024, 3:27am UTC](https://forum.habana.ai/t/problem-to-train-the-local-dataset-using-llama2-fine-tuning-with-low-rank-adaptations-lora-on-intel-gaudi-2-ai-accelerator/1249/1 "2024-07-21T03:27:24Z")
followed procedure as mentioned link: [Llama2 Fine-Tuning with Low-Rank Adaptations (LoRA) on Intel® Gaudi®2 AI Accelerator - Intel Gaudi Developers (habana.ai)](https://developer.habana.ai/tutorials/pytorch/llama2-fine-tuning-with-low-rank-adaptations-lora-on-intel-gaudi2-ai-accelerator/)
how to train a model with local dataset, tried method as blow mentioned python3 ../gaudi\_spawn.py --use\_deepspeed --world\_size 8 run\_lora\_clm.py
–model\_name\_or\_path meta-llama/CodeLlama-7b-Instruct-hf
–deepspeed llama2\_ds\_zero2\_config.json
–train\_file /optimum-habana/examples/language-modeling/train-data.jsonl
–bf16 True
–output\_dir ./lora\_out
–num\_train\_epochs 2
–max\_seq\_len 2048
–per\_device\_train\_batch\_size 10
–per\_device\_eval\_batch\_size 10
–gradient\_checkpointing
–evaluation\_strategy epoch
–eval\_delay 2
–save\_strategy no
–learning\_rate 0.0018
–warmup\_ratio 0.03
–lr\_scheduler\_type “cosine”
–logging\_steps 1
–dataset\_concatenation
–attn\_softmax\_bf16 True
–do\_train
–do\_eval
–use\_habana
–use\_lazy\_mode
–pipelining\_fwd\_bwd
–throughput\_warmup\_steps 3
–lora\_rank 4
–lora\_target\_modules “q\_proj” “v\_proj” “k\_proj” “o\_proj”
–validation\_split\_percentage 4
# After Execution above got error like : raceback (most recent call last):
File “/optimum-habana/examples/language-modeling/run\_lora\_clm.py”, line 754, in
main()
File “/optimum-habana/examples/language-modeling/run\_lora\_clm.py”, line 529, in main
raise ValueError(“Unsupported dataset”)
ValueError: Unsupported dataset
Please suggest the best way to train a model using local dataset with intel Gaudi 2 AI accelerator
---
### Author:  [@Sayantan\_S](https://forum.habana.ai/u/Sayantan_S)
#### Post date: [July 21, 2024, 3:34am UTC](https://forum.habana.ai/t/problem-to-train-the-local-dataset-using-llama2-fine-tuning-with-low-rank-adaptations-lora-on-intel-gaudi-2-ai-accelerator/1249/2 "2024-07-21T03:34:36Z")
The cmd line you posted says:
–train\_file /optimum-habana/examples/language-modeling/train-data.jsonl
Is the extension “jsonl”, or is it a typo and is it supposed to be “json”?
I’ll check and get back what is the expected format of the dataset
---
### Author:  [@Afzal](https://forum.habana.ai/u/Afzal)
#### Post date: [July 22, 2024, 4:16pm UTC](https://forum.habana.ai/t/problem-to-train-the-local-dataset-using-llama2-fine-tuning-with-low-rank-adaptations-lora-on-intel-gaudi-2-ai-accelerator/1249/3 "2024-07-22T16:16:13Z")
Yes this is jsonl extension, I have tried with different extension as well like .csv, json and parquet but getting same error. I request please help me to fix the issue, if required we will have a working session to get the things clear.
Thanks, in advance,
---
### Author:  [@Sayantan\_S](https://forum.habana.ai/u/Sayantan_S)
#### Post date: [July 22, 2024, 5:21pm UTC](https://forum.habana.ai/t/problem-to-train-the-local-dataset-using-llama2-fine-tuning-with-low-rank-adaptations-lora-on-intel-gaudi-2-ai-accelerator/1249/4 "2024-07-22T17:21:32Z")
The [readme](https://github.com/huggingface/optimum-habana/tree/main/examples/language-modeling) has an example:
```auto
python3 run_lora_clm.py \
--model_name_or_path huggyllama/llama-7b \
--dataset_name tatsu-lab/alpaca \
...
```
tatsu-lab/alpaca is a sample dataset
Please check the preprocessing section on run\_lora\_clm.py to check what transformations happen on the data.
First loaded [here](https://github.com/huggingface/optimum-habana/blob/v1.12.1/examples/language-modeling/run_lora_clm.py#L470)
sample from the dataset at this point:
```auto
raw_datasets["train"]
Dataset({
features: ['instruction', 'input', 'output', 'text'],
num_rows: 52002
})
(Pdb) raw_datasets["train"][0]
{'instruction': 'Give three tips for staying healthy.', 'input': '', 'output': '1.Eat a balanced diet and make sure to include plenty of fruits and vegetables. \n2. Exercise regularly to keep your body active and strong. \n3. Get enough sleep and maintain a consistent sleep schedule.', 'text': 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGive three tips for staying healthy.\n\n### Response:\n1.Eat a balanced diet and make sure to include plenty of fruits and vegetables. \n2. Exercise regularly to keep your body active and strong. \n3. Get enough sleep and maintain a consistent sleep schedule.'}
```
At this [point](https://github.com/huggingface/optimum-habana/blob/v1.12.1/examples/language-modeling/run_lora_clm.py#L485):
```auto
raw_datasets["train"]
Dataset({
features: ['instruction', 'input', 'output', 'text'],
num_rows: 49922
})
(Pdb) raw_datasets["train"][0]
{'instruction': 'Compare and contrast the use of web technologies between web 1.0 and web 2.0', 'input': '', 'output': 'The technology used in the original web (web 1.0) was limited to static webpages, and the primary focus was on the visual aspects of web design. Web 1.0 technologies lacked most of the interactivity and personalization that is commonplace today.\n\nThe use of web technologies in web 2.0 are much more complex, with the focus being on the development of dynamic webpages that can offer user interaction, personalization, and social interaction. Content is the main emphasis, and user content and interaction is encouraged. This encourages the development of communities and apps using extensive backend programming.', 'text': 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCompare and contrast the use of web technologies between web 1.0 and web 2.0\n\n### Response:\nThe technology used in the original web (web 1.0) was limited to static webpages, and the primary focus was on the visual aspects of web design. Web 1.0 technologies lacked most of the interactivity and personalization that is commonplace today.\n\nThe use of web technologies in web 2.0 are much more complex, with the focus being on the development of dynamic webpages that can offer user interaction, personalization, and social interaction. Content is the main emphasis, and user content and interaction is encouraged. This encourages the development of communities and apps using extensive backend programming.'}
```
Finally [this](https://github.com/huggingface/optimum-habana/blob/v1.12.1/examples/language-modeling/run_lora_clm.py#L538) gets called, creating prompts:
```auto
(Pdb) type(prompts)
(Pdb) prompts.keys()
dict_keys(['source', 'target'])
(Pdb) prompts['source'][0]
'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCompare and contrast the use of web technologies between web 1.0 and web 2.0\n\n### Response:'
(Pdb) prompts['target'][0]
'The technology used in the original web (web 1.0) was limited to static webpages, and the primary focus was on the visual aspects of web design. Web 1.0 technologies lacked most of the interactivity and personalization that is commonplace today.\n\nThe use of web technologies in web 2.0 are much more complex, with the focus being on the development of dynamic webpages that can offer user interaction, personalization, and social interaction. Content is the main emphasis, and user content and interaction is encouraged. This encourages the development of communities and apps using extensive backend programming.'
```
Finally after all this, we get:
```auto
raw_datasets['train']
Dataset({
features: ['prompt_sources', 'prompt_targets'],
num_rows: 49922
})
(Pdb) raw_datasets['train'][0]
{'prompt_sources': 'Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nCompare and contrast the use of web technologies between web 1.0 and web 2.0\n\n### Response:', 'prompt_targets': 'The technology used in the original web (web 1.0) was limited to static webpages, and the primary focus was on the visual aspects of web design. Web 1.0 technologies lacked most of the interactivity and personalization that is commonplace today.\n\nThe use of web technologies in web 2.0 are much more complex, with the focus being on the development of dynamic webpages that can offer user interaction, personalization, and social interaction. Content is the main emphasis, and user content and interaction is encouraged. This encourages the development of communities and apps using extensive backend programming.'}
```
So basically, you need a dataset with 2 keys, ‘prompt\_sources’, ‘prompt\_targets’