Can I train DistilBERT with Gaudi?

Training
1

Can I train DistilBERT with Gaudi? How?

1 Like
2

Hi Jonathan,
Distilbert for PyTorch is under progress. You can track Distilbert (and other models) progress in our roadmap here. As for Distilbert for Tensorflow, you can follow the instructions on the migration guide here and port your Distilbert model to Gaudi. If you run into issues, please reach out to us.

2 Likes
3

@Sayantan_S Why wouldn’t Distilbert work by default on PyTorch?

Why isn’t it just a matter of importing the Habana module properly using load_habana_module())?

What would be the changes required in case I would like to make port myself?

4

Hi @gustavozomer, we now support Distilbert on PyTorch. Please see our reference model here