Is there any way to check the health/status of Gaudi cards? mem usage, processes running, etc

The hl-smi utility tool can provide this information on querying the status of Gaudi cards, including total and available memory. This is documented in the System Management Tools guide.