Advisory: Issue with S3 connectivity; Error: File system scheme 's3' not implemented

Training FAQ
,
1

When running the TensorFlow version 2.5.1 or 2.6.0 in a DL1 instance, there may be connectivity issues between the instance and AWS S3 storage, as the Tensorflow IO plugins have been moved to a different module. Future releases of SynapseAI Software will have the tensorflow_io package included in Docker and Base AMI releases. To make Tensorflow 2.5.1 or 2.6.0 able to access AWS S3 Storage (example s3://…), the tensorflow_io package is needed. To enable tensorflow_io, there are 2 steps:

First Step: Install the pip package tensorflow_io in the console:

for TensorFlow 2.5.1
set this Environment variable: TF_USE_MODULAR_FILESYSTEM=1
export PYTHON=/usr/bin/python3.7
${PYTHON} -m pip install --no-deps tensorflow_io==0.19.0
${PYTHON} -m pip install --no-deps tensorflow-io-gcs-filesystem==0.19.0

for TensorFlow 2.6.0
export PYTHON=/usr/bin/python3.7
${PYTHON} -m pip install --no-deps tensorflow_io==0.21.0
${PYTHON} -m pip install --no-deps tensorflow-io-gcs-filesystem==0.21.0

Note: for the Python environment, the following is loaded by default in Habana TensorFlow
python3.7: Ubuntu18.04, Amazon Linux 2
python3.8: Ubuntu20.04, RHEL8.3, Centos8.3

Second Step: After importing tensorflow, import tensorflow_io - In the python code
import tensorflow as tf
import tensorflow_io as tfio

Below is a quick example to verify the fix

Before, this is the typical error observed:
import tensorflow as tf
tf.io.gfile.exists(’#S3 LINK’)
Traceback (most recent call last): File “”, line 1, in File “/usr/local/lib/python3.8/dist-packages/tensorflow/python/lib/io/file_io.py”, line 291, in file_exists_v2 _pywrap_file_io.FileExists(compat.path_to_bytes(path))tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme ‘s3’ not implemented (file: ‘#S3 LINK’)

After, installing the tensorflow_io package:
import tensorflow as tf
import tensorflow_io as tfio
tf.io.gfile.exists(’#S3 LINK’)
True

2 Likes