GCP - Cloud Storage gsutil Commands

Why to Learn gsutil Command?

• In the GCP certification exam, there will be few questions based on the gsutil commands. 
• If you are working on Google Cloud projects, then also you should be knowing about these commands as these are faster way to work on Cloud Storage.

What is gsutil?

gsutil is a Command-line tool to work with Google Cloud Storage through CLI. You can install Google Cloud SDK and other command-line tools on your system like gcloud, gsutil and kubectl. 

There is another way to access these command-line tools is through Cloud Shell. Cloud Shell is an instance provisioned with 5TB persistent disk storage. This is embedded in the Google Cloud Console itself so you can access it from the browser. gcloud, gsutil and kubectl command-line tools are already installed in it. 

Useful gsutil Commands

Command to Create Bucket

gsutil mb -b on -l us-east1 gs://my-first-bucket/

mb - make bucket

-b  - to specify the uniform bucket-level access setting. Default is 'off'. It can be set as 'on' as done in the above command.

-l - to specify the location in which you want to create bucket

us-east1 - location in which you want to create bucket

my-first-bucket - name of the bucket (it should be globally unique)

This command will create a bucket my-first-bucket in us-east1 region.

Command to Create Bucket with more Specifications

gsutil mb -c nearline -l europe-west1 -p myproject1 --retention 2m gs://my-first-bucket/

mb - make bucket

-c  - to specify the storage class.

-l - to specify the location in which you want to create bucket

-p - to specify the project with which you want to associate this bucket

--retention  - to specify the retention period of objects in the bucket. It can be specified in s (seconds), d (days), m (months) or y (year). If don't specify, there won't be any retention policy applied to the objects in the bucket.

my-first-bucket - name of the bucket (it should be globally unique)

This command will create a bucket my-first-bucket in europe-west1 region with the storage class as nearline, in the myproject1 project and the objects inside the bucket will have retention period of 2 months.

Command to Upload files to Bucket

gsutil cp Data/cloud.png gs://my-first-bucket

cp - copy

Data/cloud.png - file to be copied with its path

my-first-bucket - name of the bucket (it should be globally unique)

This command will upload cloud.png file from your local folder Data to the bucket my-first-bucket.

Command to Upload all the files from a folder to Bucket

gsutil cp -r Data gs://my-first-bucket

cp - copy

-r - to specify all files from a folder

Data - name of the folder from which all files need to be uploaded

my-first-bucket - name of the bucket (it should be globally unique)

This command will upload Data folder with all its contents to the bucket my-first-bucket.

Command to Upload multiple files in Parallel Multi-threaded way to Bucket

gsutil -m cp -r Data gs://my-first-bucket

-m - copy large number of files in a parallel multi-threaded/multi-processing way

cp - copy

-r - to specify all files from a folder

Data - name of the folder from which all files need to be uploaded

my-first-bucket - name of the bucket (it should be globally unique)

This command will upload Data folder with all its contents to the bucket my-first-bucket in a parallel multi-threaded way.

Command to List Content of a Bucket

gsutil ls gs://my-first-bucket

ls - list

my-first-bucket - name of the bucket (it should be globally unique)

This command will return the object names inside the bucket my-first-bucket.

Command to List Details of Object inside a Bucket

gsutil ls -l gs://my-first-bucket/cloud.png

ls - list

-l - flag to get details of object

my-first-bucket - name of the bucket (it should be globally unique)

This command will return the object cloud.png details like size of object, Creation Date inside the bucket my-first-bucket.

Command to Download the Object from a Bucket

gsutil cp gs://my-first-bucket/cloud.png DataFolder/

cp - copy

my-first-bucket - name of the bucket (it should be globally unique)

This command will copy the object cloud.png from the bucket my-first-bucket to DataFolder on your machine.

Command to Grant Read-permission of Objects inside a Bucket to All Users

gsutil iam ch allUsers:objectViewer gs://my-first-bucket

iam ch- to change the IAM permission

allUsers:objectViewer - changing the permission to objectViewer, i.e Read access to all users.

my-first-bucket - name of the bucket (it should be globally unique)

This command will grant read access of objects inside the bucket my-first-bucket to all users.

Command to Remove Public access of Objects inside a Bucket

gsutil iam ch -d allUsers:objectViewer gs://my-first-bucket

iam ch- to change the IAM permission

-d allUsers:objectViewer - delete the objectViewer, i.e Read access to all users.

my-first-bucket - name of the bucket (it should be globally unique)

This command will remove the read access of objects inside the bucket my-first-bucket to all users.

Command to Grant Read permission of Objects inside a Bucket to Specific User

gsutil iam ch user:abc@gmail.com:objectViewer gs://my-first-bucket

iam ch- to change the IAM permission

user:abc@gmail.com:objectViewer - changing the permission to objectViewer, i.e Read access to user abc@gmail.com.

my-first-bucket - name of the bucket (it should be globally unique)

This command will grant read access of objects inside the bucket my-first-bucket to user abc@gmail.com.

Command to Grant Read-Write permission of Objects inside a Bucket to Specific User

gsutil iam ch user:abc@gmail.com:objectCreator,objectViewer gs://my-first-bucket

iam ch- to change the IAM permission

user:abc@gmail.com:objectCreator,objectViewer - changing the permission to objectCreator and objectViewer, i.e Write & Read access to user abc@gmail.com.

my-first-bucket - name of the bucket (it should be globally unique)

This command will grant read & write access of objects inside the bucket my-first-bucket to user abc@gmail.com.

Command to Remove the specific user's permissions of Objects inside a Bucket

gsutil iam ch -d user:abc@gmail.com:objectCreator, objectViewer gs://my-first-bucket

iam ch- to change the IAM permission

-d user:abc@gmail.com:objectCreator,objectViewer - delete the objectCreator, objectViewer, i.e Write & Read access from user abc@gmail.com.

my-first-bucket - name of the bucket (it should be globally unique)

This command will remove the read & write access of objects inside the bucket my-first-bucket from user abc@gmail.com.

Command to Delete an Object from a Bucket

gsutil rm gs://my-first-bucket/cloud.png

rm- to remove

my-first-bucket - name of the bucket (it should be globally unique)

cloud.png - name of the object to be removed

This command will remove the object cloud.png from the bucket my-first-bucket.

 

Command to Delete a Bucket and all contents inside the Bucket

gsutil rm -r gs://my-first-bucket

rm -r- to remove all recursively

my-first-bucket - name of the bucket (it should be globally unique)

This command will remove the bucket my-first-bucket anad all objects inside it recursively.

Command to get help on specific gsutil command line option

gsutil help cp

This command will provide you details on gsutil cp command.

Command to get details on gsutil top-level command line options

gsutil help option

This command will provide you details on all gsutil top-level command line options.