In this post I will show you how to access IBM Cloud Object Storage using Go Cloud.

Background

When photographing using RAW files, you need a lot of storage capacity to save all those photos. Many photographers are using a NAS at home or some kind of online storage SaaS like Google Drive or Dropbox Pro. But as a developer with knowledge on clouds and additionally working for a company operating its own cloud, this is too simple. Therefore, I am developing my own storage system based on cloud technology.

What I am currently doing is uploading all the files immediately to an IBM Cloud Object Storage. Why am I using IBM Cloud for this? Because in my comparison of different Cloud Storage's it had the best cost-benefit ratio for me (but compare for yourself). Simultaneously I am working on a management platform for synchronization and managing all these photos. I am planning to report from my efforts with some simple examples about how I am doing things.

This is the first post about this project showing how I interact with my IBM Cloud Object Storage bucket using Go Cloud.

What is Go Cloud?

Google announced on Google Cloud Next 2018 it's 'Portable Cloud Programming' toolkit Go Cloud Development Kit (CDK) (aka Go Cloud). It's purpose supporting teams in multi-cloud and hybrid-cloud architectures.

Basically it's a set of libraries abstracting the platform dependent parts and providing a generic and open cloud API. This means when I develop my application with this library, it requires only a few changes to change the cloud provider. So I don't have a vendor lock-in and can choose the cloud provider with the best offering.

Why is it working with IBM COS?

IBM Cloud Object Storage has a S3-compatible API, that's why it can use the S3-implementation of Go Cloud. The only thing you have to mind is to connect to a different endpoint.

Listing all objects on storage bucket

In this post I will show you how to access IBM Cloud Object Storage using Go Cloud. Let's start!

Prerequisites

  • IBM Cloud Account: If you need one create a free account here
  • Installed Go tool chain: I am currently using version 1.15.1

Creating IBM Cloud Object Storage Bucket

  1. Login to your IBM Cloud Account and search in the catalog for Object Storage. Create an instance and you will be forwarded to the Getting started page of your Cloud Storage instance.
  2. Next we have to create a Storage Bucket. Click on Create bucket and use the predefined Quick Start bucket. Next you have to enter a unique bucket name. On the next page you have a chance to upload some files. I suggest uploading two or three small files, so you can see something when executing the program later.
  3. Exit the wizard using the View bucket configuration button. On this site you need to note some details:
    • Bucket name
    • Location
    • Public endpoint (it's further down on the page, e.g. s3.eu-de.cloud-object-storage.appdomain.cloud)

View Configuration properties

  1. What are we missing now? Right, the credentials. IBM has auto-generated a service account for you, but we need HMAC credentials to access the bucket with Go Cloud. Navigate to the Service Credentials tab on the left and click on New Credential. Choose Role Reader, open the Advanced options and activate the Include HMAC Credentials option. Finish this by clicking on Add.

The created credentials will appear in the list and when opened it shows a JSON document. Extract the following three properties:

  • apikey
  • cos_hmac_keys.access_key_id
  • cos_hmac_keys.secret_access_key

HMAC Service credentials

Creating the program

Let's start using Go Cloud. Create a new Go project named go-cloud-ibm-cos and add Go Cloud as dependency:

mkdir ~/go/src/github.com/schoenenberg/go-cloud-ibm-cos # $_ is a bash shortcut for using the last argument of the previous command cd $_ # Initialize Go modules go mod init # Add Go Cloud as dependency go get -u gocloud.dev # For the s3 constructor we also need the AWS SDK go get -u github.com/aws/aws-sdk-go/aws 

Next we create a package bucketop where we create all those operations on a bucket. We separate them in another package to be able to reuse them later on. Let's start with the ListObjects function, which shall return all objects in the bucket:

// file: ./pkg/bucketop/list.go package bucketop import ( "context" "io" "gocloud.dev/blob" ) // ListObjects lists all objects saved in the provided bucket func ListObjects( ctx context.Context, bkt *blob.Bucket, ) (*[]blob.ListObject, error) { // Init result array of Objects objects := make([]blob.ListObject, 0) // Do a list request on the bucket iter := bkt.List(nil) // Iterate through each element of the iterator for { obj, err := iter.Next(ctx) if err == io.EOF { break } else if err != nil { return nil, err } objects = append(objects, *obj) } return &objects, nil } 

The cool thing about this function is we just provide a *blob.Bucket as an argument and this already abstract the underlying Object Storage technology, so we can use this function also with e.g. Google Cloud Storage and Azure Blob Storage.

Finally, we are developing a simple application to list all the files in a bucket using our ListObjects function.

// file: ./bin/list_objects/main.go package main import ( "context" "fmt" "log" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/credentials" "github.com/aws/aws-sdk-go/aws/session" "gocloud.dev/blob/s3blob" "github.com/schoenenberg/go-cloud-ibm-cos/pkg/bucketop" ) // Enter your credentials here! const ( endpoint = "<public endpoint>" region = "<location>" bucketName = "<bucket name>" apiKey = "<apikey>" secret = "<secret_access_key>" keyId = "<access_key_id>" ) func main() { ctx := context.Background() // Create a session with s3 constructor sess, err := session.NewSession(&aws.Config{ Region: aws.String(region), Endpoint: aws.String(endpoint), Credentials: credentials.NewStaticCredentials( keyId, secret, apiKey, ), }) if err != nil { log.Fatalln(err) } // Open the bucket bucket, err := s3blob.OpenBucket( ctx, sess, bucketName, nil, ) if err != nil { log.Fatalln(err) } defer bucket.Close() // Use our ListObjects function objs, err := bucketop.ListObjects(ctx, bucket) if err != nil { log.Fatalln(err) } // And print all objects with its size for _, obj := range *objs { fmt.Printf("%s - Size %.2f MB\n", obj.Key, float64(obj.Size)/1000000.0, ) } } 

What we are basically doing is creating a session, opening the bucket and executing our ListObjects function. Afterwards, we print the result. To execute this example run go run ./bin/list_objects on your command line:

go run ./bin/list_objects # example.txt - Size 0.08 MB # hello_world.txt - Size 0.22 MB 

This is it! If you don't want to copy&paste the code from this post you can also clone or fork my repo from Github.

Summary

We developed a small application using Google's Go Cloud Development Kit to access an IBM Cloud Storage bucket and listing all the files in it.

References

Disclaimer: The postings on this site are my own and don't necessarily represent my employer IBM's positions, strategies or opinions.