Gocloud — designing for three clouds at once

I contributed to Gocloud, a Go library that aims to give you a single API for AWS, GCP, and Azure across common service categories — storage, secrets, pub/sub, etc. The project surfaces a hard problem: what’s actually shareable across cloud APIs, and what’s irreducibly per-cloud? Here is what I learned.

The premise

A Go application wants to read from object storage. AWS calls it S3; GCP calls it Cloud Storage; Azure calls it Blob Storage. Conceptually it’s the same thing — a put / get / list interface over named blobs in named buckets.

The naive answer: write one Go interface that covers the common case, implement it three times. blob.Read("my-bucket/my-key") works regardless of where the bucket actually lives.

The detailed answer: every cloud has nuance that the unified interface either has to expose (compromising the abstraction) or hide (compromising the use case).

What works

For the common case, the abstraction holds. The Gocloud blob package exposes:

type Bucket interface {
    NewReader(ctx context.Context, key string, opts *ReaderOptions) (io.ReadCloser, error)
    NewWriter(ctx context.Context, key string, opts *WriterOptions) (io.WriteCloser, error)
    Delete(ctx context.Context, key string) error
    List(ctx context.Context, opts *ListOptions) ListIterator
}

Three implementations (S3, GCS, Azure Blob) satisfy the interface. For an application that just needs put/get, switching clouds is a URL change.

The patterns that work cleanly:

Storage (object stores). Put / get / list / delete. Strong semantic alignment across clouds.
Pub/sub. Topic publish + subscription consume. Some semantic gaps (delivery guarantees, ordering) but the common case works.
Secrets. Get / set / version. Aligned enough across clouds that a unified API is meaningful.

What doesn’t work

The patterns that resist unification:

Compute. Each cloud’s compute primitive has a different shape. A VM on EC2 isn’t a VM on GCE isn’t a VM on Azure VMs; the network, IAM, image, and lifecycle models all differ.
IAM. The mental models are different enough that a unified IAM API would hide more than it abstracted.
Managed databases. RDS / Cloud SQL / Azure SQL look similar but the management surface (snapshots, replicas, parameter groups) doesn’t generalise.

Gocloud wisely doesn’t try to unify these. The library’s scope is the categories where unification adds value; everything else stays per-cloud.

The leaky abstraction problem

Even in the categories where unification works, there are leaks.

Examples I hit:

S3 supports pre-signed URLs with rich options. GCS supports signed URLs with different options. Azure Blob supports SAS tokens with yet another shape. The unified SignedURL method has to pick one shape and hope; the per-cloud options leak out as ProviderOptions.
Object metadata has different size limits and naming rules per cloud. The library normalises to a lowest-common- denominator interface; advanced users who need cloud-specific metadata fall back to provider-specific code.
Eventual consistency semantics differ. GCS is strongly consistent for most operations; S3 has post-write consistency for new objects but eventual for overwrites; Azure Blob has its own model. The unified API hides this; the application developer needs to know.

The library handles these by accepting the leak and documenting it. Pretending the leaks don’t exist would be worse.

When unified cloud APIs make sense

Three use cases I’ve seen pay back:

Open-source libraries that integrate with cloud storage. A library like Hugo (static site generator) wants to upload to “the cloud” without forcing users to a specific provider. Gocloud is great for this.
Multi-tenant SaaS where customers BYO storage. Customer A wants you to write to their S3; customer B wants GCS. The unified API lets you support both without forking the ingestion pipeline.
Migration in progress. An application moving from AWS to GCP wants to support both during the transition. The unified API lets you swap one bucket at a time.

When it doesn’t make sense:

Application is single-cloud and likely to stay that way. The abstraction is cost, not benefit. Use the native SDK.
You need the cloud-specific advanced features. The library hides them; you’ll fight the abstraction.
The unified API doesn’t cover your category. Don’t force a square peg.

What I learned about API design from contributing

Three takeaways that transfer beyond Gocloud:

An interface that covers 80% of use cases is more valuable than one that tries to cover 100%. The 20% that doesn’t fit should fall back to provider-specific code cleanly. Trying to cover 100% leads to either bloated interfaces or compromise-everywhere abstractions.
Document the leaks explicitly. A reader who knows what the abstraction doesn’t cover can plan around it. A reader who thinks the abstraction is total will hit edges painfully.
The hard part isn’t the unified interface; it’s the per-provider implementation. Each provider’s SDK has its own conventions, retry behaviours, error shapes. Normalising them into the unified interface is most of the actual work.

The library is a useful tool when the use case fits and a wrong choice when it doesn’t. Like most abstractions, the discipline is knowing which one you have.

What’s changed since

Gocloud has stayed alive and well-maintained. The categories the library covers have grown slightly; the team has resisted the pressure to grow without strong use cases. The result is a small, focused library that does what it says.

For a Go developer working multi-cloud: Gocloud is worth checking before reaching for three separate SDKs. For a Go developer working single-cloud: use the native SDK and don’t introduce the abstraction tax.

The contribution itself was a good way to study API design under the constraint of unifying genuinely different systems. The patterns I learned about what to expose and what to hide have shown up in every API I’ve designed since.