Working on an ever-evolving ecosystem of Go micro-services being modified by hundreds of developers every day can be a daunting experience. No matter how much event-driven architecture simplifies the dependency graph, inevitably, even the most atomic teams will spend a great deal of time communicating. Over the last few years of using Go for large projects, I’ve converged on a few conventions around organizing structs that have made collaborating across timezones and sharing repos across team barriers significantly easier.

The golden rule of defining Go packages is the single responsibility principle (SRP): each package should only be responsible for a single part of the program’s functionality. The package’s surface of exported versus private members functions like a blast door, designed to prevent internal code changes from spilling out to the rest of the program. But even well-bounded packages don’t exist in isolation. They work together, communicating using data linked to shared structs. A change to one of these structs will require modifying multiple packages.

Early in a project, these changes are often harmless; after all, adding a SubTitle field to the product.Product struct would be useless without the calling package supplying the value. But as the project grows in complexity, these types of modifications will result in cascades of unforeseen consequences. The scope of change spirals out of control and problem with implicitly mapped data sources suffer incompatiblies.

To avoid these repercussions, I like to apply the lessons of SRP and organize my structs into three categories:

  • Data Transfer Objects (DTO) are for public communication.
  • Internal Transfer Objects (IDO) are for internal communication.
  • Models are for communicating with datastores.

In the project repo, the packages usually end up organized as follows:

├── apps/
|   ├── some-app/
|   |   └── models.go
|   └── another-app/
|       └── models.go
├── internal/
|   ├── idos/
|   └── internal-widget/
└── pkg/
    ├── dtos/
    └── shared-gizmo/

DTOs are for Public Communication

Data Transfer Objects are used to communicate with outside integrations. Full backward-compatibility must always be maintained, as we don’t have control over the external integrations. Optional fields may be added, but existing fields cannot be renamed, removed, or have their data types changed.

By creating the package as pkg/dtos in the git repo, other projects can import them directly. Should the project communicate with different “kinds” of external integrations, create distinct DTO packages to prevent a needed change – a DTO used for web traffic shouldn’t require modifying a DTO used to write messages to Kafka.

└── pkg/
    └── dtos/
        ├── avro/
        └── web/

If the DTOs are shared implicitly via encoding mechanisms such as JSON, unit-tests can be written to fail if a change breaks backward-compatibility.

func TestMyDTO_UnMarshalJSON(t *testing.T) {
    t.Parallel()
    // Hand-encoded JSON representing an external integration.
    jsonPayload := `{"id":"some-id", "token":"some-token"}`
    var sut MyDTO
    if err := json.Unmarshal([]byte(jsonPayload), &MyDTO); err != nil {
        t.Fatalf("couldn't unmarshal json payload: %q.", err.Error())
    }
    if sut.ID != "some-id" {
        t.Errorf("Expected an ID of 'some-id' but got %q.", sut.ID)
    }
    // …

IDOs are for Internal Communication

Internal Data Objects are for communicating between the non-exported packages of a project’s repo. Any packages nested in the pkg directory cannot import IDOs - if this functionality is needed, a DTO should be used instead. Backwards-compatibility can be broken as long as all affected deployments are updated together (such as in the case of a Monorepo).

By creating the package as internal/idos projects outside of the repo cannot import it.

└── internal/
    └── idos/

Depending upon the size and complexity of the project, multiple IDO packages could be beneficial.

└── internal/
    ├── a/
    |   └──b/
    |      ├──c/
    |      └──more-idos/
    └── some-idos/

Models for Database storage

Models are for communicating with datastores, most commonly databases. Models are exported (by capitalizing the first letter of their name) to work with datastore packages but must not be shared between project packages.

By creating a models.go file in the relevant package, anyone looking at a database issue has a better chance of locating the offended model in the repo.

└── apps/
    ├── some-app/
    |   └── models.go
    └── another-app/
        └── models.go

If a package has lots of models, it may be useful to split them across files while keeping them identifiable as models.

└── apps/
    └── some-app/
        ├── mdl-tasks.go
        └── mdl-results.go

Interoperability

Decoupling our communication objects into separate packages will require writing functions to convert between types. I’ve found the following conventions helpful for managing this overhead. Functions that convert to DTOs or IDOs can be done via receiver functions, as the target package is accessible.

func (m MyModel) ToDTO() dtos.MyDTO

Conversely, functions that convert to models should not be receiver functions, as the target struct shouldn’t be accessible alongside the source struct.

func MyModelFromIDO(src idos.MyIDO) MyModel

It’s helpful to make distinctions between inbound and outbound data objects. An incoming request DTO may become a request IDO, which is then validated and processed, resulting in a database record. On the outbound side, the processing could result directly in a response DTO. Carefully defining this flow will keep the project cleaner and reduce the number of conversions that need to happen.

struct conversion flow

Other Thoughts

  • All conversion functions should be unit tested to hedge against data loss/mutation.
  • DTOs can be used to communicate internally and externally, as long as backward-compatibility is maintained.
  • If a struct contains annotations of multiple kinds, such as bson mixed with json, its intents have not been safely decoupled.
  • Most projects will not benefit from IDOs; IDO packages should be created when they would be beneficial.

See Also