Since the introduction of modules in Go version 1.11, developers have been expeditiously migrating away from the woes of the GOPATH-based workspace. Many of these early-adopters discovered that getting this beta-grade feature to work inside red-taped corporate bureaucracies was nearly impossible. This has resulted in hacks of convoluted git submodule dependency graphs, git repositories filled with templates intended only to be copy/pasted, and monorepos that flatten microservices into distributed monoliths. Worse yet, once these disorderly hacks were established they became institutions in and of themselves, laying in wait to trip up unsuspecting system operators!

Fortunately for us, module support is finally production-grade as of version 1.14! We can now achieve our goal of creating Go packages, organized using module support, that can be used as common libraries throughout the restrictive environment of Initech. Any projects importing these libraries are now able to manage them just like any other dependency using commands like go get -u, go mod download, go mod tidy, and go mod verify.

The Setup

At Initech many of our next-generation microservices will need to directly interact with that in-house legacy catalog server Bill Lumbergh’s monetization strategy won’t allow us to replace.

yeah

Let’s create a Go module, using go mod init, to interact with the legacy service and throw it on our company’s internal git server at git.initech.dev/projects/legacy-catalog. Now, when tasked with writing a new storefront microservice we can simply import our legacy-catalog module without having to reinvent the wheel!

import (
        ...
        "strings"

        catalog "git.initech.dev/projects/legacy-catalog"
        ...
)

And, because we’re hip enough to write microservices using Go we’re also going to containerize our application using the following Dockerfile:

FROM golang:latest as GOLANG-BUILDER

COPY /src /src

RUN go build -o /app/storefront /src/main.go

FROM alpine

COPY --from=GOLANG-BUILDER /app/storefront /app/storefront

CMD ["/app/storefront", "--port 80"]

Problem #1: Local Version Control Doesn’t Use HTTPS

When our Docker build compiles our application all packages imported by the codebase must be downloaded. In the background the go compiler uses go mod download to fetch our legacy-catalog module, but we get this error:

go: git.initech.dev/projects/legacy-catalog@v0.0.0-20200402015453-ed8fdcc94fed: unrecognized import path "git.initech.dev/projects/legacy-catalog": https fetch: Get "https://git.initech.dev/projects/legacy-catalog?go-get=1": dial tcp 172.30.1.109:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Turns out Go’s tooling does the right thing and downloads everything over HTTPS by default. As much as we’d like to enable HTTPS, here at Initech, central IT doesn’t “waste” time maintaining certificates on some server that only developers use. They have more pressing issues. Besides, Milton has the bosses completely convinced that encrypted communication isn’t needed behind the safety of our enterprise-grade firewall.

Fortunately with Go 1.14 a new environmental variable was added just for this situation. Using GOINSECURE we can instruct the Go tools to not only allow downloads over HTTP connections but also to skip certificate validation completely (for that HTTPS box with the certificate that expired many months ago). Simply populate this environment variable with a comma separated list of glob-patterns matching the insecure sources needed:

export GOINSECURE="git.initech.dev,*.initech.com"`

(on Windows use $Env:GOINSECURE="git.initech.dev,*.initech.com")

Problem #2: Local Version Control is Private

Now that the secure download problem has been solved let’s try to build again!

go: git.initech.dev/projects/legacy-catalog@v0.0.0-20200402015453-ed8fdcc94fed/go.mod: verifying module: git.initech.dev/projects/legacy-catalog@v0.0.0-20200402015453-ed8fdcc94fed/go.mod: reading https://sum.golang.org/lookup/git.initech.dev/!p!r!o!j!e!c!t!s/legacy-catalog v0.0.0-20200402015453-ed8fdcc94fed: 410 Gone
	server response: not found: git.initech.dev/projects/legacy-catalog v0.0.0-20200402015453-ed8fdcc94fed: unrecognized import path git.initech.dev/projects/legacy-catalog": https fetch: Get "https://git.initech.dev/projects/legacy-catalog?go-get=1": dial tcp 10.11.3.141:443: connect: connection refused

Hmmmmm. Looks like Go has another safeguard in place that clashes with Initech’s security! To prevent Go from having its own left-pad controversy, module downloads are permanently mirrored at proxy.golang.org. Should a public package be deleted the mirrored contents will still be available, and since all downloads go through this proxy by default no import statements have to be changed. Furthermore, this public proxy validates all content against a public checksum database; should this step fail the downloaded will be rejected as the contents have been discretely (aka maliciously) changed.

When Go attempts to download our legacy-catalog it’s trying to download it through this public proxy, and because the proxy cannot access our private repo the download is failing. To work around we can disable the public proxy using by setting GONOPROXY=none and can skip the public checksum validation by setting GOPRIVATE="*.initech.com". Better yet, we can kill two birds with one stone:

go env -w GOPRIVATE="*.initech.dev"

A nicer setup would have been to setup our own Go module datastore and proxy such as Athens - but who has time for that? Besides, now we only have one last Initech twist to work around…

Problem #3: Swapping Module URLs

Central IT has thoroughly firewalled the production environment; developers spend their time in a lower environment know as “dev” whereas system operators spend their time in “production” (the highest of environments). To further enhance security through obscurity, a wall of confusion has been established. Operators must compile the code themselves in the production environment if they are to deploy anything.

The Initech operators copy all git repositories from “dev” to “production”, from git.initech.dev to git.initech.com, and kick off the build. As developers we are now open to the ridicule of the operators as the build fails:

go: git.initech.dev/projects/legacy-catalog@v0.0.0-20200402015453-ed8fdcc94fed: unrecognized import path "git.initech.dev/projects/legacy-catalog": (https fetch: Get "https://git.initech.dev/projects/legacy-catalog?go-get=1": dial tcp 172.30.1.109:443: i/o timeout)

Our code imports git.initech.dev/projects/legacy-catalog. But in the production world it needs to be imported from git.initech.com/projects/legacy-catalog. Using Go 1.14 and the command line we can quickly modify our import paths with one command:

go mod edit --replace=git.initech.dev/projects/legacy-catalog=git.initech.com/projects/legacy-catalog@latest

This command adds the following line to the project’s go.mod file that instructs the Go tooling to use the production-environment url whenever it sees the development-environment url.

replace git.initech.dev/projects/legacy-catalog => git.initech.dev/projects/legacy-catalog v0.0.0-20200402015453-ed8fdcc94fed

Putting it all together

Let’s put all these workarounds into a new Dockerfile, including a build argument that lets us trigger go mod edit --replace as needed:

FROM golang:latest as GOLANG-BUILDER

ARG ALTERNATE_VCS=""

ENV GOINSECURE="git.initech.dev,*.prod.initech.com"

COPY /src /src

RUN go env -w GOPRIVATE=*.initech.dev,*.initech.com

RUN if [[ ! -z "$ALTERNATE_VCS" ]] ; then \
        go mod edit --replace=git.initech.dev/="$ALTERNATE_VCS" \
    fi;

RUN go build -o /app/storefront /src/main.go

FROM alpine

COPY --from=GOLANG-BUILDER /app/storefront /app/storefront

CMD ["/app/storefront", "--port 80"]

Now both environments can build our code! Developers continue to build normally and system operators can use it with just a simple tweak (docker build --build-arg ALTERNATE_VCS="git.initech.com/").

Final Thoughts

As developers it’s easy (and possibly even therapeutic) to get snarky towards the choices central IT makes. But the reality is we will never know the constraints, pressures, and pure chaos that go into these decisions. Sometimes development hacks, even nasty ones, will be needed. Let’s not forget, much of what was covered here are workarounds in and of themselves. If Central I.T. gets HTTPS going on the git server remove GOINSECURE from all your builds. The only way to prevent hacks from snowballing is to continuously circle back, reevaluate, and make corrections whenever viable - lest we become the target of scorn from the operators!

Additional Reading