Since the introduction of modules in Go version 1.11, developers have been expeditiously migrating away from the woes of the
GOPATH-based workspace. Many of these early-adopters discovered that getting this beta-grade feature to work inside red-taped corporate bureaucracies was nearly impossible. This has resulted in hacks of convoluted git submodule dependency graphs, git repositories filled with templates intended only to be copy/pasted, and monorepos that flatten microservices into distributed monoliths. Worse yet, once these disorderly hacks were established they became institutions in and of themselves, laying in wait to trip up unsuspecting system operators!
Fortunately for us, module support is finally production-grade as of version 1.14! We can now achieve our goal of creating Go packages, organized using module support, that can be used as common libraries throughout the restrictive environment of Initech. Any projects importing these libraries are now able to manage them just like any other dependency using commands like
go get -u,
go mod download,
go mod tidy, and
go mod verify.
At Initech many of our next-generation microservices will need to directly interact with that in-house legacy catalog server Bill Lumbergh’s monetization strategy won’t allow us to replace.
Let’s create a Go module, using
go mod init, to interact with the legacy service and throw it on our company’s internal git server at
git.initech.dev/projects/legacy-catalog. Now, when tasked with writing a new storefront microservice we can simply import our
legacy-catalog module without having to reinvent the wheel!
import ( ... "strings" catalog "git.initech.dev/projects/legacy-catalog" ... )
And, because we’re hip enough to write microservices using Go we’re also going to containerize our application using the following Dockerfile:
FROM golang:latest as GOLANG-BUILDER COPY /src /src RUN go build -o /app/storefront /src/main.go FROM alpine COPY --from=GOLANG-BUILDER /app/storefront /app/storefront CMD ["/app/storefront", "--port 80"]
Problem #1: Local Version Control Doesn’t Use HTTPS
When our Docker build compiles our application all packages imported by the codebase must be downloaded. In the background the go compiler uses
go mod download to fetch our
legacy-catalog module, but we get this error:
go: email@example.com: unrecognized import path "git.initech.dev/projects/legacy-catalog": https fetch: Get "https://git.initech.dev/projects/legacy-catalog?go-get=1": dial tcp 172.30.1.109:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Turns out Go’s tooling does the right thing and downloads everything over HTTPS by default. As much as we’d like to enable HTTPS, here at Initech, central IT doesn’t “waste” time maintaining certificates on some server that only developers use. They have more pressing issues. Besides, Milton has the bosses completely convinced that encrypted communication isn’t needed behind the safety of our enterprise-grade firewall.
Fortunately with Go 1.14 a new environmental variable was added just for this situation. Using
GOINSECURE we can instruct the Go tools to not only allow downloads over HTTP connections but also to skip certificate validation completely (for that HTTPS box with the certificate that expired many months ago). Simply populate this environment variable with a comma separated list of glob-patterns matching the insecure sources needed:
(on Windows use
Problem #2: Local Version Control is Private
Now that the secure download problem has been solved let’s try to build again!
go: firstname.lastname@example.org/go.mod: verifying module: email@example.com/go.mod: reading https://sum.golang.org/lookup/git.initech.dev/!p!r!o!j!e!c!t!s/legacy-catalog v0.0.0-20200402015453-ed8fdcc94fed: 410 Gone server response: not found: git.initech.dev/projects/legacy-catalog v0.0.0-20200402015453-ed8fdcc94fed: unrecognized import path git.initech.dev/projects/legacy-catalog": https fetch: Get "https://git.initech.dev/projects/legacy-catalog?go-get=1": dial tcp 10.11.3.141:443: connect: connection refused
Hmmmmm. Looks like Go has another safeguard in place that clashes with Initech’s security! To prevent Go from having its own left-pad controversy, module downloads are permanently mirrored at proxy.golang.org. Should a public package be deleted the mirrored contents will still be available, and since all downloads go through this proxy by default no import statements have to be changed. Furthermore, this public proxy validates all content against a public checksum database; should this step fail the downloaded will be rejected as the contents have been discretely (aka maliciously) changed.
When Go attempts to download our
legacy-catalog it’s trying to download it through this public proxy, and because the proxy cannot access our private repo the download is failing. To work around we can disable the public proxy using by setting
GONOPROXY=none and can skip the public checksum validation by setting
GOPRIVATE="*.initech.com". Better yet, we can kill two birds with one stone:
go env -w GOPRIVATE="*.initech.dev"
A nicer setup would have been to setup our own Go module datastore and proxy such as Athens - but who has time for that? Besides, now we only have one last Initech twist to work around…
Problem #3: Swapping Module URLs
Central IT has thoroughly firewalled the production environment; developers spend their time in a lower environment know as “dev” whereas system operators spend their time in “production” (the highest of environments). To further enhance security through obscurity, a wall of confusion has been established. Operators must compile the code themselves in the production environment if they are to deploy anything.
The Initech operators copy all git repositories from “dev” to “production”, from
git.initech.com, and kick off the build. As developers we are now open to the ridicule of the operators as the build fails:
go: firstname.lastname@example.org: unrecognized import path "git.initech.dev/projects/legacy-catalog": (https fetch: Get "https://git.initech.dev/projects/legacy-catalog?go-get=1": dial tcp 172.30.1.109:443: i/o timeout)
Our code imports
git.initech.dev/projects/legacy-catalog. But in the production world it needs to be imported from
git.initech.com/projects/legacy-catalog. Using Go 1.14 and the command line we can quickly modify our import paths with one command:
`go mod edit --replace=git.initech.dev/projects/legacy-catalog=git.initech.com/projects/legacy-catalog@latest`
This command adds the following line to the project’s
go.mod file that instructs the Go tooling to use the production-environment url whenever it sees the development-environment url.
replace git.initech.dev/projects/legacy-catalog => git.initech.dev/projects/legacy-catalog v0.0.0-20200402015453-ed8fdcc94fed
Putting it all together
Let’s put all these workarounds into a new Dockerfile, including a build argument that lets us trigger
go mod edit --replace as needed:
FROM golang:latest as GOLANG-BUILDER ARG ALTERNATE_VCS="" ENV GOINSECURE="git.initech.dev,*.prod.initech.com" COPY /src /src RUN go env -w GOPRIVATE=*.initech.dev,*.initech.com RUN if [[ ! -z "$ALTERNATE_VCS" ]] ; then \ go mod edit --replace=git.initech.dev/="$ALTERNATE_VCS" \ fi; RUN go build -o /app/storefront /src/main.go FROM alpine COPY --from=GOLANG-BUILDER /app/storefront /app/storefront CMD ["/app/storefront", "--port 80"]
Now both environments can build our code! Developers continue to build normally and system operators can use it with just a simple tweak (
docker build --build-arg ALTERNATE_VCS="git.initech.com/").
As developers it’s easy (and possibly even therapeutic) to get snarky towards the choices central IT makes. But the reality is we will never know the constraints, pressures, and pure chaos that go into these decisions. Sometimes development hacks, even nasty ones, will be needed. Let’s not forget, much of what was covered here are workarounds in and of themselves. If Central I.T. gets HTTPS going on the git server remove
GOINSECURE from all your builds. The only way to prevent hacks from snowballing is to continuously circle back, reevaluate, and make corrections whenever viable - lest we become the target of scorn from the operators!
2020-04-07 21:47 +0000