If all Gophers abandoned the GOPATH build model and embraced the Go module build model, if all legacy Go package authors added go.mod to their legacy packages, and if all Go module authors adhered strictly to the semantic versioning (semver) specification, then Go would solve the “dependency hell” problem once and for all.
But the reality is not so rosy! The “dependency hell problem” in Go still exists . In this article, we’ll talk about the “scenarios” of dependency hell in Go, its causes and solutions.
1. What is “dependency hell”?
The “dependency hell” problem is not unique to one programming language, but is a widespread problem in software development and distribution. A broad interpretation of the problem from Wikipedia is as follows.
Dependency problems arise when several packages have dependencies on the same shared package or library, but they depend on different, incompatible versions of the shared package. If only one version of the shared package or library can be installed, the user may need to resolve the problem by obtaining a newer or older version of the dependent package. In turn, this may break other dependencies.
In the software development build world, we face the same “dependency hell” problem. Since words are always hard to understand, let’s use a more intuitive diagram to illustrate what “dependency hell” is in the software build process.
We see in this diagram that package P1 depends on package P3 V1.5, package P2 depends on package P3 V2.0, and the app depends on both package P1 and package P2, so the question arises: which version of package P3 should be used by the build tool when building the app: V1.5 or V2.0? The premise of this problem is that Apps are only allowed to contain one version of package P3 .
If the V1.5 and V2.0 versions of P3 are incompatible, then the build tool will fail to build the app regardless of which version of package P3 is selected. The developer can only intervene to solve the problem manually by either upgrading P1’s dependency on P3 to V2.0, or downgrading P2’s dependency on P3 to V1.5. However, P1 and P2 are in most cases third-party open source packages, and the developers of the app have limited influence on the authors of the P1 and P2 packages, so the success rate of this manual solution is not high either.
The “dependency hell” (also known as the “diamond dependency” problem) has a long history. Various programming languages have been trying to solve this problem for decades, and there are several good solutions that can help developers. Let’s take a look at the Go language solution.
2. Go’s solution
In the days of GOPATH build mode, Go build tools could not automatically solve the above “dependency hell” problem, but after the introduction of Go module build mode in Go 1.11, Go module build mode has matured over the years and become the standard Go build mode.
The Go module build pattern is a partial solution to this problem: sematic import versioning, that is, adding the package major version prefix to the package import path. version prefix .
With “sematic import versioning”, the Go solution to the above problem is as follows.
We see that Go enables P1 and P2 to each use their own dependent version by breaking the premise that “only one version of package P3 is allowed in the app”. However, this assumes that P1 and P2 depend on different major version numbers of P3. In Go, due to the semantic import versioning mechanism, mods with different major version numbers are treated as different mods, even if they originate from the same repository (e.g. v1.5 and v2.0 from the same P3 as above are treated as two different mods).
Of course, this solution comes with a cost! The first cost is that the size of the built app binary becomes larger, because the binary contains multiple versions of P3’s code; the second cost, which may not be a cost, is more to do with the fact that the types, variables, and identifiers between different versions of the module cannot be mixed, as we see in go-redis/redis, an open source project, as an example. go-redis/redis has the latest major version v8.11.4, and the version without go.mod enabled is v6.x.x. We mix these two versions together as follows
|
|
The Go compiler will report the following error when compiling this code.
|
|
That is, Client and redis Client under redis v8 are not of the same type and cannot be mixed together even if they have the same internal fields.
So, does it mean that with the Go module build mechanism, the “dependency hell” problem is completely removed from Go development? No, it doesn’t. The “dependency hell” problem still exists, so let’s take a look at the cases where it still occurs.
3. In which cases “dependency hell” still exists
1) Dependency on legacy Go packages without go.mod
It has been many years since the Go module was introduced into the Go language, but there are still a lot of legacy Go packages in the Go community that have not yet had the go.mod file added. For such go packages, the Go command’s handling strategy is roughly as follows.
- For go packages that are not yet tagged, then they are treated as equivalent to v0/v1
When the go command caches a go package in the local mod cache, it will synthesize a go.mod file, e.g.
|
|
- For go packages that have been tagged and the tag’s major version number is <2, then they are also treated in the same way as v0/v1
When the go command caches such a go package in the local mod cache, it will also compose a go.mod file, e.g.
- For tagged and tag’s major version number >= 2, the Go command, after downloading the package to the mod cache, will also compose a go.mod file for the go package, which is named vX.Y.Z+incompatible.mod, such as the following example.
In the above three cases, the module root path in the synthesized .mod file is without the vN suffix, regardless of whether the tag is played or not, and regardless of whether the tag major version >= 2. Take v2.6.1+incompatible.mod as an example, its content is as follows.
We see that the composite mod file also does not contain the require code block for the third-party package that the legacy package itself depends on. So how does a project that relies on the lz4 legacy package determine the version of the third-party dependencies of lz4? And where are the versions of the third-party packages that lz4 depends on recorded? Let’s take the app dependency github.com/pierrec/lz4 as an example and look at the following diagram.
When doing dependency analysis, the go mod command will determine the version of lz4 according to import github.com/pierrec/lz4 in the source code. Since there is no vN suffix, the go command will look for go.mod in the source code below v2 of lz4, but lz4 has not added go.mod until then, so it can only determine the version of lz4 according to the legacy So we have to follow the legacy model to determine the version of lz4. Here we determine v2.6.1+incompatible, which the go command uses as a direct dependency of the app module.
|
|
The go command then also analyzes the lz4 dependencies and records them in the app module’s go.mod as indirect dependencies.
|
|
Documenting third-party dependencies of directly dependent legacy packages in your own go.mod is to satisfy the requirements of a go.mod-based reproducible build .
There! Now that we understand how the go command handles legacy go packages, let’s see how the go command handles the “diamond dependency” situation. The conclusion is given directly in the following figure.
In this diagram, we have P1 relying on v1.0.1 of lz4 and P2 relying on v2.6.1+incompatible of lz4, both of which are tagged under legacy (no go.mod added). how does the go command choose the lz4 version when the app relies on both P1 and P2? The Go command simply selects the minimum version that satisfies both P1 and P2: v2.6.1+incompatible. here Go seems to make the assumption that legacy packages must be forward compatible with newer versions of older versions . For the lz4 package, this assumption is correct, and we will not encounter problems with the build and execution of the app.
However if this assumption is not true, e.g. lz4 v2.6.1 is a release that is not compatible with v1.0.1, then the build of the app will encounter an error . There is nothing more the go command can do in this case but to intervene manually! So how to intervene? There are only the following means.
- Submit an issue to urge the P1 author to upgrade the lz4 dependency to the latest v2.6.1
This is not only inefficient, but it is likely that the P1 author will not even bother with you.
- fork a P1, modify it yourself, and then make the app depend on your forked P1
This method works, but then you have to maintain a forked P1 by yourself, which invariably adds an extra burden to yourself.
- vendor down, modify it yourself, and maintain it in the vendor directory
This means is also feasible, but the subsequent only use vendor mode build, and to maintain a local P1, also adds an extra burden to yourself.
Isn’t there a better way? No! The transition from a legacy project to one that embraces go modules is bound to be a bumpy one.
2) Conflicts in packages that depend on the go module mechanism
After looking at the legacy package, let’s look at the conflict problem of packages that depend on the go module mechanism. With the understanding of the above example as a basis, it is easier to understand the following example, let’s look at the following diagram.
This example is also very simple. Both P1 and P2 depend on module P3, which depends on v1.1.0 and v1.2.0 of P3, respectively, and as in the previous example, the app depends on both P1 and P2, thus forming the “diamond structure” on the right side of the diagram. However, another mechanism of Go module: Minimum Version Selection (MVS) can solve this dependency problem very well.
The MVS mechanism is also based on semantic versioning, which selects the minimum available version that satisfies the dependency of the app. Therefore, according to the semantic version specification, they are compatible versions, so the go command selects the least available version that satisfies the app dependency as v1.2.0 (at this point the highest version of P3 is v1.7.0, and the go command does not select the highest version).
However there is often a disconnect between theoretical and practical conventions, and if the author of P3 does not recognize an incompatible change to v1.1.0 when releasing v1.2.0, then in this case an incompatible v1.1.0 version of P3 will cause the build of the P1 package to fail! This is the most common “dependency hell” problem when using the go module mechanism for dependency packages.
Is it hard to identify incompatible changes? Not really, but it is tedious and mind-numbing. So as a module author, how do you try to avoid releasing incompatible module versions without following the semantic versioning specification?
The Go community’s approach falls into two categories.
- Extreme practices.
- not releasing version 1.0, always at 0.x.y, no commitment that the new version is compatible with the old one.
- Upgrade the major version number every time a slightly larger version is released, which avoids the tedious problem of checking for incompatible changes, and certainly does not create a “dependency hell” for the community.
- General practice.
- Check for incompatible changes and only upgrade the major version if there is an incompatible change.
The official Go blog has published an article called “Keeping Your Modules Compatible” to show you how to check for incompatible changes in new changes. team has developed a tool called gorelease to help Go developers detect the existence of incompatible changes between new versions and old versions (of course, the tool is also likely to not fully scan out incompatible (of course, the tool may not be able to fully scan for incompatible changes), you can check the detailed usage of gorelease at here.
3. future thinking
The Rust language, which has been growing rapidly in recent years, seems to be ahead of Go in facing the problem of “dependency hell”, as explained in How Rust Solved Dependency Hell explains Rust’s solution to this problem in general. The principle is roughly based on the use of name modifiers, where two incompatible versions of the same dependency can coexist in a Rust application, each with an identifier that is unique in the application, and Rust uses package names, version numbers, etc. as name modifiers to achieve this.
So, will Go modules be able to do this in the future? I think it is possible and compatible with the current mechanism of go module, the introduction of Go module is also a “namespace” concept, with the basis of the problem like Rust. But whether the Go team wants to do this is a different story, because once the Go language world as mentioned in the beginning of this article, then the existing mechanism can also be a good solution to the “dependency hell” problem.