Golang — vgo vs dep — Dependency Management Tools Explained

As a leading Golang development company in San Francisco and Minsk, we are on the frontlines of navigating the ever-evolving landscape of implementing dependency management tools in Golang. In this article, we’ll discuss at length what makes a dependency management tool effective in Golang web development as well as take a deep dive into the differences between vgo and dep.

Nowadays, even writing a simple application is tough to do from scratch. A lot of times, even writing a simple application is easier to build if you use different libraries and extensions written by third parties and can really alter the cost of creating an app. You may be scratching your head and asking yourself, “Why do we need to be concerned about managing the dependencies for all these libraries?” It’s pretty simple. These libraries evolve over their lifetime and if we update them to their latest version, there is a decent chance that our app’s functionality might break unexpectedly. One day, your app is working just fine and then all of a sudden it’s not. After banging your head against your desk you think again, “Why the hell is this happening to me?”

It’s highly likely that your app needs some kind of dependency management. This is why the creators of popular programming languages pay such close attention to this area. In many of the programming languages that have been around for a long time, their dependency management is pretty much clear and written in stone (like Maven/Gradle in Java).

On the other end of the spectrum, however, you have Golang. If you want to build web apps with Go language, you are going to face some dependency management hurdles. Golang is still a relatively young programming language and has some minor kinks to work out. Even to this day, the Go community is still searching for the silver bullet to solve this problem.

Golang’s popularity is growing among engineers and businesses looking to be rewarded by all of Golang’s benefits. To start, gopkg.in introduced version compatibility through the import path. Then, on June 19th, 2015, in version 1.5, the Go toolchain included an experimental vendoring flag. The official dependency management tool at the time was dep. The breakthrough came during the official Go proposal presented by Russ Cox on March 20, 2018: vgo. Russ’s proposal was accepted by the community on May 21, 2018.

Before we review vgo and what it can do, let’s review some of the requirements a good dependency management tool should include. To start, it should be able to enumerate, download and store dependencies in our code. This part is obvious. It would be great if the tool could manage “transitive” dependencies (dependencies of the other dependencies etc.) as well as provide the ability to solve or, at least, show us where there are conflicts in the dependencies. The tool should keep different versions of the same dependency so we can easily switch from one to another. It’s helpful if the dependencies are stored in some central repository so they can be used in a number of projects or that they can create a local mirror of a central repository where the dependencies are cached. The tool needs to be simple and easy to use.

So, what was wrong with dep? First of all, there wasn’t a central storage for dependencies and their versions. All the dependencies were kept in the vendor folder and only one version was available at a time. This caused a lot of annoying problems to pop up like excluding vendor from go vet, go lint and go test tools as well as issues that appeared in versioning control systems. Conflicts couldn’t be resolved and we couldn’t use different versions of the same project. According to the dep guidelines, dep prefers to have ranges over a specific version i.e. if you say version = “0.1.6” then dep can take any version less than “0.2.0” like “0.1.9”. If you want to use an exact version you should use version the = sign “=0.1.6”. This is not an intuitive behavior. Of course, there’s a way to point to the precise version but it looks a bit ugly.

vgo supports the same semantics as gopkg.in. Russ Cox named this approach the convention semantic import versioning. The rule is defined as: “If an old package and a new package have the same import path, the new package must be backwards compatible with the old package”

How does vgo keeps these dependencies? All the dependencies are zipped and stored in a temporary folder $GOPATH/src/v/cache. The name of the file contains a version or date as well as a hash of the commit, if the dependency was taken by the commit hashcode.

There are a number of files in the temporary folder for every dependency: *.zip contains the compressed source code, the *.info contains the meta information for the dependency and the *.mod file contains the transitive dependencies. The *.ziphash file contains the hash of the zip for checking the consistency of the dependency.

$GOPATH/src/v/ folder contains the versions unzipped, like the vendor folder, but by version:

vgo supports automatic transformation out-of-the-box for the most popular vendor experiment tools, chief among them is dep. Let’s take a look at how we can migrate from dep to vgo, looking at the following simple project managed by dep, as an example. First of all, let’s install vgo:

go get -u golang.org/x/vgo

Kicking off a project with vgo is quite simple and done the same way for a project without dependency management or managed by dep. Let’s move a simple project containing a few dependencies and dep over to vgo. Gopkg.toml file contains two dependencies:

Let’s build it with vgo:

vgo build

It finished with an error because there is no go file in the root of the project. Go.mod has not been created. We believe it is a temporary vgo issue. But now, we need to create a noop.go file in the root. Let’s do it and re-run the command again:

As you can see, vgo used the information, stored it in the Gopkg.lock file and created the descriptor go.mod in the project’s root.

The go.mod contains the list of all the managed dependencies:

vgo should use the exact version described in the go.mod file. It is the oldest version still acceptable by our application. On the other hand, dep takes the newest version in the range, which can be more a more recent update than the one from the toml file.

In go.mod, we can see two dependencies on github.com/gin-gonic/gin and github.com/leonelquinteros/gorand like in Gopkg.toml. You might be asking yourself, ”why are there now an additional 6 dependencies?” This is because vgo takes into account Gopkg.lock and not Gopkg.toml. vgo decides how many dependencies to take into consideration and the process is still unclear. But what will happen with the dependency on github.com/gin-gonic/gin? It was pointed to 1.2, but in go.mod we see that it points to the hash of the commit related to the tag 1.2. Why does it happen like that? If we go to the tags list — we can see the 1.2 version. Let’s use vgo to print the list of available versions:

vgo list -t github.com/go-kit/kit

We do not see the version 1.2 as well as 1.1, 1.0 or 0.6. It seems this is happening because of the semantic import versioning paradigm of vgo. It is ignoring any versions that is not in the vn.n.n form. Let’s hope this issue is temporary as well. Well, let’s build the executable for the simple Golang web development project. The syntax of the command is similar to the go build. The command will compile and build our project using the vgo dependencies repo. $ vgo build -o simple ./cmd Please note, that the zipped sources (*.zip and *.ziohash files) are downloaded and placed into the vgo cache folder when they are needed to build the binary with the command above. Another very important point to take note of is that we can no longer use the go build or the go install command for building our project with vgo dependencies management. This is because vgo can’t be used with the vendor experiment and the go build will not find our dependencies in the common places. Let’s check out the list of our dependencies and any available updates (i.e. the most up to date version), if they exist:

vgo list -m -u

If a package has an updated version, we can use that version with the following command:

vgo get github.com/stretchr/testify

If we want to choose a specific version of a package, we just point to the version like this:

vgo get github.com/stretchr/testify@v1.2.1

And now if we list the dependencies, we see that our Golang web development project is dependent on the following version:

vgo list -m -u

What if we want to apply all the available updates that are currently available? You can by executing the following command:

vgo get -u

vgo build -o cmd/myservice cmd/main.go

What if we want to use a dependency locally? For example, the dependency repository is also in our project and we want to use the latest development version. In dep, it was impossible, but it is quite easy with vgo. We need to put the following directive into our go.mod file:

replace github.com/stretchr/testify v1.2.2 => /Users/anovikau/go/src/github.com/stretchr/testify

Our updated go.mod:

Now vgo will use the local version. The location of the file can be outside the $GOPATH as well:

The important thing to understand is that vgo does not destroy our /vendor/ folder, GOPKG.tolm, and GOPKG.lock files so we can use them at the same time with dep, in order to try and test it. But in production, we can use the same dep approach we did before. It is possible to have another version for a given package or a tree of packages starting from a certain root inside the same project. Just create an empty go.mod file inside the folder and run:

vgo build

As a result, the go.mod file will be populated by the dependencies from the start, exactly for this package. And we can add and change the dependencies here like normal:

Please note, that by doing it in this way, it lets us have different dependency versions inside the same project. It will be compiled and work! The packages that had separately specified dependency version will use that version and the rest will use the version from the root go.mod. According to the vgo papers, we can export the dependencies managed by vgo into the vendor folder in order to work with the standard toolchain:

vgo vendor

Ooops, it does not work yet:

#The outcome is:

• vgo has it’s central repo so there won’t be duplicates of the same versions for every project like in go
• moving from a dep project to a vgo one is simple. We need to use vgo build for this. go build will not take into account all the vgo dependencies. It’s useful because we can still use dep or an old dependency manager at the same time (in production, for example)
• different versions are cached locally so there is no need to download them every time
• in an entire project (same binary), we can have dependencies on different versions, per module (a package or a package tree)
• we can use a local (temporary) version of a dependency manager for development needs

#What’s missing still?

• we still cannot configure the path to the vgo cache folder (repository) and because of this we can not switch from one dependency repository to another
• we cannot use local mirrors for the dependencies so every developer in the same group will download the same dependencies remotely
• there isn’t any clear control of transitive dependency tree or what the source of a given transitive dependency is
• there is no tool for marking conflicts and resolving them
• some things are still not working as expected or do not work at all

Our conclusion is that vgo is definitely a step forward. It gives more structure and you can work with a centralized dependency management tool. The vgo tool is still being developed and it may be better to wait for the next version Go 1.11 to use it in production. But you can start off by experimenting with it in parallel with dep.

Here is a short cheatsheet for vgo commands you need to know and a short description:

• vgo build - initiates the vgo project and builds executables
• vgo list -m -u - lists the dependencies and possible updates
• vgo list -t github.com/go-kit/kit - lists all the available versions for the package
• vgo get github.com/go-kit/kit - gets the latest dependency and changes the go.mod file accordingly
• vgo get github.com/go-kit/kit@v0.0.7 - gets the precise version of the dependency
• vgo get github.com/stretchr/objx@v0.0.0–20180106011353-facf9a85c22f - the same as above, gets the precise version of the dependency
• vgo get github.com/go-kit/kit@’<v1.6′ - less than specified version of the dependency
• vgo get -u - update all the dependencies
• vgo vendor - creates a vendor folder like dep — does not work yet
• vgo test - runs the tests of the module
• vgo test all - runs the tests of the module and all the dependencies — seems not working
• replace github.com/go-kit/kit v0.7.0 => “/Users/someone/go/src/github.com/go-kit/kit” - replaces a certain dependency from the location

#UPDATE

On June 14, 2018, a number of updates were released that added to the functionality of vgo. You can now do a number of things with the new vgo mod command. While it is still under development, we have noted in some cases that it can be unstable. What can we do with vgo mod? There are a number of flags that the command can accept:

• vgo mod -init - initializes and writes a new go.mod to the current directory, thus creating a new module rooted in the current directory
• vgo mod -sync - synchronizes go.mod with the source code in the module. It adds any missing modules necessary to build the current module’s packages and dependencies and then removes any unused modules that aren’t being used anymore
• vgo mod -verify - checks that the current modules dependencies, which are stored in a local downloaded source cache, have not been modified since being downloaded. If all the modules are unchanged, -verify prints “all modules verified.” Otherwise, it identifies which modules have been changed and causes ‘go mod’ to exit with a non-zero status
• vgo mod -vendor - this flag resets the module’s vendor directory to include any packages needed to build and test all the module’s packages and their dependencies
• vgo mod -json - prints the go.mod file in JSON format corresponding to these Go types:

type Module struct {
 Path string
 Version string
 }type GoMod struct {
 Module Module
 Require []Module
 Exclude []Module
 Replace []struct{ Old, New Module }
 }

Note that this only describes the go.mod file itself, not other modules referred to indirectly. For the full set of modules available to a build, use ‘vgo list -m -json all’

• vgo mod -fix - updates go.mod to use canonical version identifiers and to be semantically consistent
• vgo mod -packages - prints a list of packages in the module
• vgo mod -fmt - reformats the go.mod file without making other changes. This reformatting is also implied by any other changes that use or rewrite the go.mod file. The only time this flag is needed is if there are no other flags specified, as in ‘vgo mod -fmt’
• vgo mod -require=path@version and vgo mod -droprequire=path - add and drop a requirement on the given module path and version. Note that this overrides any existing requirements on path

More information about this can be found at this page.