April 18, 2016
Speeding up npm in virtual environments
Ryan Buckheit
Updated on November 21, 2018
At Plaid, we spend a lot of time thinking about the tools we use on the engineering side, which has helped us to automate time-consuming tasks and focus on empowering our customers. We wanted to peel back the curtain a bit to share some of the tools—and decisions—that help Plaid run.
Recently, the Plaid team introduced a developer environment built on Vagrant and Chef. The transition made development easier but created an unexpected problem: npm install nearly ground to a halt in VirtualBox. Installs were 10 times slower than running on standard hardware. We ultimately chose to build a tool called npmserve and offload our installs to a remote server. It was an interesting experience getting there and exploring some of the open-source solutions to npm pain points, which we thought we’d share.
Our initial investigation revealed some quick wins to make our development environment faster. We found that Vagrant’s file synchronization was slow, so we started storing our node_modules outside of the shared directory. Configuring Vagrant with a bit more memory and a single cpu core helped as well. As we dove deeper, we found more and more Vagrant and npm dials to turn. We even got a good laugh out of a GitHub ticket about disabling the npm progress spinner to increase performance. Many of the tweaks yielded improvements, but installs in particular were still crawling:
We started considering more drastic approaches to address the issue. We investigated an alternative package manager called ied, but found that it hadn’t quite stabilized to the point where we were comfortable with replacing npm outright. Next we got excited about freight, which allows clients to offload their npm install to a remote server. Freight’s model was to rebuild native extensions on the client after downloading in order to match the extensions to the client architecture. Unfortunately, this rebuild carries a substantial performance penalty and also caused a compile error with one of our dependencies.
While Freight didn’t quite stick, we were inspired by the idea of offloading installs to a server. To that end, we built npmserve, a simple server that exposes npm as a service and allows clients to build and delete node_modules remotely. At Plaid, we installed it on a server running the same npm and Ubuntu version as our Vagrant boxes. Our clients now upload their package.json to the server and in return receive an archive containing their modules. This approach has a huge advantage from a caching perspective: The first client to request a build for a particular dependency configuration triggers a full build, but any subsequent client simply downloads the pre-built archive. Building our dependencies is now much faster:
Finally, we topped the tool off with a simple web UI so developers can administer the builds stored on the server:
If you’d like to check out npmserve, take a look at the GitHub links below. The implementation is a bit rough, but we’re excited to make improvements. In the future we’d like to improve build metadata persistence and add authentication to the server. We’d also like to add support for multiple architectures.
What problems have you encountered with npm or virtual environments? We’d love to hear about your experience and the tools you’ve built to make development more efficient!