June 09, 2020
How we migrated our website from Handlebars to React in 12 weeks
Updated on June 19, 2020
TL;DR: We built an automated tool that let us incrementally migrate, while keeping up regular feature work and maintenance.
As a company, Plaid has reached critical mass and is rapidly expanding our product offerings and website to serve new markets. Today, we connect over 11,000 financial institutions across the United States, Canada, and parts of Europe. We power thousands of applications that users rely on to manage their financial lives and our website has scaled to serve millions of visitors every year. As we continue to grow, so do the requirements for plaid.com.
Our website is the nexus of the Plaid user ecosystem. Customers like Venmo and Robinhood visit it to learn about new features and products, developers use it to reference API documentation in order to build new applications, and end users seek details about our data practices and policies. Making this information readily available to global audiences is imperative to the success of our international growth.
As we began our expansion into new markets, we noticed the number of incoming change requests increased with each additional locale. Growth experiments no longer pertained to North American markets and what historically worked in the United States was not working in the United Kingdom. Serving users at a global scale demanded faster iterations, more experimentation, and localized content. As internationalization marched on, our team would need to effectively handle the ever increasing complexity of maintaining a globalized static website.
Migration approaches considered
We’re constantly shipping updates to plaid.com in order to deliver the best experience for our users, so it wasn’t acceptable for us to pause work while undergoing the migration. We needed to approach this in a way that would allow the team to keep up momentum, while avoiding any unnecessary downtime. This is how we thought about it:
1 // Build a new website from scratch
Building a new website from scratch is often an extremely time consuming challenge for a small team. At the time, our web development team consisted of only three engineers, and we couldn’t allocate resources to starting over. Our hands were tied with regular maintenance and new feature requests, making it impractical to consider taking on such a massive initiative.
2 // Compile Handlebars templates to HTML then wrap the output in React
Using a flat file generator (e.g. Panini), we could compile all of the Handlebars templates to HTML and then wrap the output in a React component. This is a better solution than starting from scratch. It’s a cheap and easy way to get the ball rolling, but at what cost? Compiling to static HTML would defeat the purpose of using a templating language. We would lose all of the benefits of Handlebars, such as conditionally rendering components, and iterating over data. Our website would be frozen in its current state and we’d be left to manually abstract all of our data and components from each compiled page.
3 // Server-side render the website and incrementally migrate each page
Similar to Hulu’s solution, we also considered rendering the templates server-side with the Handlebars view engine for Express.js. This would allow us to unblock the international expansion by enabling dynamic content sooner. However, as we incrementally migrated pages, we would need to maintain several top-level components like primary navigation, supporting dynamic internationalization logic, in both Handlebars and React. This context switching would present new challenges in how duplicate components are maintained making our website susceptible to errors and inconsistencies across pages.
4 // Compile Handlebars templates directly to React
All of the options considered above came with tradeoffs that we couldn’t justify at the time, so we asked ourselves, “could we build a script to handle most of the migration for us and manually migrate only a small number of complex pages?”.
Building a compiler
The engineering teams at Plaid prefer to write technical specifications for larger projects that generally take more than a month to complete. We do this because it forces engineers to think through all of the details of the design, gather feedback early, and accurately scope the task at hand. Part of that process sometimes involves building a proof-of-concept in order to test theories early, and ensure the project will succeed.
While thinking through the different approaches to the migration, we discovered the handlebars-to-jsx library. It did a great job of compiling some of our Handlebars expressions, but could only assemble very simple components. When it came to more complex templates that invoked other components (i.e. partials), the library was unable to compile without throwing an error.
Since our website contained over 900 invocations of partials throughout 146 files, the library, in its current state, was only capable of compiling 35% percent of our website. This inspired us to build our own compiler to use in conjunction with the library.
Since, for our purposes, we were only dealing with simple markup and a few Handlebar helpers, there was no need to parse our code into an Abstract Syntax Tree before compiling. In fact, the compiler we built was actually quite simple. At a high level, it just read the contents of a file into an array and applied several reducer functions to transform each line of code. Below is an example of the first compiler function we wrote for our proof-of-concept:
This function coupled with the Handlebars to JSX library, would turn a Handlebars template like this:
Into a React component like this:
We had to write an additional 10 reducers to handle other helpers and edges cases not included in the handlebars-to-jsx library. After that, we were able to compile almost the entire website using custom directives to ignore erroneous blocks of code. With more than 400 files processed, this resulted in:
82% compiled with no errors
3% compiled with minor warnings
9% partially compiled
5% required manual migration
One of the biggest challenges of a website migration is the transition from old to new systems. By automating the brunt of the migration, we estimated the team would save a minimum of 120 hours of work while enabling a smooth transition from new to old. Manually rewriting hundreds of files from Handlebars to JSX was shaved off the migration process by our compiler, not to mention a lot of stress.
We wanted to keep the migration within the same repository as the legacy website so that we could share dependencies, make it easy to reason about, and run both websites concurrently while we migrated the remaining templates and tested our changes in production. However, Next.js required that we upgrade some of our legacy dependencies like Babel and Webpack. Upon upgrading, we discovered that Browsersync was no longer compatible with the latest version of Babel. We didn’t want to change our regular developer workflow, so we forked the unmaintained dependencies, fixed the errors, and published them on npm.
Without the Gulp task runner watching for incremental changes and recompiling them into the new website, contributors would be left making duplicate changes while both websites existed. One changeset for the legacy codebase, and the same again for the new site. They would also need to deal with the mental overhead of “what has been migrated, and what has not?” Over time, this list would inevitably begin to slip our minds and make it very difficult to keep abreast the migration.
While interviewing new candidates for the team, we often pitched that the migration to Next.js would enable us to write UI components using React. In fact, during our migration we were able to double the team to 6 members, thanks in part to this effort. Interview candidates often expressed a lot of interest in working with Next.js and our newest hires are happy they did not have to endure the legacy tech stack.
What we learned
While migrating the website, we learned a lot about our legacy code base:
The preexisting code was littered with invalid syntax which caused the compiler and React to choke. By spending more time testing our API documentation we uncovered code samples that were often missing important tokens because of Handlebars markdown compilation bugs. After testing the website for launch, we ended up documenting 49 low priority bugs unrelated to the migration, and we were able to resolve all these bugs before going live.
Our API docs were a mixbag of Markdown, markup, and Handlebars that required an additional custom compilation step on top of the Handlebars-to-JSX compiler and our hand rolled compilers. Compilers all the way down.
A thorough audit of dependencies led to deleting several HTML, CSS, and JS artifacts. Additionally, we were using dependencies that are no longer maintained and we plan to replace as we continue to iterate.
What we would do differently
Migrating plaid.com off of these legacy technologies was long overdue. In fact, it probably should have started before expanding into new markets so that we were well equipped for our speedy growth. Now that we’ve migrated, we will need to retroactively patch up some of our temporary workarounds from initiatives aimed at expanding into parts of Europe.
The Next.js migration is but a small fraction of some of the amazing work we’re doing on the Plaid WebDev team. If you’re interested in solving the challenges we’ll face as both our team and the company continue to grow, consider applying for one of our open roles.