What Does a Bundler Actually Do?

What is the job of a web application? A server component delivers a bunch of HTML, JavaScript, CSS, and other files (e.g., images and text) via HTTP to a browser. The browser puts the pieces together and renders a complete webpage. Regardless of which technology stack is used, this process should – above all – be as fast as possible, look good on all browsers, and of course offer a good user interface.

For the purposes of this article, an example application will be considered that uses the JavaScript framework React on the client side. This library offers a whole array of tools that create easily digestible pieces from the JavaScript source text for browsers. This example therefore offers us a good overview of the most commonly used technologies. But the concepts presented here can just as easily be used on classic, server-side rendered applications.

The Connection

The main problem plaguing all web applications (and even simple websites) is the network connection. It may be slow and unreliable or cut out at any time. Nowhere near all users are connected by high-speed fibre.

And even with a perfect connection, there are of course limits. For example, browsers have to decide whether to request the files required for a webpage sequentially or in parallel. Parallel connections have the advantage that downloads are faster, but they also have overhead with each connection.

A webpage can – possibly transitively – demand huge resources

Modern browsers use sophisticated heuristics or newer protocols such as HTTP/2 or HTTP/3 (aka “QUIC”) to squeeze out the last little bit of performance from the connection. As a developer, however, one rarely has any influence over this.

The much bigger lever is reducing the number of necessary requests from the outset. There are two techniques to achieve this, which can be combined with one another:

Aggressive caching
The consolidation of multiple resources

Frontend tools can do this for the developer, hence the term “bundler.” This is a building step comparable to static linking. For modern tools however this falls short, as alongside the combination of JavaScript files (see next section) they also process other types of resources, such as images and styles. As a result, the terms “frontend build tool” and “asset pipeline” are sometimes also used. But I’ll return to that later.

The Combination

So, how can the term “bundling” be understood? Some resources, distributed across multiple individual files, can be combined without causing problems. As an example: Multiple CSS files, bundled via <link> in HTML code, can be consolidated into a single file in which they are easily concatenated. Instead of multiple necessary GET requests to the server, now only one is needed.

This is also possible with JavaScript. Indeed not just possible but necessary, for libraries like React consist internally of countless individual files, exactly as one is used to for example from the typical backend programming languages like Java. Although modern browsers can also resolve imported references thanks to the JavaScript module standard (more on this later too), this leads to the problem of “cascading requests”:

frontend.js file is loaded via <script>.
frontend.js imports lib.js.
lib.js imports lib/dom.js.
lib/dom.js imports lib/util.js.

… and so on, each line generating an HTTP request. The browser can only start to execute the JavaScript code once all (transitive) dependencies have been loaded.

Bundling optimizes this by examining all imports during building. Instead of delivering multiple JavaScript files, these files are concatenated by preparing them as a combination of framework and application code.

Reduction of the required resources through bundling

A further pitfall in the bundling of JavaScript files is that these are blocked by the browser during their execution at exactly the point where the <script> tag is located. So, when these appear in the <head> of the HTML file, the browser pauses the rendering and instead downloads the rest of the code. Meanwhile, the browser window remains blank.

For completeness, it should be mentioned that one can delay the processing of the script tag to shortly before the end of the file using the defer attribute. Files loaded with type="module" are automatically loaded with a delay; the defer attribute is redundant here.

There are various options here, depending on the architecture. In the case of single-page applications, the JavaScript code is responsible for the rendering of content. One therefore has no choice but to wait until it – including the framework – arrives at the browser. If one chooses server-side rendering instead, one can for example move the <script> tag to the end of the HTML code (i.e., before </body>).

In addition, through the analysis of import paths, bundlers also take on a translation of the Node world into the browser world. By convention, packages installed by npm land in the node_modules folder. If one then imports React via import React from "react", under the hood the file node_modules/react/index.js is loaded (to put it in simple terms). This mechanism can be individually configured in each npm package in the package.json file. This is however alien to the browser. It will attempt to make a GET request to fetch the react resource, leading to a 404.

An alternative approach to this problem are so-called import maps, with which URL remapping for imported modules can be defined. They are however currently lacking browser support, and they don’t help in all situations faced during frontend development.

The Cache

A user often loads a webpage more than once. Especially in the case of server-side rendering, the web server delivers multiple HTML pages with a substantial overlapping of resources – the subpages usually all use the same style sheet.

Caching therefore must be considered to reduce the loading times of repeated requests. This could be the subject of an entire article of its own, so I can only give a brief introduction to the browser cache here.

Responding to a GET request, a web server can determine the desired caching behaviour. Most web servers do this automatically. When the browser requests a resource, the server replies not only with its content but also delivers specific metadata, such as modification date and hash:

Last-Modified: Wed, 21 Oct 2015 07:28:00 GMT
ETag: "33a64df551425fcc55e4d42a148795d9f25f89d4"

The content including metadata then lands in the browser cache. If the user requests the resource again, for example when following a link, the browser sends specific headers to the server:

If-None-Match: "33a64df551425fcc55e4d42a148795d9f25f89d4"

This way, the browser indicates to the server that it already has the content with this hash. If the resource on the server has not been modified, the server replies with the status code 304 Not Modified. This in turn signals to the browser that it can load the content from the cache. The server therefore does not have to send the content of the resource, and ends its reply with an empty body.

Simplified procedure of an HTTP request, where the browser first queries its cache

Correct caching therefore helps to keep transfer volumes small. However, bundling sometimes torpedoes this behaviour. Staying with the example of React: In addition to the framework code (approx. 132 kB in Version 17.0.2), we also have application code. Both land in the same file, which as a result is handled by the server as a single resource.

Unfortunately, both changes to the code and framework updates result in the bundle changing. In the worst case, the browser must therefore download 132 kB of unchanged React code anew with every bug fix because the web server doesn’t know exactly what has changed, but only that something, somewhere, has changed. Although this may not sound like a lot, it quickly adds up when other libraries in addition to React come into play.

The Splitting

The antidote to this is called “code splitting.” Instead of lumping all relevant source files including libraries into a single file, a more intelligent approach is used. Usually, external dependencies land in one file and application-specific code in another file. A bundler makes sure that the import path is correctly implemented, so that the development workflow doesn’t have to be changed. Only the configuration files have to be adjusted.

Code splitting can also be done manually to a certain degree, as the configuration is complicated in places. For this purpose, many libraries have “prebundled” and compressed files available for download, in the case of React for example react.production.min.js. Now instead of writing import React from "react" in your code, it is sufficient to access the global variable React, for the provided bundle exports all functionality in this global variable.

In practice, the header of an HTML file could then look as follows:

<script src="react.production.min.js"></script>
<script src="react-dom.production.min.js"></script>
<script src="app.js"></script>

However, I advise against this approach when further libraries are incorporated, as things quickly become confusing.

Some frontend frameworks, including React, also allow the further partitioning of the application code. More recent versions of JavaScript understand so-called dynamic imports. With this technology, which can be integrated into React components, a global set of components can be used everywhere, while individual pages can still download their specific code. This takes place automatically in the background: The browser already begins to render the framework before the content can be available. An example of this can be found in the React documentation:

import React, { Suspense } from 'react';

const InnerComponent =
  React.lazy(() => import('./OtherComponent'));

const MyComponent = () => (
  <div>
    <Suspense fallback={<div>Loading...</div>}>
      <OtherComponent />
    </Suspense>
  </div>
);

Bundlers can recognize these dynamic imports and split the application code accordingly. To do so, browsers must support the ESM standard (which, as of October 2021, over 90% do). Alternatively, the files can be generated in a different module standard. I would advise against this, though, as this incurs greater complexity.

React lazy loading: The browser can begin rendering before all the components are available. The workflow may differ if a different framework is used

Logically, overly fine-grained partitioning leads to too many HTTP requests whenever a page is viewed. Because, despite caching, the browser must send a request to the web server for each resource.

The Immutability

Ideally, the browser shouldn’t send this request if the resource has not changed on the server. But how would the browser to know? The solution here is “fingerprinting.”

Instead of delivering a script via the URL /app.js, one can simply insert the corresponding hash: /app.js becomes /app.48de1f9.js. The file name and the hash – the fingerprint – must now be referenced in the HTML code.

And then the server must be configured so that with these files the following header is set:

Cache-Control: public,max-age=31536000,immutable

This way, the server tells the browser that the files under this path never change. They can therefore be unconditionally loaded from the cache. This accelerates repeated page loading considerably, as the fastest request is one that doesn’t actually happen.

For this, the HTML header given above must be changed as follows:

<script src="react.production.min.e4de6fa.js"></script>
<script src="react-dom.production.min.cd3664e.js"></script>
<script src="app.48de1f9.js"></script>

If the content of the file changes at a later time, the hash also changes, and thus also the reference in HTML. This way, the browser recognizes that a new resource must be requested. A bundler can automatically adapt the hashes (see next section).

One typically only uses this technology for the so-called assets: styles, JavaScript, images, text, etc. Fingerprints are not used for HTML pages, as otherwise the URLs would change with every update, which of course would be contradictory to the idea of stable links.

Better caching behaviour due to fingerprinting

In summary: ETag-based caching for HTML, fingerprint+immutable for everything else. A bundler ensures that after bundling, the names of the generated files contain the correct fingerprint.

As it is very difficult to combine multiple source files of certain asset classes like images and text, fingerprinting is an important performance technology. Paired with code splitting, it is a powerful tool.

Incidentally, it is not necessary to store the script with the file name app.48de1f9.js in the web server. One can also correspondingly configure the web server so that the URL is rewritten. But beware: Resources with an out-of-date fingerprint should result in a 404. Full-blown web frameworks such as Rails offer this without extensive configuration efforts.

The References

Unfortunately, fingerprinting also has an influence on the actual content of the resources. In the previous section, I mentioned that at the very least the <script> tags have to be adjusted. But other assets are also affected. If for example one uses SVG files for icons, their paths are referenced in CSS:

.icon {
  background-image: url("/cool-icon.svg");
}

But due to fingerprinting, the URL is now suddenly different. The CSS declaration must therefore be correspondingly adjusted by the bundler:

.icon {
  background-image: url("/cool-icon.2008714.svg");
}

It is therefore important that the bundler is configured in such a way that it processes all classes of assets. Internally, they use a dependency tree that encompasses not only JavaScript files but also styles, etc.

In the case of React applications, one often goes one step further still and “imports” styles directly into the JavaScript code:

import styles from "./styles.css";

However, this is a feature that doesn’t actually exist in JavaScript. In a way, the bundler therefore extends the language, which although practical isn’t portable. As the different bundlers can differ quite significantly in the way they handle such imports, I will not go into further detail here.

The Translation

Language extensions are – at least with React – unavoidable. This is because React uses the JavaScript dialect JSX, which allows HTML tags to be used in JavaScript. Example:

const tree = (
  <ul>
    { items.map(item => <li>{item}</li>) }
  </ul>
)

At runtime, this code generates an HTML list (<ul>) with multiple entries (<li>) stremming from the items array. But browsers cannot understand this syntax. This isn’t the place to philosophize about the logic of JSX; instead, I want to explain how actual JavaScript code can be created from it. Cue the “transpiler” …

A transpiler, aka transcompiler, is a tool with which a JavaScript dialect can be translated into a different one. The most popular of these is Babel, whose name already hints at the biblical levels of confusion regarding language dialects. Babel can scrunch the JSX code above and spits out the following JavaScript code:

const tree = React.createElement(
  "ul",
  null,
  {
    children: items.map(item =>
      React.createElement("li", null, item)
    )
  }
)

The code is now freed of all idiosyncratic features and all browsers are able to work with it, provided React has been loaded as a dependency. Right?

Unfortunately, Babel often has to do even more, as modern JavaScript code often uses additional features that, although standardized, are not available in all browsers. Internet Explorer 11, which is still used by some applications (and users), is the main culprit here. It is stuck about a decade in the past, so doesn’t even support the class keyword. Babel can remove such features as well, and – if you want it to – also generate prehistoric JavaScript code.

The technology-agnostic database Browserslist keeps tabs on browser versions and their support of language versions. To transpile, create a configuration file that looks for example as follows:

Last 2 versions
IE 11
> 5% in DE

Babel automatically evaluates this file and configures itself so that the generated code is supported by the two most recent versions of every browser, as well as IE 11 and those with more than 5% market share. All features not supported by the applicable browsers are transpiled by Babel. Caution: The compilation does not always resemble the original. Sometimes Babel has to use every trick in the book to emulate new features.

A translation process whereby from React code a bundle is created that works with ES5. Normally, one wouldn’t include React itself in the bundle, but incorporate it as separate script in the HTML code

A common source of errors with the use of Babel is its often poorly understood configuration. In many projects experimental, not yet standardized JavaScript features are freely applied (so-called proposals in Stage 1, 2, or 3). Their specification and implementation can still change, meaning that an update in Babel version can result in the code no longer working.

As an aside, transpilers can also work with alternative programming languages, such as TypeScript. As far as they are concerned, these are just another JavaScript dialect. The additional challenges resulting from this will not be considered here.

The Polyfill

Anyone thinking that the support of old browser versions is now covered has unfortunately forgotten about Web APIs. Because not only is JavaScript as a language evolving, but browsers are rolling out new APIs at breakneck speed. For example, with custom elements one can – as the name suggests – define unique HTML tags that enrich the content with additional functions.

For the unfortunate 5% of users whose browsers don’t know what to do with this (thanks IE!), Babel is unable to help because as a mere transpiler it knows nothing about the programming interfaces of the browsers. The same applies for example to fetch, the significantly easier to operate successor to XMLHttpRequest.

What to do? The solution is called polyfill, a set of very different technologies named after a brand of putty. There is a polyfill for many modern APIs. It is normally a snippet of JavaScript code that emulates the API with the features of the browser as closely as possible. This doesn’t always work completely, especially when the new feature is deeply anchored in the browser. For fetch, the appropriate polyfill implements the functionality based on XMLHttpRequest and in so doing gives it a new appearance.

It must however be noted that each polyfill inflates the size of the webpage. When designing the architecture, it must therefore be decided whether the “progressive enhancement” approach would perhaps be better. In this case, one designs the site so that it works completely without JavaScript. Modern browsers gain additional features as a bonus. Custom elements are perfect for this. Following this approach, barriers are removed, so that the web application remains usable for everyone (including users with old smartphones or screen readers).

Progressive enhancement however only works to a limited extent with single-page applications. In these cases, one is often forced to apply large amounts of putty.

Both approaches have in common that, in order to be able to use them correctly, a certain amount of preparatory work is required to determine the concrete demands of the project and thus put things on the right track for the development. If you decide to go with polyfills, bundlers can automatically inject these into the generated bundles. Sometimes the necessary polyfills can even be inferred from the source text and the Browserslist configuration.

At the same time it must be noted that the transpiling of language features, as done by Babel, usually preserves the semantics, whereas polyfills, by their nature, have to make compromises. It is therefore essential to read the documentation.

The Images

Whereas Babel can deal admirably with JavaScript, ailing browser support or preprocessing also affects other types of assets. For example, modern browsers can cope with the image formats WEBP and AVIF, which produce substantially smaller files than JPG. HTML 5 offers the option of declaring multiple different image formats and leaving the choice up to the browser:

<picture>
  <source type="image/avif" srcset="pic.avif">
  <source type="image/webp" srcset="pic.webp">
  <img alt="A dog with a hat" src="pic.jpg">
</picture>

A browser unable to deal with this simply loads the JPG format as usual. But if it is able to render AVIF, it loads that version instead. As of October 2021, only about two thirds of web users worldwide can view AVIF files. These users benefit from significantly faster loading times.

This problem can be made as complicated as you like, because in order to speed up the website on all platforms, one can also deliver differently sized images for different display sizes. As many bundlers give up at this stage at the latest, there are a few specialized third-party providers that scale, convert, and optimize images on the fly, depending on the request. With caching support, of course. That technology is called “content negotiation,” where the provider evaluates the request header of the browser in order to determine what the browser can deal with.

Viewed from a technical perspective, the transcoding and scaling of images doesn’t really have anything to do with the processing of JavaScript, so isn’t one of the core competencies of bundlers. Nonetheless, many asset pipelines are able to deal with this, as they support for example the inlining of especially small resources. For example, an image can be optimized as follows: From

<img alt="Loading" src="spinner.gif">

<img alt="Loading" src="data:image/gif;base64,iVBORw0KGgoAAAANSUhEUgAAA...">

This substitution can take place in HTML, CSS, or in JavaScript resources. Similar deliberations on the trade-off between cacheability and cascading requests as with code splitting are also needed.

The Compressing

There is one more important job undertaken by bundlers: So-called “tree-shaking,” aka “dead code elimination.”

On the server side, it is usually unimportant how large the executable artefact is. All transitive dependencies – even if they’re not actually needed in the end – are included in the image. But as I have already made clear here, on the client side, bandwidth is precious.

Tree-shaking describes the process of the bundler removing unused source files from the generated bundle. This isn’t necessarily a complicated job, as a dependency tree is needed anyway for fingerprinting.

In theory anyway. In fact, many bundlers support tree-shaking below the file level, namely on the level of individual definitions. Considering constants for example, they have to determine if they are “pure.” For instance, the following definition should not be deleted, even if x isn’t accessed:

function f() {
  console.log("hello world!");
  return 3;
}

const x = f();

The definition const x = 3 in contrast is pure and can be disposed of if necessary.

There are additional technologies for compression, such as the renaming of long class, function, and variable names. It almost goes without saying that a language as dynamic as JavaScript, where field access is also possible by string, has many pitfalls. Tests should therefore also be undertaken on the processed artefacts where possible.

The Development

If you have successfully improved the performance of a web application thanks to of a bundler, the result that is executed and displayed by the browser often only has a limited resemblance to the original. Especially the last step described above kills readability.

But there is a remedy to this as well. In the debug mode, one can configure bundlers so that they create a source map. The – more or less unchanged – source text is placed in the header of the compiled file as a comment. This of course makes a mockery of all attempts at size reduction, so this should not be used in production. But during the development phase, browsers can use the source map for example to reconstruct accurate stack traces, which significantly simplifies the search for errors.

Concrete benchmarks are however unavoidable. For this, one can use the web inspector provided by all modern browsers, which can display the network requests with and without cache. This quickly allows bottlenecks to be identified. Automatic tools are another building block and can be used in the search for other problems, such as incorrect display on mobile end devices.

I always advise viewing the newly developed page on a smartphone using mobile internet on the train. Because if it works fine then, it will work fine everywhere.

Artikel