lib-web-prerender

• The cached page has a freshness window — cacheSeconds: 300 (5 minutes) in web-shop-simo
• For those 5 minutes, a visitor can see the old price in a cached page
• Once 5 minutes pass, the next visitor still gets served the (now stale) page once more — but that request also silently triggers a background rebuild
• The visitor after that gets the new price

Worst case: stale pricing for roughly cacheSeconds + one more visit's worth of time. This is a deliberate speed-vs-freshness trade-off — lower cacheSeconds if pricing needs to be fresher, at the cost of more traffic to the prerender service.

• Every cached page is stamped with a buildHash (in web-shop-simo: the CI build ID)
• After a deploy, the new server instance has a new buildHash
• Old cached pages in Redis still carry the old buildHash
• On the next request, the library sees the mismatch and refuses to serve the old page — a fresh render happens while it rebuilds

This is what stops old cached pages referencing deleted JS/CSS files after a deploy — no manual cache flush needed.

shouldSkipPrerender(req, res) runs first. If it returns truthy the middleware immediately calls next() and does nothing else — no Redis access, no prerender request. See When Prerender Is Skipped for the full rule set.

If a getBuildHash(req) function was supplied it is awaited and overwrites options.buildHash for this request. Then getUrlHash(req, res) is awaited — consumers use this to add their own dimension (e.g. an experiment bucket or brand) into the cache key. It defaults to an empty string.

getCacheableUrls() returns { cacheableUrl, cacheableBotUrl } — the user key (includes the URL hash) and the bot key (prefixed static-, no URL hash). See Cache Keys Explained.

If geoLocationConfig is an array, getGeoLocationOverride() finds the first entry whose url appears in req.url and returns it with a default accuracy of 10. This is forwarded to the prerender service so a location-specific page can be rendered (used for store/coverage style pages).

One object carrying everything downstream helpers need: all resolved options, both cache keys, req.cookies, the optional geolocation, and — importantly — req, res and next themselves. Every helper takes this single object.

1. Googlebot and a bot cache exists → returnStaticCachedPage(). Sends the script-stripped HTML.
2. A user cache entry exists → returnCachedPage(). Validates freshness and build, may refresh in the background, then sends or defers.
3. Nothing cached → log, write the isCurrentlyPrerendering marker, fire forwardToPrerenderServer() without awaiting, and call next() so this visitor gets a normal client-side render.

export { default } from './middleware/prerender'

Simply: the front door. It says "the thing this package gives you is the prerender middleware".

Technically: the package's main is index.js relative to the published lib/, so this compiles to lib/index.js. Note that only the middleware is re-exported — the client-side helpers are reached by deep path.

export { default } from './prerender'

Simply: the same thing one level down, so import … from './middleware' works.

Simply: the manager. It does no detailed work itself — it asks the right specialist at the right moment and decides whether to answer the visitor or hand the job back to the website.

Technically: a factory prerender(customOptions) that closes over resolved options and returns an async Express middleware.

Setup phase (runs once)

1. { ...defaultOptions, ...customOptions, ...fixPrerenderServerUrl(...) } — note the URL fix is spread last, so it always wins.
2. In local/development, options.redis ||= new MockRedis(). Marked TODO: deprecated pending the boilerplate migration.
3. initShouldSkipPrerender(options) is called once and its returned closure reused.

Per-request phase

1. await shouldSkipPrerender(req, res) → next() and return.
2. getBuildHash / getUrlHash resolution.
3. getCacheableUrls → both keys.
4. getGeoLocationOverride, only when geoLocationConfig is an array.
5. Build prerenderOptions, carrying req/res/next.
6. The three-way branch on Googlebot / cache hit / miss.

Why it is written this way

• One options object passed everywhere keeps helper signatures uniform and makes them trivially testable.
• The miss branch does not await forwardToPrerenderServer, which is what keeps latency flat on a cold cache.
• Two exists calls, not get — the branch only needs to know whether a key is present; the body is fetched inside the chosen handler.

Simply: the sensible-defaults sheet. Anything a consumer does not specify comes from here.

Technically: a frozen-by-convention object literal. The one live expression is buildHash: uuidv4(), evaluated at import time — i.e. once per Node process. See the warning in Build Hash & Releases.

Note what is absent: environment, skipChecker, getBuildHash, alwaysPrerenderRoutes, geoLocationConfig, skipReturnCachedPage, authCookiePrefix and redis have no defaults and are simply undefined unless supplied.

Simply: the bouncer. It answers one question — "is it safe and worthwhile to serve this visitor a shared page?" If not, the library stands aside.

Technically: a curried function; the outer call binds options at start-up, the inner async call evaluates per request. Returns true to skip.

Evaluation order: hard skips → alwaysPrerenderRoutes override → [hasPrerenderHeader, hasJourney, skipChecker?] combined with .some(Boolean). Full rule table in When Prerender Is Skipped.

Why it matters: this file is the privacy boundary. A bug here means a logged-in customer's page could be cached and served to a stranger. Treat every change as security-relevant.

Simply: finds the cookie that says whether someone is signed in.

const cookiePrefix = authCookiePrefix || process.env.AUTH_COOKIE_PREFIX || 'eShop-auth'
return req.cookies[`${cookiePrefix}_p_id_token`] || req.cookies.Session || {}

Returning {} on failure is deliberate: the caller reads session.assuranceLevel, and undefined >= 3 is false, so an absent session correctly means "not logged in" rather than throwing. It has no test file — the behaviour is covered through initShouldSkipPrerender's tests.

Simply: if someone configured prerender.internal instead of https://prerender.internal, this adds the missing bit.

Returns { prerenderServerUrl } rather than a bare string so it can be spread over the options object in one expression. Note the filename (…ServiceUrl.js) differs from its folder and its test file (…ServerUrl.test.js) — harmless, but confusing when grepping.

Simply: a pretend Redis, so you can develop without running the real one.

export default class MockRedis {
  store = {}
  exists = async (key) => !!this.store[key]
  set = async (key, value) => (this.store[key] = value)
  get = async (key) => this.store[key]
}

It ignores the 'ex' TTL arguments entirely and is per-process, so nothing expires and nothing is shared. Only injected when NODE_ENV is local or development.

Simply: makes the two labels — one for people, one for Google.

Technically: calls getCacheableUrl twice. The bot call passes isBot: true and deliberately omits urlHash, so crawlers get one canonical entry per URL rather than one per experiment variant.

Simply: writes the label for one page.

Technically: splits req.url on ?, parses the query with qs, forces a trailing slash on the path, then appends — in this order — the URL hash, the retained query params, and the retained cookies. Result:

`${isBot ? 'static-' : ''}${environment}${urlWithoutQuery}${queryString}`

qs is used but is not declared in package.json — it resolves transitively (via Express). Worth knowing if dependencies are ever pruned.

Simply: one small string-builder used for two different jobs.

Technically: a private _getParamString with an isForCacheKey switch that selects the separator and terminator, exposed as two named functions:

A key that is present but empty contributes just its name — the presence of a parameter can be meaningful even without a value.

Simply: throws away the query parameters nobody declared as important, and returns what is left with a ? in front.

Very close to appendQueryToCache but separate because it starts a fresh string and prefixes ?. Returns '' when there is no query string or nothing to retain. Used for the URL sent to the prerender service, not for the cache key.

Simply: "if this page is about a place, tell the renderer to pretend it is standing there."

Finds the first geoLocationConfig entry whose url is a substring of req.url, returns it with accuracy defaulted to 10, and returns undefined when nothing matches. The result is forwarded in the prerender request body.

Simply: "does the visitor say it is Google?"

return !!headers['user-agent'] && headers['user-agent'].includes('Googlebot')

A substring check on a spoofable header. That is acceptable because the worst outcome is that someone receives a script-free page. Other crawlers do not get the static branch — for the broader question, returnCachedPage uses the isbot package instead.

Simply: the quality inspector on the hot counter. It picks up the ready-made page and runs through a checklist before handing it over — is it finished? is it from the current menu? is it too old? Depending on the answers it serves it, orders a replacement, or waves the visitor through to the kitchen.

Technically: reads and JSON-parses the entry, then evaluates eight states in a fixed order. Full table in Cache States & Staleness.

The three things it can do

• next() — let the app render client-side.
• res.status(...).send(...) with X-Prerender-Cache: true.
• Fire forwardToPrerenderServer(options) without awaiting — a background refresh.

Details that are easy to miss

• The absence of statusCode is how a placeholder is recognised. The single-flight marker written by setIsCurrentlyPrerenderingInCache has only isCurrentlyPrerendering and timeOfCache, so "no status code" means "no real page here yet".
• The expiry branch does not return. It triggers the refresh and then falls through to send the stale page — that is the stale-while-revalidate behaviour.
• The geolocation branch inverts the usual logic: if the page has a geolocation override and the visitor is not a bot (checked with isbot), the cached page is not served. A location-specific cached page is for crawlers; real users get a live render.
• The status code is replayed. A cached 404 is sent as a 404, not a 200.
• The file begins with stray ;;. Cosmetic, harmless, but it is really there.

Simply: hands Google the script-free copy. No checks, no timers — read it and send it.

Technically: redis.get(cacheableBotUrl) → parse → set X-Prerender-Cache: true → res.status(...).send(...). Any error is warned and next() is called.

Deliberately much simpler than returnCachedPage: no freshness check, no build-hash check, no background refresh. A crawler does not execute the page, so a slightly old static copy is harmless, and its Redis TTL of 72h is the only lifetime control.

Simply: the cook. It asks the rendering service for a fully-built page, checks the result is good, tidies up the HTML, and puts two copies on the counter — one for people, one for Google.

Step by step

1. buildPrerenderUrl → { requestUrl, urlToRender }.
2. getRequestOptions → the axios config.
3. await axios(options).
4. getIsCacheable(response, buildHash, disable…). If false → handleInvalidPrerender and stop; nothing is cached.
5. replacePlaceholders(response.data.content, [...defaultReplacements(...), ...htmlReplacements]) — consumer replacements run last, so they can override.
6. Write the bot entry: removeScriptTags(completeHtml), status, timestamp, TTL 259200.
7. Write the user entry: full HTML, apiData, status, timestamp, isCurrentlyPrerendering: false, buildHash, TTL 86400.

It writes to Redis directly with redis.set(...) rather than going through cachePageResponse, which is why the TTLs appear here as literals.

Simply: works out two addresses — where to send the render request, and which page the renderer should load.

requestUrl:   `${prerenderServerUrl}/render`
urlToRender:  `${appServerEndpoint}${urlToPrerender}`

appServerEndpoint is decided by environment:

The boilerplate check runs after the local check, so it wins when both apply. The default case — telling the prerender service to render prerenderServerUrl itself — is why defaultReplacements later rewrites that host to cdnDomain in the cached HTML.

Simply: the page path plus only the query parameters that matter.

Splits req.url and re-joins it with getFilteredQueryString(...). This keeps the rendered page aligned with the cache key: if a parameter is not worth its own cache entry, it is not sent to the renderer either.

Simply: fills in the form that is sent to the rendering service.

{
  method: 'post',
  url: requestUrl,                 // <prerenderServerUrl>/render
  followRedirect: false,
  withCredentials: true,
  headers: { Cookie?: '…' },       // only if cookiesToRetain is non-empty
  timeout: prerenderRequestTimeout,
  data: {
    renderType: 'html',
    javascript: 'window.prerenderData=window._hydratedData',
    url: urlToRender,
    geoLocation?: { … },
  },
  httpsAgent: new HttpsAgent({ rejectUnauthorized: false }),
  validateStatus: () => true,
}

The three lines worth understanding

• javascript: a snippet the headless browser runs after the page settles. It copies the app's accumulated _hydratedData onto prerenderData, which is how the captured API responses come back in the response body.
• validateStatus: () => true: axios never throws on an HTTP status. getIsCacheable owns that judgement instead — that is how a legitimate 404 can be cached.
• rejectUnauthorized: false: accepts self-signed certificates on the internal prerender host. A new HttpsAgent is constructed per request, so the keep-alive pooling agentkeepalive provides is not actually shared between requests.

followRedirect is not an axios option (axios uses maxRedirects), so it has no effect. validateStatus is also spelled without the axios validateStatus casing issue — it is correct — but the property axios reads is validateStatus only in v1; confirm against the pinned axios version before relying on it.

Simply: the taste test. Three questions, and a "no" to any of them means the page is thrown away rather than stored.

Each failure logs a distinct [Prerender] warning, which makes this the single most useful file to grep for when pages mysteriously never cache.

const isCurrentBuild = (html, buildIdentifier) => html.includes(buildIdentifier)

Simply: "does this page mention the current version number?" One line, no test file, but load-bearing — it is the reason a deploy invalidates the cache.

Simply: the standing list of find-and-replace edits every cached page gets.

Technically: builds an array of { placeholder, replacement } — two always, three more conditionally. Full table in The HTML Contract.

Why each edit exists

• _isPrerender → false: the cached HTML was produced during a prerender but will be served to real browsers, which must run their analytics and browser-only code.
• _hydratedData → JSON: inlines the captured API responses so the client boots without repeating them. This is the pay-off of the whole hydrate mechanism.
• vfukTarget: same idea for Adobe Target personalisation data, added only when the render captured vfukTargetData.
• prerenderServerUrl → cdnDomain: the only global RegExp in the list. Without it, cached pages would ask browsers to fetch assets from the internal prerender host. Skipped when NODE_ENV === 'local'.
• headUtagToReplace: swaps a placeholder script tag for the real Tealium utag.sync.js, which must load synchronously in <head>.

Simply: actually performs the edits, one after another.

replacementList.forEach((item) => {
  let placeholderKey = item.placeholder
  if (process.env.PRERENDER_BOILERPLATE === 'true') {
    placeholderKey = item.boilerplatePlaceholder || item.placeholder
  }
  amendedHtml = amendedHtml.replace(placeholderKey, item.replacement)
})

The PRERENDER_BOILERPLATE branch is marked TODO: deprecated — it exists so the same library version works against both the old markup and the new boilerplate markup during migration.

Folder is replacePlaceHolders (capital H) but the file is replacePlaceholders.js (lower h). Case-sensitive filesystems care; macOS does not.

Simply: takes the interactive bits out of the copy meant for Google, but keeps the machine-readable product information.

html.match(/<script(?:.*?)>(?:[\S\s]*?)<\/script>/gi)
  // remove each match unless it contains 'application/ld+json'

Regex-based rather than DOM-based because it must be fast and there is no parser available server-side here. Wrapped in try/catch returning the original HTML on failure — a bot page with scripts is better than no bot page.

Keeping JSON-LD is the entire point: that is where structured product, price and breadcrumb data lives, and it is what rich search results are built from.

Simply: writes an object into Redis as text, and refuses to crash if Redis is unhappy.

await redis.set(cacheableUrl, JSON.stringify(cachedObject), 'ex', 86400)

Swallows errors with a warning — a failed cache write must never break a response. Note the 24h TTL is hard-coded here too, so a placeholder marker also lives for 24h if never overwritten.

Simply: puts up a "being cooked" sign so several visitors do not all order the same dish at once.

Technically: two modes:

• With an existing cacheObject — sets the flag on it and rewrites it, so the currently-cached HTML survives while the refresh runs. On failure it falls back to writing a bare marker.
• Without one — writes a bare marker { isCurrentlyPrerendering, timeOfCache }. This is the object whose missing statusCode tells returnCachedPage "no real page here yet".

The distinction matters: a cold miss must not serve anything, but an expiring page must keep serving while it refreshes.

Simply: takes the "being cooked" sign down when the cooking failed, so someone can try again.

Reads the entry, sets isCurrentlyPrerendering = false, writes it back, then sets its local redis variable to null (a no-op for the caller — reassigning a parameter does not affect the caller's reference).

Simply: "ask for this data, but if the prerender already fetched it, just use that."

Technically: returns a Promise. If window._hydratedData[nameOfData] exists, resolves it immediately — arrays as-is, objects spread with { isCachedData: true, ... }. Otherwise it initialises _hydratedData and the outstandingRequiredRequests counter, optionally increments that counter, awaits dataCall(), stores the result under the key, decrements, and resolves.

Why this exists: without it, a prerendered page would arrive fully drawn and then immediately re-request every API on boot — the visitor would see a flash of loading states and the cache would have saved nothing but the first paint. Full usage guidance in Using hydrate().

Uses the new Promise(async (resolve, reject) => …) anti-pattern. It works because the body is wrapped in try/catch, but a throw outside that catch would be lost.

Simply: "the page is not finished until this has come back."

The counter half of hydrate with no caching: increment, await, decrement. Use it for work the render must wait for but whose result should not be stored — and remember that if it rejects, the counter is never decremented, so the render is permanently judged incomplete and the page will never cache.

Simply: for pages with no data to fetch, this says "nothing to wait for — go ahead and cache me."

if (window._isPrerender) {
  window._hydratedData = window._hydratedData || {}
  window._hydratedData.outstandingRequiredRequests =
    window._hydratedData.outstandingRequiredRequests || 0
}

Without it, a purely static page never touches hydrate, so outstandingRequiredRequests is undefined, undefined === 0 is false, and getIsCacheable rejects the render. A tiny file that fixes a confusing "my static page won't cache" failure.

Simply: for parts of the page that load their code separately, this shows the version the prerender already drew instead of an empty box while the code downloads.

Technically: a higher-order component factory. Given { module: { scope, name, getComponent }, fallback, errorComponent } it:

1. Computes compId = `${scope}-${name}` and grabs document.getElementById(compId) at factory time — the prerendered markup.
2. Returns a component that calls useLoadDynamicImport(module).
3. While loading: if prerendered HTML was found, renders it back via dangerouslySetInnerHTML; otherwise renders the supplied fallback.
4. On error: renders errorComponent or a plain <div>Error</div>.
5. Once resolved: renders the real component inside <div id={compId}>, which is what makes the ID available for the next prerender.

The wrapper div id is the whole mechanism — it is the anchor that links a prerendered fragment to the component that will eventually replace it.

Simply: downloads the component's code and reports whether it is ready.

A hook that calls module.getComponent() in an effect keyed on [scope, name] and returns { dynamicComponent, isLoading, error }. Stores the component with setDynamicComponent(() => module.default) — the functional form, because React would otherwise treat a function value as a state updater.

uuidv4() runs once per Node process. Behind a load balancer each instance has a different hash, so every instance treats the others' cache entries as "old build" and regenerates them. Effective cache hit rate collapses to near zero, silently — no errors, just slow pages.

Fix: always pass a deployment-stable buildHash (webpack bundle hash, image tag, release SHA).

html.includes('') is always true. If buildHash resolves to an empty string, the build check passes for everything and release invalidation stops working — while every log line looks healthy.

The flag has no timestamp of its own and no timeout. If the process dies between setting the flag and writing the result, or if handleInvalidPrerender itself throws (it has no try/catch and JSON.parse(null) yields null), the flag stays set. Every later request sees "someone is already rendering this" and falls through to CSR until the Redis TTL expires.

Symptom: one URL stops caching and never recovers while others are fine.

Workaround: delete that cache key in Redis.

if (options.getBuildHash) options.buildHash = await options.getBuildHash(req) writes to the closed-over options object, not a per-request copy. Harmless when getBuildHash is constant, but if it varies per request, concurrent requests can read each other's value.

Cache-key fragments follow the array order, not the URL order or alphabetical order. Swapping two entries produces different keys for identical pages, so every entry becomes a cold miss on deploy.

It is Array.includes(req.url), so the query string is part of the comparison and there is no pattern matching. '/plans' will not match /plans?segment=business or /plans/.

Only the prerenderServerUrl → cdnDomain rule uses a global RegExp. If a marker such as var _hydratedData = {}; appears twice in your template, the second one survives and overwrites the injected data at runtime.

initShouldSkipPrerender reads req.cookies.basketId directly. Without cookie-parser mounted first, req.cookies is undefined and the property access throws.

The decrement happens after await dataCall(). If the call rejects, the catch path rejects the promise without decrementing, so outstandingRequiredRequests stays above zero and getIsCacheable rejects every render of that page. hydrate with isRequired: true has the same shape.

TLS certificate verification is disabled for the call to the prerender service, and a fresh HttpsAgent is created per request — so the keep-alive pooling that agentkeepalive exists to provide is not actually reused. Acceptable for an internal host with a self-signed certificate, but worth being explicit about.

forwardToPrerenderServer is never awaited. Its own try/catch catches everything and logs a console.warn. There is no metric, no error propagation and no retry beyond the next request — so a prerender service outage looks like "the site is a bit slower" unless someone is reading logs.

86400 appears in forwardToPrerenderServer and cachePageResponse; 259200 in forwardToPrerenderServer; http://localhost:8000 in buildPrerenderUrl. None are options, so a shop running its dev server on a different port cannot prerender locally without the boilerplate env vars.

• Folder fixPrerenderServerUrl/ contains fixPrerenderServiceUrl.js.
• Folder replacePlaceHolders/ contains replacePlaceholders.js.
• returnCachedPage.js starts with a stray ;;.
• prerender.js imports returnStaticCachedPage by full file path while every other import uses the folder.
• CHANGELOG.md is empty; sonar-project.properties is the untouched template.
• Several npm scripts point at directories that do not exist.
• followRedirect in the axios options is not an axios option (it would be maxRedirects).

1. Is X-Prerender-Cache ever present on a response? If never, the middleware may be skipping.
2. Is prerenderServerUrl set? A falsy value disables the library entirely and logs nothing.
3. Is prerender() mounted before your renderer, and after cookieParser()?
4. Do you have a basketId, JourneyID or session cookie in your browser? Try a private window.
5. Look for BuildHash not found or Required requests unsuccessful — those are outright rejections.

Most likely a stuck isCurrentlyPrerendering flag. Inspect the key in Redis; if it holds a marker with no statusCode, delete it. Second possibility: a required API call on that page rejects, so the counter never returns to zero.

Almost always the buildHash. Locally there is one process, so the uuid default is consistent; in production every instance disagrees. Look for a steady stream of Cached page is old build warnings — that is the fingerprint.

The hydration data did not make it in. Check that var _hydratedData = {}; appears in the template exactly once and byte-for-byte, that your API calls are wrapped in hydrate() with stable keys, and — if you are on the new boilerplate — that PRERENDER_BOILERPLATE=true is actually set to the string 'true'.

The CDN rewrite. Outside NODE_ENV=local, every occurrence of prerenderServerUrl is replaced with cdnDomain — or with an empty string if cdnDomain is unset, which can mangle asset URLs. Set cdnDomain explicitly.

The bot cache is only ever written during a user-driven prerender, never by the bot branch itself. A URL that only crawlers visit will have no static entry. Add it to alwaysPrerenderRoutes or drive traffic to warm it, and verify the content survives removeScriptTags — anything rendered by client-side JavaScript will be gone.

Expected within the stale-while-revalidate window: an expired page is served once while its replacement is built. Lower cacheSeconds to shrink the window. If staleness lasts far longer, the background refresh is failing — check the warn-level logs.

The x-prerender header is not reaching the middleware. That header is what makes the middleware recognise the render service's own request and skip itself. Check that nothing between the render service and the app strips it.