Parse dynamic params on the client

This is part of a larger effort to remove dynamic params from server
responses except in cases where they are needed to render a Server
Component. If a param is not rendered by a Server Component, then it can
omitted from the cache key, and cached responses can be reused across
pages with different param values.

In this step, I've implemented client parsing of the params from
the response headers.

The basic approach is to split the URL into parts, then traverse the
route tree to pick off the param values, taking care to skip over things
like interception routes.

Notably, this is not how the server parses param values. The server
gets the params from the regex that's also used for routing. Originally,
I thought I'd send the regex to the client and use it there, too.
However, this ended up being needlessly complicated, because the server
regexes are also used for things like interception route matching. But
this is already encapsulated by the structure of the route tree. So it's
easier to just walk the tree and pick off the params.

My main hesitation is that there's drift between the server and client
params parsing that we'll need to keep in sync. However, I think the
solution is actually to update the server to also pick the params off
the URL, rather than use the ones passed from the base router. It's
conceptually cleaner, since it's less likely that extra concerns from
the base server leaking into the application layer. As an example,
compare the code needed to get a catchall param value in the client
versus the server implementation.

Note: Although the ultimate goal is to remove the dynamic params from
the response body (e.g. FlightRouterState), I have not done so yet this
PR. The rest of the work will be split up into multiple subsequent PRs.

Author: acdlite

packages/next/errors.json

@@ -764,5 +764,6 @@
   "763": "\\`unstable_rootParams\\` must not be used within a client component. Next.js should be preventing it from being included in client components statically, but did not in this case.",
   "764": "Missing workStore in %s",
   "765": "Route %s used %s inside a Route Handler. Support for this API in Route Handlers is planned for a future version of Next.js.",
-  "766": "%s was used inside a Server Action. This is not supported. Functions from 'next/root-params' can only be called in the context of a route."
+  "766": "%s was used inside a Server Action. This is not supported. Functions from 'next/root-params' can only be called in the context of a route.",
+  "767": "An unexpected response was received from the server."
 }

packages/next/src/client/components/router-reducer/create-initial-router-state.ts

@@ -37,7 +37,26 @@ export function createInitialRouterState({
   // This is to ensure that when the RSC payload streamed to the client, crawlers don't interpret it
   // as a URL that should be crawled.
   const initialCanonicalUrl = initialCanonicalUrlParts.join('/')
-  const normalizedFlightData = getFlightDataPartsFromPath(initialFlightData[0])
+
+  // TODO: Eventually we want to always read the rendered params from the URL
+  // and/or the x-rewritten-path header, so that we can omit them from the
+  // response body. This lets reuse cached responses if params aren't referenced
+  // anywhere in the actual page data, e.g. if they're only accessed by client
+  // components. However, during the initial render, there's no way to access
+  // the headers. For a partially dynamic page, this is OK, because there's
+  // going to be a dynamic server render regardless, so we can send the URL
+  // in the resume body. For a completely static page, though, there's no
+  // dynamic server render, so that won't work.
+  //
+  // Instead, we'll perform a HEAD request and read the rewritten URL from
+  // that response.
+  const renderedPathname = new URL(initialCanonicalUrl, 'http://localhost')
+    .pathname
+
+  const normalizedFlightData = getFlightDataPartsFromPath(
+    initialFlightData[0],
+    renderedPathname
+  )
   const {
     tree: initialTree,
     seedData: initialSeedData,

packages/next/src/client/components/router-reducer/fetch-server-response.ts

@@ -31,6 +31,7 @@ import {
 } from '../../flight-data-helpers'
 import { getAppBuildId } from '../../app-build-id'
 import { setCacheBustingSearchParam } from './set-cache-busting-search-param'
+import { getRenderedPathname } from '../../route-params'
 
 const createFromReadableStream =
   createFromReadableStreamBrowser as (typeof import('react-server-dom-webpack/client.browser'))['createFromReadableStream']
@@ -235,8 +236,21 @@ export async function fetchServerResponse(
       return doMpaNavigation(res.url)
     }
 
+    let renderedPathname
+    if (process.env.__NEXT_CLIENT_SEGMENT_CACHE) {
+      // Read the URL from the response object.
+      renderedPathname = getRenderedPathname(res)
+    } else {
+      // Before Segment Cache is enabled, we should not rely on the new
+      // rewrite headers (x-rewritten-path, x-rewritten-query) because that
+      // is a breaking change. Read the URL from the response body.
+      const renderedUrlParts = response.c
+      renderedPathname = new URL(renderedUrlParts.join('/'), 'http://localhost')
+        .pathname
+    }
+
     return {
-      flightData: normalizeFlightData(response.f),
+      flightData: normalizeFlightData(response.f, renderedPathname),
       canonicalUrl: canonicalUrl,
       couldBeIntercepted: interception,
       prerendered: response.S,

packages/next/src/client/components/router-reducer/reducers/server-action-reducer.ts

@@ -55,6 +55,7 @@ import {
   omitUnusedArgs,
 } from '../../../../shared/lib/server-reference-info'
 import { revalidateEntireCache } from '../../segment-cache'
+import { getRenderedPathname } from '../../../route-params'
 
 const createFromFetch =
   createFromFetchBrowser as (typeof import('react-server-dom-webpack/client.browser'))['createFromFetch']
@@ -180,9 +181,25 @@ async function fetchServerAction(
       Promise.resolve(res),
       { callServer, findSourceMapURL, temporaryReferences }
     )
+
+    let renderedPathname
+    if (process.env.__NEXT_CLIENT_SEGMENT_CACHE) {
+      // Read the URL from the response object.
+      renderedPathname = getRenderedPathname(res)
+    } else {
+      // Before Segment Cache is enabled, we should not rely on the new
+      // rewrite headers (x-rewritten-path, x-rewritten-query) because that
+      // is a breaking change. Read the URL from the response body.
+      const canonicalUrlParts = response.c
+      renderedPathname = new URL(
+        canonicalUrlParts.join('/'),
+        'http://localhost'
+      ).pathname
+    }
+
     // An internal redirect can send an RSC response, but does not have a useful `actionResult`.
     actionResult = redirectLocation ? undefined : response.a
-    actionFlightData = normalizeFlightData(response.f)
+    actionFlightData = normalizeFlightData(response.f, renderedPathname)
   } else {
     // An external redirect doesn't contain RSC data.
     actionResult = undefined

packages/next/src/client/components/segment-cache-impl/cache-key.ts

@@ -1,6 +1,3 @@
-import { NEXT_REWRITTEN_QUERY_HEADER } from '../app-router-headers'
-import type { RSCResponse } from '../router-reducer/fetch-server-response'
-
 // TypeScript trick to simulate opaque types, like in Flow.
 type Opaque<K, T> = T & { __brand: K }
 
@@ -32,18 +29,3 @@ export function createCacheKey(
   } as RouteCacheKey
   return cacheKey
 }
-
-export function getRenderedSearch(response: RSCResponse): NormalizedSearch {
-  // If the server performed a rewrite, the search params used to render the
-  // page will be different from the params in the request URL. In this case,
-  // the response will include a header that gives the rewritten search query.
-  const rewrittenQuery = response.headers.get(NEXT_REWRITTEN_QUERY_HEADER)
-  if (rewrittenQuery !== null) {
-    return (
-      rewrittenQuery === '' ? '' : '?' + rewrittenQuery
-    ) as NormalizedSearch
-  }
-  // If the header is not present, there was no rewrite, so we use the search
-  // query of the response URL.
-  return new URL(response.url).search as NormalizedSearch
-}

packages/next/src/client/components/segment-cache-impl/cache.ts

@@ -9,6 +9,7 @@ import type {
 } from '../../../shared/lib/app-router-context.shared-runtime'
 import type {
   CacheNodeSeedData,
+  DynamicParamTypesShort,
   Segment as FlightRouterStateSegment,
 } from '../../../server/app-render/types'
 import { HasLoadingBoundary } from '../../../server/app-render/types'
@@ -42,7 +43,13 @@ import type {
   NormalizedSearch,
   RouteCacheKey,
 } from './cache-key'
-import { getRenderedSearch } from './cache-key'
+import {
+  doesStaticSegmentAppearInURL,
+  getRenderedPathname,
+  getRenderedSearch,
+  parseDynamicParamFromURLPart,
+  type RouteParam,
+} from '../../route-params'
 import { createTupleMap, type TupleMap, type Prefix } from './tuple-map'
 import { createLRU } from './lru'
 import {
@@ -88,7 +95,10 @@ import { FetchStrategy } from '../segment-cache'
 
 export type RouteTree = {
   key: string
+  // TODO: Remove the `segment` field, now that it can be reconstructed
+  // from `param`.
   segment: FlightRouterStateSegment
+  param: RouteParam | null
   slots: null | {
     [parallelRouteKey: string]: RouteTree
   }
@@ -851,19 +861,60 @@ function rejectSegmentCacheEntry(
   }
 }
 
-function convertRootTreePrefetchToRouteTree(rootTree: RootTreePrefetch) {
-  return convertTreePrefetchToRouteTree(rootTree.tree, ROOT_SEGMENT_KEY)
+function convertRootTreePrefetchToRouteTree(
+  rootTree: RootTreePrefetch,
+  renderedPathname: string
+) {
+  // Remove trailing and leading slashes
+  const pathnameParts = renderedPathname.replace(/^\/|\/$/g, '').split('/')
+  const index = 0
+  return convertTreePrefetchToRouteTree(
+    rootTree.tree,
+    ROOT_SEGMENT_KEY,
+    pathnameParts,
+    index
+  )
 }
 
 function convertTreePrefetchToRouteTree(
   prefetch: TreePrefetch,
-  key: string
+  key: string,
+  pathnameParts: Array<string>,
+  pathnamePartsIndex: number
 ): RouteTree {
   // Converts the route tree sent by the server into the format used by the
   // cache. The cached version of the tree includes additional fields, such as a
   // cache key for each segment. Since this is frequently accessed, we compute
   // it once instead of on every access. This same cache key is also used to
   // request the segment from the server.
+
+  const segment = prefetch.segment
+
+  let doesAppearInURL: boolean
+  let param: RouteParam | null = null
+  if (Array.isArray(segment)) {
+    // This segment is parameterized. Get the param from the pathname.
+    const paramType = segment[2] as DynamicParamTypesShort
+    param = {
+      name: segment[0],
+      value: parseDynamicParamFromURLPart(
+        paramType,
+        pathnameParts,
+        pathnamePartsIndex
+      ),
+      type: paramType,
+    }
+    doesAppearInURL = true
+  } else {
+    doesAppearInURL = doesStaticSegmentAppearInURL(segment)
+  }
+
+  // Only increment the index if the segment appears in the URL. If it's a
+  // "virtual" segment, like a route group, it remains the same.
+  const childPathnamePartsIndex = doesAppearInURL
+    ? pathnamePartsIndex + 1
+    : pathnamePartsIndex
+
   let slots: { [parallelRouteKey: string]: RouteTree } | null = null
   const prefetchSlots = prefetch.slots
   if (prefetchSlots !== null) {
@@ -881,13 +932,17 @@ function convertTreePrefetchToRouteTree(
       )
       slots[parallelRouteKey] = convertTreePrefetchToRouteTree(
         childPrefetch,
-        childKey
+        childKey,
+        pathnameParts,
+        childPathnamePartsIndex
       )
     }
   }
+
   return {
     key,
-    segment: prefetch.segment,
+    segment,
+    param,
     slots,
     isRootLayout: prefetch.isRootLayout,
     // This field is only relevant to dynamic routes. For a PPR/static route,
@@ -935,26 +990,39 @@ function convertFlightRouterStateToRouteTree(
       slots[parallelRouteKey] = childTree
     }
   }
-
-  // The navigation implementation expects the search params to be included
-  // in the segment. However, in the case of a static response, the search
-  // params are omitted. So the client needs to add them back in when reading
-  // from the Segment Cache.
-  //
-  // For consistency, we'll do this for dynamic responses, too.
-  //
-  // TODO: We should move search params out of FlightRouterState and handle them
-  // entirely on the client, similar to our plan for dynamic params.
   const originalSegment = flightRouterState[0]
-  const segmentWithoutSearchParams =
-    typeof originalSegment === 'string' &&
-    originalSegment.startsWith(PAGE_SEGMENT_KEY)
-      ? PAGE_SEGMENT_KEY
-      : originalSegment
+
+  let segment: FlightRouterStateSegment
+  let param: RouteParam | null = null
+  if (Array.isArray(originalSegment)) {
+    const paramValue = originalSegment[3]
+    param = {
+      name: originalSegment[0],
+      value: paramValue === undefined ? null : paramValue,
+      type: originalSegment[2] as DynamicParamTypesShort,
+    }
+    segment = originalSegment
+  } else {
+    // The navigation implementation expects the search params to be included
+    // in the segment. However, in the case of a static response, the search
+    // params are omitted. So the client needs to add them back in when reading
+    // from the Segment Cache.
+    //
+    // For consistency, we'll do this for dynamic responses, too.
+    //
+    // TODO: We should move search params out of FlightRouterState and handle
+    // them entirely on the client, similar to our plan for dynamic params.
+    segment =
+      typeof originalSegment === 'string' &&
+      originalSegment.startsWith(PAGE_SEGMENT_KEY)
+        ? PAGE_SEGMENT_KEY
+        : originalSegment
+  }
 
   return {
     key,
-    segment: segmentWithoutSearchParams,
+    segment,
+    param,
     slots,
     isRootLayout: flightRouterState[4] === true,
     hasLoadingBoundary:
@@ -1139,15 +1207,21 @@ export async function fetchRouteOnCacheMiss(
         return null
       }
 
-      // Get the search params that were used to render the target page. This may
-      // be different from the search params in the request URL, if the page
+      // Get the params that were used to render the target page. These may
+      // be different from the params in the request URL, if the page
       // was rewritten.
+      const renderedPathname = getRenderedPathname(response)
       const renderedSearch = getRenderedSearch(response)
 
+      const routeTree = convertRootTreePrefetchToRouteTree(
+        serverData,
+        renderedPathname
+      )
+
       const staleTimeMs = serverData.staleTime * 1000
       fulfillRouteCacheEntry(
         entry,
-        convertRootTreePrefetchToRouteTree(serverData),
+        routeTree,
         serverData.head,
         serverData.isHeadPartial,
         Date.now() + staleTimeMs,
@@ -1458,7 +1532,15 @@ function writeDynamicTreeResponseIntoCache(
   canonicalUrl: string,
   routeIsPPREnabled: boolean
 ) {
-  const normalizedFlightDataResult = normalizeFlightData(serverData.f)
+  // Get the URL that was used to render the target page. This may be different
+  // from the URL in the request URL, if the page was rewritten.
+  const renderedSearch = getRenderedSearch(response)
+  const renderedPathname = getRenderedPathname(response)
+
+  const normalizedFlightDataResult = normalizeFlightData(
+    serverData.f,
+    renderedPathname
+  )
   if (
     // A string result means navigating to this route will result in an
     // MPA navigation.
@@ -1492,11 +1574,6 @@ function writeDynamicTreeResponseIntoCache(
   const isResponsePartial =
     response.headers.get(NEXT_DID_POSTPONE_HEADER) === '1'
 
-  // Get the search params that were used to render the target page. This may
-  // be different from the search params in the request URL, if the page
-  // was rewritten.
-  const renderedSearch = getRenderedSearch(response)
-
   const fulfilledEntry = fulfillRouteCacheEntry(
     entry,
     convertRootFlightRouterStateToRouteTree(flightRouterState),
@@ -1566,7 +1643,12 @@ function writeDynamicRenderResponseIntoCache(
     }
     return null
   }
-  const flightDatas = normalizeFlightData(serverData.f)
+
+  // Get the URL that was used to render the target page. This may be different
+  // from the URL in the request URL, if the page was rewritten.
+  const renderedPathname = getRenderedPathname(response)
+
+  const flightDatas = normalizeFlightData(serverData.f, renderedPathname)
   if (typeof flightDatas === 'string') {
     // This means navigating to this route will result in an MPA navigation.
     // TODO: We should cache this, too, so that the MPA navigation is immediate.

packages/next/src/client/components/segment-cache.ts

@@ -19,6 +19,7 @@
 
 export type { NavigationResult } from './segment-cache-impl/navigation'
 export type { PrefetchTask } from './segment-cache-impl/scheduler'
+export type { NormalizedSearch } from './segment-cache-impl/cache-key'
 
 const notEnabled: any = () => {
   throw new Error(