[Segment Cache] Support for non-PPR projects/pages (#73960)

Based on:

- #73945
- #73853
- #73667
- #73877

---

This implements support for the Segment Cache in projects/pages where
PPR is not enabled.

I originally thought the Segment Cache would be tied to the roll out of
PPR, because to take full advantage of per-segment caching, you need PPR
to generate static shells for each segment. However, as I was
investigating how to support the incremental PPR opt-in API, where some
pages support PPR and others don't — perhaps just by falling back to the
old cache implementation — I realized that there are benefits to
per-segment caching even if the requests themselves are not per-segment.

For example, when performing a non-PPR-style prefetch, the server diffs
the prefetched page against a description of the base page sent by the
client. In the old caching implementation, the base tree represented
whatever the current page was at the time of the prefetch. In the
Segment Cache implementation, we improve on this by telling the server
to exclude any shared layouts that are already in the client cache.

This ended up requiring more code than I would have preferred, because
dynamic server responses and per-segment server responses have
differentformats and constraints. However, I realized I was going to
have to implement most of this regardless in order to support `<Link
prefetch={true}>`, which performs a full dynamic request.

Once more of the pieces are settled, we can simplify the implementation
by unifying/redesigning some of the data structures, especially
FlightRouterState, which has become overloaded with many different
concerns. But we need to land some of our experiments first — one reason
there's more code than you might expect is because of all the different
combinations of features/experiments we need to support simultaneously.

While it's likely that PPR will be enabled by default sometime within
the next few release cycles, supporting non-PPR projects means we have a
pathway to rolling out additional prefetching improvements (like
prioritization and cancellation) independently of the PPR release.

packages/next/src/client/components/app-router.tsx

@@ -280,7 +280,12 @@ function Router({
           ? // Unlike the old implementation, the Segment Cache doesn't store its
             // data in the router reducer state; it writes into a global mutable
             // cache. So we don't need to dispatch an action.
-            (href) => prefetchWithSegmentCache(href, actionQueue.state.nextUrl)
+            (href) =>
+              prefetchWithSegmentCache(
+                href,
+                actionQueue.state.nextUrl,
+                actionQueue.state.tree
+              )
           : (href, options) => {
               // Use the old prefetch implementation.
               const url = createPrefetchURL(href)

packages/next/src/client/components/router-reducer/prefetch-cache-utils.ts

@@ -397,7 +397,7 @@ export function prunePrefetchCache(
 const DYNAMIC_STALETIME_MS =
   Number(process.env.__NEXT_CLIENT_ROUTER_DYNAMIC_STALETIME) * 1000
 
-const STATIC_STALETIME_MS =
+export const STATIC_STALETIME_MS =
   Number(process.env.__NEXT_CLIENT_ROUTER_STATIC_STALETIME) * 1000
 
 function getPrefetchEntryCacheStatus({