chore!: refactor MapeoManager to accept Fastify server instance

docs/guides/media-server.md → docs/guides/mapeo-http-server.md

@@ -1,28 +1,40 @@
-# Mapeo's Media Server
+# Mapeo's HTTP Server
 
-Each Mapeo manager instance includes an embedded HTTP server that is responsible for serving media assets over HTTP. Each server is responsible for handling requests for assets that can live in any Mapeo project (the URL structure reflects this, as we will show later on).
+Each Mapeo manager instance requires an adjacent Fastify server instance that is responsible for serving assets over HTTP. Each Fastify instance is responsible for handling requests for assets that can live in any Mapeo project (the URL structure reflects this, as we will show later on).
 
 Some boilerplate for getting started with a Mapeo project:
 
 ```js
+// Create Fastify instance
+const fastify = Fastify()
+
 // Create the manager instance (truncated for brevity)
-const manager = new MapeoManager({...})
+const manager = new MapeoManager({ fastify, ... })
+
+// Start the HTTP server (awaitable but no need to await in most cases, unless you need to immediately access the HTTP endpoints)
+fastify.listen()
 
-// Start the media server (no need to await in most cases, unless you need to immediately access the HTTP endpoints)
-manager.startMediaServer()
+// (optional) Create FastifyController instance for managing the starting and stopping the Fastify server (handles it more gracefully and allows pausing and restarting)
+// This is useful if you are working in a context that needs to pause or restart the server frequently.
+// e.g.
+//   const fastifyController = new FastifyController({ fastify })
+//   fastifyController.start()
 
 // Create a project
 const projectPublicId = await manager.createProject()
 
 // Get the project instance
 const project = await manager.getProject(projectPublicId)
+
+// Whenever you need to stop the HTTP server, use the controller
+await fastifyController.stop()
 ```
 
 The example code in the following sections assume that some variation of the above has been done already.
 
 ## Working with blobs
 
-Blobs represent any binary objects. In the case of Mapeo, that will most likely be media assets such as photos, videos, and audio files. Mapeo provides a project-scoped API that is used for creating and retrieving blobs. Combined with the media server, applications can access them using HTTP requests.
+Blobs represent any binary objects. In the case of Mapeo, that will most likely be media assets such as photos, videos, and audio files. Mapeo provides a project-scoped API that is used for creating and retrieving blobs. Combined with the HTTP server, applications can access them using HTTP requests.
 
 In the case of an observation record, there can be any number of references to "attachments" (in most cases, an image). In order to create these attachments we need to work with a project's blob API, which can be accessed using `project.$blobs`.
 
@@ -54,7 +66,7 @@ const observation = await project.observation.create({
 })
 ```
 
-The attachment provides the information that is needed to create a HTTP URL that can be used to access the asset from the media server:
+The attachment provides the information that is needed to create a HTTP URL that can be used to access the asset from the HTTP server:
 
 ```js
 // If you don't already have the observation record, you may need to retrieve it by doing the following
@@ -81,7 +93,7 @@ http://{HOST_NAME}:{PORT}/blobs/{PROJECT_PUBLIC_ID}/{DRIVE_DISCOVERY_ID}/{TYPE}/
 Explanation of the different parts of this URL:
 
 - `HOST_NAME`: Hostname of the server. Defaults to `127.0.0.1` (localhost)
-- `PORT`: Port that's being listened on. A random available port is used when the media server is started.
+- `PORT`: Port that's being listened on. A random available port is used when the HTTP server is started.
 - `PROJECT_PUBLIC_ID`: The public ID used to identify the project of interest.
 - `DRIVE_DISCOVERY_ID`: The discovery ID of the Hyperdrive instance where the blob of interest is located.
 - `TYPE`: The asset type. Can be `'photo'`, `'video'`, or `'audio'`.
@@ -104,7 +116,7 @@ You can then use this URL with anything that uses HTTP to fetch media. Some exam
 
 ## Working with icons
 
-Icons are primarily used in the context of project presets, where they are displayed as visual representations of a particular category when recording observations. Mapeo provides a project-scoped API for creating and retrieving icons. Combined with the media server, applications can access them using HTTP requests.
+Icons are primarily used in the context of project presets, where they are displayed as visual representations of a particular category when recording observations. Mapeo provides a project-scoped API for creating and retrieving icons. Combined with the HTTP server, applications can access them using HTTP requests.
 
 In order to create an icon we need to work with a project's icon API, which can be accessed using `project.$icons`:
 
@@ -183,7 +195,7 @@ http://{HOST_NAME}:{PORT}/icons/{PROJECT_PUBLIC_ID}/{ICON_ID}/{SIZE}{PIXEL_DENSI
 Explanation of the different parts of this URL:
 
 - `HOST_NAME`: Hostname of the server. Defaults to `127.0.0.1` (localhost)
-- `PORT`: Port that's being listened on. A random available port is used when the media server is started.
+- `PORT`: Port that's being listened on. A random available port is used when the HTTP server is started.
 - `PROJECT_PUBLIC_ID`: The public ID used to identify the project of interest.
 - `ICON_ID`: The ID of the icon record associated with the asset.
 - `SIZE`: The denoted size of the asset. Can be `'small'`, `'medium'`, or `'large'`.

src/fastify-controller.js

@@ -0,0 +1,84 @@
+import { promisify } from 'util'
+import StateMachine from 'start-stop-state-machine'
+
+/**
+ * @typedef {Object} StartOpts
+ *
+ * @property {string} [host]
+ * @property {number} [port]
+ */
+
+// Class to properly manage the server lifecycle of a Fastify instance
+export class FastifyController {
+  #fastify
+  #fastifyStarted
+  #serverState
+
+  /**
+   * @param {Object} opts
+   * @param {import('fastify').FastifyInstance} opts.fastify
+   */
+  constructor({ fastify }) {
+    this.#fastifyStarted = false
+
+    this.#fastify = fastify
+
+    this.#serverState = new StateMachine({
+      start: this.#startServer.bind(this),
+      stop: this.#stopServer.bind(this),
+    })
+  }
+
+  /**
+   * @param {StartOpts} [opts]
+   */
+  async #startServer({ host = '127.0.0.1', port = 0 } = {}) {
+    if (!this.#fastifyStarted) {
+      this.#fastifyStarted = true
+      await this.#fastify.listen({ host, port })
+      return
+    }
+
+    const { server } = this.#fastify
+
+    await new Promise((res, rej) => {
+      server.listen.call(server, { host, port })
+
+      server.once('listening', onListening)
+      server.once('error', onError)
+
+      function onListening() {
+        server.removeListener('error', onError)
+        res(null)
+      }
+
+      /**
+       * @param {Error} err
+       */
+      function onError(err) {
+        server.removeListener('listening', onListening)
+        rej(err)
+      }
+    })
+  }
+
+  async #stopServer() {
+    const { server } = this.#fastify
+    await promisify(server.close.bind(server))()
+  }
+
+  /**
+   * @param {StartOpts} [opts]
+   */
+  async start(opts) {
+    await this.#serverState.start(opts)
+  }
+
+  async started() {
+    return this.#serverState.started()
+  }
+
+  async stop() {
+    await this.#serverState.stop()
+  }
+}

src/fastify-plugins/utils.js

@@ -0,0 +1,32 @@
+import { once } from 'node:events'
+
+/**
+ * @param {import('node:http').Server} server
+ * @param {{ timeout?: number }} [options]
+ * @returns {Promise<string>}
+ */
+export async function getFastifyServerAddress(server, { timeout } = {}) {
+  const address = server.address()
+
+  if (!address) {
+    await once(server, 'listening', {
+      signal: timeout ? AbortSignal.timeout(timeout) : undefined,
+    })
+    return getFastifyServerAddress(server)
+  }
+
+  if (typeof address === 'string') {
+    return address
+  }
+
+  // Full address construction for non unix-socket address
+  // https://github.com/fastify/fastify/blob/7aa802ed224b91ca559edec469a6b903e89a7f88/lib/server.js#L413
+  let addr = ''
+  if (address.address.indexOf(':') === -1) {
+    addr += address.address + ':' + address.port
+  } else {
+    addr += '[' + address.address + ']:' + address.port
+  }
+
+  return 'http://' + addr
+}

src/index.js

@@ -1 +1,2 @@
+export { FastifyController } from './fastify-controller.js'
 export { MapeoManager } from './mapeo-manager.js'

src/mapeo-manager.js

@@ -11,6 +11,7 @@ import { TypedEmitter } from 'tiny-typed-emitter'
 import { IndexWriter } from './index-writer/index.js'
 import {
   MapeoProject,
+  kBlobStore,
   kProjectLeave,
   kSetOwnDeviceInfo,
 } from './mapeo-project.js'
@@ -30,9 +31,11 @@ import {
   projectKeyToPublicId,
 } from './utils.js'
 import { RandomAccessFilePool } from './core-manager/random-access-file-pool.js'
+import BlobServerPlugin from './fastify-plugins/blobs.js'
+import IconServerPlugin from './fastify-plugins/icons.js'
+import { getFastifyServerAddress } from './fastify-plugins/utils.js'
 import { LocalPeers } from './local-peers.js'
 import { InviteApi } from './invite-api.js'
-import { MediaServer } from './media-server.js'
 import { LocalDiscovery } from './discovery/local-discovery.js'
 import { Capabilities } from './capabilities.js'
 import NoiseSecretStream from '@hyperswarm/secret-stream'
@@ -50,6 +53,10 @@ const CLIENT_SQLITE_FILE_NAME = 'client.db'
 // other things e.g. SQLite and other parts of the app.
 const MAX_FILE_DESCRIPTORS = 768
 
+// Prefix names for routes registered with http server
+const BLOBS_PREFIX = 'blobs'
+const ICONS_PREFIX = 'icons'
+
 export const kRPC = Symbol('rpc')
 export const kManagerReplicate = Symbol('replicate manager')
 
@@ -80,7 +87,7 @@ export class MapeoManager extends TypedEmitter {
   #deviceId
   #localPeers
   #invite
-  #mediaServer
+  #fastify
   #localDiscovery
   #loggerBase
   #l
@@ -92,15 +99,15 @@ export class MapeoManager extends TypedEmitter {
    * @param {string} opts.projectMigrationsFolder path for drizzle migrations folder for project database
    * @param {string} opts.clientMigrationsFolder path for drizzle migrations folder for client database
    * @param {string | import('./types.js').CoreStorage} opts.coreStorage Folder for hypercore storage or a function that returns a RandomAccessStorage instance
-   * @param {{ port?: number, logger: import('fastify').FastifyServerOptions['logger'] }} [opts.mediaServerOpts]
+   * @param {import('fastify').FastifyInstance} opts.fastify Fastify server instance
    */
   constructor({
     rootKey,
     dbFolder,
     projectMigrationsFolder,
     clientMigrationsFolder,
     coreStorage,
-    mediaServerOpts,
+    fastify,
   }) {
     super()
     this.#keyManager = new KeyManager(rootKey)
@@ -160,8 +167,16 @@ export class MapeoManager extends TypedEmitter {
       this.#coreStorage = coreStorage
     }
 
-    this.#mediaServer = new MediaServer({
-      logger: mediaServerOpts?.logger,
+    this.#fastify = fastify
+    this.#fastify.register(BlobServerPlugin, {
+      prefix: BLOBS_PREFIX,
+      getBlobStore: async (projectPublicId) => {
+        const project = await this.getProject(projectPublicId)
+        return project[kBlobStore]
+      },
+    })
+    this.#fastify.register(IconServerPlugin, {
+      prefix: ICONS_PREFIX,
       getProject: this.getProject.bind(this),
     })
 
@@ -199,6 +214,35 @@ export class MapeoManager extends TypedEmitter {
     return this.#replicate(noiseStream)
   }
 
+  /**
+   * @param {'blobs' | 'icons' | 'maps'} mediaType
+   * @returns {Promise<string>}
+   */
+  async #getMediaBaseUrl(mediaType) {
+    /** @type {string | null} */
+    let prefix = null
+
+    switch (mediaType) {
+      case 'blobs': {
+        prefix = BLOBS_PREFIX
+        break
+      }
+      case 'icons': {
+        prefix = ICONS_PREFIX
+        break
+      }
+      default: {
+        throw new Error(`Unsupported media type ${mediaType}`)
+      }
+    }
+
+    const base = await getFastifyServerAddress(this.#fastify.server, {
+      timeout: 5000,
+    })
+
+    return base + '/' + prefix
+  }
+
   /**
    * @param {NoiseSecretStream<any>} noiseStream
    */
@@ -403,9 +447,7 @@ export class MapeoManager extends TypedEmitter {
       sharedIndexWriter: this.#projectSettingsIndexWriter,
       localPeers: this.#localPeers,
       logger: this.#loggerBase,
-      getMediaBaseUrl: this.#mediaServer.getMediaAddress.bind(
-        this.#mediaServer
-      ),
+      getMediaBaseUrl: this.#getMediaBaseUrl.bind(this),
     })
   }
 
@@ -654,17 +696,6 @@ export class MapeoManager extends TypedEmitter {
     return this.#invite
   }
 
-  /**
-   * @param {import('./media-server.js').StartOpts} [opts]
-   */
-  async startMediaServer(opts) {
-    await this.#mediaServer.start(opts)
-  }
-
-  async stopMediaServer() {
-    await this.#mediaServer.stop()
-  }
-
   async startLocalPeerDiscovery() {
     return this.#localDiscovery.start()
   }