Appview - serve feed skeletons (#1265)

* proxy timeline skeleton construction to appview * add getFeedSkeleton to appview * mount route * smart proxy feed skeletons * tests * proper error code * only proxy specific feeds * build branch * update proxyable feed logic, should use feed publisher rather than generator * fix feed proxy tests, configure feed publisher (in addition to generator) * hotfix: prevent user-supplied rkey on posts with createRecord (#1313) * prevent user-supplied rkey on posts with createRecord * allow empty-string rkey parameter Co-authored-by: devin ivy <[email protected]> --------- Co-authored-by: devin ivy <[email protected]> * add slurs to reserved words (#1318) * add slurs to reserved words (#1314) * Update reserved.ts Add slurs to reserved words * Update reserved.ts fix typo * Update reserved.ts to clean up the slur list * linting * pluralise --------- Co-authored-by: jess <[email protected]> * identifier: tweaks and additions to slur list (#1319) * Refactor appview repo subscription for memleak (#1308) * refactor to remove closure in loop * move consecutive item out of p-queue * Handle validation improvements (#1336) * Handle matches and false positives for unacceptable words in handles * move handle validation logic to pds * missed merge * add cfg flag * encode lists * fix build issues * move words to cfg * tidy --------- Co-authored-by: Jaz Volpert <[email protected]> * Allow moderators to take and reverse actor takedowns (#1330) allow moderators to take and reverse actor takedowns * ✨ Allow searching reports by moderator did (#1283) * ✨ Allow searching reports by moderator did * ✅ Remove .only flags on tests * ✅ Update snapshot * ✅ Add checks for did match in actions * v0.4.1 * Make sequencer leader behavior optional on pds (#1250) * make sequencer leader behavior optional on pds * tidy * use 127.0.0.1 in with-test-db.sh for colima (#1297) So, since Docker Desktop has licensing issues, some folks use colima for running containers on their macOS machines (The licensing exempted CLI-only version of Docker only exists on Linux). Unfortunately, colima binds host ports only on the IPv4 localhost address (`127.0.0.1`) while the atproto postgres clients will attempt to connect to the IPv6 localhost address (`::1`) that macOS sets in /etc/hosts. See abiosoft/colima#583 and lima-vm/lima#1330 for the tickets against colima. (Docker Desktop binds to both IPv4 and IPv6 localhost addresses and so doesn't have this issue.) To workaround this silly issue, we can use `localhost` within the docker containers and docker-compose, but need to set the `DB_POSTGRES_URL` env var to use the IPv4 localhost explicitly. (Asking folks to edit /etc/hosts causes other tools to break and will be overridden on each OS upgrade.) * Subscription util tests (#1295) * consecutive list tests * flesh out subscription util tests --------- Co-authored-by: Devin Ivy <[email protected]> * Content reporting on record fields (#1351) * content reporting on record fields * fix test * tests * tidy * Check rkey contents just for non-tids (#1353) check rkey content for non-tids * ✨ Added new procedure for sending admin email (#1312) * 🚧 Added new lexicon for sending admin email * ✨ Add moderation mailer * ✨ Switch to text email content from html * 🧹 Cleanup some early implementation code and reflect PR reivew * ✨ Use smtp host instead of gmail service config * ✨ Move to using single smtp url * v0.4.2 * Patch up a couple sqlite tests (#1355) patch up a couple sqlite tests * enable feeds & build branch * disable branch building & enable without proxy header --------- Co-authored-by: Devin Ivy <[email protected]> Co-authored-by: David Buchanan <[email protected]> Co-authored-by: jess <[email protected]> Co-authored-by: bnewbold <[email protected]> Co-authored-by: Jaz Volpert <[email protected]> Co-authored-by: Foysal Ahamed <[email protected]> Co-authored-by: Jeff Hodges <[email protected]>
bluesky-social · Jul 19, 2023 · 2836ee3 · 2836ee3
1 parent 8c32ed3
commit 2836ee3
Show file tree

Hide file tree

Showing 8 changed files with 913 additions and 10 deletions.
diff --git a/packages/bsky/src/api/app/bsky/feed/getFeedSkeleton.ts b/packages/bsky/src/api/app/bsky/feed/getFeedSkeleton.ts
@@ -0,0 +1,39 @@
+import { InvalidRequestError } from '@atproto/xrpc-server'
+import { Server } from '../../../../lexicon'
+import AppContext from '../../../../context'
+
+export default function (server: Server, ctx: AppContext) {
+  server.app.bsky.feed.getFeedSkeleton({
+    auth: ctx.authVerifierAnyAudience,
+    handler: async ({ params, auth }) => {
+      const { feed } = params
+      const viewer = auth.credentials.did
+      const localAlgo = ctx.algos[feed]
+
+      if (!localAlgo) {
+        throw new InvalidRequestError('Unknown feed', 'UnknownFeed')
+      }
+
+      const { cursor, feedItems } = await localAlgo(ctx, params, viewer)
+
+      const skeleton = feedItems.map((item) => ({
+        post: item.postUri,
+        reason:
+          item.uri === item.postUri
+            ? undefined
+            : {
+                $type: 'app.bsky.feed.defs#skeletonReasonRepost',
+                repost: item.uri,
+              },
+      }))
+
+      return {
+        encoding: 'application/json',
+        body: {
+          cursor,
+          feed: skeleton,
+        },
+      }
+    },
+  })
+}
diff --git a/packages/bsky/src/api/index.ts b/packages/bsky/src/api/index.ts
@@ -7,6 +7,7 @@ import getAuthorFeed from './app/bsky/feed/getAuthorFeed'
 import getFeed from './app/bsky/feed/getFeed'
 import getFeedGenerator from './app/bsky/feed/getFeedGenerator'
 import getFeedGenerators from './app/bsky/feed/getFeedGenerators'
+import getFeedSkeleton from './app/bsky/feed/getFeedSkeleton'
 import getLikes from './app/bsky/feed/getLikes'
 import getPostThread from './app/bsky/feed/getPostThread'
 import getPosts from './app/bsky/feed/getPosts'
@@ -59,6 +60,7 @@ export default function (server: Server, ctx: AppContext) {
   getFeed(server, ctx)
   getFeedGenerator(server, ctx)
   getFeedGenerators(server, ctx)
+  getFeedSkeleton(server, ctx)
   getLikes(server, ctx)
   getPostThread(server, ctx)
   getPosts(server, ctx)

diff --git a/packages/dev-env/src/pds.ts b/packages/dev-env/src/pds.ts
@@ -95,6 +95,7 @@ export class TestPds {
       repoSigningKey,
       plcRotationKey,
       config,
+      algos: cfg.algos,
     })
 
     await server.start()

diff --git a/packages/dev-env/src/types.ts b/packages/dev-env/src/types.ts
@@ -10,6 +10,7 @@ export type PdsConfig = Partial<pds.ServerConfig> & {
   plcUrl: string
   migration?: string
   enableInProcessAppView?: boolean
+  algos?: pds.MountedAlgos
 }
 
 export type BskyConfig = Partial<bsky.ServerConfig> & {

diff --git a/packages/pds/src/app-view/api/app/bsky/feed/getFeed.ts b/packages/pds/src/app-view/api/app/bsky/feed/getFeed.ts
@@ -11,7 +11,7 @@ import {
   PoorlyFormattedDidDocumentError,
   getFeedGen,
 } from '@atproto/identity'
-import { AtpAgent, AppBskyFeedGetFeedSkeleton } from '@atproto/api'
+import { AtpAgent, AppBskyFeedGetFeedSkeleton, AtUri } from '@atproto/api'
 import { SkeletonFeedPost } from '../../../../../lexicon/types/app/bsky/feed/defs'
 import { QueryParams as GetFeedParams } from '../../../../../lexicon/types/app/bsky/feed/getFeed'
 import { OutputSchema as SkeletonOutput } from '../../../../../lexicon/types/app/bsky/feed/getFeedSkeleton'
@@ -20,11 +20,26 @@ import AppContext from '../../../../../context'
 import { FeedRow } from '../../../../services/feed'
 import { AlgoResponse } from '../../../../../feed-gen/types'
 
+// temp hardcoded feeds that we can proxy to appview
+const PROXYABLE_FEEDS = [
+  'with-friends',
+  'bsky-team',
+  'hot-classic',
+  'best-of-follows',
+  'mutuals',
+]
+
 export default function (server: Server, ctx: AppContext) {
+  const isProxyableFeed = (feed: string): boolean => {
+    const uri = new AtUri(feed)
+    return feed in ctx.algos && PROXYABLE_FEEDS.includes(uri.rkey)
+  }
+
   server.app.bsky.feed.getFeed({
     auth: ctx.accessVerifier,
     handler: async ({ req, params, auth }) => {
       const requester = auth.credentials.did
+
       if (ctx.canProxyRead(req)) {
         const { data: feed } =
           await ctx.appviewAgent.api.app.bsky.feed.getFeedGenerator(
@@ -40,18 +55,35 @@ export default function (server: Server, ctx: AppContext) {
           body: res.data,
         }
       }
+      let algoRes: AlgoResponse
+      const timerSkele = new ServerTimer('skele').start()
 
-      const { feed } = params
-      const feedService = ctx.services.appView.feed(ctx.db)
-      const localAlgo = ctx.algos[feed]
+      if (ctx.cfg.bskyAppViewEndpoint && isProxyableFeed(params.feed)) {
+        // this is a temporary solution to smart proxy bsky feeds to the appview
+        const res = await ctx.appviewAgent.api.app.bsky.feed.getFeedSkeleton(
+          params,
+          await ctx.serviceAuthHeaders(requester),
+        )
+        algoRes = await filterMutesAndBlocks(
+          ctx,
+          res.data,
+          params.limit,
+          requester,
+        )
+      } else {
+        const { feed } = params
+        const localAlgo = ctx.algos[feed]
+        algoRes =
+          localAlgo !== undefined
+            ? await localAlgo(ctx, params, requester)
+            : await skeletonFromFeedGen(ctx, params, requester)
+      }
 
-      const timerSkele = new ServerTimer('skele').start()
-      const { feedItems, ...rest } =
-        localAlgo !== undefined
-          ? await localAlgo(ctx, params, requester)
-          : await skeletonFromFeedGen(ctx, params, requester)
       timerSkele.stop()
 
+      const feedService = ctx.services.appView.feed(ctx.db)
+      const { feedItems, ...rest } = algoRes
+
       const timerHydr = new ServerTimer('hydr').start()
       const hydrated = await feedService.hydrateFeed(feedItems, requester)
       timerHydr.stop()

diff --git a/packages/pds/src/index.ts b/packages/pds/src/index.ts
@@ -46,6 +46,7 @@ import { LabelCache } from './label-cache'
 import { ContentReporter } from './content-reporter'
 import { ModerationService } from './services/moderation'
 
+export type { MountedAlgos } from './feed-gen/types'
 export type { ServerConfigValues } from './config'
 export { ServerConfig } from './config'
 export { Database } from './db'