Threadsが動画でとても長いファイルURLを連合してくるので投稿を保存できない #14292

tai-cha · 2024-07-22T20:10:49Z

💡 Summary

Threadsの動画付き投稿に照会をかけたとき、正常に取得できない

(Threadsの動画URLがとても長い、画像では普通)

処理のログ

2024-07-22T19:50:57.808631778Z ERR  *   [api]   Internal error occurred in ap/show: value too long for type character varying(512) {
2024-07-22T19:50:57.808643725Z   ep: 'ap/show',
2024-07-22T19:50:57.808733669Z   ps: {
2024-07-22T19:50:57.808748371Z     uri: 'https://threads.net/ap/users/17841465704449480/post/17877931239051219/',
2024-07-22T19:50:57.808811925Z     i: '***masked***'
2024-07-22T19:50:57.808818008Z   },
2024-07-22T19:50:57.808820807Z   e: {
2024-07-22T19:50:57.808901990Z     message: 'value too long for type character varying(512)',
2024-07-22T19:50:57.808908128Z     code: 'QueryFailedError',
2024-07-22T19:50:57.808969938Z     stack: 'QueryFailedError: value too long for type character varying(512)\n' +
2024-07-22T19:50:57.808976147Z       '    at PostgresQueryRunner.query (/misskey/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)\n' +
2024-07-22T19:50:57.809049280Z       '    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n' +
2024-07-22T19:50:57.809055434Z       '    at async InsertQueryBuilder.execute (/misskey/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)\n' +
2024-07-22T19:50:57.809058606Z       '    at async DriveService.addFile (file:///misskey/packages/backend/built/core/DriveService.js:501:24)\n' +
2024-07-22T19:50:57.809125290Z       '    at async DriveService.uploadFromUrl (file:///misskey/packages/backend/built/core/DriveService.js:701:31)\n' +
2024-07-22T19:50:57.809131365Z       '    at async ApImageService.createImage (file:///misskey/packages/backend/built/core/activitypub/models/ApImageService.js:68:22)\n' +
2024-07-22T19:50:57.809200208Z       '    at async ApImageService.resolveImage (file:///misskey/packages/backend/built/core/activitypub/models/ApImageService.js:95:16)\n' +
2024-07-22T19:50:57.809206340Z       '    at async ApNoteService.createNote (file:///misskey/packages/backend/built/core/activitypub/models/ApNoteService.js:193:26)\n' +
2024-07-22T19:50:57.809209348Z       '    at async _class.fetchAny (file:///misskey/packages/backend/built/server/api/endpoints/ap/show.js:150:141)\n' +
2024-07-22T19:50:57.809285627Z       '    at async file:///misskey/packages/backend/built/server/api/endpoints/ap/show.js:111:28\n' +
2024-07-22T19:50:57.809361242Z       '    at async ApiCallService.call (file:///misskey/packages/backend/built/server/api/ApiCallService.js:364:20)',
2024-07-22T19:50:57.809367305Z     id: '4085fa9e-7356-4cfb-977a-ce989c921ee8'
2024-07-22T19:50:57.809370324Z   }
2024-07-22T19:50:57.809431248Z }
2024-07-22T19:50:57.809893423Z QueryFailedError: value too long for type character varying(512)
2024-07-22T19:50:57.809904840Z     at PostgresQueryRunner.query (/misskey/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
2024-07-22T19:50:57.809972772Z     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-07-22T19:50:57.809985838Z     at async InsertQueryBuilder.execute (/misskey/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
2024-07-22T19:50:57.810061691Z     at async DriveService.addFile (file:///misskey/packages/backend/built/core/DriveService.js:501:24)
2024-07-22T19:50:57.810068008Z     at async DriveService.uploadFromUrl (file:///misskey/packages/backend/built/core/DriveService.js:701:31)
2024-07-22T19:50:57.810130011Z     at async ApImageService.createImage (file:///misskey/packages/backend/built/core/activitypub/models/ApImageService.js:68:22)
2024-07-22T19:50:57.810136336Z     at async ApImageService.resolveImage (file:///misskey/packages/backend/built/core/activitypub/models/ApImageService.js:95:16)
2024-07-22T19:50:57.810210589Z     at async ApNoteService.createNote (file:///misskey/packages/backend/built/core/activitypub/models/ApNoteService.js:193:26)2024-07-22T19:50:57.810216952Z     at async _class.fetchAny (file:///misskey/packages/backend/built/server/api/endpoints/ap/show.js:150:141)
2024-07-22T19:50:57.810220186Z     at async file:///misskey/packages/backend/built/server/api/endpoints/ap/show.js:111:28
2024-07-22T19:50:57.810282230Z     at async ApiCallService.call (file:///misskey/packages/backend/built/server/api/ApiCallService.js:364:20) {
2024-07-22T19:50:57.810360871Z   query: 'INSERT INTO "drive_file"("id", "userId", "userHost", "md5", "name", "type", "size", "comment", "blurhash", "properties", "storedInternal", "url", "thumbnailUrl", "webpublicUrl", "webpublicType", "accessKey", "thumbnailAccessKey", "webpublicAccessKey", "uri", "src", "folderId", "isSensitive", "maybeSensitive", "maybePorn", "isLink", "requestHeaders", "requestIp") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, DEFAULT, DEFAULT, DEFAULT, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24) RETURNING "properties", "isSensitive", "maybeSensitive", "maybePorn", "isLink", "requestHeaders"',
2024-07-22T19:50:57.810369487Z   parameters: [
2024-07-22T19:50:57.810431822Z     '9w0yuuiw1m',
2024-07-22T19:50:57.810437743Z     '9uyq36pdl3',
2024-07-22T19:50:57.810506976Z     'threads.net',
2024-07-22T19:50:57.810513006Z     'ced6a727163eb3a27ec4b55a447329bd',
2024-07-22T19:50:57.810515962Z     'An-p7dgy80SMO2BjpA5qIsyk_eZAghX-2w9lWQpltcT5DROtAXAZAqzTNhLHrcZ6_QQ7DH-f6mYlbkokeLqYi0dM.mp4',
2024-07-22T19:50:57.810578783Z     'video/mp4',
2024-07-22T19:50:57.810584650Z     0,
2024-07-22T19:50:57.810667111Z     null,
2024-07-22T19:50:57.810672993Z     null,
2024-07-22T19:50:57.810688782Z     '{}',
2024-07-22T19:50:57.810763981Z     false,
2024-07-22T19:50:57.810769961Z     'https://scontent-nrt1-2.cdninstagram.com/o1/v/t16/f2/m69/An-p7dgy80SMO2BjpA5qIsyk_eZAghX-2w9lWQpltcT5DROtAXAZAqzTNhLHrcZ6_QQ7DH-f6mYlbkokeLqYi0dM?_nc_ht=scontent-nrt1-2.cdninstagram.com&_nc_cat=107&vs=1139900233980148_4074901013&_nc_vs=HBksFQIYOnBhc3N0aHJvdWdoX2V2ZXJzdG9yZS9HQkdSUUFOVmdVZzRqVElFQU04Y05IUnJwQU1FYnBSMUFBQUYVAALIAQAVAhg6cGFzc3Rocm91Z2hfZXZlcnN0b3JlL0dFbXY5aHJXZEZLT2tzb0RBREd2OFZlUEpGSVBia1lMQUFBRhUCAsgBACgAGAAbAYgHdXNlX29pbAExFQAAJr7oqYr09IBAFQIoAkMzLBdAKjEm6XjU_hgWZGFzaF9iYXNlbGluZV8xMDgwcF92MREAdeoHAA&ccb=9-4&oh=00_AYDhbIuQqQJ2NxzM35uKjBoYCpVU1nLMKW4XyGA_JlRJBQ&oe=66A0ACA2&_nc_sid=1d576d',
2024-07-22T19:50:57.810774379Z     'ce3a05aa-3c9f-4b33-b138-218c9fca8321',
2024-07-22T19:50:57.810777120Z     'thumbnail-75a788e8-a953-4864-86ad-63d15f63c9a7',
2024-07-22T19:50:57.810855615Z     'webpublic-7d0b9b95-57ba-42e3-a8e6-dc5b24d2b994',
2024-07-22T19:50:57.810861574Z     'https://scontent-nrt1-2.cdninstagram.com/o1/v/t16/f2/m69/An-p7dgy80SMO2BjpA5qIsyk_eZAghX-2w9lWQpltcT5DROtAXAZAqzTNhLHrcZ6_QQ7DH-f6mYlbkokeLqYi0dM?_nc_ht=scontent-nrt1-2.cdninstagram.com&_nc_cat=107&vs=1139900233980148_4074901013&_nc_vs=HBksFQIYOnBhc3N0aHJvdWdoX2V2ZXJzdG9yZS9HQkdSUUFOVmdVZzRqVElFQU04Y05IUnJwQU1FYnBSMUFBQUYVAALIAQAVAhg6cGFzc3Rocm91Z2hfZXZlcnN0b3JlL0dFbXY5aHJXZEZLT2tzb0RBREd2OFZlUEpGSVBia1lMQUFBRhUCAsgBACgAGAAbAYgHdXNlX29pbAExFQAAJr7oqYr09IBAFQIoAkMzLBdAKjEm6XjU_hgWZGFzaF9iYXNlbGluZV8xMDgwcF92MREAdeoHAA&ccb=9-4&oh=00_AYDhbIuQqQJ2NxzM35uKjBoYCpVU1nLMKW4XyGA_JlRJBQ&oe=66A0ACA2&_nc_sid=1d576d',
2024-07-22T19:50:57.810924471Z     'https://scontent-nrt1-2.cdninstagram.com/o1/v/t16/f2/m69/An-p7dgy80SMO2BjpA5qIsyk_eZAghX-2w9lWQpltcT5DROtAXAZAqzTNhLHrcZ6_QQ7DH-f6mYlbkokeLqYi0dM?_nc_ht=scontent-nrt1-2.cdninstagram.com&_nc_cat=107&vs=1139900233980148_4074901013&_nc_vs=HBksFQIYOnBhc3N0aHJvdWdoX2V2ZXJzdG9yZS9HQkdSUUFOVmdVZzRqVElFQU04Y05IUnJwQU1FYnBSMUFBQUYVAALIAQAVAhg6cGFzc3Rocm91Z2hfZXZlcnN0b3JlL0dFbXY5aHJXZEZLT2tzb0RBREd2OFZlUEpGSVBia1lMQUFBRhUCAsgBACgAGAAbAYgHdXNlX29pbAExFQAAJr7oqYr09IBAFQIoAkMzLBdAKjEm6XjU_hgWZGFzaF9iYXNlbGluZV8xMDgwcF92MREAdeoHAA&ccb=9-4&oh=00_AYDhbIuQqQJ2NxzM35uKjBoYCpVU1nLMKW4XyGA_JlRJBQ&oe=66A0ACA2&_nc_sid=1d576d',
2024-07-22T19:50:57.810993959Z     null,
2024-07-22T19:50:57.810999851Z     false,
2024-07-22T19:50:57.811060655Z     false,
2024-07-22T19:50:57.811065711Z     false,
2024-07-22T19:50:57.811068562Z     true,
2024-07-22T19:50:57.811335898Z     null,
2024-07-22T19:50:57.811349715Z     null
2024-07-22T19:50:57.811352926Z   ],
2024-07-22T19:50:57.811355870Z   driverError: error: value too long for type character varying(512)
2024-07-22T19:50:57.811358922Z       at /misskey/node_modules/.pnpm/[email protected]/node_modules/pg/lib/client.js:526:17
2024-07-22T19:50:57.811362183Z       at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-07-22T19:50:57.811373779Z       at async PostgresQueryRunner.query (/misskey/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)
2024-07-22T19:50:57.811376939Z       at async InsertQueryBuilder.execute (/misskey/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33)
2024-07-22T19:50:57.811379846Z       at async DriveService.addFile (file:///misskey/packages/backend/built/core/DriveService.js:501:24)
2024-07-22T19:50:57.811382708Z       at async DriveService.uploadFromUrl (file:///misskey/packages/backend/built/core/DriveService.js:701:31)
2024-07-22T19:50:57.811385432Z       at async ApImageService.createImage (file:///misskey/packages/backend/built/core/activitypub/models/ApImageService.js:68:22)
2024-07-22T19:50:57.811388221Z       at async ApImageService.resolveImage (file:///misskey/packages/backend/built/core/activitypub/models/ApImageService.js:95:16)
2024-07-22T19:50:57.811407126Z       at async ApNoteService.createNote (file:///misskey/packages/backend/built/core/activitypub/models/ApNoteService.js:193:26)
2024-07-22T19:50:57.811410071Z       at async _class.fetchAny (file:///misskey/packages/backend/built/server/api/endpoints/ap/show.js:150:141)
2024-07-22T19:50:57.811412926Z       at async file:///misskey/packages/backend/built/server/api/endpoints/ap/show.js:111:28
2024-07-22T19:50:57.811415782Z       at async ApiCallService.call (file:///misskey/packages/backend/built/server/api/ApiCallService.js:364:20) {
2024-07-22T19:50:57.811418631Z     length: 99,
2024-07-22T19:50:57.811421279Z     severity: 'ERROR',
2024-07-22T19:50:57.811423911Z     code: '22001',
2024-07-22T19:50:57.811426464Z     detail: undefined,
2024-07-22T19:50:57.811429007Z     hint: undefined,
2024-07-22T19:50:57.811431598Z     position: undefined,
2024-07-22T19:50:57.811434126Z     internalPosition: undefined,
2024-07-22T19:50:57.811437268Z     internalQuery: undefined,
2024-07-22T19:50:57.811439994Z     where: undefined,
2024-07-22T19:50:57.811442668Z     schema: undefined,
2024-07-22T19:50:57.811445273Z     table: undefined,
2024-07-22T19:50:57.811447818Z     column: undefined,
2024-07-22T19:50:57.811450362Z     dataType: undefined,
2024-07-22T19:50:57.811452862Z     constraint: undefined,
2024-07-22T19:50:57.811458984Z     file: 'varchar.c',
2024-07-22T19:50:57.811461757Z     line: '632',
2024-07-22T19:50:57.811473141Z     routine: 'varchar'
2024-07-22T19:50:57.811476140Z   },
2024-07-22T19:50:57.811478824Z   length: 99,
2024-07-22T19:50:57.811481508Z   severity: 'ERROR',
2024-07-22T19:50:57.811484264Z   code: '22001',
2024-07-22T19:50:57.811486958Z   detail: undefined,
2024-07-22T19:50:57.811489531Z   hint: undefined,
2024-07-22T19:50:57.811492143Z   position: undefined,
2024-07-22T19:50:57.811494639Z   internalPosition: undefined,
2024-07-22T19:50:57.811497202Z   internalQuery: undefined,
2024-07-22T19:50:57.811499735Z   where: undefined,
2024-07-22T19:50:57.811502259Z   schema: undefined,
2024-07-22T19:50:57.811504947Z   table: undefined,
2024-07-22T19:50:57.811507538Z   column: undefined,
2024-07-22T19:50:57.811510109Z   dataType: undefined,
2024-07-22T19:50:57.811512726Z   constraint: undefined,
2024-07-22T19:50:57.811515203Z   file: 'varchar.c',
2024-07-22T19:50:57.811517725Z   line: '632',
2024-07-22T19:50:57.811520454Z   routine: 'varchar'
2024-07-22T19:50:57.811523291Z } 4085fa9e-7356-4cfb-977a-ce989c921ee8

🥰 Expected Behavior

Threadsの動画付き投稿を受け取ったり照会したとき、正常に投稿を取得できる

🤬 Actual Behavior

Threadsの動画付き投稿に照会をかけたとき、正常に取得できない

📝 Steps to Reproduce

https://threads.net/ap/users/17841465704449480/post/17877931239051219/を照会する (SignedGet必須)（元投稿 https://www.threads.net/@tainyanpu/post/C9u54UJy1VC ）
投稿が取得不可能

💻 Frontend Environment

* Model and OS of the device(s):
* Browser:
* Server URL: https://mi.taichan.site
* Misskey: 2024.5.0

🛰 Backend Environment (for server admin)

* Installation Method or Hosting Service: docker with "Misskey install shell script"
* Misskey: 2024.5.0
* Node:
* PostgreSQL:
* Redis:
* OS and Architecture:

Do you want to address this bug yourself?

Yes, I will patch the bug myself and send a pull request

The text was updated successfully, but these errors were encountered:

tai-cha · 2024-07-22T20:13:59Z

あくまでURLの長さに対してDBが足りないだけなのでfederationタグはつけていない

anatawa12 · 2024-07-23T02:37:16Z

対応するために長さを長くするとしてどこまで長くするかという議論になると思いますが、一つの参考文献として思いついた HTTP Semantics には 8000 octetsが示されてます。
HTTP Semanticsは実際に送受信するときのお話ですし、リソースのIDとして以外の情報が含まれているために長く設定されてるようなので、流石にリソース用のURLとしては長過ぎるような気もするのでもっと短くちょうど良さそうな文献がほしいとは思います。

https://www.rfc-editor.org/rfc/rfc9110.html#section-4.1

It is RECOMMENDED that all senders and recipients support, at a minimum, URIs with lengths of 8000 octets in protocol elements. Note that this implies some structures and on-wire representations (for example, the request line in HTTP/1.1) will necessarily be larger in some cases.

kakkokari-gtyih · 2024-07-23T04:58:08Z

https://httpd.apache.org/docs/current/ja/mod/core.html#limitrequestline Apache: 8190バイト
https://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers Nginx: 8000バイト
- https://nginx.org/en/docs/http/ngx_http_core_module.html#client_header_buffer_size Nginx: 1000バイト

tai-cha · 2024-07-24T13:33:25Z

実用上は2048も長さがあれば十分な気がする（要出典）（TEXT型を採用して長さの制限自体をしないということも考えられなくもないけどそこまでする必要があまりないと思う）

anatawa12 · 2024-07-24T14:24:24Z

Mastodonはcharacter varyingでこれはpostgres環境では長さ制限がないそうです

https://github.com/mastodon/mastodon/blob/06f070d86d448b97c082c038220becaec8a038ce/db/migrate/20160905150353_create_media_attachments.rb#L14
https://github.com/rails/rails/blob/72741db81e26a2e8c6dd63ca3bfa98aff6ab961b/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb#L136

tai-cha · 2024-07-27T06:11:45Z

これ、しゅいろさんの意見さえ聞いたらすぐ定義書き換えで対応できる気もする
migration書くだけな気がするので

syuilo · 2024-07-27T06:28:32Z

具体的にどれくらいの長さかというのは分かるかしら

anatawa12 · 2024-07-27T06:44:39Z

Secineralyr#70 (comment) 曰く 739?

syuilo · 2024-07-27T09:50:58Z

1024にするかしら

tai-cha · 2024-07-30T00:23:50Z

1024, ok

tai-cha added the 🐛Bug Unexpected behavior label Jul 22, 2024

tai-cha changed the title ~~Threadsがとても長いファイルURLを連合してくる~~ Threadsが動画でとても長いファイルURLを連合してくる Jul 22, 2024

tai-cha added the packages/backend Server side specific issue/PR label Jul 22, 2024

tai-cha changed the title ~~Threadsが動画でとても長いファイルURLを連合してくる~~ Threadsが動画でとても長いファイルURLを連合してくるので投稿を保存できない Jul 22, 2024

tai-cha mentioned this issue Jul 22, 2024

Threadsのリソースurlが長すぎて入らない Secineralyr/misskey.dream#70

Closed

Sayamame-beans mentioned this issue Jul 22, 2024

URLからアップロードができないファイルがある #14216

Open

1 task

tai-cha self-assigned this Jul 30, 2024

tai-cha mentioned this issue Jul 30, 2024

Fix(backend): ドライブのファイルのurl, uri, src の上限引き上げ #14323

Merged

5 tasks

syuilo closed this as completed in #14323 Jul 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Threadsが動画でとても長いファイルURLを連合してくるので投稿を保存できない #14292

Threadsが動画でとても長いファイルURLを連合してくるので投稿を保存できない #14292

tai-cha commented Jul 22, 2024 •

edited

Loading

tai-cha commented Jul 22, 2024

anatawa12 commented Jul 23, 2024 •

edited

Loading

kakkokari-gtyih commented Jul 23, 2024 •

edited

Loading

tai-cha commented Jul 24, 2024 •

edited

Loading

anatawa12 commented Jul 24, 2024

tai-cha commented Jul 27, 2024 •

edited

Loading

syuilo commented Jul 27, 2024

anatawa12 commented Jul 27, 2024

syuilo commented Jul 27, 2024

tai-cha commented Jul 30, 2024

Threadsが動画でとても長いファイルURLを連合してくるので投稿を保存できない #14292

Threadsが動画でとても長いファイルURLを連合してくるので投稿を保存できない #14292

Comments

tai-cha commented Jul 22, 2024 • edited Loading

💡 Summary

🥰 Expected Behavior

🤬 Actual Behavior

📝 Steps to Reproduce

💻 Frontend Environment

🛰 Backend Environment (for server admin)

Do you want to address this bug yourself?

tai-cha commented Jul 22, 2024

anatawa12 commented Jul 23, 2024 • edited Loading

kakkokari-gtyih commented Jul 23, 2024 • edited Loading

tai-cha commented Jul 24, 2024 • edited Loading

anatawa12 commented Jul 24, 2024

tai-cha commented Jul 27, 2024 • edited Loading

syuilo commented Jul 27, 2024

anatawa12 commented Jul 27, 2024

syuilo commented Jul 27, 2024

tai-cha commented Jul 30, 2024

tai-cha commented Jul 22, 2024 •

edited

Loading

anatawa12 commented Jul 23, 2024 •

edited

Loading

kakkokari-gtyih commented Jul 23, 2024 •

edited

Loading

tai-cha commented Jul 24, 2024 •

edited

Loading

tai-cha commented Jul 27, 2024 •

edited

Loading