504 gateway timeout error while accessing tags, categories in discussion forum

[sravani-64]

Hi team,
We have setup discussion forum in DEV environment running on 4.10 version,
We are not able to access categories and tags after enabling discussions

After checking network tab,
Request URL:https://{{host}}/discussion/forum/tags
Request Method:POST
Status Code:504 Gateway Timeout
We have checked nodebb logs and found below information

SB:Category read
SB:Category read

<--- Last few GCs --->

[97:0x5da1590] 334500 ms: Mark-sweep 506.4 (521.3) -> 502.5 (521.5) MB, 772.2 / 0.0 ms (average mu = 0.279, current mu = 0.238) allocation failure; scavenge might not succeed
[97:0x5da1590] 335507 ms: Mark-sweep 506.6 (521.5) -> 502.8 (521.8) MB, 711.8 / 0.0 ms (average mu = 0.286, current mu = 0.293) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xb7b3e0 node::Abort() [/usr/local/bin/node]
2: 0xa8c8aa [/usr/local/bin/node]
3: 0xd69100 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
4: 0xd694a7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
5: 0xf46ba5 [/usr/local/bin/node]
6: 0xf47aa8 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/usr/local/bin/node]
7: 0xf57fb3 [/usr/local/bin/node]
8: 0xf58e28 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
9: 0xf3378e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0xf34b57 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
11: 0xf15d2a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/bin/node]
12: 0x12dacdf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
13: 0x1707b79 [/usr/local/bin/node]
[cluster] Child Process (97) has exited (code: null, signal: SIGABRT)
[cluster] Spinning up another process...
2023-07-03T07:13:13.300Z [4567/108] - info: Initializing NodeBB v2.8.10 http://0.0.0.0:4567/discussions/
(node:108) [LRU_CACHE_UNBOUNDED] UnboundedCacheWarning: TTL caching without ttlAutopurge, max, or maxSize can result in unbounded memory consumption.
(Use node --trace-warnings ... to show where the warning was created)
2023-07-03T07:13:14.747Z [4567/108] - info: [socket.io] Restricting access to origin: http://0.0.0.0:*
2023-07-03T07:13:14.768Z [4567/108] - warn: [plugins] "@appyhigh/nodebb-plugin-session-sharing-appyhigh" is active but not installed.
2023-07-03T07:13:14.768Z [4567/108] - warn: [plugins] "@dasmeta/nodebb-plugin-upload" is active but not installed.
2023-07-03T07:13:14.768Z [4567/108] - warn: [plugins] "@ktr41n/nodebb-plugin-emailer-mailjet" is active but not installed.
2023-07-03T07:13:14.769Z [4567/108] - warn: [plugins] "@nodebb-community/nodebb-plugin-s3-uploads-updated" is active but not installed.
2023-07-03T07:13:14.769Z [4567/108] - warn: [plugins] "nodebb-plugin-attendance" is active but not installed.
2023-07-03T07:13:14.769Z [4567/108] - warn: [plugins] "@nodebb/nodebb-plugin-cdn" is active but not installed.
2023-07-03T07:13:14.769Z [4567/108] - warn: [plugins] "@nodebb/nodebb-plugin-node-ldap" is active but not installed.
2023-07-03T07:13:14.769Z [4567/108] - warn: [plugins] "@nodebb/nodebb-plugin-topic-ratings" is active but not installed.
2023-07-03T07:13:14.769Z [4567/108] - warn: [plugins] "@phits/nodebb-plugin-discord-notification" is active but not installed.
2023-07-03T07:13:14.769Z [4567/108] - warn: [plugins] "nodebb-plugin-adsense" is active but not installed.
2023-07-03T07:13:14.769Z [4567/108] - warn: [plugins] "nodebb-plugin-archiver" is active but not installed.
2023-07-03T07:13:14.769Z [4567/108] - warn: [plugins] "nodebb-plugin-arma3-slotting" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-assetsdomain" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-audio-embed" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-beep" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-calendar" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-canned-responses" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-category-carousel" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-category-join-group" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-category-notifications" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-category-sort-by-title" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-category-tags" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-categoryheader" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-composer-quill" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-contact-page" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-customize" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-email-blacklist" is active but not installed.
2023-07-03T07:13:14.770Z [4567/108] - warn: [plugins] "nodebb-plugin-emailer" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-emailer-sendgrid" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-embed-gmap" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-emoji-apple" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-emoji-cubicopp" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-emoji-fontawesome" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-emoji-one" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-emoji-twemoji-v2" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-emoji-vital" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-extended-markdown" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-featured-topics-extended" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-github-embed" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-glossary" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-google-analytics" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-gravatar" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-header-extend" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-href2video" is active but not installed.
2023-07-03T07:13:14.771Z [4567/108] - warn: [plugins] "nodebb-plugin-href2video-bilibili" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-href2video-local" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-html5videoplayer" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-imagemagick" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-imgbed" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-math" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-medals" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-meilisearch" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-mermaid" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-minecraft-integration" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-newsletter" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-osm-map" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-poll" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-post-placeholder" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-pushbullet" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-question-and-answer" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-random-topics" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-restrict-usernames" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-sanitizeconfig" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-sanitizehtml" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-schema" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-sensitiveword" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-seo" is active but not installed.
2023-07-03T07:13:14.772Z [4567/108] - warn: [plugins] "nodebb-plugin-session-sharing" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-session-sharing-lit-protocol" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-share-bar" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-smsverification" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-social-poster" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-solr" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-sso-auth0" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-sso-ethereum-wallet" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-sso-facebook" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-sso-google" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-sso-oauth-faforever" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-sso-patreon" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-sso-qq-fix" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-sso-twitter" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-sso-wordpress" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-support-forum" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-svg-captcha" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-tenor-gif" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-title-bypass" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-topic-move-notification" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-topic-redirect" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-two-way-block" is active but not installed.
2023-07-03T07:13:14.773Z [4567/108] - warn: [plugins] "nodebb-plugin-web3-tip" is active but not installed.
2023-07-03T07:13:14.774Z [4567/108] - warn: [plugins] "nodebb-plugin-ws-dice" is active but not installed.
2023-07-03T07:13:14.774Z [4567/108] - warn: [plugins] "nodebb-plugin-ws-extended-markdown" is active but not installed.
2023-07-03T07:13:14.774Z [4567/108] - warn: [plugins] "nodebb-plugin-ws-markdown" is active but not installed.
2023-07-03T07:13:14.774Z [4567/108] - warn: [plugins] "nodebb-widget-discord-embed" is active but not installed.
2023-07-03T07:13:15.127Z [4567/108] - warn: [plugins/nodebb-plugin-azurestorage] The plugin.json field "library" is deprecated. Please use the package.json field "main" instead.
2023-07-03T07:13:15.270Z [4567/108] - warn: [plugins] Unable to load library for: nodebb-plugin-azurestorage
2023-07-03T07:13:15.271Z [4567/108] - error: Error: Cannot find module './meta'
Require stack:

• /usr/src/app/src/plugins/index.js
• /usr/src/app/src/meta/configs.js
• /usr/src/app/src/meta/index.js
• /usr/src/app/src/start.js
• /usr/src/app/require-main.js
• /usr/src/app/app.js
  at Module._resolveFilename (node:internal/modules/cjs/loader:1075:15)
  at Module._load (node:internal/modules/cjs/loader:920:27)
  at Module.require (node:internal/modules/cjs/loader:1141:19)
  at Object. (/usr/src/app/node_modules/nodebb-plugin-azurestorage/library.js:14:23)
  at Module._compile (node:internal/modules/cjs/loader:1254:14)
  at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
  at Module.load (node:internal/modules/cjs/loader:1117:32)
  at Module._load (node:internal/modules/cjs/loader:958:12)
  at Module.require (node:internal/modules/cjs/loader:1141:19)
  at require (node:internal/modules/cjs/helpers:110:18)
  nodebb plugin sunbird oidc working
  2023-07-03T07:13:15.424Z [4567/108] - warn: [plugins] File not found: /usr/src/app/node_modules/nodebb-plugin-create-forum/public/js/admin.js (Ignoring)
  2023-07-03T07:13:15.940Z [4567/108] - warn: [plugins/nodebb-plugin-telemetry-plugin] The plugin.json field "library" is deprecated. Please use the package.json field "main" instead.
  2023-07-03T07:13:15.941Z [4567/108] - warn: [plugins] Unable to load library for: nodebb-plugin-telemetry-plugin
  2023-07-03T07:13:15.941Z [4567/108] - error: Error: Cannot find module './user'
  Require stack:
• /usr/src/app/src/plugins/index.js
• /usr/src/app/src/meta/configs.js
• /usr/src/app/src/meta/index.js
• /usr/src/app/src/start.js
• /usr/src/app/require-main.js
• /usr/src/app/app.js
  at Module._resolveFilename (node:internal/modules/cjs/loader:1075:15)
  at Module._load (node:internal/modules/cjs/loader:920:27)
  at Module.require (node:internal/modules/cjs/loader:1141:19)
  at Object. (/usr/src/app/node_modules/nodebb-plugin-telemetry-plugin/library.js:1:26)
  at Module._compile (node:internal/modules/cjs/loader:1254:14)
  at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
  at Module.load (node:internal/modules/cjs/loader:1117:32)
  at Module._load (node:internal/modules/cjs/loader:958:12)
  at Module.require (node:internal/modules/cjs/loader:1141:19)
  at require (node:internal/modules/cjs/helpers:110:18)
  at Plugins.requireLibrary (/usr/src/app/src/plugins/index.js:63:39)
  at registerHooks (/usr/src/app/src/plugins/load.js:160:13)
  at Plugins.loadPlugin (/usr/src/app/src/plugins/load.js:123:4)
  at async Plugins.reload (/usr/src/app/src/plugins/index.js:113:3)
  at async Plugins.init (/usr/src/app/src/plugins/index.js:86:2)
  at async initializeNodeBB (/usr/src/app/src/webserver.js:101:2)
  at async exports.listen (/usr/src/app/src/webserver.js:85:2)
  at async start.start (/usr/src/app/src/start.js:45:3)

2023-07-03T07:13:15.944Z [4567/108] - warn: [plugins/load] The following plugins may not be compatible with your version of NodeBB. This may cause unintended behaviour or crashing. In the event of an unresponsive NodeBB caused by this plugin, run ./nodebb reset -p PLUGINNAME to disable it.

• nodebb-rewards-essentials
• nodebb-plugin-azurestorage
• nodebb-plugin-create-forum
• nodebb-plugin-dbsearch
• nodebb-plugin-sunbird-oidc
• nodebb-plugin-telemetry-plugin

2023-07-03T07:13:15.945Z [4567/108] - warn: [plugins/load] DEPRECATION The hook filter:router.page has been deprecated as of v1.15.3, and slated for removal in v2.1.0. Please use response:router.page instead. The following plugins are still listening for this hook:

• nodebb-plugin-write-api
  2023-07-03T07:13:15.957Z [4567/108] - warn: [helpers.setupAdminPageRoute(/admin/plugins/2factor)] passing middleware as the third param is deprecated, it can now be safely removed
  2023-07-03T07:13:15.958Z [4567/108] - warn: [helpers.setupPageRoute(/user/:userslug/2factor)] passing middleware as the third param is deprecated, it can now be safely removed
  2023-07-03T07:13:15.958Z [4567/108] - warn: [helpers.setupPageRoute(/login/2fa)] passing middleware as the third param is deprecated, it can now be safely removed
  2023-07-03T07:13:15.958Z [4567/108] - warn: [helpers.setupPageRoute(/login/2fa/backup)] passing middleware as the third param is deprecated, it can now be safely removed
  Redis db connection {"url":"http://0.0.0.0:4567/discussions/","secret":"","database":"redis","port":"4567","redis":{"host":"","port":"6379","password":"",""}}
  2023-07-03T07:13:15.978Z [4567/108] - info: [plugins/spam-be-gone] Settings loaded
  2023-07-03T07:13:15.999Z [4567/108] - info: Sunbird SSO will not be available until it is configured!
  Redis db connected.
  2023-07-03T07:13:16.027Z [4567/108] - info: [api] Adding 5 route(s) to api/v3/plugins
  2023-07-03T07:13:16.036Z [4567/108] - info: [router] Routes added
  2023-07-03T07:13:16.038Z [4567/108] - info: 🎉 NodeBB Ready
  2023-07-03T07:13:16.040Z [4567/108] - info: 🤝 Enabling 'trust proxy'
  2023-07-03T07:13:16.042Z [4567/108] - info: 📡 NodeBB is now listening on: 0.0.0.0:4567
  2023-07-03T07:13:16.042Z [4567/108] - info: 🔗 Canonical URL: http://0.0.0.0:4567/discussions/

Can You pls help to resolve this error? @keshavprasadms , @santhosh-tg
CC @gandham-santhosh

[chitranshu-keshav]

@rajeevsathish @vinukumar-vs Can you please get back on this issue

[sravani-64]

Discussion forum error is resolved now.
Below are the steps involved in Discussion forum setup on release-4.10.0

1. Provision nodebb by running build and deploy nodebb jobs from Jenkins and login to https://{{host}}/discussions/login with username and password available in secrets.yaml file

2. Follow the steps https://github.com/project-sunbird/sunbird-devops/blob/bee2ee6f39baa087eba22646d7919493552b10e1/private_repo/ansible/inventory/dev/Core/secrets.yml#L113 to generate master token and update the value in secrets.yaml file

3. Redeploy discussionsmw

4. Login to portal and create groups

5. While creating groups, we found two errors acceptformAPI and googlerecaptcha from network console, which were resolved by modifying incorrect field value of groupsTnc and googleRecaptcha **field was set in system_settings table
   Groups are created and activity and members can be added

6. For enabling discussions, v1/form/read API was updated and landed into error mentioned in ticket

7. Every time after clicking enable discussions, we can see popup enabled discussions successfully and forum was created

8. After clicking on forum, category, tags and my discussion couldn't be accessed with below error
   
   Because of this error, nodebb pod was going down with CPU and Memory utilization going beyond limits

9. As the pod was going down due to health check failure, we increased timeout seconds and finally disabled liveness and readiness from nodebb deployment

10. To troubleshoot this issue we redeployed nodebb service with various versions

11. We changed Redis database values in nodebb configmaps to new but couldn't find any progress

12. When we login to nodebb UI https://{{host}}/discussions/login, we were able to create discussion and came to an assumption that sunbird nodebb integration is incorrect

13. Finally identified the issue was with incorrect PID value updated in v1/form/read API

14. PID value has to be fetched after creating a new category in nodebbUI and add the same value in v1/form/create API form . The link for form/create is https://project-sunbird.atlassian.net/wiki/spaces/SBDES/pages/2281537544/Discussion+forum+Key+processes+to+enable+disable+DF.

16. Create a category and hover on the category name to get the PID

17. Add PID value to form update API
18. Discussion forum started working

The ideal approach to enable discussions is, first create category in nodebb UI and update form API with PID fetched from category created