Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* spike lambda renderer
* working on node errors
* i think this is in a working state now
* Added prerender manager script and copied the lambda script as the worker script
* Added worker code; Handle the sitemap as well
* Removed lambda-specific stuff
* Restored the original prerender script
* Commented out sitemap code and restored code that waits 1 minute and checks that the work queue message attributes are 0
* Fixed lint and typecheck errors; Actually delete processed messages from the queue
* Reverted change to script/entry.js
* Fixed some mistakes in the manage script; Added more logging
* Repurposed Tom's lambda Dockerfile
Fixed a bunch of errors in the prerendering script
* Fixed crash when trying to get messages from an empty queue
* Removed local proxy; Fixed lint errors
* Made the saveS3Page function async
* Reduced S3 concurrency to prevent errors and fixed Cluster deletion
* Use c6i instaces instead of t2
* Reverted credentials code
* Use node work_threads in the work script so a single instance of the script can use multiple cores and avoid network errors due to CPU contention
* Fixed errors in the threading code
* Send a single page to each thread rather than sets of 10
* Since we wait for the threads on each loop iteration to send the delete messages, there will always be threads available when we get 10 new messages, so simplify the logic
* Use asyncPool for prepareBooks
Use a promise for the work queues rather than polling
* Handle different message types
Queue and work sitemap jobs
* Write sitemaps to S3 instead of locally
* Upload sitemaps with text/xml content type
* Removed stats that don't work with distributed jobs
* Don't block the queuing thread waiting for all work threads to complete each batch of messages
* Removed more unused stats
* Create a separate concourse dir for prerender-workers
* Exclude already-uploaded files from verify-production-urls
* Replaced echo with print to fix shellcheck issue
* Fixed lint error that is only caught on CI for some reason
* Improved some comments
* Removed concourse/prerender-workers, to be re-added on the next PR
* Reverted verify-production-urls
* Removed unused Server export
* Added a comment explaining the sqsHeartbeat function
* Converted the task switch statement to a mapping of task functions
Replaced function factories with the use of function.bind()
* Replaced some more undefined checks with assertDefined
* Combined prerender-workers code with old prerender code without modifying old code behavior
* Combined the sitemap functions some more
* Combined the old and new page rendering code as much as possible
* Removed duplicated comment
* Use more threads for bigger instances
* Replaced CodeVersion and ReleaseIdSuffix parameters with ReleaseId and SanitizedReleaseId
* Updated some comments
* Added a random BUILD_ID suffix, used to keep stack and resource names unique
* Changed the Application tags to Rex Prerender
* Moved content route stuff (sitemap etc) to a separate file
Rewrote some of the types using existing generic types
* Removed unnecessary assertDefineds in getArchivePage() and getArchiveBook()
* Changed most contentPages functions to work with SerializedPageMatch instead of PageMatch
renderAndSavePage() now calls deserializePageMatch() to add { route: content } to its match argument
* Removed commented-out RedriveAllowPolicy
* Decomposed writeS3File function and replaced bind() calls
* Moved import './setup' to contentPages
* Split renderManifest() into its own file
* Format the ISO date using date-fns-tz instead of toISOString()
* Use lodash/fp/chunk instead of a manual loop to chunk pages
* Unnest page task and sitemapIndex task payloads
* More typechecks for message payloads
* Removed Loadable.preloadAll() from manage.ts and moved loader initialization to the beginning of the file
* Removed commented-out log messages
* Improved comments about checking that the SQS queue is empty
* Rewrote most of manage.ts to make the logic easier to follow
* Rewrote findOutputValue() for clarity
* manage() pieces now pass some arguments around; got rid of all non-const globals
* Fixed undefined variable error
* Take BuildId from the stack name rather than passing it in as a parameter
* Pass the PrerenderImageTag to the workers stack rather than a SanitizedReleaseId
* Re-added CodeVersion parameter, required for the workers to run
* Increased workers stack create/delete timeout
* Don't set execArgv for worker_threads
Add a global hook to handle all unhandledRejections and exit
Split some of thread.ts into smaller functions
* Removed max-old-space-size and added vm.max_map_count instead
* Added tests for assertObject()
* Removed remaining commented-out console.log()
* Make savePage return Promise<unknown> and make savePageAsset async
* Use lodash's once function to initialize the S3Client
* Make saveFile return Promise<unknown> and make renderSitemap* functions use an async version of writeAssetFile
* Integrate prerender workers with Concourse (#1408)
* Readded concourse/prerender-workers
* Renamed prerender-workers/* to prerender/manage.*
* Renamed prerender to prerender:local and prerender:manage to prerender:fleet
* Renamed prerender target to release and made it the default
Removed entrypoint and cmd from release target
* Use a prerender-mode parameter to switch prerendering modes
No longer need to run yarn install and yarn build on the rex-web image and cache node_modules
* Removed -prerender suffix from Rex image tag
* Fixed lint errors
* Downcased ci target; Build release target from utils instead of CI
* Re-added removed calls to yarn install and yarn build
* Allow execute on concourse/prerender/script.bash
* Added AWS secrets and BUCKET_NAME to prerender task
Read IMAGE_TAG from build-config
* Re-added -prerender- to the prerender stack name
* Give the large Docker image another try
* Run prerender from the image, not from the repo
* Also set the correct paths for update-content
* Build Docker image with build-configs as build args
Use PUBLIC_URL during yarn build and store it in the image
* Might as well store all build configs in the image
* Try the other Docker env variable format to see if it handles quotes better
* Create unquoted config file to use with oci-build-task
* Increased the prerender stack creation timeout
* Increased the workers stack creation/deletion timeout to 5 minutes
* Revert "update content (#1468)"
This reverts commit 31c7350.
* Pass the document to wrapSolutions
* Fixed the redirect for Fisica Universitaria
* Handle unicode URIs in update-redirects-data
* Also call decodeURI in prepareRedirects
* Add a maximum number of SQS heartbeats
Make failed messages immediately visible to other workers
* Reorganized and DRYed up code that changes ReceiptHandle visibility timeout
Co-authored-by: staxly[bot] <35789409+staxly[bot]@users.noreply.github.com>
Co-authored-by: Thomas Woodward <[email protected]>
Co-authored-by: tom <[email protected]>
Co-authored-by: staxly[bot] <35789409+staxly[bot]@users.noreply.github.com>- Loading branch information
3 people
authored
May 24, 2022
1 parent
8977214
commit 920b1f4