Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Arquivo.pt #353

Open
waybackarchiver opened this issue Mar 22, 2023 · 0 comments
Open

Add support for Arquivo.pt #353

waybackarchiver opened this issue Mar 22, 2023 · 0 comments
Labels
documentation Improvements or additions to documentation enhancement New feature or request go Pull requests that update Go code

Comments

@waybackarchiver
Copy link
Contributor

waybackarchiver commented Mar 22, 2023

Wayback

Use the SavePageNow record webpages on Arquivo.pt: https://arquivo.pt/services/savepagenow?l=en

Make a request to https://arquivo.pt/save/now/record/<uri> and extract the saving result from headers.

The saving result is shown below.

% curl -I https://arquivo.pt/save/now/record/https://github.com/wabarc/wayback
HTTP/1.1 200 OK
Date: Thu, 29 Feb 2024 06:59:51 GMT
Server: Apache
X-Archive-Orig-Server: GitHub.com
X-Archive-Orig-Date: Thu, 29 Feb 2024 06:59:51 GMT
Content-Type: text/html; charset=utf-8
X-Archive-Orig-Vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Encoding, Accept, X-Requested-With
X-Archive-Orig-ETag: W/"d34e32dca443c7feb7df540cdb3bf2ab"
X-Archive-Orig-Cache-Control: max-age=0, private, must-revalidate
X-Archive-Orig-Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Archive-Orig-X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-Archive-Orig-X-XSS-Protection: 0
Referrer-Policy: no-referrer-when-downgrade
X-Archive-Orig-Content-Security-Policy: default-src 'none'; base-uri 'self'; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com api.githubcopilot.com objects-origin.githubusercontent.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ productionresultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ productionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows.net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core.windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.blob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionresultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ productionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.windows.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.core.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultssa18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com secured-user-images.githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.githubusercontent.com opengraph.githubassets.com github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/ private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/
Set-Cookie: _gh_sess=Rvpjmwb1bxsK7rGvBlFGQ%2FOpjqDiA08cFqtL6Y8Q07In%2Fjpzfg7ByTn%2FZeVeK39U4QB50qYhYnmlBVfygORxs1KK0OYGMFFae%2FAw1k%2FGaU7ITjjVJvaQXvyyDLFQBm04UBjVoydD9L7nMDPg%2FO1vp9rX7gelBgf4PV9Cio%2FvDQL%2BB362%2BypOq%2BCIvPQZeqinImphuTgrG%2FwgDWzTyGPj0k2yZhfcvlUX07Z%2BO6M9gNJSSW4EjQWkR2ny9i%2BY0ICJgvmJkKLQkymAv4Ia290d0A%3D%3D--cnLhREymwatjYNpV--djJ9a8UJo2%2F9xBzXEgTQhg%3D%3D; HttpOnly; Path=/save/now/record/https://github.com/; SameSite=Lax; Secure
Set-Cookie: _octo=GH1.1.1877508287.1709189991; Path=/save/now/record/https://github.com/; SameSite=Lax; Secure
Set-Cookie: logged_in=no; HttpOnly; Path=/save/now/record/https://github.com/; SameSite=Lax; Secure
Accept-Ranges: bytes
X-GitHub-Request-Id: CE48:190C8D:3254478:332DCA3:65E02B66
Memento-Datetime: Thu, 29 Feb 2024 06:59:51 GMT
Link: <https://github.com/wabarc/wayback>; rel="original", <https://arquivo.pt/save/now/record/https://github.com/wabarc/wayback>; rel="timegate", <https://arquivo.pt/save/now/record/timemap/link/https://github.com/wabarc/wayback>; rel="timemap"; type="application/link-format", <https://arquivo.pt/save/now/record/20240229065951/https://github.com/wabarc/wayback>; rel="memento"; datetime="Thu, 29 Feb 2024 06:59:51 GMT"; collection="live"
Vary: accept-datetime,Accept-Encoding
Content-Location: https://arquivo.pt/save/now/record/20240229065951/https://github.com/wabarc/wayback
Content-Security-Policy: default-src 'unsafe-eval' 'unsafe-inline' 'self' data: blob: mediastream: ws: wss: ; form-action 'self'
Cache-Control: max-age=300, public, must-revalidate
Strict-Transport-Security: max-age=31536000; preload
X-XSS-Protection: 1; mode=block
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: X-Requested-With

Playback

APIs: https://arquivo.pt/api or https://github.com/arquivo/pwa-technologies/wiki/APIs

Memento API: https://github.com/arquivo/pwa-technologies/wiki/Memento--API
TimeGate: https://arquivo.pt/wayback/

Example: https://arquivo.pt/textsearch?versionHistory=https://github.com/wabarc/wayback

{
  "serviceName" : "Arquivo.pt - the Portuguese web-archive",
  "linkToService" : "https://arquivo.pt/",
  "next_page" : "https://arquivo.pt/textsearch?versionHistory=https://github.com/wabarc/wayback&offset=50",
  "estimated_nr_results" : 3,
  "response_items" : [ {
    "title" : "https://github.com/wabarc/wayback",
    "originalURL" : "https://github.com/wabarc/wayback",
    "linkToArchive" : "https://arquivo.pt/wayback/20230319121752/https://github.com/wabarc/wayback",
    "tstamp" : "20230319121752",
    "contentLength" : 48985,
    "digest" : "ZASZQVWBPU4RGDCN4SBWWSRIGOUAKDGZ",
    "mimeType" : "text/html",
    "linkToScreenshot" : "https://arquivo.pt/screenshot?url=https%3A%2F%2Farquivo.pt%2FnoFrame%2Freplay%2F20230319121752%2Fhttps%3A%2F%2Fgithub.com%2Fwabarc%2Fwayback",
    "linkToNoFrame" : "https://arquivo.pt/noFrame/replay/20230319121752/https://github.com/wabarc/wayback",
    "linkToMetadata" : "https://arquivo.pt/textsearch?metadata=https%3A%2F%2Fgithub.com%2Fwabarc%2Fwayback%2F20230319121752",
    "linkToOriginalFile" : "https://arquivo.pt/noFrame/replay/20230319121752id_/https://github.com/wabarc/wayback",
    "fileName" : "save-merged-p108.arquivo.pt-20230319153144-61U13F81OW.warc.gz",
    "collection" : "SAWP2",
    "offset" : 72755013,
    "statusCode" : 200
  }, {
    "title" : "https://github.com/wabarc/wayback",
    "originalURL" : "https://github.com/wabarc/wayback",
    "linkToArchive" : "https://arquivo.pt/wayback/20230319121639/https://github.com/wabarc/wayback",
    "tstamp" : "20230319121639",
    "contentLength" : 48986,
    "digest" : "NLHYHH6YSRRVK6SAUCLMIDYHDZP4RSX2",
    "mimeType" : "text/html",
    "linkToScreenshot" : "https://arquivo.pt/screenshot?url=https%3A%2F%2Farquivo.pt%2FnoFrame%2Freplay%2F20230319121639%2Fhttps%3A%2F%2Fgithub.com%2Fwabarc%2Fwayback",
    "linkToNoFrame" : "https://arquivo.pt/noFrame/replay/20230319121639/https://github.com/wabarc/wayback",
    "linkToMetadata" : "https://arquivo.pt/textsearch?metadata=https%3A%2F%2Fgithub.com%2Fwabarc%2Fwayback%2F20230319121639",
    "linkToOriginalFile" : "https://arquivo.pt/noFrame/replay/20230319121639id_/https://github.com/wabarc/wayback",
    "fileName" : "save-merged-p108.arquivo.pt-20230319153144-61U13F81OW.warc.gz",
    "collection" : "SAWP2",
    "offset" : 32546305,
    "statusCode" : 200
  }, {
    "title" : "https://github.com/wabarc/wayback",
    "originalURL" : "https://github.com/wabarc/wayback",
    "linkToArchive" : "https://arquivo.pt/wayback/20230319030556/https://github.com/wabarc/wayback",
    "tstamp" : "20230319030556",
    "contentLength" : 49504,
    "digest" : "6J3KE7C64NEHRYEP6XMYDVWL3YK37N2P",
    "mimeType" : "text/html",
    "linkToScreenshot" : "https://arquivo.pt/screenshot?url=https%3A%2F%2Farquivo.pt%2FnoFrame%2Freplay%2F20230319030556%2Fhttps%3A%2F%2Fgithub.com%2Fwabarc%2Fwayback",
    "linkToNoFrame" : "https://arquivo.pt/noFrame/replay/20230319030556/https://github.com/wabarc/wayback",
    "linkToMetadata" : "https://arquivo.pt/textsearch?metadata=https%3A%2F%2Fgithub.com%2Fwabarc%2Fwayback%2F20230319030556",
    "linkToOriginalFile" : "https://arquivo.pt/noFrame/replay/20230319030556id_/https://github.com/wabarc/wayback",
    "fileName" : "save-merged-p108.arquivo.pt-20230319153144-61U13F81OW.warc.gz",
    "collection" : "SAWP2",
    "offset" : 91480610,
    "statusCode" : 200
  } ]
}
@waybackarchiver waybackarchiver added documentation Improvements or additions to documentation enhancement New feature or request go Pull requests that update Go code labels Mar 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation enhancement New feature or request go Pull requests that update Go code
Projects
None yet
Development

No branches or pull requests

1 participant