Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy electricityMap static assets to GCS #2869

Merged
merged 9 commits into from
Jan 4, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 24 additions & 24 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,18 @@ services:
ports: ['8000:8000']
volumes:
- './config:/home/config'
- './web/.eslintrc:/home/web/.eslintrc'
- './web/generate-geometries.js:/home/web/generate-geometries.js'
- './web/locales:/home/web/locales'
- './web/locales-config.json:/home/web/locales-config.json'
- './web/package.json:/home/web/package.json'
- './web/public:/home/web/public'
- './web/server.js:/home/web/server.js'
- './web/src:/home/web/src'
- './web/third_party_maps:/home/web/third_party_maps'
- './web/topogen.sh:/home/web/topogen.sh'
- './web/views:/home/web/views'
- './web/webpack.config.js:/home/web/webpack.config.js'
- './web/.eslintrc:/home/src/electricitymap/contrib/web/.eslintrc'
- './web/generate-geometries.js:/home/src/electricitymap/contrib/web/generate-geometries.js'
- './web/locales:/home/src/electricitymap/contrib/web/locales'
- './web/locales-config.json:/home/src/electricitymap/contrib/web/locales-config.json'
- './web/package.json:/home/src/electricitymap/contrib/web/package.json'
- './web/public:/home/src/electricitymap/contrib/web/public'
- './web/server.js:/home/src/electricitymap/contrib/web/server.js'
- './web/src:/home/src/electricitymap/contrib/web/src'
- './web/third_party_maps:/home/src/electricitymap/contrib/web/third_party_maps'
- './web/topogen.sh:/home/src/electricitymap/contrib/web/topogen.sh'
- './web/views:/home/src/electricitymap/contrib/web/views'
- './web/webpack.config.js:/home/src/electricitymap/contrib/web/webpack.config.js'
web-watch:
build:
context: .
Expand All @@ -48,15 +48,15 @@ services:
- NODE_ENV=development
volumes:
- './config:/home/config'
- './web/.eslintrc:/home/web/.eslintrc'
- './web/generate-geometries.js:/home/web/generate-geometries.js'
- './web/locales:/home/web/locales'
- './web/locales-config.json:/home/web/locales-config.json'
- './web/package.json:/home/web/package.json'
- './web/public:/home/web/public'
- './web/server.js:/home/web/server.js'
- './web/src:/home/web/src'
- './web/third_party_maps:/home/web/third_party_maps'
- './web/topogen.sh:/home/web/topogen.sh'
- './web/views:/home/web/views'
- './web/webpack.config.js:/home/web/webpack.config.js'
- './web/.eslintrc:/home/src/electricitymap/contrib/web/.eslintrc'
- './web/generate-geometries.js:/home/src/electricitymap/contrib/web/generate-geometries.js'
- './web/locales:/home/src/electricitymap/contrib/web/locales'
- './web/locales-config.json:/home/src/electricitymap/contrib/web/locales-config.json'
- './web/package.json:/home/src/electricitymap/contrib/web/package.json'
- './web/public:/home/src/electricitymap/contrib/web/public'
- './web/server.js:/home/src/electricitymap/contrib/web/server.js'
- './web/src:/home/src/electricitymap/contrib/web/src'
- './web/third_party_maps:/home/src/electricitymap/contrib/web/third_party_maps'
- './web/topogen.sh:/home/src/electricitymap/contrib/web/topogen.sh'
- './web/views:/home/src/electricitymap/contrib/web/views'
- './web/webpack.config.js:/home/src/electricitymap/contrib/web/webpack.config.js'
24 changes: 21 additions & 3 deletions web/BUILD.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,36 @@ steps:
commands:
- yarn

# This script is used to generate zonegeometries.json
build:
environment:
ELECTRICITYMAP_PUBLIC_TOKEN: ${ELECTRICITYMAP_PUBLIC_TOKEN}
inputs:
# geometries
- generate-geometries.js
- topogen.sh
- third_party_maps
# source code
- ../config/{exchanges,zones}.json
- locales
- ./{locales-config.json,translation-status.js}
- public/{css,fonts,images,apple-app-site-association,browserconfig.xml,manifest.json}
- src
- views
- ./{.babelrc,.eslintrc,server.js,webpack.config.js}
commands:
- mkdir -p public/dist
- mkdir -p src
- bash topogen.sh
- yarn build-release
outputs:
- public/dist/zonegeometries.json
- src/world.json
- public/dist
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

brick now also builds the frontend

tag: eu.gcr.io/tmrow-152415/electricitymap_web

deploy:
image: google/cloud-sdk:243.0.0
commands:
- gsutil -m cp -a public-read -r public/* gs://static.electricitymap.org/public_web
secrets:
gcloud:
src: ~/.config/gcloud
target: /root/.config/gcloud
2 changes: 1 addition & 1 deletion web/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
FROM node:12.13.1
WORKDIR /home/web
WORKDIR /home/src/electricitymap/contrib/web
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in order to keep consistent with brick paths


# Install dependencies
RUN apt-get update && apt-get install -y jq unzip
Expand Down
74 changes: 39 additions & 35 deletions web/server.js
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ const {
getSingleTranslationStatusJSON,
getTranslationStatusJSON,
getTranslationStatusSVG,
} = require(__dirname + '/translation-status');
} = require(`${__dirname}/translation-status`);
const {
localeToFacebookLocale,
supportedFacebookLocales,
Expand All @@ -26,7 +26,7 @@ const app = express();
const server = http.Server(app);

// Constants
const STATIC_PATH = process.env['STATIC_PATH'] || (__dirname + '/public');
const STATIC_PATH = process.env.STATIC_PATH || (`${__dirname}/public`);

// * Common
app.use(compression()); // Cloudflare already does gzip but we do it anyway
Expand All @@ -46,7 +46,7 @@ i18n.configure({
// where to store json files - defaults to './locales' relative to modules directory
// note: detected locales are always lowercase
locales,
directory: __dirname + '/locales',
directory: `${__dirname}/locales`,
defaultLocale: 'en',
queryParameter: 'lang',
objectNotation: true,
Expand Down Expand Up @@ -85,24 +85,18 @@ function translateWithLocale(locale, keyStr) {
// * Long-term caching
function getHash(key, ext, obj) {
let filename;
if (typeof obj.assetsByChunkName[key] == 'string') {
if (typeof obj.assetsByChunkName[key] === 'string') {
filename = obj.assetsByChunkName[key];
} else {
// assume list
filename = obj.assetsByChunkName[key]
.filter((d) => d.match(new RegExp('\.' + ext + '$')))[0]
.filter(d => d.match(new RegExp(`\.${ext}$`)))[0];
}
return filename.replace('.' + ext, '').replace(key + '.', '');
return filename.replace(`.${ext}`, '').replace(`${key}.`, '');
}

const manifest = JSON.parse(fs.readFileSync(`${STATIC_PATH}/dist/manifest.json`));

// * Error handling
function handleError(err) {
if (!err) return;
console.error(err);
}

app.get('/health', (req, res) => res.json({ status: 'ok' }));
app.get('/clientVersion', (req, res) => res.send(version));

Expand All @@ -115,10 +109,8 @@ app.get('/translationstatus', (req, res) => res.json(getTranslationStatusJSON(lo
app.get('/translationstatus/:language', (req, res) => res.json(getSingleTranslationStatusJSON(req.params.language)));

// API
app.get('/v1/*', (req, res) =>
res.redirect(301, `https://api.electricitymap.org${req.originalUrl}`));
app.get('/v2/*', (req, res) =>
res.redirect(301, `https://api.electricitymap.org${req.originalUrl}`));
app.get('/v1/*', (req, res) => res.redirect(301, `https://api.electricitymap.org${req.originalUrl}`));
app.get('/v2/*', (req, res) => res.redirect(301, `https://api.electricitymap.org${req.originalUrl}`));

// Source maps
app.all('/dist/*.map', (req, res, next) => {
Expand All @@ -143,20 +135,20 @@ app.use('/', (req, res) => {
// redirect everyone except the Facebook crawler,
// else, we will lose all likes
const isTmrowCo = req.get('host').indexOf('electricitymap.tmrow') !== -1;
const isNonWWW = req.get('host') === 'electricitymap.org' ||
req.get('host') === 'live.electricitymap.org';
const isNonWWW = req.get('host') === 'electricitymap.org'
|| req.get('host') === 'live.electricitymap.org';
const isStaging = req.get('host') === 'staging.electricitymap.org';
const isHTTPS = req.secure;
const isLocalhost = req.hostname === 'localhost'; // hostname is without port

// Redirect all non-facebook, non-staging, non-(www.* or *.tmrow.co)
if (!isStaging && (isNonWWW || isTmrowCo) && (req.headers['user-agent'] || '').indexOf('facebookexternalhit') == -1) {
res.redirect(301, 'https://www.electricitymap.org' + req.originalUrl);
res.redirect(301, `https://www.electricitymap.org${req.originalUrl}`);
// Redirect all non-HTTPS and non localhost
// Warning: this can't happen here because Cloudfare is the HTTPS proxy.
// Node only receives HTTP traffic.
} else if (false && !isHTTPS && !isLocalhost) {
res.redirect(301, 'https://www.electricitymap.org' + req.originalUrl);
res.redirect(301, `https://www.electricitymap.org${req.originalUrl}`);
} else {
// Set locale if facebook requests it
if (req.query.fb_locale) {
Expand All @@ -166,7 +158,7 @@ app.use('/', (req, res) => {
res.setLocale(lr[0]);
}
const { locale } = res;
const fullUrl = 'https://www.electricitymap.org' + req.originalUrl;
const fullUrl = `https://www.electricitymap.org${req.originalUrl}`;

// basic auth for premium access
if (process.env.BASIC_AUTH_CREDENTIALS) {
Expand All @@ -186,19 +178,17 @@ app.use('/', (req, res) => {
res.end('Access denied');
return;
}
res.cookie('electricitymap-token', process.env['ELECTRICITYMAP_TOKEN']);
res.cookie('electricitymap-token', process.env.ELECTRICITYMAP_TOKEN);
}
res.render('pages/index', {
alternateUrls: locales.map(function(l) {
alternateUrls: locales.map((l) => {
if (fullUrl.indexOf('lang') !== -1) {
return fullUrl.replace('lang=' + req.query.lang, 'lang=' + l)
} else {
if (Object.keys(req.query).length) {
return fullUrl + '&lang=' + l;
} else {
return fullUrl.replace('?', '') + '?lang=' + l;
}
return fullUrl.replace(`lang=${req.query.lang}`, `lang=${l}`);
}
if (Object.keys(req.query).length) {
return `${fullUrl}&lang=${l}`;
}
return `${fullUrl.replace('?', '')}?lang=${l}`;
}),
bundleHash: getHash('bundle', 'js', manifest),
vendorHash: getHash('vendor', 'js', manifest),
Expand All @@ -207,14 +197,22 @@ app.use('/', (req, res) => {
// Make the paths absolute as that's required for BrowserHistory routing
// to work normally and it's also ok when used with the https:// protocol
// as resources are mounted to a fixed location.
resolvePath: function(relativePath) { return '/' + relativePath; },
// Note: `resolvePath` is executed on the client as well,
// as it is used in react components. We can't therefore include any variables
// in its closure. It would be better to pass a `pathPrefix` instead.
resolvePath: (!isProduction || isStaging)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the important non-lint change to make sure we use GCS in production

? relativePath => `/${relativePath}`
: relativePath =>
// Note we here point to static hosting in order to make
// sure we can serve older bundle versions
`https://static.electricitymap.org/public_web/${relativePath}`,
fullUrl,
locale,
locales: { en: localeConfigs['en'], [locale]: localeConfigs[locale] },
locales: { en: localeConfigs.en, [locale]: localeConfigs[locale] },
supportedLocales: locales,
FBLocale: localeToFacebookLocale[locale],
supportedFBLocales: supportedFacebookLocales,
'__': function() {
__() {
const argsArray = Array.prototype.slice.call(arguments);
// Prepend the first argument which is the locale
argsArray.unshift(locale);
Expand All @@ -224,7 +222,13 @@ app.use('/', (req, res) => {
}
});

if (isProduction) {
app.get('/*', (req, res) =>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

legacy redirect (but shouldn't be necessary)

// Redirect all requests except root to static
res.redirect(`https://static.electricitymap.org/public_web${req.originalUrl}`));
}

// Start the application
server.listen(process.env['PORT'], () => {
console.log(`Listening on *:${process.env['PORT']}`);
server.listen(process.env.PORT, () => {
console.log(`Listening on *:${process.env.PORT}`);
});
2 changes: 1 addition & 1 deletion web/src/world.json

Large diffs are not rendered by default.