Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy electricityMap static assets to GCS #2869

Merged
merged 9 commits into from
Jan 4, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ jobs:
docker-compose build
# Make sure files are available outside of container
CONTAINER_ID=$(docker create eu.gcr.io/tmrow-152415/electricitymap_web:latest)
docker cp $CONTAINER_ID:/home/web/public/dist web/public/dist
docker cp $CONTAINER_ID:/home/src/electricitymap/contrib/web/public/dist web/public/dist
no_output_timeout: 30m
- run:
name: Start environment
Expand Down
48 changes: 24 additions & 24 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,18 @@ services:
ports: ['8000:8000']
volumes:
- './config:/home/config'
- './web/.eslintrc:/home/web/.eslintrc'
- './web/generate-geometries.js:/home/web/generate-geometries.js'
- './web/locales:/home/web/locales'
- './web/locales-config.json:/home/web/locales-config.json'
- './web/package.json:/home/web/package.json'
- './web/public:/home/web/public'
- './web/server.js:/home/web/server.js'
- './web/src:/home/web/src'
- './web/third_party_maps:/home/web/third_party_maps'
- './web/topogen.sh:/home/web/topogen.sh'
- './web/views:/home/web/views'
- './web/webpack.config.js:/home/web/webpack.config.js'
- './web/.eslintrc:/home/src/electricitymap/contrib/web/.eslintrc'
- './web/generate-geometries.js:/home/src/electricitymap/contrib/web/generate-geometries.js'
- './web/locales:/home/src/electricitymap/contrib/web/locales'
- './web/locales-config.json:/home/src/electricitymap/contrib/web/locales-config.json'
- './web/package.json:/home/src/electricitymap/contrib/web/package.json'
- './web/public:/home/src/electricitymap/contrib/web/public'
- './web/server.js:/home/src/electricitymap/contrib/web/server.js'
- './web/src:/home/src/electricitymap/contrib/web/src'
- './web/third_party_maps:/home/src/electricitymap/contrib/web/third_party_maps'
- './web/topogen.sh:/home/src/electricitymap/contrib/web/topogen.sh'
- './web/views:/home/src/electricitymap/contrib/web/views'
- './web/webpack.config.js:/home/src/electricitymap/contrib/web/webpack.config.js'
web-watch:
build:
context: .
Expand All @@ -48,15 +48,15 @@ services:
- NODE_ENV=development
volumes:
- './config:/home/config'
- './web/.eslintrc:/home/web/.eslintrc'
- './web/generate-geometries.js:/home/web/generate-geometries.js'
- './web/locales:/home/web/locales'
- './web/locales-config.json:/home/web/locales-config.json'
- './web/package.json:/home/web/package.json'
- './web/public:/home/web/public'
- './web/server.js:/home/web/server.js'
- './web/src:/home/web/src'
- './web/third_party_maps:/home/web/third_party_maps'
- './web/topogen.sh:/home/web/topogen.sh'
- './web/views:/home/web/views'
- './web/webpack.config.js:/home/web/webpack.config.js'
- './web/.eslintrc:/home/src/electricitymap/contrib/web/.eslintrc'
- './web/generate-geometries.js:/home/src/electricitymap/contrib/web/generate-geometries.js'
- './web/locales:/home/src/electricitymap/contrib/web/locales'
- './web/locales-config.json:/home/src/electricitymap/contrib/web/locales-config.json'
- './web/package.json:/home/src/electricitymap/contrib/web/package.json'
- './web/public:/home/src/electricitymap/contrib/web/public'
- './web/server.js:/home/src/electricitymap/contrib/web/server.js'
- './web/src:/home/src/electricitymap/contrib/web/src'
- './web/third_party_maps:/home/src/electricitymap/contrib/web/third_party_maps'
- './web/topogen.sh:/home/src/electricitymap/contrib/web/topogen.sh'
- './web/views:/home/src/electricitymap/contrib/web/views'
- './web/webpack.config.js:/home/src/electricitymap/contrib/web/webpack.config.js'
24 changes: 21 additions & 3 deletions web/BUILD.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,36 @@ steps:
commands:
- yarn

# This script is used to generate zonegeometries.json
build:
environment:
ELECTRICITYMAP_PUBLIC_TOKEN: ${BRICK_ELECTRICITYMAP_PUBLIC_TOKEN}
inputs:
# geometries
- generate-geometries.js
- topogen.sh
- third_party_maps
# source code
- ../config/{exchanges,zones}.json
- locales
- ./{locales-config.json,translation-status.js}
- public/{css,fonts,images,apple-app-site-association,browserconfig.xml,manifest.json}
- src
- views
- ./{.babelrc,.eslintrc,server.js,webpack.config.js}
commands:
- mkdir -p public/dist
- mkdir -p src
- bash topogen.sh
- yarn build-release
outputs:
- public/dist/zonegeometries.json
- src/world.json
- public/dist
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

brick now also builds the frontend

tag: eu.gcr.io/tmrow-152415/electricitymap_web

deploy:
image: google/cloud-sdk:243.0.0
commands:
- gsutil -m cp -a public-read -r public/* gs://static.electricitymap.org/public_web
secrets:
gcloud:
src: ~/.config/gcloud
target: /root/.config/gcloud
18 changes: 9 additions & 9 deletions web/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,26 +1,26 @@
FROM node:12.13.1
WORKDIR /home/web
WORKDIR /home/src/electricitymap/contrib/web
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in order to keep consistent with brick paths


# Install dependencies
RUN apt-get update && apt-get install -y jq unzip
ADD web/package.json /home/web/package.json
ADD web/yarn.lock /home/web/yarn.lock
ADD web/package.json ./package.json
ADD web/yarn.lock ./yarn.lock
RUN yarn

# Generate map
RUN mkdir -p public/dist/
ADD web/build /home/web/build
ADD web/third_party_maps /home/web/third_party_maps
ADD web/generate-geometries.js web/topogen.sh /home/web/
ADD web/src/world.json /home/web/src/world.json
ADD web/build ./build
ADD web/third_party_maps ./third_party_maps
ADD web/generate-geometries.js web/topogen.sh ./
ADD web/src/world.json ./src/world.json
RUN bash topogen.sh

ARG ELECTRICITYMAP_PUBLIC_TOKEN

# Build
# (note: this will override the world.json that was previously created)
ADD config /home/config
ADD web /home/web
ADD config /home/src/electricitymap/contrib/config
ADD web ./
RUN yarn lint && yarn build-release

EXPOSE 8000
Expand Down
1 change: 0 additions & 1 deletion web/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@
"babel-polyfill": "^6.26.0",
"babel-preset-es2015": "^6.24.1",
"babel-runtime": "^6.26.0",
"clean-webpack-plugin": "^3.0.0",
"colors": "^1.3.2",
"css-loader": "^0.28.10",
"eslint": "^5.5.0",
Expand Down
74 changes: 39 additions & 35 deletions web/server.js
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ const {
getSingleTranslationStatusJSON,
getTranslationStatusJSON,
getTranslationStatusSVG,
} = require(__dirname + '/translation-status');
} = require(`${__dirname}/translation-status`);
const {
localeToFacebookLocale,
supportedFacebookLocales,
Expand All @@ -26,7 +26,7 @@ const app = express();
const server = http.Server(app);

// Constants
const STATIC_PATH = process.env['STATIC_PATH'] || (__dirname + '/public');
const STATIC_PATH = process.env.STATIC_PATH || (`${__dirname}/public`);

// * Common
app.use(compression()); // Cloudflare already does gzip but we do it anyway
Expand All @@ -46,7 +46,7 @@ i18n.configure({
// where to store json files - defaults to './locales' relative to modules directory
// note: detected locales are always lowercase
locales,
directory: __dirname + '/locales',
directory: `${__dirname}/locales`,
defaultLocale: 'en',
queryParameter: 'lang',
objectNotation: true,
Expand Down Expand Up @@ -85,24 +85,18 @@ function translateWithLocale(locale, keyStr) {
// * Long-term caching
function getHash(key, ext, obj) {
let filename;
if (typeof obj.assetsByChunkName[key] == 'string') {
if (typeof obj.assetsByChunkName[key] === 'string') {
filename = obj.assetsByChunkName[key];
} else {
// assume list
filename = obj.assetsByChunkName[key]
.filter((d) => d.match(new RegExp('\.' + ext + '$')))[0]
.filter(d => d.match(new RegExp(`\.${ext}$`)))[0];
}
return filename.replace('.' + ext, '').replace(key + '.', '');
return filename.replace(`.${ext}`, '').replace(`${key}.`, '');
}

const manifest = JSON.parse(fs.readFileSync(`${STATIC_PATH}/dist/manifest.json`));

// * Error handling
function handleError(err) {
if (!err) return;
console.error(err);
}

app.get('/health', (req, res) => res.json({ status: 'ok' }));
app.get('/clientVersion', (req, res) => res.send(version));

Expand All @@ -115,10 +109,8 @@ app.get('/translationstatus', (req, res) => res.json(getTranslationStatusJSON(lo
app.get('/translationstatus/:language', (req, res) => res.json(getSingleTranslationStatusJSON(req.params.language)));

// API
app.get('/v1/*', (req, res) =>
res.redirect(301, `https://api.electricitymap.org${req.originalUrl}`));
app.get('/v2/*', (req, res) =>
res.redirect(301, `https://api.electricitymap.org${req.originalUrl}`));
app.get('/v1/*', (req, res) => res.redirect(301, `https://api.electricitymap.org${req.originalUrl}`));
app.get('/v2/*', (req, res) => res.redirect(301, `https://api.electricitymap.org${req.originalUrl}`));

// Source maps
app.all('/dist/*.map', (req, res, next) => {
Expand All @@ -143,20 +135,20 @@ app.use('/', (req, res) => {
// redirect everyone except the Facebook crawler,
// else, we will lose all likes
const isTmrowCo = req.get('host').indexOf('electricitymap.tmrow') !== -1;
const isNonWWW = req.get('host') === 'electricitymap.org' ||
req.get('host') === 'live.electricitymap.org';
const isNonWWW = req.get('host') === 'electricitymap.org'
|| req.get('host') === 'live.electricitymap.org';
const isStaging = req.get('host') === 'staging.electricitymap.org';
const isHTTPS = req.secure;
const isLocalhost = req.hostname === 'localhost'; // hostname is without port

// Redirect all non-facebook, non-staging, non-(www.* or *.tmrow.co)
if (!isStaging && (isNonWWW || isTmrowCo) && (req.headers['user-agent'] || '').indexOf('facebookexternalhit') == -1) {
res.redirect(301, 'https://www.electricitymap.org' + req.originalUrl);
res.redirect(301, `https://www.electricitymap.org${req.originalUrl}`);
// Redirect all non-HTTPS and non localhost
// Warning: this can't happen here because Cloudfare is the HTTPS proxy.
// Node only receives HTTP traffic.
} else if (false && !isHTTPS && !isLocalhost) {
res.redirect(301, 'https://www.electricitymap.org' + req.originalUrl);
res.redirect(301, `https://www.electricitymap.org${req.originalUrl}`);
} else {
// Set locale if facebook requests it
if (req.query.fb_locale) {
Expand All @@ -166,7 +158,7 @@ app.use('/', (req, res) => {
res.setLocale(lr[0]);
}
const { locale } = res;
const fullUrl = 'https://www.electricitymap.org' + req.originalUrl;
const fullUrl = `https://www.electricitymap.org${req.originalUrl}`;

// basic auth for premium access
if (process.env.BASIC_AUTH_CREDENTIALS) {
Expand All @@ -186,19 +178,17 @@ app.use('/', (req, res) => {
res.end('Access denied');
return;
}
res.cookie('electricitymap-token', process.env['ELECTRICITYMAP_TOKEN']);
res.cookie('electricitymap-token', process.env.ELECTRICITYMAP_TOKEN);
}
res.render('pages/index', {
alternateUrls: locales.map(function(l) {
alternateUrls: locales.map((l) => {
if (fullUrl.indexOf('lang') !== -1) {
return fullUrl.replace('lang=' + req.query.lang, 'lang=' + l)
} else {
if (Object.keys(req.query).length) {
return fullUrl + '&lang=' + l;
} else {
return fullUrl.replace('?', '') + '?lang=' + l;
}
return fullUrl.replace(`lang=${req.query.lang}`, `lang=${l}`);
}
if (Object.keys(req.query).length) {
return `${fullUrl}&lang=${l}`;
}
return `${fullUrl.replace('?', '')}?lang=${l}`;
}),
bundleHash: getHash('bundle', 'js', manifest),
vendorHash: getHash('vendor', 'js', manifest),
Expand All @@ -207,14 +197,22 @@ app.use('/', (req, res) => {
// Make the paths absolute as that's required for BrowserHistory routing
// to work normally and it's also ok when used with the https:// protocol
// as resources are mounted to a fixed location.
resolvePath: function(relativePath) { return '/' + relativePath; },
// Note: `resolvePath` is executed on the client as well,
// as it is used in react components. We can't therefore include any variables
// in its closure. It would be better to pass a `pathPrefix` instead.
resolvePath: (!isProduction || isStaging)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the important non-lint change to make sure we use GCS in production

? relativePath => `/${relativePath}`
: relativePath =>
// Note we here point to static hosting in order to make
// sure we can serve older bundle versions
`https://static.electricitymap.org/public_web/${relativePath}`,
fullUrl,
locale,
locales: { en: localeConfigs['en'], [locale]: localeConfigs[locale] },
locales: { en: localeConfigs.en, [locale]: localeConfigs[locale] },
supportedLocales: locales,
FBLocale: localeToFacebookLocale[locale],
supportedFBLocales: supportedFacebookLocales,
'__': function() {
__() {
const argsArray = Array.prototype.slice.call(arguments);
// Prepend the first argument which is the locale
argsArray.unshift(locale);
Expand All @@ -224,7 +222,13 @@ app.use('/', (req, res) => {
}
});

if (isProduction) {
app.get('/*', (req, res) =>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

legacy redirect (but shouldn't be necessary)

// Redirect all requests except root to static
res.redirect(`https://static.electricitymap.org/public_web${req.originalUrl}`));
}

// Start the application
server.listen(process.env['PORT'], () => {
console.log(`Listening on *:${process.env['PORT']}`);
server.listen(process.env.PORT, () => {
console.log(`Listening on *:${process.env.PORT}`);
});
2 changes: 1 addition & 1 deletion web/src/world.json

Large diffs are not rendered by default.

2 changes: 0 additions & 2 deletions web/webpack.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ const autoprefixer = require('autoprefixer');

const MiniCssExtractPlugin = require('mini-css-extract-plugin');
const OptimizeCssAssetsPlugin = require('optimize-css-assets-webpack-plugin');
const { CleanWebpackPlugin } = require('clean-webpack-plugin');

const isProduction = process.env.NODE_ENV === 'production';

Expand Down Expand Up @@ -41,7 +40,6 @@ module.exports = {
],
},
plugins: [
new CleanWebpackPlugin(),
new OptimizeCssAssetsPlugin(),
new MiniCssExtractPlugin({
filename: '[name].' + (isProduction ? '[chunkhash]' : 'dev') + '.css',
Expand Down