Journal

This is a journal of activity. Put stuff here first, then later figure out where else it should go.

Journal 2019

Aug 4, 2020

Where are we now, 1 year later? What is the state of this project?
Documenting the Architecture and Operations

Updating angular from 7 to 8
https://update.angular.io/#7.0:8.0

and then from 8 to 9
https://update.angular.io/#8.0:9.1

and then finally to angular 10

Aug 12, 2020

Circular progress indicators
As of 8/12/20, the package below was last published around 7/15/20
https://www.npmjs.com/package/angular-svg-round-progressbar

Aug 13, 2020

Need to address this
Security rules on database access: https://firebase.google.com/docs/firestore/security/insecure-rules
Current rules: https://console.firebase.google.com/project/yourvotecounts-bd737/database/firestore/rules

Aug 13, 2020

Custom Phone Pipe
See this commit to see how I did it - very cool
https://github.com/bdunklau/yourvotecounts/commit/826fb1ab7e475f0deaa843a5f1507c55812c185b

Aug 14, 2020

Phone number validation
See this commit
https://github.com/bdunklau/yourvotecounts/commit/11120a65a3551776867659fb825252357f47b6d6
And this article for reference
https://fiyazhasan.me/angular-forms-validation-updateon-blur/

Aug 14, 2020

Adding new screens - getting Karma unit tests to pass
Look at what went in to getting them to pass
https://github.com/bdunklau/yourvotecounts/commit/f4bd7eb0935377e62165ed9709ce3d99c548a11c

Aug 16, 2020

How does TelePatriot-Web display video?...
TelePatriot-Web Video Chat Code Trace
See also the Really Detailed Code Trace

Aug 17, 2020

Added cors support to the generateTwilioToken() firebase function
See this commit:
https://github.com/bdunklau/yourvotecounts/commit/0f0c1d0fa239580e4f0bee07703d73da68ccdf93
Got the solution from this video - worked great
https://www.youtube.com/watch?v=H0rZnxRV7_k

Aug 18, 2020 Recording Rooms

You don't have to create the room first. You can simply connect to a room from the angular component and the room will be created. Recording will be enabled by default as long as you configure the default room that way. See this page for the default room config:
https://www.twilio.com/console/video/configure

Aug 23, 2020 FFMPEG Splice and Concatenate

Great article on splicing from 2015
https://markheath.net/post/cut-and-concatenate-with-ffmpeg

Concatenate
See #2 concat demuxer
https://stackoverflow.com/questions/7333232/how-to-concatenate-two-mp4-files-using-ffmpeg/11175851#11175851

Aug 23, 2020 Use Google Virtual Machine

Probably need to use a vm and a python script to splice and concatenate the videos
How do we call a python script running on a vm? (We did it in TelePatriot)
Answer: You have a node server/express app running on the vm and you have the IP address of the vm
Look at the vm instance in the TelePatriot project
Start the vm
See External IP when running
Click SSH to get a terminal

Aug 23, 2020 Downloading the composition file

To download the composition file to a vm, see app.post('/publish'... in index.js

Aug 23, 2020 Calling FFMPEG from NodeJS

This package apparently lets us call ffmpeg functions from nodejs
https://github.com/fluent-ffmpeg/node-fluent-ffmpeg

Aug 24, 2020 New project yourvotecounts-vm

https://github.com/bdunklau/yourvotecounts-vm - copied from https://github.com/bdunklau/TelePatriot-GCloud
VM instance: https://console.cloud.google.com/compute/instances?project=yourvotecounts-bd737&instancessize=50
Had to install node, npm and git
Created Firewall Rule: Added port 7000 after port 80 to the "default-allow-http" rule
See External IP and go to that IP at port 7000, i.e. http://34.68.114.174:7000/
See bottom of https://github.com/bdunklau/yourvotecounts-vm/blob/master/nodejs/index.js

app.listen(7000, function() {
    console.log('app listeneing on port 7000')
})

Aug 24, 2020 Starting/Stopping the VM

Go to VM Instances page
Click SSH -> Open in browser window

cd ~/yourvotecounts-vm/nodejs 
./start-node

And to stop node

./stop-node

Aug 25, 2020 Producing the Video

See Producing the Video

Aug 27, 2020 Marking Time

Working the design doc called Marking Time

Aug 27, 2020 Install ffmpeg on vm

vm instance > SSH

sudo apt update
sudo apt install ffmpeg
ffmpeg -version

source: https://linuxize.com/post/how-to-install-ffmpeg-on-ubuntu-18-04/

Aug 29, 2020 Download composition to the vm

curl -H 'Content-Type: application/json' -d '{ 
    "RoomSid": "RMa538d10712f334f4830f7694483f952d", 
    "twilio_account_sid": "ACce7e5e5cbf309ac4eb81b6579793a1b1", 
    "twilio_auth_token": "SEE TWILIO", 
    "domain": "video.twilio.com", 
    "MediaUri": "/v1/Compositions/CJ2601577fa348e97e367f218417e49920/Media", 
    "CompositionSid": "CJ2601577fa348e97e367f218417e49920", 
    "Ttl": 3600, 
    "firebase_functions_host": "us-central1-yourvotecounts-bd737.cloudfunctions.net", 
    "firebase_function": "/downloadComplete"
}' -X POST http://34.68.114.174:7000/downloadComposition

for cut-paste...

curl -H 'Content-Type: application/json' -d '{ "RoomSid": "RMa538d10712f334f4830f7694483f952d", "twilio_account_sid": "ACce7e5e5cbf309ac4eb81b6579793a1b1", "twilio_auth_token": "SEE TWILIO", "domain": "video.twilio.com", "MediaUri": "/v1/Compositions/CJ2601577fa348e97e367f218417e49920/Media", "CompositionSid": "CJ2601577fa348e97e367f218417e49920", "Ttl": 3600, "firebase_functions_host": "us-central1-yourvotecounts-bd737.cloudfunctions.net", "firebase_function": "/downloadComplete"}' -X POST http://34.68.114.174:7000/downloadComposition

Aug 29, 2020 Cutting up the Video from vm

See: splice and concatenate with ffmpeg
Example

curl -H "Content-Type: application/json" -d 
'{"tempEditFolder": "/home/bdunklau/videos/CJdf0664dfe3b22c30d201cb7a52e91ec2", 
"compositionFile": "/home/bdunklau/videos/CJdf0664dfe3b22c30d201cb7a52e91ec2.mp4",
"roomObj": {"mark_time": [{"start_recording": "00:00:02", "duration": "00:00:08"},
{"start_recording": "00:00:12", "duration": "00:00:08"},
{"start_recording": "00:00:22", "duration": "00:00:03"}]}}'
 -X POST  "http://34.68.114.174:7000/cutVideo"

as one line for cut-paste...

curl -H "Content-Type: application/json" -d '{"tempEditFolder":"/home/bdunklau/videos/CJdf0664dfe3b22c30d201cb7a52e91ec2","compositionFile":"/home/bdunklau/videos/CJdf0664dfe3b22c30d201cb7a52e91ec2.mp4","roomObj": {"mark_time": [{"start_recording": "00:00:02", "duration": "00:00:08"},{"start_recording":"00:00:12","duration":"00:00:08"},{"start_recording": "00:00:22", "duration": "00:00:03"}]}}' -X POST  "http://34.68.114.174:7000/cutVideo"

Aug 29, 2020 Cutting up the Video from firebase function

See: splice and concatenate with ffmpeg
Example

curl -H "Content-Type: application/json" -d \
'{"compositionFile": "/home/bdunklau/videos/COMPOSITION_SID.mp4", \
"RoomSid": "ROOM_SID", \
"tempEditFolder":  "/home/bdunklau/videos/COMPOSITION_SID", \
"downloadComplete": true}' \
-X POST  "https://us-central1-yourvotecounts-bd737.cloudfunctions.net/downloadComplete"

as one line for cut-paste...

curl -H "Content-Type: application/json" -d '{"compositionFile": "/home/bdunklau/videos/COMPOSITION_SID.mp4", "RoomSid":"ROOM_SID", "tempEditFolder":  "/home/bdunklau/videos/COMPOSITION_SID", "downloadComplete": true}' -X POST  "https://us-central1-yourvotecounts-bd737.cloudfunctions.net/downloadComplete"

Aug 30, 2020 npm request going away

Here are some alternatives
https://github.com/request/request/issues/3143#issue-427486416

Sept 2, 2020 Securely connecting to vm instances

https://cloud.google.com/solutions/connecting-securely
https://cloud.google.com/load-balancing/docs/ssl-certificates/google-managed-certs

Sept 2, 2020 Updating Operations page

Deploy Production (Angular)
Deploy Production (vm code)

Sept 3, 2020 SeeSaw on Vimeo

https://developer.vimeo.com/apps/187911
[email protected] / J........11
Uploading via API
https://developer.vimeo.com/api/guides/videos/upload
https://developer.vimeo.com/api/upload/videos

Sept 3, 2020 res.download() is not the way to go!

https://stackoverflow.com/a/24977085

Sept 3, 2020 DIY Cloud FTP Storage

https://5best.cloud/cloud-ftp-storage-diy/

Sept 3, 2020 Back to Google Cloud Storage

Can we make .mp4 files public to the internet? Yes - see Storage Browser > yourvotecounts-bd737.appspot.com > this .mp4
See Cloud Storage API
See Uploading objects > Making data public
Notice this code sample

/**
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// const bucketName = 'Name of a bucket, e.g. my-bucket';
// const filename = 'File to make public, e.g. file.txt';

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');

// Creates a client
const storage = new Storage();

async function makePublic() {
  // Makes the file public
  await storage.bucket(bucketName).file(filename).makePublic();

  console.log(`gs://${bucketName}/${filename} is now public.`);
}

makePublic().catch(console.error);

Sept 5, 2020 Why don't .mp4 files play on iOS?

The error you get in Safari console: "Failed to load resource: Plug-in handled load"
See...
https://github.com/meteor/meteor/issues/3229
https://stackoverflow.com/a/50354182
https://philna.sh/blog/2018/10/23/service-workers-beware-safaris-range-request/
https://stackoverflow.com/a/61903166 <-- maybe this
How to Convert MP4 to HLS

The Magical ffmpeg command

ffmpeg -i CJ9b9c4b45bc8f02c9724c0717592073b3.mp4 -c:v h264 -flags +cgop -g 30 -hls_time 10 -hls_list_size 0  -hls_segment_filename 'CJ9b9c4b45bc8f02c972
4c0717592073b3%03d.ts' CJ9b9c4b45bc8f02c9724c0717592073b3.m3u8

work in progress...
See ~/videos/hls on the vm
See the storage bucket

Sept 9, 2020 Design: Producing the Video

Misc cURL

Sept 9, 2020 Looking up Elected Officials

Google Civic Information API
Google Places API

Sept 20, 2020 Angular Server Side Rendering (SSR)

Why SSR?
Primarily so that social media sites will be able to unfurl /view-video pages
https://angular.io/guide/universal
https://blog.angular-university.io/angular-universal/
https://www.youtube.com/watch?v=Ttfvv_tw3Qc <-- this video solved the "window is not defined" problem

Commits

Look at these commits that got us SSR:
https://github.com/bdunklau/yourvotecounts/commit/6b732e353b00cebffb1bd9f499af31d133598db2
https://github.com/bdunklau/yourvotecounts/commit/9ca87652073f8d33f7a65a57840b89dd9f0ca9ec (see inline comments)

Sept 22, 2020 FFMPEG - Screenshots

Here's how you take a screenshot of a video file
source: https://www.junian.net/tech/ffmpeg-video-screenshot/

ffmpeg -i input.mp4 -ss 00:00:01 -vframes 1 output.jpg

00:00:01 means take the screenshot 1 second in to the video

Go to Compute Engine instance

SSH in

cd ~/videos

Pick any .mp4 and run the command above

Sept 25, 2020 Angular SSR - Big Problem

https://github.com/angular/angularfire/issues/2420
Workaround: don't use toPromise() and await. Use Observables and subscribe() (big hassle, more code)

Sept 26, 2020 Deploying to App Engine

Oct 5, 2020 More Angular Universal Crap

Look at Commits that happened between Oct 5 and Oct 25 and create some entries here

Oct 25, 2020 Firestore Security Rules

Get started with Cloud Firestore Security Rules
Fix insecure rules
Security rules version '2' - specifically notice this part...

Overlapping match statements

It's possible for a document to match more than one match statement. 
In the case where multiple allow expressions match a request, the access 
is allowed if any of the conditions is true

To view the project's security rules
Firebase console > click project > Cloud Firestore > Rules

Journal

Journal 2019

Aug 4, 2020

Aug 12, 2020

Aug 13, 2020

Aug 13, 2020

Aug 14, 2020

Aug 14, 2020

Aug 16, 2020

Aug 17, 2020

Aug 18, 2020 Recording Rooms

Aug 23, 2020 FFMPEG Splice and Concatenate

Aug 23, 2020 Use Google Virtual Machine

Aug 23, 2020 Downloading the composition file

Aug 23, 2020 Calling FFMPEG from NodeJS

Aug 24, 2020 New project yourvotecounts-vm

Aug 24, 2020 Starting/Stopping the VM

Aug 25, 2020 Producing the Video

Aug 27, 2020 Marking Time

Aug 27, 2020 Install ffmpeg on vm

Aug 29, 2020 Download composition to the vm

Aug 29, 2020 Cutting up the Video from vm

Aug 29, 2020 Cutting up the Video from firebase function

Aug 30, 2020 npm request going away

Sept 2, 2020 Securely connecting to vm instances

Sept 2, 2020 Updating Operations page

Sept 3, 2020 SeeSaw on Vimeo

Sept 3, 2020 res.download() is not the way to go!

Sept 3, 2020 DIY Cloud FTP Storage

Sept 3, 2020 Back to Google Cloud Storage

Sept 5, 2020 Why don't .mp4 files play on iOS?

Sept 9, 2020 Design: Producing the Video

Sept 9, 2020 Looking up Elected Officials

See also the API Credentials page

Sept 20, 2020 Angular Server Side Rendering (SSR)

Commits

See also Operations

Sept 22, 2020 FFMPEG - Screenshots

Go to Compute Engine instance

See also Misc cURL

Sept 25, 2020 Angular SSR - Big Problem

Sept 26, 2020 Deploying to App Engine

Oct 5, 2020 More Angular Universal Crap

Look at Commits that happened between Oct 5 and Oct 25 and create some entries here

Oct 25, 2020 Firestore Security Rules

Clone this wiki locally