Importing Your Own Podcast
This guide assumes you have done all the steps in Getting started (for developers)!
Getting your own podcast content into Shortcut involves two major steps: obtaining a transcript and aligning the transcript. Along the way we'll be organizing our files and serving them on a local http server.
Create a directory on your hard drive where you're going to store the episode data for Shortcut. We'll call this the "Shortcut data directory". This will eventually be served on a local http server, so if you already have a directory that serves on localhost, drop it in there. Otherwise just create a directory wherever it makes sense for you.
Pick an episode of your podcast that you want to test Shortcut with and decide on a unique ID for the episode. Many podcasts just use their episode number as the unique ID, since no two episodes share the same episode number. The ID can be any alphanumeric string, so if it's your third episode, "3" or "s01e03" or "ep3" can all work.
Create a subdirectory in your Shortcut data directory with the same name as the ID of the episode. Grab the mp3 of the episode (convert it to mp3 if it's in another format) and put it in that new directory. Make sure the mp3 file has the same name as the ID, too.
Let's say your episode ID is "s01e03". At this point you should have a folder structure like this:
/path/to/data/
/path/to/data/s01e03/s01e03.mp3
You'll want a clean text transcript of your podcast audio. If you already have one, you can skip down to "aligning the transcript".
You can use any transcription service you like, including doing it yourself, but for fast and cheap transcription we recommend Temi, which provides fast, accurate machine-generated transcription at $0.10/minute, so only $6 for a one-hour podcast episode.
You can request a free transcription as part of their trial, so let's do that now. Go to https://www.temi.com and click "Select Audio/Video File" on their landing page. It'll ask you a few questions (hopefully you have pretty decent audio quality in your podcast recording). Click "proceed" to move on, and select your test podcast episode. Give them a valid email address and click "Send". In about 5-10 minutes you'll get an email with a link to a tool that lets you preview your episode transcript.
Once you get that email and click the link to go to their Transcript Editor tool, you'll see what looks like a word processor. You can press play to hear a preview of the transcript, and you can directly edit the transcript as well. You might find their tutorial video helpful here, too. Once you're happy with the transcript (or once you get impatient and decide "good enough"), you can select "Save" and then "Download". You'll be prompted with some options -- next to "Speaker names" and "Timestamps" select "Remove". This will give you just the text of the speech without any extra data. Then select "TXT" as the filetype and click "Download". You'll get a plain text file with the transcription of your speech, which you can go ahead and save.
Save it to the same directory where your mp3 is stored, with the same ID-as-name scheme. In the case of our episode "s01e03", the folder structure now looks like this:
/path/to/data/
/path/to/data/s01e03/s01e03.mp3
/path/to/data/s01e03/s01e03.txt
Alignment is the process of taking spoken word audio and transcript of that spoken word and coming up with precise timing data for each word. We use an open-source aligner called Gentle to do this. Go to the Gentle website and download the DMG for MacOS. When you run Gentle it will give you a small prompt, and you should then click "Open in browser". This will take you to a local alignment server that Gentle runs on its own. It looks like this:
Select your mp3 for your test episode, and then paste in the whole transcript from your associated txt file. Leave the default options unchecked and click "Align". This occurs in approximately 1x-5x real-time depending on the speed of your computer, so if you have a 60 minute episode, alignment might take ten minutes to an hour. Go have lunch or something.
When the alignment process is finished, you'll get a preview page that looks like this:
Left-click on the "JSON" link in the top right -- this will take you to a raw JSON rendering of all the alignment data. Save this as a .json file using our same ID naming scheme from above. Now your directory should look something like this:
/path/to/data/
/path/to/data/s01e03/s01e03.mp3
/path/to/data/s01e03/s01e03.txt
/path/to/data/s01e03/s01e03.json
There are two ways to give Shortcut the metadata for your episodes:
- manually specify everything in an
episodes.jsonfile (what the sample project does) - point Shortcut to your complete RSS feed with your entire show history
We highly recommend using the RSS method since it's one less manual step for you as Shortcut will automatically derive episode titles, publish dates, etc from the RSS. If you don't publish your full RSS history, please consider doing so! Even hundreds of episodes will come out to a sub-1MB file. However, we understand that some podcasts just can't do this, so we also provide the manual JSON method for those shows.
Open a text editor to an empty file and enter the following JSON for your test episode, with the appropriate metadata included:
[
{
"number": "s01e03",
"description": "This is the greatest single episode of spoken word anything ever recorded.",
"original_air_date": "2017-07-01T00:00:00",
"title": "My Test Episode"
}
]Save this in the Shortcut data directory as episodes.json. Now your directory should look like:
/path/to/data/
/path/to/data/episodes.json
/path/to/data/s01e03/s01e03.mp3
/path/to/data/s01e03/s01e03.txt
/path/to/data/s01e03/s01e03.json
In your server/.env file, set RSS_FEED to the URL of your full RSS feed, like so:
RSS_FEED=http://example.com/feed.xmlNote: if you have this set, your server will now locally serve your RSS feed from http://localhost:3000/rss. This endpoint returns a 404 if no feed (or a bad feed) is specified.
If you browse to http://localhost:8000/#/admin you will see an administration panel with every episode of your podcast.
You will be asked for a username and password. The username is admin, the default password is 1234. Use this for now while testing, then immediately change the password to something else by changing ADMIN_PASSWORD in your server/.env file. Please generate a secure password!
In the case of an RSS configuration you will see literally all the episodes; in the case of the episodes.json configuration you will see just the episodes you specified in that file.
Each episode has an on/off switch which should be set to "off" by default. Switch this to "on" for your test episode, then go back to http://localhost:8000 and reload the page if necessary to see the episode you just enabled listed on the home page.
At this point, you can click the episode but it will return an error because we have a couple more steps to do.
We use the HLS streaming audio format to enable Shortcut to work well in mobile web browsers. So we need to convert each episode to HLS. In the command line, in the directory where your episode's mp3 file lives, run the following ffmpeg command, replacing s01e03 with whatever your episode ID is:
ffmpeg -i s01e03.mp3 -map 0:a -profile:v baseline -level 3.0 -start_number 0 -hls_time 10 -hls_list_size 0 -hls_segment_filename 's01e03%03d.ts' -f hls s01e03.m3u8This will create many *.ts files (ten second audio chunks that get streamed) and a s01e03.m3u8 file that contains the metadata for all the streaming (what order to play the chunks in, etc). Your directory structure should now look something like:
/path/to/data/
/path/to/data/episodes.json
/path/to/data/s01e03/s01e03.mp3
/path/to/data/s01e03/s01e03.m3u8
/path/to/data/s01e03/s01e03.txt
/path/to/data/s01e03/s01e03.json
/path/to/data/s01e03/s01e03000.ts
/path/to/data/s01e03/s01e03001.ts
/path/to/data/s01e03/s01e03002.ts
...(etc)
Serve your files from an http server. This could be anywhere (including a web host) but it needs to be CORS-enabled. If you've already got apache, nginx, or something similar installed locally, just serve these files with CORS headers.
If you don't already have a local dev http server, one simple way to do that is by using http-server, a simple http server for Node.js. Just follow these steps in the terminal:
npm install -g http-server
cd /path/to/data/
http-server --corsThis will start a CORS-enabled http server at http://localhost:8080.
Edit client/src/config/dev.js and change dataBucket to http://localhost:8080/. Make sure to include the trailing slash!
Open up server/.env and change DATA_BUCKET to http://localhost:8080/. Make sure to include the trailing slash!
Run npm start to start your server up. Also make sure your client is running with npm run serve.
Go to the following URL:
http://localhost:3000/api/ABC123/update/s01e03 (ABC123 is your API_HASH variable set in server/.env. If you've already changed it, simply use your new API_HASH value.)
This is an API endpoint that will take the contents of s01e03.json and put the transcript data into a format that Shortcut understands. It will also analyze the corresponding s01e03.mp3 file and create waveform metadata. All this data will be put into a new file called s01e03-data.json and uploaded to s3.
It should take up to a few minutes and then display the success message.
At this point you should be able to browse to http://localhost:8000, select your episode, and make some clips!