RSS configuration and a simple admin panel #96

dariusk · 2018-02-15T23:17:53Z

Major new feature which closes epic #76, including sub-issues #80, #81, #82, #90, #95

Creates a new RSS_FEED server .env variable. If this points to a valid RSS feed, it returns a JSON-parsed version of all the feed data if a GET request is made to http://localhost:3000/rss (in dev, replace with the server name as needed)
Creates a new useRss variable in client/src/config/base.js which tells the client to use the RSS data instead of accessing episodes.json for metadata
Creates an administration panel at http://localhost:8000/#/admin which shows all episodes specified in either episodes.json or the RSS feed. An administrator can provide credentials to access this panel and then enable/disable episodes from appearing on the front page as needed. Episodes are considered disabled by default. 86d2f8f
New ADMIN_PASSWORD server .env variable, which is the password for simple http auth for the admin panel. 0fddb27
New API endpoint at http://localhost:3000/admin/setEpisode. It's a POST and you must send guid (string) and enabled (bool) in the form body to enable/disable episodes by GUID
We now store a guid behind the scenes for each episode. In the case of the old-style episodes.json config, the guid is simply the episode number, since no two episodes will presumably share an episode number. However, if RSS configuration is enabled, it will use the item:guid tag from the RSS feed itself as the guid, and if no guid is specified it uses the canonical uri for the item
In the case of RSS configuration, Shortcut now infers episode numbers either from the itunes:episode and itunes:season tags in the RSS, or it numbers them on its own in chronological order by publication date.
Shortcut now uses flat-cache to store the small amount of persistent data needed to keep track of which episodes are enabled/disabled by the admin. This is stored in a file called adminData.json at the server root. 86d2f8f

Documentation follows. This will replace the Create the episode metadata section in the Importing Your Own Podcast wiki article.

Provide the episode metadata

There are two ways to give Shortcut the metadata for your episodes:

manually specify everything in an episodes.json file (what the sample project does)
point Shortcut to your complete RSS feed with your entire show history

We highly recommend using the RSS method since it's one less manual step for you as Shortcut will automatically derive episode titles, publish dates, etc from the RSS. If you don't publish your full RSS history, please consider doing so! Even hundreds of episodes will come out to a sub-1MB file. However, we understand that some podcasts just can't do this, so we also provide the manual JSON method for those shows.

Option 1: Manually specify episode data

Open a text editor to an empty file and enter the following JSON for your test episode, with the appropriate metadata included:

[
  {
    "number": "s01e03",
    "description": "This is the greatest single episode of spoken word anything ever recorded.",
    "original_air_date": "2017-07-01T00:00:00",
    "title": "My Test Episode"
  }
]

Save this in the Shortcut data directory as episodes.json. Now your directory should look like:

/path/to/data/
/path/to/data/episodes.json
/path/to/data/s01e03/s01e03.mp3
/path/to/data/s01e03/s01e03.txt
/path/to/data/s01e03/s01e03.json

Option 2: Point the server to your full RSS feed

In your server/.env file, set RSS_FEED to the URL of your full RSS feed, like so:

RSS_FEED=http://example.com/feed.xml

Note: if you have this set, your server will now locally serve your RSS feed from http://localhost:3000/rss. This endpoint returns a 404 if no feed (or a bad feed) is specified.

Enable your test episode

If you browse to http://localhost:8000/#/admin you will see an administration panel with every episode of your podcast.

You will be asked for a username and password. The username is admin, the default password is 1234. Use this for now while testing, then immediately change the password to something else by changing ADMIN_PASSWORD in your server/.env file. Please generate a secure password!

In the case of an RSS configuration you will see literally all the episodes; in the case of the episodes.json configuration you will see just the episodes you specified in that file.

Each episode has an on/off switch which should be set to "off" by default. Switch this to "on" for your test episode, then go back to http://localhost:8000 and reload the page if necessary to see the episode you just enabled listed on the home page.

At this point, you can click the episode but it will return an error because we have a couple more steps to do.

This enables us to get around any CORS issues with the podcast RSS feed. We just re-serve it from our server with CORS already enabled. By parsing the RSS serverside and serving it up as plain json, we no longer have to include a costly XML parsing step on the client. #80

There is now the option to point shortcut's client and server to an RSS feed URL instead of maintaining an `episodes.json` file with manual metadata. The admin sets the `useRSS` option in client/src/config/default.js to `true` and provides a feed URL in the server's .env file `RSS_FEED`. At every point in the code where `episodes.json` might be accessed for metadata about all episodes, the code now branches if RSS is enabled and instead parses the RSS feed. This closes #82.

There is now an admin panel that can be reached at the `/#/admin` route. The panel shows a list of episodes that Shortcut knows about, as well as their state as enabled or disabled (disabled by default). This state is stored in `server/adminData.json` on the server. This closes #81, because we no longer need a specific user-defined GUID to say "episode 3098281ABD23 is enabled" -- now we use the `guid` tag defined in the rss spec.

Fixes the following warning reported: npm WARN [email protected] No repository field.

Whilst Yeoman may have been used during initial, one-off generation of the project it is unlikelt to be re-run. Remove the configuration file from upstream repository. Unlikely to be run again as: (1) yeoman itself isn't a dependency of this project, and (2) low likelihood of re-running a project generator after v1.0.0 has been released without a major refactor.

Use new abstracted name in client/ documentation.

Update documentation and example commands in server/ to reflect new project name.

Documentation and package.json provide guidance to using nodemon to ease the development process. However, it was missing from devDependencies in server/package.json and a manual install instruction in server/README.md As npm has functionality to reflect dependencies that should not be run in --production (or when the NODE_ENV environment variable is set to production) then let's utilize that. This reduces the number of manual commands that must be run to setup the recommended shortcut development environment.

Rename the existing test to *Test.js, so that shortcut-client's test runner automatically finds the existing test for ClippingContainerComponent. This improves code coverage as measured by lines to 25.57% (from 23.56%). See below: BEFORE --------------------------------|----------|----------|----------|----------|----------------| File | % Stmts | % Branch | % Funcs | % Lines |Uncovered Lines | --------------------------------|----------|----------|----------|----------|----------------| ... src/components/ | 45.99 | 34.85 | 65.77 | 27.17 | | ClippingContainerComponent.js | 45.96 | 25.62 | 74.19 | 22.43 |... 441,442,444 | ... --------------------------------|----------|----------|----------|----------|----------------| All files | 42.71 | 35.54 | 62.18 | 23.56 | | --------------------------------|----------|----------|----------|----------|----------------| AFTER --------------------------------|----------|----------|----------|----------|----------------| File | % Stmts | % Branch | % Funcs | % Lines |Uncovered Lines | --------------------------------|----------|----------|----------|----------|----------------| ... src/components/ | 48 | 37.02 | 66.15 | 29.78 | | ClippingContainerComponent.js | 63.35 | 41.32 | 77.42 | 45.79 |... 441,442,444 | ... --------------------------------|----------|----------|----------|----------|----------------| All files | 44.25 | 37.16 | 62.95 | 25.57 | | --------------------------------|----------|----------|----------|----------|----------------|

Without this peer dependency of [email protected], the following error message is reported when running `npm install`: [email protected] /home/rhyskidd/Coding/shortcut/client └── UNMET PEER DEPENDENCY flexboxgrid@^6.3.0 npm WARN [email protected] requires a peer of flexboxgrid@^6.3.0 but none was installed. Reported with: $ npm --version 3.5.2 $ nodejs --version v6.11.4 For more on peer dependencies, see: https://stackoverflow.com/a/22004559/4206728

If no RSS_FEED is specified on the server, the `rss/` endpoint immediately returns a 404 instead of attempting a GET request. Relevant to #95.

Fixes a bug where two cache objects were created; we now create a single cache object on server init, and then pass it into `routes/all-episode-data.js`. Forcing the entire cache to update every time, since this is only ever called during occasional admin-only tasks. (all `cache.save()` calls are now passed `true`); Also updates the Admin panel to grab the full list of episodes completely unfiltered from the server (via a call to `/recent?filter=0`). Before this commit, disabling an episode from appearing on the front page would also disable it from appearing in the admin panel! (The unfiltered endpoint doesn't need to be secure since it's pulling from the canonical rss anyway.)

This commit makes the admin panel control what appears on the front page even if Shortcut is set up to use `episodes.json` as its metadata source instead of RSS.

therewasaguy

looks great, @dariusk!

Left some comments / questions, mostly minor things

therewasaguy · 2018-02-18T18:17:34Z

client/src/components/Main.js

    if (env === 'dev') {
+      let devUrl = useRSS ? apiEndpoint + '/rss' : apiEndpoint + '/recent';


It seems that the /rss and /recent endpoints return the same result. I'm probably missing something, but why do we need to be able to access both from the client (and are we only hitting the /rss endpoint in dev)? It'd be nice if we could get rid of this useRSS flag on the client so that only the server needs to know the datasource.

Good call. I had originally envisioned the /rss endpoint to be a simple local mirror of the RSS feed to get around CORS issues, but I think it makes sense to just keep /recent and have it render the rss data if that's enabled on the server. And then no useRSS flag!

Re: only hitting that endpoint in dev, that's technically true but what is really happening is in production, we render the front page using the marko template, which loads the episodes using the allEpisodeData function, which in turn checks for RSS and preloads those episodes, or falls back to the old explicit json method.

therewasaguy · 2018-02-18T18:23:37Z

server/package.json

    "striptags": "^2.1.1",
    "twit": "^2.2.9",
    "urlsafe-base64": "^1.0.0",
-    "waveform-util": "github:howlingeverett/waveform-util"
+    "waveform-util": "github:FeelTrainCoop/waveform-util"


therewasaguy · 2018-02-19T02:50:18Z

server/routes/all-episode-data.js

+    request.get({url: rssFeed}, function(err, resp, body) {
+      if (!err) {
+        let episodes = cache.getKey('episodes') || [];
+        console.log('AAAA',episodes);


we can remove this console.log (and the leftover one above on line 17 while at it, I think I'm to blame for that one 🙃 )

therewasaguy · 2018-02-19T02:56:06Z

client/src/components/Admin.js

+          </div>
+        </div>
+        <div className="content episodes">
+          <h3 className="recent-episodes">Enable/Disable Episodes</h3>


Users can see this before authenticating. Maybe that's ok cuz why would they be on this page in the first place? But we could wait until the withCredentials request above succeeds, then set component state to authenticated and render the text that says "Admin Panel" etc

Doing exactly this, good call.

therewasaguy · 2018-02-19T02:56:55Z

client/src/components/Admin.js

+          enabled: switches[index].checked
+        }
+      });
+      this.forceUpdate()


could we do this without forceUpdate? Maybe use the POST response to update the component state? I've read not to use forceUpdate, but probably not a big deal here and I do like how simple this component is otherwise.

Unfortunately I was unable to figure out the React lifecycle stuff to make an update happen in a way that made sense here without just doing a forceUpdate. I'm okay with it for now because it's admin-facing, but generally I wouldn't do it if it were a listener-facing feature.

therewasaguy · 2018-02-19T16:40:21Z

client/src/components/Admin.js

+        return episode;
+      });
+      this.setState({
+        switches: tempSwitches


maybe set "authenticated" here?

therewasaguy · 2018-02-19T16:53:15Z

server/README.md

- `docker build -t tal-server .`
- `docker run -p 3000:3000 -d tal-server`
+- `docker build -t shortcut-server .`
+- `docker run -p 3000:3000 -d shortcut-server`
 - server will be running at docker-machine ip, i.e. `http://192.168.99.100:3000/`

 ### Run with Node:


I think some of the es6 node requires 6.x or higher? It says 4.4.3 down here so we should update and maybe add engines to package.json

I deleted the line in question to avoid confusion

therewasaguy · 2018-02-19T16:55:06Z

server/package.json

  "author": {
    "name": "Darius Kazemi",
    "email": "[email protected]"
  },
  "license": "GPL-3.0",
+  "devDependencies": {
+    "nodemon": "^1.10.0"
+  },
  "dependencies": {


add { "engines" : { "node" : ">=6" } } here?

therewasaguy · 2018-02-19T17:00:11Z

server/routes/all-episode-data.js

+            // filter out inactive episodes specified in the .env file
+            .filter((episode) => !inactiveEpisodes.includes(episode.number))
+            // filter out inactive episodes (only include explicitly active episodes in the admin panel)
+            // unless we passed `true` to `episodes` in which case, include all episodes


when do we pass true instead of an array of episodes?

This stuff was left over from an earlier instance of the code, before I gave the option to return filtered or unfiltered episodes. Fixing now!

therewasaguy · 2018-02-19T17:32:48Z

server/routes/admin.js

+// return the state of enabled/disabled episodes
+// any episode not in this list is considered disabled by default
+router.get('/getEpisodes', function (req, res) {
+  return res.json(req.app.get('cache').getKey('episodes'));


do we want to check the RSS feed for new episodes here?

Otherwise, I'm not totally clear on is when/how we check the RSS feed for new episodes—it seems this only happens when we boo the server or if we're in development mode and hit /rss on the Landing page from the frontend

Hmmm, good point. As it stands the workflow would be: publish new episode, get your transcript in order, reboot the server, enable in admin panel. Probably shouldn't have to reboot the server every time. Maybe a button on the admin pane that says "refresh all episodes" which updates the episode cache from the live RSS.

Actually, better idea, and closer to what you suggested: just update the cache itself here, since it's an admin pane. Just need to make sure it happens after authentication.

@therewasaguy

* Explicitly declare Node.js compatibility in the package.json * Replace a console.log with a thrown Error object * Safety-check the parent object of a subobject before testing if the subobject exists From @therewasaguy's PR feedback.

Infinite scroll is overengineering for this particular UX element. Even the weakest smartphones can render ~1000 plaintext divs no problem, and while getting all the way down to the bottom might be a problem, as long as we don't provide a search feature (which is forthcoming) at least rendering all the episodes on one landing page lets a user ctrl+F or "find in page" on mobile to find the particular episode they are looking for. Even pagination (the usual alternative to infinite scroll) would make this impossible.

@therewasaguy

This commit makes it so we render a blank page on the Admin page until the user successfully authenticates. Previously we rendered an empty Admin panel but probably best to obfuscate all Admin functions prior rendering instead of giving an empty placeholder admin panel. (Per @therewasaguy's suggestion)

@therewasaguy

Per @therewasaguy's [review](#96 (comment)) the '/rss' and '/recent' endpoints return the same thing if you're set to RSS mode. So no reason to have a dedicated '/rss' endpoint, and also no more need for a `useRSS` configuration variable on the client. The client just asks the server for recent episodes, and the server knows if it needs to retrieve those from the RSS feed or from the JSON file.

Cleaned up some jshint warnings and updated moment which was throwing a bad warning.

The all-episode-data route used to have some gnarly logic built in but I removed that a while back. However, some confusing comments stayed in, and also some application logic that we didn't need was in there too. This commit removes those.

Accessing the admin panel will update the cache locally. It's just a GET request to the RSS feed (in the case of RSS), which we wouldn't want a normal user to intialize but is fine if an admin does it. The server endpoint is behind an auth gate so only an admin will ever kick it off.

dariusk and others added 19 commits December 30, 2017 16:33

security update per vulnerability alert

a7560d1

Add upstream repository to server/package.json

5cf456f

Fixes the following warning reported: npm WARN [email protected] No repository field.

client: Update client project name to abstract away from tal

5a8c678

Use new abstracted name in client/ documentation.

server: Update server project name to abstract away from tal

d377be7

Update documentation and example commands in server/ to reflect new project name.

Additional error state on RSS endpoint

7c5dbef

If no RSS_FEED is specified on the server, the `rss/` endpoint immediately returns a 404 instead of attempting a GET request. Relevant to #95.

test server response for text vs parsed array

60be559

Display title instead of guid on admin pane

adff6cf

Fix make sure cache is always initialized with empty array

7ed397e

Make admin panel compatible with episodes.json

a1a8f2d

This commit makes the admin panel control what appears on the front page even if Shortcut is set up to use `episodes.json` as its metadata source instead of RSS.

Setting RSS to false as default so Getting Started docs work simply

e8fe92b

Admin password controlled by .env file

0fddb27

dariusk requested a review from therewasaguy February 15, 2018 23:17

therewasaguy reviewed Feb 19, 2018

View reviewed changes

dariusk added 6 commits March 13, 2018 11:08

Small changes based on @therewasaguy's feedback

67ddd35

* Explicitly declare Node.js compatibility in the package.json * Replace a console.log with a thrown Error object * Safety-check the parent object of a subobject before testing if the subobject exists From @therewasaguy's PR feedback.

Remove confusing note

fbdaa59

Cleaning up warnings

7c1bc26

Cleaned up some jshint warnings and updated moment which was throwing a bad warning.

dariusk force-pushed the 80-enable-rss branch from bed0c62 to 7c1bc26 Compare March 14, 2018 18:57

dariusk added 2 commits March 14, 2018 12:05

Remove some old logic/comments

615dcd1

The all-episode-data route used to have some gnarly logic built in but I removed that a while back. However, some confusing comments stayed in, and also some application logic that we didn't need was in there too. This commit removes those.

Merge branch 'master' into 80-enable-rss

8fecf48

dariusk merged commit af99380 into master Mar 14, 2018

dariusk deleted the 80-enable-rss branch March 14, 2018 22:27

dariusk mentioned this pull request Mar 15, 2018

Create a simple admin panel to manage podcast content #56

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RSS configuration and a simple admin panel #96

RSS configuration and a simple admin panel #96

dariusk commented Feb 15, 2018 •

edited

Loading

therewasaguy left a comment

therewasaguy Feb 18, 2018

dariusk Mar 13, 2018

therewasaguy Feb 18, 2018

therewasaguy Feb 19, 2018

therewasaguy Feb 19, 2018

dariusk Mar 13, 2018

therewasaguy Feb 19, 2018

dariusk Mar 13, 2018

therewasaguy Feb 19, 2018

therewasaguy Feb 19, 2018

dariusk Mar 14, 2018

therewasaguy Feb 19, 2018

therewasaguy Feb 19, 2018

dariusk Mar 14, 2018

therewasaguy Feb 19, 2018

dariusk Mar 13, 2018

dariusk Mar 14, 2018

		if (env === 'dev') {
		let devUrl = useRSS ? apiEndpoint + '/rss' : apiEndpoint + '/recent';

RSS configuration and a simple admin panel #96

RSS configuration and a simple admin panel #96

Conversation

dariusk commented Feb 15, 2018 • edited Loading

Provide the episode metadata

Option 1: Manually specify episode data

Option 2: Point the server to your full RSS feed

Enable your test episode

therewasaguy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dariusk commented Feb 15, 2018 •

edited

Loading