Add support for video captions #910

douzebis · 2021-05-23T21:23:36Z

fix #239
(add subtitles for video feature #239)

Generate HTML5 subtitle tracks dynamically based on the contents of the folder
where the video is located.

For example: given a movie 'video.mkv', the player will look for VTT subtitle
files named 'video.XX.vtt' and '.video.XX.vtt', where XX is the two-letter
code for the language.

Signed-off-by: Frederic Ruget [email protected]

beardhatcode

Wow, you really reworked this, well done 👍.

Did you try this out on your instance, does it work well?

Converting a string to HTML like this patch suggests at the moment could be dangerous (XSS-attacks). Instead, I suggest you use v-for (see this link for more info)

If you make changes you can just do them here, I will rewrite the git history to tidy things up afterwards 🙂

src/components/Videos.vue

beardhatcode · 2021-05-23T22:24:10Z

Note to self: Things to check before merging when this is reworked:

handles files with emoji's in the name correctly
has tests in the video spec and in the naughty files spec
- I have no idea how to test this, I'm hoping there will be something like videoElement.captions or some value we can query form Plyr
check if this works on shared pages

douzebis · 2021-05-24T05:43:04Z

There is still a problem with the test Node / node12.x
I don't know what this test does.
Is it possible to run it on my dev setup?

beardhatcode · 2021-05-24T19:30:04Z

/compile fixup /

beardhatcode · 2021-05-24T19:33:27Z

That test means that you generated the JS in the wrong way (not with make build-js-production) or something like that (but its not clear from the docs how you should do that, so thats not your fault). I asked a bot to do it now.

beardhatcode · 2021-05-24T20:24:58Z

Cool it works 😄

But if I close the viewer and reopen it again it no longer works.We'll need to sort that out.

douzebis · 2021-05-24T20:45:13Z

Its weird.
I can reproduce this (the subtitles do not re-appear when I close and re-open the viewer), but only if the browser's debug window is closed (right-click 'inspect').
When the debug window is open, the subtitles re-appear as expected.

This needs to be investigated...

douzebis · 2021-05-24T22:44:10Z

Should be better now.
I believe the root of the problem is that 'tracks' was not properly declared and was not always reactive.

skjnldsv · 2021-05-25T06:14:32Z

Hey!

Thanks for diving into this!

EDIT: I mislooked the PR, I saw a glimpse at some code, but seemed to behave properly, ignore previous comment 😉

skjnldsv · 2021-05-25T06:20:24Z

src/components/Videos.vue

+					ko: 'Korean',
+					kr: 'Kanuri',
+					ks: 'Kashmiri',
+					ku: 'Kurdish',
+					kv: 'Komi',
+					kw: 'Cornish',
+					ky: 'Kirghiz',
+					la: 'Latin',
+					lb: 'Luxembourgish',
+					lg: 'Ganda',
+					li: 'Limburgan',
+					ln: 'Lingala',
+					lo: 'Lao',
+					lt: 'Lithuanian',
+					lu: 'Luba-Katanga',
+					lv: 'Latvian',
+					mg: 'Malagasy',
+					mh: 'Marshallese',
+					mi: 'Maori',
+					mk: 'Macedonian',
+					ml: 'Malayalam',
+					mn: 'Mongolian',
+					mr: 'Marathi',
+					ms: 'Malay',
+					mt: 'Maltese',
+					my: 'Burmese',
+					na: 'Nauru',
+					nb: 'Norwegian Bokmål',
+					nd: 'Ndebele, North',
+					ne: 'Nepali',
+					ng: 'Ndonga',
+					nl: 'Dutch',
+					nn: 'Norwegian Nynorsk',
+					no: 'Norwegian',
+					nr: 'Ndebele, South',
+					nv: 'Navajo',
+					ny: 'Nyanja',
+					oc: 'Occitan',
+					oj: 'Ojibwa',
+					om: 'Oromo',
+					or: 'Oriya',
+					os: 'Ossetian',
+					pa: 'Panjabi',
+					pi: 'Pali',
+					pl: 'Polish',
+					ps: 'Pushto',
+					pt: 'Portuguese',
+					qu: 'Quechua',
+					rm: 'Romansh',
+					rn: 'Rundi',
+					ro: 'Romanian',
+					ru: 'Russian',
+					rw: 'Kinyarwanda',
+					sa: 'Sanskrit',
+					sc: 'Sardinian',
+					sd: 'Sindhi',
+					se: 'Northern Sami',
+					sg: 'Sango',
+					si: 'Sinhala',
+					sk: 'Slovak',
+					sl: 'Slovenian',
+					sm: 'Samoan',
+					sn: 'Shona',
+					so: 'Somali',
+					sq: 'Albanian',
+					sr: 'Serbian',
+					ss: 'Swati',
+					st: 'Sotho',
+					su: 'Sundanese',
+					sv: 'Swedish',
+					sw: 'Swahili',
+					ta: 'Tamil',
+					te: 'Telugu',
+					tg: 'Tajik',
+					th: 'Thai',
+					ti: 'Tigrinya',
+					tk: 'Turkmen',
+					tl: 'Tagalog',
+					tn: 'Tswana',
+					to: 'Tonga',
+					tr: 'Turkish',
+					ts: 'Tsonga',
+					tt: 'Tatar',
+					tw: 'Twi',
+					ty: 'Tahitian',
+					ug: 'Uighur',
+					uk: 'Ukrainian',
+					ur: 'Urdu',
+					uz: 'Uzbek',
+					ve: 'Venda',
+					vi: 'Vietnamese',
+					vo: 'Volapük',
+					wa: 'Walloon',
+					wo: 'Wolof',
+					xh: 'Xhosa',
+					yi: 'Yiddish',
+					yo: 'Yoruba',
+					za: 'Zhuang',
+					zh: 'Chinese',
+					zu: 'Zulu',


This most likely already exists, but I can see where maintaining this list could be an issue.
We already maintain such list here: https://github.com/nextcloud/server/blob/9de329a4c2327767d86bd7f594b232eb56af0d01/resources/locales.json

We have a dedicated php method to get those ressources.
I'll suggest we use that instead (you can either have this injected in the page from a InitialState or maybe it's already available
https://github.com/nextcloud/server/blob/9de329a4c2327767d86bd7f594b232eb56af0d01/lib/public/L10N/IFactory.php#L88

Thanks @skjnldsv... so I would like to invoke L10N's php function 'findLanguageFromLocale()' from 'fetchTracks()'.
Please would you provide guidance as to how to import this function into Videos.vue?

public function findLanguageFromLocale(string $app = 'core', string $locale = null)

Why do you need the findLanguageFromLocale ?
All of the locale computation should be done on the php side

skjnldsv · 2021-05-25T06:23:33Z

src/components/Videos.vue

+				const davDir = this.davPath.replace(/[^/]*$/, '')
+				const videoRoot = this.basename.replace(/[.][^.]+$/, '')
+				// Create caption tracks for the HTML5 player
+				// E.g.: <file>.mkv: look for <file>.xx.vtt or .<file>.xx.vtt
+				const capTracks = []
+				for (const file of folder) {
+					const basename = file.basename
+					const index = basename.indexOf(videoRoot)
+					// Consider only file... or .file...
+					if (!(index === 0 || (index === 1 && basename[0] === '.'))) {
+						continue
+					}
+					const suffix = basename.slice(videoRoot.length + index)
+					// Consider only ...xx.vtt
+					if (suffix.search(/^[.]..[.]vtt$/) !== 0) {
+						continue
+					}
+					const lang = suffix.slice(1, 3)
+					const language = languages[lang] || lang
+					// an array of objects with src, label and lang keys
+					capTracks.push({
+						src: davDir + basename,
+						label: language,
+						srclang: lang,
+					})
+				}
+				this.tracks = capTracks


I'm wondering if that should not be better to have all of this in a dedicated php Controller that, given a specific fileName, that would return an array of LANG => filename.

I think the code would benefit from it, especially because the Locale list is managed by a php method.

Not sure how to do that (writing a dedicated php Controller).
Please would you point to a similar Controller in NextCloud source, which I could use as a template?

Yes, look our documentation about ocsControllers :)

https://docs.nextcloud.com/server/latest/developer_manual/basics/controllers.html
https://github.com/nextcloud/forms/blob/master/lib/Controller/ApiController.php

I think the controller method should:

Receive the filename as a parameter

Search for existing files with the same filename+langcode using the list of known locale retrieved via the findAvailableLocales of the L10N/IFactory using Dependency Injection

Return an array of langcode => filename so the viewer can use it and build the list accordingly. Should be much faster than our dav endpoint.

OK. I take the challenge :)

By the way, the current proposal does not use the dav endpoint to look for candidate vtt files -- I reckon getFileList() works with the local filesystem?
So we may not see a performance improvement with your proposal.
But in any case the code will be cleaner and easier to maintain.

reckon getFileList() works with the local filesystem?

getFileList is using the dav endpoint.

viewer/src/services/FileList.js

Line 37 in a5eec09

const response = await client.getDirectoryContents(fixedPath, Object.assign({

src/components/Videos.vue

skjnldsv · 2021-05-25T15:56:10Z

Btw, please change your target branch to master, and fix following potential conflicts 😬

Generate HTML5 subtitle tracks dynamically based on the contents of the folder where the video is located. For example: given a movie 'video.mkv', the player will look for VTT subtitle files named 'video.XX.vtt' and '.video.XX.vtt', where XX is the two-letter code for the language. Signed-off-by: Frederic Ruget <[email protected]>

Signed-off-by: Frederic Ruget <[email protected]>

Instead fetchTracks() can be called from options(). Signed-off-by: Frederic Ruget <[email protected]>

Signed-off-by: Frederic Ruget <[email protected]>

Signed-off-by: npmbuildbot-nextcloud[bot] <npmbuildbot-nextcloud[bot]@users.noreply.github.com>

Property track was not properly declared, and sometimes was not reactive. E.g. closing and re-opening the player would make subtitles disappear. Fixing entails declaring tracks in the data() function associated with the Videos vue. Signed-off-by: Frederic Ruget <[email protected]>

(Templates are not made to call methods.) Signed-off-by: Frederic Ruget <[email protected]>

douzebis · 2021-05-25T17:02:51Z

Btw, please change your target branch to master, and fix following potential conflicts 😬

OK, I just tested with master. The viewer builds OK.
Unfortunately I'm unable to test with my instance of NextCloud (stable 21):
"The files of the app Viewer (viewer) were not replaced correctly. Make sure it is a version compatible with the server."

I don't know how easy it would be for me to setup an additional instance of NextCloud for testing master... and I don't want to break my production instance :)

beardhatcode · 2021-05-25T17:07:02Z

@douzebis I rebased your changes on master (you might need to re-clone your repository, or git fetch; git reset origin/stable21)

beardhatcode · 2021-05-25T17:40:04Z

@douzebis you should just be able to take the built files of master and use them with stable21. Thats no problem (stable21 is practically the same as master). Does that work for you?

beardhatcode · 2021-05-25T18:34:16Z

One other thing you can do @douzebis is run the program ./cypress/start_server.sh that will start an extra Nextcloud server on your local machine with the curretn version of the viewer app at http://localhost:8082/ (you cna login as user "test" with pass "test", or as "admin" pass "admin" )

douzebis · 2021-05-27T20:25:42Z

Well I could not find time to work on the requested changes but I still managed to do a big mess by triggering github's 'fetch upstream' button... and updating my forked version of 'stable21' to 'master' :( :(

The last big 'push' is my attempt at reverting to previous stable21 state.

Please advise, I can re-fork from scratch and redo the modifications to src/components/Videos.vue, so that you no longer have to deal with this mess...

douzebis · 2021-05-27T20:31:54Z

@douzebis you should just be able to take the built files of master and use them with stable21. Thats no problem (stable21 is practically the same as master). Does that work for you?

Nope, unfortunately it does not work.
When navigating to the welcome portal (https://nextcloud.domain), nextcloud complains about:
"The files of the app Viewer (viewer) were not replaced correctly. Make sure it is a version compatible with the server."

This may have to do with the components version numbers in 'master' not being compatible with those in 'stable21'?

beardhatcode · 2021-05-27T20:36:19Z

@douzebis don't worry about it, just continue your work here, I'll rebase your changes when everything is done

beardhatcode · 2021-05-27T20:41:48Z

Alternatively, @douzebis , I can put this branch back on stable21 for you, just let me know.

skjnldsv · 2021-05-28T07:35:19Z

#915

skjnldsv · 2021-05-28T07:37:38Z

Moved and rebased to new branch here, will be much easier.
You can edit and continue your work there 😉

douzebis mentioned this pull request May 23, 2021

Add support for movie captions #907

Closed

beardhatcode requested changes May 23, 2021

View reviewed changes

src/components/Videos.vue Outdated Show resolved Hide resolved

src/components/Videos.vue Outdated Show resolved Hide resolved

src/components/Videos.vue Show resolved Hide resolved

src/components/Videos.vue Outdated Show resolved Hide resolved

This was linked to issues May 23, 2021

add subtitles for video feature #239

Open

Support for subtitles in the video player #879

Closed

douzebis requested a review from beardhatcode May 24, 2021 05:40

skjnldsv added 3. to review Waiting for reviews enhancement New feature or request labels May 25, 2021

skjnldsv reviewed May 25, 2021

View reviewed changes

src/components/Videos.vue Outdated Show resolved Hide resolved

skjnldsv reviewed May 25, 2021

View reviewed changes

src/components/Videos.vue Outdated Show resolved Hide resolved

skjnldsv added 2. developing Work in progress and removed 3. to review Waiting for reviews labels May 25, 2021

skjnldsv added this to the Nextcloud 22 milestone May 25, 2021

beardhatcode changed the base branch from stable21 to master May 25, 2021 16:36

beardhatcode changed the base branch from master to stable21 May 25, 2021 16:41

douzebis added 5 commits May 25, 2021 18:44

Fix typo in condition

088b678

Signed-off-by: Frederic Ruget <[email protected]>

Use v-for instead of v-html to create tracks

f2216ad

Signed-off-by: Frederic Ruget <[email protected]>

Simplify the code (no need for extra async function)

e1eacde

Signed-off-by: Frederic Ruget <[email protected]>

Get rid of v-html (and associated warning)

f681942

Instead fetchTracks() can be called from options(). Signed-off-by: Frederic Ruget <[email protected]>

douzebis and others added 7 commits May 25, 2021 18:46

Add js/viewer-main.js* (necessary??)

ce00b04

Signed-off-by: Frederic Ruget <[email protected]>

Clean up (cosmetic)

1362f92

Signed-off-by: Frederic Ruget <[email protected]>

Fix typo (lang -> srclang)

15d412e

Signed-off-by: Frederic Ruget <[email protected]>

Improve code readability -- use: for (item of items)

be25019

Signed-off-by: Frederic Ruget <[email protected]>

fixup! Improve code readability -- use: for (item of items)

9f05ce1

Signed-off-by: npmbuildbot-nextcloud[bot] <npmbuildbot-nextcloud[bot]@users.noreply.github.com>

Move fetchData() from template to methods

01fc82e

(Templates are not made to call methods.) Signed-off-by: Frederic Ruget <[email protected]>

beardhatcode force-pushed the stable21 branch from 262eb93 to 01fc82e Compare May 25, 2021 17:06

beardhatcode changed the base branch from stable21 to master May 25, 2021 17:06

MorrisJobke mentioned this pull request May 26, 2021

22.0.0 beta 2 nextcloud/server#27111

Merged

98 tasks

skjnldsv modified the milestones: Nextcloud 22, Nextcloud 23 May 26, 2021

douzebis added 2 commits May 27, 2021 20:10

Try and fix the mess I did with github 'fetch upstream'

a9cfee1

Try and fix the mess I did with github 'fetch upstream'

31cb81a

beardhatcode changed the base branch from master to stable21 May 27, 2021 20:36

beardhatcode changed the base branch from stable21 to master May 27, 2021 20:36

skjnldsv closed this May 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for video captions #910

Add support for video captions #910

douzebis commented May 23, 2021 •

edited by skjnldsv

Loading

beardhatcode left a comment •

edited

Loading

beardhatcode commented May 23, 2021 •

edited

Loading

douzebis commented May 24, 2021

beardhatcode commented May 24, 2021

beardhatcode commented May 24, 2021

beardhatcode commented May 24, 2021

douzebis commented May 24, 2021

douzebis commented May 24, 2021

skjnldsv commented May 25, 2021 •

edited

Loading

skjnldsv May 25, 2021

douzebis May 25, 2021

skjnldsv May 26, 2021

skjnldsv May 25, 2021

douzebis May 25, 2021

skjnldsv May 25, 2021

skjnldsv May 25, 2021 •

edited

Loading

douzebis May 25, 2021

skjnldsv May 26, 2021

skjnldsv commented May 25, 2021 •

edited

Loading

douzebis commented May 25, 2021

beardhatcode commented May 25, 2021 •

edited

Loading

beardhatcode commented May 25, 2021

beardhatcode commented May 25, 2021

douzebis commented May 27, 2021

douzebis commented May 27, 2021

beardhatcode commented May 27, 2021

beardhatcode commented May 27, 2021

skjnldsv commented May 28, 2021

skjnldsv commented May 28, 2021

Add support for video captions #910

Add support for video captions #910

Conversation

douzebis commented May 23, 2021 • edited by skjnldsv Loading

beardhatcode left a comment • edited Loading

Choose a reason for hiding this comment

beardhatcode commented May 23, 2021 • edited Loading

douzebis commented May 24, 2021

beardhatcode commented May 24, 2021

beardhatcode commented May 24, 2021

beardhatcode commented May 24, 2021

douzebis commented May 24, 2021

douzebis commented May 24, 2021

skjnldsv commented May 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skjnldsv May 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skjnldsv commented May 25, 2021 • edited Loading

douzebis commented May 25, 2021

beardhatcode commented May 25, 2021 • edited Loading

beardhatcode commented May 25, 2021

beardhatcode commented May 25, 2021

douzebis commented May 27, 2021

douzebis commented May 27, 2021

beardhatcode commented May 27, 2021

beardhatcode commented May 27, 2021

skjnldsv commented May 28, 2021

skjnldsv commented May 28, 2021

douzebis commented May 23, 2021 •

edited by skjnldsv

Loading

beardhatcode left a comment •

edited

Loading

beardhatcode commented May 23, 2021 •

edited

Loading

skjnldsv commented May 25, 2021 •

edited

Loading

skjnldsv May 25, 2021 •

edited

Loading

skjnldsv commented May 25, 2021 •

edited

Loading

beardhatcode commented May 25, 2021 •

edited

Loading