Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for video captions #910

Closed
wants to merge 14 commits into from
Closed
Show file tree
Hide file tree
Changes from 12 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions js/viewer-main.js

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion js/viewer-main.js.map

Large diffs are not rendered by default.

239 changes: 239 additions & 0 deletions src/components/Videos.vue
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,13 @@
@canplay="doneLoading"
@loadedmetadata="onLoadedMetadata">

<track v-for="track in tracks"
:key="track"
:src="track.src"
:label="track.label"
kind="captions"
:srclang="track.srclang">

<!-- Omitting `type` on purpose because most of the
browsers auto detect the appropriate codec.
Having it set force the browser to comply to
Expand All @@ -58,6 +65,8 @@
import Vue from 'vue'
import VuePlyr from '@skjnldsv/vue-plyr'
import '@skjnldsv/vue-plyr/dist/vue-plyr.css'
import { extractFilePaths } from '../utils/fileUtils'
import getFileList from '../services/FileList'

const liveExt = ['jpg', 'jpeg', 'png']
const liveExtRegex = new RegExp(`\\.(${liveExt.join('|')})$`, 'i')
Expand All @@ -67,6 +76,12 @@ Vue.use(VuePlyr)
export default {
name: 'Videos',

data() {
// This is required so that tracks is declared and reactive
// Otherwise updates may fail to make it to plyr
return { tracks: [], }
},

computed: {
livePhoto() {
return this.fileList.find(file => {
Expand All @@ -85,6 +100,7 @@ export default {
options() {
return {
autoplay: this.active === true,
captions: { active: false, language: 'auto', update: true },
controls: ['play-large', 'play', 'progress', 'current-time', 'mute', 'volume', 'captions', 'settings', 'fullscreen'],
loadSprite: false,
}
Expand Down Expand Up @@ -119,8 +135,231 @@ export default {
},

onLoadedMetadata() {

this.fetchTracks()
this.updateVideoSize()
},

fetchTracks() {
const dirPath = extractFilePaths(this.filename)[0]
getFileList(dirPath).then(folder => {
// ISO code for languages
// See https://www.loc.gov/standards/iso639-2/php/code_list.php
const languages = {
aa: 'Afar',
ab: 'Abkhazian',
ae: 'Avestan',
af: 'Afrikaans',
ak: 'Akan',
am: 'Amharic',
an: 'Aragonese',
ar: 'Arabic',
as: 'Assamese',
av: 'Avaric',
ay: 'Aymara',
az: 'Azerbaijani',
ba: 'Bashkir',
be: 'Belarusian',
bg: 'Bulgarian',
bh: 'Bihari languages',
bi: 'Bislama',
bm: 'Bambara',
bn: 'Bengali',
bo: 'Tibetan',
br: 'Breton',
bs: 'Bosnian',
ca: 'Catalan',
ce: 'Chechen',
ch: 'Chamorro',
co: 'Corsican',
cr: 'Cree',
cs: 'Czech',
cu: 'Church Slavic',
cv: 'Chuvash',
cy: 'Welsh',
da: 'Danish',
de: 'German',
dv: 'Divehi',
dz: 'Dzongkha',
ee: 'Ewe',
el: 'Greek',
en: 'English',
eo: 'Esperanto',
es: 'Spanish',
et: 'Estonian',
eu: 'Basque',
fa: 'Persian',
ff: 'Fulah',
fi: 'Finnish',
fj: 'Fijian',
fo: 'Faroese',
fr: 'French',
fy: 'Western Frisian',
ga: 'Irish',
gd: 'Gaelic',
gl: 'Galician',
gn: 'Guarani',
gu: 'Gujarati',
gv: 'Manx',
ha: 'Hausa',
he: 'Hebrew',
hi: 'Hindi',
ho: 'Hiri Motu',
hr: 'Croatian',
ht: 'Haitian',
hu: 'Hungarian',
hy: 'Armenian',
hz: 'Herero',
ia: 'Interlingua',
id: 'Indonesian',
ie: 'Interlingue',
ig: 'Igbo',
ii: 'Sichuan Yi',
ik: 'Inupiaq',
io: 'Ido',
is: 'Icelandic',
it: 'Italian',
iu: 'Inuktitut',
ja: 'Japanese',
jv: 'Javanese',
ka: 'Georgian',
kg: 'Kongo',
ki: 'Kikuyu',
kj: 'Kuanyama',
kk: 'Kazakh',
kl: 'Kalaallisut',
km: 'Central Khmer',
kn: 'Kannada',
ko: 'Korean',
kr: 'Kanuri',
ks: 'Kashmiri',
ku: 'Kurdish',
kv: 'Komi',
kw: 'Cornish',
ky: 'Kirghiz',
la: 'Latin',
lb: 'Luxembourgish',
lg: 'Ganda',
li: 'Limburgan',
ln: 'Lingala',
lo: 'Lao',
lt: 'Lithuanian',
lu: 'Luba-Katanga',
lv: 'Latvian',
mg: 'Malagasy',
mh: 'Marshallese',
mi: 'Maori',
mk: 'Macedonian',
ml: 'Malayalam',
mn: 'Mongolian',
mr: 'Marathi',
ms: 'Malay',
mt: 'Maltese',
my: 'Burmese',
na: 'Nauru',
nb: 'Norwegian Bokmål',
nd: 'Ndebele, North',
ne: 'Nepali',
ng: 'Ndonga',
nl: 'Dutch',
nn: 'Norwegian Nynorsk',
no: 'Norwegian',
nr: 'Ndebele, South',
nv: 'Navajo',
ny: 'Nyanja',
oc: 'Occitan',
oj: 'Ojibwa',
om: 'Oromo',
or: 'Oriya',
os: 'Ossetian',
pa: 'Panjabi',
pi: 'Pali',
pl: 'Polish',
ps: 'Pushto',
pt: 'Portuguese',
qu: 'Quechua',
rm: 'Romansh',
rn: 'Rundi',
ro: 'Romanian',
ru: 'Russian',
rw: 'Kinyarwanda',
sa: 'Sanskrit',
sc: 'Sardinian',
sd: 'Sindhi',
se: 'Northern Sami',
sg: 'Sango',
si: 'Sinhala',
sk: 'Slovak',
sl: 'Slovenian',
sm: 'Samoan',
sn: 'Shona',
so: 'Somali',
sq: 'Albanian',
sr: 'Serbian',
ss: 'Swati',
st: 'Sotho',
su: 'Sundanese',
sv: 'Swedish',
sw: 'Swahili',
ta: 'Tamil',
te: 'Telugu',
tg: 'Tajik',
th: 'Thai',
ti: 'Tigrinya',
tk: 'Turkmen',
tl: 'Tagalog',
tn: 'Tswana',
to: 'Tonga',
tr: 'Turkish',
ts: 'Tsonga',
tt: 'Tatar',
tw: 'Twi',
ty: 'Tahitian',
ug: 'Uighur',
uk: 'Ukrainian',
ur: 'Urdu',
uz: 'Uzbek',
ve: 'Venda',
vi: 'Vietnamese',
vo: 'Volapük',
wa: 'Walloon',
wo: 'Wolof',
xh: 'Xhosa',
yi: 'Yiddish',
yo: 'Yoruba',
za: 'Zhuang',
zh: 'Chinese',
zu: 'Zulu',
Comment on lines +148 to +332
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This most likely already exists, but I can see where maintaining this list could be an issue.
We already maintain such list here: https://github.com/nextcloud/server/blob/9de329a4c2327767d86bd7f594b232eb56af0d01/resources/locales.json

We have a dedicated php method to get those ressources.
I'll suggest we use that instead (you can either have this injected in the page from a InitialState or maybe it's already available
https://github.com/nextcloud/server/blob/9de329a4c2327767d86bd7f594b232eb56af0d01/lib/public/L10N/IFactory.php#L88

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @skjnldsv... so I would like to invoke L10N's php function 'findLanguageFromLocale()' from 'fetchTracks()'.
Please would you provide guidance as to how to import this function into Videos.vue?

public function findLanguageFromLocale(string $app = 'core', string $locale = null)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need the findLanguageFromLocale ?
All of the locale computation should be done on the php side

}
const davDir = this.davPath.replace(/[^/]*$/, '')
const videoRoot = this.basename.replace(/[.][^.]+$/, '')
// Create caption tracks for the HTML5 player
// E.g.: <file>.mkv: look for <file>.xx.vtt or .<file>.xx.vtt
const capTracks = []
for (const file of folder) {
const basename = file.basename
const index = basename.indexOf(videoRoot)
// Consider only file... or .file...
if (!(index === 0 || (index === 1 && basename[0] === '.'))) {
continue
}
const suffix = basename.slice(videoRoot.length + index)
// Consider only ...xx.vtt
if (suffix.search(/^[.]..[.]vtt$/) !== 0) {
continue
}
const lang = suffix.slice(1, 3)
const language = languages[lang] || lang
// an array of objects with src, label and lang keys
capTracks.push({
src: davDir + basename,
label: language,
srclang: lang,
})
}
this.tracks = capTracks
beardhatcode marked this conversation as resolved.
Show resolved Hide resolved
Comment on lines +334 to +360
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if that should not be better to have all of this in a dedicated php Controller that, given a specific fileName, that would return an array of LANG => filename.

I think the code would benefit from it, especially because the Locale list is managed by a php method.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how to do that (writing a dedicated php Controller).
Please would you point to a similar Controller in NextCloud source, which I could use as a template?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

@skjnldsv skjnldsv May 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the controller method should:

  1. Receive the filename as a parameter
  2. Search for existing files with the same filename+langcode using the list of known locale retrieved via the findAvailableLocales of the L10N/IFactory using Dependency Injection
  3. Return an array of langcode => filename so the viewer can use it and build the list accordingly. Should be much faster than our dav endpoint.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I take the challenge :)

By the way, the current proposal does not use the dav endpoint to look for candidate vtt files -- I reckon getFileList() works with the local filesystem?
So we may not see a performance improvement with your proposal.
But in any case the code will be cleaner and easier to maintain.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reckon getFileList() works with the local filesystem?

getFileList is using the dav endpoint.

const response = await client.getDirectoryContents(fixedPath, Object.assign({

})
},
},
}
</script>
Expand Down