Where to set $FFMPEG_PATH

[yyzguy]

The script for the default encoding, ffmpeg_edl_ac3.sh, suggests a path can be set for a different version of ffmpeg. Where is the appropriate place to set this variable? Modify the script? Somewhere else?

(Goal: use default script but use ffmpeg optimized for Apple Silicon installed with Homebrew, specifically to use GPUs)

From ffmpeg_edl_ac3.sh:

use dirname $0 to get path to ourselves unless $FFMPEG_PATH is set, in which case use that

[mackworth]

So there's a few possibilities.

1. <Edit: this is WRONG:> You can copy ffmpeg_edl_ac3.sh to the directory where your homebrew ffmpeg is, and then call it from a copy of the Format.
2. You could copy the script wherever you like and replace $(dirname "$0")/../MacOS/ffmpeg with /opt/homebrew/bin/ffmpeg
3. you could create a parent script that sets the path variable and then calls the sub script.

I'd personally do # 2.

What sort of speedup are you seeing? How does it compare to Handbrake?

[yyzguy]

I had to do a combination of option 1 and option 2. Option 1 by itself failed when it came to encoding. When I also changed the script to use ffmpeg_path=/opt/homebrew/bin/ffmpeg, it worked!

I didn't notice a significant change in encoding time between the homebrew version and the built-in version. I was hoping the homebrew version of ffmpeg for Apple Silicon would have used the GPU. It didn't.

I hadn't previously tried the Handbrake format. Indeed it appears to be much faster than ffmpeg. However, it too did not use the GPU.

Because it normally takes quite a bit of time to encode, I created a 5 minute show on the TiVo for testing. For a 5 minute show, it wasn't easy to note the start and stop time of the encoding process. The resulting mp4 files had a create time and modify time that were only 1 minute different, so I couldn't use those timestamps to accurately figure how much time the process took.

I changed the logging for Download to Detail. I was hoping to figure out the exact times for the encoding only, but I'm struggling to figure out which steps in the log would indicate the beginning and end of encoding. Can you tell me what to look at in the log file to determine the encoding time?

Perhaps I should put the log level back to normal. I'll give that a try tomorrow.

[yyzguy]

I just gave it another try, using a stopwatch to time the encoding.
Both the built-in ffmpeg and homebrew ffmpeg took about 1m30s to encode
Handbrake took about 45s to encode

For a 5 minute TiVo show, handbrake was about twice as fast as ffmpeg. Good to know.
Roughly encoding/ShowLenght = 1/6.67 with HandBrake. Pretty good!

Also ffmpeg appeared to use much more CPU than Handbrake. ffmpeg was around 730%, while HandBrake was around 130%. I have no idea how many cores were being used nor which cores (Performance, Efficiency or both).

[mackworth]

Sorry about the path thing, the dirname is specifically to find the ffmpeg at its known location versus cTiVo, not versus the script itself. I marked that as wrong above. Glad you figured it out.

For download timings, set Debug level to Major, and then open the log in Console. Then you can filter in the Search field on DL status, which will show you all of the transitions in the downloading process. "Encoding" will give you the start time and "Encoded" will give you the stop time. Console will also watch the file in real time (although you may have to refresh the search field to remove blank lines), so you don't have to keep opening the log.

FYI, stat -x file.mp4 will give you the create/last modify times including seconds.

I am surprised that the latest ffmpeg isn't faster. (see this article) Did you confirm that you've got version 6 (ffmpeg -version)? I know MacPorts is still doing version 4.x because 5.x breaks a lot of other apps.

Also which encoder did you specify? If you used the "Default" Format in cTiVo and only changed the Encoder to use field, then it will specify -vcodec h264, which is software based.

Note there's a lot of discussion about GPU acceleration and quality in cTiVo issues. Search for videotoolbox

[yyzguy]

Thanks for the tips. I'll do some more investigation, especially regarding videotoolbox. I found this additional article .

I initially hadn't used the HandBrake option because I was under the impression that HandBrake was merely a GUI for ffmpeg. I've seen several articles that suggest exactly that.

So, now I'm a bit confused on how HandBrake differs from ffmpeg. Handbrake definitely encoded about 50% faster than ffmpeg did.

Here's the output of my local homebrew copy of ffmpeg (version 7.0.2)?:
% ffmpeg -version
ffmpeg version 7.0.2 Copyright (c) 2000-2024 the FFmpeg developers built with Apple clang version 15.0.0 (clang-1500.3.9.4) configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/7.0.2-with-options --enable-shared --cc=clang --host-cflags= --host-ldflags= --enable-gpl --enable-libaom --enable-libdav1d --enable-libharfbuzz --enable-libmp3lame --enable-libopus --enable-libsnappy --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-demuxer=dash --enable-opencl --enable-audiotoolbox --enable-videotoolbox --enable-neon --disable-htmlpages libavutil 59. 8.100 / 59. 8.100 libavcodec 61. 3.100 / 61. 3.100 libavformat 61. 1.100 / 61. 1.100 libavdevice 61. 1.100 / 61. 1.100 libavfilter 10. 1.100 / 10. 1.100 libswscale 8. 1.100 / 8. 1.100 libswresample 5. 1.100 / 5. 1.100 libpostproc 58. 1.100 / 58. 1.100

[yyzguy]

Adding to my previous comment:
It does look like HandBrake indeed uses ffmpeg. Specifically, cTiVo's encoder "HB HW Default" calls HandBrakeCLI.sh script which appears to use ffmpeg with hardware acceleration.

[mackworth]

So, yes, HandBrake uses ffmpeg under the hood, but not calling the separate executable; they do use ffmpeg libraries but seem to be particularly clever about how they do it, e.g. multitasking, and GPU usage etc. They provide the HandBrakeCLI executable to do the same things from a command line. The CLI in cTiVo is built on ffmpeg 6.1.

My HandBrakeCLI.sh script adds the feature of supporting a cut-list for commercials, just like our ffmpeg_edl_ac3.sh (which it's based on) does for ffmpeg. The script primarily uses the HandBrakeCLI executable, but is quite complicated. Initially, it does call ffmpeg, but only to get header info from the video to determine duration and what kinds of audio streams are present. Then it calls handBrakeCLI multiple times to actually do the encoding of each non-commercial segment , then finally calls ffmpeg to merge the segments together. Or if it's not cutting commercials, then it just calls handBrakeCLI to do the encoding of the entire file.

And yes, I see that in April they released v7.0 including a "multi-threaded ffmpeg CLI". So even more I would have expected it to match HandBrake's speed.

I did a few tests myself, downloading and converting a 5 minute (152MB file) in nine different scenarios. Time in seconds below for each scenario. Note that HandBrake HW Default outputs as H265, so smaller and better quality. HW H264 I find unusably bad.

HW encoding (mostly H265)
52.2 Handbrake HW Default but with HandBrakeCLI 1.8.2
52.3 Handbrake HW Default (HandBrakeCLI 1.7.3; H265) unchanged
62.5 Default but with ffmpeg 7.0 and HW 264 (bad quality)
65.3 Default but with ffmpeg 7.0 and HW 265 (FYI incompatible with QuickTime Player, but ok in VLC)
72.3 Default but with HW H265 (file incompatible with QuickTime Player, but ok in VLC)

SW encoding (all H264)
130.8 Handbrake Default, but with HandBrake CLI 1.8.2 H264
137.0 "Handbrake Default" (handbrakeCLI 1.7.3 H264) unchanged
169.0 Default, but with ffmpeg 7.0 h264
181.0 "Default" (ffmpeg 4.4; h264) unchanged

So I'm not seeing much reason (at least performance-wise) to spend the day it takes just to build the static ffmpeg, and HandbrakeCLI HW still is best (and 1.8.2 hasn't improved it much.)

A couple of testing tips btw.
o You already figured out the first one: use a small show!
o Be sure to turn on Allow Duplicate Downloads in Adv Prefs to avoid having to delete the download entry every time.
o Set the filename template to:
["Test" / MainTitle " - " SeriesEpNumber | OriginalAirDate ["-" ExtraEpisode][" - " EpisodeTitle | Guests]]["Movies" / MainTitle " (" MovieYear ")"] [Format] [Options]. This appends the current Format and options (e.g. cutting commercials) to the file name, so if you download the same file with three different Formats, you can tell which was which.
o If you set Concurrent Encoders to 1, then it won't try and run two encoders at once, so you can queue up multiple downloads and let them all run. (It will overlap downloading with encoding, but downloading doesn't use much CPU, so encode times won't change much.)

Don't forget to turn the last two off when done testing.

[mackworth]

For any who may want to go down this rabbit hole or just who just want to run the latest ffmpeg or HandBrakeCLI, but using the Default scripts, attached are modified versions that should make it easy:

For HandBrakeCLI,

1. download the latest HandBrake CLI.
2. Open the DMG, and copy the HandBrakeCLI executable to a directory, e.g. Downloads.
3. Download and unzip the HandBrakeCLI.sh script and put in the same directory (required).
4. In cTiVo>Edit Formats Duplicate HB Default and rename to HB Default New CLI or something.
5. Change the Encoder field to point to the script, e.g. ~/Downloads/HandBrakeCLI.sh (note tilde).
6. Save and use your new Format.

For ffmpeg,

1. Install the latest ffmpeg using homebrew into the default location.
2. Download and unzip the revised ffmpeg_edl_ac3.sh script to a directory, e.g. Downloads.
3. In cTiVo>Edit Formats> Duplicate Default and rename to Default New ffmpeg or something.
4. Change the Encoder field to point to the script, e.g. ~/Downloads/ffmpeg_edl_ac3.sh (note tilde).
5. Save and use your new Format.

In both cases, updating to a future version (by download or homebrew) won't require any of those steps.

HandBrakeCLI.sh.zip
ffmpeg_edl_ac3.sh.zip

[yyzguy]

It appears you and I were doing a fair amount of testing simultaneously and a bit differently too. I wish I had seen some of your tips prior to running my own tests.

I'll post my results later, but I have a couple of comments/questions

When you were doing hardware encoding (h265), did you happen to look at Activity Monitor and look at GPU usage? In my testing, I didn't see any GPU activity. So, I'm not sure what's performing the HW encoding. (I'm using the regular M3 Apple Silicon on MacBook Air)

Your source file was 5MB ? For my 5 minute test clip, the TiVo reports the size as being .220 GB (220MB). I just performed a download and used the hidden Decrypt Only and received a 228MB .ts file. It appears to be mpeg-2.

For your results, what are the units you're showing? File size or perhaps encoding time? (e.g. your first result showing 52.2, Is that 52.2 seconds to encode or perhaps 52.2 KB for the file size.

Something you did, that I neglected to do was view the final output. I didn't bother looking at the file sizes nor did I actually do any playback to see the quality. (Big mistake, it's kind of the entire purpose of using cTiVo!. I got too involved with the technolgy and lost site of why I'm doing this exercise). The files are in my trash, but I'm not sure which is which anymore, as I didn't use your filename template (ARGH). I can see that there's files in two different size approximations. The smaller files are about half the size of the larger files.

I'll repeat the testing later or tomorrow and use your new shell scripts.

For testing, is there a way to not have to re-download the file each time? cTiVo appears to put the download file in a randomly named temporary file and deletes it when it's done. Perhaps there's way to keep a copy and re-use it rather than download each time. If so, I might actually use a larger test file.

Thank you for your extensive testing.

[mackworth]

Sorry; typo on 5Min, actually 152MB. (You can show more columns in the Shows window to see MB/show). And yes, that’s seconds for units.

No way to download from a file or cache prior versions. I started to build that long ago (some disabled code still there), but it gets very tricky in the multitasking and the metadata from the TiVo, and the only real use is testing like this.

one thing I have done when doing comprehensive speed testing is to save the decoded file and just run the same command line cTiVo does for the encoding (reported in the log) in a Terminal batch file. That way I could do dozens of tests.

In addition to file quality, note also the distinction between H265 (aka HEVC) and H264.

[mackworth]

Oh, and I found this anonymous factoid:

VTB doesn't use the GPU; never has used the main GPU shaders. It uses separate encoder hardware on the SoC.

[yyzguy]

Indeed the spec sheet for the M3 silicon mentions “video encode engine” but I haven’t found much information about it. https://support.apple.com/en-us/118551

[yyzguy]

Thank you so much for the new shell scripts. I’m hoping to get an opportunity to try them out today, but might not happen until tomorrow

[yyzguy]

one thing I have done when doing comprehensive speed testing is to save the decoded file and just run the same command line cTiVo does for the encoding (reported in the log) in a Terminal batch file. That way I could do dozens of tests.

I'm not seeing the command line in the log. Which logging option would show the command line?

[mackworth]

If you set TaskChain to Detail (in Adv Prefs) and run a download, it should show you the configured chain of all tasks, including their launchPath. Otherwise, it only shows the launch path in case of an error. Look for Task Name: encode

[yyzguy]

The new Handbrake shell script you provided worked great on my 5 minute test file.
When I used it on the show I actually care about (A 1.5 hour concert), the end result was a video file without any sound :(

Using your troubleshooting suggestion (getting the arguments from the log file), I discovered something I didn't expect:
Using the current HandBrakeCLI (version 1.8.2), the audio was dropped. The file info shows no audio track.

Using the version of HandBrakeCLI included with cTiVo (version 1.7.3), the audio came through as expected. The file info indeed shows "Stereo" which is the same as the source file.

Furthermore, the resulting file size is smaller for the version without audio (as to be expected).

This appears to be an issue with the 1.8.2 version of HandbrakeCLI. For some reason it's not capturing the audio from SOME sources. It was able to grab the audio from my test file, so I have to assume TiVo has different ways of encoding the audio, depending on the original source material.

We covered alot of ground in this thread. I appreciate your work on this. You've given me a nice set of tools for downloading my concerts from TiVo and converting them into a modern format, using hardware encoding. Indeed Apple Silicon is doing hardware encoding with its video encoding engine. I'm not sure what to make of the problem with the latest HandBrakeCLI but that's not a problem with cTiVo.

I have a bunch of concerts saved on my TiVo going back 12 years that I want to download and save on a Mac. My current TiVo Romeo is 10 years old and I sure don't want to lose my concert collection.

Thank you so much!

[mackworth]

You can see in the Finder>Get Info exactly which streams are in the file. And there'a an app called MediaInfo to see the specifics about each stream. The script looks inside each file and handles audio differently based on what's there (it tries to pass through AC3, but also create a mix down stereo, but you can control each of those by command line.)

My guess is that it isn't the length of the clip, but different kinds of audio in each stream and that something changed in the command line calls for audio streams. I'd love to know what the change is to fix the script accordingly. Note that it logs the exact audio parameters to HandbrakeCLI in the subprocess log file (in logs/handbrake.log in the temp files directory) under Audio: . That file is deleted at the end of encoding, but you can see it while encoding is in process, OR turn on Don't delete temp files (for debug only) in Adv Prefs, OR by canceling a download while debug level is on Detail.

[yyzguy]

You can see in the Finder>Get Info exactly which streams are in the file.
Finder>Get Info is wonky when it comes to finding the audio information. When a video is downloaded with cTiVo using "Decrypted Tivo" some end up with a .ts extension some with .mpg extension.

When the show had a .ts extension, Get Info doesn't show any audio information. When using Get Info on an .mpg file, it does show audio information.

Merely changing the .ts extension to .mpg (no changes to the actual file), GetInfo does show Audio information, the same as a native .mpg file. So whatever MacOS is doing GetInfo seems to depend on the file extension.

There'a an app called MediaInfo to see the specifics about each stream.

What a fantastic resource. It shows way more information than GetInfo. Thank you.

Indeed my 5 minute test files and the concert have different audio formats, which explains the differences on the final output.

I'd love to know what the change is to fix the script accordingly.

I'm starting to get a bit confused and overwhelmed with all the various test, using different versions of HB, HBcli, along with all the logs, different codecs and debugging information.

If you are still interested in debugging and troubleshooting your newer script I'd be happy to upload the decrypted Tivo files to my Google Drive and post a link where they can be downloaded. Otherwise, I'm pleased the built-in "HB HW Default" for its speed, use of Apple's media encode engine and the output quality.

To summarize:

For the concert that's been giving me trouble, the older HandbrakeCLI (1.7.3) which is included with cTiVo works the best and includes audio. The newer HandBrakeCLI (1.8.2) along with the update script lost the audio on the concert file.

I have several other concerts I'd like to download and convert. I'll start with using the built-in "HB HW Default" and with any luck, that will be the solution.

When I started this little project, I had know idea what we'd be getting into and I really appreciate the effort you've put in to help me.

[mackworth]

You’re welcome, and glad the HB Default HW is working for you. It’s my choice in general. But I would like to know what’s wrong with the new version of HandBrakeCLI

I'd be happy to upload the decrypted Tivo files to my Google Drive and post a link where they can be downloaded

Please do!

FYI, .ts is for Transport Stream, very similar to an MPG file, but not quite.

[yyzguy]

Here's the link to a Google Drive folder containing two files.

Update: Google Tries to process the video files so they can be streamed without downloading. I had to gzip the two files and create a new link. If you were relying on a previous version of this post, I had to change the link.

This one should work. [[Link deleted]]

The 5 minute test file is called joe.ts. It converts without any difficulty

The concert is an .mpg file called stones.mpg.

I've shortened the names to make command lines less complicated than they would otherwise be.

It's quite possible the mpg file was transferred from a previous TiVo. That would have been about 10 years ago, so I can't recall for certain. Might that explain why it's an mpg vs ts?

I tried using the GUI version of HandBrake to convert the file and it crashes upon opening it. I've opened a support case with HandBrake developers along with the application log. I have other TiVo files that crash in the same way. The test file doesn't cause a crash, so it's a problem with certain files only.

Please let me know when you have the files so I can stop sharing them. By default anyone with the link has read-only access.

[mackworth]

Thanks! Ihave both videos.

[yyzguy]

Thank you deleting the link for me. I have also removed the file from Google Drive.

[yyzguy]

Me, yesterday:

I tried using the GUI version of HandBrake to convert the file and it crashes upon opening it. I've opened a support case with HandBrake developers along with the application log.

I opened a support ticket with HandBrake and they found the problem and fixed it. I am able to open the troublesome files without having a crash.

Here's a link to the ticket including a link to their new snapshot. The snapshot includes updated version of the HandBrakeCLI as well. I'm really not sure if the crash was an issue with the GUI or something within their CLI (They didn't provide an explanation of what they did to fix the problem)

If you encounter crashes/errors with the CLI, might want to give the snapshot a try.

I'm impressed with how quickly they were able to identify and fix the problem.

https://forum.handbrake.fr/viewtopic.php?p=206995#p206995