Get from AudioInputStream to MediaAudio

[Wattful]

My goal is to get from javax.sound.sampled.AudioInputStream to humble MediaAudio.
I thought that MediaAudioConverter would be useful as its javadoc states that it "Converts from Humble MediaAudio to byte arrays that can be used in Java's sound system and back." However, there is no method defined in the interface that can convert from java audio to MediaAudio.

Is there any way to get from AudioInputStream to MediaAudio? Am I missing something obvious?

[javaspeak]

@Wattful did you resolve your issue?

Also does anyone know if humble-video can record both audio and webcam and create a single video file (with audio and video in it) e.g. mp4?

[Wattful]

I never resolved my issue, ended up using FFMpeg.

[javaspeak]

@Wattful thanks for your answer - did you use a wrapper library that wrapped FFMpeg like:

• https://github.com/a-schild/jave2
• https://github.com/bytedeco/javacv/
• https://github.com/bramp/ffmpeg-cli-wrapper

or did you use something like Runtime.exec(..) and execute ffmpeg command line tool directly?

I am super interested in knowing the best way to record audio and video and save it in a single file instead of 2 separate files.

I am using Java for the rest of the app, so a Java wrapper solution or Java solution would be good. I do not understand why the JDK does not provide such basic functionality in the jdk / jvm itself instead of depending on these third party libraries. Maybe it is a licensing issue - I do not know. However, if web browsers can record video and audio then so should it be an easy matter for the jdk.

Thank you for your time

[javaspeak]

@Wattful

This one:

 https://trashcode.io/post/d/Javacv-FFmpeg-merge-videomp4-and-audiomp3;jsessionid=2F8FAA2B817B6275B22CAE45356E0620

merges a mp3 file and a mp4 file into a mp4 file. It is using JavaCV as the wrapper library for FFMpeg

For the use case of recording from webcam and mic I am not sure if I should

(i) Record the webcam and create a mp4. And record audio separately into another file like mp3 and then finally merge the audio and video files together.

or (ii) if there is a way to record video frames and audio samples simultaneously and create a mp4 in one go.

Do you have any experience of what is possible @Wattful with library code?

[Wattful]

It turns out that our use cases are extremely similar. I also used FFMpeg to merge audio and video into a single file, although mine was pre-generated as opposed to live.

The tool that I wrote is open-source, you can check it out for yourself: https://github.com/Wattful/AskReddit-video/. Specifically this line is where I merge audio and video together. As you can see, I used FFMpeg command line with Runtime.exec().

Out of the two strategies you propose, I used strategy one (two wouldn't have made sense because my input wasn't live). I'm not sure if number two is possible using FFMpeg, and if it is, it's probably much harder to work with than strategy one.

[javaspeak]

@Wattful Thank you for your answer.

I have included this comment here for the benefit of others.

I came across this example:

https://github.com/bytedeco/javacv/blob/master/samples/WebcamAndMicrophoneCapture.java

The above example is about recording live video from the webcam and audio from the mic and producing a mp4 file. It is using javacv which internally uses stuff like FFMPeg.

It has permission issues with jdk 14 and MacOS Catalina. However it works with jdk 16 if you run your IDE from the terminal.

My IDE is STS (based on eclipse). I think it is missing a camera permission in its info.plist.

If I run the IDE from the terminal. it is probably looking at a different info.plist file in the terminal, which has the correct entries for Catalina. Catalina introduced security for the webcam which has stopped past stuff working.

This is the version of Java which worked for me with the above link.

 java 16.0.1 2021-04-20
 Java(TM) SE Runtime Environment (build 16.0.1+9-24)
 Java HotSpot(TM) 64-Bit Server VM (build 16.0.1+9-24, mixed mode, sharing)

For the above link example my pom.xml file looked like:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.spotadev.javacv</groupId>
<artifactId>javacv_exp</artifactId>
<version>1.0-RELEASE</version>
<packaging>jar</packaging>
<name>webcam_exp</name>
<description>javacv experiment to see if can record webcam and mic at the same time</description>

<repositories>

    <!--
    <repository>
        <id>Sonatype OSS Snapshot Repository</id>
        <url>http://oss.sonatype.org/content/repositories/snapshots</url>
    </repository>
    -->

    <repository>
        <id>mavenCentral()</id>
        <url>https://maven.ecs.soton.ac.uk/content/groups/maven.openimaj.org/</url>
    </repository>
</repositories>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <java.version>1.16</java.version>
    <maven.compiler.source>1.16</maven.compiler.source>
    <maven.compiler.target>1.16</maven.compiler.target>
</properties>

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.4.5</version>
</parent>

<dependencies>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
        <exclusions>
            <exclusion>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-logging</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>org.testng</groupId>
        <artifactId>testng</artifactId>
        <version>7.3.0</version>
        <scope>test</scope>
    </dependency>
    
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-log4j2</artifactId>
    </dependency>
    
    <dependency>
        <groupId>org.bytedeco</groupId>
        <artifactId>javacv-platform</artifactId>
        <version>1.5.5</version>
    </dependency>
    
</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.8.0</version>
            <configuration>
                <source>16</source> 
                <target>16</target>
            </configuration>
        </plugin>
    </plugins>
</build>
</project>

I have extra java spring boot dependencies that are not needed for the example. I added them because I like dependency injection.

I am rewriting the example in the above link so it is more readable and is not hard coded to a specific webcam or mic.