Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add opus custom support #18

Open
wants to merge 41 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 35 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
b96e0ad
Enable Opus CUSTOM_MODES
emlynmac Aug 23, 2021
144063f
Add comma
emlynmac Aug 23, 2021
fb5c31c
Add CELT headers / source
emlynmac Aug 23, 2021
04fa515
Include base header / source makefiles
emlynmac Aug 23, 2021
db32719
Wrap variadic ctl functions for swift to import
emlynmac Aug 23, 2021
d1be9a6
Wrap up some variadic functions
emlynmac Aug 23, 2021
80f7e0f
rename to copuswrapper
emlynmac Aug 23, 2021
dc5f841
Move external header to export directory
emlynmac Aug 23, 2021
4e452fd
Add custom mode extensions to support Jamulus
emlynmac Mar 18, 2022
6104386
Add jamulus custom configuration
emlynmac Mar 18, 2022
7273180
Enable custom encoder / decoders to create higher level wrapper class…
emlynmac Mar 18, 2022
40be05d
Enable access outside the module
emlynmac Mar 18, 2022
593766d
Add custom decode / encode functions and contain frameSize
emlynmac Mar 20, 2022
605b9ec
Fix up encoder bug
emlynmac Mar 21, 2022
fefebd7
Set frame size in encode
emlynmac Mar 21, 2022
3204c1a
Generalize the ioctl call
emlynmac Mar 21, 2022
b0c9e3b
make the framesize public
emlynmac Mar 21, 2022
c898e20
Enable sample count to be passed in as a multiplier of frameSize
emlynmac Mar 31, 2022
61b6a4b
Rename method parameter to reflect correct usage
emlynmac Apr 1, 2022
9fa606f
Ensure error code is passed back
emlynmac Apr 3, 2022
21164e1
Add explicit return
emlynmac Apr 3, 2022
bada8cf
Allocate a buffer properly
emlynmac Apr 10, 2022
ed39aac
Handle the case of null data, aka packet loss
emlynmac Apr 14, 2022
2c72a72
Add documentation; add some safety
emlynmac Apr 18, 2022
3ff24bf
Merge branch 'alta:main' into main
emlynmac Apr 19, 2022
566f056
Merge branch 'main' into jamulus-coders
emlynmac Apr 19, 2022
b41d414
Tidy some changes prior to PR back to fork source
emlynmac Apr 19, 2022
40fb047
More whitespace formatting
emlynmac Apr 19, 2022
e3788e3
Update submodule to correct (master branch version)
emlynmac Apr 19, 2022
8b656b3
Apply swiftformat rules
emlynmac Apr 19, 2022
e7cb649
Revert swiftformat changes that cause changes to master
emlynmac Apr 19, 2022
a7f8272
One more swiftformat change revert
emlynmac Apr 19, 2022
a5b29aa
Change return type on ctl wrapper method
emlynmac Apr 20, 2022
712b0ff
Merge opus custom with existing en/de-coders
emlynmac Apr 20, 2022
33c68f4
Apply swiftformat
emlynmac Apr 20, 2022
033520f
Rename to customFrameSize
emlynmac Apr 21, 2022
aa3db6f
One more rename
emlynmac Apr 21, 2022
42d044f
Ensure that the compressed size is actually passed down
emlynmac Apr 21, 2022
f5f8735
Always build optimized builds of Opus
emlynmac Jan 3, 2023
00d42cf
Merge branch 'main_upstream'
emlynmac May 8, 2023
a48104d
Merge branch 'main' into jamulus-coders
emlynmac May 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
.build
.DS_Store
DerivedData
.swiftpm
15 changes: 14 additions & 1 deletion Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@ let package = Package(
name: "Copus",
targets: ["Copus"]
),
.library(
name: "Copuswrapper",
targets: ["Copuswrapper"]
),
.library(
name: "Opus",
targets: ["Opus", "Copus"]
Expand Down Expand Up @@ -88,6 +92,7 @@ let package = Package(
.headerSearchPath("silk/float"),

.define("OPUS_BUILD"),
.define("CUSTOM_MODES"),
.define("VAR_ARRAYS", to: "1"),
.define("FLOATING_POINT"), // Enable Opus floating-point mode

Expand All @@ -105,9 +110,17 @@ let package = Package(
.define("HAVE_UNISTD_H", to: "1"),
]
),
.target(
name: "Copuswrapper",
dependencies: ["Copus"],
publicHeadersPath: "include",
cSettings: [
.headerSearchPath("."),
]
),
.target(
name: "Opus",
dependencies: ["Copus"]
dependencies: ["Copus", "Copuswrapper"]
),
.testTarget(
name: "OpusTests",
Expand Down
11 changes: 11 additions & 0 deletions Sources/Copuswrapper/include/variadic-wrapper.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#ifndef __OPUS_VARIADIC_WRAPPER_H__
#define __OPUS_VARIADIC_WRAPPER_H__

//#include ""
#include <opus_defines.h>
#include <opus_custom.h>

int opus_custom_encoder_ctl_wrapper(OpusCustomEncoder *OPUS_RESTRICT st, int request, opus_int32 val);
int opus_custom_decoder_ctl_wrapper(OpusCustomDecoder *OPUS_RESTRICT st, int request, opus_int32 val);

#endif
11 changes: 11 additions & 0 deletions Sources/Copuswrapper/variadic-wrapper.c
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#include "variadic-wrapper.h"

int opus_custom_encoder_ctl_wrapper(OpusCustomEncoder *OPUS_RESTRICT st, int request, opus_int32 val)
{
return opus_custom_encoder_ctl(st, request, val);
}

int opus_custom_decoder_ctl_wrapper(OpusCustomDecoder *OPUS_RESTRICT st, int request, opus_int32 val)
{
return opus_custom_decoder_ctl(st, request, val);
}
95 changes: 95 additions & 0 deletions Sources/Opus/Opus.Custom.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
import AVFoundation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What’s the purpose of this file? Can it be deleted?

(I think it should be, if Opus.Encoder and Opus.Decoder can support nonstandard frame sizes.

import Copus
import Copuswrapper
import Foundation

public extension Opus {
///
/// Implements a custom opus encoder / decoder.
/// Custom implementations can have non-standard frame sizes
///
final class Custom {
private let opusCustomMode: OpaquePointer
let encoder: Opus.Encoder
let decoder: Opus.Decoder
private let format: AVAudioFormat
public let frameSize: Int32

public init(format: AVAudioFormat,
application _: Application = .audio,
frameSize: UInt32 = 128) throws
{
if !format.isValidOpusPCMFormat {
throw Opus.Error.badArgument
}
self.format = format
self.frameSize = Int32(frameSize)

var error: Opus.Error = .ok

// Create custom parameters
guard let customMode = opus_custom_mode_create(
Int32(format.sampleRate),
Int32(frameSize),
&error.rawValue
) else { throw error }
opusCustomMode = customMode

// Create custom encoder
encoder = try Opus.Encoder(customOpus: opusCustomMode,
format: format,
frameSize: frameSize)
// Create custom decoder
decoder = try Opus.Decoder(customOpus: opusCustomMode,
format: format,
frameSize: frameSize)
}

deinit {
opus_custom_mode_destroy(opusCustomMode)
}

///
/// Wrapper onto the opus_custom_encoder_ctl function
/// https://www.opus-codec.org/docs/opus_api-1.3.1/group__opus__encoderctls.html
/// - Parameter request The Opus CTL to change
/// - Parameter value The value to set it to
///
/// - Returns Opus.Error code
public func encoderCtl(request: Int32, value: Int32) -> Opus.Error {
Opus.Error(
rawValue: opus_custom_encoder_ctl_wrapper(encoder.encoder, request, value)
)
}

///
/// Encode a PCM buffer to data using the custom mode configuration and max size
/// - parameter avData Audio data to compress
/// - parameter compressedSize Opus packet size to compress to
/// - Returns Data containing the Opus packet
public func encode(_ avData: AVAudioPCMBuffer,
compressedSize: Int) throws -> Data
{
try encoder.encode(avData, compressedSize: compressedSize)
}

///
/// Decode an opus packet
/// If the data is empty, it is treated as a dropped packet
/// - Parameter data Compressed data
/// - Parameter compressedPacketSize Number of bytes of data
/// - Parameter sampleMultiplier Frame size multiplier if greater than one
/// - Returns Uncompressed audio buffer
public func decode(_ data: Data,
compressedPacketSize: Int32,
sampleMultiplier: Int32 = 1) throws -> AVAudioPCMBuffer
{
guard data.isEmpty || data.count == compressedPacketSize else {
throw Opus.Error.bufferTooSmall
}
return try decoder.decode(data,
compressedPacketSize: compressedPacketSize,
sampleMultiplier: sampleMultiplier)
}
}
}
120 changes: 92 additions & 28 deletions Sources/Opus/Opus.Decoder.swift
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ public extension Opus {
class Decoder {
let format: AVAudioFormat
let decoder: OpaquePointer
let customFrameSize: Int32?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can probably be a non-optional int that defaults to 0.

We’ll also need to add a let customMode: OpaquePointer here (see below).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually no, this is a way to avoid having a duplicate un-needed pointer.
If you have a custom frame size, then your decoder is a custom one.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There’s no function in the Opus library to extract the frame size used to initialize an OpusCustomMode, hence storing it here.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't follow your point here; can you clarify?


// TODO: throw an error if format is unsupported
public init(format: AVAudioFormat, application _: Application = .audio) throws {
customFrameSize = nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is made non-optional, this can be customFrameSize = 0, with customMode = nil.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And then you need 2 new vars; customMode and customFrameSize.
Using an optional for customFrameSize means you don't need a separate variable.

if !format.isValidOpusPCMFormat {
throw Opus.Error.badArgument
}
Expand All @@ -22,6 +23,27 @@ public extension Opus {
}
}

public init(customOpus: OpaquePointer,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the encoder and decoder typically exist on different machines, I think Decoder should own the OpusCustomMode and create it here.

It can be freed in deinit.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree - custom encoders and decoders need the custom object; you need to have a custom object in order to use either.
It maybe in your use case you have encoding on one machine and decoding on another, but you have to have the same custom instance for both if you're using them together.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The constructor for an OpusCustomMode takes two arguments: a sample rate and a target frame size in number of samples, which must be 64-1024 plus some additional constraints on prime factors.

The sample rate can be derived from the AVAudioFormat in the constructor along with a customFrameSize parameter to the custom constructor for an Encoder and Decoder.

It’s probably useful to have common helper code to aid in the creation of a valid OpusCustomMode that throws an error, but isn’t part of this package’s public API.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It’s probably useful to have common helper code to aid in the creation of a valid OpusCustomMode that throws an error, but isn’t part of this package’s public API.

That is the Opus.Custom.swift file.

format: AVAudioFormat,
frameSize: UInt32) throws
emlynmac marked this conversation as resolved.
Show resolved Hide resolved
{
customFrameSize = Int32(frameSize)
if !format.isValidOpusPCMFormat {
throw Opus.Error.badArgument
}

self.format = format

// Initialize Opus decoder
var error: Opus.Error = .ok
decoder = opus_custom_decoder_create(customOpus,
Int32(format.channelCount),
&error.rawValue)
if error != .ok {
throw error
}
}

deinit {
opus_decoder_destroy(decoder)
}
Expand All @@ -38,28 +60,42 @@ public extension Opus {
// MARK: Public decode methods

public extension Opus.Decoder {
func decode(_ input: Data) throws -> AVAudioPCMBuffer {
func decode(_ input: Data,
compressedPacketSize: Int32? = nil,
sampleMultiplier: Int32 = 1) throws -> AVAudioPCMBuffer
{
try input.withUnsafeBytes {
var output: AVAudioPCMBuffer
let input = $0.bindMemory(to: UInt8.self)
let sampleCount = opus_decoder_get_nb_samples(decoder, input.baseAddress!, Int32($0.count))
if sampleCount < 0 {
throw Opus.Error(sampleCount)
if compressedPacketSize != nil, let frameSize = customFrameSize {
output = AVAudioPCMBuffer(
pcmFormat: format,
frameCapacity: AVAudioFrameCount(frameSize * sampleMultiplier)
)!
} else {
let sampleCount = opus_decoder_get_nb_samples(decoder, input.baseAddress!, Int32($0.count))
if sampleCount < 0 {
throw Opus.Error(sampleCount)
}
output = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(sampleCount))!
}
let output = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(sampleCount))!
try decode(input, to: output)

try decode(input, to: output, packetSize: compressedPacketSize)
return output
}
}

func decode(_ input: UnsafeBufferPointer<UInt8>, to output: AVAudioPCMBuffer) throws {
func decode(_ input: UnsafeBufferPointer<UInt8>, to output: AVAudioPCMBuffer,
packetSize: Int32? = nil) throws
{
let decodedCount: Int
switch output.format.commonFormat {
case .pcmFormatInt16:
let output = UnsafeMutableBufferPointer(start: output.int16ChannelData![0], count: Int(output.frameCapacity))
decodedCount = try decode(input, to: output)
decodedCount = try decode(input, to: output, packetSize: packetSize)
case .pcmFormatFloat32:
let output = UnsafeMutableBufferPointer(start: output.floatChannelData![0], count: Int(output.frameCapacity))
decodedCount = try decode(input, to: output)
decodedCount = try decode(input, to: output, packetSize: packetSize)
default:
throw Opus.Error.badArgument
}
Expand All @@ -73,30 +109,58 @@ public extension Opus.Decoder {
// MARK: Private decode methods

extension Opus.Decoder {
private func decode(_ input: UnsafeBufferPointer<UInt8>, to output: UnsafeMutableBufferPointer<Int16>) throws -> Int {
let decodedCount = opus_decode(
decoder,
input.baseAddress!,
Int32(input.count),
output.baseAddress!,
Int32(output.count),
0
)
private func decode(_ input: UnsafeBufferPointer<UInt8>,
to output: UnsafeMutableBufferPointer<Int16>,
packetSize: Int32?) throws -> Int
{
var decodedCount: Int32 = 0
if let size = packetSize {
decodedCount = opus_custom_decode(
decoder,
input.isEmpty ? nil : input.baseAddress,
size,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is packetSize passed as an argument here (and other decode methods) when it should just be equal to Int32(input.count)?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

packet size is needed for the custom decoder to know how much data it has.
In the case of a dropped packet, input is nil, but the encoder still needs to know how much data got dropped

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment at the top-level of this PR regarding dropped packets.

If the decoder was initialized with a custom frame size, then either use that, or have decodeDroppedPacket accept an (optional?) frame size argument.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Custom frame size and compressed packet size are not the same thing. Custom encode / decode need to know both in order to work.

output.baseAddress!,
Int32(output.count)
)
} else {
decodedCount = opus_decode(
decoder,
input.isEmpty ? nil : input.baseAddress,
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change means that if you pass an empty data packet, opus decoder gets the nil it needs to signify a dropped packet.

Int32(input.count),
output.baseAddress!,
Int32(output.count),
0
)
}
if decodedCount < 0 {
throw Opus.Error(decodedCount)
}
return Int(decodedCount)
}

private func decode(_ input: UnsafeBufferPointer<UInt8>, to output: UnsafeMutableBufferPointer<Float32>) throws -> Int {
let decodedCount = opus_decode_float(
decoder,
input.baseAddress!,
Int32(input.count),
output.baseAddress!,
Int32(output.count),
0
)
private func decode(_ input: UnsafeBufferPointer<UInt8>,
to output: UnsafeMutableBufferPointer<Float32>,
packetSize: Int32?) throws -> Int
{
var decodedCount: Int32 = 0
if let size = packetSize {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If possible, this should branch on the presence of a non-nil customMode instance variable, which implies the decoder was initialized with a custom frame size.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've implemented this a different way; using the optional presence of a custom frame size.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When decoding a stream of packets, is the frame size consistent, or does it vary? If it’s consistent, then this parameter shouldn’t exist.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The size can be multiples of the underlying frame size

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does it need the packetSize argument? Can that be inferred from input.count?

Copy link
Author

@emlynmac emlynmac Apr 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, now I've had my coffee...

Compressed packet size != frame size.

Both are needed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, great. Then can that be taken from the customFrameSize instance variable?

decodedCount = opus_custom_decode_float(
decoder,
input.isEmpty ? nil : input.baseAddress,
size,
output.baseAddress!,
Int32(output.count)
)
} else {
decodedCount = opus_decode_float(
decoder,
input.isEmpty ? nil : input.baseAddress,
Int32(input.count),
output.baseAddress!,
Int32(output.count),
0
)
}
if decodedCount < 0 {
throw Opus.Error(decodedCount)
}
Expand Down
Loading