Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ISDK-2241: Co-Viewing example with custom AudioDevice #325

Open
wants to merge 97 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 76 commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
6c96009
Add new project with empty template.
ceaglest Oct 29, 2018
06c933c
Add ExampleAVPlayerView.
ceaglest Oct 30, 2018
b65f914
WIP - Play and pause video.
ceaglest Oct 30, 2018
d74becf
Play automatically for now.
ceaglest Oct 30, 2018
e62c97a
Added ExampleAVPlayerSource which captures from an AVPlayerItem.
ceaglest Oct 30, 2018
e5ff609
WIP - Add an AVAudioMix to the asset.
ceaglest Oct 31, 2018
f3204b4
Attempt to request an IOSurface.
ceaglest Oct 31, 2018
d18f28c
Add ExampleAVPlayerAudioTap.
ceaglest Oct 31, 2018
65a7d5e
Comment out IOSurface request, it crashes on device.
ceaglest Oct 31, 2018
2e8486b
Drop in a copy of ExampleCoreAudioDevice, and a bridging header.
ceaglest Oct 31, 2018
8113a51
Spacing.
ceaglest Oct 31, 2018
c45ce61
Add project to Podfile, and consume TPCircularBuffer.
ceaglest Oct 31, 2018
5d29ca4
Rename to AVPlayerAudioDevice, WIP - AudioTap.
ceaglest Nov 1, 2018
20345d2
WIP - Produce audio using the ring buffer.
ceaglest Nov 1, 2018
27891e3
WIP - Playback of MTAudioProcessingTap.
ceaglest Nov 1, 2018
d8eede2
Create an MTAudioProcessingTap.
ceaglest Nov 1, 2018
2b90a9d
Hook up the audio device, almost there.
ceaglest Nov 1, 2018
eee3408
Fix the crash consuming buffers.
ceaglest Nov 1, 2018
de4ee11
Use a format converter.
ceaglest Nov 1, 2018
cb64ff6
Workaround for device crashes.
ceaglest Nov 1, 2018
d0079a9
Co-viewing app ui (#2)
piyushtank Nov 1, 2018
12bdc2c
Minor.
ceaglest Nov 1, 2018
14f507c
WIP - Adding capturing capabilities.
ceaglest Nov 1, 2018
2314119
UI tweaks - always share audio, mirror camera.
ceaglest Nov 2, 2018
2542a60
Recording is almost working (distorted).
ceaglest Nov 2, 2018
06fc4fb
Fix recording distortion.
ceaglest Nov 2, 2018
0aace23
Use a multichannel mixer for playback.
ceaglest Nov 2, 2018
15d44da
Print UI messages to the console for now.
ceaglest Nov 2, 2018
a54a440
ExampleAVPlayerSource is a TVIVideoCapturer.
ceaglest Nov 2, 2018
26f8c45
WIP - Add a recording mixer.
ceaglest Nov 2, 2018
d08d26c
Init is at least working.
ceaglest Nov 2, 2018
a265fab
Render into our buffer list.
ceaglest Nov 2, 2018
4552a1c
Internal refactor to prepare for generic output.
ceaglest Nov 2, 2018
325a448
Player track rendering at remote side (#3)
piyushtank Nov 2, 2018
fc12e6c
WIP - Use a generic output.
ceaglest Nov 2, 2018
1340251
Ordering.
ceaglest Nov 2, 2018
7177887
dicsonnect button (#5)
piyushtank Nov 3, 2018
705a113
Merge branch 'tweek/co-viewing' into tweek/co-viewing-recording-mixer
ceaglest Nov 3, 2018
c018ae0
Add an early return and remove dead code.
ceaglest Nov 3, 2018
0d0cbb1
Fix compiler errors.
ceaglest Nov 3, 2018
ac6bba8
Add a disconnect button.
ceaglest Nov 3, 2018
a8b7ccf
Request 480p buffers for streaming.
ceaglest Nov 3, 2018
ab84ab8
Lower the deployment target to iOS 11.0 for now.
ceaglest Nov 3, 2018
f1344c1
Fix the supported video format.
ceaglest Nov 3, 2018
4f9d495
Metadata.
ceaglest Nov 3, 2018
2526fa7
Tweak the UI, use Twilio red.
ceaglest Nov 3, 2018
b109c4a
Comments and logging.
ceaglest Nov 3, 2018
bbdea1f
Nearly working recording mixer solution.
ceaglest Nov 3, 2018
d8218be
Working playback + recording solution.
ceaglest Nov 3, 2018
eae005e
Merge pull request #4 from ceaglest/tweek/co-viewing-recording-mixer
ceaglest Nov 3, 2018
5cd4231
Don’t share the camera on the simulator.
ceaglest Nov 4, 2018
b4b5228
Observe AVPlayerItem status and tracks.
ceaglest Nov 4, 2018
dc42e24
Remove unused recording mixer code.
ceaglest Nov 4, 2018
9e7099b
Improve button style, reset local mirroring and access token each time.
ceaglest Nov 4, 2018
0fe18fc
Cleanup usage of self.audioDevice.
ceaglest Nov 4, 2018
d3787fb
Cleanup more state on disconnect/failure.
ceaglest Nov 4, 2018
cab5eed
Several improvements.
ceaglest Nov 4, 2018
7ada723
Cleanup content, initial animation, and tap to fit/fill AVP video.
ceaglest Nov 5, 2018
0718c4b
Significant rework of ExampleAVPlayerSource.
ceaglest Nov 5, 2018
13fd7aa
Comment out noisy logs.
ceaglest Nov 5, 2018
3748e84
Add another trailer.
ceaglest Nov 5, 2018
66d3878
Remove remote player view for presenters.
ceaglest Nov 5, 2018
34e2e22
Fix initial state of ExampleAVPlayerView.contentMode.
ceaglest Nov 5, 2018
b4c855b
Hide the status bar as well.
ceaglest Nov 5, 2018
5ea855a
Add a conditional check.
ceaglest Nov 5, 2018
a38df60
Add .mp4 and .mov document handling + background modes.
ceaglest Nov 7, 2018
60fbf70
Support stopping/starting of the audio device.
ceaglest Nov 7, 2018
f0aa053
Add test content.
ceaglest Nov 7, 2018
9edc998
Temporary fix for video vs full range content.
ceaglest Nov 8, 2018
f67df1f
Disallow opening documents in place.
ceaglest Nov 8, 2018
79b0b13
Improve some view controller teardown.
ceaglest Nov 8, 2018
0adf389
Refactor of audio pipeline.
ceaglest Nov 8, 2018
b96a6bd
Revert video level changes.
ceaglest Nov 8, 2018
4bc0459
Separate MTAudioProcessingTap code into a separate file.
ceaglest Nov 8, 2018
e631fd4
WIP - SRC
ceaglest Nov 8, 2018
65a42bb
Use 48khz sample content that doesn’t need resampling.
ceaglest Nov 9, 2018
c7275b8
Working sample rate conversion to 48 kHz.
ceaglest Nov 9, 2018
c5294b4
Use a 20 millisecond duration for all example TVIAudioDevices.
ceaglest Nov 9, 2018
6659a3e
Use more bandwidth for presenter audio, restore 44.1 kHz content.
ceaglest Nov 9, 2018
6b70a34
Comment out printf statements in realtime code.
ceaglest Nov 9, 2018
a517531
Review feedback - document important audio device properties.
ceaglest Nov 10, 2018
b9a2c6c
Remove dispatch_semaphore, address ASBD naming.
ceaglest Nov 10, 2018
1def76f
Use the regular token server URL.
ceaglest Nov 10, 2018
1a26281
Remove commented KVO code.
ceaglest Nov 10, 2018
9c51e00
Explicitly request an IOSurface.
ceaglest Nov 10, 2018
7156ed2
Dynamically create and destroy remotePlayerView.
ceaglest Nov 10, 2018
32ccba6
Use C++14.
ceaglest Nov 10, 2018
f63f649
Improve ExampleAVPlayerSource documentation.
ceaglest Nov 10, 2018
e75ee3e
Reduce the button / video view margins from 10 to 4 points.
ceaglest Nov 10, 2018
3555d2d
Comments and disable AVAudioMix update code.
ceaglest Nov 12, 2018
f924c8e
Produce buffers with timestamps for playback.
ceaglest Nov 12, 2018
6d97213
Set the preferred number of input channels, cleanup logging.
ceaglest Nov 12, 2018
d066e15
Attempt to fix memory management of MTAudioProcessingTap.
ceaglest Nov 12, 2018
4980f30
Time synchronization for audio playback.
ceaglest Nov 12, 2018
287b8e2
Merge branch 'master' into tweek/co-viewing
ceaglest Dec 2, 2018
47be1d8
Arbitrary loads for media only.
ceaglest Dec 2, 2018
d316d29
Support 1-channel output devices properly (like AirPods in HFP).
ceaglest Dec 2, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
377 changes: 377 additions & 0 deletions CoViewingExample.xcodeproj/project.pbxproj

Large diffs are not rendered by default.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>IDEDidComputeMac32BitWarning</key>
<true/>
</dict>
</plist>
59 changes: 59 additions & 0 deletions CoViewingExample/AppDelegate.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
//
// AppDelegate.swift
// CoViewingExample
//
// Copyright © 2018 Twilio Inc. All rights reserved.
//

import UIKit

@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {

var window: UIWindow?


func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
print("didFinishLaunchingWithOptions:", launchOptions as Any)
if let options = launchOptions,
let videoUrl = options[UIApplication.LaunchOptionsKey.url] as? URL {
let rootVC = window?.rootViewController as! ViewController
rootVC.startPresenter(contentUrl: videoUrl)
}
return true
}

func application(_ app: UIApplication, open url: URL, options: [UIApplication.OpenURLOptionsKey : Any] = [:]) -> Bool {
print("app:openURL:", url, " options:", options as Any)

let rootVC = window?.rootViewController as! ViewController
rootVC.startPresenter(contentUrl: url)

return true
}

func applicationWillResignActive(_ application: UIApplication) {
// Sent when the application is about to move from active to inactive state. This can occur for certain types of temporary interruptions (such as an incoming phone call or SMS message) or when the user quits the application and it begins the transition to the background state.
// Use this method to pause ongoing tasks, disable timers, and invalidate graphics rendering callbacks. Games should use this method to pause the game.
}

func applicationDidEnterBackground(_ application: UIApplication) {
// Use this method to release shared resources, save user data, invalidate timers, and store enough application state information to restore your application to its current state in case it is terminated later.
// If your application supports background execution, this method is called instead of applicationWillTerminate: when the user quits.
}

func applicationWillEnterForeground(_ application: UIApplication) {
// Called as part of the transition from the background to the active state; here you can undo many of the changes made on entering the background.
}

func applicationDidBecomeActive(_ application: UIApplication) {
// Restart any tasks that were paused (or not yet started) while the application was inactive. If the application was previously in the background, optionally refresh the user interface.
}

func applicationWillTerminate(_ application: UIApplication) {
// Called when the application is about to terminate. Save data if appropriate. See also applicationDidEnterBackground:.
}


}

98 changes: 98 additions & 0 deletions CoViewingExample/Assets.xcassets/AppIcon.appiconset/Contents.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
{
"images" : [
{
"idiom" : "iphone",
"size" : "20x20",
"scale" : "2x"
},
{
"idiom" : "iphone",
"size" : "20x20",
"scale" : "3x"
},
{
"idiom" : "iphone",
"size" : "29x29",
"scale" : "2x"
},
{
"idiom" : "iphone",
"size" : "29x29",
"scale" : "3x"
},
{
"idiom" : "iphone",
"size" : "40x40",
"scale" : "2x"
},
{
"idiom" : "iphone",
"size" : "40x40",
"scale" : "3x"
},
{
"idiom" : "iphone",
"size" : "60x60",
"scale" : "2x"
},
{
"idiom" : "iphone",
"size" : "60x60",
"scale" : "3x"
},
{
"idiom" : "ipad",
"size" : "20x20",
"scale" : "1x"
},
{
"idiom" : "ipad",
"size" : "20x20",
"scale" : "2x"
},
{
"idiom" : "ipad",
"size" : "29x29",
"scale" : "1x"
},
{
"idiom" : "ipad",
"size" : "29x29",
"scale" : "2x"
},
{
"idiom" : "ipad",
"size" : "40x40",
"scale" : "1x"
},
{
"idiom" : "ipad",
"size" : "40x40",
"scale" : "2x"
},
{
"idiom" : "ipad",
"size" : "76x76",
"scale" : "1x"
},
{
"idiom" : "ipad",
"size" : "76x76",
"scale" : "2x"
},
{
"idiom" : "ipad",
"size" : "83.5x83.5",
"scale" : "2x"
},
{
"idiom" : "ios-marketing",
"size" : "1024x1024",
"scale" : "1x"
}
],
"info" : {
"version" : 1,
"author" : "xcode"
}
}
6 changes: 6 additions & 0 deletions CoViewingExample/Assets.xcassets/Contents.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"info" : {
"version" : 1,
"author" : "xcode"
}
}
8 changes: 8 additions & 0 deletions CoViewingExample/AudioDevices/AudioDevices-Bridging-Header.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
//
// AudioDevices-Bridging-Header.h
// CoViewingExample
//
// Copyright © 2018 Twilio Inc. All rights reserved.
//

#import "ExampleAVPlayerAudioDevice.h"
27 changes: 27 additions & 0 deletions CoViewingExample/AudioDevices/ExampleAVPlayerAudioDevice.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
//
// ExampleAVPlayerAudioDevice.h
// CoViewingExample
//
// Copyright © 2018 Twilio, Inc. All rights reserved.
//

#import <TwilioVideo/TwilioVideo.h>

/*
* ExampleAVPlayerAudioDevice uses a VoiceProcessingIO audio unit to play audio from an MTAudioProcessingTap
* attached to an AVPlayerItem. The AVPlayer audio is mixed with Room audio provided by Twilio.
* The microphone input, and MTAudioProcessingTap output are mixed into a single recorded stream.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

*/
@interface ExampleAVPlayerAudioDevice : NSObject <TVIAudioDevice>

- (void)audioTapDidPrepare;

/*
* Creates a processing tap bound to the device instance.
*
* @return An `MTAudioProcessingTap`, or NULL if there is an error. The caller assumes all ownership
* of the tap, and should call CFRelease when they are finished with it.
*/
- (nullable MTAudioProcessingTapRef)createProcessingTap;

@end
Loading