-
Notifications
You must be signed in to change notification settings - Fork 178
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ISDK-2241: Co-Viewing example with custom AudioDevice #325
Open
ceaglest
wants to merge
97
commits into
twilio:master
Choose a base branch
from
ceaglest:tweek/co-viewing
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 76 commits
Commits
Show all changes
97 commits
Select commit
Hold shift + click to select a range
6c96009
Add new project with empty template.
ceaglest 06c933c
Add ExampleAVPlayerView.
ceaglest b65f914
WIP - Play and pause video.
ceaglest d74becf
Play automatically for now.
ceaglest e62c97a
Added ExampleAVPlayerSource which captures from an AVPlayerItem.
ceaglest e5ff609
WIP - Add an AVAudioMix to the asset.
ceaglest f3204b4
Attempt to request an IOSurface.
ceaglest d18f28c
Add ExampleAVPlayerAudioTap.
ceaglest 65a7d5e
Comment out IOSurface request, it crashes on device.
ceaglest 2e8486b
Drop in a copy of ExampleCoreAudioDevice, and a bridging header.
ceaglest 8113a51
Spacing.
ceaglest c45ce61
Add project to Podfile, and consume TPCircularBuffer.
ceaglest 5d29ca4
Rename to AVPlayerAudioDevice, WIP - AudioTap.
ceaglest 20345d2
WIP - Produce audio using the ring buffer.
ceaglest 27891e3
WIP - Playback of MTAudioProcessingTap.
ceaglest d8eede2
Create an MTAudioProcessingTap.
ceaglest 2b90a9d
Hook up the audio device, almost there.
ceaglest eee3408
Fix the crash consuming buffers.
ceaglest de4ee11
Use a format converter.
ceaglest cb64ff6
Workaround for device crashes.
ceaglest d0079a9
Co-viewing app ui (#2)
piyushtank 12bdc2c
Minor.
ceaglest 14f507c
WIP - Adding capturing capabilities.
ceaglest 2314119
UI tweaks - always share audio, mirror camera.
ceaglest 2542a60
Recording is almost working (distorted).
ceaglest 06fc4fb
Fix recording distortion.
ceaglest 0aace23
Use a multichannel mixer for playback.
ceaglest 15d44da
Print UI messages to the console for now.
ceaglest a54a440
ExampleAVPlayerSource is a TVIVideoCapturer.
ceaglest 26f8c45
WIP - Add a recording mixer.
ceaglest d08d26c
Init is at least working.
ceaglest a265fab
Render into our buffer list.
ceaglest 4552a1c
Internal refactor to prepare for generic output.
ceaglest 325a448
Player track rendering at remote side (#3)
piyushtank fc12e6c
WIP - Use a generic output.
ceaglest 1340251
Ordering.
ceaglest 7177887
dicsonnect button (#5)
piyushtank 705a113
Merge branch 'tweek/co-viewing' into tweek/co-viewing-recording-mixer
ceaglest c018ae0
Add an early return and remove dead code.
ceaglest 0d0cbb1
Fix compiler errors.
ceaglest ac6bba8
Add a disconnect button.
ceaglest a8b7ccf
Request 480p buffers for streaming.
ceaglest ab84ab8
Lower the deployment target to iOS 11.0 for now.
ceaglest f1344c1
Fix the supported video format.
ceaglest 4f9d495
Metadata.
ceaglest 2526fa7
Tweak the UI, use Twilio red.
ceaglest b109c4a
Comments and logging.
ceaglest bbdea1f
Nearly working recording mixer solution.
ceaglest d8218be
Working playback + recording solution.
ceaglest eae005e
Merge pull request #4 from ceaglest/tweek/co-viewing-recording-mixer
ceaglest 5cd4231
Don’t share the camera on the simulator.
ceaglest b4b5228
Observe AVPlayerItem status and tracks.
ceaglest dc42e24
Remove unused recording mixer code.
ceaglest 9e7099b
Improve button style, reset local mirroring and access token each time.
ceaglest 0fe18fc
Cleanup usage of self.audioDevice.
ceaglest d3787fb
Cleanup more state on disconnect/failure.
ceaglest cab5eed
Several improvements.
ceaglest 7ada723
Cleanup content, initial animation, and tap to fit/fill AVP video.
ceaglest 0718c4b
Significant rework of ExampleAVPlayerSource.
ceaglest 13fd7aa
Comment out noisy logs.
ceaglest 3748e84
Add another trailer.
ceaglest 66d3878
Remove remote player view for presenters.
ceaglest 34e2e22
Fix initial state of ExampleAVPlayerView.contentMode.
ceaglest b4c855b
Hide the status bar as well.
ceaglest 5ea855a
Add a conditional check.
ceaglest a38df60
Add .mp4 and .mov document handling + background modes.
ceaglest 60fbf70
Support stopping/starting of the audio device.
ceaglest f0aa053
Add test content.
ceaglest 9edc998
Temporary fix for video vs full range content.
ceaglest f67df1f
Disallow opening documents in place.
ceaglest 79b0b13
Improve some view controller teardown.
ceaglest 0adf389
Refactor of audio pipeline.
ceaglest b96a6bd
Revert video level changes.
ceaglest 4bc0459
Separate MTAudioProcessingTap code into a separate file.
ceaglest e631fd4
WIP - SRC
ceaglest 65a42bb
Use 48khz sample content that doesn’t need resampling.
ceaglest c7275b8
Working sample rate conversion to 48 kHz.
ceaglest c5294b4
Use a 20 millisecond duration for all example TVIAudioDevices.
ceaglest 6659a3e
Use more bandwidth for presenter audio, restore 44.1 kHz content.
ceaglest 6b70a34
Comment out printf statements in realtime code.
ceaglest a517531
Review feedback - document important audio device properties.
ceaglest b9a2c6c
Remove dispatch_semaphore, address ASBD naming.
ceaglest 1def76f
Use the regular token server URL.
ceaglest 1a26281
Remove commented KVO code.
ceaglest 9c51e00
Explicitly request an IOSurface.
ceaglest 7156ed2
Dynamically create and destroy remotePlayerView.
ceaglest 32ccba6
Use C++14.
ceaglest f63f649
Improve ExampleAVPlayerSource documentation.
ceaglest e75ee3e
Reduce the button / video view margins from 10 to 4 points.
ceaglest 3555d2d
Comments and disable AVAudioMix update code.
ceaglest f924c8e
Produce buffers with timestamps for playback.
ceaglest 6d97213
Set the preferred number of input channels, cleanup logging.
ceaglest d066e15
Attempt to fix memory management of MTAudioProcessingTap.
ceaglest 4980f30
Time synchronization for audio playback.
ceaglest 287b8e2
Merge branch 'master' into tweek/co-viewing
ceaglest 47be1d8
Arbitrary loads for media only.
ceaglest d316d29
Support 1-channel output devices properly (like AirPods in HFP).
ceaglest File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
7 changes: 7 additions & 0 deletions
7
CoViewingExample.xcodeproj/project.xcworkspace/contents.xcworkspacedata
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
8 changes: 8 additions & 0 deletions
8
CoViewingExample.xcodeproj/project.xcworkspace/xcshareddata/IDEWorkspaceChecks.plist
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
<?xml version="1.0" encoding="UTF-8"?> | ||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> | ||
<plist version="1.0"> | ||
<dict> | ||
<key>IDEDidComputeMac32BitWarning</key> | ||
<true/> | ||
</dict> | ||
</plist> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
// | ||
// AppDelegate.swift | ||
// CoViewingExample | ||
// | ||
// Copyright © 2018 Twilio Inc. All rights reserved. | ||
// | ||
|
||
import UIKit | ||
|
||
@UIApplicationMain | ||
class AppDelegate: UIResponder, UIApplicationDelegate { | ||
|
||
var window: UIWindow? | ||
|
||
|
||
func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool { | ||
print("didFinishLaunchingWithOptions:", launchOptions as Any) | ||
if let options = launchOptions, | ||
let videoUrl = options[UIApplication.LaunchOptionsKey.url] as? URL { | ||
let rootVC = window?.rootViewController as! ViewController | ||
rootVC.startPresenter(contentUrl: videoUrl) | ||
} | ||
return true | ||
} | ||
|
||
func application(_ app: UIApplication, open url: URL, options: [UIApplication.OpenURLOptionsKey : Any] = [:]) -> Bool { | ||
print("app:openURL:", url, " options:", options as Any) | ||
|
||
let rootVC = window?.rootViewController as! ViewController | ||
rootVC.startPresenter(contentUrl: url) | ||
|
||
return true | ||
} | ||
|
||
func applicationWillResignActive(_ application: UIApplication) { | ||
// Sent when the application is about to move from active to inactive state. This can occur for certain types of temporary interruptions (such as an incoming phone call or SMS message) or when the user quits the application and it begins the transition to the background state. | ||
// Use this method to pause ongoing tasks, disable timers, and invalidate graphics rendering callbacks. Games should use this method to pause the game. | ||
} | ||
|
||
func applicationDidEnterBackground(_ application: UIApplication) { | ||
// Use this method to release shared resources, save user data, invalidate timers, and store enough application state information to restore your application to its current state in case it is terminated later. | ||
// If your application supports background execution, this method is called instead of applicationWillTerminate: when the user quits. | ||
} | ||
|
||
func applicationWillEnterForeground(_ application: UIApplication) { | ||
// Called as part of the transition from the background to the active state; here you can undo many of the changes made on entering the background. | ||
} | ||
|
||
func applicationDidBecomeActive(_ application: UIApplication) { | ||
// Restart any tasks that were paused (or not yet started) while the application was inactive. If the application was previously in the background, optionally refresh the user interface. | ||
} | ||
|
||
func applicationWillTerminate(_ application: UIApplication) { | ||
// Called when the application is about to terminate. Save data if appropriate. See also applicationDidEnterBackground:. | ||
} | ||
|
||
|
||
} | ||
|
98 changes: 98 additions & 0 deletions
98
CoViewingExample/Assets.xcassets/AppIcon.appiconset/Contents.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,98 @@ | ||
{ | ||
"images" : [ | ||
{ | ||
"idiom" : "iphone", | ||
"size" : "20x20", | ||
"scale" : "2x" | ||
}, | ||
{ | ||
"idiom" : "iphone", | ||
"size" : "20x20", | ||
"scale" : "3x" | ||
}, | ||
{ | ||
"idiom" : "iphone", | ||
"size" : "29x29", | ||
"scale" : "2x" | ||
}, | ||
{ | ||
"idiom" : "iphone", | ||
"size" : "29x29", | ||
"scale" : "3x" | ||
}, | ||
{ | ||
"idiom" : "iphone", | ||
"size" : "40x40", | ||
"scale" : "2x" | ||
}, | ||
{ | ||
"idiom" : "iphone", | ||
"size" : "40x40", | ||
"scale" : "3x" | ||
}, | ||
{ | ||
"idiom" : "iphone", | ||
"size" : "60x60", | ||
"scale" : "2x" | ||
}, | ||
{ | ||
"idiom" : "iphone", | ||
"size" : "60x60", | ||
"scale" : "3x" | ||
}, | ||
{ | ||
"idiom" : "ipad", | ||
"size" : "20x20", | ||
"scale" : "1x" | ||
}, | ||
{ | ||
"idiom" : "ipad", | ||
"size" : "20x20", | ||
"scale" : "2x" | ||
}, | ||
{ | ||
"idiom" : "ipad", | ||
"size" : "29x29", | ||
"scale" : "1x" | ||
}, | ||
{ | ||
"idiom" : "ipad", | ||
"size" : "29x29", | ||
"scale" : "2x" | ||
}, | ||
{ | ||
"idiom" : "ipad", | ||
"size" : "40x40", | ||
"scale" : "1x" | ||
}, | ||
{ | ||
"idiom" : "ipad", | ||
"size" : "40x40", | ||
"scale" : "2x" | ||
}, | ||
{ | ||
"idiom" : "ipad", | ||
"size" : "76x76", | ||
"scale" : "1x" | ||
}, | ||
{ | ||
"idiom" : "ipad", | ||
"size" : "76x76", | ||
"scale" : "2x" | ||
}, | ||
{ | ||
"idiom" : "ipad", | ||
"size" : "83.5x83.5", | ||
"scale" : "2x" | ||
}, | ||
{ | ||
"idiom" : "ios-marketing", | ||
"size" : "1024x1024", | ||
"scale" : "1x" | ||
} | ||
], | ||
"info" : { | ||
"version" : 1, | ||
"author" : "xcode" | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
{ | ||
"info" : { | ||
"version" : 1, | ||
"author" : "xcode" | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
// | ||
// AudioDevices-Bridging-Header.h | ||
// CoViewingExample | ||
// | ||
// Copyright © 2018 Twilio Inc. All rights reserved. | ||
// | ||
|
||
#import "ExampleAVPlayerAudioDevice.h" |
27 changes: 27 additions & 0 deletions
27
CoViewingExample/AudioDevices/ExampleAVPlayerAudioDevice.h
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
// | ||
// ExampleAVPlayerAudioDevice.h | ||
// CoViewingExample | ||
// | ||
// Copyright © 2018 Twilio, Inc. All rights reserved. | ||
// | ||
|
||
#import <TwilioVideo/TwilioVideo.h> | ||
|
||
/* | ||
* ExampleAVPlayerAudioDevice uses a VoiceProcessingIO audio unit to play audio from an MTAudioProcessingTap | ||
* attached to an AVPlayerItem. The AVPlayer audio is mixed with Room audio provided by Twilio. | ||
* The microphone input, and MTAudioProcessingTap output are mixed into a single recorded stream. | ||
*/ | ||
@interface ExampleAVPlayerAudioDevice : NSObject <TVIAudioDevice> | ||
|
||
- (void)audioTapDidPrepare; | ||
|
||
/* | ||
* Creates a processing tap bound to the device instance. | ||
* | ||
* @return An `MTAudioProcessingTap`, or NULL if there is an error. The caller assumes all ownership | ||
* of the tap, and should call CFRelease when they are finished with it. | ||
*/ | ||
- (nullable MTAudioProcessingTapRef)createProcessingTap; | ||
|
||
@end |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍