Decorate your footage with Swift and Sora
Nice to meet you, I'm Okoshi, and this is my first time working part-time on the iTour project.
As the title suggests, I would like to summarize here how to decorate the video using Swift and Sora.

Technology used
The video is displayed using a technology called WebRTC, and the video is decorated using Swift's pre-prepared functions.
WebRTC is a so-called video chat that allows real-time communication, but this time it is possibleSoraI will use a library called

Now let's move on to setup.

About setup

Roughly speaking, Sora prepares in advancesample projectからgit　cloneafter thatCarthagejust run
I think there are multiple sample projects,Carthageto runOnly in DecoStreamingSample and RealTimeStreamingSampleis fine.
For the detailed setup method from 1Click here for more information..

Notes

Please prepare the actual machine in advance because it works only with the actual machine.

Distributor side

The distribution side will edit the code of the DecoStreamingSample project.

1.
sora-ios-sdk-samples/DecoStreamingSample/DecoStreamingSample/Classes/PublisherVideoViewController.swiftAdd the following code as a global variable in

class PublisherVideoViewController: UIViewController{ /*Variable added here*/ //Preparation when converting from uiimage to ciimage private var ciChange:CIImage? //Prepare an image in advance to send the mask to the sub private static let allMasks: [String: UIImage] = { let pikaExMask:UIImage = UIImage(named: "pikaExtend.jpg")! let pikaSmiMask:UIImage = UIImage(named: "pikaSmile.jpg")! return [ "Extended Pikachu": pikaExMask, " Smiling Pikachu": pikaSmiMask ] }() // currently selected image private var currentMasks: UIImage? }

Change the image name and variable name to something appropriate.

2.Assert.xcassetsPlease put an appropriate image inside by drag & drop.
before putting画像の名前@2x.jpgLet's change it to (for iPad)
iPhone6 plusな ど
is@3xAndiPad2,iPad mini1Is@1xbe added after the name. (For more informationClick here for more information.)

3. Add the processing for displaying the image to the viewer side to the video capture function.

func captureOutput(_ captureOutput: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { /* Code to add here.Let's write it below the filtering process */ //Display the mask on the sub side if let mask = currentMasks { guard let maskBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } //UIImage -> CIImage conversion let context = CIContext (options: nil) ciChange = CIImage(image: mask) context.render(ciChange!, to: maskBuffer) //Send mask to sub mediaStream.send(videoFrame: VideoFrame(from: sampleBuffer)) } else { // filter is not selected, so the captured video will be delivered as is. mediaStream.send(videoFrame: VideoFrame(from: sampleBuffer)) } }

4. Add processing when selecting an image.It may be easier to understand if you write it below the process when the filter is selected.

 /** Defines the behavior when pressing the filter selection button.See the definition in Main.storyboard for details. */ @IBAction func onFilterButton(_ sender: UIBarButtonItem) { //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ } /* from here Add */ //Button that can select an image @IBAction func onMaskButton(_ sender: UIBarButtonItem) { let alertController = UIAlertController(title: "Select Mask", message: nil, preferredStyle: .actionSheet) for (name, mask) in PublisherVideoViewController.allMasks.sorted(by: { $0.0 < $1.0 }) { let action = UIAlertAction(title: name, style: .default) { [weak self] _ in self?.currentMasks = mask } alertController.addAction(action) } alertController.popoverPresentationController?.sourceView = self.view; //specify view for ipad let noMaskAction = UIAlertAction(title: "no mask", style: .destructive) { [weak self] _ in self?.currentMasks = nil } alertController.addAction(noMaskAction) present(alertController, animated: true, completion: nil) }

 

5.Main.storyboardto customize the UI.Select the desired button from the lower right toolbar and drag and drop it as shown in the image below.button itemsput it inside.Then right-click on the button added to synchronize with the process described earlierSent ActionsToonMaskButtonLet's add a function. 
6.SoraSDKManager.swiftEnter the URL of the server at hand in the targetURL described in .

/** This is the connection destination URL of Sora SDK.Please specify the connection destination of your Sora. */ private static let targetURL: URL = URL(string: “change here”)!

Editing on the distributor side is now complete.

viewer side

On the viewer side, edit the code of the RealTimeStreamingSample project.
While saying that, it is written on the distributor side6の項目のみis.The file name should also exist with the same name.

Build

1. Press the upper left triangle button to build. (Although I wrote it at the beginning, it works only on the actual machine)

2. On the viewer side (RealTimeStreamingSample)他の配信を見るSince there is a button, tap it and make settings for viewing.

3. On the distributor side (DecoStreamingSample), make settings for distribution.Let's move to the next screen with the same settings on both the viewing and the distributor side.

4. I think there is a button added as an image button in the distribution screen on the distributor side.Press it and see if the image appears on the screen.
It is a success if the same image appears on the screen on the viewer side.

This is the end of the explanation.good job for today.