macOS Getting frames into display link with CV

[jcl43]

Hi all, I am working on an application that involves real time video editing with Core Framework in Cocoa. For testing we basically have a DVI cam hooked up to a Mac Book Pro and need to be able to edit the stream it produces. I successfully got the video into quick time using QTKit, but am having trouble aquiring the frames with a display link. The example in the Core Video guide shows it initializing the video source from file with this method:

* (id)initWithFilePath: (NSString*)theFilePath


, but I need to get the frames I can already access into the display link for OpenGL rendering and Core Image effects. If anyone has any ideas or advice I would greatly appreciate it. Thanks in advance!

 

[Sayer]

In the Core Video Programming Guide, Core Video Tasks, at the very end of Obtaining Frames using Display Link it talks about getting frames from QuickTime. Basically you grab a frame of video as an OpenGL texture and pass it back to Core Video for manipulation somehow.

I think that is the direction you need to go (literally and figuratively).

 

[jcl43]

Thanks, im going to give it a try right now. When using this implementation would I set the delegate to be an NSOpenGLView? Right now i have it output via QTCaptureView. But if I want to do openGL rendering/Core Image effects on the frame does it need to be an NSOpenGLView?

 

[jcl43]

maybe posting my code this far will help

//
//  VideoDisplayController.m
//  Prototype492
//
//  Created on 3/7/08.
//

#import "VideoDisplayController.h"

@implementation VideoDisplayController 

/*
    start the video capture session if it's not running
*/
- (IBAction)startVideoCapture:(id)sender
{

	if(![captureSession isRunning])
	{
	    [captureSession startRunning];
	}
}

/*
    stop the video capture session if it's running
*/
	
- (IBAction)stopVideoCapture:(id)sender
{
    if([captureSession isRunning])
	{
	    [captureSession stopRunning];
	}
	
}


/*
initialization of UI elements
*/
- (void)awakeFromNib
{
    NSLog(@"initializing video controller");

    if(!captureSession)	{

        //creating capture session

        captureSession = [[QTCaptureSession alloc] init];
	

	    //flags for connecting inputs and outputs
	    BOOL successConnect = NO;
	    NSError *error = nil;
	
		
	    //finding and creating the input device and adding it to the capture session
	    //if unsuccessful, send a message to error log


        //note: using QTMediaTypeMuxed for DV camera... otherwise use QTMediaTypeVideo
        videoCaptureDevice = [QTCaptureDevice defaultInputDeviceWithMediaType:QTMediaTypeMuxed];
    
	    //if the video capture device is found, attempt to open it and add it to the capture session
        if(videoCaptureDevice)
	    {
            successConnect = [videoCaptureDevice open:&error];

		
		    if(!successConnect)
		    {
            
		        NSLog(@"error opening video capture device");
                videoCaptureDevice = nil;

		        //other error handling code goes here
		    }

            captureVideoDeviceInput = [[QTCaptureDeviceInput alloc] initWithDevice:videoCaptureDevice];
     	
	
            //attempt to add capture device input to the capture session
		    successConnect = [captureSession addInput:captureVideoDeviceInput error:&error];

		    if(!successConnect)
		    {
		        NSLog(@"error adding video capture device input to capture session");
			
			    //other error handling code goes here
	        }
        
            //adding decompressed video output to capture session
			
			decompressedVideoOutput = [[QTCaptureDecompressedVideoOutput alloc] init];
			
			//assigning delegate to decompressed video output
			
			[decompressedVideoOutput setDelegate: self];
			
			successConnect = [captureSession addOutput:decompressedVideoOutput error:&error];

            if(!successConnect)
			{
			    NSLog(@"error adding decompressed video output to capture session");
				
				//other error handling code goes here
			
			}

	
            //linking capture session to the capture view
		    [captureView setCaptureSession:captureSession];
	    
		    //[captureSession startRunning];
	    }
        else
	    {
	        NSLog(@"couldn't find video capture device");
	    }
    }
}


/*
delegate method called when decompressedVideoOutput receives a frame
it is not called in the main thread, so a synhronized block is used
delegates of decompressedVideoOutput receive this message 
and then can used the provided video frame for further image processing

captureOutput - the QTCaptureDecompressedVideoOutput instance that sent the frame (decompressedVideoOutput)
videoFrame - Core Video image buffer containing the decompressed frame
sampleBuffer sample buffer containing additional info about the frame
connection - connnection from which vdieo was received
*/


- (void)captureOutput:(QTCaptureOutput *)captureOutput 
        didOutputVideoFrame:(CVImageBufferRef)videoFrame
        withSampleBuffer:(QTSampleBuffer *)sampleBuffer 
        fromConnection:(QTCaptureConnection *)connection
{
    CVImageBufferRef releasedImageBuffer;


    CVBufferRetain(videoFrame);
	

    @synchronized(self)
	{
        //basically, have frame to be released refer to the current frame
		//then update the reference to the current frame with the next frame in the "video stream"
	    releasedImageBuffer = currentImageBuffer;
        currentImageBuffer = videoFrame;
	}

    CVBufferRelease(releasedImageBuffer);

}





/*
event that's fired when window is closed
stop the session and close any opened devices
*/
- (void)windowWillClose:(NSNotification *)notification
{
    if([captureSession isRunning])
	{
	    [captureSession stopRunning];
    }
	
	if([videoCaptureDevice isOpen])
	{
	    [videoCaptureDevice close];
	}

}

/*
event that fires when memory is to be deallocated when object is killed
*/
- (void)dealloc
{

    [captureSession release];
	[captureVideoDeviceInput release];
    [decompressedVideoOutput release];
	
	[super dealloc];
}


@end


//
//  VideoDisplayController.h
//  Prototype492v1.1
//
//  Updated on 3/8/08.
//

#import <Cocoa/Cocoa.h>
#import <QTKit/QTkit.h>
#import "VideoView.h"

@interface VideoDisplayController : NSOpenGLView 
{
    QTCaptureSession *captureSession;
    QTCaptureDevice *videoCaptureDevice;
	QTCaptureDeviceInput *captureVideoDeviceInput;
    QTCaptureDecompressedVideoOutput *decompressedVideoOutput;
   
    //stores most recent frame grabbed from a CVImageBufferRef
	CVImageBufferRef currentImageBuffer;


    //this is the panel on which the video from the DV camera will be displayed
    IBOutlet QTCaptureView *captureView;


    //IBOutlet VideoView *openGLVideoView;

}



- (IBAction)startVideoCapture:(id)sender;
- (IBAction)stopVideoCapture:(id)sender;


@end

thanks again for any help

 

[Sayer]

Yes, you use NSOpenGLView.

There is sample code called CIVideoDemoGL that does a lot of what you want to do and is easier to point to than copy/pasting it into a message board.

 

[jcl43]

Thanks for the info. That seriously helped me out a ton. I still need to figure out how to get the frames from the dvi came...in this example it opens from a file i believe.

 

[Sayer]

QTKit now has video capture support in Leopard and its fairly easy to do.

Lemme look for the sample code I was studying earlier (for a big-time pro-level contract job with a rather large company).

Working...

The sample code is called "StillImage" but it does a lot of what you want e.g. capturing from a live video source and doing something with the frames. Here's some of it:

        mCaptureSession = [[QTCaptureSession alloc] init];
        
        // Find a video device
        QTCaptureDevice *device = [QTCaptureDevice defaultInputDeviceWithMediaType:QTMediaTypeVideo];
        success = [device open:&error];
        if (!success) {
            [[NSAlert alertWithError:error] runModal];
            return;
        }
        
        // Add a device input for that device to the capture session
        mCaptureDeviceInput = [[QTCaptureDeviceInput alloc] initWithDevice:device];
        success = [mCaptureSession addInput:mCaptureDeviceInput error:&error];
        if (!success) {
            [[NSAlert alertWithError:error] runModal];
            return;
        }
        
        // Add a decompressed video output that returns raw frames to the session
        mCaptureDecompressedVideoOutput = [[QTCaptureDecompressedVideoOutput alloc] init];
        [mCaptureDecompressedVideoOutput setDelegate:self];
        success = [mCaptureSession addOutput:mCaptureDecompressedVideoOutput error:&error];
        if (!success) {
            [[NSAlert alertWithError:error] runModal];
            return;
        }
        
        // Preview the video from the session in the document window
        [mCaptureView setCaptureSession:mCaptureSession];
        
        // Start the session
        [mCaptureSession startRunning];

Note that QTKit's video capture is Leopard only (for use in Cocoa apps anyway).

In this implementation you create a few delegate methods (callbacks) that hand off a frame of raw decompressed video that you can manipulate.

// This delegate method is called whenever the QTCaptureDecompressedVideoOutput receives a frame
- (void)captureOutput:(QTCaptureOutput *)captureOutput didOutputVideoFrame:(CVImageBufferRef)videoFrame withSampleBuffer:(QTSampleBuffer *)sampleBuffer fromConnection:(QTCaptureConnection *)connection
{
    // Store the latest frame
	// This must be done in a @synchronized block because this delegate method is not called on the main thread
    CVImageBufferRef imageBufferToRelease;
    
    CVBufferRetain(videoFrame);
    
    @synchronized (self) {
        imageBufferToRelease = mCurrentImageBuffer;
        mCurrentImageBuffer = videoFrame;
    }
    
    CVBufferRelease(imageBufferToRelease);
}

 

[fantastico]

Sayer, from what I'm seeing in jcl43's code, he is already using QTKit to capture the frames from a DV Cam, and even storing the current frame in a CVBufferRef...very similar to what you just posted.

In fact, in the original post, he states:I SUCCESSFULLY GOT THE VIDEO INTO QUICK TIME USING QTKIT, but am having trouble aquiring the frames with a display link.