Optimizing Access to Raw Camera Frames on Android – Part 2.

In the previous post I have shown a way how to utilize the private Camera API on Android 1.5. If you run that code on later Android OS versions it will crash. If you don’t want to release a separate app for all OS versions – which would be very ugly – you have to solve this problem somehow. At first let’s check what has changed on Android 1.6 compared to 1.5. You can browse the source code at the time of this writing on GitHub. The file you should check out first is ui/Camera.h.
One major part that changed is the callback handling. The recording/preview callbacks from 1.5 have been moved to a separate CameraListener class on 1.6. There are other changes too, but those don’t seem to be that severe.

Let’s assume that we were able to utilize the 1.6 Camera API too, just as we did with on 1.5. There are still some problems after that. First, it would be good if we could put the camera handling codes under a common interface and the rest of our code won’t have to bother about on which Android version it will run. Secondly, and this one is more important than the previous one, we have to find a way to separate the private API calls from the main codebase. Because System.loadLibrary() could fail on us if our code can call both 1.5 and 1.6 codes.

These problems can be solved by applying the Wrapper/Adapter design pattern for the first one, and applying the Bridge design pattern combined with dynamic loading for the second one. So we create a camera interface, e.g. let’s call it MyICam. It will be capable of encapsulating the common methods that can be called on a Camera – like startPreview() -, it will have it’s own error messages, camera flags and constants that can serve both the 1.5 and 1.6 APIs. Next we create the OS specific CupCam and DonutCam classes that are implementing the MyICam interface. We compile these into two separate dynamic libraries, the CupCam in the 1.5 Android source, the DonutCam in the 1.6 Android source. And at last we create a manager class that loads the right camera class for the current OS it runs on and returns a MyICam pointer hiding the concrete implementation from the rest of the code.

Let’s start with a possible implementation for the interface MyICam.

MyICam.h.


#ifndef MYICAM_H
#define MYICAM_H

#include <ui/Surface.h>

namespace android{

typedef void (*frame_cb)(void* mem, void *cookie);
typedef void (*error_cb)(int err, void *cookie);
typedef void (*autofocus_cb)(bool focused, void *cookie);

class MyICam{

public:
	//flags
	static const int F_JUSTPREVIEW = 0x00;
	static const int F_CAMCORDER = 0x01;
	static const int F_CAMERA = 0x05;
	//errors
	static const int M_NO_ERROR = 0;
	static const int M_DEAD_OBJECT = -32;
	static const int M_UNKNOWN_ERROR = 0x80000000;

	static const int PREVIEW_FORMAT_YUV420SP = 0x01;
	static const int PREVIEW_FORMAT_YUV420P = 0x02;

	int previewFormat;

	virtual ~MyICam(){};
	virtual void setSurface(int* surface)=0;
	virtual bool initCamera()=0;
	virtual void releaseCamera()=0;
	virtual void setRecCallback(frame_cb cb, void* cookie, int flag=F_CAMERA)=0;
	virtual void setErrCallback(error_cb ecb, void* cookie)=0;
	virtual void setAutoFocusCallback(autofocus_cb cb, void* cookie)=0;
	virtual void startPreview()=0;
	virtual void stopPreview()=0;
	virtual void autoFocus()=0;

	int getPreviewFormat() {return previewFormat;}
	void setPreviewFormat(int f) {previewFormat=f;}

};

// the types of the class factories
// These must be implemented on the name of "create" and "destroy"
typedef MyICam* createCamera_t();
typedef void destroyCamera_t(MyICam*);

}//namespace

#endif /* MYICAM_H */

The interface has a rich set of features, even some of which you don’t necessarily need in your project, like autofocus handling. Most device’s camera returns frames in YUV420SP format, so you could erase those too. The narrower the feature set of the interface, the less work you have to do to implement them in the concrete classes.
The last two typedefs could be confusing at first. They are needed because dlopen() and dlsym() are inherently C APIs and can’t deal with c++ objects. We have to implement two C functions – create and destroy -, that can be loaded by dlsym(). They will be responsible to create and delete the c++ camera objects for us.

Let’s see an implementation for DonutCam.h and DonutCam.cpp.

DonutCam.h


#ifndef DONUTCAM_H
#define DONUTCAM_H

#include "MyICam.h"

#include "android_runtime/AndroidRuntime.h"

#include "IMemory.h"
#include <ui/Surface.h>
#include "ui/Camera.h"

namespace android{

struct CamContext{
	frame_cb rec_cb;
	void* rec_cb_cookie;
};

struct AutoFocusContext{
	autofocus_cb af_cb;
	void* af_cb_cookie;
};

class DonutCam : public MyICam {
	bool hasCamera;
	bool hasListener;
	sp<Camera> camera;
	sp<Surface> mSurface;
	error_cb err_cb;
	void* err_cb_cookie;
	int rec_flag;
	CamContext* context;
	AutoFocusContext* aFContext;
	autofocus_cb autoFocusCallback;

public:
	DonutCam();
	virtual ~DonutCam();
	virtual void setSurface(int* surface);
	virtual bool initCamera();
	virtual void releaseCamera();
	virtual void setRecCallback(frame_cb cb, void* cookie, int flag=F_CAMERA);
	virtual void setErrCallback(error_cb ecb, void* cookie);
	virtual void setAutoFocusCallback(autofocus_cb cb, void* cookie);
	virtual void startPreview();
	virtual void stopPreview();
	virtual void autoFocus();
};

extern "C" MyICam* create() {
    return new DonutCam;
}

extern "C" void destroy(MyICam* iCam) {
    delete iCam;
}

}//namespace

#endif /* DONUTCAM_H */

At next, let’s see the interesting parts of DonutCam.cpp, e.g. where it differs most from CupCam.cpp.

DonutCam.cpp


namespace android{

class MyCamListener: public CameraListener
{
public:
    MyCamListener();
    ~MyCamListener() {release();}
    void setRecCallback(CamContext* context);
    void setAutoFocusCallback(AutoFocusContext* context);
    virtual void notify(int32_t msgType, int32_t ext1, int32_t ext2);
    virtual void postData(int32_t msgType, const sp<IMemory>& dataPtr);
    virtual void postDataTimestamp(nsecs_t timestamp, int32_t msgType, const sp<IMemory>& dataPtr);
    void release();

private:
    void copyAndPost(JNIEnv* env, const sp& dataPtr, int msgType);

    jobject     mCameraJObjectWeak;     // weak reference to java object
    jclass      mCameraJClass;          // strong reference to java class
    sp  mCamera;                // strong reference to native object
    CamContext* mContext;
    AutoFocusContext* mAFContext;
};

MyCamListener::MyCamListener(){	
	mContext = NULL;
	mAFContext = NULL;
}
void MyCamListener::setRecCallback(CamContext* context) {
	mContext = context;
}
void MyCamListener::setAutoFocusCallback(AutoFocusContext* context) {
	mAFContext = context;
}
void MyCamListener::notify(int32_t msgType, int32_t ext1, int32_t ext2){
	if (mAFContext && msgType == CAMERA_MSG_FOCUS) {
		if (mAFContext->af_cb){
			mAFContext->af_cb(ext1, NULL);
		}
	}
}
void MyCamListener::postDataTimestamp(nsecs_t timestamp, int32_t msgType, const sp<IMemory>& dataPtr){	
	postData(msgType, dataPtr);
}
void MyCamListener::postData(int32_t msgType, const sp<IMemory>& dataPtr){
	if ((dataPtr != NULL) && (mContext != NULL) && (mContext->rec_cb != NULL)) {
		ssize_t offset;
		size_t size;
		sp<IMemoryHeap> heap = dataPtr->getMemory(&offset, &size);
		unsigned char* caaInBuf = ((unsigned char*)heap->base()) + offset;
		mContext->rec_cb(caaInBuf, mContext->rec_cb_cookie);
	}
}

sp<CamListener> listener;

//rest of DonutCam.cpp ...

The rest of the CupCam and DonutCam codes are not very interesting after the previous post.

As I mentioned above we need a some kind of manager class that loads the appropriate camera object for the given OS version. Let’s call this class LoadManager.

LoadManager.h


//...includes
#include "MyICam.h"

namespace android{

class LoadManager {
	int SDK;
	char* dataDir;
public:
	LoadManager();
	~LoadManager();

	void setSDKVersion(int sdk);
	void setDataDir(const char* datadir);

	MyICam* createCam();
	void destroyCam(MyICam*);
};

}//namespace

LoadManager.cpp


namespace android {

createCamera_t* createCamera;
destroyCamera_t* destroyCamera;
void* handle = NULL;

void LoadManager::setSDKVersion(int sdk) {
	this->SDK = sdk;
}

MyICam* LoadManager::createCam(){
	char *error=NULL;
	if(!handle){
		char dir[150];
		switch(SDK){
		case 3:
			snprintf(dir, 150, "%s/lib/libcupcam.so",dataDir);
			handle = dlopen(dir, RTLD_LAZY);
			if(!handle){
				//Need to handle it later!
			}
			break;
		case 4:
			snprintf(dir, 150, "%s/lib/libdonutcam.so",dataDir);
			handle = dlopen(dir, RTLD_LAZY);
			if(!handle){
				//Need to handle it later!
			}
			break;
		}

		createCamera = (createCamera_t*) dlsym(handle, "create");
		const char* dlsym_error = dlerror();
		if(dlsym_error){
			//Need to handle it later!
		}

		destroyCamera = (destroyCamera_t*) dlsym(handle, "destroy");
		dlsym_error = dlerror();
		if (dlsym_error) {
			//Need to handle it later!
		}
	}
	return createCamera();
}


void LoadManager::destroyCam(MyICam* iCam){
	destroyCamera(iCam);
}

//...rest of LoadManager

Now we have everything that is needed to support both 1.5 and 1.6 versions at the same time within one application. We still need to compile the CupCam and DonutCam sources into separate so-s, but it’s an easy task after the previous post. We didn’t handle if something goes wrong during dlopen() or dlsym(), but it is not very apparent what we could do in that case. I’ll cover it in the following blog post.