Monthly Archives: September 2011

Optimizing Access to Raw Camera Frames on Android – Part 1.

If you want to do some real time processing on the raw camera frames, e.g., you are working on an augmented reality app, then ensuring a high fps rate is essential. If you also want to support the 1.5 OS version of Android named cupcake, then you will quickly find that the camera API has a major flaw. On OS versions below 2.2 you are forced to use the old setPreviewCallback function. The problem with this method is that it allocates a new buffer for every preview frame, copies the frame to this buffer and gives it back in a callback. Apart from allocating memory is generally slow, the gc have to be called constantly to clean up the buffers. Setting the fps to 15 and the preview frame to the relative low resolution CIF format (352*288) will result in 2280960 byte allocation in every second. If we take a G1 phone with 192MB RAM – from which only 15-20MB can be used by a given application – it makes clear that this API can be very limiting.

I’ll show a method that bypasses the public preview callback API using private APIs. Using private APIs in production code is dangerous as it can broke on devices you have never saw or after a firmware update. I’ll also show some methods to minimize this risk by implementing fallback mechanisms in later blog posts.

You will need a linux or mac to build the Android source code yourself. Building the source codes on mac is a bit tricky. To get the source go to source.android.com. After the download, build it with make -j2. If this is the first time you see the Android source you should definitely explore it a bit. (***At the time of this writing the android source code can’t be downloaded because the kernel.org has been hacked a few weeks before. Try alternate download locations, maybe here.)

In the following sections I’ll create a small C++ app that utilizes the private Camera APIs. It would help a lot if you are familiar with the Android NDK (especially with JNI), however the app will be built inside the Android source and not with the NDK.

First create a directory under cupcake_src/external/ and name it mycam. After that create the following files CupCam.h, CupCam.cpp, native.cpp, Main.h, Main.cpp and Android.mk.

CupCam.h.


#ifndef CUPCAM_H
#define CUPCAM_H

//352*288*3/2=152064
#define BUFLENGTH 152064

//STRING8
#include "jni.h"
#include "JNIHelp.h"
#include "android_runtime/AndroidRuntime.h"

#include "utils/IMemory.h"
#include <ui/Surface.h>
#include "ui/Camera.h"

namespace android{

typedef void (*frame_cb)(void* mem, void *cookie);
typedef void (*error_cb)(int err, void *cookie);

struct CamContext{
	frame_cb rec_cb;
	void* rec_cb_cookie;
};

class CupCam {
	bool hasCamera;
	sp<Camera> camera;
	sp<Surface> mSurface;
	error_cb err_cb;
	void* err_cb_cookie;
	int rec_flag;
	CamContext* mCamContext;

public:
	CupCam();
	virtual ~CupCam();
	virtual void setSurface(int* surface);
	virtual bool initCamera();
	virtual void releaseCamera();
	virtual void setRecordingCallback(frame_cb cb, void* cookie, int flag=FLAG_CAMERA);
	virtual void setErrorCallback(error_cb ecb, void* cookie);
	virtual void startPreview();
	virtual void stopPreview();
};

}//namespace

#endif /* CUPCAM_H */

This class will handle us the Camera in native code bypassing the public Java Camera API.

CupCam.cpp


#define LOG_TAG "MyCupCam"
#include <utils/Log.h>

#define DEBUG_LOG 0

#include "CupCam.h"

namespace android{

volatile bool isDeleting=false;

void main_rec_cb(const sp<IMemory>& mem, void *cookie){

    CamContext* context = reinterpret_cast<CamContext*>(cookie);
    if(context == NULL) {
    	LOGE_IF(DEBUG_LOG,"context is NULL in main_rec_cb");
        return;
    }
    ssize_t offset;
    size_t size;
    sp<IMemoryHeap> heap = mem->getMemory(&offset, &size);
    unsigned char* inBuf = ((unsigned char*)heap->base()) + offset;

    if(!isDeleting){
    	context->rec_cb(inBuf, context->rec_cb_cookie);
    }
}

CupCam::CupCam() {
	LOGD_IF(DEBUG_LOG, "constructor");
	hasCamera = false;
	err_cb = NULL;
	err_cb_cookie = NULL;
	mCamContext = new CamContext();
	mCamContext->rec_cb = NULL;
	mCamContext->rec_cb_cookie = NULL;
	rec_flag=FRAME_CALLBACK_FLAG_NOOP;
}

CupCam::~CupCam() {
	LOGD_IF(DEBUG_LOG, "destructor");
	releaseCamera();
	if(mCamContext){
		delete mCamContext;
	}
}

void CupCam::setSurface(int* surface){
	LOGD_IF(DEBUG_LOG, "setSurface");
	mSurface = reinterpret_cast<Surface*> (surface);
}

bool CupCam::initCamera(){
	LOGD_IF(DEBUG_LOG, "initCamera");
	camera = Camera::connect();
	//make sure camera hardware is alive
	if(camera->getStatus() != NO_ERROR){
		LOGD_IF(DEBUG_LOG, "camera initialization failed");
		return false;
	}

	camera->setErrorCallback(err_cb, err_cb_cookie);

	if(camera->setPreviewDisplay(mSurface) != NO_ERROR){
		LOGD_IF(DEBUG_LOG, "setPreviewDisplay failed");
		return false;
	}

	const char* params = "preview-format=yuv420sp;preview-frame-rate=15;"
			"picture-size=355x288"
			";preview-size=355x288"
			";antibanding=auto;antibanding-values=off,50hz,60hz,auto;"
			"effect-values=mono,negative,solarize,pastel,mosaic,resize,sepia,posterize,whiteboard,blackboard,aqua;"
			"jpeg-quality=100;jpeg-thumbnail-height=240;jpeg-thumbnail-quality=90;jpeg-thumbnail-width=320;"
			"luma-adaptation=0;nightshot-mode=0;picture-format=jpeg;"
			"whitebalance=auto;whitebalance-values=auto,custom,incandescent,fluorescent,daylight,cloudy,twilight,shade";

	String8 params8 = String8(params, 510);
	camera->setParameters(params8);
	if(mCamContext->rec_cb){
		camera->setPreviewCallback(main_rec_cb, mCamContext, rec_flag);
		isDeleting=false;
	}
	hasCamera = true;
	return true;
}

void CupCam::releaseCamera(){
	LOGD_IF(DEBUG_LOG, "releaseCamera");
	if(hasCamera){
		isDeleting=true;
		camera->setPreviewCallback(NULL, NULL, FRAME_CALLBACK_FLAG_NOOP);
		camera->setErrorCallback(NULL, NULL);
		camera->disconnect();
		hasCamera = false;
	}
}

void CupCam::setRecordingCallback(frame_cb cb, void* cookie, int flag){
	LOGD_IF(DEBUG_LOG, "setRecordingCallback");
	CamContext* temp = new CamContext();
	temp->rec_cb = cb;
	temp->rec_cb_cookie = cookie;
	rec_flag=flag;
	if(hasCamera){
		if(temp->rec_cb == NULL){
			isDeleting=true;
			camera->setPreviewCallback(NULL, NULL, FRAME_CALLBACK_FLAG_NOOP);
		} else{
			camera->setPreviewCallback(main_rec_cb, temp, rec_flag);
			isDeleting=false;
		}
	}
	delete mCamContext;
	mCamContext=temp;
}

void CupCam::setErrorCallback(error_cb ecb, void* cookie){
	LOGD_IF(DEBUG_LOG, "setErrorCallback");
	err_cb=ecb;
	err_cb_cookie=cookie;
	if(hasCamera){
		camera->setErrorCallback(err_cb, err_cb_cookie);
	}
}

void CupCam::startPreview(){
	LOGD_IF(DEBUG_LOG, "startPreview");
	if(hasCamera){
		camera->startPreview();
	}
}

void CupCam::stopPreview(){
	LOGD_IF(DEBUG_LOG, "stopPreview");
	if(hasCamera){
		camera->stopPreview();
	}
}

}//namespace

Let’s continue with the JNI glue Native.java and Native.cpp.

Native.java


package com.example;

import android.content.Context;

public class CopyOfNative {
	static {
		System.loadLibrary("Native");
	}

	public static native void initCamera(Object surface);
	public static native void releaseCamera();
	public static native void startPreview();
	public static native void stopPreview();
}

Native.cpp


#ifndef LOG_TAG
#define LOG_TAG "Native"
#include <utils/Log.h>
#endif

#include "jni.h"
#include "JNIHelp.h"
#include "android_runtime/AndroidRuntime.h"

#include "Main.h"

using namespace android;

Main* main=NULL;

static void initCamera(JNIEnv *env, jobject thiz, jobject jSurface){
	jclass surfaceClass = env->FindClass("android/view/Surface");
	jfieldID surfaceField = env->GetFieldID(surfaceClass, "mSurface", "I");

	int* surface = (int*)(env->GetIntField(jSurface, surfaceField));

	main->initCamera(surface);
}

static void releaseCamera(JNIEnv *env, jobject thiz){
	main->releaseCamera();
}

static void startPreview(JNIEnv *env, jobject thiz){
	main->startPreview();
}

static void stopPreview(JNIEnv *env, jobject thiz){
	main->stopPreview();
}

//Path to the Java part of the jni glue, e.g., com/example/Native
static const char *classPathName = "your/path/to/Native";
static JNINativeMethod methods[] = {
	{ "releaseCamera", "()V", (void*) releaseCamera },
	{ "initCamera", "(Ljava/lang/Object;)V", (void*) initCamera },
	{ "startPreview", "()V", (void*) startPreview },
	{ "stopPreview", "()V", (void*) stopPreview }
};

//Register several native methods for one class.
static int registerNativeMethods(JNIEnv* env, const char* className,
		JNINativeMethod* gMethods, int numMethods) {
	jclass clazz = env->FindClass(className);
	if (clazz == NULL) {
		LOGE_IF(DEBUG_LOG,"Native registration unable to find class '%s'", className);
		return JNI_FALSE;
	}
	if (env->RegisterNatives(clazz, gMethods, numMethods) < 0) {
		LOGE_IF(DEBUG_LOG,"RegisterNatives failed for '%s'", className);
		return JNI_FALSE;
	}
	return JNI_TRUE;
}

//Register native methods for all classes we know about.
static int registerNatives(JNIEnv* env) {
	if (!registerNativeMethods(env, classPathName, methods, sizeof(methods)
			/ sizeof(methods[0]))) {
		return JNI_FALSE;
	}
	return JNI_TRUE;
}

//This is called by the VM when the shared library is first loaded.
jint JNI_OnLoad(JavaVM* vm, void* reserved) {
	JNIEnv* env = NULL;
	jint result = -1;

	if (vm->GetEnv((void**) &env, JNI_VERSION_1_4) != JNI_OK) {
		LOGE_IF(DEBUG_LOG,"ERROR: GetEnv failed\n");
		goto bail;
	}
	assert(env != NULL);

	if (registerNatives(env) < 0) {
		LOGE_IF(DEBUG_LOG,"ERROR: native registration failed\n");
		goto bail;
	}

	/* success -- return valid version number */
	result = JNI_VERSION_1_4;

	main = new Main();

	bail: return result;
}

The CupCam and the JNI glue assumes a few things about your code. First, the heavy processing on the camera frames will take place at Main.cpp. Second, you have to send “down” a surface object through Native.initCamera() on which the Camera preview frames will be shown.

Let’s see a dummy implementation for some functions in the Main class.


static void rec_callback(void* mem, void* cookie){
	Main* c = (Main*) cookie;
	c->recordingCallback(mem);
}

void Main::recordingCallback(void* mem){
	tUint8 *memBuf = (tUint8 *) mem;
	memcpy(buf, memBuf, BUFLENGTH);
	//do some stuff
}

void Main::releaseCamera(){
	CupCam->releaseCamera();
}

void Main::initCamera(int* surface){
	CupCam->setSurface(surface);
	CupCam->initCamera();
}

void Main::startPreview(){
	CupCam->startPreview();
}

void Main::stopPreview(){
	CupCam->stopPreview();
}

Starting and stopping the preview callbacks are best left to the Surface object. Some events can be controlled only in the Java level. A simple implementation for the CameraPreview.java could be the following.

CameraPreview.java


class CameraPreview extends SurfaceView implements SurfaceHolder.Callback {
	private final static String TAG = "CameraPreview";

	protected static SurfaceHolder mHolder;

	CameraPreview(Context context) {
		super(context);
		init();
	}

	public CameraPreview(Context context, AttributeSet attrs) {
		super(context, attrs);
		init();
	}

	private void init() {
		setFocusable(true);
		mHolder = getHolder();
		mHolder.addCallback(this);
		mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
	}

	public void surfaceCreated(SurfaceHolder holder) {
		Native.initCamera(mHolder.getSurface());
		Native.startPreview();
	}

	public void surfaceDestroyed(SurfaceHolder holder) {
			Native.stopPreview();
			Native.releaseCamera();
	}

	public void surfaceChanged(SurfaceHolder holder, int format, int w, int h) {
		//empty
	}
}

All we need is to compile the native codes into a dynamic library. For this we would need an Android.mk makefile.

Android.mk


LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)

LOCAL_ARM_MODE := arm

LOCAL_SRC_FILES := \
native.cpp \
Main.cpp \
CupCam.cpp

LOCAL_MODULE := libNative

LOCAL_C_INCLUDES := \
$(LOCAL_PATH) \
$(LOCAL_PATH)/includes \
$(ANDR_ROOT)/frameworks/base/camera/libcameraservice \
$(ANDR_ROOT)/frameworks/base/include/media \
$(ANDR_ROOT)/frameworks/base/include/binder \
$(ANDR_ROOT)/frameworks/base/include/utils

LOCAL_CFLAGS += -w -Wno-trigraphs -Wreturn-type -Wunused-variable -std=gnu99 -fPIC -O3 -fno-strict-aliasing -Wno-write-strings 

LOCAL_MODULE_SUFFIX := $(HOST_JNILIB_SUFFIX)

LOCAL_SHARED_LIBRARIES := \
	libandroid_runtime \
	libnativehelper \
	libcutils \
	libutils \
	libcameraservice \
	libmedia \
	libdl \
	libui \
	liblog \
    libicuuc \
    libicui18n \
    libsqlite
	
LOCAL_C_INCLUDES += $(JNI_H_INCLUDE)
LOCAL_LDLIBS := -lpthread -ld

LOCAL_PRELINK_MODULE := false

include $(BUILD_SHARED_LIBRARY)

Compiling code in the Android source is as easy as using make.


$ cd cupcake_source/external/mycam/
$ source ../../build/envsetup.sh
$ mm

The compiled library can be found at cupcake_source/system/lib/libNative.so. You have to copy this so to your Android project’s /libs/armeabi/ folder. If there are no such folders, create them as you would do if you were using Android NDK.

The code is not ready for release. The biggest problem that it works only on 1.5 OS. I’ll make it much more smarter in the following blog posts.