Bringing the power of OpenCV to open web platfrom

Sajjad Taheri

Donald Bren School of Computer Science, UCIrvine
 

Previous works

WebRTC

• Real-time communication for the Web
  • HTML5 Tags and JavaScript API
    • Access media devices
    • Establish peer-to-peer connections
    • Transfer data and media contents
  • Set of underlying protocols
• Industry effort
  • Supported by Chrome, Firefox, Opera
• Individual and enterprise users

WebRTCBench

• Previous work conducted at UC Irvine supported by Intel SSG grants
• Measures complete features/aspects of WebRTC
  • Browser independent measurements
    • Complete call measurement
    • Including signaling and ICE
    • Media Stream Measurements
    • Data channel Measurements
• Paper published at ESTIMedia 2015
• Source code available at GitHub

• High performance, cross-platform processing of color/depth
• A JavaScript Open Source CV Library
  • Same functionality as Open CV routines
  • Vectorization of core routines

Towards perceptual computing and computer vision for the web

Last Year Agenda

OpenCV

1. A popular Open Source Library for real-time computer vision
   1. Image/video processing
   2. object detection
   3. image coding
2. SIMD acceleration
3. Bindings for other languages and platforms
   1. Python, Java, Android

OpenCV on Web

• OpenCV has an active community
• Having the experience on web with good performance would be great

• Work conducted by Mozilla FoxEye project as well
• Collaboration with Mozilla to push the work into official OpenCV repo

asm.js

asm.js is typed subet of JavaScript offering near native performance

OpenCV C++ code

LLVM Bitcode

From C++ to JavaScript

Emscripten is a source to source compiler from Mozilla

Binding glue code

LLVM Bitcode

Binding Generation

• "asm.js" names are mangled

• Bind C++ entities to JavaScript so can be used as a normal

• extracts all the symbols with OpenCV Wrap macros
  • enumerations and constants
  • properties
  • class functions
  • functions
  • classes
• generates appropriate binding for each type

C++ code

JS code

void erode(){
   // Read image
   Mat image,dst;
   image = imread("image.jpg", CV_LOAD_IMAGE_COLOR);  
 
   // Create a structuring element
   int erosion_size = 6;  
   Mat element = getStructuringElement(cv::MORPH_RECT,
   cv::Size(2 * erosion_size + 1, 2 * erosion_size + 1),
   cv::Point(erosion_size, erosion_size) );
 
   // Apply erosion or dilation on the image
   erode(image,dst,element);  
}

function erode(){
    // Read image
    var src = cv.imread("image.jpg", 
                        cv.CV_LOAD_IMAGE_COLOR.value);  
    var dst = new cv.Mat();

    // Create a structuring element
    var erosion_type = cv.cv_MorphShapes.MORPH_RECT.value;
    var erosion_size=new cv.Size(2*Control.erosion_size+1,
         2*Control.erosion_size+1 );
    var element = cv.getStructuringElement( erosion_type 
        ,erosion_size  ) ;

    
    // Apply erosion or dilation on the image   
    cv.erode(src, dst, element) ;
				
}

C++ vs JS code

function __ZN2cv8haar_cvtL7convertERKNS_6StringES3_($oldcascade,
                                                    $newcascade){
}

CV_WRAP static bool convert(const String& oldcascade, 
                            const String& newcascade){
}

CV_EXPORTS_W void erode( InputArray src, 
                         OutputArray dst, 
                         InputArray kernel,
                         Point anchor = Point(-1,-1), 
                         int iterations = 1,
                         int borderType = BORDER_CONSTANT,
                         const Scalar& borderValue = morphologyDefaultBorderValue());

Results

• Most modules are covered

  • core
  • image/video processing
  • object detection
  • feature extraction
  • image codecs
  • machine learning
  • shape analysis

60+ classes

about 800 functions(including class methods)

50+ Enums

• Function overloading
• Default parameters

Performance

• How good it is actually.....

Demos

Image processing

Features extractions

Object detection using HAAR

Simd.js

• JavaScript API to expose SIMD capabilities to browsers
• Based on Intel SSE2 and ARM NEON
• My summer work with Mozilla on gl-matrix showed promising results
  • 3.6x speedup for 4 by 4 matrix multiplication
•  

Next steps

Thanks!

Backup Slides

MSER: Sample class from OpenCV

class CV_EXPORTS_W MSER : public Feature2D
{
public:
    //! the full constructor
    CV_WRAP static Ptr<MSER> create( int _delta=5, int _min_area=60, int _max_area=14400,
          double _max_variation=0.25, double _min_diversity=.2,
          int _max_evolution=200, double _area_threshold=1.01,
          double _min_margin=0.003, int _edge_blur_size=5 );

    CV_WRAP virtual void detectRegions( InputArray image,
                                        CV_OUT std::vector<std::vector<Point> >& msers,
                                        std::vector<Rect>& bboxes ) = 0;

    CV_WRAP virtual void setDelta(int delta) = 0;
    CV_WRAP virtual int getDelta() const = 0;

    CV_WRAP virtual void setMinArea(int minArea) = 0;
    CV_WRAP virtual int getMinArea() const = 0;

    CV_WRAP virtual void setMaxArea(int maxArea) = 0;
    CV_WRAP virtual int getMaxArea() const = 0;

    CV_WRAP virtual void setPass2Only(bool f) = 0;
    CV_WRAP virtual bool getPass2Only() const = 0;
};

• OpenCV uses Macros to identify symbols to binding generators

• Image/Video are coming from HTML Canvas

• interface to grab/show image from HTML canvas

• Canvas API gets images in RGBA format

  • OpenCV likes RGB better

• Canvas only supports data with 8 bits per pixels

  • depth data are represented with 16 bits per pixels

  • work in progress to fix this

Input/Output Interface

Binding code for MSER Class

emscripten::class_<cv::MSER ,base<Feature2D>>("MSER")
    
    .function("getPass2Only", 
              select_overload<bool(cv::MSER&)>(&Wrappers::MSER_getPass2Only_wrap),
              pure_virtual())
    .function("setMinArea", 
              select_overload<void(cv::MSER&,int)>(&Wrappers::MSER_setMinArea_wrap),
              pure_virtual())
    .function("getDelta", 
              select_overload<int(cv::MSER&)>(&Wrappers::MSER_getDelta_wrap),
              pure_virtual())
    .class_function("create",
              select_overload<Ptr<MSER>(int,int,int,double,double,int,double,double,int)>(&Wrappers::create_MSER_wrapper))
    .function("getMaxArea", 
              select_overload<int(cv::MSER&)>(&Wrappers::MSER_getMaxArea_wrap),
              pure_virtual())
    .function("setMaxArea", select_overload<void(cv::MSER&,int)>(&Wrappers::MSER_setMaxArea_wrap),
              pure_virtual())
    .function("setPass2Only", select_overload<void(cv::MSER&,bool)>(&Wrappers::MSER_setPass2Only_wrap),
              pure_virtual())
    .function("getMinArea", select_overload<int(cv::MSER&)>(&Wrappers::MSER_getMinArea_wrap),
              pure_virtual())
    .function("detectRegions", 
              select_overload<void(cv::MSER&,
                        const cv::Mat&, 
                        std::vector<std::vector<Point>>&,
                        std::vector<Rect>&)>(&Wrappers::MSER_detectRegions_wrap),
              pure_virtual())
    .function("setDelta", 
              select_overload<void(cv::MSER&,int)>(&Wrappers::MSER_setDelta_wrap),
              pure_virtual())
;