Feature Measurement

from 360° street images

Panoramic image from Google Streetview

Features that can be be extracted

Features that can be be extracted

(Latitude?, Longitude?)

(Latitude?, Longitude?)

(Latitude?, Longitude?)

(Latitude?, Longitude?)

Photogrammetry concept in Street View Feature Extraction

Get coordinates of Point from position A

Photogrammetry concept in Street View Feature Extraction

Position A

Position B

Point of interest

Photogrammetry concept in Street View Feature Extraction

Position A

Position B

Point of interest

Photogrammetry concept in Street View Feature Extraction

B

A

b

a

P

( NB  x  tan ( b ) )  -  ( NA  x tan ( a ) )  -  EB  +  EA

 tan ( b )  -  tan ( a )

 ( EB  x  cot ( b ) )  -  ( EA  x cot ( a ) )  -  NB  +  NA

cot ( b )  -  cot ( a )

NP =

EP =

(EP, NP)

(EB, NB)

(EA, NA)

Heading A

Heading B

http://www.hugha.co.uk/PhotoInt/SphericalPanos.htm

Photogrammetry concept in Street View Feature Extraction

http://www.hugha.co.uk/PhotoInt/SphericalPanos.htm

A

P

(EP, NP)

Photogrammetry concept in Street View Feature Extraction

Horizontal Angle = (horizontal pixel difference between the reference and the point / horizontal pixels for the image) x Horizontal FOV

Vertical Angle = (vertical pixel difference between the horizon and the point / vertical pixels for the image) x Vertical FOV

Horizontal FOV

Vertical FOV

Automated Feature Detection

from images

Identifying Features

Identifying Features

Histogram of Oriented Gradients (HOG)

For every pixel in an image, it can be calculated in which direction the color changes the fastest.

The gradient would point into the direction where the image is getting darker.

https://kiriproject.wordpress.com/tag/computer-vision/

Histogram of Oriented Gradients (HOG)

An image is separated into cells

https://kiriproject.wordpress.com/tag/computer-vision/

Histogram of Oriented Gradients (HOG)

Most of the time the histograms are visualized in angular histogram form

Example of a STOP sign HOG using Matlab+Python

Histogram of Oriented Gradients (HOG)

Example of a street view image HOG

Histogram of Oriented Gradients (HOG)

A sliding window for each image and scale will slide through the image to detect trained image

Scale-invariant feature transform (SIFT)

SIFT is another descriptor developed based on HOG

Keypoints are extracted from trained images (keypoints localization)

Scale-invariant feature transform (SIFT)

SIFT features

Particular scales and orientations are generated to keypoints ensuring invariance to image location, scale and rotation

Scale-invariant feature transform (SIFT)

Automated Feature Extraction

from 360° street images

Objective

Automatically extract features from streetview images by classifying the feature type and identify its location

Proposed Strategy

Input

Single Image detection

Multi-view  sign positioning

Multi-view sign classification

Output

Input

Single Image detection

Multi-view  sign positioning

Multi-view sign classification

Output

1. Sequence of street view images
2. Image information
   • Capturing position (Latitude & longitude)
   • Camera heading (° facing North)

Input

Single Image detection

Multi-view  sign positioning

Multi-view sign classification

Output

1st stage: HOG detector

Identify if street view image has the feature

• HOG use less resource
• provide positive & negative training data
• retrieve pixel locations of all features

• based on bounding boxes from 1st stage
• algorithm multiplies the huge training data
• generalize type of feature (e.g. warning signs, shop signs, road features etc.)

Input

Single Image detection

Multi-view  sign positioning

Multi-view sign classification

Output

• Identifies detection from consecutive captured images
• Single detection will be identified as false positive
• Apply trigonometry to estimate location

Input

Single Image detection

Multi-view  sign positioning

Multi-view sign classification

Output

• Further accurately classifies feature based on multiple angles of detection
• Text recognition for street signs

Input

Single Image detection

Multi-view  sign positioning

Multi-view sign classification

Output

Challenges

1. Source data
   • Coverage wise
   • Accuracy wise
2. Complicated features
   • Obstructions
   • Look alike features
   • Non-standardized signs
3. Requires large set of training data
   • 10,000 - 50,000 data even for a specific detection