Visual Search at Pinterest

11

Paper: Visual Search at Pinterest by Jing, et al. Visual Discovery, Pinterest and University of California, Berkeley

Some Fundamentals

I’ve started learning fundamentals from Andrew Ng’s Coursera course. After that this deep learning lecture series was very helpful. For fundamentals about terminology of Convolutional Neural Networks (CNN), I recommend this video:

Overview

In this paper, Pinterest team describe:

a) How they approached the build vs buy decision for visual searching. Presented scalable and cost-effective visual search solution using widely available tools

b) Their experiences deploying visual search solutions in two of their product applications: Related Pins, and Similar Looks

Related Pins is a feature that recommends Pins based on the Pins the user is currently viewing.

Similar Looks allowed users to select a visual query for regions of interest (e.g: a bag or a pair of shoes) and identified visually similar pins for users to explore or purchase.

On a related note, this article from Google’s research blog is worth a read, and this piece from Baidu search.

Visual Search Architecture at Pinterest

Features is a fundamental concept in computer vision. A feature may be specific items of interest in the image such as points, edges, or objects. Pinterest architecture uses feature extraction from local and deep features. The deep features are based on convolutional neural networks (CNN) based on AlexNet and VGG architectures.

Pinterest is using Caffe deep learning framework, developed by Berkeley Vision and Learning Center.

Object detection is a two-step process using textual metadata as the first step, and followed by the visual signals. For the visual part, CNNs are used for semantic prediction tasks: classification, detection, and segmentation.

Incremental Fingerprinting Service

There is an excellent description on what Pinterest is calling their Fingerprint service …

We built a system called the Incremental Fingerprinting

Service, which computes image features for all Pinterest images using a cluster of workers on Amazon EC2. It incrementally updates the collection of features under two main change scenarios: new images uploaded to Pinterest, and feature evolution (features added/modified by engineers).

We copy these features into various forms for more convenient access by other jobs: features are merged to form a fingerprint containing all available features of an image, and fingerprints are copied into sharded, sorted files for random access by image signature (MD5 hash). These joined fingerprint files are regularly re-materialized, but the expensive feature computation needs only be done once per image.

I recommend this paper for anyone has remote interest in image search area. There is an excellent write up on Related Pins and Similar Looks applications. This paper also has some insights in Pinterest’s A/B testing and how they did live experiments.

Paper: Visual Search at Pinterest by Jing, et al. Visual Discovery, Pinterest and University of California, Berkeley

P.S: Don’t miss Andrej Karpathy’s academic website on Deep Learning.

Related Posts

My New Stories

11product strategy pitfalls
11Customer needs
11Confirmation bias in Product Discovery