Fact: Clarifai’s Search API allows you to search your images and video by visual similarity. Lesser known fact: it also lets you search your media by geolocation data! Learn more about this feature and how to put it into action for your own developer application.

The Search API allows you to send images to Clarifai and have them indexed using concepts and their visual representation. After they are indexed, you can search over your inputs using concepts (e.g. dog) or images (e.g. visually similar dogs). Clarifai can extend this search by adding extra data points onto our inputs such as geolocation data. A search by geolocation acts as a filter of inputs so you get only results within a specified range.

We’ll look at attaching geolocation data to your inputs and then query that data with different measurements of distance to see the change in results. For this tutorial, we are going to use Node.js. Let’s get started by installing the official Clarifai JavaScript client with

npm install clarifai

Adding Inputs

You need to sign up for Clarifai and create an application before you can get started. Inputs are added using either a URL or bytes. Along with your input source we’ll add a geo object that will contain keys for GPS coordinates (longitude and latitude). Remember that the coordinate system is using the cardinal directions: North, South, East, and West. North and East will be positive numbers and South and West will be negative numbers.

Below we will add an image of the Statue of Liberty using a URL from Wikipedia:

const Clarifai = require('clarifai')
const app = new Clarifai.App({ apiKey: 'YOUR_API_KEY_HERE' })

app.inputs.create({
  url: "https://upload.wikimedia.org/wikipedia/commons/thumb/a/a1/Statue_of_Liberty_7.jpg/1200px-Statue_of_Liberty_7.jpg",
  geo: {
    latitude: 40.689247,
    longitude: -74.044502
  }
})

and use a local image of the Golden Gate Bridge from HISTORY and a function for file-to-base64 conversion:

const Clarifai = require('clarifai')
const app = new Clarifai.App({ apiKey: 'YOUR_API_KEY_HERE' })

const convert_bytes = (img) => {
  const img_file = fs.readFileSync(img)
  return new Buffer(img_file).toString('base64')
}

app.inputs.create({
  base64: convert_bytes('./golden_gate.jpg')
  geo: {
    latitude: 37.807812,
    longitude: -122.475164
  }
})

Searching for Images

Once your images are uploaded, you will be able to use them with search. When searching by geolocation, you can refine your results using a single point and some radius given ‘withinMiles’, ‘withinKilometers’, ‘withinDegrees’, or ‘withinRadians’.

Let’s say we only want results of images within a mile of the Empire State Building in New York City (because it’s the best city in the world, naturally). Our search would look like this:

app.inputs.search({
  input: {
    geo: {
      latitude: 40.748817,
      longitude: -73.985428,
      type: 'withinMiles',
      value: 1.0
    }
  }
}).then((response) => { console.log(response.hits)})

// Response
[]

The above example returned no results due to how small the range is. If we were to increase this value we will get our hit.

app.inputs.search({
  input: {
    geo: {
      latitude: 40.748817,
      longitude: -73.985428,
      type: 'withinMiles',
      value: 7.0
    }
  }
}).then((response) => { console.log(response.hits)})

// Response
[{ 
  score: 1,
  input: { 
    id: 'd7b80aac52f14399b98a9472fc201e64',
    data: [ 
      { 
        score: 1,
        input: { 
          id: 'd7b80aac52f14399b98a9472fc201e64',
          data: { 
            image: { 
              url: 'https://upload.wikimedia.org/wikipedia/commons/thumb/a/a1/Statue_of_Liberty_7.jpg/1200px-Statue_of_Liberty_7.jpg' },
              concepts: [
                { 
                id: 'statue',
                  name: 'statue',
                  value: 1,
                  app_id: 'fb70b904750c4891aecddf82082181c2' 
                },
                { 
                  id: 'bridge',
                  name: 'bridge',
                  value: 0,
                  app_id: 'fb70b904750c4891aecddf82082181c2' 
                }
              ],
              metadata: {},
              geo: { 
                geo_point: { 
                  longitude: -74.0445, 
                  latitude: 40.689247 
                }
              }
          },
          created_at: '2017-09-05T17:52:38.616686Z',
          modified_at: '2017-09-05T17:52:39.029363Z',
          status: [Object] 
        } 
      } 
    ],
    created_at: '2017-09-05T17:52:38.616686Z',
    modified_at: '2017-09-05T17:52:39.029363Z',
    status: [Object] 
  } 
}]

The search result will give us an array of matches. Each object within the array is one of our inputs that matches the search criteria. This also includes a score that will indicate how much of a match this the result is to our query. Whenever you want to access any of the data related to a specific object you would need to access its input.data key. You can see what the custom concepts, geolocation data, or custom metadata it has attached to it. You can also find the input’s original url or bytes in this object.

Conclusion

The Search API is a powerful tool and searching using geolocation is just one of many useful features for developers. You can do much more with the Search API like:

Search using Public or Custom concepts
Search using your own custom metadata on inputs
Combine search options usingANDing

If you have any questions, concerns, or even friendly notes feel free to reach out to us over at hackers@clarifai.com or comment below!

Getting Started with Search by Geo Location

Adding Inputs

Searching for Images

Conclusion