How to Search Through Different Social Media Platforms with Node.js

Shahed Nasser - May 9 '21 - - Dev Community

This article was originally posted on my personal blog

In this tutorial, we'll go over how to search through each of Twitter, Instagram, Tumblr, and Pinterest. We'll be setting up a server with express with a simple setup just to demonstrate the functionalities.

You can find the code for this tutorial in this GitHub repository.

Project Setup

Create a directory for the project then switch to it:

mkdir social-search
cd social-search
Enter fullscreen mode Exit fullscreen mode

Next, let's initialize it with NPM:

npm init
Enter fullscreen mode Exit fullscreen mode

You can enter any information that's relevant for you, or just leave the default values.

Once done, we'll install some initial packages we'll need:

npm i express axios dotenv
Enter fullscreen mode Exit fullscreen mode

Where express is for our server, axios is to send requests which we'll use with some of the social media platforms, and dotenv which we'll use to store necessary tokens in .env.

Then, create a file called index.js in the root with the following content:

require('dotenv').config()
const app = require('express')()
const axios = require('axios')

app.listen(3000, function () {
    console.log("started server for social search!")
})
Enter fullscreen mode Exit fullscreen mode

This will create a simple server that listens on port 3000. Last thing we'll need to do to start our server is add a start script in package.json:

"scripts": {
    "start": "node index.js"
},
Enter fullscreen mode Exit fullscreen mode

That's it! To test our server, let's run:

npm start
Enter fullscreen mode Exit fullscreen mode

If everything worked correctly, you'll see a message saying "started server for social search!"

Now, we're ready to start searching through social media.

Searching Twitter

To search Twitter, you first need to create a new app on twitter's developer portal. Depending on whether this is your first app or not, you might need to enter some extra information, but generally, you just need to enter the name of the app.

App Name

Then, you will be given the API key and the API secret. Make sure to copy them and save them somewhere.

Once the app is created, go to "Keys and Tokens" tab on the app's page. Under "Authentication Tokens", click on "Generate" for Access Token and Secret. You will be given two extra keys that you need to copy and save somewhere.

Back to our project, create .env file which will hold the keys:

TWITTER_API_KEY=
TWITTER_API_SECRET=
TWITTER_ACCESS_TOKEN=
TWITTER_ACCESS_TOKEN_SECRET=
Enter fullscreen mode Exit fullscreen mode

where the first 2 keys are the first 2 you received, and the second 2 are the ones you generated.

To access Twitter's API easily, we'll use the library twitter-api-client. It provides an easy to use interface to access Twitter's APIs. We need to install it first with NPM:

npm i twitter-api-client
Enter fullscreen mode Exit fullscreen mode

Now, go to index.js, require the library and initialize the Twitter client:

const TwitterClient = require('twitter-api-client').TwitterClient

const twitterClient = new TwitterClient({
    apiKey: process.env.TWITTER_API_KEY,
    apiSecret: process.env.TWITTER_API_SECRET,
    accessToken: process.env.TWITTER_ACCESS_TOKEN,
    accessTokenSecret: process.env.TWITTER_ACCESS_TOKEN_SECRET,
  });
Enter fullscreen mode Exit fullscreen mode

Since we've added at the beginning of the file require('dotenv').config(), we can now use the keys we added in .env through process.env.

Next, we'll add a new GET route that takes query parameter, which we'll use to search through Twitter:

app.get('/twitter/:query', function (req, res) {
});
Enter fullscreen mode Exit fullscreen mode

Inside the callback function for this route, we need to search through twitter for tweets including this given query. To search tweets, we'll use the twitterClient.tweets.search which takes as a parameter an object that includes various properties. The one we'll use is q which we'll pass the query:

const query = req.params.query

twitterClient.tweets.search({
    q: query
}).then((response) => {
    res.send(response)
}).catch((err) => {
        console.error(err)
        res.status(500).send('An error occurred, please try again later.')
})
Enter fullscreen mode Exit fullscreen mode

When we get a response, we're just sending it back. If an error occurs, we're printing it to the console and sending a status 500 response.

That's it! To test it out, start the server:

npm start
Enter fullscreen mode Exit fullscreen mode

Then go to the route we specified, passing it any query you want. For example, localhost:3000/twitter/cats. If everything is done correctly, you should see a JSON object that holds an array of tweets.

Twitter Response

You can look at the documentation for twitter-api-client to understand more the options you can pass, the response, etc...

Searching Instagram

Instagram APIs are a mess. Long story short, now there's Hashtag Search that can be accessed through Instagram Graph API. However, this only allows you to search through posts of a certain user, not search in general. Not to mention that this API needs you to access a bunch of endpoints in order to finally get the posts with the hashtag you're querying.

There's another unofficial way to search through Instagram. There's an endpoint you can find at https://www.instagram.com/explore/tags/KEYWORD/?__a=1, where when you replace KEYWORD with the query, you'll get a JSON response with all the posts using the hashtag.

We'll create a new endpoint that allows us to search through Instagram:

app.get('/instagram/:query', function (req, res) {

});
Enter fullscreen mode Exit fullscreen mode

Inside the callback, we'll use axios to send a GET request to the endpoint we mentioned above, then send back the response once received:

const query = req.params.query

axios.get('https://www.instagram.com/explore/tags/' + query + '/?__a=1')
        .then((response) => {
            res.send(response.data)
        })
        .catch((err) => {
            console.error(err)
            res.status(500).send('An error occurred, please try again later.')
        })
Enter fullscreen mode Exit fullscreen mode

That's it. To test it out, start the server:

npm start
Enter fullscreen mode Exit fullscreen mode

Then, go to the route we just created, passing it the query you want. For example, localhost:3000/instagram/cats. If everything was done correctly, the response will be a JSON object with the posts using the hashtag cats.

Searching Tumblr

To search through Tumblr, first register a new application. You'll need a tumblr account to do that. You'll have to enter information like Application name, Application URL (you can just enter http://example.com), etc...

Once done, you'll be redirected to the Applications page and it will show you the newly created app. You can see the OAuth Consumer Key. Copy it and add it to .env:

TUMBLR_CONSUMER_KEY=
Enter fullscreen mode Exit fullscreen mode

To search posts by tags, we can use the /tagged endpoint. We'll pass it the API key, which is the Consumer key we just added to .env, and the tag which is the query we're searching.

First, we'll create the new GET route:

app.get('/tumblr/:query', function (req, res) {
    const query = req.params.query
});
Enter fullscreen mode Exit fullscreen mode

As usual, we're creating a route that takes a query as a parameter.

Then, we'll send a GET request to Tumblr's /tagged endpoint using axios passing it the parameter as explained above:

axios.get('https://api.tumblr.com/v2/tagged?tag=' + query + '&api_key=' + process.env.TUMBLR_CONSUMER_KEY)
        .then((response) => {
            res.send(response.data)
        })
        .catch((err) => {
            console.error(err)
            res.status(500).send('An error occurred, please try again later.')
        })
Enter fullscreen mode Exit fullscreen mode

As before, we're just sending back the response once received.

To test it out, start the server:

npm start
Enter fullscreen mode Exit fullscreen mode

Then, go to the route we created passing it the query you want. For example, localhost:3000/tumblr/cats. If everything is working correctly, the response will be a JSON response with an array of posts using the tag cats.

Searching Pinterest

Pinterest doesn't have an API that allows you to search through their pins. To search through Pinterest, we'll have to use Google's Custom Search JSON API. This API allows you to retreive search results in JSON format. We'll use it to search through Pinterest. It's not optimal, but at the moment of writing this, it's the easiest way. You should note that this API provides 100 queries per day for free, then you'll be billed 5$ for every additional 1000 query.

First, you'll need to get an API key to use. Just click the "Get a Key" button. It will ask you to create a project or use an existing one, then you'll be provided with an API key. Copy the key and add it to .env:

CUSTOM_SEARCH_API_KEY=
Enter fullscreen mode Exit fullscreen mode

Next, you'll need to create a "Programmable Search". You can do that here. You'll need to enter the URL of the site you'll be searching. In our case it will be pinterest.com. You can also change the language and Name of the Programmable Search. Once done, click Create. It will take you to a page that will show you a bunch of options. Choose "Control Panel" button.

Control Panel

Once you're on the Control Panel, copy the "Search Engine ID" and add it to .env:

CUSTOM_SEARCH_CX=
Enter fullscreen mode Exit fullscreen mode

We'll need to pass this as a parameter when we send our requests to the API.

Let's start by creating the GET route to query Pinterest:

app.get('/pinterest/:query', function (req, res) {
    const query = req.params.query


});
Enter fullscreen mode Exit fullscreen mode

Next, we'll send a request to the Custom Search API endpoint passing it the API key, CX which is the Search Engine ID, and the query we're searching:

axios.get('https://www.googleapis.com/customsearch/v1?key=' + process.env.CUSTOM_SEARCH_API_KEY + 
        '&cx=' + process.env.CUSTOM_SEARCH_CX + '&q=' + query)
        .then((response) => {
            res.send(response.data.items)
        })
        .catch((err) => {
            console.error(err)
            res.status(500).send('An error occurred, please try again later.')
        })
Enter fullscreen mode Exit fullscreen mode

As usual, we're just sending back the response received.

To test it out, start the server:

npm start
Enter fullscreen mode Exit fullscreen mode

Then, go to the route we just created, passing it a query. For example, localhost:3000/pinterest/cats. You'll see a JSON array, each object having the title of the pin, the link to it, and other information.

Conclusion

In this tutorial, we went over searching Twitter, Instagram, Tumblr, and Pinterest. In the next one, we'll go through searching more social media platforms!

Image
2N5133
Image
The Compound Component Pattern in React
Image
2N4341
Image
YouTube Download Videos and Audios Save from in Media File Formats

youtube, downloader, converter, mp4
Image
Develop .NET MAUI Apps Using VS Code Without Traditional IDEs
Image
2N4248
Image
Exploring JavaScript: From Scripting to Object-Oriented Programming
Image
How Best Folding Wheelchair Was The Most Talked About Trend In 2023
Image
2N3565
Image
2N1597
Image
RC4136DC
Image
10 Tips For Lightweight Folding Wheelchair That Are Unexpected
Image
AM9513APC
Image
DS1213B
Image
Why P6 is typically the Premier Betting Link for Gamblers throughout Vietnam
Image
HPR1000
Image
Die Bedeutung der Seebestattung
Image
Think You're Cut Out For Bi Fold Door Repairs? Answer This Question
Image
LS352
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .