Go Scrapper API

Vignesh Muthukumaran - Jul 3 '22 - - Dev Community

As I was exploring further about Go, I came across APIs in Go, which was very easy to build. So instead of going the traditional way and building a CRUD API, I decided to write a simple scrapper API. The API scraps Goodreads quotes page and returns a JSON of quotes with their authors.

Here I am using 2 modules,

  • Gin - For quickly building the API
  • Colly - For web scrapping.

I have built it and deployed it on Heroku.

Commands I ran from start to finish

To initiate the current folder/project as a module

go mod init vigneshm.me/go-quotes-api
Enter fullscreen mode Exit fullscreen mode

This creates a file called go.mod. This has the name of the project and the version of Go it uses.

To import modules

go get github.com/gin-gonic/gin
go get github.com/gocolly/colly
Enter fullscreen mode Exit fullscreen mode

This imports the modules and creates a go.sum file to add checksums for the modules it downloads. Also, the go.mod page gets filled with all the dependent modules info.

Next, I filled in the main.go with my code.

Now, for the part where I deploy the app in Heroku.

Install Heroku CLI

npm install -g heroku
Enter fullscreen mode Exit fullscreen mode

Login to Heroku

heroku login -i
Enter fullscreen mode Exit fullscreen mode

Initate a Git repo and commit the code

git init
git add .
git commit -m "initial commit"
Enter fullscreen mode Exit fullscreen mode

Create a Procfile for Heroku

touch Procfile
echo "web: bin/go-quotes-api" > Procfile
Enter fullscreen mode Exit fullscreen mode

This tells Heroku to run the command on startup.

Create Heroku App

heroku create go-quote-api
git push heroku master
Enter fullscreen mode Exit fullscreen mode

This creates the Heroku app and pushes the code.

Heroku takes this code runs the build and exposes the app at https://app-name.herokuapp.com/

My app is available at https://go-quote-api.herokuapp.com/quotes/dumbledore

The last word can be replaced with any search criteria to search for quotes from your favorite character or book. Code can be found at https://github.com/vigneshm243/go-quotes-api.

Happy Coding!


Thanks for reading. Leave a ❤️ if you liked it and comment and let me know what you think.
Originally published at vigneshm.me

. . . . . . . . . . . . . . . . . . .