John Au-YeungSubscribe to my email list now at http://jauyeung.net/subscribe/
Follow me on Twitter at https://twitter.com/AuMayeung
Many more articles at https://medium.com/@hohanga
To prevent overloading your servers, adding a rate limit to your API is a good option to solve this problem. We can block excessive requests from being accepted by blocking it with route middleware to prevent route code from executing if too many requests are sent from one IP address.
Express apps can use the express-rate-limit package to limit the number of requests being accepted by the back end app. It is very simple to use. We just have to specify the rate limit per some amount of time for request to be accepted.
For example, to limit a request to accepting 5 requests per minute from one IP address, we put:
const rateLimit = require("express-rate-limit");
const limiter = rateLimit({
windowMs: 60 * 1000,
max: 5
});
app.use("/api/", limiter, (req, res) => {...});
The limiter is a middleware that is added before the route callback and it is executed before the route callback is called if the rate limit is not reach.
In this article, we will build a video converter to convert the source video into the format of the user’s choice. We will use the fluent-ffmpeg package for running the conversions. Because the jobs are long running, we will also create a job queue so that it will run in the background. The rate limit per minute can be set by us in an environment variable.
FFMPEG is a command line video and audio processing program that has a lot of capabilities. It also supports lots of formats for video conversion.
Developers have done the hard work for us by creating a Node.js wrapper for FFMPEG. The package is called fluent-ffmpeg. gIt is located at https://github.com/fluent-ffmpeg/node-fluent-ffmpeg. This package allows us to run FFMPEG commands by calling the built in functions.
We will use Express for the back end and React for the front end.
Back End
To get started, create a project folder and a backend folder inside it. In the backend folder, run npx express-generator to generate the files for the Express framework.
Next run npm i in the backend folder to download the packages in package.json.
Then we have to install our own packages. We need Babel to use import in our app. Also, we will use the Bull package for background jobs, CORS package for cross domain requests with front end, fluent-ffmpeg for converting videos, Multer for file upload, Dotenv for managing environment variables, express-rate-limit for limiting requests to our app, Sequelize for ORM and SQLite3 for or database.
Run npm i @babel/cli @babel/core @babel/node @babel/preset-env bull cors dotenv fluent-ffmpeg multer sequelize sqlite3 express-rate-limit to install all the packages.
Next add .babelrc file to the backend folder and add:
{
"presets": [
"@babel/preset-env"
]
}
to enable the latest JavaScript features, and in the scripts section of package.json , replace the existing code with:
"start": "nodemon --exec npm run babel-node -- ./bin/www",
"babel-node": "babel-node"
to run with Babel instead of the regular Node runtime.
Next we create the Sequelize code by running npx sequelize-cli init.
Then in config.json that’s just created by running the command above, we replace the existing code with:
{
"development": {
"dialect": "sqlite",
"storage": "development.db"
},
"test": {
"dialect": "sqlite",
"storage": "test.db"
},
"production": {
"dialect": "sqlite",
"storage": "production.db"
}
}
Next we need to create our model and migration. We run:
npx sequelize-cli --name VideoConversion --attributes filePath:string,convertedFilePath:string,outputFormat:string,status:enum
To create the model and migration for the VideoConversions table.
In the newly created migration file, replace the existing code with:
"use strict";
module.exports = {
up: (queryInterface, Sequelize) => {
return queryInterface.createTable("VideoConversions", {
id: {
allowNull: false,
autoIncrement: true,
primaryKey: true,
type: Sequelize.INTEGER
},
filePath: {
type: Sequelize.STRING
},
convertedFilePath: {
type: Sequelize.STRING
},
outputFormat: {
type: Sequelize.STRING
},
status: {
type: Sequelize.ENUM,
values: ["pending", "done", "cancelled"],
defaultValue: "pending"
},
createdAt: {
allowNull: false,
type: Sequelize.DATE
},
updatedAt: {
allowNull: false,
type: Sequelize.DATE
}
});
},
down: (queryInterface, Sequelize) => {
return queryInterface.dropTable("VideoConversions");
}
};
to add the constants for our enum.
Then in models/videoconversion.js , replace the existing code with:
"use strict";
module.exports = (sequelize, DataTypes) => {
const VideoConversion = sequelize.define(
"VideoConversion",
{
filePath: DataTypes.STRING,
convertedFilePath: DataTypes.STRING,
outputFormat: DataTypes.STRING,
status: {
type: DataTypes.ENUM("pending", "done", "cancelled"),
defaultValue: "pending"
}
},
{}
);
VideoConversion.associate = function(models) {
// associations can be defined here
};
return VideoConversion;
};
to add the enum constants to the model.
Next run npx sequelize-init db:migrate to create our database.
Then create the files folder in the back end folder for storing the files.
Next we create our video processing job queue. Create a queues folder and inside it, create videoQueue.js file and add:
const Queue = require("bull");
const videoQueue = new Queue("video transcoding");
const models = require("../models");
var ffmpeg = require("fluent-ffmpeg");
const fs = require("fs");
const convertVideo = (path, format) => {
const fileName = path.replace(/\.[^/.]+$/, "");
const convertedFilePath = `${fileName}_${+new Date()}.${format}`;
return new Promise((resolve, reject) => {
ffmpeg(`${__dirname}/../files/${path}`)
.setFfmpegPath(process.env.FFMPEG_PATH)
.setFfprobePath(process.env.FFPROBE_PATH)
.toFormat(format)
.on("start", commandLine => {
console.log(`Spawned Ffmpeg with command: ${commandLine}`);
})
.on("error", (err, stdout, stderr) => {
console.log(err, stdout, stderr);
reject(err);
})
.on("end", (stdout, stderr) => {
console.log(stdout, stderr);
resolve({ convertedFilePath });
})
.saveToFile(`${__dirname}/../files/${convertedFilePath}`);
});
};
videoQueue.process(async job => {
const { id, path, outputFormat } = job.data;
try {
const conversions = await models.VideoConversion.findAll({ where: { id } });
const conv = conversions[0];
if (conv.status == "cancelled") {
return Promise.resolve();
}
const pathObj = await convertVideo(path, outputFormat);
const convertedFilePath = pathObj.convertedFilePath;
const conversion = await models.VideoConversion.update(
{ convertedFilePath, status: "done" },
{
where: { id }
}
);
Promise.resolve(conversion);
} catch (error) {
Promise.reject(error);
}
});
export { videoQueue };
In the convertVideo function, we use fluent-ffmpeg to get the video file, then set the FFMPEG and FFProbe paths from the environment variables. Then we call toFormat to convert it to the format we specify. We log in the start, error, and end handlers to see the outputs and resolve our promise on the end event. When conversion is done, we save it to a new file.
videoQueue is a Bull queue that processes jobs in the background sequentially. Redis is required to run the queue, we will need a Ubuntu Linux installation. We run the follwing commands in Ubuntu to install and run Redis:
$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install redis-server
$ redis-server
In the callback of the videoQueue.process function, we call the convertVideo function and update the path of the converted file and the status of the given job when the job is done.
Next we create our routes. Create a conversions.js file in the routes folder and add:
var express = require("express");
var router = express.Router();
const models = require("../models");
var multer = require("multer");
const fs = require("fs").promises;
const path = require("path");
import { videoQueue } from "../queues/videoQueue";
const rateLimit = require("express-rate-limit");
const limiter = rateLimit({
windowMs: 60000,
max: process.env.CALL_PER_MINUTE || 10,
message: {
error: "Too many requests"
}
});
const storage = multer.diskStorage({
destination: (req, file, cb) => {
cb(null, "./files");
},
filename: (req, file, cb) => {
cb(null, `${+new Date()}_${file.originalname}`);
}
});
const upload = multer({ storage });
router.get("/", async (req, res, next) => {
const conversions = await models.VideoConversion.findAll();
res.json(conversions);
});
router.post("/", limiter, upload.single("video"), async (req, res, next) => {
const data = { ...req.body, filePath: req.file.path };
const conversion = await models.VideoConversion.create(data);
res.json(conversion);
});
router.delete("/:id", async (req, res, next) => {
const id = req.params.id;
const conversions = await models.VideoConversion.findAll({ where: { id } });
const conversion = conversions[0];
try {
await fs.unlink(`${__dirname}/../${conversion.filePath}`);
if (conversion.convertedFilePath) {
await fs.unlink(`${__dirname}/../files/${conversion.convertedFilePath}`);
}
} catch (error) {
} finally {
await models.VideoConversion.destroy({ where: { id } });
res.json({});
}
});
router.put("/cancel/:id", async (req, res, next) => {
const id = req.params.id;
const conversion = await models.VideoConversion.update(
{ status: "cancelled" },
{
where: { id }
}
);
res.json(conversion);
});
router.get("/start/:id", limiter, async (req, res, next) => {
const id = req.params.id;
const conversions = await models.VideoConversion.findAll({ where: { id } });
const conversion = conversions[0];
const outputFormat = conversion.outputFormat;
const filePath = path.basename(conversion.filePath);
await videoQueue.add({ id, path: filePath, outputFormat });
res.json({});
});
module.exports = router;
In the POST / route, we accept the file upload with the Multer package. We add the job and save the file to the files folder that we created before. We save it with the file’s original name in the filename function in the object we passed into the diskStorage function and specified the file be saved in the files folder in the destination function.
The GET route / route gets the jobs added. DELETE / deletes the job with the given ID along with the source file of the job. PUT /cancel/:id route sets status to cancelled .
And the GET /start/:id route add the job with the given ID to the queue we created earlier.
We added limiter object here to use express-rate-limit to limit the number of API calls to POST / route and the GET /start/:id route to prevent too many jobs from be added and started respectively.
In app.js , we replace the existing code with:
require("dotenv").config();
var createError = require("http-errors");
var express = require("express");
var path = require("path");
var cookieParser = require("cookie-parser");
var logger = require("morgan");
var cors = require("cors");
var indexRouter = require("./routes/index");
var conversionsRouter = require("./routes/conversions");
var app = express();
// view engine setup
app.set("views", path.join(__dirname, "views"));
app.set("view engine", "jade");
app.use(logger("dev"));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, "public")));
app.use(express.static(path.join(__dirname, "files")));
app.use(cors());
app.use("/", indexRouter);
app.use("/conversions", conversionsRouter);
// catch 404 and forward to error handler
app.use(function(req, res, next) {
next(createError(404));
});
// error handler
app.use(function(err, req, res, next) {
// set locals, only providing error in development
res.locals.message = err.message;
res.locals.error = req.app.get("env") === "development" ? err : {};
// render the error page
res.status(err.status || 500);
res.render("error");
});
module.exports = app;
to add the CORS add-on to enable cross domain communication, expose the files folder to the public, and expose the conversions routes we created earlier to the public.
To add the environment variables, create an .env file in the backend folder and add:
FFMPEG\_PATH='c:\\ffmpeg\\bin\\ffmpeg.exe'
FFPROBE\_PATH='c:\\ffmpeg\\bin\\ffprobe.exe'
CALL\_PER\_MINUTE=5
Change the paths to the FFMPEG and FFProbe paths in your computer and configure CALL_PER_MINUTE to whatever you like as long as it is more than zero.
Front End
With back end done, we can move on to the front end. In the project’s root folder, run npx create-react-app frontend to create the front end files.
Next we install some packages. We need Axios for making HTTP requests, Formik for form value handling, MobX for state management, React Router for routing URLs to our pages, and Bootstrap for styling.
Run npm i axios bootstrap formik mobx mobx-react react-bootstrap react-router-dom to install the packages.
Next we replace the existing code in App.js with:
import React from "react";
import { Router, Route } from "react-router-dom";
import "./App.css";
import { createBrowserHistory as createHistory } from "history";
import HomePage from "./HomePage";
import { ConversionsStore } from "./store";
import TopBar from "./TopBar";
const conversionsStore = new ConversionsStore();
const history = createHistory();
function App() {
return (
<div className="App">
<TopBar />
<Router history={history}>
<Route
path="/"
exact
component={props => (
<HomePage {...props} conversionsStore={conversionsStore} />
)}
/>
</Router>
</div>
);
}
export default App;
We add the top bar and the routes in this file.
In App.css , we replace the existing code with:
.page {
padding: 20px;
}
.button {
margin-right: 10px;
}
to adding padding and margins to our page and buttons.
Next create HomePage.js in the src folder and add:
import React from "react";
import Table from "react-bootstrap/Table";
import Button from "react-bootstrap/Button";
import ButtonToolbar from "react-bootstrap/ButtonToolbar";
import { observer } from "mobx-react";
import {
getJobs,
addJob,
deleteJob,
cancel,
startJob,
APIURL
} from "./request";
import { Formik } from "formik";
import Form from "react-bootstrap/Form";
import Col from "react-bootstrap/Col";
function HomePage({ conversionsStore }) {
const fileRef = React.createRef();
const [file, setFile] = React.useState(null);
const [fileName, setFileName] = React.useState("");
const [initialized, setInitialized] = React.useState(false);
const onChange = event => {
setFile(event.target.files[0]);
setFileName(event.target.files[0].name);
};
const openFileDialog = () => {
fileRef.current.click();
};
const handleSubmit = async evt => {
if (!file) {
return;
}
let bodyFormData = new FormData();
bodyFormData.set("outputFormat", evt.outputFormat);
bodyFormData.append("video", file);
try {
await addJob(bodyFormData);
} catch (error) {
alert(error.response.statusText);
} finally {
getConversionJobs();
}
};
const getConversionJobs = async () => {
const response = await getJobs();
conversionsStore.setConversions(response.data);
};
const deleteConversionJob = async id => {
await deleteJob(id);
getConversionJobs();
};
const cancelConversionJob = async id => {
await cancel(id);
getConversionJobs();
};
const startConversionJob = async id => {
await startJob(id);
getConversionJobs();
};
React.useEffect(() => {
if (!initialized) {
getConversionJobs();
setInitialized(true);
}
});
return (
<div className="page">
<h1 className="text-center">Convert Video</h1>
<Formik onSubmit={handleSubmit} initialValues={{ outputFormat: "mp4" }}>
{({
handleSubmit,
handleChange,
handleBlur,
values,
touched,
isInvalid,
errors
}) => (
<Form noValidate onSubmit={handleSubmit}>
<Form.Row>
<Form.Group
as={Col}
md="12"
controlId="outputFormat"
defaultValue="mp4"
>
<Form.Label>Output Format</Form.Label>
<Form.Control
as="select"
value={values.outputFormat || "mp4"}
onChange={handleChange}
isInvalid={touched.outputFormat && errors.outputFormat}
>
<option value="mov">mov</option>
<option value="webm">webm</option>
<option value="mp4">mp4</option>
<option value="mpeg">mpeg</option>
<option value="3gp">3gp</option>
</Form.Control>
<Form.Control.Feedback type="invalid">
{errors.outputFormat}
</Form.Control.Feedback>
</Form.Group>
</Form.Row>
<Form.Row>
<Form.Group as={Col} md="12" controlId="video">
<input
type="file"
style={{ display: "none" }}
ref={fileRef}
onChange={onChange}
name="video"
/>
<ButtonToolbar>
<Button
className="button"
onClick={openFileDialog}
type="button"
>
Upload
</Button>
<span>{fileName}</span>
</ButtonToolbar>
</Form.Group>
</Form.Row>
<Button type="submit">Add Job</Button>
</Form>
)}
</Formik>
<br />
<Table>
<thead>
<tr>
<th>File Name</th>
<th>Converted File</th>
<th>Output Format</th>
<th>Status</th>
<th>Start</th>
<th>Cancel</th>
<th>Delete</th>
</tr>
</thead>
<tbody>
{conversionsStore.conversions.map((c, i) => {
return (
<tr key={i}>
<td>{c.filePath}</td>
<td>{c.status}</td>
<td>{c.outputFormat}</td>
<td>
{c.convertedFilePath ? (
<a href={`${APIURL}/${c.convertedFilePath}`}>Open</a>
) : (
"Not Available"
)}
</td>
<td>
<Button
className="button"
type="button"
onClick={startConversionJob.bind(this, c.id)}
>
Start
</Button>
</td>
<td>
<Button
className="button"
type="button"
onClick={cancelConversionJob.bind(this, c.id)}
>
Cancel
</Button>
</td>
<td>
<Button
className="button"
type="button"
onClick={deleteConversionJob.bind(this, c.id)}
>
Delete
</Button>
</td>
</tr>
);
})}
</tbody>
</Table>
</div>
);
}
export default observer(HomePage)
This is the home page of our app. We have a drop down for selecting the format of the file, a upload button for select the file for conversion, and a table for displaying the video conversion jobs with the status and file names of the source and converted files.
We also have buttons to start, cancel and delete each job.
To add file upload, we have a hidden file input and in the onChange handler of the file input, we set the file. The Upload button’s onClick handler will click the file input to open the upload file dialog.
We get latest jobs by calling getConversionJobs we we first load the page, and when we start, cancel and delete jobs. The job data are stored in the MobX store that we will create later. We wrap observer in our HomePage in the last line to always get the latest values from the store.
Next create request.js and the src folder and add:
const axios = require("axios");
export const APIURL = "http://localhost:3000";
export const getJobs = () => axios.get(`${APIURL}/conversions`);
export const addJob = data =>
axios({
method: "post",
url: `${APIURL}/conversions`,
data,
config: { headers: { "Content-Type": "multipart/form-data" } }
});
export const cancel = id => axios.put(`${APIURL}/conversions/cancel/${id}`, {});
export const deleteJob = id =>
axios.delete(`${APIURL}/conversions/${id}`);
export const startJob = id => axios.get(`${APIURL}/conversions/start/${id}`);
The HTTP requests that we make to the back end are all here. They were used on the HomePage.
Next create the MobX store by creating store.js file in the src folder. In there add:
import { observable, action, decorate } from "mobx";
class ConversionsStore {
conversions = [];
setConversions(conversions) {
this.conversions = conversions;
}
}
ConversionsStore = decorate(ConversionsStore, {
conversions: observable,
setConversions: action
});
export { ConversionsStore };
This is a simple store which stores the contacts the conversions array is where we store the contacts for the whole app. The setConversions function let us set contacts from any component where we pass in the this store object to.
Next create TopBar.js in the src folder and add:
import React from "react";
import Navbar from "react-bootstrap/Navbar";
import Nav from "react-bootstrap/Nav";
function TopBar() {
return (
<Navbar bg="primary" expand="lg" variant="dark">
<Navbar.Brand href="#home">Video Converter</Navbar.Brand>
<Navbar.Toggle aria-controls="basic-navbar-nav" />
<Navbar.Collapse id="basic-navbar-nav">
<Nav className="mr-auto">
<Nav.Link href="/">Home</Nav.Link>
</Nav>
</Navbar.Collapse>
</Navbar>
);
}
export default TopBar;
This contains the React Bootstrap Navbar to show a top bar with a link to the home page and the name of the app.
In index.html , we replace the existing code with:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<link rel="shortcut icon" href="%PUBLIC_URL%/favicon.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="theme-color" content="#000000" />
<meta
name="description"
content="Web site created using create-react-app"
/>
<link rel="apple-touch-icon" href="logo192.png" />
<!--
manifest.json provides metadata used when your web app is installed on a
user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
-->
<link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
<!--
Notice the use of %PUBLIC_URL% in the tags above.
It will be replaced with the URL of the `public` folder during the build.
Only files inside the `public` folder can be referenced from the HTML.
Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
work correctly both with client-side routing and a non-root public URL.
Learn how to configure a non-root public URL by running `npm run build`.
-->
<title>Video Converter</title>
<link
rel="stylesheet"
href="https://maxcdn.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css"
integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T"
crossorigin="anonymous"
/>
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root"></div>
<!--
This HTML file is a template.
If you open it directly in the browser, you will see an empty page.
You can add webfonts, meta tags, or analytics to this file.
The build step will place the bundled scripts into the <body> tag.
To begin the development, run `npm start` or `yarn start`.
To create a production bundle, use `npm run build` or `yarn build`.
-->
</body>
</html>
to add the Bootstrap CSS and change the title.
After writing all that code, we can run our app. Before running anything, install nodemon by running npm i -g nodemon so that we don’t have to restart back end ourselves when files change.
Then run back end by running npm start in the backend folder and npm start in the frontend folder, then choose ‘yes’ if you’re asked to run it from a different port.
