Hugo Di FrancescoThere are occasions when running a Python/Ruby/PHP shell script from Node.js is necessary. This post looks at best practices around leveraging child_process.spawn to encapsulate this call in Node.js/JavaScript.
The goal here is to have an interoperability layer between Node.js and an outside shell. This is a quick workaround if some other part of your system isn’t developed in JavaScript.
We’ll use spawn over exec because we’re talking about passing data, and potentially large amounts of it. To understand the difference between child_process.spawn and child_process.exec (see “Difference between spawn and exec of Node.js child_process”).
The long and short of it is use exec for small amounts of data (under 200k) using a Buffer interface and spawn for larger amounts using a stream interface.
spawn has a more verbose syntax for some of the use-cases we’ll look at, but it’s more serviceable for integrating with Ruby/Python/PHP since we might get more data than a couple of lines of text.
Full examples github.com/HugoDF/node-run-python.
Table of contents:
- Call a shell command and log it
- Call Python for its version
- Call a Python script from Node
- Pass arguments to a Python script from Node.js using child_process.spawn
- Read child_process.spawn output from Node.js
- Handle errors from child_process.spawn
- Pass structured data from Python/Ruby to Node.js/JavaScript
The following examples contain 2 sections.
The part that actually runs the shell command, usually a function called run.They also contain an IIFE (“immediately invoked function expression”) that actually calls it ie. (async () => { await run() })(). This IIFE is a nice pattern enabled by async/await (see Async JS: history, patterns and gotchas) but it’s just there for illustration purposes since it represents the call to the wrapped spawn call from another part of your application.
Call a shell command and log it
Using spawn is overkill in this situation since echo is only going to return what’s passed to it.
The example is pretty self-explanatory and shows how to use child_process.spawn to “shell out” and read that data back.
spawn takes the executable to call as the first parameter and optionally an array of options/parameters for the executable as the second parameter.
const { spawn } = require('child_process');
function run() {
const process = spawn('echo', ['foo']);
process.stdout.on(
'data',
(data) => console.log(data.toString())
);
}
(() => {
try {
run()
// process.exit(0)
} catch (e) {
console.error(e.stack);
process.exit(1);
}
})();
$ node run.js
foo
Call Python for its version
We’ll move quite quickly to showcase how we would do something similar to the above with python. Note again how --version is passed inside of an array.
We also create a nice logger to differentiate between stdout and stderr and bind to them. Since spawn returns an instance which has stdout and stderr event emitters, we can bind our logOutput function to 'data' event using .on('data', () => { /* our callback function */ }).
Another interesting tidbit is that python --version outputs the version to stderr. The inconsistencies around whether *NIX executables use exit codes, stderr and stdout on success/error are a quirk that we’ll have to bear in mind while integrating Python/Ruby/other with Node.js.
const { spawn } = require('child_process')
const logOutput = (name) => (data) => console.log(`[${name}] ${data.toString()}`)
function run() {
const process = spawn('python', ['--version']);
process.stdout.on(
'data',
logOutput('stdout')
);
process.stderr.on(
'data',
logOutput('stderr')
);
}
(() => {
try {
run()
// process.exit(0)
} catch (e) {
console.error(e.stack);
process.exit(1);
}
})();
Output:
sh
$ node run.js
[stderr] Python 2.7.13
## Call a Python script from Node
We’ll now run a fully-fledged Python script (although it could just as well be Ruby, PHP, shell etc.) from Node.js.
This is `script.py`, it just logs out `argv` (the “argument vector”, ie. `['path/to/executable', /* command line arguments]`)
py
import sys
print(sys.argv)
Like in the previous example, we’ll just call spawn with `python` with the path to the Python script (`./script.py`) in the second parameter.
Here comes another gotcha of integrating scripts in this fashion. In this example, the path to the script is based on the working directory from which `node` is called.
There are workaround of course using the `path` module and `__dirname`, which for example could resolve a `other-script.py` co-located with the JavaScript file/Node module calling `spawn` using: `require('path').resolve(__dirname, './other-script.py')`.
js
const { spawn } = require('child_process')
const logOutput = (name) => (data) => console.log([${name}] ${data.toString()})
function run() {
const process = spawn('python', ['./script.py']);
process.stdout.on(
'data',
logOutput('stdout')
);
process.stderr.on(
'data',
logOutput('stderr')
);
}
(() => {
try {
run()
// process.exit(0)
} catch (e) {
console.error(e.stack);
process.exit(1);
}
})();
Output:
sh
$ node run.js
[stdout] ['./script.py']
## Pass arguments to a Python script from Node.js using child\_process.spawn
The next step of integration is to be able to pass data from the Node/JavaScript code to the Pytonh script.
In order to do this, we’ll just passed more shell arguments using the arguments array (second parameter to `spawn`).
js
const { spawn } = require('child_process')
const logOutput = (name) => (data) => console.log([${name}] ${data.toString()})
function run() {
const process = spawn('python', ['./script.py', 'my', 'args']);
process.stdout.on(
'data',
logOutput('stdout')
);
process.stderr.on(
'data',
logOutput('stderr')
);
}
(() => {
try {
run()
// process.exit(0)
} catch (e) {
console.error(e.stack);
process.exit(1);
}
})();
Our `script.py` will also just log out the `argv` except the first element (which is the path to the script).
py
import sys
print(sys.argv)[1:]
Here’s the output:
sh
$ node run.js
[stdout] ['my', 'args']
## Read child\_process.spawn output from Node.js
It’s nice to be able to pass data down to the Python script, but we’re still not able to get the data from the Python script back in a format that we’re able to leverage in our Node.js/JavaScript application.
The solution to this is to wrap the whole `spawn` -calling function into a Promise. This allows us to decide when we want to `resolve` or `reject`.
To keep track of the Python script’s output stream(s), we manually buffer the output using arrays (one for `stdout` and another for `stderr`).
We also add a listener for `'exit'` using `spawn().on('exit', (code, signal) => { /* probably call resolve() */ })`. This is where we will tend to `resolve`/`reject` the Promise’s value(s) from the Python/Ruby/other script.
js
const { spawn } = require('child_process')
const logOutput = (name) => (data) => console.log([${name}] ${data})
function run() {
return new Promise((resolve, reject) => {
const process = spawn('python', ['./script.py', 'my', 'args']);
const out = []
process.stdout.on(
'data',
(data) => {
out.push(data.toString());
logOutput('stdout')(data);
}
);
const err = []
process.stderr.on(
'data',
(data) => {
err.push(data.toString());
logOutput('stderr')(data);
}
);
process.on('exit', (code, signal) => {
logOutput('exit')(`${code} (${signal})`)
resolve(out);
});
});
}
(async () => {
try {
const output = await run()
logOutput('main')(output)
process.exit(0)
} catch (e) {
console.error(e.stack);
process.exit(1);
}
})();
Output:
$ node run.js
[stdout] ['my', 'args']
[main] ['my', 'args']
## Handle errors from child\_process.spawn
Next up we need to handle errors from the Python/Ruby/shell script at the Node.js/JavaScript level.
The main way that a \*NIX executable signals that it errored is by using a `1` exit code. That’s why the `.on('exit'` handler now does a check against `code === 0` before deciding whether to resolve or reject with value(s).
js
const { spawn } = require('child_process')
const logOutput = (name) => (data) => console.log([${name}] ${data})
function run() {
return new Promise((resolve, reject) => {
const process = spawn('python', ['./script.py', 'my', 'args']);
const out = []
process.stdout.on(
'data',
(data) => {
out.push(data.toString());
logOutput('stdout')(data);
}
);
const err = []
process.stderr.on(
'data',
(data) => {
err.push(data.toString());
logOutput('stderr')(data);
}
);
process.on('exit', (code, signal) => {
logOutput('exit')(`${code} (${signal})`)
if (code === 0) {
resolve(out);
} else {
reject(new Error(err.join('\n')))
}
});
});
}
(async () => {
try {
const output = await run()
logOutput('main')(output)
process.exit(0)
} catch (e) {
console.error('Error during script execution ', e.stack);
process.exit(1);
}
})();
Output:
sh
$ node run.js
[stderr] Traceback (most recent call last):
File "./script.py", line 3, in
print(sy.argv)[1:]
NameError: name 'sy' is not defined
Error during script execution Error: Traceback (most recent call last):
File "./script.py", line 3, in
print(sy.argv)[1:]
NameError: name 'sy' is not defined
at ChildProcess.process.on (/app/run.js:33:16)
at ChildProcess.emit (events.js:182:13)
at Process.ChildProcess._handle.onexit (internal/child_process.js:240:12)
## Pass structured data from Python/Ruby to Node.js/JavaScript
The final step to full integration between Ruby/Python/PHP/shell scripts and our Node.js/JavaScript application layer is to be able to pass structured data back from the script up to Node.js/JavaScript.
The simplest structured data format that tends to be available in both Python/Ruby/PHP and Node.js/JavaScript is JSON.
In the Python script, we print the `json.dumps()` output of a dictionary, see `script.py`:
py
import sys
import json
send_message_back = {
'arguments': sys.argv[1:],
'message': """Hello,
This is my message.
To the world"""
}
print(json.dumps(send_message_back))
In Node, we add some JSON-parsing logic (using `JSON.parse`) in the `'exit'` handler.
A gotcha at this point is if, for example `JSON.parse()` fails due to badly-formed JSON, we need to propagate that error up, hence the try/catch where the `catch` clause `reject`-s the potential error: `try { resolve(JSON.parse(out[0])) } catch(e) { reject(e) }`.
js
const { spawn } = require('child_process')
const logOutput = (name) => (message) => console.log([${name}] ${message})
function run() {
return new Promise((resolve, reject) => {
const process = spawn('python', ['./script.py', 'my', 'args']);
const out = []
process.stdout.on(
'data',
(data) => {
out.push(data.toString());
logOutput('stdout')(data);
}
);
const err = []
process.stderr.on(
'data',
(data) => {
err.push(data.toString());
logOutput('stderr')(data);
}
);
process.on('exit', (code, signal) => {
logOutput('exit')(`${code} (${signal})`)
if (code !== 0) {
reject(new Error(err.join('\n')))
return
}
try {
resolve(JSON.parse(out[0]));
} catch(e) {
reject(e);
}
});
});
}
(async () => {
try {
const output = await run()
logOutput('main')(output.message)
process.exit(0)
} catch (e) {
console.error('Error during script execution ', e.stack);
process.exit(1);
}
})();
Output
$ node run.js
[stdout] {"message": "Hello,\nThis is my message.\n\nTo the world", "arguments": ["my", "args"]}
[main] Hello,
This is my message.
To the world
I’ve got mentoring spots open at [https://mentorcruise.com/mentor/HugoDiFrancesco/](https://mentorcruise.com/mentor/HugoDiFrancesco/), so do that if you want Node.js/JavaScript/career mentoring or feel free to tweet at me [@hugo\_\_df](https://twitter.com/hugo__df)
[<svg xmlns="http://www.w3.org/2000/svg" style="height:12px;width:auto;position:relative;vertical-align:middle;top:-2px;fill:white" viewbox="0 0 32 32"><title>unsplash-logo</title>
<path d="M10 9V0h12v9H10zm12 5h10v18H0V14h10v9h12v-9z"></path></svg>Elaine Casap](https://unsplash.com/@ecasap?utm_medium=referral&utm_campaign=photographer-credit&utm_content=creditBadge "Download free do whatever you want high-resolution photos from Elaine Casap")
