An HTTP request parser with rust and pest.rs

protium - May 23 '22 - - Dev Community

Introduction

I happened to have learned about parsing expression grammars a few days ago and got really excited about writing my own grammar.
As I was missing this VS Code extension when working on neovim, an idea popped up:
What if I write an HTTP grammar and an execute request from an interactive prompt?

Preview:

preview

Preparation

For our grammar, we need to make sure we understand the HTTP standard defined here.



Request = Request-Line ; Section 5.1
        *(( general-header        ; Section 4.5
         | request-header         ; Section 5.3
         | entity-header ) CRLF)  ; Section 7.1
        CRLF
        [ message-body ]          ; Section 4.3
Enter fullscreen mode Exit fullscreen mode

Let's breakout the definition

Request

A Request is conformed by a Request-Line optionally followed by headers and a new line (a Carriage Return and Line Feed) then another new line. After the headers we find the message-body.

Request-Line 



Request-Line = Method SP Request-URI SP HTTP-Version CRLF
Enter fullscreen mode Exit fullscreen mode

Here we should fine a Method followed by a white space. Then the Request-URI followed by a white space. Finally the HTTP-Version.

An example of a request can be GET http://www.w3.org/pub/WWW/TheProject.html HTTP/1.\r\n (note the end of the line)

Request-Header

Defined as



equest-header = Accept                   ; Section 14.1
      | Accept-Charset           ; Section 14.2
      | Accept-Encoding          ; Section 14.3
      | Accept-Language          ; Section 14.4
      | Authorization            ; Section 14.8
      | Expect                   ; Section 14.20
      | From                     ; Section 14.22
      | Host                     ; Section 14.23
      | If-Match                 ; Section 14.24
      | If-Modified-Since        ; Section 14.25
      | If-None-Match            ; Section 14.26
      | If-Range                 ; Section 14.27
      | If-Unmodified-Since      ; Section 14.28
      | Max-Forwards             ; Section 14.31
      | Proxy-Authorization      ; Section 14.34
      | Range                    ; Section 14.35
      | Referer                  ; Section 14.36
      | TE                       ; Section 14.39
      | User-Agent               ; Section 14.43
Enter fullscreen mode Exit fullscreen mode

let's open one of those sections to check the format.



Authorization = "Authorization" ":" credentials
Enter fullscreen mode Exit fullscreen mode

Basically this header is the word Authorization followed by : and then the credentials.

Message-Body 



message-body = entity-body
                    | <entity-body encoded as per Transfer-Encoding>
Enter fullscreen mode Exit fullscreen mode

For simplicity let's assume that the body can be anything that is not a request line nor a header.

Now we have enough information to write our http grammar, more specific a set of rules.

Pest

To write our grammar we will use pest. From the website:

pest is a general purpose parser written in Rust with a focus on accessibility, correctness, and performance. It uses parsing expression grammars (or PEG) as input, which are similar in spirit to regular expressions, but which offer the enhanced expressivity needed to parse complex languages.

Sounds great!

Pest has its own syntax for writing grammar rules. A rule is defined as follows:



my_rule = { ... }
Enter fullscreen mode Exit fullscreen mode

Let's create a new rule my_rule= { "test" }. This rule will match the word test inside an input. So if we want to parse a file that only contains "test" or new lines we need to define the shape of a line with the delimiters SOI and EOI (start and end of input respectively)



testy_file = {
    SOI
    ~ (my_rule | NEWLINE)*
    ~ EOI
}
Enter fullscreen mode Exit fullscreen mode

Let's test this out in the online grammar editor:

editor

The parser has identified a testy_file that contains to matches for the rule my_rule. So far so good.

HTTP Grammar

Now we are ready for writing our grammar (are we?). We need to write this grammar as close as possible to the standard, since it's defined in a similar way, with rules. We can imagine something like this



request = { 
    request_line ~
    headers? ~
    NEWLINE ~
    body?
}
Enter fullscreen mode Exit fullscreen mode

Our request rule says: a request_line optionally followed by headers followed by a new line, then optionally followed by the body. With optionally I make reference to the ? mark which is a repetition operator (those who are familiar with regex expressions should understand this right away) and means that something can occur zero or one times.
For our request_line we will match the standard as follows:



request_line = _{ method ~ " "+ ~ uri ~ " "+ ~ "HTTP/" ~ version ~ NEWLINE }
Enter fullscreen mode Exit fullscreen mode

We are talking about a method followed by a blank space then the uri then another blank space then the http version followed by a NEWLINE. NEWLINE is a pest built-in rule that matches "\n" | "\r\n" | "\r".

Let's break the request line.

For method we just need to match any of the http methods defined in the standard



Method = "OPTIONS"                ; Section 9.2
        | "GET"                    ; Section 9.3
        | "HEAD"                   ; Section 9.4
        | "POST"                   ; Section 9.5
        | "PUT"                    ; Section 9.6
        | "DELETE"                 ; Section 9.7
        | "TRACE"                  ; Section 9.8
        | "CONNECT"                ; Section 9.9
Enter fullscreen mode Exit fullscreen mode

Therefore our method rule should be



method = { ("GET" | "DELETE" | "POST" | "PUT" | ...) }
Enter fullscreen mode Exit fullscreen mode

Here our rule says: match the exact word GET or DELETE or ... you get the idea.

Next, for simplicity purpuses, we will say that the uri is anything that is not a blank space



uri = { (!whitespace ~ ANY)+ }
whitespace = _{ " " | "\t" }
Enter fullscreen mode Exit fullscreen mode

Note: here we also defined that the white space can be a space or a tab. Note that _ means that the rule is silent; it does not produce tokens.

The HTTP version goes like:



version = { (ASCII_DIGIT | ".")+ }
Enter fullscreen mode Exit fullscreen mode

Meaning: any digit or a dot character. In our request_line the version is defined as the word "HTTP/" followed by the version.

Finally, we require a NEWLINE at the end of our request line.

Let's test this out on the editor.

request line

Perfect! Note that the whitespace rule does not produce a token, we only see method, uri and version. Let's continue with the headers.



header = { header_name ~ ":" ~ whitespace ~ header_value ~ NEWLINE }
header_name = { (!(":" | NEWLINE) ~ ANY)+ }
header_value = { (!NEWLINE ~ ANY)+ }
Enter fullscreen mode Exit fullscreen mode

In here we say that a header_name is anything except for : , because this character is a header separator, it delimits the header name. We also don't want a new line, we will see why bellow.
The name is followed by the separator : and then 1 white space.
The header_value is anything except for a new line, because the new line delimits a header. After the value we encounter a new line.

Let's test this again:

headers

Here we can se that the parser matches 1 header as { header_name: "auth", header_value: "token" }. Did you notice that it also say headers > header ? That is because we also want a rule that can matches a 1 or more header rules. We define this rule as:



headers = { header+ }
Enter fullscreen mode Exit fullscreen mode

This way the pest parser will produce a token headers with a list of header matches.

Lastly the body:



body = { ANY+ }
Enter fullscreen mode Exit fullscreen mode

The body is anything after the headers

body

So far our grammar can parse 1 request from the input. Coming back to the original idea for this post, we want to be able to parse multiple http requests from a file. But here we have a conflict because our body rule will match anything after the headers, and that anything can be another request.
To solve this problem we need the help of a ✨ delimiter ✨

The .http file syntax

To delimit each http request in our file we will make use of 3 # symbols (as the VS Code rest client does).
Let's go add this to our grammar:



delimiter = { "#"{3} ~ NEWLINE+ }
Enter fullscreen mode Exit fullscreen mode

Our delimiter is exactly 3 number signs followed by 1 or more new lines. Then we can say that the body is anything except for a delimiter:



body = { (!delimiter ~ ANY)+ }
Enter fullscreen mode Exit fullscreen mode

Let's test this:

multiple requests

Nice! So finally we can define what our http file should look like:



file = { SOI ~ (delimiter | request)* ~ EOI}
Enter fullscreen mode Exit fullscreen mode

Our http file is composed by delimiters or requests, 0 or more of them.
We have completed our http grammar, now it's time to get rusty

get schwifty

Parsing http files

We will refer to this example for setting up the base code 



extern crate pest;
#[macro_use]
extern crate pest_derive;

use pest::Parser;

#[derive(Parser)]
#[grammar = "grammar.pest"]
struct HttpParser;
Enter fullscreen mode Exit fullscreen mode

We will use the following file for testing



GET https://protiumx.github.io HTTP/1.1
authorization: token

###

POST https://rq-rust.free.beeceptor.com/api HTTP/1.1

{
  "hello": "hola"
}
Enter fullscreen mode Exit fullscreen mode

Note: I set up a mock api in https://beeceptor.com/. It might not be available by the time you are reading this. But you can use any endpoint that accepts POST.

Let's go ahead an parse the file



fn main() {
    let unparsed_file = fs::read_to_string("test.http").expect("cannot read file");

    let file = HttpParser::parse(Rule::file, &unparsed_file)
        .expect("unsuccessful parse") // unwrap the parse result
        .next().unwrap(); // get and unwrap the `file` rule; never fails

        println!("{:#?}", file);
}
Enter fullscreen mode Exit fullscreen mode

we get: 



Pair {
    rule: file,
    span: Span {
        str: "GET https://protiumx.github.io HTTP/1.1\nauthorzation: token\n\n###\n\nPOST https://rq-rust.free.beeceptor.com/api HTTP/1.1\n\n{\n  \"hello\": \"hola\"\n}\n",
        start: 0,
        end: 142,
    },
    inner: [
        Pair {
            rule: request,
            span: Span {
                str: "GET https://protiumx.github.io HTTP/1.1\nauthorzation: token\n\n",
                start: 0,
                end: 61,
            },
            inner: [
                Pair {
                    rule: method,
                    span: Span {
                        str: "GET",
                        start: 0,
                        end: 3,
                    },
                    inner: [],
                },
...
Enter fullscreen mode Exit fullscreen mode

The Pair struct has a pretty clear structure and we could use to print each inner Pair. But we want to get rusty so we will put some effort.

We will define an HttpFile struct. This struct will contain a vector of HttpRequest.
Let's define them:



use std::collections::HashMap;

#[derive(Debug)]
pub enum HttpMethod {
    Get,
    Post,
    Put,
    Delete,
    // ...
}

#[derive(Debug)]
pub struct HttpRequest {
    pub method: HttpMethod,
    pub url: String,
    pub version: String,
    pub headers: HashMap<String, String>,
    pub body: String,
}

#[derive(Debug)]
pub struct HttpFile {
    pub requests: Vec<HttpRequest>,
}
Enter fullscreen mode Exit fullscreen mode

And to get even more rusty we will implement the TryFrom trait to each struct. Let's implement it for the HttpFile



impl<'i> TryFrom<Pair<'i, Rule>> for HttpFile {
    type Error = Error<Rule>;

    fn try_from(pair: Pair<Rule>) -> Result<Self, Self::Error> {
        let iterator = pair.into_inner();
        let mut requests = vec![];
        for item in iterator {
            match item.as_rule() {
                Rule::EOI => {
                    break;
                }
                Rule::request => {
                    requests.push(item.try_into()?);
                }
                _ => {}
            }
        }
        Ok(Self { requests })
    }
}
Enter fullscreen mode Exit fullscreen mode

Note: Pair<'i, Rule> has the lifetime of the input.
As we know, the file rule can contain 0 or more of (delimiter | request). This means that in its content we can find: delimiter, request or End of File.
Here we will try to parse each request, so we need the TryFrom trait for the HttpRequest struct



impl<'i> TryFrom<Pair<'i, Rule>> for HttpRequest {
    type Error = Error<Rule>;

    fn try_from(pair: Pair<'i, Rule>) -> Result<Self, Self::Error> {
        let mut iterator = pair.into_inner();
        let mut ret = Self {
            method: iterator.next().unwrap().try_into()?,
            url: iterator.next().unwrap().as_str().to_string(),
            version: iterator.next().unwrap().as_str().to_string(),
            headers: HashMap::new(),
            body: String::new(),
        };

        for item in iterator {
            match item.as_rule() {
                Rule::headers => {
                    ret.parse_headers(item.into_inner());
                }
                Rule::body => {
                    ret.body = item.as_str().to_string();
                }
                _ => {
                    unreachable!();
                }
            }
        }

        Ok(ret)
    }
}
Enter fullscreen mode Exit fullscreen mode

egghead

Each request can have 5 inner matches: method, url, version, headers and body.
The first 3 do not have inner rules, so we can just extract them as &str. After this, the iterator can only have headers or body pairs. For the headers we perform a similar operation:



impl HttpRequest {
    fn parse_headers(&mut self, pairs: Pairs<Rule>) {
        for item in pairs {
            let mut kv = item.into_inner();
            let key = kv.next().unwrap().as_str().to_string();
            let value = kv.next().unwrap().as_str().to_string();
            self.headers.insert(key, value);
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Grapes! We are pretty much done with the parser. Now let's use the parsed content to make the requests.

reqwest

After a quick check of the available rust http client libraries I opted for reqwest. It has a pretty simple API and it seems to be among the most used libraries for this matters. But I'm a bit concerned about all its dependencies so I might try ureq later.

For this part I just followed the examples on the docs website and ended up with this code:



#[derive(Default)]
pub struct HttpClient {
    pub client: reqwest::blocking::Client,
}

impl HttpClient {
    pub fn new() -> Self {
        let mut headers = header::HeaderMap::new();
        headers.insert(
            header::CONTENT_TYPE,
            header::HeaderValue::from_static("application/json"),
        );
        headers.insert(
            header::ACCEPT,
            header::HeaderValue::from_static("application/json"),
        );

        let client = reqwest::blocking::Client::builder()
            .timeout(Duration::from_secs(10))
            .default_headers(headers)
            .no_gzip()
            .build()
            .unwrap();
        HttpClient { client }
    }

    pub fn execute(&self, req: &HttpRequest) -> Result<(), Box<dyn std::error::Error>> {
        let request = self
            .client
            .request(Method::from_str(req.method.to_string().as_str())?, &req.url);

        let headers: header::HeaderMap = (&req.headers).try_into()?;

        let body = req.body.clone();
        let res = request.headers(headers).body(body).send()?;

        println!("{}\n", res.text()?);

        Ok(())
    }
}
Enter fullscreen mode Exit fullscreen mode

What we can see in here:

  • Use the ClientBuilder to add some default headers and config for the request
  • Parse our HttpMethod into reqwest::Method. Note: HttpMethod implements the Display trait.
  • Parse our HashMap<String, String of headers into reqwest::header::HeaderMap. This is possible because it implements the TryFrom<HashMap<String, String>> trait.
  • Finally clone the body and send the request. We output the body of the request as text (this consumes the body)

Note: We are not trimming the body, perhaps we should.

No we are ready for the last part: the interactive prompt

Lazyness kicks in

At this point I felt a bit lazy to implement the prompt so I searched for existing solutions, one of them being inquire. It has a pretty straight forward API, so to show an interactive select we just need this lines:



    let http_file = parse(&file_content).unwrap();

    let request = Select::new("Select requests to execute:", http_file.requests)
        .prompt()
        .unwrap();

    let client = HttpClient::new();
    client.execute(&request)?;
Enter fullscreen mode Exit fullscreen mode

Note that the HttpRequest implements the Display trait.

The result:

prompt

Gorgeous.

Unintroduction

This project turned out to be quite fun! I ended up naming it rq and you can see its source here

My goal is to have a full interactive prompt that allows you to execute many request without existing the prompt mode.
If you would like to collaborate I have created a few TODO cards here.

That's it. Thanks for reading 👽

Others posts:

Image
Metode Main Judi Online dengan Optimis
Image
Implementacion de videollamadas multiplataforma iOS - Android

ios, mobile, jitsi, ionic
Image
I didn't know it was this exciting !

devchallenge, hacktoberfestchallenge, hacktoberfest, opensource
Image
I didn't know it was this exciting !

devchallenge, hacktoberfestchallenge, hacktoberfest, opensource
Image
The Reason Everyone Is Talking About Biggest Crypto Casino Right Now
Image
Happy? Halloween? 🤔

halloween, learning, discuss
Image
Isoleren met glaswol: tips en advies van Essent
Image
15 Surprising Facts About Mesothelioma Asbestos Attorney
Image
The 15 Things Your Boss Would Like You To Know You'd Known About Wheelchairs Self Propelled
Image
Untuk Pemula Main Judi Online ! Panduan Permainan Poker Judi Online Untuk Beberapa Pemula Pemain Judi Online!
Image
Meble, które odmieniają wnętrza: 10 inspirujących pomysłów na aranżację!
Image
Making a crypto wallet app with React Native

programming, reactnative, mobile, cryptocurrency
Image
Are You Getting The Most From Your Asbestos Exposure Lawyer?
Image
Halloween.dev 🎃 JavaScript Programming Challenges

halloween, javascript, webdev, beginners
Image
e
Image
Arahan Masuk Taruhan di Agen Judi Online Indonesia
Image
Unlocking the Mystery: Deciphering the Enigmatic Code of URL's Glob Patterns in cy.intercept()

cypress, qa, automation, testing
Image
15 Reasons Not To Be Ignoring Asbestos Cancer Claim
Image
Building an AI-Powered iOS Chat App with Amazon Bedrock and Swift

aws, serverless, ai, ios
. . . . . . . . . . . . . . . . . . . . . . . . . . . .