Have you ever encountered a "broken pipe" error in the applications you have developed, during database operations or when making requests to the systems you have connected over the network?

In this article, we will look at the causes and solutions of some errors that you may encounter in network operations while using Golang, wherever TCP connection is established. These network operations can be operations that you perform by establishing a database connection or by making an HTTP request.

Although Golang is mentioned in the topic and content, you can apply the explained approach here in many programming languages. You can even apply the approaches to be explained when designing a system, communicating with multiple applications, using Reverse Proxy, Load Balancer, and client-server communications.

Connection Pools are Everywhere

Most application developers that make network communication use the connection pool structure consciously or unconsciously. Because it is a costly operation to open TCP connections over and over, "pool" structures are well suited to improve application performance and resource utilization. HTTP packets, database drivers or libraries, web frameworks, load balancers and reverse proxies, in short, most systems that communicate via TCP in the background (if there is no inherent contradiction) embrace "connection pool".

HTTP Client - Server Communication with Golang

Let's take a look at how a connection pool has a place in HTTP operations and how it is used in the Golang HTTP package.

HTTP Keep Alive

When you write an HTTP server with Go and send a request via a standard Go HTTP client, the opened "connection" will be stored on a pool to be used again in future requests, unless you specify otherwise.

Now let's create a simple HTTP server instance with the following code:

The server is running on the "8080" port, and it outputs the connection statuses on the server with the ConnState function as console output.

Now let's write the HTTP client code to send our request:

We use the httptrace package to monitor client connection status. The thing to note here is that the http response is read completely with the io.ReadAll() function. If we do not do this, the connection keep-alive function will not be performed.

Let's run our server and client code and look at the outputs:

Our server application prints the connection status as New for the first request and remains Active throughout the request. When the request is completed, the opened connections switches to the Idle state.

Subsequent requests switch between Active and Idle states. Because a new connection is not opened, the first connection is used again and again.

Let's look at the output of our Client application during this process:

When the first request is sent, we see that the connection start process takes place. When the request is completed, the connection switches to the Idle state and the same connection is used again for subsequent requests.

Go maintains a connection pool on the Transport type within the HTTP client.

In the screenshot below, we can see the idle conn pool in the Transport definition.

type Transport struct {
   idleMu       sync.Mutex
   closeIdle    bool                                // user has requested to close all idle conns
   idleConn     map[connectMethodKey][]*persistConn // most recently used at end
   idleConnWait map[connectMethodKey]wantConnQueue  // waiting getConns
   idleLRU      connLRU
...
}

Idle Connection Settings

If we want, we can make various settings for idle connections on both client and server.

We can make the following settings for idle connections to be kept on the HTTP client:

http.Client{
  Transport: http.Transport{
   DisableKeepAlives:   false,
   MaxIdleConns:        100,
   MaxIdleConnsPerHost: 5,
   IdleConnTimeout:     60 ,
}}

DisableKeepAlives : Does not maintain keep alive connections if set as true

MaxIdleConns : Total connection number can be hold on Transport

MaxIdleConnsPerHost : Total connection number can be hold for a specific host on Transport

IdleConnTimeout : Idle connection timeout on the pool

Creating a TCP Connection in Golang

We made a quick introduction to the connection pool structure with HTTP requests and saw its implementation on a real example.

Now let's create a TCP server ourselves and establish a connection.

We created a TCP server listening on port 8080 with the Golang net package. And we processed all incoming connections with a goroutine.

We created a client that makes TCP connection requests using the net package. We sent the hello data over the connection we created in the loop.

Now let's run our program and look at the application outputs.

We ran our server application and immediately after that we also ran our client application. Our server application has received a new request and printed the received data to the screen.

Our client application also successfully wrote the size of the data it sent to the screen.

Broken Pipe Error

So what happens if we close our server application while both applications are running?

Now, after running the example above, let's end the server application for a while and look at the output.

After closing our server application, our client application gives an output as above.

After the server is shut down, our client application encounters an error as seen. The error message appears exactly as write tcp 127.0.0.1:58766->127.0.0.1:8080: write: broken pipe.

As it can be understood from here, if an opened TCP connection is closed by the server, it cannot be notified until the client side writes.

Lets Create Our Own TCP Connection Pool

We have seen how the Broken pipe error happens. If an opened TCP connection is closed by the server while it is still in use, client applications receive an error as expected.

Let's create a connection pool to bring this error closer to real life examples.

This is quite basic pool. Let's run our client application again.

When we close the server after a while after running the client application, we see that the connection received from the pool on the client side encounters with a broken pipe error.

In our examples we are talking about closing the server application, but this may not always be the case. In fact, the important thing here is to close the connection on the server side.

An idle timeout value can be given for connections created on the server side, and the connections of requests that are satisfied after a certain period of time can be closed automatically.

In such a case, a connection opened and used by the client and stored in the idle pool may be closed by the server. In such a scenario, the client will encounter a broken pipe error when trying to reuse the connection from the idle pool.

Where can we encounter with this broken pipe error?

Database Connections: The libraries we use while connecting to databases can contain pools. We may use packages containing pools. In this case, we should consider that we may receive an error on the application side against the idle connections closed by the database.

Reverse Proxy & Load Balancer Applications: Many reverse proxies and load balancer applications have a connection pool structure on their own. In the configuration of such applications, the idle connection timeout values must be less than the timeout value specified on the server side, so that the connection is terminated by the client before the server shuts down.

See you on the next article, I wish the bug free developments to all of you.

Let's Connect:
Twitter
Github

You can support me on Github: Support mstrYoda on Github