Reusing HTTP connections in client-server applications

I’m working on a clientserver application which uses HTTP as a transport protocol for API requests, and sometimes there are occasions with a need to execute a few hundreds requests, such as data import or synchronization.

With default Apache HTTP server settings and default LWP::UserAgent options, every new request would result in a new HTTP session, and each time a DNS query is sent out. So, a synchronization process with a thousand object floods the DNS service with the same requests for the HTTP server name. This results in delays, and some public DNS servers apply rate limits which cause DNS lookup failures (had this with a domain hosted at Godaddy name servers).

HTTP 1.1 protocol supports reusing of persistent connections, but it’s not enabled by default in Apache and in the client.

In Apache HTTP server, the following options need to be configured:

  KeepAlive On
  MaxKeepAliveRequests 500

In the Perl client program, LWP::UserAgent needs the keep-alive option:

  my $ua = LWP::UserAgent->new(keep_alive => 1);

With these modifications, the DNS queries are only sent on every 500th API request, and the HTTP connection is reused between the requests, which saves CPU time on the server. This speeds up the whole process significantly, and also prevents the DNS failures caused by rate limiting.

Advertisements

,

  1. Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: