I’m working on a client–server application which uses HTTP as a transport protocol for API requests, and sometimes there are occasions with a need to execute a few hundreds requests, such as data import or synchronization.
With default Apache HTTP server settings and default LWP::UserAgent options, every new request would result in a new HTTP session, and each time a DNS query is sent out. So, a synchronization process with a thousand object floods the DNS service with the same requests for the HTTP server name. This results in delays, and some public DNS servers apply rate limits which cause DNS lookup failures (had this with a domain hosted at Godaddy name servers).
HTTP 1.1 protocol supports reusing of persistent connections, but it’s not enabled by default in Apache and in the client.
In Apache HTTP server, the following options need to be configured:
KeepAlive On MaxKeepAliveRequests 500
In the Perl client program, LWP::UserAgent needs the keep-alive option:
my $ua = LWP::UserAgent->new(keep_alive => 1);
With these modifications, the DNS queries are only sent on every 500th API request, and the HTTP connection is reused between the requests, which saves CPU time on the server. This speeds up the whole process significantly, and also prevents the DNS failures caused by rate limiting.