4 Easy Tests to Quickly Identify Web Performance Issues

Down the rabbit hole.

That probably best sums up two things:  car repairs and web performance optimization.

In nearly 95% of web performance cases we investigate, the ultimate cause of poor performance is the application itself.  However, few customers want to hear that their application is the cause.  Fixing application issues is costly and often slow.   They really want to hear us say, “we’ve enabled the go faster setting in PHP and your site is blazing fast now.”   Unfortunately, there is no such setting.

While rudimentary, we’ve use the following test framework to quickly rule out server capacity and configuration issues.   Our goal is to find the underlying cause of the performance problem and fix it.  These tests help us focus on the right issue as quickly as possible — saving our customer’s time and money.

4 Easy Web Performance Tests

To do this, we use these four simple tests:

• Test static content
• Test a simple php page (phpinfo)
• Test simple SQL query.
• Test your web application

How to Test

For testing, I use apachebench, curl and webpagetest.org.   I mainly use curl to fetch HTTP headers:

[bash]

[jeffh@office ~]$ curl -I www.rackaid.com/
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 12 Oct 2015 18:31:45 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff

[/bash]

I review the headers to mainly to assure they are sane and someone has not done something stupid.

I then use apachebench to get a baseline performance

[bash]

ab -c 10 -n 100 https://test.rackaid.com/images/logo.png

[/bash]

I run apachebench against each of the four test targets.

Yes, this is a rudimentary test, and you will not likely find the smoking gun, but you will get a hint at where to look: server, network, database or application programming.

Understanding Test Results [su_table]

[/su_table]

Slow Static Content

I had a customer reporting periodic site slow downs.  He said sometimes the site would load quickly but the images would take a long time to populate the page.  This was unusual as modern web servers can handle static content very quickly.  Even an entry level cloud or VPS unit can serve 100’s of images per second.  His page only had 34 static files.

So we started doing some testing by starting by running apachebench against static content.

Here’s what we found:

CHART SHOWING SLOWING AT HIGH CONCURRENCY

This was odd.  The server had no load during these test.  Disk IO was minimal and CPU usage barely changed.   We begin to dig into the network and there it was:

conntrack issue

The firewall was using connection tracking.   What was happening is that when his site was busy or under a brute force SSH attack, the connection tracking queue was filling up.  As a result, the server started dropping packets.  Under severe conditions, this was causing intermittment site performance issues.

The static file test quickly revealed we had a network issue not a web application programming or database issue.  By using this simple test, we identified and fixed the issue in about 30 minutes.

Slow PHP Page

Similarly to above, we had another customer reporting interrmittment slow downs for their site.  They said when they visited certain sites they would be very slow and then suddenly start responding quicky.

So the first thing we did was a static content test. That was fine.

So we then loaded up a PHP info page and did some benchmarking with apache bench.  In general the benchmarking was fine; however, I did discover an issue when checking headers.

Often before benchamrking or testing a site, I find it wise to restart the entire application stack.  This helps clear out any caching that could impact your results.

What I noticed is that when I query the PHP info page just after a restart, it would takes seconds to load.  Once loaded, then it would respond in miliscons on subsequet requests.

CHART SHOWING FIRST vs SUBSEQUENT REQEUST

So I knew we had a general PHP configuration issue.   When I dug into the server, we found that FastCGI had been configured in a way to spin down all processes for a site when it was not in use.   As a result, during the first request, FastCGI had to do a complete startup process.  This was very slow on this server due to CPU constraints.  Once started, opcode caching and the persistent nature of FastCGI enabled fast server responses.

This also explained why the server would suddenly overload.  Digging into the logs, we found loads would spike when bots happened to hit multiple sites at once.  This was a cPanel system with about 100 shared hosting accounts.

So we switched the PHP module from FastCGI to ModRUID.  This resolved the issue.