Wordpress Bandwidth

[Barney McGrew]

Since changing my site over to Wordpress my bandwidth has increased greatly, and in December, I nearly touched 100GB for the month, quite worrying as December is one of the quieter months for my site.

I have installed the plugin WP Super Cache but it does not seem to have made a great deal of difference. This may be down to the fact there are so many settings I may not have tweeked it enough, I have simply left it as a default install.

Any advice on how to reduce or configure this would be greatly appreciated as obviously 100GB is way too much.

[Rosco]

you could use robots.txt to stop search engines going into your images folder?

[Barney McGrew]

I shall do that as a matter of course, however, there are now 25% less images on the wordpress site than there was previously, so not sure if that will make an impact. The site has jumped from 25GB a month to nearly 100GB

[Mr Anderson]

I would robots.txt wp-admin, wp-content and wp-includes. I'd also check my stats to see where my traffic was coming from and if there's anything out of the ordinary then look at maybe blocking certain visitors if necessary. However using WordPress will often result in a bump in traffic as you'll likely be notifying Pingomatic every time you write a post, which then updates various SEs very quickly.

[Barney McGrew]

OK, I will look to do that also. There are no blogs on the site, simply static pages which I presume will increase it as they are all created on the fly each time.

I will go block those directories now, see if that reduces it any.

[rutty]

If Google is causing you some issues you can log into Google Analytics and throttle their visits to your site. I had to do this last year.

Most of my bandwidth is used up by robots of various descriptions so I'd highly recommend using robots.txt to block any/all image directories for certain

[sarahA]

I can't see the super cache reducing your bandwidth. That's designed for slow sites where the processing of php/mysql is slowing the loading of pages down so the super cache has simply created a html version of the pages to load instead. Of course if you have regular visitors then the super cache may help but the browser would cache the dynamic page too anyway if the browser settings were set to do that, and so shouldn't make much of a difference.

Generating the pages also shouldn't use much extra bandwidth as it's the processing that's used, but bandwidth is simply downloading the files, and if you check your stats you'll see that a visit to the front page will load what the front page requires eg. the front page and it's associated css and graphics.

My guess is it's the increase in search engines and also potentially the increase in visitors/page views. As Mr A said, check your stats, compare the number of human visitors and the average page views with stats from your old site. Check the spiders usage but be aware that plenty of bots will be classified as human. Also plenty of bots unfortunately do not follow the robots.txt rules.

If you have AW Stats then you can also view which file types are causing the greatest bandwidth usage. I used to find PDFs were getting downloaded a lot and had to restrict visits to this using robots.txt. You can use robots.txt to slow down the visits too, again, only the major/decent bots will read the rules before entering the site.

if you find one or two bots that are using more bandwidth than you want and you're not fussed on who they are (eg. from unknown sites) then you could use htaccess to just block them completely.

[suedenem]

** Apologies in advance for the verbose reply - stick with it though if you want to make bandwidth savings **

Unfortunately everyone has missed the most obvious cause of this - a badly optimised site!

Blocking bots might cut your bandwidth bill, but be careful of cutting off desirable traffic.  Extra bandwidth use might be an acceptable cost for visitors gained from Google Images, for example - make sure you look at it in the round.

As Sarah says, robots.txt will only skin the cat if the bot behaves.  It's probably worth adding a blocklist of bots to httpd.conf/.htaccess anyway.  The same caution about blocking desirable traffic applies; try this blocklist as a starting point:

http://www.javascriptkit.com/howto/htaccess13.shtml

Bandwidth use by bots is likely to be a bit of a red herring, though.  You say that you've had a big increase since changing your site architecture - whilst Wordpress isn't intrinsically a bandwidth hog, techniques used by many (most) theme developers lack optimisation.

(In fact, it's not just WP themes.  The web community at large seems to have lost the art of optimised code, server configuration and graphics).

You can make significant bandwidth reductions by looking at the following areas:

GZIP Compression
Your server automagically gzips files served as HTML before sending, if the browser requesting them indicates it can handle the ungzipping (90-95% of all browsers, nowadays).  This is courtesy of Super Cache, which has a GZIP compression setting.

The effect of this is that the bandwidth used to transfer your home page's HTML has decreased from around 30.7KB to 6.43KB - whilst 23KB it may seem insignificant, it's a decrease of 80% and mounts up over the entire site.

The good news is that both your main site and forum (traditionally a resource/bandwidth hog) compresses HTML content, likely thanks to the software being used.

HOWEVER - HTML is only a small part of each page.  Your site is serving (a ridiculus!) 12 JS files and 4 CSS files for the home page, all of which are being transferred uncompressed.

Your JS files total about 225KB and CSS files just shy of 100K - compressed, the bandwidth savings could be around 260K for each page view.

You should only try to compress 'plain text' (not binary) files.  It seems like mod_deflate is already installed as an Apache module, so you should be able to add the following to your httpd.conf/.htaccess file and get immediate bandwidth savings:

Code:

AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript

Estimated bandwidth savings: 250MB per 1000 unique page views (WP site only).


Image Optimisation
Another art lost in recent years...

You have about 1MB of images on your home page alone - mainly the slideshow of 4 images.

Use smushit to strip out any unnecessary image meta information.  You could shave about 20KB from slide1.jpg, 32KB from slide2.jpg, 50KB from slide3.jpg and 30KB from slide4.jpg for example - a total of 132KB for someone visiting your home page for the first time, without the image quality dropping in any way.

If you can live with a slight drop in image quality, the following savings can be made if you 'save for web' in Photoshop at 60% quality, and then smush them:

Image	Before	After (PS Optimised)	After (PS Optimised and Smushed)	Total Saving
Slide 1	208KB	70KB	68KB	140KB
Slide 2	208KB	81KB	80KB	128KB
Slide 2	269KB	109KB	106KB	163KB
Slide 4	270KB	95KB	91KB	179KB
TOTAL	955KB	355KB	345KB	610KB

Estimated bandwidth savings:
129MB per 1000 unique home page views (Smushed).
596MB per 1000 unique home page views (Optimised for web and smushed).
   

Javascript
You could let Google host your jQuery and SWFObject files.  They'll hardly ever be down, and it's free:

http://code.google.com/apis/libraries/

Estimated bandwidth savings: 80MB per 1000 unique home page views

Fonts
Using cufon for fancy fonts increases bandwidth use by the size of the cufon library (18KB) and the JS font (42KB).

Consider whether defining standard fonts in the CSS font stack would suffice, or if either the free Google Font Library or services such as Typekit (both hosted) would provide an equally good (or better) font with no bandwidth costs.

• http://code.google.com/webfonts
• https://typekit.com/plans

Estimated bandwidth savings: 60MB per 1000 unique home page views


Other savings...
There are numerous other areas which you could look at, but I'd concentrate on those above first - it's possible that you could save 1GB per 1,000 visitors to just your home page and around 400MB per 1,000 visitors to other pages.

If you still need to cut back after changes above are made, look at setting appropriate file expiry headers to ensure files are cacheable, cutting back further on the use of JS/CSS libraries (which are easy to drop into sites, but necessarily bloated compared to custom-made code), changing the design to make use of less bandwidth-hungry and fewer images and so on.

Incidentally, if you do implement any of the above, post back here to let us know what reductions have been made.

[Barney McGrew]

Thanks Sarah and Suedenem for the advice above. I shall spend tonight going through this. Some of it looks a little above me, but I will give it a go and of course report the results over the next week.

Incidentally, the forum has not seen any significant increase in traffic and has remained the same for the past few years. The high bandwidth on my AW stats shows the increase I dropped my site and uploaded the word press, so its got to be in there

[Barney McGrew]

I have just looked at the AW stats of my site and took a screen grab. There is some large numbers on the JS files and also on the PHP element of it. These stats were taken from the December stats as this is the only full month WP has been online.
http://www.carlhaslam.com/testimages/stats.jpg

[Mr Anderson]

Those don't tell you much, have a look at the stats for specific files instead.

[Barney McGrew]

Cant work out how to do it individually. I just noticed that php gives a bandwidth of 19GB etc which look excessive. I shall see if I can look further ?

[Mr Anderson]

Cant work out how to do it individually. I just noticed that php gives a bandwidth of 19GB etc which look excessive. I shall see if I can look further ?

That's web pages served with a .php extension rather than a .html one.

For individual pages have a look at the section beneath the file type section, titled pages-url (top 25), and accessed via the viewed link in the navigation section of the left-hand menu.

[Barney McGrew]

This is what I get http://www.carlhaslam.com/testimages/stats1.png

[suedenem]

Your stats confirm that robots aren't slurping too aggressively.  I'd personally remove the /wp-content/images/ disallow from robots.txt - images are increasingly important for good SEO practice, and blocking them could have a negative impact.

You've made a good start, though.  Enabling gzip compression for JS has reduced the bandwidth used for the main libraries on the homepage from 227KB to 72KB - a 68% reduction.  Extrapolated over your overall JS b/w consumption, it should reduce it from 9.61GB to just over 3GB per month.

You could cut most of this out altogether if you call jQuery and SWFobject from the Google API library rather than from your own site.

Gzip compression has reduced b/w used by CSS by around 81% - saving about 3.1GB per month.

10GB of bandwidth is being used by thumb.php.  Your slideshow images are funneled through this file, hence using so much.  Run them through smushit.com (it's dead easy!) and you'll save 1.65GB per month without any loss of image quality.  If you're willing to compromise on quality, running the 4 images through Photoshop's 'Save for Web' at 60% quality would save you just under 6GB per month.  Download a trial from the Adobe site if you haven't got it - it'll be worth it.

Keep us updated :-)

[Barney McGrew]

I have CS5 so have saved the slider images for web and then used Smushit. Also did the 4 smaller images on the home page to help it along. I have also installed a plugin which claims that it uses the Google versions of the JS files rather than my own, not sure how I can check if that is doing its job or not?

I also added the deflate code to the htaccess file as above, correctly I think, however, I am not sure how I can check its working or not.

Also took out the disallow images within the robots.

Thanks for your continued help on this, much appreciated. 

[sarahA]

Your jquery is being called from Google Check in your source and in the header you'll find the call to jquery and you can see the URL.

[suedenem]

I have CS5 so have saved the slider images for web and then used Smushit. Also did the 4 smaller images on the home page to help it along.

Slider images are still mahussive for me.  Have you uploaded them?  You might need to clear your WP cache (probably somewhere in the Supercache settings), or even rename the images and re-upload via whatever method your theme suggests.

I have also installed a plugin which claims that it uses the Google versions of the JS files rather than my own, not sure how I can check if that is doing its job or not?

It seems to be working - your code (view source) is now showing:

Code:

<script type='text/javascript' src='http://ajax.googleapis.com/ajax/libs/swfobject/2.2/swfobject.js'></script>
<script type='text/javascript' src='http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js'></script>

...which means it's calling from the Google API site  

I also added the deflate code to the htaccess file as above, correctly I think, however, I am not sure how I can check its working or not.

Yes, this is working.  Go to:

http://web-sniffer.net/

and put in the URL of one of your CSS/JS files, such as:

http://www.fireservice.co.uk/wp-content/themes/awake/style.css

Make sure "Accept-Encoding: gzip" is ticked.  In the response header, you'll see that "Content-Encoding" is set to 'gzip'.  Below that, it'll say "Content (encoded: 13.81 KiB / decoded: 75.76 KiB)", which shows the difference in size during transit (decoded is what it'd be without the deflate code; encoded is what it is with it).

It's considerably smaller, which is good for your bandwidth bill and liked by both users and search engines who prefer quick download speeds.

Also took out the disallow images within the robots.

That's the call I'd make - I know others prefer to have it in almost as a default.  Keep an eye on your stats - if bots become a nuisance you can always add it back in.

Thanks for your continued help on this, much appreciated.

NP  It'll be interesting to see how much you can reduce your b/w by without adversely affecting the quality of your site, so post some renewed stat screenshots after a week  

[Barney McGrew]

Ah, thats good news. I also took Sarahs advice and got rid of the WP Super Cache ? However, I have discovered a folder in lib/cache which I deleted the contents from. I then got a 500 error and could not recover from it. Not sure what happened but managed to get it back again when I uploaded my old .htaccess file. I have since edited the file to include the deflate values.

What happened and is this cache required. ?

[Barney McGrew]

Just deleted the cache folder and it created itself once I loaded my site up again, hopefully that will have cleared it

[suedenem]

I'm not sure if Sarah was explicitly advising you to remove WP Super Cache - more just questioning whether it would reduce b/w.  There might still be merit in keeping it to reduce the amount of DB queries, assuming that's what it's intended to do (sorry - WP extensions not my area... maybe Sarah could advise).

Digging a bit deeper, you need to upload the optimised slide gfx to the following locations:

• http://www.fireservice.co.uk/wp-content/images/various/slider1.jpg
• http://www.fireservice.co.uk/wp-content/images/various/slider2.jpg
• http://www.fireservice.co.uk/wp-content/images/various/slider3.jpg
• http://www.fireservice.co.uk/wp-content/images/various/slider4.jpg

That should do the trick :-)

[sarahA]

To be honest i don't use any cache WP plugin, never needed to, but then again I'll usually tidy up a theme that I get out the WP repository (as I can't design for toffee!). Depends on your visitor load. I would personally recommend to get your site tidied up as you're doing and not use the cache plugin, then see how things go. If things are still loading slow then try the plugin, however I think all of suedenem's suggestions are spot on. To be honest I didn't even think to go and check the size of images etc. but then again my head isn't all there these days!

Also, as a side note, beware of plugins that ship with their own versions of jquery. Put the Google option aside, WP ships with jQuery and it only needs loading once which can be done using a WP function that will ensure it's only loaded once (regardless of the number of function calls for it), but plenty of plugin and theme developers don't learn that and just whack jquery into their plugin/theme and load it into the page with a script call, thereby loading jQuery over and over. Sorry, it annoys me 

[suedenem]

It's been a couple of weeks since you made these changes - have they made a difference?

Can your stats stats package give you a breakdown for the last two weeks?  Post them here if it can 

[Barney McGrew]

Its difficult to see as I have not had a whole month with the changes. However, the scale does seem to have dropped, I have just used under 70GB of bandwidth, still high compared to the 25GB I had been using for the past 3 or 4 years. ?

Here are a couple of stats
http://www.carlhaslam.com/testimages/jan1.png
http://www.carlhaslam.com/testimages/jan2.png
http://www.carlhaslam.com/testimages/jan3.png

[suedenem]

It's dropped quite significantly.  The graph below shows average transfers over the 5 days prior to the date (so the data point for 22 shows the average daily bandwidth for 17/1-22/1):



You can see the big reduction at day 22 - i.e. the average over the first 5 full days after you implemented the changes.  You're in line to use around 50-55GB per month at the moment.

still high compared to the 25GB I had been using for the past 3 or 4 years. ?

Yes, if you were comparing like-for-like, but you're not.  I assume that this latest design is much richer, with animated slideshows and big graphics.  All this uses bigger files which means more bandwidth.

You can still cut down quite a bit.  Your slideshow images are still unnecessarily using a massive amount of bandwidth.  Download this file, unzip it and upload the images to:

/wp-content/images/various/

You'll save around 100KB for each unique home page view (around 20%), without any drop in quality.  This will probably shave another 5GB/mo or so off.

If you don't mind a slight drop in quality, use these images instead, and you'll save a further 215KB or so - around 15GB/mo compared to your current images.

[Barney McGrew]

Heres a strange one. The slideshow images I have within the various folder are actually around 30kb in size as I really reduced them in CS5 using the save for web option. However, they still seem high when used within the slide show, even after refreshing. I cant understand this ?

Thanks for your continued look in on this suedenem, Im learning a great deal in trying to reduce this bandwidth. In terms of the site, its no richer I don;t think, simply a WP theme I liked and quite a few less images than before. The only real difference I have noticed is that many of the images I now use are png as opposed to the jpg I used previously.?

[Mr Anderson]

It's either using a saved version of the larger images which you've removed, or it's creating them on the fly. Eg.

http://www.fireservice.co.uk/wp-content/themes/awake/lib/scripts/thumb.php?src=/wp-content/images/various/slide2.jpg&w=900&h=350&zc=1&q=100

[Barney McGrew]

Thats exactly what its doing, creating them on the fly I think. I think I can change it to a single image file, which may be worth doing just to see if it changes it. I think I choose a bad theme

[Barney McGrew]

I got rid of the moving front banner displaying those 3 pictures and have left it for a month. Here are the results.

I have managed to almost half the bandwidth by simply carrying out all the steps above, so thank you for all your suggestions.

[ih8mondays]

Is that top graph (mar 2011) correct?

a) It's mostly blue?
b) It's only the 5th March?