Wordpress - Google Indexing

[charlie]

added a few new pages to my site yesterday .. not long after I set the actual site up but it appears as though google is only indexing the index page and no others .... Is there something I should be doing that I'm missing ?

www.grumples.co.uk

[Jem]

It's a brand new site, it'll take a while to be fully indexed

[Dom]

Well there's no set time on how long it takes Google to crawl and index new sites, so you'll probably just have to wait.

But you can make sure that you don't have a robots.txt file in the root directory that disallows search crawlers, and you could also try installing this well-known WordPress add-on:

http://wordpress.org/extend/plugins/google-sitemap-generator/

It's a Google Sitemap generator that allows Google to better index your site by creating an XML sitemap according to the open sitemap specifications... or something.

It automatically updates as you create new content, but you can also manually update it through the control panel.

[charlie]

cheers .. the plug in looks useful .. will try that later .. ta :-)

[familychoice]

I'd watch that one.

I've just had problems with a Wordpress site being hacked (dodgy code inserted into the root index page). All the plugins were up to date, the template was fine, latest version of Wordpress. All passwords and FTP accounts changed, and a full reinstall of all the core files didn't make any difference, half an hour later it'd been hacked again.

I then noticed that the XML sitemap files that'd been created matched the time the index page was changed. I removed these and switched off the plugin and haven't had any issues (touch wood) since.

I'll amend this post if it does get hacked again, but all the indications are pointing to a vulnerability with the plugin.

[Mr Anderson]

I'd consider whether all of the content needs to be in pages, because you can frequently get a lot more (quick) SE coverage from blog posts compared to static pages. As an example section - does commentary and current affairs warrant being a series of static pages, or would that be better as a chronological set of posts? I'd argue the latter. Having this section as a category of blog posts would then allow you to automatically post chronological excerpts to the category page, which can replace the main heading in the top, or a list of links to posts in that section (despite having two pages in this section you have no links from the top menu page for it).

With blog posts you have the benefit of being able to ping external sites every time you make a post. The main one to ping is pingomatic which then updates a lot of other sites and you'll quickly find a lot of spider love action going on. Google blog search for example loves blog posts and you'll generally find yourself in there within minutes, and from there into the main search results.

To set up the ping sites in your admin go to Settings > Writing > Update Services. As I said, rpc.pingomatic.com is the main one to have included, but there are others:

http://rpc.pingomatic.com/
http://www.britishblogs.co.uk/xmlrpc.php
http://api.moreover.com/ping
http://api.moreover.com/RPC2
http://api.my.yahoo.com/RPC2
http://api.my.yahoo.com/rss/ping
http://blogsearch.google.com/ping/RPC2
http://ping.blo.gs/
http://ping.feedburner.com
http://ping.syndic8.com/xmlrpc.php
http://ping.weblogalot.com/rpc.php
http://ping.weblogs.se/
http://rpc.icerocket.com:10080/
http://rpc.newsgator.com/
http://rpc.technorati.com/rpc/ping
http://rpc.weblogs.com/RPC2

(Some of those are also covered by Pingomatic IIRC, you should be able to check at http://pingomatic.com/ and adjust as needed. Some aren't, and some may require you to put a confirmation meta in your site (eg. britishblogs)).

Also, have a look at the markup structure in use. You have pages with sub-headings that aren't marked up as headings.

And finally, is the first result in Google for Grumples just an unfortunate co-incidence?

[Jem]

I'll amend this post if it does get hacked again, but all the indications are pointing to a vulnerability with the plugin.

WordPress has no plugin checking / quality control - I have a list of insecure plugins on my laptop at home. This would not surprise me.

ETA: just had a quick look at the plugin and there was nothing obvious in the way of gaping holes (unlike a site I was looking at earlier  ) What other plugins were you using?

[Dom]

Hmm, perhaps that particular plug-in was responsible for one of my sites being hacked a few weeks ago too. I thought it might have been a dodgy template, but perhaps not. Having said that, since I removed all the malicious code etc, the site has been fine, and I haven't disabled the Google Sitemaps plug-in.

You'd think a popular plug-in like that would have had bugs/vulnerabilities reported fairly often.

[Mr Anderson]

I've never had a problem with that specific sitemap plugin (which isn't to say that there's no issue with it); but there are several different ones with similar names, was it specifically that one? I do know of another that had an issue with a XSS exploit, which they're meant to have fixed, but not heard anything anywhere regarding the one linked.

[familychoice]

I'll amend this post if it does get hacked again, but all the indications are pointing to a vulnerability with the plugin.

WordPress has no plugin checking / quality control - I have a list of insecure plugins on my laptop at home. This would not surprise me.

ETA: just had a quick look at the plugin and there was nothing obvious in the way of gaping holes (unlike a site I was looking at earlier  ) What other plugins were you using?

Hardly any for that one: Nextgen gallery, Wickett Twitter thing, Contact form 7 and a dynamic header plugin.

I've never had a problem with that specific sitemap plugin (which isn't to say that there's no issue with it); but there are several different ones with similar names, was it specifically that one?

Yeah, I use it in nearly all of my sites so I'll have to keep an eye on them too.

Maybe it's a coincidence that the hacks have stopped, but as no new content had been added or sitempas generated, the XML files in the root shouldn't have been updated, and the fact they corresponded with the time the index file was changed makes it the most likely candidate in my book - unless the hacks were causing the plugin to update the sitemap!

[charlie]

ok thanks ... Mr A ... I'm kinda new to wordpress and thought it would be better to lay the site out in a conventional manner, but thinking ahead, I expect to be adding a lot more articles as I go and I can see the menu getting pretty crowded, especially as I guess it's not a site people will browse through too much ... I'm aiming it more at search engines and will be including adsense ads on once the article I;ve copied over from my hubpages account get de-indexed ..

so .. all in all, do you think categories/posts would be a better way to go than pages/subpages ?

If so .... is there an easy way I can change my pages to posts or do I have to re-do them all ?

ta

[familychoice]

so .. all in all, do you think categories/posts would be a better way to go than pages/subpages ?

It really depends on how much content you're going to have - but if you're going to be uploading loads of articles then personally I'd use posts and categories. I tend to use pages for more 'static' content such as About Us and Contact forms.

[charlie]

think you're right ... any idea how to do a quick convert from pages to posts ?

[Mr Anderson]

so .. all in all, do you think categories/posts would be a better way to go than pages/subpages ?

Mike's answered as I would, with static content being pages, the rest posts, but I'd add that it helps to understand the purpose of both types - if time has any relevance to the content, and/or if there's to be multiple subsequent similar/related entries then posts are generally more ideally suited. If it's one-off (aside from occasional content that fits as sub-pages), time independent, content then pages are generally better suited.

If so .... is there an easy way I can change my pages to posts or do I have to re-do them all ?

http://www.briandgoad.com/downloads/ptypeconverter/

[Mr Anderson]

pages are generally better suited.

If so .... is there an easy way I can change my pages to posts or do I have to re-do them all ?

http://www.briandgoad.com/downloads/ptypeconverter/

I would add to that, though, that you're likely to lose a lot of the SEO benefit of the content being posts if converting pages to posts like that. As I think it's unlikely that the site will ping the remote services I mentioned above. As you're hoping for SEO benefits it might be worth actually recreating them manually so that when the post is made you ping Pingomatic (if you have it set up to). It should be relatively quick to do - Add new post > copy and paste the title > copy and paste the content > pick a category > publish.

You can of course also manually ping them by visiting http://pingomatic.com and filling the form in, but that would be longer winded than just manually converting to posts I think.

[charlie]

yep .. did a copy and paste with the commentary section articles and was fairly quick so may set aside an hour or so to do that .. haven't set the pingomatic thing up yet but will before I change the others

cheers

[sarahA]

Maybe it's a coincidence that the hacks have stopped, but as no new content had been added or sitempas generated, the XML files in the root shouldn't have been updated, and the fact they corresponded with the time the index file was changed makes it the most likely candidate in my book - unless the hacks were causing the plugin to update the sitemap!

Or the hack actually modified that file too? Were any other sites on the same server hacked in a similar manner? WP or not? Could simply have been someone who'd gained access to the whole server and modified index.* files. Your sitemap file date changing could be due to another unrelated reason (page/post added/removed usually)

[charlie]

ah pingomatic is automatically set up in the latest WP themes 

[familychoice]

Or the hack actually modified that file too? Were any other sites on the same server hacked in a similar manner? WP or not? Could simply have been someone who'd gained access to the whole server and modified index.* files. Your sitemap file date changing could be due to another unrelated reason (page/post added/removed usually)

No, none of my other WP sites have been affected.

Nothing else changed on the site, no new pages or posts so the sitemap updating was a bit weird, could be as you've said that the hack changed that as well, or maybe it was used to gain entry to the root. Maybe the plugin was malfunctioning and changing files it shouldn't.

The file was being changed within half an hour of my reloading the original each time, and this went on for a whole day - when I switched off the plugin and removed the sitemap files it stopped so unless it happens again I'd say there could be a link there somewhere. I'm keeping it turned off just in case!

[suedenem]

TBH, I'd steer clear of unnecessary plugins until you've mastered the basics by hand.  The site is tiny at the moment, so there's hardly the need to automate things anyway.

To start with, do the following search to see what pages are indexed...

site:http://www.grumples.co.uk/

Two things should jump out here:

Google Webmaster Tools advert.  If you've not already done so, sign up, add your site and verify it.  It'll give you information about when you site was last crawled, any problems the Googlebot encountered and so on.

Tag archives.  There's absolutely no point having these indexed - it clutters your results page, detracts from the user experience and uses up your 'crawl budget'.  Google will only crawl and index so many pages in any one crawl, so you should prioritise your most important pages - block Googlebot from crawling these pages by adding this to your robots.txt:

Disallow: /tag/


A couple of other things should help expedite full indexing:

XML Sitemap.  As discussed above, a sitemap will tell Google where to look.  Generate one at www.xml-sitemaps.com.  

Remove any pages that you don't want indexing - login pages, forgotten password, tag archives and so on.  Exclude these in your robots.txt file as above.

Then add this to your robots.txt:

Sitemap: http://www.grumples.co.uk/sitemap.xml

Once you get going with writing look at a plugin to automate the sitemap, but do it by hand initially.


Links.  Beg, borrow and steal links from other sites.  Leave comments in blogs, forums.  Submit to directories.  Write something so good that others will want to link to it,  and link to it from Chant4 pages.


Sue.


PS - your first post starts in the third person and veers into first.  Makes you sound a bit schizophrenic ;-)

PPS - sign up with Google Analytics and track traffic from the start.  You'll be able to make informed decisions about what to write for if you can see what's popular.

PPPS - read through the 'bandwidth' thread and apply the principles to this site.  B/W probably won't become an issue with this site, but it'll help keep it nice and quick.  Site speed can have a marginal effect in some Google searches.

[charlie]

Thanks Sue ... looks like some very useful advice there and I'll look into those ... currently changing pages to posts and then will look at these. It is pretty small but the plan is, when I've got it working properly, to add an article more or less every day so I expect it to grow to a decent size.  I haven't added the adsense yet as several articles have been taken from an account I had at another site and they are still indexed in google so I need to wait for them to vanish first.


thanks

[suedenem]

Analytics, not Adsense ;-)

It'll help you find out what pages are attracting visitors and help you build your site in the medium and long term.  Start collecting data as soon as possible, though.

It's especially important if you want to drive revenue from your site.  You'll be able to see which are your most lucrative posts and do more on that theme, for example.

[charlie]

Doh .. yep .. I have an account .. I'll add that site too .. cheers 

[sarahA]

Tag archives.  There's absolutely no point having these indexed - it clutters your results page, detracts from the user experience and uses up your 'crawl budget'.  Google will only crawl and index so many pages in any one crawl, so you should prioritise your most important pages - block Googlebot from crawling these pages by adding this to your robots.txt:

Disallow: /tag/

To be honest I would add archives to that list too eg. http://www.grumples.co.uk/2011/ whilst it's not linked to, it's still there. You just end up having too much duplicated content all over the shop. Have your pages, posts and category pages indexed, block the rest from the search engines (same goes for the author section too eg. /author/grumples/).

Also, any reason why you have the date linked to the post? It's a bit odd! It may be typical of that theme, I've never really looked at it as it's a confusing theme to edit IMO (really don't know why it's released with WP as the default one). Your post is linked to via the title, I would unlink the post date as the anchor text is of no relevance (sorry, appreciate you weren't looking for critic but when you ask, there's an answer!)

[charlie]

Thanks Sarah

No .. it's all good .. I'm really new to WP so all a learning curve ... will trawl through the suggestions on here over the weekend .. cheers all 

[charlie]

one quick question .. all my articles appear in FULL on the home page ... I presume that stops at some point or it's gonna be a hell of a scroll !!!!

[Dom]

You can put <!--more--> within the code of your articles (or click the button to break the article at the appropriate place) and that should stop full articles being displayed. Instead, you'll get the articles shown up until you put the break in, with a link that takes users to the full article.

[Mr Anderson]

one quick question .. all my articles appear in FULL on the home page ... I presume that stops at some point or it's gonna be a hell of a scroll !!!!

Under admin > settings > read (I think, OTTOMH) you can set how many to display. IIRC the default is 10. This can still be a lot on a single page if your posts are very long, so you might want to consider either reducing the size, or posting excerpts instead of full posts to the home page.

[sarahA]

I would suggest excerpts like Dom and Mr A said, so that the content on the post page is not duplicated again on the home page (same goes for category pages, but I think that's already the case).

[charlie]

I've amended the front page to only show the latest ... i wanted it as exceprts only but there didn't seem to be that option apart from for rss feeds