Posts Tagged ‘WordPress’

I’ve been meaning to revisit my WordPress 404 Errors for Spam Comments but somehow it kept slipping through the cracks. I was at a bit of a loose end so I did a quick run through my spam comment log and my site’s error logs…

There were 424 spam comments to my site in December 2010. Of those 74 had an associated 404 error for the comment page that WordPress would have made had the comment been approved. All the comments and 404 errors that paired had IP addresses that matched. I’d say that suggests that the spambots are checking if they succeeded to get a comment posted rather than someone auditing to check the inbound links they’ve paid for exist.

The test for a comment page that causes the 404 error isn’t happening immediately so the bots have been coded to allow for comments being delayed. I can’t think of a reason a comment would be delayed other than moderation. Most of the comments are so blatantly spam that I can’t see most of them being approved. I did a quick sample of the ones I looked at the ones with long lists of links weren’t tested to see if they got through and the random garbage text with a couple of links were sometimes tested.

The most heavily checked spam comments were short like “A very interesting post thanks for writing it!” or “Thanks For the exelent info. I’ll be back in the future. Thanks again! [Link to drugs spam site]“. Maybe a few of the short ones like the second one might slip past someone who’s taken the drugs from the site linked to? I imagine some like the first one will get through on a high traffic blog with lots of comments to moderate.

The original Knight Rider used to end with the refrain of Wilton Knight’s last words to Michael…

One man can make a difference

Tonight I thought I’d share how one page (and a tiny bit of PHP code for WordPress) made a difference to my site.  At least its made a difference for Google.  I’ll apologise for the over stretched Knight Rider references now.

I’d noticed in Google Webmaster tools that around five hundred of my site’s pages were no longer in the Google index that used to be.  I’m not sure if this was the result of Google’s Caffeine launch but the timing of the fall is quite close. I’d also seen a 5% drop in visits to the site but that could be just a random fluctuation in the numbers.

There are some pages I exclude deliberately using robot meta tags like tag index pages and category pages.  They don’t have any useful content so I didn’t want to risk a duplicate content penalty.  Even deducting those from the total number of missing pages and 300+ pages were no longer in the index.

A bit of investigation (the obligatory bit of any Knight Rider episode where Michael would wander round a warehouse filled with cardboard boxes with a torch) and deduction hinted that the missing pages were old blog posts.  The missing ones were buried deep in the blog.  They are in the XML sitemap but Google’s spider just wasn’t getting deep enough through links to think they should be in the index. I’d trusted the sitemap to keep real content in the index but it doesn”t seem to be doing that anymore.  The XML sitemap had become C.A.R.R. to my site’s K.I.T.T!

I can’t just drive into the back of the FLAG mobile unit and get Bonnie to fix the problem.  I had to come up with a solution for myself.  So I added a rather uninteresting Too Many Ideas Contents page that lists every page on my blog by title. It was pretty easy to set up in WordPress with a little bit of code in a plug-in…

Source code

function show_all_posts($atts) {
	$posts = get_posts('numberposts=-1&orderby=title&order=ASC');
 
	foreach($posts as $post)
	{
		$text.='<li><a href="'.$post->post_name.'">'.$post->post_title.'</a></li>';
	}
 
	return '<ul>'.$text.'</ul>';
}
 
add_shortcode('show_all_posts', 'show_all_posts');

I dropped the [show_all_posts] shortcode into a page and magically the list appeared.

Then all I had left to do was wait and see if it worked…

The impact was like Michael hitting Turbo Boost – the day after I published the page Google indexed 500+ pages on my site rather than the 200 it was doing in June and July.  Its done that every day for a week now and the number of pages has actually risen slightly to a high of 617 pages.

The 300 missing pages are now back in Google’s index so I think I can safely say…

One Page Can Make a Difference

Ever since I rebuilt my website with WordPress and imported my blog from blogspot I’ve kept an eye on my server error logs.  I like to make sure people who arrive here from the old blog or from other links end up where they were looking to go. While I’ve redirects that catch almost all of the possible routes in I want to be sure and checking the error logs gives me a way to spot any I’ve missed.

I just went through last months logs.  Of the around 250 error pages served there was a rather annonymous blog page that needed redirection setting up for.  It accounted for 5 of the hits.  10 were typos or calls to deleted pages. 60 were bots trying crude hack attacks on the site trying known vulnerabilities in a variety of software. That left around 170 that I’d not been able to explain.  Every one a call to a comments page that when I checked didn’t exist. The address was a unique combination of a unique comment number and the page the comment had been posted on.

2006/02/more-thoughts-on-the-game-without-a-snappy-title/comment-page-1/#comment-9940

I’ve seen 404s like this for months and couldn’t work out why they were appearing.

Then it struck me what they were.  Those comment pages were for comments that Akismet caught as spam. The only way (short of brute force which would show up in the logs) that someone could know the combination of the comments unique ID and the page it had been posted on was to be the original poster (or the spam system that posted it) .  Those 404 pages must either be the spam bot coming back later to see if it worked or some other system running quality control before paying out for links to a site having been created.

I’ve not had a chance to match up spam messages to 404s because I keep my spam logs clear but I’m going to keep an eye on it and see if they support the idea.   I’m intrigued to see how often they check a comment, if they come fromt the same IP as the spam message and how long after posting they check.

I noticed awhile ago that I was blogging less.  Partly I put it down to frustration with the time it took me to add a page to my web site.  If I couldn’t easily add a page about something I didn’t need to blog about the page.  Partly it was a disatisfaction with the blog and site not being integrated.  Partly it was a concern about what I was blogging about.  I like posting about fun little discoveries – you tube videos, interesting web pages, announcement and all the other stuff thats going on.  However I was always picky about how much of it I posted.  I knew it could easily swamp out my own stuff in a huge noise to ratio disaster.  Its bad enough that I choose to post on several different themes I’m interested in rather than focusing on one theme.  Thats why I use categories and tags to classify my blog posts.

So I delt with the first two problems by switching my site to using WordPress as a Content Management System.  I know some people will say WordPress isn’t a CMS.  Frankly if it isn’t a Web CMS then every other Web CMS I’ve ever worked with isn’t either including the one those people are touting.  Yes it’s a blogging platform but that doesn’t stop it also being a CMS.

The third problem, the noise to content issue, I’ve pretty much put to rest by the happy accident of using microblogging.  Now I signed up to Twitter sometime in 2007 to see what all the fuss was about.  I didn’t really do anything with it till this summer when I needed to know about it for work (and probably foolishly removed my few early tweets since I was using it to test a development at work).  Anyway I’ve been using Twitter for a couple of months now for amongst other things the noise stuff and just consolidating the posts to my blog once a week.  Twitter has the added bonus that unlike blog comments, which tend not to end up being conversational because the lack of a consistent alerting system, twitter can have at least a bit of a discourse.

And the point of this post? I’ve no idea really.  Maybe that blogging, like the web, is still evolving.

Anyway I’m off to do something more useful instead.  Just as soon as I’ve checked Twitter…

• DIY, cleaning and gardening done. Now for something less boring instead… #
• RT @guardiantech The clock cannot be turned back | Peter Thompson http://bit.ly/FbCBs #
• RT @CameronCollie How To Lose a Job Via Facebook via @jearle and @markhughes3000 #
• @richeym Good to see the FA's Respect programme has really been taken to heart at footballs highest levels in reply to richeym #
• @jearle Do you need a rumour starting that tw3.it is under threat? I can start it for you if you want. My rates are reasonable. in reply to jearle #
• @jearle http://tw3.it to be bought by someotherurlshortneruvneverheardof.com. Will that do? I can get it out there for £1m (or nothing) in reply to jearle #
• RT @fieldusThe King, the Moor and the Sheriff. Amusing: http://tinyurl.com/kqqu34 #henrylouisgates #
• impworksRT @fieldus The King, the Moor and the Sheriff. Amusing: http://tinyurl.com/kqqu34 #henrylouisgates #
• Having a quick look at Juitter http://bit.ly/51xSk before heading home #
• RT @lbutlr: http://bit.ly/2bIxya NO BABBYS INZ SHREDDERS!!! via @jearle #
• RT @WeAreConnect Five things to know about Google Caffeine http://tinyurl.com/kpsku2 #
• Awful Library Books http://bit.ly/lm9lI (found via the guardian who didn't think we needed the URL) #
• 2 Sites updated to RT wordpressWordPress 2.8.4 #
• RT @jearle This Fox News report (YouTube) on NHS Terrorists is the single most disgusting piece of propaganda to date: http://tw3.it/MT #
• RT @KimKnox RT: @McKelvie: Graham Linehan's Channel 4 interview on #welovethenhs countering anti-NHS US Right rhetoric #
• Rare eye complaint a few years ago: From GP to Nurses to Consultant and treated in less than 3 hours for free #welovethenhs #
• Wondering how many people can't claim their blog because #technorati normalise urls and doesn't cope with redirects? #

So glad I switched this site to WordPress – adding new pages is so much easier.

Tonight I’ve added a new page for the simple Vue Stereo Image Camera Creation Script I wrote back in 2007.

• @KimKnox SpaceFluffin aX da univers, On d Starship Enterprise under Capt Knox. SFin aX da univers, 1ly goin fwd cauz we cnt fnd reverse. in reply to KimKnox #
• @KimKnox lol It was in my head messing up my editing so I thought I’d share it in reply to KimKnox #
• Doubt they ment this for working out comparative value of money stolen at different famous robberies http://bit.ly/c8VC6 #
• A good days editing almost got a complete piece to send off. Now a little light relaxation before bed… #
• listening to “Johnny Clegg & Savuka – One (Hu)’ Man One Vote” ♫ http://blip.fm/~b1stj #
• listening to “Tasmin Archer – Lords of the New Church” ♫ http://blip.fm/~b1t4h #
• listening to “nik kershaw one world” ♫ http://blip.fm/~b1tkw #
• @guardiantech G-Force http://bit.ly/HyvGW but Bruckheimer’s got a short memory Cats&Dogs did spy pets 8 yrs ago in reply to guardiantech #
• listening to “Billy Bragg – Moving The Goalposts” ♫ http://blip.fm/~b2qox #
• listening to “Ofra Haza Im Nin Alu” ♫ http://blip.fm/~b2r4a #
• Haggis invented in England? http://bit.ly/1olEv5 Doesn’t really matter its Scottish now.
  Just glad I’m not in a meeting with the SAA today. #
• listening to “Sisters of Mercy – Temple of Love ’92″ ♫ http://blip.fm/~b2uo7 #
• @jearle Looks like there might be something fishy going on http://bit.ly/4hyQyv in reply to jearle #
• listening to “Nickel Creek- Hanging by a thread” ♫ http://blip.fm/~b2wmr #
• RT @criticalhits criticalhitsTop 10 Comic Book Cities from Architects Journal http://is.gd/20FDx #
• 2 Sites updated without a hitch RT wordpressWordPress 2.8.3: http://bit.ly/Qf1Cx #
• Akismet 227 : Spammers 0 #
• @very_true_thing I’ve no fish knife hang up but I did buy ice cream spoons on Sat so I can get the last of the jam out of jars – go with age in reply to very_true_thing #
• @very_true_thing It gets worse they do long handled melon spoons too. Is there a market for CDGs (Collectable Dinnerservice Games)? in reply to very_true_thing #
• RT @DanDiplo Is this the best programming related song ever? http://bit.ly/InfF0 #
• @very_true_thing I’ve seen grapefruit knives but never tomato knives – does that mean those are an Ultra rare or just a Promo? in reply to very_true_thing #
• Will you click unfollow, Oh ethereal twit spammers, By autumn? #poem #
• Back from round trip to London to see client. Liverpool a lot warmer. #
• What fuels US sitcom writers? Sort of a RT of @KenLevine http://bit.ly/aqBxN #
• What (if anything) should we read into it if Clarkson is hit by a meteorite now? #
• RT @richeym: @DanDiplo @impworks He does have a way with lyrics. Probably has the best zombie-related song too: http://bit.ly/9j3Ih #
• Is this cartoon the best comment on Bingoo deal? http://bit.ly/H8CGY #
• Quantum physics announcement on the Tube yesterday “Please use all entrances to enter the station” Are passengers wavicles? #
• Is it me or does aquiavic’s comment on my blog make no sense or is it some new sort of weird spam? http://bit.ly/3Shlg #
• Just saw a picture listed on a stock photo site described as “London Sky line 1970s”. Just one thing – when was the Gerkin built? lol #
• For the spec that just went in the recycling bin ♫ http://blip.fm/~b8xmf #
• RT: mediaguardian Twitter’s Evan Williams grilled on Demi Moore in Newsnight interview: http://bit.ly/3NNDTz [The claws come out at the end] #
• RIP John Hughes http://bit.ly/tTBGs ♫ http://blip.fm/~ba1cq #
• @KimKnox You did but don’t worry – I just saw a twitterer who posted the same tweet every 5 minutes for a day… in reply to KimKnox #
• RT @jearle Dear Tony Carter, You don’t get it, you’re an idiot. KTHXBYE. #
• RT @guardianweekly American word introduced into the office – neatnik: a person overly obsessed with tidiness, particularly IN the office #
• @KimKnox Or you didn’t and I saw it on your RSS feed lol.Too many ways to keep up these days.Should just turn everything off once a week. #
• RT @guardiantech: Twist in the tale of questioning spam http://bit.ly/sR5mq #
• RT @guardiantech: Charlie’s Angles: Why can’t local government and open source be friends? http://bit.ly/KPOVY #

• RT @wordpress: WordPress 2.8.2: http://bit.ly/12nm2b #
• Just added myself to the http://wefollow.com twitter directory under: #writer #vue #rpg #
• @SJGames Superfluous O! http://bit.ly/pEDf just after I read the Two Ronnies missing hose http://bit.ly/1p9Wp4 Conspiracy or coincidence lol #
• @DanDiplo lol and incomplete XSLT parsers used to make them very (how can I put it politely) interesting. And variables that don't vary… in reply to DanDiplo #
• A new John Lewis stock pot of slow cooked spicy black-eye beans ment dinner plans were easy http://bit.ly/XpcCS #recipe #food #
• 2 Sites updated without a hitch RT @wordpress WordPress 2.8.2: http://bit.ly/12nm2b #
• @KimKnox They're working hard analysing this to death. A fashion expert focuses on the boots not being done up right has too much spare time in reply to KimKnox #
• @KimKnox Better Paul than Joe who gave me a harsh stare at the deli shelf in M&S on two Sats. I've better things to do than stalk him. in reply to KimKnox #
• @KimKnox lol He was disguised by a forest of facial hair. I guess for a role: would have been a fire hazard on masterchef in reply to KimKnox #
• And Serif Software call me again. At least this time they handled my lack of interest and my not playing along with their script OK. #
• @AlphaComplex Enjoyed the posts but Posting way too often = No follow No more #
• Anyone know if there's a thesaurus of hashtags that are in use? #
• Posted a bit from an old, funny Valkyrie article on how #rpg characters go through doors http://bit.ly/9mmUs #
• @very_true_thing You're right – a few of those games are post Valkyrie I just keep adding to it if I think of a good new one #
• RT @severdia: #joomla 1.5.13 released – upgrade immediately http://bit.ly/25blLU #
• RT @MonteJCook Here's another game site that I enjoy: http://www.gnomestew.com/ #
• @jearle it may be arbitrary but if it make you happy who am I to knock it lol. Congratulations on 1000 followers in reply to jearle #
• @jearle I miss Valkyrie, with all its odd little production difficulties, sometimes. Always sad I never got to do a SLA Industries piece. in reply to jearle #
• @jearle Not really a surprise. It wasn't just Nightfall: Jay told me the slush pile of SLA fan articles could have filled its own mag in reply to jearle #
• @jearle Your stuff every time – helped yours were lots better lol. in reply to jearle #
• @jearle Jay used to worry about being seen as the official SLA rag with an article an issue. He'd email me for funny, grit free articles. in reply to jearle #
• @KimKnox Gratz. 9/10 for cover cheese picture although at least its not Casu Marzu in reply to KimKnox #
• @alyankovic Polka Your Eyes Out but a medley could be even better… #
• Yeh. Vue 8 Announced. Well it will be soon. #

Slogging slowly but surely though all my old blog posts categorising them so I can add better category based navigation now the sites all wordpressified.  Down from 250+ to less than 100 left to go.  Yeh!

Then I can start worrying about tagging the ones that haven’t got tags