• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar

AreWeConnected.com

Custom WordPress Websites that Rock!

  • Home
  • About
    • Hire a Tech Dude!
    • Recommendations
    • Add your testimonial
  • WordPress
    • Custom WordPress Website Build
    • The Managed EVERYTHING Website!
    • Making your site “SECURE”
  • FAQ
    • Making Your Site SECURE
  • BLOG
    • Posts about WordPress
    • Guest Post and Comment Policy
  • Contact

Check your Robots.txt file and why it’s so critical you do

Wordpress · June 12, 2013

127
SHARES
FacebookTwitterSubscribe

Holy Guacamole!  This just the 8th in the series of…

20 Secret Blogging Tips in 20 Days!

Check out the bottom of this post to see the rest.


Avocado LoveThis one might sound a little advanced for some but really it’s as easy as cutting up an avocado.

It’s setting up your robot.txt file

No, we’re not going to start texting robots.  The robots.txt file is one of those things that runs in your site’s background.  If it’s there and setup right it’s a great step towards being found in Google.   If it’s not your site is running under the radar and you probably don’t even know it.

Like the name suggests it’s a text file.  And as you already guessed it’s for the robots.  Those are the ‘Bots sent by search engines to your site.  This file simply tells them which pages and posts are ok to spider and which ones not to.

It’s nothing secret – all public sites should have them.
Here’s the robots.txt files for Google, Facebook, and even the White House!

If you have a wordpress site I bet you already have one too.  It’s probably not as fancy as Google’s or Facebook’s but maybe you don’t need it to be, eh?

Here’s the point to this post.   Don’t blow this off as something too small or inconsequential.  It’s very important stuff.

Let’s go check your robots.txt file and make sure it’s doing what it should be.

Are you ready?  Ok, here we go!

Just go to the root of your site and at the end of the .com or .net (or .org) just add a /robots.txt and then enter.

Your Robot.txt file

It’s probably just 3 lines.  Am I right?

We can decipher that.  It’s telling the spiders to ignore the two folders “wp-admin” and “wp-includes” in your server directory.  That’s ok because those two files contain stuff that makes your site work but there’s nothing in there that should be indexed by Google.

The three folders of wordpressThere’s a third folder that’s a part of your wordpress install called, “wp-content”.  As you might guess we don’t want to tell the spiders to ignore the ‘content’ – that would be bad.  If we want to be found in Google, that’s the folder we DO want them to go to.

Now, IF your robots.txt file looks like that box above AND you are happy with your entire site being crawled by spiders you have nothing to do.  Go be happy!

However, IF you want to be a little bit pickier on who can spider and what – you can either learn to adjust this file yourself (all you need is FTP access and notepad) or you can hire someone to do it (like me!).

BONUS:

If you want to exclude your site from being cached by the Internet WayBack Machine (I’m not sure why you would) here’s your text for that.

# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /

skynet plays the drums

If there’s a site you don’t want – all you need to know is the name of their Bot.
You were going to ask me how to find that out – right?
I’m a step ahead of you!  Here’s a whole database of Bots!  You’re welcome!

Oh and in case you were wondering…

“The Skynet Funding Bill is passed. The system goes on-line August 4th, 1997. Human decisions are removed from strategic defense. Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug.  Skynet has become self aware.”


Ready for a killer wordpress website?  Click HERE.  Read the rest of the series…
[listly id=”5A5″ layout=”full”]

  • Author
  • Recent Posts
Mike Mueller
Connected?
Mike Mueller
VP of Happiness at AreWeConnected.com
A former professional hand model, Mike builds Custom WordPress Websites that rock!


He's an avid hockey fan, rides a mountain bike, sometimes rides a road bike, has a few motorcycles (he had a really fast one, bought a cool orange one, rode a really slow one, and now,a perfect "BDR Weapon"). If that isn't enough, he makes cheese and sourdough bread, loves strong beer and good red wine, and poorly plays the Mandolin.
Mike Mueller
Connected?
Latest posts by Mike Mueller (see all)
  • The Ultimate Guide to Writing the Perfect Blog Post - March 14, 2023
  • 8 Questions Your Web Developer Should Have Asked - April 27, 2021
  • Slack, Chat or Discord? - April 6, 2021
127
SHARES
FacebookTwitterSubscribe
New Comment Policy:
If you see something, say something!
(shamelessly borrowed from Homeland Security but really, let's bring commenting on posts back!)

Filed Under: Wordpress

Previous Post: « Always Set a Featured Image
Next Post: The how and why of Akismet »

Primary Sidebar

Who runs this Pop Stand?

Mike Mueller
Mike@AreWeConnected.com
PH: (925) 456-4567

Connect

  • Facebook
  • Flickr
  • Instagram
  • LinkedIn
  • RSS
  • Tumblr
  • Twitter
  • YouTube

Look, it’s Just a Newsletter!

  • While I might write a bunch, the newsletter is set to go out only on Wednesdays. That's it!  Just one day a week! Can you handle that?
  • This field is for validation purposes and should be left unchanged.

FREE Downloads

Icon
1843Small Biz Websites Built so far…

Let's Build Yours!

Expert Google Workspace help

Get Started and then hire the right help!

Lighten up!

Search: (anything)

Sign In / Up

  • Account
  • Website Management Subscription
  • The Managed EVERYTHING Website!

Who to host with?

It's the #1 question I'm asked - where to find inexpensive shared hosting. Here's who I recommend.
Data Servers are a HUGE drain on the power grid. Here's a host that sources 300% of it's power from Wind and Solar! On top of that, their service ROCKS!
Check out GreenGeeks

Need more power?

WPEngine has it!

Hacked?

Sucuri Security
Creative Commons License
This site was built, is hosted, and backed up, and maintained by Mike Mueller. AreWeConnected is about the open sharing of ideas without a hidden agenda. Assume any link on this site is an affiliate link, Clicking on affiliate links costs you nothing extra (probably saves you $) and helps support this site.
PRIVACY POLICY

MANAGE YOUR ACCOUNT

...another Custom Wordpress Website by Mike Mueller Hi!