Have you ever wondered exactly how search engines read your site? What direction do they read it from…what links and pages are they indexing…and what are those bots reading within your sidebar 
Just think what drastic changes you could make along with unprecedented PR spikes you would receive if you could see your site exactly as a search engine robot saw it….
Well, I’ll spare you from fantasizing and let you in on an extremely valuable yet free tool I just came across that allows you to view your site through the eyes of search engines. Just enter your site URL (even if it’s an individual page within your site!) below and study it in amazement:
Now after you have studied this you will surely want to either:
- Instruct the robots to include certain areas
- Command the robots to exclude certain areas
or
In efforts to keep this post short and simple, I won’t go into all those details…I will simply give you one of the best resources I found concerning how to instruct search engine robots. It’s at AnteZeta.com. Just do what the article tells you and your site will be straight. I’ll also include this robot.txt generator ;)
Now I can’t take the credit for this because I stumbled on it from a Google search which led me to a post by Jennifer Laycock -who originally got famous for her ‘breast milk’ campaign back in the day.
MSB gets tons of content spidered…perhaps even a bit too much, so right now I’m going through a tedious tweaking process with my robot.txt files.
Now tell us, how do the search engines view your site 
IMPORTANT UPDATE: For anyone from a country where people read from right to left -have no fear- your content will still get spidered by Google. Just check out what Big “G” says about this on their blog.
p.s.
For more resources on affiliate marketing online read the free ebook Affiliate Marketing 101 and don’t forget to subscribe to the MSB FEED!
Link to this pageRelated Secrets from the World Wide Web
-
Make a Search Engine in PHP and MySQL Why would you want to make a search engine anyway? There already is a search engine to rule them all. You can use Google to find just about anything in the Internet and I doubt you will ever have the... - Search Engine & Directory Submission Tips Web site submission to look engines and directories is often done incorrectly. So as to get the most effective results from search engine and directory submissions, Swift Media UK offers the following internet site submission tips: Treat search engine submission...
- How To Use Search Engine Optimization to Destroy Your Competition in 3 Easy Steps Search Engine Optimization - Secrets Revealed.Let's begin: What is 'competition'? Competition is described as "one that competes", "a rival", "selling or buying goods or sevices in the same market as another", and even down to the biological level - "an...
-
SEO Tips for Blog Traffic Generation While it may be true to say content is king when it comes to blog publishing, the truth is that writing your blog content is not by far the only thing that you should be focusing on when it comes...
Related Articles here on MSB
- A Guide To Free Search Engine Optimisation Website search engine optimisation and submission There are five main search engines that you will need to submit your website to, they are Google, Msn, alexa, aol and yahoo. There are also many other...
- 6 Search Engine Tips Get The Best Search Rankings Possible If you have an online business then it is imperative that you use search engine marketing to get traffic to your business. To help you do this you need to know some search engine...
- Create A Buzz In The Internet Through Search Engine Marketing! Search engine marketing simply means promoting a site online. In this internet age there are numerous online sites for various products and services. Simply having an online site will trim down your site just...
- Search Engine Optimization – Only One Part Of A Succesful Internet Marketing Strategy SEO (Search Engine Optimization) is not the means to an end. It should only be one part of your total Internet Marketing strategy. It is a fundamental part of your strategy and it is...
- SEO For Topping Search Engine Results In case you don’t recognize, SEO really stands for Search Engine Optimization, and it is the near-science of obtaining websites to the top of search engine results. Although there are a number of alternative...
Tweet This Post!
steps to get your ex back (9)
Catering Florida (2)
Electric under floor heating (2)
3d cad (1)
aapeterjonas10 (1)
![\"Make](\"http://www.instantvideosites.biz/images/banner3.jpg\")
68 users commented in " How Search Engine Robots Read Your Site "
Follow-up comment rss or Leave a Trackback March 16th, 2009 at 3:10 pmsearch engine robots read only text not any code like java script , CSS and flash. it is give the preference to HTML tage like H1 , H2 etc
[Reply]
SECRET TIP: With flash you can use ‘alt’ tags just as would for images to get some bots to read it ;)
[Reply]
Great tool. The article is very helpful.
The Blogger Source´s last blog post..Latest WordPress Plugins #6
[Reply]
Really Search Engine Advertising Choices… house and optimization is a learned-skill, corporate marketing departments lean towards the very simple model of paid-search
[Reply]
[...] Here is the original: Making Money Online secrets revealed » How Search Engine Robots … [...]
Excellent tool. Thanks for all the great links to these free tools.
[Reply]
Something cool to play with, thanks for the find. :)
Dennis Edell´s last blog post..Google Closes Adsense Account – Google Gets Sued – Google LOSES!
[Reply]
what happens If I have a hebrew site which is directed from right to left and not left to right, does the spider read it from right to left?
[Reply]
Its too confusing for me. I have to put some efforts to get it right :)
Rahul Jadhav´s last blog post..Sunday Link Love # 2
[Reply]
Caleb
Reply:
March 9th, 2009 at 12:28 pm
Rahul: Start off by knowing that you want your most important content to be near the top and figure a way to make that happen without sacrificing the user’s experience ;)
[Reply]
for the most part, entities that rely on automated software agents called spiders, crawlers, robots and bots. These bots are the seekers of content on the Internet, and from within individual web pages. These tools are key parts in how the search engines operate.
accident claim´s last blog post..Injured at a Bus Shelter
[Reply]
I believe this is a great tool, but it is something i need to learn about. thanks for this.
Leanie Belle
How To Earn Your First 100 Dollars Online
[Reply]
Heck, you had to use a real picture of a spider, didn’t you? Ick! Well, I put in my blog URL, and it came up with a 404 page, which is freaky since I’m using the Google Sitemaps plugin. I then put in my business site, and I see many pages on my site spidered, but it’s missing a bunch as well.
Of course, now I have to go to that other site to figure out what to do about the spidering of my blog; sigh,…
Mitch´s last blog post..Keys To Leadership
[Reply]
Caleb
Reply:
March 12th, 2009 at 11:44 am
Hey Mitch, I see your blog url is now bringing up plenty of content…what was the original problem
Your answer to this may help a few bloggers out there…
[Reply]
yeah really these bots are the seekers of content on the Internet within individual web pages.
[Reply]
Caleb, what I did was go into the Google Sitemap plugin and there’s an option to open it up for robots.txt, and I selected it. Never knew I had to do that before, and wouldn’t have thought of it if you hadn’t written this post. So, I thank you for it.
Mitch´s last blog post..Spock
[Reply]
Caleb
Reply:
March 14th, 2009 at 2:23 pm
No problem…I’m just sharing
[Reply]
Mitch
Reply:
March 14th, 2009 at 2:42 pm
Yeah man! Then again, I keep coming to this post and forgetting about the image of the spider; freaks me out every time, as I have this thing about bugs and spiders, and no, it’s not a good thing. lol
Mitch´s last blog post..Massive Traffic To Your Website/Blog?
[Reply]
This is an IMPORTANT Response to קידום אתרים: google officially stated the following on their blog…
Read the rest here…
Rest assured, your content will get crawled
[Reply]
[...] is in the same niche as mine I thought it would be interesting to do a comparison using the Search Engine Spider Simulator I told you guys about on the last [...]
Sometimes bots are unable to access the website they are visiting. If a website is down, the bot may not be able to access the website. When this happens, the website may not be re-indexed, and if it happens repeatedly, the website may drop in the rankings.
[Reply]
I not like its all non sporting features as not support to java script.
accident claim´s last blog post..Claim for Severe Cut
[Reply]
Like Mitch I hate spiders, but luckily this one is frozen to the screen. I remember once when I say a huge spider on the windscreen of the car and I put on the windscreen wipers to get rid of it only to find it was on the inside. I stopped the car in the middle of the road, luckily at night when there was no traffic, and made the wife get rid of it. :D
I’ve always used a robot.txt because I read ages ago that search engines preferred sites that had them. Mine are always simple things, but perhaps I should look into fixing it up a bit. Nah, that would be too much of a hassle.
Sire´s last blog post..Googles Interest-Based Advertising Sucks
[Reply]
Caleb (Market Secrets Blogger)
Reply:
March 18th, 2009 at 5:34 pm
It’s not that much of a hassle..try using the robot.txt generator link to make it easier on you
[Reply]
Sire
Reply:
March 18th, 2009 at 6:01 pm
No worries Caleb, I did see that link in your post. Like I said I already have a basic one, and that link would make it easier if I wanted to include other robots. I’ll put it into my ‘to do’ pile.
[Reply]
Mitch
Reply:
March 18th, 2009 at 6:18 pm
Sire, I almost had a major accident once because this wasp decided to fly into the front of my car from the back window; whew! And, of course, coming here to comment to you, I once again forgot about that big hairy thing at the top; freaked me out again. Caleb, you win; the torture!
Also Sire, remember that there is a WP plugin that will take care of it all for you, if needed.
Mitch´s last blog post..Visa Black Card
[Reply]
Sire
Reply:
March 18th, 2009 at 10:25 pm
Which plugin is that Mitch? One for a robots.txt?
Sire´s last blog post..Googles Interest-Based Advertising Sucks
[Reply]
Mitch
Reply:
March 18th, 2009 at 10:34 pm
Okay, this is my last visit to the “spider” post, so get all you need now, Sire. lol I wrote about it on my post, the Google Sitemap plugin.
Mitch´s last blog post..More Free Ebooks On Internet/Forum Marketing
Sire
Reply:
March 18th, 2009 at 10:42 pm
OH, I already had that, I thought you meant a robots.txt plugin, now that is one I haven’t heard of.
Sire´s last blog post..Googles Interest-Based Advertising Sucks
Caleb
Reply:
March 19th, 2009 at 12:11 pm
Just make sure you follow Mitch’s advice on his article to ensure the bots are reading your sitemap
Caleb
Reply:
March 19th, 2009 at 12:29 pm
I hope that spider isn’t scaring away any potential readers…maybe I should’ve went with my original thought and simply used a web instead of the thing that makes the web
[Reply]
nice tool right here.. and i hope that spider doesnt come crawling out of my screen
chris´s last blog post..MOVED: tickets flight 11
[Reply]
Caleb
Reply:
March 19th, 2009 at 12:18 pm
I think Mitch is hoping the same thing
BTW: Chris, in order for your smileys to show just click the ‘Smiley’ link (above comment box) and select the one you want next time…I’ll go ahead and fix it.
And hey, TO ALL COMMENTERS: You can now include videos in your comments…just use the ‘ADD Video Comment’ link instead of the ‘Post Comment’ link to upload your own personally created video or from other sources on the web(Make sure it’s not copywrite protected!):exclaim_ee:
[Reply]
[...] spidering [...]
This tool to stimulate web crawl is fantastic. I have never seen such a tool which tell how search engines spider our website. Thanks for sharing!
[Reply]
Nice tool. I installed one on my blog too. Where do you guys get this cool stuff?
james (binaural-isochronic)´s last blog ..The Power of Appreciation
[Reply]
Really a good information on how search engines crawl web pages. Thanks for sharing! Now I think I can modify my website to improve the traffic to the site.
[Reply]
I remember once when I say a huge spider on the windscreen of the car and I put on the windscreen wipers to get rid of it only to find it was on the inside. I stopped the car in the middle of the road, luckily at night when there was no traffic.
[Reply]
Figuring out how google spiders work is almost impossible. But you gave a good try. Thanks for sharing.
[Reply]
Really Interesting how search engine Robot works. Thanks for sharing
[Reply]
Search engine robots will check a special plain text file in the root of each server called robots.txt before indexing a site. Robots.txt implements the Robots Exclusion Protocol, which allows you as a web manager, to define what parts of your site are off-limits to search engine crawlers. For example, Web managers can disallow access to the Common Gateway Interface (CGI), or private and temporary directories, because they don’t want pages in those areas indexed.
[Reply]
I happened upon a post by our buddy Caleb of The Market Secrets Blog titledHow Search Engine Robots Read Your Site. Once I got past the image of the stupid spide.
[Reply]
Be extremely careful when making changes to your robots.txt
A simple error can be catastrophic. I use the robot checking tool in Google Webmaster Central.
[Reply]
Tha’s what you called a great tool. thanks.
[Reply]
Search engine robots will check a special plain text file in the root of each server called robots.txt before indexing a site. Robots.txt implements the Robots Exclusion Protocol, which allows you as a web manager, to define what parts of your site are off-limits to search engine crawlers.
business mobile tariff´s last blog ..Phones PDAs
[Reply]
Search engine robots will check a special plain text file in the root of each server called robots.txt before indexing a site. Robots.txt implements the Robots Exclusion Protocol, which allows you as a web manager, to define what parts of your site are off-limits to search engine crawlers
liposuction´s last blog ..Phones PDAs
[Reply]
hey whether this google spider visited the site daily actually i want to know the exact time period when it checks our site.
[Reply]
if you know of a site that lists the regular expressions to identify each of the bots for the major engines, that would help to some extent. However, I know that MSN frequently hits my sites guised as a user (without msnbot in the user agent), and I really want to avoid serving up content to a search engines based on its ip address (I’m doing ip-based content delivery).
[Reply]
Robots.txt implements the Robots Exclusion Protocol, which allows you as a web manager, to define what parts of your site are off-limits to search engine crawlers.
sleigh beds´s last blog ..The truth behind the demise of cabinet made furniture pt3
[Reply]
Its very worthy to read this post. Its very informative and useful for all of us. We are looking forward for more info.
[Reply]
This tool is really helpful in SEO endeavors. It’s worth knowing how search engines spider websites.
[Reply]
Its a subject that fascinates me and I cant get enough of talking and reading about. The mystery puzzle that we are all trying to figure out.
Enjoyed reading the post
[Reply]
Caleb
Reply:
August 11th, 2009 at 2:06 pm
Figuring this out will truly put you ahead of the game
[Reply]
Hmm. . Although you can use CSS display:none attribute for that, Its not considered good practice because thats the technique used by spammers to get their sites listed on search engines .. Of lately, search engines have stopped indexing hidden divs .
[Reply]
Awesome tool. Thanks!
[Reply]
Good tool, thank you for sharing.
[Reply]
Thanks for the insight. There is just so much to learn (and so little time).
[Reply]
Caleb
Reply:
November 2nd, 2009 at 6:03 pm
Take one thing at a time and just focus in on it young jedi.
[Reply]
Don’t forget off-page SEO!
[Reply]
Now if the spider could just whisper to me exactly what it is I need to do to rank for what it is I want, I’d be golden ;-)
Till then,
Jean
[Reply]
The higher a website ranks, the higher advertisers will pay its owner. So if a website owner wishes to gain income from search engine marketing such as paid, links, pay-per-click and even banner ads; how much they will actually get depends on page ranking.
[Reply]
Hi, guys, thank you for the post and comments ) i liked )
[Reply]
Great tools thanks! You have some seriously educational and useful things on your blog.
You have made a regular reader out of me!
Treasure Coach Review´s last blog ..Online Business Success
[Reply]
Leave a Reply