SEO Tips – Using Robots.txt Files

Robot PicAll too often web developers and internet marketers get bogged down in big lofty ideas about Automotive SEO best practices. When this happens the basics can be forgotten and details overlooked that may have a significant effect on ranking. This blog will help the budding SEO-ers out there learn the basics, and keep the more advanced SEO-ers honest.

First topic will be the use of robots.txt files on your site. Robots.txt files, usually reside in the root directory of your web server, and are used to prevent search engine spiders or crawlers from accessing certain parts of your site. Now I know what you’re thinking, “why would I want to do that?” but there are plenty of good reasons you might not want search engine crawlers to index your entire site.

One of the main reasons you might want to limit a crawlers access to your site is if you sell copyrighted products or downloadable software on your site. Search engines may be able to find and index your intellectual property and inadvertently make it available to any user who knows how to find it. To prevent this you can create a simple text file in a program like text edit on a Mac or notepad on a PC.  This text file will have some very simple code in it telling the search engine crawlers what they can and can’t index on your site. There is a great tutorial on how to create your own robots.txt file here.

Additionally you can use a robots.txt file to limit how and when these crawlers can index your site. Sometimes Crawlers can use up a lot of bandwidth on your website so you may want to dictate how often they crawl your site or even force them to only crawl during non-peak hours.

Do you want to learn more about SEO? Maybe you need some help finding direction for your SEO strategies. Contact Wikimotive for more information on SEO tactics and a free evaluation of your stratagies.


Dealership or Dealer Group *

Name *

Email *

Phone Number *

Request More Information

7 − = four

At Wikimotive, we believe in an aggressive, manually-delivered automotive SEO strategy. No automation. Transparent results.

If you’re an aggressive car dealership that wants to make an impact on your sales very quickly, let’s talk.

Our service includes:

  • Complete revamp of onsite content – make it completely unique to your dealership
  • Ongoing content added monthly
  • Link-earning for domain and page authority (not the link-building that you’re probably getting)
  • Social signal campaigns that actually work (shocker!)
  • Localization strategies – think mobile
  • Citations, directories, the basics you’re probably not getting now
  • Complete search marketing roadmap

We’re not for every dealer, just those who want to dominate!