The beta version of your new site should be blocked from any search spiders so they can?t spider that content. If search spiders are able to access your beta site and spider that content, this could become a huge problem with search engines displaying incorrect results and could potentially hurt your rankings.
You can implement several different measures to ensure spiders do not spider your beta site:
A (robots.txt) File
A robots.txt file is a text file that?s placed on your root domain that tells search spiders which directories they should not spider. You can disallow specific pages or entire subfolders. A robots.txt file that is disallowing several different elements looks like this:
User-agent: *
Disallow: /features_terms_service.php
Disallow: /temp/
Disallow: /beta/
As you can see, each specific page or entire subfolder call-out needs to be on a new line. You can create a robots.txt file with the help of Google in your Google Webmaster Central account.
Data Migration Software
