There you are. A big redesign or CMS migration is looming, and you’re ready to unleash a crawl on the new site in a test environment. You fire up your favorite crawling tool and trigger the crawl… and it lasts all of three seconds.
Yes, there’s an obvious problem. The staging server is protected and behind some type of security measure that inhibits you from freely crawling it. Sweat begins to form on your brow as you wonder how you will get the crawl completed.
At this point, you could choose to manually check all the pages, but you might end up in a padded room whispering something about header response codes taking over the world.
Or you could continue to click “crawl” and repeatedly crawl a login page, but that won’t help either. OR you could snap out of it and figure out a way to crawl the site in staging, which would enable you to analyze the crawl data and save SEO. Yes, that’s the ticket.
Some of you might be saying, “Hey, this is easy to get around!” Well, it’s important to understand that it’s not so easy sometimes. In my experience, I’ve helped a number of clients that used a staging setup that was just not easy to access and crawl. And for those situations, you might need to use alternative methods.
Comments