SEO Optimization: Scraping Search Results


It is common knowledge that search engines such as Google, Bing and Yahoo built their empires by scraping others content. When it comes to SEO experts scraping these search engines, it becomes a breach their user terms and conditions. This is ironical. However, that does not mean that content owners will not do seo keyword research tutorial in a bid to improve their sites’ ranking. There are many methods to go around this. We will focus on how to scrape Google’s suggestion box.

Detecting Software and Bots

IP addresses-This is the most common method that search engines use to detect anomalies in the search patterns by SEO experts. They can always detect too many requests coming from a single IP. If a high amount of traffic is detected from a certain IP, the search engines will interrupt with a capture page.

Search Patterns-What normally happens is that search engines will link search traffic to existing matching patterns. This happens to a single IP. If there are large variations in the search pattern, they will block the user and classify that as a bot.

Simplified Method

This is the use of Python scripts on Google suggestion box. You should first conduct a search. Then, you should download the first page of the results, followed by the next 4 pages. You should then save the pages to enable parsing.

It is recommended to use script#3 as it gives the best results as compared to scripts #1 and #2. An experiment showed that script#3 is able to perform a total of 150 searches meaning it was able to pull five pages of ranking data for 150 different keywords. Google will definitely detect it but as it is not obvious, they will not block it. It will make no difference to them.

Some of the things that you could do to try and avoid detection are: Clearing cookies after every IP change. You could even disable them completely. You could also try to do some random scraping. If you follow one pattern you will be detected in no time. Trying to switch user agents between queries will also help a great deal. These actions will slow the rate at which Google detects your activities and most probably you will be able to do a perfect keyword research. It would be prudent also scrape at a slow rate. It may make Google think that you are a normal browser having fun.


These scraping tips will ensure that your keyword understanding is greatly enhanced and will help you in optimizing your site to have high rankings across all search engines. This will translate to higher traffic and higher sales.