Disguise as GoogleBot to view Hidden Content of a Website

Ever experienced this? You ask Google to search something and it will return a lot of relevant results, but if you try to open the ones with the most promising content, you are confronted with a registration page instead, and the stuff you were looking for will not be revealed to you unless you agree to a credit card transaction first. This means that Google is able to see what a normal netizen cannot see.

The reason behind this is that Google uses a Bot called GoogleBot and most of websites which force users to register or even pay in order to search and use their content, leave a backdoor open for the GoogleBot because a prominent presence in Google searches is known to generate sales leads, site hits and exposure. Examples of such sites are Expert-Exchange, Windows Magazine, .Net Magazine, Nature, and many other sites around the globe.

What if you could disguise as GoogleBot then you can also see what GoogleBot can.

How to Disguise as GoogleBot?

It is Quite simple. You just need to change your browser’s User Agent. To change your Browser’s User Agent follow the steps given below:

Copy the following code segment into a notepad file and save it as Useragent.reg or you may also download it. 

Windows Registry Editor Version 5.00[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\User Agent]@=”Googlebot/2.1″“Compatible”=”+http://www.googlebot.com/bot.html”

* Now Double-Click on the file Useragent.reg to merge the registry file into your Windows Registry.
* Now Restart your computer. This is required to apply the changes made into the Registry.
* Voila! You’re done! Now you have become GoogleBot.


How Revert back to Normal Agent?

For IE users : To restore the IE User Agent, Follow the Given Steps Below:
* Copy the following code segment into a notepad file and save it as Normalagent.reg or you may also download it:

Windows Registry Editor Version 5.00[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings\5.0\User Agent]@=”Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)”

* Now Double-Click on the file Normalagent.reg to merge the registry file into your Windows Registry.
* Now Restart your computer. This is required to apply the changes made into the Registry.

For Opera Users : Opera allows on-the-fly for switching of User Agents through its “Browser Identification” function.


For Firefox users : Just download User Agent Switcher extension for Firefox.

• Now Goto Tools -> User Agent Switcher -> Options -> Options.
• Click “User Agents”
• Click ” Add” and fill the following information  in the form

1. Description: Googlebot
2. User Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
3. App Name: Googlebot
4. App Version: 5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
5. Platform: +http://www.google.com/bot.html
6. Vendor:
7. Vendor Sub:

• Click “OK”.
• Now you may change the user agent on the fly.

Source :
http://www.hungry-hackers.com/2009/05/disguise-as-googlebot-to-view-hidden-content-of-a-website.html