Proxy Usage

D.R.T. Was designed to be a bulk scanning tool. However, in order to ensure that it performs at peak levels certain requirements have to be met.

First off, it's important to understand that most search engines (Yahoo, Google, Alexa, Overture) ban an IP after a large amount of queries. While "large amount of queries" varies from one search engine to another, sending more than 300 queries per day is considered "bulk" and grounds for temporary IP banning.

Temporary IP banning can last anywhere from 5 minutes to 3 hours. Secondary offenses may result in longer bans, and finally permanent bans.

To get around this problem D.R.T. includes proxy server support.

A proxy server is an Internet Server which serves as a bridge between your PC and a site you are querying. The purpose of the proxy server is to hide your IP address from the remote web site so that the queries looks like they are coming from the proxy server. Of course, proxy servers can get banned just as easily as a regular user can and that is why we recommend having at least 20-50 fast, non-transparent proxy servers loaded in the proxy server list.

Not only do proxy servers ensure that you do not get banned from the queried search engine but they can also speed up the search process by a significant factor. For example, Overture recognizes bulk queries from a single PC and as a result slows down the results. Using multiple proxy servers fools Overture into thinking that the queries are coming from multiple PCs and results in a quicker search.

Finding a good source of proxy servers is not hard, however it may be necessary to pay a subscription to receive daily updated proxy server lists. We strongly recommend using proxy servers with the following search engines:

• Google
• Yahoo
• Alexa
• Overture

How many proxies do I need?

The answer to the above question depends on what type of scan you are performing and how many domains are in the scan queue. Use the table below to identify how many proxies should be used for a given scan.

Domains	Overture / Alexa / Google / Yahoo Scan	MSN / PageRank Scan	Whois Scan
1-100	5 Proxies	0 Proxies	N/A
100-1,000	10 Proxies	0 Proxies	N/A
1,000-10,000	20 Proxies	5 Proxies	N/A
10,000-50,000	40 Proxies	10 Proxies	N/A
50,000-100,000	80 Proxies	15 Proxies	N/A
100,000+	160+ Proxies	25+ Proxies	N/A

 

Where can I find quality proxy servers?

Based on experience in providing our customers with proxy servers in the past, we've come to the conclusion that it is best to let our customers find proxy sources on their own. Providing lists in the past resulted in many complaints because the proxy servers were quickly banned. This is because all users utilized the same proxy servers to send bulk queries, which resulted in millions of queries and quick bans.