7 Practical Steps To Rotate Proxies In Python

A proxy is a helpful tool for more accessible, secure, and private browsing. It works by masking your IP address and using another to send requests to your target.

A static proxy works well for simple tasks. However, a site can stop you from web scraping or data parsing using a single IP. In this case, you'll need a rotating proxy for these tasks.

Rotating proxies, on the other hand, lets you use a pool of proxy IP addresses. By doing so, you can send multiple requests—each with a unique IP.

In this guide, you'll learn how to rotate proxies in Python. Discover how to do it by reading further.

How To Rotate Proxies In Python?

It can be a technical task to rotate IP addresses. Still, it's doable with a few steps. Here's how you can rotate proxies in Python:

Things You Need To Rotate Proxies In Python

There are a few things you would need to do in Python. These are:

• Python 3: You must install Python 3 for the script to run. This version is usually pre-installed in most systems. Still, it's best to download and install the latest version of Python. 
• Pip Install: Pip Install is a command in Python. It looks for the latest version and installs it. It also checks the package metadata for dependencies and installs them to ensure the package has everything it needs.
• List of proxy addresses: Data center and residential proxies are used for IP rotation. Each request is assigned a residential IP address for residential proxies.  Meanwhile, requests are handled separately by data center proxies.

Steps To Rotate Proxies in Python

Here's a guide on how to rotate proxies in Python:

Step 1: Have A Set Of Proxy Addresses

A set of proxy addresses is needed to rotate proxies in Python. There are a lot of free proxies online if you do not have any.

To get a list, save the URLs as ".txt". For instance, save it as proxies_for_rotation_list.txt.

Once you have the list, read the file and make a list with all the proxies. Do this by removing blank spaces and splitting every line.

Here's an example:

Be careful when saving the file since there'll be no test for valid strings. 

Step 2: Start Testing The Proxies

Once you have the list, test to see if the proxies work. You can write a function using ident.me. This page returns a reply with the IP.

You'll also need to download the Python request library. It's an efficient HTTP tool for this test. It works by taking one proxy from the list and calling the URL provided.

The script is often repetitive, but it will be helpful soon. The results could be either:

• If it works: you'll receive a printed status code and a reply text. This one is usually the proxy server’s IP.
• If it does not work: you'll receive a printed error for reasons like a timeout. If this happens, the proxy server isn't usable or cannot work on the request. It's usually the case when working with free proxies.

Here's an example:

Afterwards, save the script in a document. For instance, you can save it as proxies-rotation-for-python.py.

Then, you can launch it from your computer's command line:

Once done, the result should appear like this:

Step 3: Do Extra Tests To Confirm

You must do more tests to verify if the proxies are working. 

If there's an exception, it will show that the command did not work. Besides the exceptions, you should also check other things—like status codes.

You can check by recognizing specific codes as valid. For the rest, mark them as errors. It is also best to try other codes if they don't work.

Another test you can do is confirm if the response includes a specific IP address. Here's an example:

Step 4: Do The Same Process Over Each Proxy

After the testing, do the same process for every proxy on the list. You'll have to repeat the proxies using 'get' to do this step.

It's best to keep it simple through a sequence. You can use asyncio.gather and aiohttp to make the requests. Using async will make the code complicated. However, it's best for web scraping tasks.

It is set to receive only up to 10 packages for protection purposes. This way, you can reduce the number of unwanted commands.

Step 5: Separate Working Proxies From Non-Working Ones

Once you have the working proxies, you must set them apart from the non-working ones. It's best to group these addresses into three classifications:

• unchecked_proxies: when its state is unknown or still needs testing
• working_proxies: when the proxy showed success during the last test
• not_working_proxies: when the last test for the proxy failed

With these classifications, you can add and remove items with ease. It will also help you avoid having duplicate proxies.

Move the proxies from one list to another while avoiding any duplication. To do this, you can make tiny changes in how the proxy data is stored.

Include the proxy list from your file for the unchecked proxies. Here's how you can set it up:

Afterward, use commands to transfer proxies from one group to another. Use only one for every group. It will include the working proxy and exclude it from the rest.

Use the command discard to delete the proxy, whether present or not.

For instance, if a request succeeds, you can name it 'working_set'. This command will separate the working proxy from the not working or unchecked list. Then, it will add the proxy to the active list.

Next, you must change "get" to set these commands after every request. Here's an example:

You can include some trails at the script's end to check if it works. After running all proxies, the unchecked list should be empty.

Step 6: Start Rotating Proxies

The worst case is that you won't find anything on the working list.

Once you place each proxy into groups, you can use the working ones. However, you must make a function to pick a proxy.

You can use the working proxies alone or with the unchecked ones.

With sets, the function "random" is not suitable. For this reason, you must change them using the function "tuple."

From here, you can create a simple Python proxy rotator. Here's an example that picks random proxies from a set of working proxies:

You can then change the "get" command to select a proxy when you can't find any present ones.

With this step, the "proxy" value becomes unnecessary. Use it to test the first proxies like before.

Afterwards, you can leave the list and use "get" without adding it. You'll use a random proxy. It will go to the not_working list if it does not work.

If you're web scraping, you need to scrape actual content. To do this, you need to send back the reply. You can also apply for an exception.

Here's the last version:

You can include the data you need to collect below this script. You can also call the testing URL for a trial.

Another thing to do is make a customized parser using Beautifulsoup. Below is the final code on rotating proxies in Python:

Step 7: Re-Test The Failed Proxies

Re-testing the failed proxies is necessary as it lets you discover more working proxies for several reasons.

Python lets you use the function "Timers" for this step. This action runs at a specific time. It has various methods, but the result is the same. You can do this step with only three lines.

With the reset_proxy command, you can set up a timer and run it on every proxy that's not working.

Set it to around 20 seconds. It may sound too quick, but it's enough for this test. Then, you can remove a non-working proxy by returning it to uncheck afterward.

Here's an example:

Static vs. Rotating Proxies

There are two main types of proxies: static and rotating. Static proxies use one IP address, usually from a data center, for as long as you want. This is good for consistency. Rotating proxies, however, provides more anonymity and variety by switching IP addresses.

Meanwhile, a rotating proxy lets you use an entire proxy list or pool of IP addresses. These addresses change at random or fixed intervals, depending on how you set them.  

Take a closer look at the similarities and differences between these two: 

Similarities Of Static And Rotating Proxies

Both static and rotating proxies serve the same purpose. 

They mask your actual IP address. These proxies enhance your online anonymity for a better browsing experience.

Rotating and static proxies can also let you use shared IPs, depending on the exclusivity you pick for the latter. 

Differences Between Static And Rotating Proxies

With a static proxy, you can use a single IP address for specific purposes. It is generally cheaper since you’re not paying for an entire proxy pool. 

However, static proxies are more susceptible to IP bans. This issue usually happens when you make too many requests to a particular server.

With rotating proxies, you can enjoy higher security and anonymity. The chances of being banned are less since you'll use a different IP for every minute or every request. 

The only drawback to rotating proxies is that there’s a chance that you can use IPs from unrelated geolocations. 

Benefits Of Rotating Proxies

While rotating proxies in Python seems like a lot of work, doing so comes with many benefits. Below are some advantages you'll enjoy when using rotating proxies:

• Avoid trackers

Most websites track an IP address, which can lead to identity theft. 

With a rotating proxy, these trackers will stay closed since you use a different IP with every request. This lets you enjoy better security and anonymity while browsing.

• Unique Requests

Since you'll be using a proxy pool of IP addresses, you can send unique requests every time.

• Skip Anti-Bots

When making multiple requests to modern websites, you’re more likely to encounter anti-bots like CAPTCHA. However, a rotating proxy can help you avoid getting banned since it reflects different IPs for each request.

• Multiple connections with various locations

You can split your connections with a rotating proxy. This allows you to connect with different geo-restricted websites at the same time.

• Avoiding request limits

Rotating proxies help scrape large websites. Using a different IP for every request lets you remain under the radar and avoid restrictions.

• Higher Conversion Rates

Rotating proxies can help you raise conversion rates since every request you make comes from a different IP address.

Bottom Line

A rotating proxy lets you make multiple requests to a target without getting banned since you're using a unique IP address per request. 

Using the steps above, rotating proxies in Python is now doable. It seems technical initially, but keep in mind that rotating your proxies will help you with any complex tasks