Updated · Oct 02, 2023
Muninder Adavelli is a core team member and Digital Growth Strategist at Techjury. With a strong bac... | See full bio
Updated · Aug 24, 2023
Florence is a dedicated wordsmith on a mission to make technology-related topics easy-to-understand.... | See full bio
A proxy is a helpful tool for more accessible, secure, and private browsing. It works by masking your IP address and using another to send requests to your target.
A static proxy works well for simple tasks. However, a site can stop you from web scraping or data parsing using a single IP. In this case, you'll need a rotating proxy for these tasks.
Rotating proxies, on the other hand, lets you use a pool of proxy IP addresses. By doing so, you can send multiple requests—each with a unique IP.
In this guide, you'll learn how to rotate proxies in Python. Discover how to do it by reading further.
🔑 Key Takeaways
It can be a technical task to rotate IP addresses. Still, it's doable with a few steps. Here's how you can rotate proxies in Python:
There are a few things you would need to do in Python. These are:
🗒️ Related Article:
Smartproxy is a top-rated proxy provider trusted by many. Its 40 million+ proxies from 195+ locations help bypass CAPTCHAs, geo-blocks, and IP bans. It offers a free trial and has a high rating of 4.7 on Trustpilot with 89% 5-star ratings, making it one of the best in the industry.
Here's a guide on how to rotate proxies in Python:
A set of proxy addresses is needed to rotate proxies in Python. There are a lot of free proxies online if you do not have any.
To get a list, save the URLs as ".txt". For instance, save it as proxies_for_rotation_list.txt.
Free proxies are not the safest and most reliable proxies. Most of them are short-lived. Using residential or data center proxies for scraping or other projects is better.
Once you have the list, read the file and make a list with all the proxies. Do this by removing blank spaces and splitting every line.
Here's an example:
Be careful when saving the file since there'll be no test for valid strings.
Once you have the list, test to see if the proxies work. You can write a function using ident.me. This page returns a reply with the IP.
You'll also need to download the Python request library. It's an efficient HTTP tool for this test. It works by taking one proxy from the list and calling the URL provided.
The script is often repetitive, but it will be helpful soon. The results could be either:
Here's an example:
✅ Pro Tip:
When doing this test, use HTTP instead of HTTPS since most of the free proxies on the list won't support SSL.
Afterwards, save the script in a document. For instance, you can save it as proxies-rotation-for-python.py.
Then, you can launch it from your computer's command line:
Once done, the result should appear like this:
You must do more tests to verify if the proxies are working.
If there's an exception, it will show that the command did not work. Besides the exceptions, you should also check other things—like status codes.
You can check by recognizing specific codes as valid. For the rest, mark them as errors. It is also best to try other codes if they don't work.
Another test you can do is confirm if the response includes a specific IP address. Here's an example:
Step 4: Do The Same Process Over Each Proxy
After the testing, do the same process for every proxy on the list. You'll have to repeat the proxies using 'get' to do this step.
It's best to keep it simple through a sequence. You can use asyncio.gather and aiohttp to make the requests. Using async will make the code complicated. However, it's best for web scraping tasks.
It is set to receive only up to 10 packages for protection purposes. This way, you can reduce the number of unwanted commands.
Step 5: Separate Working Proxies From Non-Working Ones
Once you have the working proxies, you must set them apart from the non-working ones. It's best to group these addresses into three classifications:
With these classifications, you can add and remove items with ease. It will also help you avoid having duplicate proxies.
Move the proxies from one list to another while avoiding any duplication. To do this, you can make tiny changes in how the proxy data is stored.
Include the proxy list from your file for the unchecked proxies. Here's how you can set it up:
Afterward, use commands to transfer proxies from one group to another. Use only one for every group. It will include the working proxy and exclude it from the rest.
Use the command discard to delete the proxy, whether present or not.
For instance, if a request succeeds, you can name it 'working_set'. This command will separate the working proxy from the not working or unchecked list. Then, it will add the proxy to the active list.
Next, you must change "get" to set these commands after every request. Here's an example:
You can include some trails at the script's end to check if it works. After running all proxies, the unchecked list should be empty.
If you're using free proxies, you'll find only a few working proxies in the list. Like the one below.
The worst case is that you won't find anything on the working list.
Once you place each proxy into groups, you can use the working ones. However, you must make a function to pick a proxy.
You can use the working proxies alone or with the unchecked ones.
With sets, the function "random" is not suitable. For this reason, you must change them using the function "tuple."
From here, you can create a simple Python proxy rotator. Here's an example that picks random proxies from a set of working proxies:
You can then change the "get" command to select a proxy when you can't find any present ones.
With this step, the "proxy" value becomes unnecessary. Use it to test the first proxies like before.
Afterwards, you can leave the list and use "get" without adding it. You'll use a random proxy. It will go to the not_working list if it does not work.
If you're web scraping, you need to scrape actual content. To do this, you need to send back the reply. You can also apply for an exception.
Here's the last version:
You can include the data you need to collect below this script. You can also call the testing URL for a trial.
Another thing to do is make a customized parser using Beautifulsoup. Below is the final code on rotating proxies in Python:
Re-testing the failed proxies is necessary as it lets you discover more working proxies for several reasons.
Python lets you use the function "Timers" for this step. This action runs at a specific time. It has various methods, but the result is the same. You can do this step with only three lines.
With the reset_proxy command, you can set up a timer and run it on every proxy that's not working.
Set it to around 20 seconds. It may sound too quick, but it's enough for this test. Then, you can remove a non-working proxy by returning it to uncheck afterward.
Here's an example:
There are two main types of proxies: static and rotating. Static proxies use one IP address, usually from a data center, for as long as you want. This is good for consistency. Rotating proxies, however, provides more anonymity and variety by switching IP addresses.
Meanwhile, a rotating proxy lets you use an entire proxy list or pool of IP addresses. These addresses change at random or fixed intervals, depending on how you set them.
Take a closer look at the similarities and differences between these two:
Both static and rotating proxies serve the same purpose.
They mask your actual IP address. These proxies enhance your online anonymity for a better browsing experience.
Rotating and static proxies can also let you use shared IPs, depending on the exclusivity you pick for the latter.
With a static proxy, you can use a single IP address for specific purposes. It is generally cheaper since you’re not paying for an entire proxy pool.
However, static proxies are more susceptible to IP bans. This issue usually happens when you make too many requests to a particular server.
With rotating proxies, you can enjoy higher security and anonymity. The chances of being banned are less since you'll use a different IP for every minute or every request.
The only drawback to rotating proxies is that there’s a chance that you can use IPs from unrelated geolocations.
✅ Pro Tip
While rotating proxies in Python seems like a lot of work, doing so comes with many benefits. Below are some advantages you'll enjoy when using rotating proxies:
Most websites track an IP address, which can lead to identity theft.
With a rotating proxy, these trackers will stay closed since you use a different IP with every request. This lets you enjoy better security and anonymity while browsing.
Since you'll be using a proxy pool of IP addresses, you can send unique requests every time.
When making multiple requests to modern websites, you’re more likely to encounter anti-bots like CAPTCHA. However, a rotating proxy can help you avoid getting banned since it reflects different IPs for each request.
You can split your connections with a rotating proxy. This allows you to connect with different geo-restricted websites at the same time.
Rotating proxies help scrape large websites. Using a different IP for every request lets you remain under the radar and avoid restrictions.
Rotating proxies can help you raise conversion rates since every request you make comes from a different IP address.
🗒️ Helpful Article:
A proxy server acts as a bridge between users and the internet. When considering Proxy Server Services, key factors are usability, features, speed, cost, and assistance. We offer a guide on top proxy server services for selecting the right paid or free proxy, ensuring secure internet access.
A rotating proxy lets you make multiple requests to a target without getting banned since you're using a unique IP address per request.
Using the steps above, rotating proxies in Python is now doable. It seems technical initially, but keep in mind that rotating your proxies will help you with any complex tasks
Rotating proxies is legal in most countries. It's commonly used to request HTML and other data from a website.
A static or sticky proxy is best for simple tasks like scraping social media or buying from a particular region. A rotating proxy is best for complex or large projects like web scraping or data parsing.
Rotate in Python is a method of rotation that goes by the code "rotate()". In this method, the deque class rotates the sequence in the deque object 'n' times.
Your email address will not be published.
Updated · Oct 02, 2023