How To Get an href Attribute Using BeautifulSoup?

Out of 1.11 billion websites worldwide, nearly 95% predominantly use HTML. The interconnection between all websites is facilitated through linking. To make the linking process possible, an element known asthehref attribute plays a significant role.

The href attribute facilitates the connection of clickable links. It specifies the URL of a linked resource using a hyperlink. Href attributes make retrieving valuable information easier, leading to more efficient and accurate data extraction. 

This article covers a step-by-step guide on how to get HTML href using the bs4 BeautifulSoup. Read on. 

What is an href Attribute?

The hypertext reference attribute (or href attribute) creates a clickable hyperlink. It indicates the anchor text destination leading to a functional hyperlink on any webpage. 

Publishing webpages and websites means dealing with intricate sets of codes. To insert a clickable hyperlink, you must use the format below: 

<a href="Insert link here"> Insert text here </a>

Here’s an example of how it should look like:

<a href="https://techjury.net/scraping/"> Techjury | Techniques for Data Scraping</a>

The code will produce the following output: 

Without an href attribute, the output will appear as plain text. It will look like this: 

Href attributes specify the destination of a hyperlink, making it seamless to navigate from one webpage to another. The lack of href attributes affects the user experience on the website. 

What You Need to Extract href Attributes

Before you start scraping the href attribute, download and install the following prerequisites:

	Python –  the programming language commonly used to code and automate website data extraction. This guide uses Python v3.9.6.
	Python libraries – collections of codes for particular tasks or functions. Some libraries are not pre-built, so you must install them manually.  To get href attributes, you will need these two Python libraries: BeautifulSoup (version 4) – for extracting and parsing HTML and XML documents. Requests – used for HTTP client-like requests.
	Code editor or IDE – an application for writing or developing code. This guide uses Visual Studio Code, but you can choose any code editor.

Installation and Verification 

Securing the requirements is the first step in getting href attributes. Follow the steps below to install the prerequisites: 

Python Installation

You can easily download Python from its official website. Once you’ve installed it, run the following command to check the Python version:

The output should display the Python version you just installed. Example: 

PIP Installation

Python below version 3 may not include PIP upon installation. You must install PIP manually. To do so, you can run the following commands:

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py python get-pip.py

Verify if the installation is successful by running the command:

pip --version

The command will return a value indicating the PIP version installed on the machine. For example:

pip 23.2.1 from c:\users\appdata\local\programs\python\python39\lib\site-packages\pip (python 3.9)

BeautifulSoup Installation Using PIP

To get the href from HTML, install BeautifulSoup must come first. You can either use CMD or VS Code. You can do it by running this command: 

pip install beautifulsoup4

Verify the BeautifulSoup version you installed using:

pip list

The return value is a list of packages with their version. You should see a version with a 4.x.x. format. Example: 

Package                     Version
    ------------------          -----------   

beautifulsoup4              4.12.2

How To Get href From an <a> Tag in BeautifulSoup?

BeautifulSoup has different methods to find and extract HTML elements. In extracting anchor tags <a> containing href attributes, there are two ways to do so: find() and find_all().

Find out how each method works below. 

Method 1: find() 

The find() method locates the first matching element that meets the specified criteria. It will search through the first anchor tag with the href attribute.

Here are the steps on how to get href attributes withfind():

1. Import the BeautifulSoup library from the bs4 package.
   from bs4 import BeautifulSoup
2. Define the HTML content using this format:

<a href="URL"> Clickable text or content </a>

Example: 

html = '''<a href="https://techjury.net/scraping/"> Techjury | Techniques for Data Scraping</a>'''

3. Create and parse the BeautifulSoup object. 

soup = BeautifulSoup(html, 'html.parser')

4. Extract href attributes.

link = soup.find('a')

5. Extract the href attribute value using the get() command. 

href_att = link.get('href')

6. Display the href attribute using the print function.
   

print("href:", href_att)

Consolidate all the steps. Below is the final code on how to get the href attribute using the find() method:

from bs4 import BeautifulSoup html = '''<a href="https://techjury.net/scraping/"> Techjury | Techniques for Data Scraping</a>''' soup = BeautifulSoup(html, 'html.parser') link = soup.find('a') href_att = link.get('href') print("href:", href_att)

Method 2: find_all() 

The find_all() method returns a list of objects within the webpage. It gets all the anchor tags and their href attributes from the HTML content.

Follow the steps below to collect href using find_all(): 

1. Get the BeautifulSoup library from the bs4 package.

from bs4 import BeautifulSoup

2. Import an HTTP client to get the HTML content behind the URL and feed it to BeautifulSoup.
   import requests
3. Define the link you want to scrape.

url = "https://techjury.net/scraping/"

4. Extract the list of URLs using the get() command. 

req = requests.get(url)

5. Create and parse the BeautifulSoup object. 

soup = BeautifulSoup(req.text, "html.parser")

6. Return a list of all matching elements. With a for loop, it scans the entire document.

for link in soup.find_all('a')

7. Display all the HTML href links to the console or terminal.

print(link.get('href'))

The last step is to consolidate all the previous steps. Here’s a full view of how to get the href attribute using the find_all() method:

from bs4 import BeautifulSoup import requests url = "https://techjury.net/scraping/" req = requests.get(url) soup = BeautifulSoup(req.text, "html.parser") print("href links are as follows:") for link in soup.find_all('a'):     print(link.get('href'))

Conclusion

The href attribute makes linking more seamless and provides users with easier navigation over billions of websites. Other than that, it lets you scrape valuable data since it contains the full address of the destination page. 

Scraping href attributes with Python is a straightforward process. Python has a package library specifically used for web scraping—BeautifulSoup. With BeautifulSoup, extracting href attributes only requires minimal coding.

FAQs

By Muninder Adavelli

Muninder Adavelli is a core team member and Digital Growth Strategist at Techjury. With a strong background in marketing and a deep understanding of technology's role in digital marketing, he brings immense value to the TechJury team.