How to create a fullstack application using Django and Python Part 16

Sunday, September 15, 2024 at 8:48 PM | 6 min read

Last modified on Monday, May 25, 2026 at 12:14 PM

#fullstack development, #macOS, #django, #beautifulsoup4, #soupsieve, #html5lib, #html parser, #python3, #series, #tests, #unittest

Beautiful Soup

Photo by Ella Olsson on unsplash.com

Important Note: Before committing anything to Git or pushing anything to remote, please visit How to create a fullstack application using Django and Python Part 4 where I discuss how to add the python-dotenv package to the Django site and why it is crucial to do it. This article assumes you have a working knowledge of Git.

Table of Contents

Adding Test Client using Soup Sieve

As I mentioned in the section 15, there is a way of more generically verifying HTML inputs in templates/signup.html. We can do that with a package called Soup Sieve.

Soup Sieve is a powerful selector add-on for powerful Beautiful Soup 4. It comes pre-installed with BeautifulSoup4 4.7.0 and above, and uses the CSS selector API to query a document. In our case, it can be used to find the HTML inputs in our templates/signup.html and verify they exist there. This time we won't need to pass in complete inputs to the .assertContains() method. That was a bit of an inconvenient and long-winded approach.

Installing Beautiful Soup 4, Soup Sieve, and html5lib HTML parser

We have to install Beautiful Soup 4 before installing Soup Sieve. In fact, it's meant to be used alongside Beautiful Soup 4:

# first:
python3 pip -m install beautifulsoup4
# then:
python3 -m pip install soupsieve
# then (very important):
python3 -m pip install html5lib # needed for parsing and has to be installed separately

We also need to install html5lib. It parses the HTML that Beautiful Soup/Soup Sieve scrapes (that's what they do!).

Make sure to install all three packages at the root directory of the project (in my case django-boards directory) where the venv directory resides. Also make sure that your virtual environment has been activated before installing the packages. Then add bs4, soupsieve, and html5lib to INSTALLED_APPS in settings.py:

INSTALLED_APPS = [
    ...
    'soupsieve',
    'bs4',
    'html5lib',
]

Creating our Test Client using Beautiful Soup 4 and Soup Sieve

First, we will comment out our existing test_form_inputs in accounts/tests/test_signup_tests.py. Instead of relying on the generic test we originally had (although you could go that way if you want), we could inspect our signup.html template in the browser when we go to the url http://127.0.0.1:8000//signup/. All input attributes are in there, and much easier to read!

What I did was get all the input ids and pass them in like the following:

# add following imports first in accounts/tests/test_signup_tests.py:
import bs4
import soupsieve as sv

# then add the refactored test_form_inputs test:
def test_form_inputs(self):
    """
    The view must contain five inputs: csrf, username, email, password1, password2
    """
    self.assertContains(self.response, "<input", 7)

    # add the refactored test_form_inputs test:
    self.response = self.client.get(reverse("signup"))
    text = """
    <form method="post" novalidate="" class="signup-form">
        <input type="hidden" name="csrfmiddlewaretoken" value="5bzfyc9iidGoyInd3IYNlTrBGVLNVo09hNqsSjydsbrvupjtRELqgD8siJf94pup">

        <div class="form-group">
            <label for="id_username">Username:</label>
            <input type="text" name="username" maxlength="150" autofocus="" class="form-control " required="" aria-describedby="id_username_helptext" id="id_username" data-np-intersection-state="visible">

            <small class="form-text text-muted">
                Required. 150 characters or fewer. Letters, digits and @/./+/-/_ only.
            </small>
        </div>

        <div class="form-group">
            <label for="id_email">Email:</label>
            <input type="email" name="email" maxlength="254" class="form-control " required="" id="id_email" data-np-intersection-state="visible">
        </div>

        <div class="form-group">
            <label for="id_password1">Password:</label>
            <input type="password" name="password1" autocomplete="new-password" class="form-control " aria-describedby="id_password1_helptext" id="id_password1" data-np-intersection-state="visible">


            <small class="form-text text-muted">
                <ul><li>Your password can’t be too similar to your other personal information.</li><li>Your password must contain at least 8 characters.</li><li>Your password can’t be a commonly used password.</li><li>Your password can’t be entirely numeric.</li></ul>
            </small>

        </div>

        <div class="form-group">
            <label for="id_password2">Password confirmation:</label>
            <input type="password" name="password2" autocomplete="new-password" class="form-control " aria-describedby="id_password2_helptext" id="id_password2" data-np-intersection-state="visible">


            <small class="form-text text-muted">
                Enter the same password as before, for verification.
            </small>

        </div>

        <div class="form-group">
            <label>Password-based authentication:</label>
            <div id="id_usable_password" class="form-control "><div>
            <label for="id_usable_password_0"><input type="radio" name="usable_password" value="true" class="form-control " id="id_usable_password_0" checked="">
        Enabled</label>

        </div>
        <div>
            <label for="id_usable_password_1"><input type="radio" name="usable_password" value="false" class="form-control " id="id_usable_password_1">
        Disabled</label>

        </div>
        <div>
            <small class="form-text text-muted">
                Whether the user will be able to authenticate using a password or not. If disabled, they may still be able to authenticate using other backends, such as Single Sign-On or LDAP.
            </small>
        </div>

        <button type="submit" class="btn btn-primary btn-block">Create an account</button>
    </form>
    """
    soup = bs4.BeautifulSoup(text, "html5lib")
    sv.select(
            "form:is(.signup-form)",
            soup,
    )
    print(
        sv.select(
            "form:is(.signup-form)",
            soup,
        )
    )

You might also notice that I did not have to import the html5lib library. I only had to add it to INSTALLED_APPS in settings.py.

Breaking down the test_form_inputs test

  1. First we do as we have done to date. We get the "signup" view with self.response = self.client.get(reverse('signup')).
  2. Then we create a text object which contains the value of the element we want to target and the class it contains. I added the class beforehand. Soup Sieve/BeautifulSoup needs to be able to find a class, or an element selector, etc., in order to be able to parse the correct HTML. I wanted to be and needed to be more specific than the form element selector, so I chose signup-form as the class selector. It clearly describes its target element. All I needed to add was the form element, because it contains all the inputs we want to grab and verify in the test.
  3. Then I create a soup object (required) and set it to bs4.BeautifulSoup(text, 'html5lib') (also required in order to work), followed by sv.select() where I select the .signup-form class of the signup form so that I can view the entire form when I run my tests. The other thing I include is the line self.assertContains(self.response, "<input", 7). This tells me that I am looking for 7 inputs. And since there actually are 7 inputs in the signup form in my case, the entire test_form_inputs function passes. I print out the results of sv.select( "form:is(.signup-form)", soup, ) to see what actually returns from sv.select("form:is(.signup-form)", soup,). And that is it!

Finding all the input tags in the signup.html template using Beautiful Soup 4

I could also find just the input tags inside the signup.html template form using Beautiful Soup 4:

# place at the bottom of the test_form_inputs function
for tag in soup.find_all('input'):
    print(tag)

Which, for me, returned the following in Terminal:

<input name="csrfmiddlewaretoken" type="hidden" value="5bzfyc9iidGoyInd3IYNlTrBGVLNVo09hNqsSjydsbrvupjtRELqgD8siJf94pup"/> input tag
<input aria-describedby="id_username_helptext" autofocus="" class="form-control" data-np-intersection-state="visible" id="id_username" maxlength="150" name="username" required="" type="text"/> input tag
<input class="form-control" data-np-intersection-state="visible" id="id_email" maxlength="254" name="email" required="" type="email"/> input tag
<input aria-describedby="id_password1_helptext" autocomplete="new-password" class="form-control" data-np-intersection-state="visible" id="id_password1" name="password1" type="password"/> input tag
<input aria-describedby="id_password2_helptext" autocomplete="new-password" class="form-control" data-np-intersection-state="visible" id="id_password2" name="password2" type="password"/> input tag
<input checked="" class="form-control" id="id_usable_password_0" name="usable_password" type="radio" value="true"/> input tag
<input class="form-control" id="id_usable_password_1" name="usable_password" type="radio" value="false"/>

This also proves (in a different way) that all 7 inputs located in the signup form actually do exist, and it also confirms that there ARE 7 of them.

Running the accounts tests

After we complete creating the test, we can run the accounts tests:

python3 manage.py test accounts

The following is returned:

======================================================================
FAIL: test_csrf (accounts.tests.test_view_password_reset.PasswordResetConfirmTests.test_csrf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mariacam/Python-Development/django-boards/django_boards/accounts/tests/test_view_password_reset.py", line 110, in test_csrf
    self.assertContains(self.response, 'csrfmiddlewaretoken')
  File "/Users/mariacam/.pyenv/versions/3.12.5/lib/python3.12/site-packages/django/test/testcases.py", line 623, in assertContains
    self.assertTrue(
AssertionError: False is not true : Couldn't find 'csrfmiddlewaretoken' in the following response
b'\n<!DOCTYPE html>\n<html lang="en">\n  <head>\n    <meta charset="utf-8">\n    <meta name="description" content="A forum dedicated to all things Django" />\n    <meta name="keywords" content="django, python3" />\n    <title>\n      \n  \n    Reset your password\n  \n\n    </title>\n    <link rel="stylesheet" href="/static/css/bootstrap.min.css">\n    <link rel="stylesheet" href="/static/css/app.css">\n    \n  <link rel="stylesheet" href="/static/css/accounts.css">\n\n  </head>\n  <body>\n    \n  <div class="container">\n    <h1 class="text-center logo my-4">\n      <a href="/">Django Boards</a>\n    </h1>\n    \n  <div class="row justify-content-center">\n    <div class="col-lg-6 col-md-8 col-sm-10">\n      <div class="card">\n        <div class="card-body">\n          \n            <h3 class="card-title">Reset your password</h3>\n            <div class="alert alert-danger" role="alert">\n              It looks like you clicked on an invalid password reset link. Please try again.\n            </div>\n            <a href="/password-reset/" class="btn btn-secondary btn-block">Request a new password reset link</a>\n          \n        </div>\n      </div>\n    </div>\n  </div>\n\n  </div>\n\n    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/2.9.2/umd/popper.min.js"\n            integrity="sha512-2rNj2KJ+D8s1ceNasTIex6z4HWyOnEYLVC3FigGOmyQCZc2eBXKgOxQmo3oKLHyfcj53uz4QMsRCWNbLd32Q1g=="\n            crossorigin="anonymous"\n            referrerpolicy="no-referrer"></script>\n    <script src="https://code.jquery.com/jquery-3.7.1.min.js"\n            integrity="sha256-/JqT3SQfawRcv/BIHPThkBvs0OEvtFFmqPF/lYI/Cxo="\n            crossorigin="anonymous"></script>\n    <script src="https://cdnjs.cloudflare.com/ajax/libs/bootstrap/5.3.3/js/bootstrap.min.js"\n            integrity="512-ykZ1QQr0Jy/4ZkvKuqWn4iF3lqPZyij9iRv6sGqLRdTPkY69YX6+7wvVGmsdBbiIfN/8OdsI7HABjvEok6ZopQ=="\n            crossorigin="anonymous"\n            referrerpolicy="no-referrer"></script>\n  </body>\n</html>\n'

======================================================================
FAIL: test_form_inputs (accounts.tests.test_view_password_reset.PasswordResetConfirmTests.test_form_inputs)
The view must contain two inputs: csrf and two password fields
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mariacam/Python-Development/django-boards/django_boards/accounts/tests/test_view_password_reset.py", line 122, in test_form_inputs
    self.assertContains(self.response, '<input', 3)
  File "/Users/mariacam/.pyenv/versions/3.12.5/lib/python3.12/site-packages/django/test/testcases.py", line 614, in assertContains
    self.assertEqual(
AssertionError: 0 != 3 : Found 0 instances of '<input' (expected 3) in the following response
b'\n<!DOCTYPE html>\n<html lang="en">\n  <head>\n    <meta charset="utf-8">\n    <meta name="description" content="A forum dedicated to all things Django" />\n    <meta name="keywords" content="django, python3" />\n    <title>\n      \n  \n    Reset your password\n  \n\n    </title>\n    <link rel="stylesheet" href="/static/css/bootstrap.min.css">\n    <link rel="stylesheet" href="/static/css/app.css">\n    \n  <link rel="stylesheet" href="/static/css/accounts.css">\n\n  </head>\n  <body>\n    \n  <div class="container">\n    <h1 class="text-center logo my-4">\n      <a href="/">Django Boards</a>\n    </h1>\n    \n  <div class="row justify-content-center">\n    <div class="col-lg-6 col-md-8 col-sm-10">\n      <div class="card">\n        <div class="card-body">\n          \n            <h3 class="card-title">Reset your password</h3>\n            <div class="alert alert-danger" role="alert">\n              It looks like you clicked on an invalid password reset link. Please try again.\n            </div>\n            <a href="/password-reset/" class="btn btn-secondary btn-block">Request a new password reset link</a>\n          \n        </div>\n      </div>\n    </div>\n  </div>\n\n  </div>\n\n    <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/2.9.2/umd/popper.min.js"\n            integrity="sha512-2rNj2KJ+D8s1ceNasTIex6z4HWyOnEYLVC3FigGOmyQCZc2eBXKgOxQmo3oKLHyfcj53uz4QMsRCWNbLd32Q1g=="\n            crossorigin="anonymous"\n            referrerpolicy="no-referrer"></script>\n    <script src="https://code.jquery.com/jquery-3.7.1.min.js"\n            integrity="sha256-/JqT3SQfawRcv/BIHPThkBvs0OEvtFFmqPF/lYI/Cxo="\n            crossorigin="anonymous"></script>\n    <script src="https://cdnjs.cloudflare.com/ajax/libs/bootstrap/5.3.3/js/bootstrap.min.js"\n            integrity="512-ykZ1QQr0Jy/4ZkvKuqWn4iF3lqPZyij9iRv6sGqLRdTPkY69YX6+7wvVGmsdBbiIfN/8OdsI7HABjvEok6ZopQ=="\n            crossorigin="anonymous"\n            referrerpolicy="no-referrer"></script>\n  </body>\n</html>\n'

----------------------------------------------------------------------
Ran 46 tests in 10.107s

FAILED (failures=2)
Destroying test database for alias 'default'...

I am addressing these failures and will post the fixes in a future post when I have figured them out.

Conclusion

In this section, I demonstrated an addition to the original test_form_inputs test in which we used the .assertContains() method to verify the existence of specific inputs residing in the signup form. I installed Beautiful Soup 4 and Soup Sieve along with the html5lib HTML parser, and then re-created the test_form_inputs test using those packages. The test is much shorter, and my last approach to discovering the ids of the inputs I wanted to grab by iterating over the signup form input elements is quicker and much easier to read!