Add beautiful soup 4 dependency with example test

[sbland]

Description

Implementing beautiful soup to allow HTML testing of views.

Fixes #593

Type of change

• Documentation (non-breaking change that adds or improves the documentation)
• New feature (non-breaking change which adds functionality)
• Optimization (non-breaking, back-end change that speeds up the code)
• Technical work (non-breaking, change which is work as part of a new feature)
• Bug fix (non-breaking change which fixes an issue)
• Breaking change (whatever its nature)

Key checklist

• All tests pass (eg. python -m pytest)
• The documentation builds and looks OK (eg. mkdocs serve)
• Pre-commit hooks run successfully (eg. pre-commit run --all-files)

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added or an issue has been opened to tackle that in the future. (Indicate issue here: # (issue))

[sbland]

One issue I'm having is beautiful soup not being able to find elements unless we strip the text. See util function implemented below to get around this:

def find_stripped_text(doc, tag, text):
    """Helper function to find the stripped text of a tag."""
    return next(
        a
        for a in doc.find_all(tag)
        if re.search(rf"{text}", a.get_text(" ", strip=True))
    )

[sbland]

@AdrianDAlessandro I've added some tests here trying out user driven testing but I need to dig into how the other tests are working for mocking the db etc.

[AdrianDAlessandro]

The DB is currently mocked using the pytest fixtures in main/conftest.py. This might change if we move to using data migrations to populate the DB (see #598). You also need to mark the tests that are going to use the DB with @pytest.mark.django_db.

What might be the best approach here is to use a class-based test that has a setup method that spins up and populates the database and then mark that class as using the DB, then all subsequent tests inherit from that class.

[AdrianDAlessandro]

I like the idea of implementing some user-driven testing, although I think maybe it should be in a separate PR to the unit tests on individual views. It is also lower priority, particularly because I think we might be changing some of the user workflows (like the self-assessment form, etc).

(FYI, I'm going to mark this PR as a draft for now)

[AdrianDAlessandro]

The DB is currently mocked using the pytest fixtures in main/conftest.py. This might change if we move to using data migrations to populate the DB (see #598). You also need to mark the tests that are going to use the DB with @pytest.mark.django_db.

What might be the best approach here is to use a class-based test that has a setup method that spins up and populates the database and then mark that class as using the DB, then all subsequent tests inherit from that class.

[AdrianDAlessandro]

I presume this print is temporary for while debugging?

[AdrianDAlessandro]

I think the point of using BS4 is to not need to do any regex. Looking at the example PR I linked in the issue, they needed to put IDs on many of the html objects in order to test them. So maybe that gets around this?