Looking at the error here:
soup = BeautifulSoup(html_content,features="lxml") File "/usr/share/offsec-awae-wheels/beautifulsoup4-4.9.1-py2-none-any.whl/bs4
especially the 4-4.9.1-py2 would IMO make me think that the script is using python2 and beautifulsoup for python version 2.
So this is what I would recommend doing:
Install pip2 (This page has moved - pip documentation v23.0)
and then do pip2 (or just pip) install lxml.
Let me know if it helps!
PS: you might need to also upgraded setuptools on pip2, you can do this with pip2 install --upgrade setuptools
Edit: another PS (lol!) I’d honestly recommend just learning python3, not that python2 is much different, just a few syntax differences, but it’s just that it’s EOL now and isn’t supported anymore.