381

I'm parsing some HTML with Beautiful Soup 3, but it contains HTML entities which Beautiful Soup 3 doesn't automatically decode for me:

>>> from BeautifulSoup import BeautifulSoup

>>> soup = BeautifulSoup("<p>&pound;682m</p>")
>>> text = soup.find("p").string

>>> print text
&pound;682m

How can I decode the HTML entities in text to get "£682m" instead of "&pound;682m".

1

7 Answers 7

Reset to default
725

Python 3.4+

Use html.unescape():

import html
print(html.unescape('&pound;682m'))

FYI html.parser.HTMLParser.unescape is deprecated, and was supposed to be removed in 3.5, although it was left in by mistake. It will be removed from the language soon.

Python 2.6-3.3

You can use HTMLParser.unescape() from the standard library:

>>> try:
...     # Python 2.6-2.7 
...     from HTMLParser import HTMLParser
... except ImportError:
...     # Python 3
...     from html.parser import HTMLParser
... 
>>> h = HTMLParser()
>>> print(h.unescape('&pound;682m'))
£682m

You can also use the six compatibility library to simplify the import:

>>> from six.moves.html_parser import HTMLParser
>>> h = HTMLParser()
>>> print(h.unescape('&pound;682m'))
£682m
7
  • 9
    this method doesn't seem to escape characters like "&#8217;" on google app engine, though it works locally on python2.6. It does still decode entities (like &quot;) at least
    – gfxmonk
    Jul 10, 2010 at 14:40
  • How can an undocumented API be deprecated? Edited the answer.
    – Markus Unterwaditzer
    Jun 5, 2015 at 18:15
  • @MarkusUnterwaditzer there's no reason that an undocumented method can't be deprecated. This one throws deprecation warnings - see my edit to the answer.
    – Mark Amery
    Nov 25, 2015 at 15:06
  • 1
    Worth noting for Python 2: Special characters are replaced with their Latin-1 (ISO-8859-1) encoding counterparts. E.g., it may be necessary to h.unescape(s).encode("utf-8"). The docs: """The definition provided here contains all the entities defined by XHTML 1.0 that can be handled using simple textual substitution in the Latin-1 character set (ISO-8859-1)"""
    – anonymous coward
    Sep 5, 2018 at 15:03
  • 1
    It does not work for 'Don&‌#039;t forget that &‌pi; = 3.14 &‌amp; doesn&‌#039;t equal 3.' WHY is that?
    – canbax
    May 1, 2020 at 9:15
72

Beautiful Soup handles entity conversion. In Beautiful Soup 3, you'll need to specify the convertEntities argument to the BeautifulSoup constructor (see the 'Entity Conversion' section of the archived docs). In Beautiful Soup 4, entities get decoded automatically.

Beautiful Soup 3

>>> from BeautifulSoup import BeautifulSoup
>>> BeautifulSoup("<p>&pound;682m</p>", 
...               convertEntities=BeautifulSoup.HTML_ENTITIES)
<p>£682m</p>

Beautiful Soup 4

>>> from bs4 import BeautifulSoup
>>> BeautifulSoup("<p>&pound;682m</p>")
<html><body><p>£682m</p></body></html>
4
  • +1. No idea how I missed this in the docs: thanks for the info. I'm going to accept luc's answer tho because his uses the standard lib which I specified in the question (not important to me) and its probably of more general use to other people.
    – jkp
    Jan 18, 2010 at 16:23
  • 7
    BeautifulSoup4 uses HTMLParser, mostly. See the source
    – scharfmn
    Mar 3, 2015 at 7:53
  • 4
    How do we get the conversion in Beautiful Soup 4 without all the extraneous HTML that wasn't part of the original string? (i.e. <html> and <body>)
    – Praxiteles
    Jun 10, 2017 at 4:13
  • @Praxiteles : BeautifulSoup('&pound;682m', "html.parser") stackoverflow.com/a/14822344/4376342
    – Soitje
    Apr 20, 2018 at 10:51
16

You can use replace_entities from w3lib.html library

In [202]: from w3lib.html import replace_entities

In [203]: replace_entities("&pound;682m")
Out[203]: u'\xa3682m'

In [204]: print replace_entities("&pound;682m")
£682m
0
8

Beautiful Soup 4 allows you to set a formatter to your output

If you pass in formatter=None, Beautiful Soup will not modify strings at all on output. This is the fastest option, but it may lead to Beautiful Soup generating invalid HTML/XML, as in these examples:

print(soup.prettify(formatter=None))
# <html>
#  <body>
#   <p>
#    Il a dit <<Sacré bleu!>>
#   </p>
#  </body>
# </html>

link_soup = BeautifulSoup('<a href="http://example.com/?foo=val1&bar=val2">A link</a>')
print(link_soup.a.encode(formatter=None))
# <a href="http://example.com/?foo=val1&bar=val2">A link</a>
2
  • This doesn't answer the question. (Also, I have no idea what the docs are saying is invalid about the final bit of HTML here.)
    – Mark Amery
    Nov 29, 2015 at 14:52
  • <<Sacré bleu!>> is the invalid part, as it has unescaped < and > and will break the html around it. I know this is a late post from me, but in case anyone happens to be looking and wondered...
    – GMasucci
    Apr 22, 2016 at 15:49
1

I had a similar encoding issue. I used the normalize() method. I was getting a Unicode error using the pandas .to_html() method when exporting my data frame to an .html file in another directory. I ended up doing this and it worked...

    import unicodedata

The dataframe object can be whatever you like, let's call it table...

    table = pd.DataFrame(data,columns=['Name','Team','OVR / POT'])
    table.index+= 1

encode table data so that we can export it to out .html file in templates folder(this can be whatever location you wish :))

     #this is where the magic happens
     html_data=unicodedata.normalize('NFKD',table.to_html()).encode('ascii','ignore')

export normalized string to html file

    file = open("templates/home.html","w") 

    file.write(html_data) 

    file.close()

Reference: unicodedata documentation

1
  • This does not answer the question. A Unicode normalization form is not the same as HTML entities.
    – Adrian W
    Sep 7, 2023 at 21:22
0 
import html
  
myHtml = "<body><h1> How to use html.unescape() in Python </h1></body>"
encodedHtml = html.escape(myHtml)
print("Encoded HTML: ", encodedHtml)
decodedHtml = html.unescape(encodedHtml)
  
print("Decoded HTML: ", decodedHtml)

Output:

Encoded HTML:  &lt;body&gt;&lt;h1&gt; How to use html.unescape() in Python &lt;/h1&gt;&lt;/body&gt;
Decoded HTML:  <body><h1> How to use html.unescape() in Python </h1></body>

Demo

-5

This probably isnt relevant here. But to eliminate these html entites from an entire document, you can do something like this: (Assume document = page and please forgive the sloppy code, but if you have ideas as to how to make it better, Im all ears - Im new to this).

import re
import HTMLParser

regexp = "&.+?;" 
list_of_html = re.findall(regexp, page) #finds all html entites in page
for e in list_of_html:
    h = HTMLParser.HTMLParser()
    unescaped = h.unescape(e) #finds the unescaped value of the html entity
    page = page.replace(e, unescaped) #replaces html entity with unescaped value
1
  • 10
    No! You don't need to match HTML entities yourself and loop over them; .unescape() does that for you. I don't understand why you and Rob have posted these overcomplicated solutions that roll their own entity matching when the accepted answer already clearly shows that .unescape() can find entities in the string.
    – Mark Amery
    Nov 29, 2015 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.