Using emojis in matplotlib

Last month, as I was writing the blog post with all the statistics for the blog's two year anniversary, the "Blog statistics after two years" post, I decided that I really wanted to have a plot with emojis. From the moment I thought of this I knew it couldn't simply work and I was in for some fun.

When I first tried generating my plot with emojis, this is how it came out:

These emojis just have the outline and some are even missing. I wanted matplotlib to use a proper emoji font, like NotoColorEmoji or twemoji, which are colorful and look nice. I had both installed on my system but they weren't being automatically picked up by matplotlib.

After a bit of searching I figured out how to explicitly add the font to matplotlib:

matplotlib.font_manager.fontManager.addfont("/usr/share/fonts/noto/NotoColorEmoji.ttf")

And I also added fontname="noto color emoji" to the matplotlib call that should draw using this font (in my case xticks()).

By forcing matplotlib to use the emoji font I wanted, I no longer got those outlined emojis, in fact I didn't get any image at all 😝, just this traceback:

 Traceback (most recent call last):
   File "/home/nfraprado/emoji.py", line 3, in <module>
     matplotlib.font_manager.fontManager.addfont('/usr/share/fonts/noto/NotoColorEmoji.ttf')
   File "/usr/lib/python3.10/site-packages/matplotlib/font_manager.py", line 1092, in addfont
     font = ft2font.FT2Font(path)
 RuntimeError: In FT2Font: Could not set the fontsize (error code 0x17)

With an error code even, inviting me to dig deeper 😆. Since I'm not familiar with the inner workings of fonts, the first thing was to figure out what FT2Font was. And it turned out to be matplotlib's wrapper for FreeType, the library that renders the fonts.

The traceback error message was coming from this line, which meant that FT_Set_Char_Size(), which is a freetype function, was failing with error code 0x17. A look into freetype's error code reference revealed that it meant "invalid pixel size".

As I tried to figure out what differed the emoji fonts from normal fonts, I did notice something that seemed to be fundamental: Running fc-scan on the emoji fonts revealed that they had a pixelsize property which wasn't present in normal fonts. Furthermore, running ftview (from the freetype2-demos package) on the emoji fonts with any size resulted in the emojis being drawn in the same size, while normal fonts were correctly scaled.

At this point I needed to actually dig into the code, so I cloned the source for matplotlib and freetype, compiled, and set the environment so I could use them: installed matplotlib in a virtual environment with pip install -e . and pointed the LD_LIBRARY_PATH environment variable to the directory containing the build output from freetype.

After enabling debug logs in freetype (with FT2_DEBUG=any:5) and adding some logs of my own, I noticed that the difference in code run for the emoji fonts was due to FT_HAS_FIXED_SIZES being true, whose meaning can be seen here.

To sum up the issue here (to my understanding): emoji fonts are made by embedding bitmaps, which have a specific size. This is indicated by FT_HAS_FIXED_SIZES being true, and the size of these bitmaps shows up as the pixelsize attribute. When matplotlib is going to draw the font, it specifies the size it wants to render the font in, but since the emoji font has a fixed size, the freetype code expects the size passed in to be the same as the font size. This is because you can have multiple versions of the bitmaps, with different sizes, inside the same font, so freetype would pick the one with the size that was asked for.

With that in mind, I came up with this patch:

diff --git a/src/base/ftobjs.c b/src/base/ftobjs.c
index f66273f3d3cb..a59a61119702 100644
--- a/src/base/ftobjs.c
+++ b/src/base/ftobjs.c
@@ -3074,6 +3074,7 @@
     FT_Int   i;
     FT_Long  w, h;

+    ignore_width = 1;

     if ( !FT_HAS_FIXED_SIZES( face ) )
       return FT_THROW( Invalid_Face_Handle );
@@ -3101,8 +3102,6 @@
       FT_Bitmap_Size*  bsize = face->available_sizes + i;


-      if ( h != FT_PIX_ROUND( bsize->y_ppem ) )
-        continue;

       if ( w == FT_PIX_ROUND( bsize->x_ppem ) || ignore_width )
       {

What this patch does is force freetype to return the first bitmap option inside the font, even if it doesn't have the size we asked for.

With that change done, I then got a different error, now in matplotlib:

Traceback (most recent call last):
  File "/home/nfraprado/emoji.py", line 3, in <module>
    matplotlib.font_manager.fontManager.addfont('/usr/share/fonts/noto/NotoColorEmoji.ttf')
  File "/home/nfraprado/matplotlib/lib/matplotlib/font_manager.py", line 1134, in addfont
    prop = ttfFontProperty(font)
  File "/home/nfraprado/matplotlib/lib/matplotlib/font_manager.py", line 545, in ttfFontProperty
    raise NotImplementedError("Non-scalable fonts are not supported")
NotImplementedError: Non-scalable fonts are not supported

So, only now is matplotlib complaining that the font isn't scalable, which should have been the error to begin with... Anyway, I was curious to see if this would all work in the end, so I just removed the check:

diff --git a/lib/matplotlib/font_manager.py b/lib/matplotlib/font_manager.py
index f57fc9c051b0..08d8c28752cb 100644
--- a/lib/matplotlib/font_manager.py
+++ b/lib/matplotlib/font_manager.py
@@ -541,8 +541,6 @@ def ttfFontProperty(font):
     #  Length value is an absolute font size, e.g., 12pt
     #  Percentage values are in 'em's.  Most robust specification.

-    if not font.scalable:
-        raise NotImplementedError("Non-scalable fonts are not supported")
     size = 'scalable'

     return FontEntry(font.fname, name, style, variant, weight, stretch, size)

And finally got some output:

The emojis are black and white, and not the right size, but it's a step forward. At this point, what I was seeing looked a similar to what I had seen in a blog post and the linked issue on Github while searching around. The issue there is about the TTC format, so not the same thing, but the workaround to use a different backend that has better support for emojis, mplcairo, did sound promising.

So I installed mplcairo with pip and set the script to use it with

matplotlib.use("module://mplcairo.gtk")

And this was the output:

So the emojis actually rendered colored and scaled correctly with this backend! The only caveat was that the text on the Y-axis was rotated, but a quick search revealed that this was already fixed on the main branch, so I reinstalled mplcairo from git, and everything was golden!

I did think this investigation was going to end up in some fix being sent to either the font, freetype, or matplotlib, but with my current understanding of the issue, I'm not entirely sure what would be the right fix here. I'd need to investigate this a bit further to know, but given that I've managed to get it working for my purposes, granted with some hacky patches, I don't see myself looking more into this for now.

That said, my understanding is that mplcairo does the font scaling on its own for fixed size (emoji) fonts, so maybe making mplcairo check the available font sizes and ask freetype for the one available that is closer to the size it'll draw with, could be a way to get rid of the freetype patch. As for the matplotlib patch, maybe fixed size fonts could be allowed when the mplcairo backend is being used, since it clearly knows how to handle them. But again, more investigation needed to conclude anything.

In any case, with the two patches shown here and the mplcairo backend, I was able to generate the plot with the most frequently used emojis at the end of the "Blog statistics after two years" post. The complete script that generated the plot can be seen here.

Finally, for reproducibility purposes, it's worth to say that all of this was tested with the packages in the following commits:

• mplcairo: 74c27c3dbd54 ("Tweak path search in build-windows-wheel.")
• matplotlib: 60ae76b6b5b0 ("Merge pull request #23243 from timhoffm/take-over-22839")
• freetype: e7482ff4c2a3 ("* src/lzw/ftzopen.c (ft_lzwstate_stack_grow): Cosmetic macro change.")