Share your Jupyter Notebooks like a pro

Lucy Linder - Jan 18 '23 - - Dev Community

Are you doing data analysis with Jupyter Notebook? Here is how to create beautiful notebooks with interactive plots and host them as HTML on GitHub Pages.

I recently published an in-depth analysis of the top tags on dev.to:

dev.to is for webdevs and beginners - I have data to prove it

An in-depth data analysis of the trending articles and tags (count, reactions, comments) on dev.to.

favicon blog.derlin.ch

To back it up, I published the full analysis notebooks on GitHub Pages:

⮕ ✨ https://derlin.github.io/dev.to-is-for-web-devs-and-beginners/ ✨ (analysis.html is the main notebook)

If you visit the link above, you should see that:

  • The notebooks are available as responsive, mobile-friendly HTML pages

  • All the graphs are interactive: you can zoom and pan, view details on hover, export as a PNG, etc.

  • Some plots from the main notebook, analysis.html, are also available as standalone pages, for example, plot_articles.html

  • The root of the website contains an index of all the available files

  • (hidden, but trust me) The website is updated automatically on every push to the repository's main branch.

Let's see how you can do the same!

Table of Contents

Table of Contents generated with bitdowntoc! Read more in this article: Finally a clean and easy way to add Table of Contents to dev.to articles 🤩

Preface: about Jupyter Notebook

Jupyter Notebook is a powerful tool for interactively developing and presenting data science projects.

In short, a notebook is a single document where you can have code interspersed with their output, and also other media such as rich text, HTML, images, math equations, etc. If this doesn't ring a bell, have a look at https://www.dataquest.io/blog/jupyter-notebook-tutorial/.

There are lots of ways to install Jupyter Notebook, for example using JupyterLab or voila. I personally prefer the standalone, more lightweight version:

pip install notebook
jupyter notebook
  Serving notebooks from local directory: ...
  Jupyter Notebook 6.5.2 is running at:
   http://localhost:8888/?token=...
Enter fullscreen mode Exit fullscreen mode

This command starts a local web server where you can work on your notebooks.

Interactive plots: matplotlib → plotly

There are many plotting libraries for Python, the most well-known being matplotlib and its pyplot submodule. If you have followed a data sciences tutorial, chances are you have heard of it.

Matplotlib, however, generates plots as static images, and to be honest, I find the default styles quite unappealing: https://matplotlib.org/stable/gallery/style_sheets/style_sheets_reference.html.

This is why a switched to plotly years ago.

What is Plotly

Available in multiple languages (JavaScript, R, F#, Julia, and more), Plotly's graphing library makes fully interactive, publication-quality graphs. It comes with support for line plots, scatter plots, area charts, bar charts, error bars, box plots, histograms, heatmaps, subplots, multiple-axes, polar charts, bubble charts, and more. Cherries on the cake, it is free and open source 😉.

Plotly comes with a plotly.express submodule (usually imported as px) that you can use to generate plots from arrays or pandas DataFrame / Serie objects in a single command. If you need more control over the plot, you can switch to the plotly.graph_objects submodule, which is more low-level, and thus more powerful. I personally use both, depending on my needs.

It can be installed easily using pip:

pip install plotly
Enter fullscreen mode Exit fullscreen mode

Setting up Plotly

Let's first import some Plotly submodules:

# The easiest way to make Plotly plots:
import plotly.express as px
# Lower-level interface (if you want to test it):
import plotly.graph_objects as go
# For configuration (keep reading ;):
import plotly.io as pio
Enter fullscreen mode Exit fullscreen mode

The default template (aka style) is quite good already, but I personally prefer the white theme (see templates). To make it the default:

pio.templates.default = 'plotly_white'
Enter fullscreen mode Exit fullscreen mode

Plotly supports displaying figures in a variety of contexts. To do so, it uses something called the renderers framework. By default, Plotly attempts to autodetect an appropriate renderer at startup, but I had some surprises, especially during export. This is why I always explicitly set the default renderer:

# Let plotly know it runs inside a Jupyter Notebook
pio.renderers.default = 'notebook'
Enter fullscreen mode Exit fullscreen mode

If you edit your notebooks using other means, feel free to choose another renderer. At the time of writing, here are the available options:

print(pio.renderers)

Renderers configuration
-----------------------
    Default renderer: 'notebook'
    Available renderers:
        ['plotly_mimetype', 'jupyterlab', 'nteract', 'vscode',
         'notebook', 'notebook_connected', 'kaggle', 'azure', 'colab',
         'cocalc', 'databricks', 'json', 'png', 'jpeg', 'jpg', 'svg',
         'pdf', 'browser', 'firefox', 'chrome', 'chromium', 'iframe',
         'iframe_connected', 'sphinx_gallery', 'sphinx_gallery_png']
Enter fullscreen mode Exit fullscreen mode

We are all set. So let's create plots!

Plotting with Plotly

(I assume you are familiar with pandas and numpy. If this is not the case, check out Pandas: getting started and numpy: learn.

First, we need some random data:

import pandas as pd
import numpy as np

s = pd.Series(np.random.randn(100))
s.head()
0    1.292833
1    0.999066
2    1.335286
3   -0.657893
4    1.143734
dtype: float64
Enter fullscreen mode Exit fullscreen mode

Let's plot this meaningful (😜) Series using plotly.express:

px.line(s)
Enter fullscreen mode Exit fullscreen mode

Simple plot with Plotly

(This is a screenshot, but you can see the zoom/pan/drag, etc options on the top right, and the hover tooltip on a datapoint. For a real interactive plot, see plot_articles.html).

This looks good already, but there is always room for improvement. Let's customize a bit this plot:

fig = px.line(s)

fig.update_traces(dict(
    # Add the markers (dots)
    mode='lines+markers',
    # Customize the hover box
    # Use %{} to get the data value (supports formatting)
    # The empty <extra></extra> tag removes the part
    # of the hover displaying the trace name (variable: 0 above).
    hovertemplate='%{x} → %{y:.2f}<br>Nice hu?<extra></extra>'
))

fig.update_layout(
    title='Sample data',
    showlegend=False,
    yaxis=dict(title=None),
    xaxis=dict(title=None, tickmode='linear', dtick=5),
    margin=dict(l=0, b=0),
)
Enter fullscreen mode Exit fullscreen mode

This gives us:

Plotly plot with some layout customizations

And this is only the tip of the iceberg. You can spend hours just playing with all the options!

Other tips to make your notebook shine

Display Pandas DataFrame with style

Did you know that pandas provides methods to format and style a DataFrame? This is awesome when you want to make a table more appealing or highlight certain parts to convey information quickly.

Styling is controlled via the style attribute. Here is a simple example:

df.style
  .hide('index') # do not show the index
  .highlight_max(axis=0) # highlight the max value on each line
Enter fullscreen mode Exit fullscreen mode

pandas DataFrame with column-wise highlighted maximum value.

To learn more and play with styles, have a look at Essential Techniques to Style Pandas DataFrames on Kaggle.

Use alternate visualizations and rich content

Do not hesitate to fall back to other types of representations to spice up your notebooks, for example, show a WordCloud (I used it for the cover image of my article on dev.to tags). You can also embed rich content (audio, video, iframes, SVG, etc) thanks to the IPython.display module.

Exporting notebooks to HTML

Now that we have a nice notebook with interactive plots, how do we export it to HTML and keep the interactive parts?

Using nbconvert

We can use nbconvert to export to HTML:

jupyter nbconvert *.ipynb --to html
Enter fullscreen mode Exit fullscreen mode

As long as the proper renderer is set (pio.renderers.default = 'notebook'), this results in a very decent interactive webpage. On mobile, however, the responsiveness is a bit broken due to the cell numbering:

Suboptimal rendering of python notebook exported to HTML on mobile

Fortunately, Jupyter allows for custom styles, so we can fix it with a tiny bit of CSS.

Adding custom styles

To fix the responsiveness issue on mobile, create a jupyter folder with the following structure:

jupyter
├── custom
│   └── custom.css
Enter fullscreen mode Exit fullscreen mode

In the custom.css stylesheet, add:

/**
For this file to be processed by jupyter,
ensure you export JUPYTER_CONFIG_DIR, for example:
  JUPYTER_CONFIG_DIR=jupyter jupyter nbconvert --to html
*/

/** Make jupyter HTML export more responsive. */
@media (max-width: 600px) { 
    .jp-InputPrompt.jp-InputArea-prompt, 
    .jp-OutputPrompt.jp-OutputArea-prompt { 
        display: none !important;
    }
}
Enter fullscreen mode Exit fullscreen mode

Finally, ask Jupyter to add this custom stylesheet to all the HTML notebooks it converts by setting the JUPYTER_CONFIG_DIR before running nbconvert:

JUPYTER_CONFIG_DIR=./jupyter jupyter nbconvert *.ipynb --to html
Enter fullscreen mode Exit fullscreen mode

The HTML page is now decent on mobile too:

Responsive notebook as HTML on mobile

Exporting specific plots to standalone HTML pages

I talked about notebooks, but what about specific plots?

Once we have a plotly Figure, exporting it to HTML is as easy as calling:

fig = px.line(...) # or fig = go.Figure()
# ...
fig.write_html("my_plot.html")
Enter fullscreen mode Exit fullscreen mode

The my_plot.html file will contain all the necessary js/css to make it interactive and responsive. Nothing more to do!

Free and automated hosting on GitHub Pages with GitHub Actions

Since notebooks are rendered as static HTML files, the easiest way to host them is using GitHub Pages.

We could easily create a branch, manually generate and push the HTML files, and ask GitHub to serve the branch on Pages. But this would be such menial work. Instead, let's use GitHub Actions to automate the whole process!

Prerequisites

For this to work, we need to enable GitHub Pages in the repository on GitHub.

We will use a brand new way of publishing to GitHub Pages: using a custom GitHub Actions workflow. This is still in beta, but it works well and will likely pass to stable soon. For more info, see the official documentation.

Go to Settings > Pages, and under Source select "GitHub Actions":

Repository structure

The next sections assume:

  1. The notebooks are in one folder called dataviz ,

  2. There is a requirements.txt at the root listing all the dependencies (notebook, pandas, etc.)

If you do not have this structure, you may need to change parts of the workflow. It shouldn't be difficult, let me know in the comments if you need some help.

The manual process

Without GitHub Actions, here is how the manual process would look like:

# 0. setup python + environment + clone the repo

# 1. execute and convert notebooks
export JUPYTER_CONFIG_DIR=./jupyter
jupyter nbconvert dataviz/*.ipynb --execute --to html

# 2. copy all the HTML files into a new, empty folder
mkdir public
cp dataviz/*.html public

# 3. create an index.html with a directory listing
tree -H '.' -L 1 --noreport --charset utf-8 -o index.html

# 4. run a static server serving public
python -m http.server 8080 --directory public
Enter fullscreen mode Exit fullscreen mode

The workflow

First, we need to declare a workflow that is triggered on every push to main.

name: Publish Notebooks to Github Pages
on:
  push:
    branches:
      - main
Enter fullscreen mode Exit fullscreen mode

📘 SIDE NOTE 📘 If like me your repository contains more than notebook(s)+data, you can avoid re-publishing when nothing changed on the notebook(s) by adding an on.push.paths filter:

When using the push and pull_request events, you can configure a workflow to run based on what file paths are changed [...] If at least one path matches a pattern in the paths filter, the workflow runs.

This is the filter I use in my repo:

name: Publish Notebooks to Github Pages
on:
  push:
    branches:
      - main
    paths:
       # run only when ...
       # ... the notebooks changed
       - 'dataviz/*.ipynb'
       # ... the data used in the notebooks changed
       - '*.json'
       # ... I modified something in the workflow
       - '.github/**'
Enter fullscreen mode Exit fullscreen mode

Next, we need a job that exports the notebooks to HTML.

Since we are using jupyter commands, we need to setup python and install the dependencies using pip. We then run the nbconvert command and gather all the HTML outputs into a folder - called public in the workflow to follow best practices, but the name doesn't really matter.

The final steps are to call actions/configure-pages and actions/upload-pages-artifact, which are brand new official actions to publish to GitHub Pages (documentation):

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - uses: actions/setup-python@v2
        with:
          python-version: '3.10'
          cache: 'pip' # will run faster with a cache

      - name: Install requirements
        run: pip install -r requirements.txt

      - name: Export notebooks
        run: | 
          export JUPYTER_CONFIG_DIR=./jupyter
          jupyter nbconvert dataviz/*.ipynb --execute --to html

      - name: Prepare public folder
        id: folder # ← to reference the output of this step later
        run: |
          mkdir public
          cp dataviz/*.html public
          echo "name=public" >> $GITHUB_OUTPUT

      - name: Setup Pages
        id: pages # ← used in the deploy job (keep reading)
        uses: actions/configure-pages@v2

      - name: Upload artifact
        uses: actions/upload-pages-artifact@v1
        with:
          path: ${{ steps.folder.outputs.name }}
Enter fullscreen mode Exit fullscreen mode

Note that if we have used fig.write_html(...) in our notebooks, the standalone plots will also be published. Why? Thanks to the --execute argument passed to nbconvert. This asks Jupyter to run the notebook before the conversion, which has some nice side effects:

  • it ensures the code cell numberings will always be consistent, and

  • it makes the job fail if you committed a faulty notebook.

Once the artifact with all the HTML files is uploaded, the only missing piece is to publish it to GitHub Pages using the actions/deploy-pages. This is done in another job, which depends on the build job. Note that it won't run if the build job fails.

jobs:
  build:
    # ... see above
  deploy:
    permissions:
      pages: write
      id-token: write

    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}

    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: Deploy to GitHub Pages
        uses: actions/deploy-pages@v1
Enter fullscreen mode Exit fullscreen mode

That's it! We now have a fully automated process for publishing our notebooks on the web.

What about the directory listing?

The only problem with the workflow above is that unless you have a notebook called index.html hitting https://<username>.github.io/<repo> yields a 404 Not Found. Hugh.

To avoid this, let's serve a listing of all the HTML files available using the community action jayanta525/github-pages-directory-listing. Simply add the following between the "prepare public folder" and "setup pages" actions:

- name: Prepare public folder
  # ...

# ↓↓ add this
- name: Generate Directory Listings
  uses: jayanta525/github-pages-directory-listing@v2.0.0
  with:
    FOLDER: ${{ steps.folder.outputs.name }}

- name: Setup Pages
  # ...
Enter fullscreen mode Exit fullscreen mode

This action lists all the files inside FOLDER and generates a static index.html file allowing navigation. The result can be seen at https://derlin.github.io/dev.to-is-for-web-devs-and-beginners/.

The final workflow

The full workflow is available at https://github.com/derlin/dev.to-is-for-web-devs-and-beginners/blob/main/.github/workflows/ghpages.yaml. Use this link to see the version at the time of writing.

You may have to adapt it a bit to fit your needs, but I hope it provides a strong base to build from.

Going further

This article shows a way to share a notebook in a static form. If you want to go further and make the notebook(s) truly interactive, here are some links to check out. Remember, however, that they all require a Python server running (no easy/free hosting).

Dash

https://dash.plotly.com

Through a couple of simple patterns, Dash abstracts away all of the technologies and protocols that are required to build a full-stack web app with interactive data visualization.

Have a look at their gallery https://dash.gallery/Portal/. I used it a while back and enjoyed the experience.

Mercury

https://mercury-docs.readthedocs.io/en/latest/

Mercury is a perfect tool to convert Python notebook into a web app and share with non-programmers.

  • You define interactive widgets for your notebook with the YAML header.

  • Your users can change the values of the widgets, execute the notebook and save the result (as HTML file).

  • You can hide your code to not scare your (non-coding) collaborators.

  • Easily deploy to any server.

Never tried it, but seems interesting. Let me know!

Image
Revolutionize Your Customer Support with DeskDingo's AI-Powered Knowledge Base and Help Center

knowledgebase, helpcenter
Image
📞 Pay Per Call Marketing Specialist | Connecting Businesses with Real Customers 📲

webdev, javascript, programming, tutorial
Image
menunjuk situs slots online terbaik dengan gampang
Image
24 Hours For Improving Mental Health Tests
Image
Five Things You Didn't Know About New Key For Truck
Image
VPC y Subredes en AWS - Parte 2: Configuración de Conectividad Segura entre Recursos y hacia Internet

aws, cloud, vpc, redes
Image
One Of The Most Innovative Things That Are Happening With Exercise Bike For Sale
Image
Schildersbedrijf in Amersfoort
Image
What's The Job Market For Specializes In Asbestos Litigation Professionals Like?
Image
Why We Love Accident Lawyer Dallas (And You Should, Too!)
Image
7 Things You've Never Known About Asbestos Poisoning Compensation
Image
penyiapan penting saat sebelum taruhan judi games slots online
Image
What is Identity Verification in Blockchain ? Types, Benefits & Challenges
Image
Trap Laten Schilderen Tot 4 Gratis offertes aanvragen
Image
Full-Stack Development

webdev, beginners, fullstack, development
Image
Folding Mobility Scooters For Sale Near Me Tools To Make Your Day-To-Day Life
Image
A Personal Unsecured Loan Is Available Even If You Have Had Bad Credit
Image
What's The Reason? Adult L Shaped Bunk Beds Is Everywhere This Year
Image
Stylowe Mieszkanie: Jak Stworzyć Przestrzeń, Która Inspiruje
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .