Contents

PyViz Application

In this notebook, we will create a Dashboard with two HoloViews objects:

  1. a panel pn.widgets.Select object that contains a list of Xarray variables, and

  2. a hvPlot object that takes the selected variable on input.

hvplot

Fig. 10 Python tools for data visualization PyViz.

See also

An in-depth description of the approach quickly presented here is well discussed in a recent paper by Signell & Pothina (2019)1.

Load the required Python libraries

First of all, load the necessary libraries. These are the ones we discussed previously:

  • numpy

  • matplotlib

  • cartopy

  • panel

  • xarray

  • holoviews

  • geoviews

import os
import numpy as np
import xarray as xr

import cartopy
import cartopy.crs as ccrs
import cartopy.feature as cfeature
from cartopy.mpl.gridliner import LONGITUDE_FORMATTER, LATITUDE_FORMATTER
cartopy.config['data_dir'] = os.getenv('CARTOPY_DIR', cartopy.config.get('data_dir'))

import cmocean

import holoviews as hv
from holoviews import opts, dim

import geoviews as gv
from geoviews import tile_sources as gvts
import geoviews.feature as gf
from cartopy import crs as ccrs

import hvplot.xarray
import panel as pn

import warnings
warnings.filterwarnings("ignore", category=RuntimeWarning) 

gv.extension('bokeh')

Build a multi-file dataset

We will use the open_mfdataset function from xArray to open multiple netCDF files into a single xarray Dataset.

We will query load the GBR4km dataset from the AIMS server, so let’s first define the base URL:

base_url = "http://thredds.ereefs.aims.gov.au/thredds/dodsC/s3://aims-ereefs-public-prod/derived/ncaggregate/ereefs/GBR4_H2p0_B3p1_Cq3b_Dhnd/daily-monthly/EREEFS_AIMS-CSIRO_GBR4_H2p0_B3p1_Cq3b_Dhnd_bgc_daily-monthly-"

For the sake of the demonstration, we will only use 1 month:

month_st = 1   # Starting month 
month_ed = 1   # Ending month 
year = 2018    # Year

biofiles = [f"{base_url}{year}-{month:02}.nc" for month in range(month_st, month_ed+1)]
biofiles

['http://thredds.ereefs.aims.gov.au/thredds/dodsC/s3://aims-ereefs-public-prod/derived/ncaggregate/ereefs/GBR4_H2p0_B3p1_Cq3b_Dhnd/daily-monthly/EREEFS_AIMS-CSIRO_GBR4_H2p0_B3p1_Cq3b_Dhnd_bgc_daily-monthly-2018-01.nc']

Loading the dataset into xArray

Using xArray, we open these files into a Dataset:

ds_bio = xr.open_mfdataset(biofiles)
ds_bio

<xarray.Dataset>
Dimensions:          (k: 17, latitude: 723, longitude: 491, time: 31)
Coordinates:
    zc               (k) float64 dask.array<chunksize=(17,), meta=np.ndarray>
  * time             (time) datetime64[ns] 2018-01-01T02:00:00 ... 2018-01-31...
  * latitude         (latitude) float64 -28.7 -28.67 -28.64 ... -7.066 -7.036
  * longitude        (longitude) float64 142.2 142.2 142.2 ... 156.8 156.8 156.9
Dimensions without coordinates: k
Data variables: (12/101)
    alk              (time, k, latitude, longitude) float32 dask.array<chunksize=(31, 17, 723, 491), meta=np.ndarray>
    BOD              (time, k, latitude, longitude) float32 dask.array<chunksize=(31, 17, 723, 491), meta=np.ndarray>
    Chl_a_sum        (time, k, latitude, longitude) float32 dask.array<chunksize=(31, 17, 723, 491), meta=np.ndarray>
    CO32             (time, k, latitude, longitude) float32 dask.array<chunksize=(31, 17, 723, 491), meta=np.ndarray>
    DIC              (time, k, latitude, longitude) float32 dask.array<chunksize=(31, 17, 723, 491), meta=np.ndarray>
    DIN              (time, k, latitude, longitude) float32 dask.array<chunksize=(31, 17, 723, 491), meta=np.ndarray>
    ...               ...
    SGH_N            (time, latitude, longitude) float32 dask.array<chunksize=(31, 723, 491), meta=np.ndarray>
    SGH_N_pr         (time, latitude, longitude) float32 dask.array<chunksize=(31, 723, 491), meta=np.ndarray>
    SGHROOT_N        (time, latitude, longitude) float32 dask.array<chunksize=(31, 723, 491), meta=np.ndarray>
    SGROOT_N         (time, latitude, longitude) float32 dask.array<chunksize=(31, 723, 491), meta=np.ndarray>
    TSSM             (time, latitude, longitude) float32 dask.array<chunksize=(31, 723, 491), meta=np.ndarray>
    Zenith2D         (time, latitude, longitude) float32 dask.array<chunksize=(31, 723, 491), meta=np.ndarray>
Attributes: (12/20)
    Conventions:                     CF-1.0
    NCO:                             netCDF Operators version 4.7.7 (Homepage...
    RunID:                           2
    _CoordSysBuilder:                ucar.nc2.dataset.conv.CF1Convention
    aims_ncaggregate_buildDate:      2020-08-21T23:07:30+10:00
    aims_ncaggregate_datasetId:      products__ncaggregate__ereefs__GBR4_H2p0...
    ...                              ...
    paramfile:                       /home/bai155/EMS_solar2/gbr4_H2p0_B3p1_C...
    paramhead:                       eReefs 4 km grid. SOURCE Catchments with...
    technical_guide_link:            https://eatlas.org.au/pydio/public/aims-...
    technical_guide_publish_date:    2020-08-18
    title:                           eReefs AIMS-CSIRO GBR4 BioGeoChemical 3....
    DODS_EXTRA.Unlimited_Dimension:  time

Change coordinate name

To make it easier to process the xArray dataset, we change the zc coordinate to the same name as its dimension (i.e. k).

This is done like this:

# Creation of a new coordinate k with the same array as zc
ds_bio.coords['k'] = ('zc',ds_bio.zc)

# Swapping `zc` with `k`
ds_bio = ds_bio.swap_dims({'zc':'k'})

# Ok we can now safely remove `zc`
ds_bio = ds_bio.drop(['zc'])
ds_bio

<xarray.Dataset>
Dimensions:          (k: 17, latitude: 723, longitude: 491, time: 31)
Coordinates:
  * time             (time) datetime64[ns] 2018-01-01T02:00:00 ... 2018-01-31...
  * latitude         (latitude) float64 -28.7 -28.67 -28.64 ... -7.066 -7.036
  * longitude        (longitude) float64 142.2 142.2 142.2 ... 156.8 156.8 156.9
  * k                (k) float64 -145.0 -120.0 -103.0 -88.0 ... -3.0 -1.5 -0.5
Data variables: (12/101)
    alk              (time, k, latitude, longitude) float32 dask.array<chunksize=(31, 17, 723, 491), meta=np.ndarray>
    BOD              (time, k, latitude, longitude) float32 dask.array<chunksize=(31, 17, 723, 491), meta=np.ndarray>
    Chl_a_sum        (time, k, latitude, longitude) float32 dask.array<chunksize=(31, 17, 723, 491), meta=np.ndarray>
    CO32             (time, k, latitude, longitude) float32 dask.array<chunksize=(31, 17, 723, 491), meta=np.ndarray>
    DIC              (time, k, latitude, longitude) float32 dask.array<chunksize=(31, 17, 723, 491), meta=np.ndarray>
    DIN              (time, k, latitude, longitude) float32 dask.array<chunksize=(31, 17, 723, 491), meta=np.ndarray>
    ...               ...
    SGH_N            (time, latitude, longitude) float32 dask.array<chunksize=(31, 723, 491), meta=np.ndarray>
    SGH_N_pr         (time, latitude, longitude) float32 dask.array<chunksize=(31, 723, 491), meta=np.ndarray>
    SGHROOT_N        (time, latitude, longitude) float32 dask.array<chunksize=(31, 723, 491), meta=np.ndarray>
    SGROOT_N         (time, latitude, longitude) float32 dask.array<chunksize=(31, 723, 491), meta=np.ndarray>
    TSSM             (time, latitude, longitude) float32 dask.array<chunksize=(31, 723, 491), meta=np.ndarray>
    Zenith2D         (time, latitude, longitude) float32 dask.array<chunksize=(31, 723, 491), meta=np.ndarray>
Attributes: (12/20)
    Conventions:                     CF-1.0
    NCO:                             netCDF Operators version 4.7.7 (Homepage...
    RunID:                           2
    _CoordSysBuilder:                ucar.nc2.dataset.conv.CF1Convention
    aims_ncaggregate_buildDate:      2020-08-21T23:07:30+10:00
    aims_ncaggregate_datasetId:      products__ncaggregate__ereefs__GBR4_H2p0...
    ...                              ...
    paramfile:                       /home/bai155/EMS_solar2/gbr4_H2p0_B3p1_C...
    paramhead:                       eReefs 4 km grid. SOURCE Catchments with...
    technical_guide_link:            https://eatlas.org.au/pydio/public/aims-...
    technical_guide_publish_date:    2020-08-18
    title:                           eReefs AIMS-CSIRO GBR4 BioGeoChemical 3....
    DODS_EXTRA.Unlimited_Dimension:  time

Simple dashboard

We will use the quadmesh function to quickly rasterize the output to the requested width and height and to create a simple dashboard for interactive, dynamic visualization of eReefs data.

Note

Using the controls on the right, the user can select the pan and wheel_zoom, which enables dynamic exploration of the temperature value in the GBR.

Zooming into the GBR on the eReefs modelled temperature, using the pan and wheel zoom controls.

Tip

By selecting the hover control it allows data values to be displayed along with their coordinates.

var = 'temp'
base_map = gvts.EsriImagery  # ESRI satellite image as background

# Get title from dataset variables attributes
label = f'{ds_bio[var].long_name}: {ds_bio[var].units}'

# Build the quadmesh
mesh = ds_bio[var][:,:].hvplot.quadmesh(x='longitude',y='latitude',
                                        crs=ccrs.PlateCarree(), cmap='jet',
                                        rasterize=True, groupby=list(ds_bio[var].dims[:2]), 
                                        title=label, width=600,height=600)
overlay = (base_map * mesh).opts(active_tools=['wheel_zoom', 'pan'])

# Define the slider as panel widgets
widgets = {dim: pn.widgets.Select for dim in ds_bio[var].dims[:2]}

# Combine everything in a dashboard
dashboard = pn.pane.HoloViews(overlay, widgets=widgets).layout

dashboard

Adding Dashboard functionalities

At this point we have learned how to build interactive apps and dashboards with Panel, how to quickly build visualizations with hvPlot, and add custom interactivity by using HoloViews.

We will now work on putting all of this together to build a more complex, and efficient data processing pipelines, controlled by Panel widgets.

Defining panel widgets

# Define the existing variables in the xArray Dataset
rho_vars = []
for var in ds_bio.data_vars:
    if len(ds_bio[var].dims) > 0:
        rho_vars.append(var)
        

# Define the panel widget for the Xarray variables
var_select = pn.widgets.Select(name='Select variables:', options=rho_vars, 
                               value='temp')

# Define the panel widget for the background maps
base_map_select = pn.widgets.Select(name='Choose underlying map:', 
                                    options=gvts.tile_sources, 
                                    value=gvts.EsriImagery)

# Define the panel widget for the different colormap
color_select = pn.widgets.Select(name='Pick a colormap', options= sorted([
    'cet_bgy', 'cet_bkr', 'cet_bgyw', 'cet_bky', 'cet_kbc', 'cet_coolwarm', 
    'cet_blues', 'cet_gwv', 'cet_bmw', 'cet_bjy', 'cet_bmy', 'cet_bwy', 'cet_kgy', 
    'cet_cwr', 'cet_gray', 'cet_dimgray', 'cet_fire', 'kb', 'cet_kg', 'cet_kr',
    'cet_colorwheel', 'cet_isolium', 'cet_rainbow', 'cet_bgy_r', 'cet_bkr_r', 
    'cet_bgyw_r', 'cet_bky_r', 'cet_kbc_r', 'cet_coolwarm_r', 'cet_blues_r', 
    'cet_gwv_r', 'cet_bmw_r', 'cet_bjy_r', 'cet_bmy_r', 'cet_bwy_r', 'cet_kgy_r', 
    'cet_cwr_r', 'cet_gray_r', 'cet_dimgray_r', 'cet_fire_r', 'kb_r', 'cet_kg_r', 
    'cet_kr_r', 'cet_colorwheel_r', 'cet_isolium_r', 'cet_rainbow_r', 'jet'], 
    key=str.casefold), value='jet')

Defining the plotting functions

This function is the same as the one we defined for the simple dashboard above but it allows for the different variables defined in the panel widgets to be interactively chosen…

def plot(var=None, base_map=None, cmap='jet'):
    
    base_map = base_map or base_map_select.value
    var = var or var_select.value
    
    label = f'{ds_bio[var].long_name}: {ds_bio[var].units}'
    
    mesh = ds_bio[var].hvplot.quadmesh(x='longitude', y='latitude', rasterize=True, title=label,
                                    width=600, height=600, crs=ccrs.PlateCarree(),
                                    groupby=list(ds_bio[var].dims[:-2]), 
                                    cmap=cmap)
    
    mesh = mesh.redim.default(**{d: ds_bio[d].values.max() for d in ds_bio[var].dims[:-2]})
    overlay = (base_map * mesh.opts(alpha=0.9)).opts(active_tools=['wheel_zoom', 'pan'])
    widgets = {dim: pn.widgets.Select for dim in ds_bio[var].dims[:-2]}
    
    return pn.pane.HoloViews(overlay).layout #, widgets=widgets).layout

Widgets value selection functions:

def on_var_select(event):
    var = event.obj.value
    dashboard[-1] = plot(var=var)
    
def on_base_map_select(event):
    base_map = event.obj.value
    dashboard[-1] = plot(base_map=base_map)
    
def on_color_select(event):
    cmap = event.obj.value
    dashboard[-1] = plot(cmap=cmap)
    
var_select.param.watch(on_var_select, parameter_names=['value']);
base_map_select.param.watch(on_base_map_select, parameter_names=['value']);
color_select.param.watch(on_color_select, parameter_names=['value']);

Advanced dashboard

widget = pn.widgets.StaticText(name='', value='High-level dashboarding solution for interactive visualisation', 
                               style={'font-size': "14px", 'font-style': "bold"})

selection_widget = pn.Row(var_select, color_select, base_map_select)

dashboard = pn.Column(widget, selection_widget, plot(var_select.value))
box = pn.WidgetBox('# eReefs App', dashboard)

box.servable()

As you can see, the resulting object is rendered in the notebook (above), and it’s usable as long as you have Python running on this code. You can also launch this app as a standalone server outside of the notebook, because we’ve marked the relevant object .servable(). That declaration means that if someone later runs this notebook as a server process (using panel serve --show ereefs_app.ipynb), your browser will open a separate window with the serveable object ready to explore or share, just like the screenshot at the top of this notebook.

#! panel serve --show --port 5009 ereefs_app.ipynb

Note

This web page was generated from a Jupyter notebook and not all interactivity will work on this website.

1

Signell & Pothina: Analysis and Visualization of Coastal Ocean Model Data in the Cloud, 2019.

Box streaming Particles tracking