Python, Flask and Web-visualizations

GSFC PUG - OCT 2015

Welcome, Goddard Python User's Group!

This guide outlines the steps we took in our October 6th, 2015 Meetup to write a web application, for data visualization, in Python using the Flask microframework.

  • Setup

    • Hello World
    • URL Routing
    • Templates

  • User Inputs

    • URL Variable Inputs
    • Forms (Flask-WTForms)

  • Data Display

    • Static (matplotlib, ggplot, seaborn)
    • Interactive (bokeh, mpld3, plotly)

  • Upgaded Interface

    • Layouts (Bootstrap, Foundation)
    • Document Object Model (jQuery)
    • Call backs (AJAX, Databases)
    • Custom visualizations (d3.js, nvd3)
  • What is flask?
    • Flask takes care of the "Web Server Gateway Interface" (WSGI)

When a user makes a request in the client, flask will bring you back into the python program on the server side.

If a user asks for a yearly average dataset, you can use pandas to provide the sampling, and flask again helps with sending the data back to the user.

Anything you can do in python, you are suddenly able to interact with via the flask module.

There are also flask add-ons, which handle even more amazing tasks, like local language translataion, thwarting hackers, and creating beautiful barebones websites in seconds.

Let's get started!

Follow Along

I've tried to orgainze all the codes here:

git clone https://github.com/jakebrinkmann/2015-PUG-flask-data-vis.git 
cd 2015-PUG-flask-data-vis

Dependencies

Starting off, we will need:

  • python 2.7
  • flask
  • An up-to-date modern browser

I will use other packages along the way, but these are really all you need

Assumptions

  • You are using a UNIX-based OS
  • You have not used flask before
  • You have little or no web-developer experience

If you have used flask before, or know web development... let's make the next talk together!

Getting Started

git checkout version-0.1
  • Hello World
  • URL Routing
  • Templates

Flask Hello World

git checkout tags/v0.1.0

Flask is quick to get running

2015-PUG-flask-data-vis/run.py
In [ ]:
from flask import Flask

app = Flask(__name__) # WSGI application

@app.route('/')
def hello_world():
    return 'Hello from Flask!'

if __name__ == '__main__':
    app.run()

Congratulations, your first web app! It will be running at http://localhost:5000 (only accessible by your machine)

Flask can generate HTML from within Python

git checkout tags/v0.1.1
In [ ]:
from flask import Flask

app = Flask(__name__)

@app.route('/')
def show_html():
    return '<h1>Big heading</h1><p>And some <b>paragraph</b> text</p>'

if __name__ == '__main__':
    app.run()

Debugging

git checkout tags/v0.1.2

If you enable debug support the server will reload itself on code changes, and it will also provide you with a helpful debugger if things go wrong

In [ ]:
from flask import Flask

app = Flask(__name__)

@app.route('/')
def create_error():
    return 'a %s' % a

if __name__ == '__main__':
    app.run(debug=True)

Warning

The interactive debugger allows the execution of arbitrary code.
This makes it a major security risk and therefore it must never be used on production machines.

app.run(debug=os.environ.get('DEBUG', False))

Route Handling

git checkout tags/v0.1.3

Flask can also handle more complex redirects, and request routes

In [ ]:
from flask import Flask, url_for

app = Flask(__name__)

@app.route('/proj')
def projects():
    return '<h1>Project Page</h1><p><ul><li>PUG Data Visualization</li></ul></p>'


@app.route('/')
def hello_world():
    return '<a href="'+ url_for('projects') +'">Projects Page</a>'

if __name__ == '__main__':
    app.run()

App Layout

git checkout tags/v0.1.4

It is important to seperate the presentation from the logic

In [ ]:
! mkdir app
! mkdir app/static
! mkdir app/templates
! mkdir tmp
! touch app/__init__.py
! touch app/views.py
! touch run.py

 2015-PUG-flask-data-vis/app/views.py
In [ ]:
from app import app

@app.route('/')
@app.route('/index')
def index():
    return "Hello, World!"

 2015-PUG-flask-data-vis/app/__init__.py
In [ ]:
from flask import Flask

app = Flask(__name__)
from app import views

 2015-PUG-flask-data-vis/run.py
In [ ]:
from app import app
app.run(debug=True)

Templates

git checkout tags/v0.1.5
2015-PUG-flask-data-vis/app/views.py
In [ ]:
from app import app

@app.route('/')
@app.route('/index')
def index():
    user = {'nickname': 'Dr. Robert'}  # fake user
    return '''
<html>
  <head>
    <title>Home Page</title>
  </head>
  <body>
    <h1>Hello, ''' + user['nickname'] + '''</h1>
  </body>
</html>
'''

Jinja2 Template Engine

Keeps the logic of your flask application separate from the layout of your web pages using place holders

Jinja Templates are great, and the syntax is fairly straight forward

Useful for flask, but also useful for other HTML tasks, like sending emails with preset formating

git checkout v0.1.6

Control statements, like if, for, and also access to object attributes

  2015-PUG-flask-data-vis/app/templates/index.html
In [ ]:
<html>
  <head>
    {% if title %}
    <title>{{ title }} - PUG</title>
    {% else %}
    <title>2015-PUG-Data-Vis</title>
    {% endif %}
  </head>
  <body>
      <h1>Hello, {{ user.nickname }}!</h1>
  </body>
</html>

2015-PUG-flask-data-vis/app/views.py
In [ ]:
from flask import render_template
from app import app

@app.route('/')
@app.route('/index')
def index():
    user = {'nickname': 'Dr. Robert'} 
    return render_template('index.html',
                           title='Home',
                           user=user)

User Inputs

git checkout version-0.2
  • URL Variable Inputs
  • Forms (Flask-WTForms)

URL Variables

git checkout tags/v0.2.0

Add variable parts to a URL.
Optionall converters exist:

  • int - accepts integers
  • float - like int but for floating point values
  • path - like the default but also accepts slashes
     2015-PUG-flask-data-vis/app/views.py
In [ ]:
from flask import render_template
from app import app

@app.route('/post/<int:post_id>')
def show_post(post_id):
    # show the post with the given id, the id is an integer
    return 'Post %d' % post_id

@app.route('/<username>')
@app.route('/index/<username>')
def index(username):
    user = {'nickname': username}
    return render_template('index.html',
                           title='Home',
                           user=user)

HTML Forms

Allow the user to pass inputs back to the server via HTTP "POST" method

Using Flask-WTForms addon is very helpful for building input forms

git checkout tags/v0.2.1
 2015-PUG-flask-data-vis/config.py
In [ ]:
WTF_CSRF_ENABLED = True
SECRET_KEY = 'SHHH!'

 2015-PUG-flask-data-vis/app/__init__.py
In [ ]:
from flask import Flask

app = Flask(__name__)
app.config.from_object('config')
from app import views

 2015-PUG-flask-data-vis/app/views.py
In [ ]:
from flask import render_template, request
from flask_wtf import Form
from wtforms.fields.html5 import DecimalRangeField
from app import app

class MyForm(Form):
    my_slider = DecimalRangeField('Slider')

@app.route('/', methods=('GET', 'POST'))
def index():
    form = MyForm()
    user = {'nickname': 'Dr. Robert'}
    if request.method == 'POST':
        value = request.form['my_slider']
        user['nickname'] = value
    return render_template('index.html',
                           title='Home',
                           user=user,
                           form=form)

 2015-PUG-flask-data-vis/app/templates/index.html
In [ ]:
<html>
  <head>
    {% if title %}
    <title>{{ title }} - PUG</title>
    {% else %}
    <title>2015-PUG-Data-Vis</title>
    {% endif %}
  </head>
  <body>
      <h1>Hello, {{ user.nickname }}!</h1>
      <form method="POST" action="/">
          {{form.hidden_tag()}}
          <p>{{ form.my_slider.label }}: {{ form.my_slider }}</p>
          <input type="submit" value="Go">
      </form>
  </body>
</html>

Assume the worst of your users (They're hackers)

HTTP communications can be faked, so make sure you parse your inputs before using them directly

In [1]:
from flask import Markup
Markup('<strong>Hello %s!</strong>') % '<blink>hacker</blink>'
Out[1]:
Markup(u'<strong>Hello &lt;blink&gt;hacker&lt;/blink&gt;!</strong>')

Data Display

git checkout version-0.3
  • Static (matplotlib, ggplot, seaborn)
  • Interactive (bokeh, mpld3, plotly)

Static Plotting

Most of us probably already use static plotting libraries

  • matplotlib
  • ggplot
  • seaborn
git checkout tags/v0.3.0
 2015-PUG-flask-data-vis/app/views.py
In [ ]:
from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
from matplotlib.figure import Figure

@app.route('/<int:mag>/plot.png')
def plot(mag):
    fig = Figure()
    axis = fig.add_subplot(1, 1, 1)

    xs = range(100)
    ys = [mag * random.randint(1, 50) for x in xs]

    axis.plot(xs, ys)
    canvas = FigureCanvas(fig)
    output = StringIO.StringIO()
    canvas.print_png(output)
    response = make_response(output.getvalue())
    response.mimetype = 'image/png'
    return response

 2015-PUG-flask-data-vis/app/templates/index.html
In [ ]:
  <body>
      <form method="POST" action="/">
          {{form.hidden_tag()}}
          <p>{{ form.my_slider.label }}: {{ form.my_slider }}</p>
          <input type="submit" value="Go">
      </form>
      <img src="{{ url_for('plot', mag=mag) }}"></img>
  </body>

Interactive Plotting

It is nice to generate unique plots, especially when exploring the data

  • bokeh
  • mpld3 
git checkout tags/v0.3.3
 2015-PUG-flask-data-vis/app/views.py
In [ ]:
from bokeh.embed import components
from bokeh.plotting import figure

@app.route('/plot/')
def hello():
    fig = figure(title="Polynomial")
    fig.line(x, [i ** 2 for i in x], color=color, line_width=2)
    script, div = components(fig)
    return render_template(
        'index.html',
        script=script,
        div=div,
    )

 2015-PUG-flask-data-vis/app/templates/index.html
In [ ]:
  <link rel="stylesheet" href="http://cdn.pydata.org/bokeh/release/bokeh-0.9.1.min.css" type="text/css" />
  <script type="text/javascript" src="http://cdn.pydata.org/bokeh/release/bokeh-0.9.1.min.js"></script>
...
  <body>
      {{ div | safe }}
      {{ script | safe }}
  </body>

Upgraded Interface

  • Layouts (Bootstrap, Foundation)
  • Document Object Model (jQuery)
  • Call backs (AJAX, Databases)
  • Custom visualizations (d3.js, nvd3)

Javascript Awesome

git checkout version-0.4

JavaScript can provide very helpful client-side frameworks (think module imports in Python), along with helpful interactions with the "Document Object Model" (or page), just like Python (compiled in the browser)

  • Bootstrap, Foundation, etc. -- Layouts, buttons, themes, etc.
  • JQUERY -- Interaction with elements
  • AJAX -- Asynchronous refresh
In [ ]:
  <!-- Latest compiled and minified CSS -->
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
  <!-- Optional theme -->
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap-theme.min.css">
...
    <nav class="navbar navbar-inverse navbar-fixed-top">
      <div class="container">
...
        <input type="submit" value="Go" class="btn btn-md btn-success">
...
      </div>
    </nav>

Javascript Plotting

  • d3
  • nvd3
  • dygraphs
  • Google Charts

Plotting can be customized to meet your needs using re-useable libraries.

Depolyment

Couple of options for deployment

  • In-App
  • pythonanywhere, heroku
  • Apache2, nginx, RPi server, etc.

In-App

If you trust your internal network, open up a port to listen on all public IPs:

app.run(port=8080, host='0.0.0.0', debug=False)

Then, others can access your host via requests at:

http://yourcomputer:8080/

Hosting Site

Leave out the app.run() call, and setup configurations as directed by the host

pythonanywhere

heroku

Setting up Apache Server

You'll want to setup Apache to be able to call python, using FastCGI

FcgidIPCDir /tmp
AddHandler fcgid-script .fcgi
ServerAlias my-web-app
<VirtualHost *:80>
    DocumentRoot /var/www/datavis/app/static
    Alias /static /var/www/datavis/app/static
    ScriptAlias / /var/www/datavis/runp-mysql.fcgi/
</VirtualHost>

Where your python script is:

 /var/www/datavis/runp-mysql.fcgi

Then, register the DNS on your computer, and it can be accessed at:

http://my-web-app/

CONCLUSIONS ==================================

Common Pitfalls

Things which seemed to pop up reliably to trip me up early on

Get out of Flask/Jinja's way

  • Using loops, other Jinja controls
  • Keeping logic seperate from UI/UX
  • Pre-process & ready the data

Outsource processing to Javascript

  • Javascript is complied language, and is getting faster
    • Out-source some processes to the client browser
  • Imports can overwrite, or depend on, other imports
    • For example, bootstrap depends on jQuery, which must be imported (sourced) first

Determine the website type

  • Flask is awesome for when you need the server backend
  • Use Jekyll (or similar) for static webpages
  • Drupal (or similar) for multi-user content

References

flask docs

The Flask Mega-Tutorial

CodeCademy: JavaScript