Flask app 4 - word counts with AngularJS polling the back-end

bogotobogo.com site search:

Note

In the previous section (Flask app 3 : word counts with Redis task queue), we setup the Redis task queue. In this tutorial, we'll use AngularJS to poll the back-end to see if the task is complete and the data is made available. Then, update the DOM.

Github source : akadrone-flask

Angular - index.html

To use Angular, we need to add the following to templates/index.html:

<script src="//ajax.googleapis.com/ajax/libs/angularjs/1.4.9/angular.min.js"></script>

We also need to add the following directives as well:

ng-app:
ng-controller:
ng-submit:

So, we bootstrapped Angular. In other words, we're telling Angular to treat this HTML document as an Angular application. We'll add a controller, and then a function called getResults() which will be triggered on the form submission.

Here is the code:

<!DOCTYPE html>
<html ng-app="WordcountApp">
  <head>
    <title>Wordcount</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet" media="screen">

    <style>
      .container {
        max-width: 1000px;
      }
    </style>
  </head>
  <body ng-controller="WordcountController">
    <div class="container">
      <div class="row">
        <div class="col-sm-5 col-sm-offset-1">
          <h1>Wordcount</h1>
          <br>
          <form role="form" ng-submit="getResults()">
            <div class="form-group">
              <input type="text" name="url" class="form-control" id="url-box" placeholder="Enter URL..." style="max-width: 300px;">
            </div>
            <button type="submit" class="btn btn-default">Submit</button>
          </form>
          <br>
          {% for error in errors %}
            <h4>{{ error }}</h4>
          {% endfor %}
          <br>
        </div>
        <div class="col-sm-5 col-sm-offset-1">
          {% if results %}
            <h2>Frequencies</h2>
            <br>
            <div id="results">
              <table class="table table-striped" style="max-width: 300px;">
                <thead>
                  <tr>
                    <th>Word</th>
                    <th>Count</th>
                  </tr>
                </thead>
                {% for result in results%}
                  <tr>
                    <td>{{ result[0] }}</td>
                    <td>{{ result[1] }}</td>
                  </tr>
                {% endfor %}
              </table>
            </div>
          {% endif %}
        </div>
      </div>
    </div>
    <br><br>
    <script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
    <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js"></script>
    <script src="//ajax.googleapis.com/ajax/libs/angularjs/1.4.9/angular.min.js"></script>
    <script src="{{ url_for('static', filename='main.js') }}"></script>
  </body>
</html>

static/main.js

Create a static directory, and then add a file main.js to that directory.

Here is the basic code:

(function () {

  'use strict';

  angular.module('WordcountApp', [])

  .controller('WordcountController', ['$scope', '$log',
    function($scope, $log) {
    $scope.getResults = function() {
      $log.log("test");
    };
  }

  ]);

}());

When the form is submitted, getResults() is called. This function simply logs the text "test" to the JavaScript console in the browser as shown in the picture below:

In the code, we're using Dependency Injection (DI) to "inject" the $scope object and $log service.

The injected $scope is a way of communicating between the View and Controller.

Angular takes this approach because it allows us to lessen the worries of dependencies and to focus on unit-testing. We can, for example, configure the DI framework to use mock-objects for underlying components instead of real services during our unit tests.

So, when we change a variable attached to $scope in Controller, the variable in View will automatically updated vice versa (data binding).

Angular $http service

We'll use the Angular $http service to handle "POST" request asynchronously. With this, the form submission no longer sends a POST request to the back end.

Here is the updated static/main.js:

(function () {

  'use strict';

  angular.module('WordcountApp', [])

	.controller('WordcountController', ['$scope', '$log', '$http',
	  function($scope, $log, $http) {

	  $scope.getResults = function() {

	    $log.log("test");

	    // get the URL from the input
	    var userInput = $scope.url;

	    // fire the API request
	    $http.post('/start', {"url": userInput}).
	      success(function(results) {
	        $log.log(results);
	      }).
	      error(function(error) {
	        $log.log(error);
	      });

	  };

	}
	]);

}());

We need to update the input element in index.html.

From:

<input type="text" name="url" class="form-control" id="url-box" placeholder="Enter URL..." style="max-width: 300px;">

To:

<input type="text" ng-model="url" name="url" class="form-control" id="url-box" placeholder="Enter URL..." style="max-width: 300px;">

With the injected $http service, we grabbed the URL from the input box (via ng-model="url"), and then issued a POST request to the back-end. The success and error callbacks handle the response. In the case of a 200 response, it will be handled by the success handler, and logs the response to the console.

Before we test our app again, we need to refactor the back-end to put the /start endpoint.

Refactoring Redis job creation

We want to refactor the Redis job creation from the index view function, and then add it to a new view function called get_counts() in aka.py:

import os
import requests
import operator
import re
import nltk
import json

from flask import Flask, render_template, request
from flask.ext.sqlalchemy import SQLAlchemy
from stop_words import stops
from collections import Counter
from bs4 import BeautifulSoup
import rq
from rq import Queue
from rq.job import Job
from worker import conn
from flask import jsonify

app = Flask(__name__)

app.config.from_object('config')
app_settings = app.config['APP_SETTINGS']
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = True
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://localhost/wordcount_dev'

db = SQLAlchemy(app)

q = Queue(connection=conn)

from models import *

def count_and_save_words(url):

    errors = []

    try:
        r = requests.get(url)
    except:
        errors.append(
            "Unable to get URL. Please make sure it's valid and try again."
        )
        return {"error": errors}

    # text processing
    raw = BeautifulSoup(r.text).get_text()
    nltk.data.path.append('./nltk_data/')  # set the path
    tokens = nltk.word_tokenize(raw)
    text = nltk.Text(tokens)

    # remove punctuation, count raw words
    nonPunct = re.compile('.*[A-Za-z].*')
    raw_words = [w for w in text if nonPunct.match(w)]
    raw_word_count = Counter(raw_words)

    # stop words
    no_stop_words = [w for w in raw_words if w.lower() not in stops]
    no_stop_words_count = Counter(no_stop_words)

    # save the results
    try:
        result = Result(
            url=url,
            result_all=raw_word_count,
            result_no_stop_words=no_stop_words_count
        )
        db.session.add(result)
        db.session.commit()
        return result.id
    except:
        errors.append("Unable to add item to database.")
        return {"error": errors}

@app.route('/', methods=['GET', 'POST'])
def index():
    return render_template('index.html')

@app.route('/start', methods=['POST'])
def get_counts():
    # get url
    data = json.loads(request.data.decode())
    url = data["url"]
    if 'http://' not in url[:7]:
        url = 'http://' + url
    # start job
    job = q.enqueue_call(
        func=count_and_save_words, args=(url,), result_ttl=5000
    )
    # return created job id
    return job.get_id()

@app.route("/results/<job_key>", methods=['GET'])
def get_results(job_key):

    job = Job.fetch(job_key, connection=conn)
    print 'job=',job

    if job.is_finished:
        result = Result.query.filter_by(id=job.result).first()
        results = sorted(
            result.result_no_stop_words.items(),
            key=operator.itemgetter(1),
            reverse=True
        )[:10]
        return jsonify(results)

    else:
        return "Nay!", 202

if __name__ == '__main__':
    app.run()

If we submit a new URL, we can see the job id in our JavaScript console:

Polling - main.js

Here is the main.js where we updated the controller:

(function () {

  'use strict';

  angular.module('WordcountApp', [])

  .controller('WordcountController', ['$scope', '$log', '$http', '$timeout',
    function($scope, $log, $http, $timeout) {

    $scope.getResults = function() {

      $log.log('test');

      // get the URL from the input
      var userInput = $scope.url;

      // fire the API request
      $http.post('/start', {'url': userInput}).
        success(function(results) {
          $log.log(results);
          getWordCount(results);
        }).
        error(function(error) {
          $log.log(error);
        });

    };

    function getWordCount(jobID) {

      var timeout = '';

      var poller = function() {
        // fire another request
        $http.get('/results/'+jobID).
          success(function(data, status, headers, config) {
            if(status === 202) {
              $log.log(data, status);
            } else if (status === 200){
              $log.log(data);
              $timeout.cancel(timeout);
              return false;
            }
            // continue to call the poller() function every 2 seconds
            // until the timeout is cancelled
            timeout = $timeout(poller, 2000);
          });
      };
      poller();
    }

  }]);

}());

Note that we injected the $timeout service into the controller.

If HTTP request is successful, the code will call the getWordCount() function.

Within the poller() function, we called the /results/job_id endpoint.

timeout = $timeout(poller, 2000);

Let's test the update. We may see something like this in Javascript console:

Via the $timeout service, the poller() function continues to fire every 2 seconds until the timeout is cancelled when a 200 response is returned along with the word counts:

In our case, we get 202 response once and then got 200 and word counts.

Updating templates/index.html

We'll add a required attribute to the input tag, which indicate that the input box must be filled out before the form can be submitted.

Our new polling will be handled completely on the client side using Angular tags utilizing double curly braces, {{}}. So, we don't need the the server side Jinja2 template tags.

<!DOCTYPE html>
<html ng-app="WordcountApp">
  <head>
    <title>Wordcount</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet" media="screen">

    <style>
      .container {
        max-width: 1000px;
      }
    </style>
  </head>
  <body ng-controller="WordcountController">
    <div class="container">
      <div class="row">
        <div class="col-sm-5 col-sm-offset-1">
          <h1>Wordcount</h1>
          <br>
            <form role="form" ng-submit="getResults()">
              <div class="form-group">
                <input type="text" name="url" class="form-control" id="url-box" placeholder="Enter URL..." style="max-width: 300px;" ng-model="url" required>
              </div>
              <button type="submit" class="btn btn-default">Submit</button>
            </form>
          <br>
          {% for error in errors %}
            <h4>{{ error }}</h4>
          {% endfor %}
          <br>
        </div>
        <div class="col-sm-5 col-sm-offset-1">
          <h2>Frequencies</h2>
          <br>
          {% raw %}
          <div id="results">           
              {{wordcounts}}            
          </div>
          {% endraw %}
        </div>
      </div>
    </div>
    <br><br>
    <script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
    <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js"></script>
    <script src="//ajax.googleapis.com/ajax/libs/angularjs/1.4.9/angular.min.js"></script>
    <script src="{{ url_for('static', filename='main.js') }}"></script>
  </body>
</html>

As we can see from the code, we escaped the Jinja2 tags using {% raw %} and {% endraw %}.

Updating poller() in static/main.js

Let's update the success handler in the poller() function in static/main.js:

(function () {

  'use strict';

  angular.module('WordcountApp', [])

  .controller('WordcountController', ['$scope', '$log', '$http', '$timeout',
    function($scope, $log, $http, $timeout) {

    $scope.getResults = function() {

      $log.log('test');

      // get the URL from the input
      var userInput = $scope.url;

      // fire the API request
      $http.post('/start', {'url': userInput}).
        success(function(results) {
          $log.log(results);
          getWordCount(results);
        }).
        error(function(error) {
          $log.log(error);
        });

    };

    function getWordCount(jobID) {

      var timeout = '';

      var poller = function() {
        // fire another request
        $http.get('/results/'+jobID).
          success(function(data, status, headers, config) {
            if(status === 202) {
              $log.log(data, status);
            } else if (status === 200){
              $log.log(data);
              $scope.wordcounts = data;
    	      $timeout.cancel(timeout);
              return false;
            }
            // continue to call the poller() function every 2 seconds
            // until the timeout is cancelled
            timeout = $timeout(poller, 2000);
          });
      };
      poller();
    }

  }]);

}());

In the code, we attached the results to the $scope object so that it's available in the View.

Now we can see the $scope object (frequency data)in our view.

Pretty view

We need to reconstruct the return value like this:

['color',13] to {'color':13}

aka.py:

import os
import requests
import operator
import re
import nltk
import json

from flask import Flask, render_template, request
from flask.ext.sqlalchemy import SQLAlchemy
from stop_words import stops
from collections import Counter
from bs4 import BeautifulSoup
import rq
from rq import Queue
from rq.job import Job
from worker import conn
from flask import jsonify
from flask import Response

app = Flask(__name__)

app.config.from_object('config')
app_settings = app.config['APP_SETTINGS']
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = True
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://localhost/wordcount_dev'

db = SQLAlchemy(app)

q = Queue(connection=conn)

from models import *

def count_and_save_words(url):

    errors = []

    try:
        r = requests.get(url)
    except:
        errors.append(
            "Unable to get URL. Please make sure it's valid and try again."
        )
        return {"error": errors}

    # text processing
    raw = BeautifulSoup(r.text).get_text()
    nltk.data.path.append('./nltk_data/')  # set the path
    tokens = nltk.word_tokenize(raw)
    text = nltk.Text(tokens)

    # remove punctuation, count raw words
    nonPunct = re.compile('.*[A-Za-z].*')
    raw_words = [w for w in text if nonPunct.match(w)]
    raw_word_count = Counter(raw_words)

    # stop words
    no_stop_words = [w for w in raw_words if w.lower() not in stops]
    no_stop_words_count = Counter(no_stop_words)

    # save the results
    try:
        result = Result(
            url=url,
            result_all=raw_word_count,
            result_no_stop_words=no_stop_words_count
        )
        db.session.add(result)
        db.session.commit()
        return result.id
    except:
        errors.append("Unable to add item to database.")
        return {"error": errors}

@app.route('/', methods=['GET', 'POST'])
def index():
    return render_template('index.html')


@app.route('/start', methods=['POST'])
def get_counts():
    # get url
    data = json.loads(request.data.decode())
    url = data["url"]
    if 'http://' not in url[:7]:
        url = 'http://' + url
    # start job
    job = q.enqueue_call(
        func=count_and_save_words, args=(url,), result_ttl=5000
    )
    # return created job id
    return job.get_id()

@app.route("/results/", methods=['GET'])
def get_results(job_key):

    job = Job.fetch(job_key, connection=conn)

    if job.is_finished:
        result = Result.query.filter_by(id=job.result).first()
        results = sorted(
            result.result_no_stop_words.items(),
            key=operator.itemgetter(1),
            reverse=True
        )[:10]

        results_dict = {}
        for t in results:
            results_dict[t[0]] = t[1]

        return jsonify(results_dict)

    else:
        return "Nay!", 202


if __name__ == '__main__':
    app.run()

To display the dictionary properly on our view, we want to modify templates/index.html:

<!DOCTYPE html>
<html ng-app="WordcountApp">
  <head>
    <title>Wordcount</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet" media="screen">

    <style>
      .container {
        max-width: 1000px;
      }
    </style>
  </head>
  <body ng-controller="WordcountController">
    <div class="container">
      <div class="row">
        <div class="col-sm-5 col-sm-offset-1">
          <h1>Wordcount</h1>
          <br>
            <form role="form" ng-submit="getResults()">
              <div class="form-group">
                <input type="text" name="url" class="form-control" id="url-box" placeholder="Enter URL..." style="max-width: 300px;" ng-model="url" required>
              </div>
              <button type="submit" class="btn btn-default">Submit</button>
            </form>
          <br>
          {% for error in errors %}
            <h4>{{ error }}</h4>
          {% endfor %}
          <br>
        </div>
        <div class="col-sm-5 col-sm-offset-1">
          <h2>Frequencies</h2>
          <br>
          <div id="results">           
            <table class="table table-striped">
              <thead>
                <tr>
                  <th>Word</th>
                  <th>Count</th>
                </tr>
              </thead>
              <tbody>
              {% raw %}
                <tr ng-repeat="(key, val) in wordcounts">               
                  <td>{{key}}</td>
                  <td>{{val}}</td>                 
                </tr>
              {% endraw %}
              </tbody>
            </table>         
          </div>
        </div>
      </div>
    </div>
    <br><br>
    <script src="//code.jquery.com/jquery-1.11.0.min.js"></script>
    <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js"></script>
    <script src="//ajax.googleapis.com/ajax/libs/angularjs/1.4.9/angular.min.js"></script>
    <script src="{{ url_for('static', filename='main.js') }}"></script>
  </body>
</html>