How to use Flask-APScheduler in your Python 3 Flask application to run multiple tasks in parallel, from a single HTTP request

When you build an API endpoint that serves HTTP requests to work on long-running tasks, consider using a scheduler. Instead of holding up a HTTP client until a task is completed, you can return an identifier for the client to query the task status later. In the meantime, your HTTP server can offload the task to a scheduler which will complete it and update the status.

When you are building your HTTP server with Python 3 Flask, Flask-APScheduler gives you the facilities to schedule tasks to be executed in the background.

In this post, we look at how we can get Flask-APScheduler in your Python 3 Flask application to run multiple tasks in parallel, from a single HTTP request.

Installing Flask-APScheduler

In order to use Flask-APScheduler, we will need to install it into our Python environment:

pip install Flask-APScheduler

Flask-APScheduler built-in trigger types

Since Flask-APScheduler is based on APScheduler, it comes with three built-in trigger types:

• date: use when you want to run the job just once at a certain point of time
• interval: use when you want to run the job at fixed intervals of time
• cron: use when you want to run the job periodically at certain time(s) of day

As shown above, the first trigger type is what we will need for run multiple tasks in parallel from a single HTTP request.

Example Python 3 Flask application that run multiple tasks in parallel, from a single HTTP request

In order to see the effects of using Flask-APScheduler, let's build a simple Flask application:

from flask import Flask
from flask_apscheduler import APScheduler

import time

app = Flask(__name__)
scheduler = APScheduler()
scheduler.init_app(app)
scheduler.start()

@app.route('/')
def welcome():
    return 'Welcome to flask_apscheduler demo', 200

@app.route('/run-tasks')
def run_tasks():
    for i in range(10):
        app.apscheduler.add_job(func=scheduled_task, trigger='date', args=[i], id='j'+str(i))

    return 'Scheduled several long running tasks.', 200

def scheduled_task(task_id):
    for i in range(10):
        time.sleep(1)
        print('Task {} running iteration {}'.format(task_id, i))
        
app.run(host='0.0.0.0', port=12345)

As an illustration for using Flask-APScheduler, we create the above script that will run a web server at port 12345.

Initializing Flask and APScheduler

After importing the dependencies that are needed, we create a Flask object and a APScheduler object. Once we had created these two objects, we use scheduler.init_app(app) to associate our APScheduler object with our Flask object.

Starting the APScheduler object

When we had associated the APScheduler object with our Flask object, we then start the APScheduler object running at the background. Given that, we can then add tasks to the APScheduler object to run our tasks later.

Scheduling jobs for APScheduler inside the run_tasks function

Next, we define two functions and decorate them with @app.route. Given that, our web server will be able to serve HTTP GET requests designated to / and /run-tasks.

When a HTTP request is received at /run-tasks, run_tasks will be run. In this case, we add 10 jobs that will run scheduled_task via app.apscheduler.add_job and the following keyword arguments:

• func=scheduled_task: the function to run afterwards is scheduled_task.
• trigger='date': an indication that we want to run the task immediately afterwards, since we did not supply an input for run_date.
• args=[i]: a list of arguments to pass to scheduled_task when APScheduler runs it.
• id='j'+str(i): an identifier for the job. When another job with the same identifier is added, it will be ignored by default. This is a good feature when we want to avoid running duplicate work while there is a same task that had not been completed yet.

Simulating long running tasks in scheduled_task

When scheduled_task is run by APScheduler, we simply print 10 statements that are spaced with 1 second delays.

Starting the web server

Finally, at the end of the script, we start our web server through app.run(host='0.0.0.0', port=12345).

Observations from running the code

When you run the Python script, a web server will listen at port 12345. After that, you can then run the following command to initiate a HTTP request to run the 10 tasks:

curl localhost:12345/run-tasks

Once you run the command, you will notice that a HTTP response is immediately returned to you. When you look at the terminal that you use for starting the script, you will notice that 10 statements are printed every second. In addition to that, at each second interval, the statements are not printed in any particular order.

A sample run of the program is as follows:

 * Serving Flask app "run_app" (lazy loading)
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:12345/ (Press CTRL+C to quit)
127.0.0.1 - - [20/Oct/2018 17:11:01] "GET /run-tasks HTTP/1.1" 200 -
Task 0 running iteration 0
Task 1 running iteration 0
Task 2 running iteration 0
Task 3 running iteration 0
Task 5 running iteration 0
Task 6 running iteration 0
Task 8 running iteration 0
Task 9 running iteration 0
Task 7 running iteration 0
Task 4 running iteration 0
Task 0 running iteration 1
Task 1 running iteration 1
Task 3 running iteration 1
Task 2 running iteration 1
Task 5 running iteration 1
Task 6 running iteration 1
Task 9 running iteration 1
Task 4 running iteration 1
Task 8 running iteration 1
Task 7 running iteration 1
Task 3 running iteration 2
Task 0 running iteration 2
Task 1 running iteration 2
Task 2 running iteration 2
Task 6 running iteration 2
Task 9 running iteration 2
Task 5 running iteration 2
Task 4 running iteration 2
Task 8 running iteration 2
Task 7 running iteration 2
Task 0 running iteration 3
Task 2 running iteration 3
Task 1 running iteration 3
Task 3 running iteration 3
Task 9 running iteration 3
Task 6 running iteration 3
Task 7 running iteration 3
Task 8 running iteration 3
Task 4 running iteration 3
Task 5 running iteration 3
Task 2 running iteration 4
Task 1 running iteration 4
Task 0 running iteration 4
Task 3 running iteration 4
Task 9 running iteration 4
Task 8 running iteration 4
Task 7 running iteration 4
Task 4 running iteration 4
Task 6 running iteration 4
Task 5 running iteration 4
Task 1 running iteration 5
Task 3 running iteration 5
Task 2 running iteration 5
Task 0 running iteration 5
Task 8 running iteration 5
Task 9 running iteration 5
Task 4 running iteration 5
Task 7 running iteration 5
Task 6 running iteration 5
Task 5 running iteration 5
Task 1 running iteration 6
Task 3 running iteration 6
Task 0 running iteration 6
Task 2 running iteration 6
Task 8 running iteration 6
Task 9 running iteration 6
Task 4 running iteration 6
Task 7 running iteration 6
Task 6 running iteration 6
Task 5 running iteration 6
Task 1 running iteration 7
Task 0 running iteration 7
Task 2 running iteration 7
Task 3 running iteration 7
Task 8 running iteration 7
Task 9 running iteration 7
Task 4 running iteration 7
Task 7 running iteration 7
Task 6 running iteration 7
Task 5 running iteration 7
Task 1 running iteration 8
Task 0 running iteration 8
Task 3 running iteration 8
Task 2 running iteration 8
Task 8 running iteration 8
Task 4 running iteration 8
Task 9 running iteration 8
Task 7 running iteration 8
Task 6 running iteration 8
Task 5 running iteration 8
Task 1 running iteration 9
Task 0 running iteration 9
Task 2 running iteration 9
Task 3 running iteration 9
Task 8 running iteration 9
Task 4 running iteration 9
Task 9 running iteration 9
Task 6 running iteration 9
Task 5 running iteration 9
Task 7 running iteration 9

• Share
• Tweet
• Share
• Save