task-queues.html

<!DOCTYPE html>
<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="description" content="Full Stack Python explains each layer of the web application stack, from the server up through the rendering in a user's browser.">
    <meta name="author" content="Matt Makai">
    <link rel="shortcut icon" href="theme/img/full-stack-python-logo-bw.png">
    <title>Task Queues - Full Stack Python</title>
    <link href="theme/css/fsp.css" rel="stylesheet">
    <!--[if lt IE 9]>
      <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
      <script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
    <![endif]-->
    <style>
    html,
    body {
      font-size: 18px;
      color: #222;
      background: #fefefe;
    }
    body {
      padding-top: 30px;
    }
    .footer {
      padding: 20px 0 30px 0;
    }
    a, a:hover {border-bottom: 1px dotted; color: #444;}
    a:hover {text-decoration: none; color: #000;}
   .logo-title {font-size: 56px; color: #403072; padding-top: 80px;
                font-family: "News Cycle", "Arial Narrow Bold", sans-serif;
                font-weight: bold; line-height: 30px; margin-left: 5px;}
    .logo-title a, .logo-title a:hover {color: #000; text-decoration: none;
                                        border-bottom: none;}
    .logo-title a:hover {color: gray;}
    .logo-image {vertical-align: top; border: none;}
    a.list-group-item.active {background: #444; border: 1px solid #222;}
    a.list-group-item.active:hover {background: #444; border: 1px solid #222;}
    #sidebar {margin-top: 30px;}

    @media (max-width: 600px) {
        .logo-header-section {
          margin: 20px 32px 0 0;
        }
    }
    </style>
        
    <script type="text/javascript">
      var _gaq = _gaq || [];
      _gaq.push(['_setAccount', 'UA-19910497-7']);
  _gaq.push(['_trackPageview']);

  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();
    </script>
  </head>

<body>
    <a href="https://github.com/makaimc/fullstackpython.github.com" class="github">
        <img style="position: absolute; top: 0; right: 0; border: 0;" src="http://s3.amazonaws.com/github/ribbons/forkme_right_darkblue_121621.png" alt="Fork me on GitHub" />
    </a>
    <div class="container">
        <div class="row">
    <div class="col-md-12">
        <div class="logo-header-section">
            <a href="/" style="text-decoration: none; border: none;"><img src="theme/img/full-stack-python-logo-bw.png" height="42" width="42" class="logo-image" /></a>
            <span class="logo-title"><a href="/">Full Stack Python</a></span>
        </div>
    </div>
</div>
        <div class="row">
    <div class="col-md-8">
      <h1>Task queues</h1>
<p>Task queues handle background work that need to be processed outside the
usual HTTP request-response cycle. These tasks are handled asynchronously 
because HTTP requests must be responded back to by the server as fast as 
possible otherwise the user experience in the web browser will suffer. The 
most common types of jobs for task queues include</p>
<ul>
<li>
<p>calculating computationally expensive data analytics</p>
</li>
<li>
<p>scheduling periodic jobs such as batch processes</p>
</li>
<li>
<p>spreading out large numbers of independent database inserts over time 
  instead of all at once</p>
</li>
<li>
<p>aggregating collected data values on a fixed interval, such as every
  15 minutes</p>
</li>
</ul>
<h2>Task queue projects</h2>
<ul>
<li>
<p>The <a href="http://www.celeryproject.org/">Celery</a> distributed task queue is the
  most commonly used Python library for handling asynchronous tasks and 
  scheduling.</p>
</li>
<li>
<p>The <a href="http://python-rq.org/">RQ (Redis Queue)</a> is a simple Python
  library for queueing jobs and processing them in the background with workers.
  RQ is backed by Redis and is designed to have a low barrier to entry.
  The <a href="http://nvie.com/posts/introducing-rq/">intro post</a> contains information
  on design decisions and how to use RQ.</p>
</li>
<li>
<p><a href="https://github.com/dcramer/taskmaster">Taskmaster</a> is a lightweight simple
  distributed queue for handling large volumes of one-off tasks. </p>
</li>
</ul>
<h2>Task queue resources</h2>
<ul>
<li>
<p><a href="http://justcramer.com/2012/05/04/distributing-work-without-celery/">Distributing work without Celery</a>
  provides a scenario in which Celery and RabbitMQ are not the right tool
  for scheduling asynchronous jobs.</p>
</li>
<li>
<p><a href="http://queues.io/">Queues.io</a> is a collection of task queue systems with
  short summaries for each one. The task queues are not all compatible with
  Python but ones that work with it are tagged with the "Python" keyword.</p>
</li>
<li>
<p><a href="http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619">Why Task Queues</a> 
  is a presentation for what task queues are and why they are needed. </p>
</li>
<li>
<p><a href="https://www.digitalocean.com/community/articles/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps">How to use Celery with RabbitMQ</a>
  is a detailed walkthrough for using these tools on an Ubuntu VPS.</p>
</li>
<li>
<p>Heroku has a clear walkthrough for using 
  <a href="https://devcenter.heroku.com/articles/python-rq">RQ for background tasks</a>.</p>
</li>
<li>
<p><a href="http://www.linuxforu.com/2013/12/introducing-celery-pythondjango/">Introducing Celery for Python+Django</a> 
  provides an introduction to the Celery task queue.</p>
</li>
<li>
<p>The "Django in Production" series by 
  <a href="https://twitter.com/robgolding63">Rob Golding</a> contains a post 
  specifically on <a href="http://www.robgolding.com/blog/2011/11/27/django-in-production-part-2---background-tasks/">Background Tasks</a>.</p>
</li>
<li>
<p><a href="http://blog.thecodepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/">Asynchronous Processing in Web Applications Part One</a> 
  and <a href="http://blog.thecodepath.com/2013/01/06/asynchronous-processing-in-web-applications-part-2-developers-need-to-understand-message-queues/">Part Two</a>
  are great reads for understanding the difference between a task queue and
  why you shouldn't use your database as one.</p>
</li>
</ul>
                                                                                                                                                                                                                                                                                                                                                                                        <br/>
            Next read the 
            <a href="/api-integration.html">api integration</a> section.
                
                                                                                                                                                                                                                                </div>
    <div class="col-md-offset-1 col-md-3" id="sidebar">
  <div class="list-group">
          <a href="/introduction.html" class="list-group-item ">Introduction</a>
          <a href="/servers.html" class="list-group-item ">Servers</a>
          <a href="/operating-systems.html" class="list-group-item ">Operating Systems</a>
          <a href="/web-servers.html" class="list-group-item ">Web Servers</a>
          <a href="/platform-as-a-service.html" class="list-group-item ">Platform-as-a-service</a>
          <a href="/databases.html" class="list-group-item ">Databases</a>
          <a href="/wsgi-servers.html" class="list-group-item ">WSGI Servers</a>
          <a href="/web-frameworks.html" class="list-group-item ">Web Frameworks</a>
          <a href="/application-dependencies.html" class="list-group-item ">Application Dependencies</a>
          <a href="/cascading-style-sheets.html" class="list-group-item ">Cascading Style Sheets</a>
          <a href="/javascript.html" class="list-group-item ">JavaScript</a>
          <a href="/static-content.html" class="list-group-item ">Static Content</a>
          <a href="/source-control.html" class="list-group-item ">Source Control</a>
          <a href="/caching.html" class="list-group-item ">Caching</a>
          <a href="/task-queues.html" class="list-group-item active">Task Queues</a>
          <a href="/api-integration.html" class="list-group-item ">API Integration</a>
          <a href="/no-sql-datastore.html" class="list-group-item ">NoSQL Data Stores</a>
          <a href="/web-application-security.html" class="list-group-item ">Web Security</a>
          <a href="/logging.html" class="list-group-item ">Logging</a>
          <a href="/monitoring.html" class="list-group-item ">Monitoring</a>
          <a href="/configuration-management.html" class="list-group-item ">Configuration Management</a>
          <a href="/web-analytics.html" class="list-group-item ">Web Analytics</a>
          <a href="/best-python-resources.html" class="list-group-item ">Best Python Resources</a>
          <a href="/about-author.html" class="list-group-item ">About the Author</a>
          <a href="/change-log.html" class="list-group-item ">Change Log</a>
      </div>
</div></div>
        <hr/>
        <div class="footer pull-right">
            <a href="http://www.mattmakai.com/" class="underline">Matt Makai</a> 2014
        </div>
    </div>

    <script src="http://code.jquery.com/jquery-2.1.0.min.js"></script>
    <script src="theme/js/bootstrap.min.js"></script>
    </body>
</html>