task-queues.html

<!DOCTYPE html>
<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="description" content="Full Stack Python shows how an entire Python web application is built and deployed. Each section of the guide explains a different key concept, from the server through the Python WSGI web framework to the front end JavaScript.">
    <meta name="author" content="Matt Makai">
    <link rel="shortcut icon" href="theme/img/full-stack-python-logo-bw.png">
    <title>Task Queues</title>
    <link href="theme/css/fsp.css" rel="stylesheet">
    <!--[if lt IE 9]>
      <script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
      <script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
    <![endif]-->
    <style>
    html,
    body {
      font-size: 18px;
      color: #222;
      background: #fefefe;
    }
    body {
      padding-top: 30px;
    }
    .footer {
      padding: 20px 0 30px 0;
    }
    a, a:hover {border-bottom: 1px dotted; color: #444;}
    a:hover {text-decoration: none; color: #000;}
   .logo-title {font-size: 56px; color: #403072; padding-top: 80px;
                font-family: "News Cycle", "Arial Narrow Bold", sans-serif;
                font-weight: bold; line-height: 30px; margin-left: 5px;}
    .logo-title a, .logo-title a:hover {color: #000; text-decoration: none;
                                        border-bottom: none;}
    .logo-title a:hover {color: gray;}
    .logo-image {vertical-align: top; border: none;}
    a.list-group-item.active {background: #444; border: 1px solid #222;}
    a.list-group-item.active:hover {background: #444; border: 1px solid #222;}
    #sidebar {margin-top: 30px;}

    @media (max-width: 600px) {
        .logo-header-section {
          margin: 20px 32px 0 0;
        }
    }
    </style>
        
    <script type="text/javascript">
      var _gaq = _gaq || [];
      _gaq.push(['_setAccount', 'UA-19910497-7']);
  _gaq.push(['_trackPageview']);

  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();
    </script>
  </head>

<body>
    <a href="https://github.com/makaimc/fullstackpython.github.com" class="github">
        <img style="position: absolute; top: 0; right: 0; border: 0;" src="http://s3.amazonaws.com/github/ribbons/forkme_right_darkblue_121621.png" alt="Fork me on GitHub" />
    </a>
    <div class="container">
        <div class="row">
    <div class="col-md-12">
        <div class="logo-header-section">
            <a href="/" style="text-decoration: none; border: none;"><img src="theme/img/full-stack-python-logo-bw.png" height="42" width="42" class="logo-image" /></a>
            <span class="logo-title"><a href="/">Full Stack Python</a></span>
        </div>
    </div>
</div>
        <div class="row">
    <div class="col-md-8">
      <h1>Task queues</h1>
<p>Task queues handle background work that need to be processed outside the
usual HTTP request-response cycle. These tasks are handled asynchronously 
because HTTP requests must be responded back to by the server as fast as 
possible otherwise the user experience in the web browser will suffer. The 
most common types of jobs for task queues include</p>
<ul>
<li>
<p>calculating computationally expensive data analytics</p>
</li>
<li>
<p>scheduling periodic jobs such as batch processes</p>
</li>
<li>
<p>spreading out large numbers of independent database inserts over time 
  instead of all at once</p>
</li>
<li>
<p>aggregating collected data values on a fixed interval, such as every
  15 minutes</p>
</li>
</ul>
<h2>Task queue projects</h2>
<ul>
<li>
<p>The <a href="http://www.celeryproject.org/">Celery</a> distributed task queue is the
  most commonly used Python library for handling asynchronous tasks and 
  scheduling.</p>
</li>
<li>
<p>The <a href="http://python-rq.org/">RQ (Redis Queue)</a> is a simple Python
  library for queueing jobs and processing them in the background with workers.
  It is backed by Redis and it is designed to have a low barrier to entry.
  It should be integrated in your web stack easily.</p>
</li>
<li>
<p><a href="https://github.com/dcramer/taskmaster">Taskmaster</a> is a lightweight simple
  distributed queue for handling large volumes of one-off tasks. </p>
</li>
</ul>
<h2>Task queue resources</h2>
<ul>
<li>
<p><a href="http://justcramer.com/2012/05/04/distributing-work-without-celery/">Distributing work without Celery</a>
  provides a scenario in which Celery and RabbitMQ are not the right tool
  for scheduling asynchronous jobs.</p>
</li>
<li>
<p><a href="http://queues.io/">Queues.io</a> is a collection of task queue systems with
  short summaries for each one. The task queues are not all compatible with
  Python but ones that work with it are tagged with the "Python" keyword.</p>
</li>
<li>
<p><a href="http://www.slideshare.net/bryanhelmig/task-queues-comorichweb-12962619">Why Task Queues</a> 
  is a presentation for what task queues are and why they are needed. </p>
</li>
<li>
<p><a href="https://www.digitalocean.com/community/articles/how-to-use-celery-with-rabbitmq-to-queue-tasks-on-an-ubuntu-vps">How to use Celery with RabbitMQ</a>
  is a detail walkthrough for using these tools on an Ubuntu VPS.</p>
</li>
<li>
<p>Heroku has a clear walkthrough for using 
  <a href="https://devcenter.heroku.com/articles/python-rq">RQ for background tasks</a>.</p>
</li>
<li>
<p><a href="http://www.linuxforu.com/2013/12/introducing-celery-pythondjango/">Introducing Celery for Python+Django</a> 
  provides an introduction to the Celery task queue.</p>
</li>
</ul>
                                                                                                                                                                                                                                                                                                                                            <br/>
            Next read the 
            <a href="/api-integration.html">api integration</a> section.
                
                                                                                                                                                                                                                                </div>
    <div class="col-md-offset-1 col-md-3" id="sidebar">
  <div class="list-group">
          <a href="/introduction.html" class="list-group-item ">Introduction</a>
          <a href="/servers.html" class="list-group-item ">Servers</a>
          <a href="/operating-systems.html" class="list-group-item ">Operating Systems</a>
          <a href="/web-servers.html" class="list-group-item ">Web Servers</a>
          <a href="/platform-as-a-service.html" class="list-group-item ">Platform-as-a-service</a>
          <a href="/databases.html" class="list-group-item ">Databases</a>
          <a href="/wsgi-servers.html" class="list-group-item ">WSGI Servers</a>
          <a href="/web-frameworks.html" class="list-group-item ">Web Frameworks</a>
          <a href="/application-dependencies.html" class="list-group-item ">Application Dependencies</a>
          <a href="/static-content.html" class="list-group-item ">Static Content</a>
          <a href="/source-control.html" class="list-group-item ">Source Control</a>
          <a href="/caching.html" class="list-group-item ">Caching</a>
          <a href="/task-queues.html" class="list-group-item active">Task Queues</a>
          <a href="/api-integration.html" class="list-group-item ">API Integration</a>
          <a href="/no-sql-datastore.html" class="list-group-item ">NoSQL Data Stores</a>
          <a href="/web-application-security.html" class="list-group-item ">Web Security</a>
          <a href="/logging.html" class="list-group-item ">Logging</a>
          <a href="/monitoring.html" class="list-group-item ">Monitoring</a>
          <a href="/configuration-management.html" class="list-group-item ">Configuration Management</a>
          <a href="/web-analytics.html" class="list-group-item ">Web Analytics</a>
          <a href="/best-python-resources.html" class="list-group-item ">Best Python Resources</a>
          <a href="/about-author.html" class="list-group-item ">About the Author</a>
          <a href="/change-log.html" class="list-group-item ">Change Log</a>
      </div>
</div></div>
        <hr/>
        <div class="footer pull-right">
            <a href="http://www.mattmakai.com/" class="underline">Matt Makai</a> 2014
        </div>
    </div>

    <script src="http://code.jquery.com/jquery-2.1.0.min.js"></script>
    <script src="theme/js/bootstrap.min.js"></script>
    </body>
</html>