An multiple application pyramid project configuration

March 22, 2012 by · Comments Off on An multiple application pyramid project configuration
Filed under: python 

In the deployment of pyramid project, I don’t want to setup a separate process for each pyramid application. So it comes the requirement to deploy multiple application under one project.

Here’s an example configuration for a multiple app project. The project has 2 application, booksearch and mybackup. The configuration file syntax is the same as production.ini which is in paste deploy format. For interpretation of each section see: http://pythonpaste.org/deploy/#composite-applications

webapp-production.ini

[composite:main]
use = egg:Paste#urlmap
/ = home
/booksearch = booksearch
/mybackup = mybackup

[app:booksearch]
use = egg:booksearch

[app:mybackup]
use = egg:mybackup

[app:home]
use = egg:Paste#static
document_root = %(here)s/htdocs

[server:main]
use = egg:waitress#main
host = 127.0.0.1
port = 9090

You can run pserve to start the project:

$pserve webapp-production.ini

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

Pyramid FAQ: from the beginning to production deployment

February 12, 2012 by · 4 Comments
Filed under: python, troubleshooting 

I started to write pyramid applications 3 weeks ago, as a python and pyramid novice.  Now the application goes to production.  In the development process, many errors and problems happened to the application, although the pyramid documentation is very good, some latest changes are not included in the reference document.

I wrote down the common problems I came across ,  and the solution I chose against the problem here.

1. How to install pyramid?

For help getting Pyramid set up, try the official install guide.

Also you may be interested at my pyramid installation guide.

2. Is there a quick start?

Yes, there is. To get the feel of how a Pyramid web application is created, go to the single file tasks tutorial page on the official site.

But be careful, the single file application is just for learning. It is single threaded which is not suitable for most production deployment. And it is not good for maintenance when all content in a single file.

3. How could I create a project for production deployment?

A scaffold project could be created for production deployment. A scaffold project means a template project. You can create project from scaffold.  See the outdated official wiki tutorial for a general guide, the tutorial is for pyramid 1.2 which is different from 1.3. For current pyramid 1.3, you should read this document: Pyramid documentation 1.3 . I’ve written  A quick start for python pyramid for quick reference of pyramid project creation.

4.When I create project by the wiki tutorial of Pyramid and run paster create xxx, it shows command not found?

Paster may not be installed. And if you are using pyraid 1.3 paster is not needed any more, see official Pyramid documentation 1.3 for detail.

5. The wiki tutorial said run “paster serve development.ini” to start pyramid. But paster does not exist.

Paster serve is replaced by pserve in pyramid 1.3.

$ ../bin/pserve development.ini

6. When I run “pserve development.ini” to start pyramid by Pyramid documentation 1.3. It shows error: “pkg_resources.DistributionNotFound: waitress”.

In pyramid 1.3a7, a new dependency waitress is introduced. You can install the packed by easy_install or pip: pip install waitress

7.When I start pyramid by pserve, it allows visiting of only 1 concurrent user.

pserve is single threaded, and is for development purpose. For multithreaded production deployment, you should take one of these wsgi server.

8. Is there a simple way to deploy pyramid as multithreaded application?

Yes, you can Gunicorn to deploy pyramid as multithreaded application.

Firstly install gunicorn and eventlet:

$easy_install gunicorn eventlet

Then create file runapp.py as the following lines:

from paste.deploy import loadapp 

application = loadapp('config:production.ini', relative_to='.')

Then run the following command to start pyramid as multithreaded application:

$gunicorn --workers=2 --worker-class=eventlet --bind 127.0.0.1:80 runapp

the worker-class means to start gunicorn in async mode(multi threaded).

 

And for single threaded deployment, you can easily run Gunicorn with `pserve production.ini` if you’ve been using with Waitress. Just replace the [server:main] section within production.ini as shown in the document: http://gunicorn.org/run.html#paster-serve

http://gunicorn.org/design.html#contents

http://gunicorn.org/configure.html#worker-class

 

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

A quick start for python pyramid

February 10, 2012 by · Comments Off on A quick start for python pyramid
Filed under: python 

Installation of pyramid

1.Install python 2.7/3.x

2.Install setuptools

For windows, please add C:\Python27\Scripts to PATH variable.

For ubuntu: apt-get install python-setuptools

3.Install virtualenv

$easy_install virtualenv

On windows, download virtualenv-1.7.tar.gz, decompress, and run : python setup.py install

create a virtual working directory mysite:
virtualenv mysite
cd mysite
source bin/activate  (Scripts\activate.bat)

Then variable $VIRTUAL_ENV will be set;  and virtualenv modified $PATH, appended bin/python,
also virtualenv installed pip at ENV/bin; so new python packages would be installed here, such as: pip install html5lib.

clear virtual env:
virtualenv –clear mysite

4. Install Pyramid and Preparation

Preparation, on unix/linux:

After you’ve got your env virtualenv installed, you may install Pyramid itself using the following commands from within the virtualenv (env) directory:

$ bin/easy_install pyramid

$bin/easy_install docutils nose coverage zope.sqlalchemy SQLAlchemy pyramid_tm waitress

Preparation, on Windows
Use your Python’s virtualenv to make a workspace:

c:\> c:\Python26\Scripts\virtualenv mysite
Switch to the pyramidtut directory:

c:\> cd mysite
Use easy_install to get Pyramid and its direct dependencies installed:

c:\mysite> Scripts\easy_install pyramid
Use easy_install to install various packages from PyPI.

c:\mysite> Scripts\easy_install docutils nose coverage zope.sqlalchemy SQLAlchemy pyramid_tm waitress
5. How to create project aimed at deployment
————————————
5.1. Making a Project
Your next step is to create a project. Pyramid supplies a variety of scaffolds to generate sample projects. We will use the pyramid_routesalchemy scaffold, which generates an application that uses SQLAlchemy and URL dispatch.

The below instructions assume your current working directory is the “virtualenv” named “pyramidtut”.

On UNIX, for pyramid 1.2:
$ bin/paster create -t pyramid_routesalchemy tutorial

On Windows, for pyramid 1.2:
c:\pyramidtut> Scripts\paster create -t pyramid_routesalchemy tutorial

On unix, for pyramid 1.3:
$ bin/pcreate –list-templates
$ bin/pcreate -s starter tutorial

On Windows, for pyramid 1.3:
scripts\pcreate –list-templates
scripts\pcreate -s starter tutorial

Note

If you are using Windows, the pyramid_routesalchemy scaffold may not deal gracefully with installation into a location that contains spaces in the path. If you experience startup problems, try putting both the virtualenv and the project into directories that do not contain spaces in their paths.

————————————–
5.2. Installing the Project in “Development Mode”
In order to do development on the project easily, you must “register” the project as a development egg in your workspace using the setup.py develop command. In order to do so, cd to the “tutorial” directory you created in Making a Project, and run the “setup.py develop” command using virtualenv Python interpreter.

On UNIX:

$ cd tutorial
$ ../bin/python setup.py develop

———————————
On Windows:

c:\pyramidtut> cd tutorial
c:\pyramidtut\tutorial> ..\Scripts\python setup.py develop

———————————
5.3 Running the Tests
After you’ve installed the project in development mode, you may run the tests for the project.

On UNIX:

$ ../bin/python setup.py test -q
On Windows:

c:\pyramidtut\tutorial> ..\Scripts\python setup.py test -q

————————–
5.4. install new project:

$ cd MyProject
$ ..\Scripts\python.exe setup.py develop

This will install adistributionrepresenting your project into the interpreter’s library set so it can be found by import statements and by other console scripts such aspserve, pshell, proutes and pviews

———————————-
5.5. Starting the Application
Start the application.

On UNIX:

$ ../bin/pserve development.ini
or
$ ../bin/pserve development.ini –reload

When –reload is passed topserve, changes to any Python module your project uses will cause the server to restart. This
typically makes development easier, as changes to Python code made within a Pyramid application is not
put into effect until the server restarts.

On Windows:

c:\pyramidtut\tutorial> ..\Scripts\pserve development.ini
———————–
5.6. Visit the Application in a Browser
In a browser, visit http://localhost:6543/. You will see the generated application’s default page.

One thing you’ll notice is the “debug toolbar” icon on right hand side of the page. You can read more about the purpose of the icon at The Debug Toolbar. It allows you to get information about your application while you develop.
———————–
5.7.Exposing Test Coverage Information

You can run the nosetests command to see test coverage information. This runs the tests in the same way that setup.py test does but provides additional “coverage” information, exposing which lines of your project are “covered” (or not covered) by the tests.

To get this functionality working, we’ll need to install a couple of other packages into our virtualenv: nose and coverage:

On UNIX:

$ ../bin/easy_install nose coverage
On Windows:

c:\pyramidtut\tutorial> ..\Scripts\easy_install nose coverage
Once nose and coverage are installed, we can actually run the coverage tests.

On UNIX:

$ ../bin/nosetests –cover-package=tutorial –cover-erase –with-coverage
On Windows:

c:\pyramidtut\tutorial> ..\Scripts\nosetests –cover-package=tutorial ^
–cover-erase –with-coverage
Looks like our package’s models module doesn’t quite have 100% test coverage.

 

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

The concurrent.futures module in python 3.2

February 7, 2012 by · Comments Off on The concurrent.futures module in python 3.2
Filed under: python 

A brief introduction on concurrent module in python 3.2 (from python docs)

Code for creating and managing concurrency is being collected in a new top-level namespace, concurrent. Its first member is a futures package which provides a uniform high-level interface for managing threads and processes.

The design for concurrent.futures was inspired by the java.util.concurrent package. In that model, a running call and its result are represented by a Future object that abstracts features common to threads, processes, and remote procedure calls. That object supports status checks (running or done), timeouts, cancellations, adding callbacks, and access to results or exceptions.

The primary offering of the new module is a pair of executor classes for launching and managing calls. The goal of the executors is to make it easier to use existing tools for making parallel calls. They save the effort needed to setup a pool of resources, launch the calls, create a results queue, add time-out handling, and limit the total number of threads, processes, or remote procedure calls.

Ideally, each application should share a single executor across multiple components so that process and thread limits can be centrally managed. This solves the design challenge that arises when each component has its own competing strategy for resource management.

Both classes share a common interface with three methods: submit() for scheduling a callable and returning a Future object; map() for scheduling many asynchronous calls at a time, and shutdown() for freeing resources. The class is a context manager and can be used in a with statement to assure that resources are automatically released when currently pending futures are done executing.

A simple of example of ThreadPoolExecutor is a launch of four parallel threads for copying files:

import concurrent.futures, shutil
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:
    e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
    e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest4.txt')

See also

PEP 3148 – Futures – Execute Computations Asynchronously
PEP written by Brian Quinlan.

Code for Threaded Parallel URL reads, an example using threads to fetch multiple web pages in parallel.

Code for computing prime numbers in parallel, an example demonstrating ProcessPoolExecutor.

Using the New Python 3.2 Concurrent Programming Features

 

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

How to convert string with timezone info to date in python

February 6, 2012 by · Comments Off on How to convert string with timezone info to date in python
Filed under: python, troubleshooting 

In python, time.strftime() and time.strptime() can be used to format a date to string, and convert string to date. But it is thought buggy in the pyhon strptime() at 2.x version.

For example, if you invoke strptime(), It does not work as expected.

from datetime import datetime

dt='Sat Jan 28 12:27:31 +0800 2012';
datetime.strptime(dt,'%a %b %d %H:%M:%S %z %Y')

error shows as:  ValueError: ‘z’ is a bad directive in format ‘%a %b %d %H:%M:%S %z %Y’

Then how can we convert string with timezone information to date?For example, given a date string “Sat Jan 28 12:27:31 +0800 2012”, you need the represented date. You can take the convenient of email.utils like this on python 2.x:

import email.utils
from datetime import datetime

dt =u'Sat Jan 28 12:27:31 +0800 2012';

# parse the date string to several parts,
#  date_tz is a list: (2012, 1, 28, 12, 27, 31, 0, 1, -1, 28800)
date_tz = email.utils.parsedate_tz(dt);  

mydate=datetime(*date_tz[:6]);  # new a datetime by the first 6 elements of the list

Footnote: I checked  python 3.2, the strptime works in the first snippet. You may reference the “What’s new in python 3.2“.

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

How to install pyramid

January 27, 2012 by · Comments Off on How to install pyramid
Filed under: python 

1.Install python 2.7/3.x

2.Install setuptools

For windows, need to add C:\Python27\Scripts to PATH variable

3. Install virtualenv

$ easy_install virtualenv

create a virtual environement mysite:
$virtualenv mysite
$cd mysite
$source bin/activate

Then a new environment variable VIRTUAL_ENV would be set. And virtualenv modified your $PATH variable too by appending bin/python to $PATH; So pythonpath could be in the current directory. virtualenv also installed pip and easy_install in ENV/bin.

How to clean virtual environment:
$virtualenv –clear mysite

4.Install pyramid

After you’ve got your env virtualenv installed, you may install Pyramid itself using the following commands from within
the virtualenv (env) directory:

$ bin/easy_install pyramid

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)