fetchmail configuration file

February 18, 2012 by · Comments Off on fetchmail configuration file
Filed under: linux 

fetchmail is a mail-retrieval and forwarding utility; it fetches mail from remote mailservers and forwards it to
your local (client) machine’s delivery system. You can then handle the retrieved mail using normal mail user
agents such as mutt(1), elm(1) or Mail(1). The fetchmail utility can be run in a daemon mode to repeatedly poll
one or more systems at a specified interval.

Below is a sample configuration of $HOME/.fetchmailrc for gmail.

The user specifies email login name; password specifies the login password, and is ‘david’ means you login the linux system as user david. If the login name isn’t correct, “unkown user” error will show when retrieving email.

For certificates generation and promail configuration, see:

http://www.andrews-corner.org/mutt.html#fetchmail

poll pop.gmail.com 
with proto POP3 
user 'david.euler' 
there with password 'xxxxyyyy' 
is 'david' here 
mda "/usr/bin/procmail -d %T" 
options 
no keep 
ssl 
sslcertck 
sslcertpath /home/david/.certs/
Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

Pyramid FAQ: from the beginning to production deployment

February 12, 2012 by · 4 Comments
Filed under: python, troubleshooting 

I started to write pyramid applications 3 weeks ago, as a python and pyramid novice.  Now the application goes to production.  In the development process, many errors and problems happened to the application, although the pyramid documentation is very good, some latest changes are not included in the reference document.

I wrote down the common problems I came across ,  and the solution I chose against the problem here.

1. How to install pyramid?

For help getting Pyramid set up, try the official install guide.

Also you may be interested at my pyramid installation guide.

2. Is there a quick start?

Yes, there is. To get the feel of how a Pyramid web application is created, go to the single file tasks tutorial page on the official site.

But be careful, the single file application is just for learning. It is single threaded which is not suitable for most production deployment. And it is not good for maintenance when all content in a single file.

3. How could I create a project for production deployment?

A scaffold project could be created for production deployment. A scaffold project means a template project. You can create project from scaffold.  See the outdated official wiki tutorial for a general guide, the tutorial is for pyramid 1.2 which is different from 1.3. For current pyramid 1.3, you should read this document: Pyramid documentation 1.3 . I’ve written  A quick start for python pyramid for quick reference of pyramid project creation.

4.When I create project by the wiki tutorial of Pyramid and run paster create xxx, it shows command not found?

Paster may not be installed. And if you are using pyraid 1.3 paster is not needed any more, see official Pyramid documentation 1.3 for detail.

5. The wiki tutorial said run “paster serve development.ini” to start pyramid. But paster does not exist.

Paster serve is replaced by pserve in pyramid 1.3.

$ ../bin/pserve development.ini

6. When I run “pserve development.ini” to start pyramid by Pyramid documentation 1.3. It shows error: “pkg_resources.DistributionNotFound: waitress”.

In pyramid 1.3a7, a new dependency waitress is introduced. You can install the packed by easy_install or pip: pip install waitress

7.When I start pyramid by pserve, it allows visiting of only 1 concurrent user.

pserve is single threaded, and is for development purpose. For multithreaded production deployment, you should take one of these wsgi server.

8. Is there a simple way to deploy pyramid as multithreaded application?

Yes, you can Gunicorn to deploy pyramid as multithreaded application.

Firstly install gunicorn and eventlet:

$easy_install gunicorn eventlet

Then create file runapp.py as the following lines:

from paste.deploy import loadapp 

application = loadapp('config:production.ini', relative_to='.')

Then run the following command to start pyramid as multithreaded application:

$gunicorn --workers=2 --worker-class=eventlet --bind 127.0.0.1:80 runapp

the worker-class means to start gunicorn in async mode(multi threaded).

 

And for single threaded deployment, you can easily run Gunicorn with `pserve production.ini` if you’ve been using with Waitress. Just replace the [server:main] section within production.ini as shown in the document: http://gunicorn.org/run.html#paster-serve

http://gunicorn.org/design.html#contents

http://gunicorn.org/configure.html#worker-class

 

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

A quick start for python pyramid

February 10, 2012 by · Comments Off on A quick start for python pyramid
Filed under: python 

Installation of pyramid

1.Install python 2.7/3.x

2.Install setuptools

For windows, please add C:\Python27\Scripts to PATH variable.

For ubuntu: apt-get install python-setuptools

3.Install virtualenv

$easy_install virtualenv

On windows, download virtualenv-1.7.tar.gz, decompress, and run : python setup.py install

create a virtual working directory mysite:
virtualenv mysite
cd mysite
source bin/activate  (Scripts\activate.bat)

Then variable $VIRTUAL_ENV will be set;  and virtualenv modified $PATH, appended bin/python,
also virtualenv installed pip at ENV/bin; so new python packages would be installed here, such as: pip install html5lib.

clear virtual env:
virtualenv –clear mysite

4. Install Pyramid and Preparation

Preparation, on unix/linux:

After you’ve got your env virtualenv installed, you may install Pyramid itself using the following commands from within the virtualenv (env) directory:

$ bin/easy_install pyramid

$bin/easy_install docutils nose coverage zope.sqlalchemy SQLAlchemy pyramid_tm waitress

Preparation, on Windows
Use your Python’s virtualenv to make a workspace:

c:\> c:\Python26\Scripts\virtualenv mysite
Switch to the pyramidtut directory:

c:\> cd mysite
Use easy_install to get Pyramid and its direct dependencies installed:

c:\mysite> Scripts\easy_install pyramid
Use easy_install to install various packages from PyPI.

c:\mysite> Scripts\easy_install docutils nose coverage zope.sqlalchemy SQLAlchemy pyramid_tm waitress
5. How to create project aimed at deployment
————————————
5.1. Making a Project
Your next step is to create a project. Pyramid supplies a variety of scaffolds to generate sample projects. We will use the pyramid_routesalchemy scaffold, which generates an application that uses SQLAlchemy and URL dispatch.

The below instructions assume your current working directory is the “virtualenv” named “pyramidtut”.

On UNIX, for pyramid 1.2:
$ bin/paster create -t pyramid_routesalchemy tutorial

On Windows, for pyramid 1.2:
c:\pyramidtut> Scripts\paster create -t pyramid_routesalchemy tutorial

On unix, for pyramid 1.3:
$ bin/pcreate –list-templates
$ bin/pcreate -s starter tutorial

On Windows, for pyramid 1.3:
scripts\pcreate –list-templates
scripts\pcreate -s starter tutorial

Note

If you are using Windows, the pyramid_routesalchemy scaffold may not deal gracefully with installation into a location that contains spaces in the path. If you experience startup problems, try putting both the virtualenv and the project into directories that do not contain spaces in their paths.

————————————–
5.2. Installing the Project in “Development Mode”
In order to do development on the project easily, you must “register” the project as a development egg in your workspace using the setup.py develop command. In order to do so, cd to the “tutorial” directory you created in Making a Project, and run the “setup.py develop” command using virtualenv Python interpreter.

On UNIX:

$ cd tutorial
$ ../bin/python setup.py develop

———————————
On Windows:

c:\pyramidtut> cd tutorial
c:\pyramidtut\tutorial> ..\Scripts\python setup.py develop

———————————
5.3 Running the Tests
After you’ve installed the project in development mode, you may run the tests for the project.

On UNIX:

$ ../bin/python setup.py test -q
On Windows:

c:\pyramidtut\tutorial> ..\Scripts\python setup.py test -q

————————–
5.4. install new project:

$ cd MyProject
$ ..\Scripts\python.exe setup.py develop

This will install adistributionrepresenting your project into the interpreter’s library set so it can be found by import statements and by other console scripts such aspserve, pshell, proutes and pviews

———————————-
5.5. Starting the Application
Start the application.

On UNIX:

$ ../bin/pserve development.ini
or
$ ../bin/pserve development.ini –reload

When –reload is passed topserve, changes to any Python module your project uses will cause the server to restart. This
typically makes development easier, as changes to Python code made within a Pyramid application is not
put into effect until the server restarts.

On Windows:

c:\pyramidtut\tutorial> ..\Scripts\pserve development.ini
———————–
5.6. Visit the Application in a Browser
In a browser, visit http://localhost:6543/. You will see the generated application’s default page.

One thing you’ll notice is the “debug toolbar” icon on right hand side of the page. You can read more about the purpose of the icon at The Debug Toolbar. It allows you to get information about your application while you develop.
———————–
5.7.Exposing Test Coverage Information

You can run the nosetests command to see test coverage information. This runs the tests in the same way that setup.py test does but provides additional “coverage” information, exposing which lines of your project are “covered” (or not covered) by the tests.

To get this functionality working, we’ll need to install a couple of other packages into our virtualenv: nose and coverage:

On UNIX:

$ ../bin/easy_install nose coverage
On Windows:

c:\pyramidtut\tutorial> ..\Scripts\easy_install nose coverage
Once nose and coverage are installed, we can actually run the coverage tests.

On UNIX:

$ ../bin/nosetests –cover-package=tutorial –cover-erase –with-coverage
On Windows:

c:\pyramidtut\tutorial> ..\Scripts\nosetests –cover-package=tutorial ^
–cover-erase –with-coverage
Looks like our package’s models module doesn’t quite have 100% test coverage.

 

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

The concurrent.futures module in python 3.2

February 7, 2012 by · Comments Off on The concurrent.futures module in python 3.2
Filed under: python 

A brief introduction on concurrent module in python 3.2 (from python docs)

Code for creating and managing concurrency is being collected in a new top-level namespace, concurrent. Its first member is a futures package which provides a uniform high-level interface for managing threads and processes.

The design for concurrent.futures was inspired by the java.util.concurrent package. In that model, a running call and its result are represented by a Future object that abstracts features common to threads, processes, and remote procedure calls. That object supports status checks (running or done), timeouts, cancellations, adding callbacks, and access to results or exceptions.

The primary offering of the new module is a pair of executor classes for launching and managing calls. The goal of the executors is to make it easier to use existing tools for making parallel calls. They save the effort needed to setup a pool of resources, launch the calls, create a results queue, add time-out handling, and limit the total number of threads, processes, or remote procedure calls.

Ideally, each application should share a single executor across multiple components so that process and thread limits can be centrally managed. This solves the design challenge that arises when each component has its own competing strategy for resource management.

Both classes share a common interface with three methods: submit() for scheduling a callable and returning a Future object; map() for scheduling many asynchronous calls at a time, and shutdown() for freeing resources. The class is a context manager and can be used in a with statement to assure that resources are automatically released when currently pending futures are done executing.

A simple of example of ThreadPoolExecutor is a launch of four parallel threads for copying files:

import concurrent.futures, shutil
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:
    e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
    e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest4.txt')

See also

PEP 3148 – Futures – Execute Computations Asynchronously
PEP written by Brian Quinlan.

Code for Threaded Parallel URL reads, an example using threads to fetch multiple web pages in parallel.

Code for computing prime numbers in parallel, an example demonstrating ProcessPoolExecutor.

Using the New Python 3.2 Concurrent Programming Features

 

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

How to convert string with timezone info to date in python

February 6, 2012 by · Comments Off on How to convert string with timezone info to date in python
Filed under: python, troubleshooting 

In python, time.strftime() and time.strptime() can be used to format a date to string, and convert string to date. But it is thought buggy in the pyhon strptime() at 2.x version.

For example, if you invoke strptime(), It does not work as expected.

from datetime import datetime

dt='Sat Jan 28 12:27:31 +0800 2012';
datetime.strptime(dt,'%a %b %d %H:%M:%S %z %Y')

error shows as:  ValueError: ‘z’ is a bad directive in format ‘%a %b %d %H:%M:%S %z %Y’

Then how can we convert string with timezone information to date?For example, given a date string “Sat Jan 28 12:27:31 +0800 2012”, you need the represented date. You can take the convenient of email.utils like this on python 2.x:

import email.utils
from datetime import datetime

dt =u'Sat Jan 28 12:27:31 +0800 2012';

# parse the date string to several parts,
#  date_tz is a list: (2012, 1, 28, 12, 27, 31, 0, 1, -1, 28800)
date_tz = email.utils.parsedate_tz(dt);  

mydate=datetime(*date_tz[:6]);  # new a datetime by the first 6 elements of the list

Footnote: I checked  python 3.2, the strptime works in the first snippet. You may reference the “What’s new in python 3.2“.

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)