Filed under: Uncategorized
1.To run unit test by classes parallel
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <parallel>classes</parallel> <threadCount>4</threadCount> </configuration> </plugin>
2.To run unit test by methods in parallel
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <parallel>methods</parallel> <threadCount>4</threadCount> </configuration> </plugin>
Filed under: python
In the deployment of pyramid project, I don’t want to setup a separate process for each pyramid application. So it comes the requirement to deploy multiple application under one project.
Here’s an example configuration for a multiple app project. The project has 2 application, booksearch and mybackup. The configuration file syntax is the same as production.ini which is in paste deploy format. For interpretation of each section see: http://pythonpaste.org/deploy/#composite-applications
[composite:main] use = egg:Paste#urlmap / = home /booksearch = booksearch /mybackup = mybackup [app:booksearch] use = egg:booksearch [app:mybackup] use = egg:mybackup [app:home] use = egg:Paste#static document_root = %(here)s/htdocs [server:main] use = egg:waitress#main host = 127.0.0.1 port = 9090
You can run pserve to start the project:
Filed under: linux, security, troubleshooting
Follow the tutorial on linode to install openvpn:
Both openvpn and dnsmasq shall be installed on the server.
And on the file /etc/openvpn/server.conf, the following line must be exist and not commented :
push “redirect-gateway def1”
By default the line is :
;push “redirect-gateway def1 bypass-dhcp”
The semi-colon should be removed, and “bypass-dhcp” should be removed too.
redirect-gateway option to configure the VPN as the default gateway (implemented on Linux and Windows only).
redirect-gateway bypass-dhcp option to add a route
allowing DHCP packets to bypass the tunnel, when the
DHCP server is non-local. Currently only implemented
on Windows clients.
Another tip: In the local directive in the begging of server.conf, ip address of your server should be configured:
On the client, just install OpenVPN GUI on windows, and download client.conf, ca.crt, client1.crt, client2.key to the OpenVPN/Config folder from the VPN server. The config folder is located at C:\Program Files\OpenVPN\config on my system.
And rename client.conf to vpn-server.ovpn (a mandatory step), start OpenVPN client and click connect.
1.How to tell if openvpn is installed correctly on the sever.
Just run netstat -na|grep 1194 to check whether it is listening, and check whether the process is running by ps. 1194 is the listening port configured at server.conf
2.Why I cann’t telnet to the 1194?
openvpn is an UDP server by default, you can’t telnet to an UDP port.
3.How could I check if my client is connected to the server?
If the OpenVPN tray turn to green, you are connected to the server.
And you can ping the vpn server, if it replies, it is connected:
4.Why I could not visit some sites on the world after connected to the openvpn server.
If your local DNS server is polluted and you are not redirect DNS lookup to VPN server, you will still not be able to open the blocked site. On the vpn server, dnsmasq must be installed, and the following exact line exist at server.conf
push “redirect-gateway def1”
5.How could check the route process to a host such as twitter from linux/windows?
on linux: traceroute twitter.com
on windows: tracert twitter.com
You can also run nslookup on windows, and host/digg on linux for debugging.
Filed under: linux, troubleshooting
The printer package does not work as it should be on my arch for HP laser jet P2035n. It costs me some hour to get it to work. Here’s the steps.
1). Install cups service
pacman -S cups cups-pdf hplip pyqt3 python2-pyqt
pacman -S system-config-printer-gnome
rc.d cups start
2). To add device and install hp driver:
1. Run ‘hp-setup’ as root.
2. Click ‘Show Advanced Options’ button and check-enable ‘Manual Discovery’.
3. Enter IP address 192.168.59.63 in the ‘…device ID…’ entry and click ‘Next’ to install driver.
3). run system-config-printer as root, to add printer
click “Add”， Select Device -> Network Printer -> input the IP address of the printer such as “192.168.1.3”, click “Find” -> “Process”
If you encounter the following error:
File “/usr/share/system-config-printer/newprinter.py”, line 1531, in getNetworkPrinterMakeModel
debugprint (host + “: ” + args)
TypeError: cannot concatenate ‘str’ and ‘list’ objects
You should comment out the error line at 1531 of newprinter.py, and then continue to add the printer.
4). Finally, you can run hp-check to see what problems exists for the printer.
We changed the home folder to /data/home/USER.
When I ssh to our centos server. It shows error “Could not chdir to home directory /home/USER: Permission denied”, however loggin ok. I must manually run cd ~ to go to the home directory.
Googled around, and found it is caused by selinux. The solution:
To disabling selinux or change it from enforcing to permissive.
change SELINUX from enforcing to permissive: SELINUX=permissive, then reboot.
For server could not be rebooted:
You can check if set correctly
fetchmail is a mail-retrieval and forwarding utility; it fetches mail from remote mailservers and forwards it to
your local (client) machine’s delivery system. You can then handle the retrieved mail using normal mail user
agents such as mutt(1), elm(1) or Mail(1). The fetchmail utility can be run in a daemon mode to repeatedly poll
one or more systems at a specified interval.
Below is a sample configuration of $HOME/.fetchmailrc for gmail.
The user specifies email login name; password specifies the login password, and is ‘david’ means you login the linux system as user david. If the login name isn’t correct, “unkown user” error will show when retrieving email.
For certificates generation and promail configuration, see:
poll pop.gmail.com with proto POP3 user 'david.euler' there with password 'xxxxyyyy' is 'david' here mda "/usr/bin/procmail -d %T" options no keep ssl sslcertck sslcertpath /home/david/.certs/
I started to write pyramid applications 3 weeks ago, as a python and pyramid novice. Now the application goes to production. In the development process, many errors and problems happened to the application, although the pyramid documentation is very good, some latest changes are not included in the reference document.
I wrote down the common problems I came across , and the solution I chose against the problem here.
1. How to install pyramid?
For help getting Pyramid set up, try the official install guide.
Also you may be interested at my pyramid installation guide.
2. Is there a quick start?
Yes, there is. To get the feel of how a Pyramid web application is created, go to the single file tasks tutorial page on the official site.
But be careful, the single file application is just for learning. It is single threaded which is not suitable for most production deployment. And it is not good for maintenance when all content in a single file.
3. How could I create a project for production deployment?
A scaffold project could be created for production deployment. A scaffold project means a template project. You can create project from scaffold. See the outdated official wiki tutorial for a general guide, the tutorial is for pyramid 1.2 which is different from 1.3. For current pyramid 1.3, you should read this document: Pyramid documentation 1.3 . I’ve written A quick start for python pyramid for quick reference of pyramid project creation.
4.When I create project by the wiki tutorial of Pyramid and run paster create xxx, it shows command not found?
Paster may not be installed. And if you are using pyraid 1.3 paster is not needed any more, see official Pyramid documentation 1.3 for detail.
5. The wiki tutorial said run “paster serve development.ini” to start pyramid. But paster does not exist.
Paster serve is replaced by pserve in pyramid 1.3.
$ ../bin/pserve development.ini
6. When I run “pserve development.ini” to start pyramid by Pyramid documentation 1.3. It shows error: “pkg_resources.DistributionNotFound: waitress”.
In pyramid 1.3a7, a new dependency waitress is introduced. You can install the packed by easy_install or pip: pip install waitress
7.When I start pyramid by pserve, it allows visiting of only 1 concurrent user.
pserve is single threaded, and is for development purpose. For multithreaded production deployment, you should take one of these wsgi server.
8. Is there a simple way to deploy pyramid as multithreaded application?
Yes, you can Gunicorn to deploy pyramid as multithreaded application.
Firstly install gunicorn and eventlet:
$easy_install gunicorn eventlet
Then create file runapp.py as the following lines:
from paste.deploy import loadapp application = loadapp('config:production.ini', relative_to='.')
Then run the following command to start pyramid as multithreaded application:
$gunicorn --workers=2 --worker-class=eventlet --bind 127.0.0.1:80 runapp
the worker-class means to start gunicorn in async mode(multi threaded).
And for single threaded deployment, you can easily run Gunicorn with `pserve production.ini` if you’ve been using with Waitress. Just replace the [server:main] section within production.ini as shown in the document: http://gunicorn.org/run.html#paster-serve
Installation of pyramid
1.Install python 2.7/3.x
For windows, please add C:\Python27\Scripts to PATH variable.
For ubuntu: apt-get install python-setuptools
On windows, download virtualenv-1.7.tar.gz, decompress, and run : python setup.py install
create a virtual working directory mysite:
source bin/activate (Scripts\activate.bat)
Then variable $VIRTUAL_ENV will be set; and virtualenv modified $PATH, appended bin/python,
also virtualenv installed pip at ENV/bin; so new python packages would be installed here, such as: pip install html5lib.
clear virtual env:
virtualenv –clear mysite
4. Install Pyramid and Preparation
Preparation, on unix/linux:
After you’ve got your env virtualenv installed, you may install Pyramid itself using the following commands from within the virtualenv (env) directory:
$ bin/easy_install pyramid
$bin/easy_install docutils nose coverage zope.sqlalchemy SQLAlchemy pyramid_tm waitress
Preparation, on Windows
Use your Python’s virtualenv to make a workspace:
c:\> c:\Python26\Scripts\virtualenv mysite
Switch to the pyramidtut directory:
c:\> cd mysite
Use easy_install to get Pyramid and its direct dependencies installed:
c:\mysite> Scripts\easy_install pyramid
Use easy_install to install various packages from PyPI.
c:\mysite> Scripts\easy_install docutils nose coverage zope.sqlalchemy SQLAlchemy pyramid_tm waitress
5. How to create project aimed at deployment
5.1. Making a Project
Your next step is to create a project. Pyramid supplies a variety of scaffolds to generate sample projects. We will use the pyramid_routesalchemy scaffold, which generates an application that uses SQLAlchemy and URL dispatch.
The below instructions assume your current working directory is the “virtualenv” named “pyramidtut”.
On UNIX, for pyramid 1.2:
$ bin/paster create -t pyramid_routesalchemy tutorial
On Windows, for pyramid 1.2:
c:\pyramidtut> Scripts\paster create -t pyramid_routesalchemy tutorial
On unix, for pyramid 1.3:
$ bin/pcreate –list-templates
$ bin/pcreate -s starter tutorial
On Windows, for pyramid 1.3:
scripts\pcreate -s starter tutorial
If you are using Windows, the pyramid_routesalchemy scaffold may not deal gracefully with installation into a location that contains spaces in the path. If you experience startup problems, try putting both the virtualenv and the project into directories that do not contain spaces in their paths.
5.2. Installing the Project in “Development Mode”
In order to do development on the project easily, you must “register” the project as a development egg in your workspace using the setup.py develop command. In order to do so, cd to the “tutorial” directory you created in Making a Project, and run the “setup.py develop” command using virtualenv Python interpreter.
$ cd tutorial
$ ../bin/python setup.py develop
c:\pyramidtut> cd tutorial
c:\pyramidtut\tutorial> ..\Scripts\python setup.py develop
5.3 Running the Tests
After you’ve installed the project in development mode, you may run the tests for the project.
$ ../bin/python setup.py test -q
c:\pyramidtut\tutorial> ..\Scripts\python setup.py test -q
5.4. install new project:
$ cd MyProject
$ ..\Scripts\python.exe setup.py develop
This will install adistributionrepresenting your project into the interpreter’s library set so it can be found by import statements and by other console scripts such aspserve, pshell, proutes and pviews
5.5. Starting the Application
Start the application.
$ ../bin/pserve development.ini
$ ../bin/pserve development.ini –reload
When –reload is passed topserve, changes to any Python module your project uses will cause the server to restart. This
typically makes development easier, as changes to Python code made within a Pyramid application is not
put into effect until the server restarts.
c:\pyramidtut\tutorial> ..\Scripts\pserve development.ini
5.6. Visit the Application in a Browser
In a browser, visit http://localhost:6543/. You will see the generated application’s default page.
One thing you’ll notice is the “debug toolbar” icon on right hand side of the page. You can read more about the purpose of the icon at The Debug Toolbar. It allows you to get information about your application while you develop.
5.7.Exposing Test Coverage Information
You can run the nosetests command to see test coverage information. This runs the tests in the same way that setup.py test does but provides additional “coverage” information, exposing which lines of your project are “covered” (or not covered) by the tests.
To get this functionality working, we’ll need to install a couple of other packages into our virtualenv: nose and coverage:
$ ../bin/easy_install nose coverage
c:\pyramidtut\tutorial> ..\Scripts\easy_install nose coverage
Once nose and coverage are installed, we can actually run the coverage tests.
$ ../bin/nosetests –cover-package=tutorial –cover-erase –with-coverage
c:\pyramidtut\tutorial> ..\Scripts\nosetests –cover-package=tutorial ^
Looks like our package’s models module doesn’t quite have 100% test coverage.
Filed under: python
A brief introduction on concurrent module in python 3.2 (from python docs)
Code for creating and managing concurrency is being collected in a new top-level namespace, concurrent. Its first member is a futures package which provides a uniform high-level interface for managing threads and processes.
The design for concurrent.futures was inspired by the java.util.concurrent package. In that model, a running call and its result are represented by a Future object that abstracts features common to threads, processes, and remote procedure calls. That object supports status checks (running or done), timeouts, cancellations, adding callbacks, and access to results or exceptions.
The primary offering of the new module is a pair of executor classes for launching and managing calls. The goal of the executors is to make it easier to use existing tools for making parallel calls. They save the effort needed to setup a pool of resources, launch the calls, create a results queue, add time-out handling, and limit the total number of threads, processes, or remote procedure calls.
Ideally, each application should share a single executor across multiple components so that process and thread limits can be centrally managed. This solves the design challenge that arises when each component has its own competing strategy for resource management.
Both classes share a common interface with three methods: submit() for scheduling a callable and returning a Future object; map() for scheduling many asynchronous calls at a time, and shutdown() for freeing resources. The class is a context manager and can be used in a with statement to assure that resources are automatically released when currently pending futures are done executing.
A simple of example of ThreadPoolExecutor is a launch of four parallel threads for copying files:
import concurrent.futures, shutil with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e: e.submit(shutil.copy, 'src1.txt', 'dest1.txt') e.submit(shutil.copy, 'src2.txt', 'dest2.txt') e.submit(shutil.copy, 'src3.txt', 'dest3.txt') e.submit(shutil.copy, 'src3.txt', 'dest4.txt')
- PEP 3148 – Futures – Execute Computations Asynchronously
- PEP written by Brian Quinlan.
Code for Threaded Parallel URL reads, an example using threads to fetch multiple web pages in parallel.
Filed under: python, troubleshooting
In python, time.strftime() and time.strptime() can be used to format a date to string, and convert string to date. But it is thought buggy in the pyhon strptime() at 2.x version.
For example, if you invoke strptime(), It does not work as expected.
from datetime import datetime dt='Sat Jan 28 12:27:31 +0800 2012'; datetime.strptime(dt,'%a %b %d %H:%M:%S %z %Y')
error shows as: ValueError: ‘z’ is a bad directive in format ‘%a %b %d %H:%M:%S %z %Y’
Then how can we convert string with timezone information to date?For example, given a date string “Sat Jan 28 12:27:31 +0800 2012”, you need the represented date. You can take the convenient of email.utils like this on python 2.x:
import email.utils from datetime import datetime dt =u'Sat Jan 28 12:27:31 +0800 2012'; # parse the date string to several parts, # date_tz is a list: (2012, 1, 28, 12, 27, 31, 0, 1, -1, 28800) date_tz = email.utils.parsedate_tz(dt); mydate=datetime(*date_tz[:6]); # new a datetime by the first 6 elements of the list
Footnote: I checked python 3.2, the strptime works in the first snippet. You may reference the “What’s new in python 3.2“.