Thoughts on search engine,cloud,big data,linux,java,lucene,hadoop

How run unit tests in parallel with testng

Filed under: Uncategorized — euler @ 9:42 pm October 6, 2013

1.To run unit test by classes parallel

 <plugin>      
  <groupId>org.apache.maven.plugins</groupId>      
  <artifactId>maven-surefire-plugin</artifactId>      
  <configuration>        
  <parallel>classes</parallel>      
    <threadCount>4</threadCount>      
  </configuration>      
 </plugin>

2.To run unit test by methods in parallel

<plugin>        
  <groupId>org.apache.maven.plugins</groupId>       
  <artifactId>maven-surefire-plugin</artifactId>        
  <configuration>        
    <parallel>methods</parallel>       
    <threadCount>4</threadCount>        
  </configuration>          
</plugin>

 

Reference:

http://books.sonatype.com/mcookbook/reference/ch07s06.html

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

An multiple application pyramid project configuration

Filed under: python — Tags: , — euler @ 10:54 pm March 22, 2012

In the deployment of pyramid project, I don’t want to setup a separate process for each pyramid application. So it comes the requirement to deploy multiple application under one project.

Here’s an example configuration for a multiple app project. The project has 2 application, booksearch and mybackup. The configuration file syntax is the same as production.ini which is in paste deploy format. For interpretation of each section see: http://pythonpaste.org/deploy/#composite-applications

webapp-production.ini

[composite:main]
use = egg:Paste#urlmap
/ = home
/booksearch = booksearch
/mybackup = mybackup

[app:booksearch]
use = egg:booksearch

[app:mybackup]
use = egg:mybackup

[app:home]
use = egg:Paste#static
document_root = %(here)s/htdocs

[server:main]
use = egg:waitress#main
host = 127.0.0.1
port = 9090

You can run pserve to start the project:

$pserve webapp-production.ini

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

Install an OpenVPN server and connect to it on windows

Filed under: linux,security,troubleshooting — euler @ 5:08 pm

Follow the tutorial on linode to install openvpn:
http://library.linode.com/networking/openvpn/debian-6-squeeze

Attention:
Both openvpn and dnsmasq shall be installed on the server.

And on the file /etc/openvpn/server.conf, the following line must be exist and not commented :
push “redirect-gateway def1″

By default the line is :
;push “redirect-gateway def1 bypass-dhcp”
The semi-colon should be removed, and “bypass-dhcp” should be removed too.

redirect-gateway option to configure the VPN as the default gateway (implemented on Linux and Windows only).

redirect-gateway bypass-dhcp option to add a route
allowing DHCP packets to bypass the tunnel, when the
DHCP server is non-local. Currently only implemented
on Windows clients.

Another tip: In the local directive in the begging of server.conf, ip address of your server should be configured:
local 102.83.214.229

On the client, just install OpenVPN GUI on windows, and download client.conf, ca.crt, client1.crt, client2.key to the OpenVPN/Config folder from the VPN server. The config folder is located at C:\Program Files\OpenVPN\config on my system.

And rename client.conf to vpn-server.ovpn (a mandatory step), start OpenVPN client and click connect.

Some faqs:
1.How to tell if openvpn is installed correctly on the sever.
Just run netstat -na|grep 1194 to check whether it is listening, and check whether the process is running by ps. 1194 is the listening port configured at server.conf

2.Why I cann’t telnet to the 1194?
openvpn is an UDP server by default, you can’t telnet to an UDP port.

3.How could I check if my client is connected to the server?
If the OpenVPN tray turn to green, you are connected to the server.
And you can ping the vpn server, if it replies, it is connected:
ping 10.8.0.1

4.Why I could not visit some sites on the world after connected to the openvpn server.
If your local DNS server is polluted and you are not redirect DNS lookup to VPN server, you will still not be able to open the blocked site. On the vpn server, dnsmasq must be installed, and the following exact line exist at server.conf
push “redirect-gateway def1″

5.How could check the route process to a host such as twitter from linux/windows?
on linux: traceroute twitter.com
on windows: tracert twitter.com

You can also run nslookup on windows, and host/digg on linux for debugging.

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

How to install a network HP printer on Arch linux

Filed under: linux,troubleshooting — Tags: — euler @ 2:54 pm March 16, 2012

The printer package does not work as it should be on my arch for HP laser jet P2035n. It costs me some hour to get it to work. Here’s the steps.

1). Install cups service
pacman -S cups cups-pdf hplip pyqt3 python2-pyqt
pacman -S system-config-printer-gnome
rc.d cups start

2). To add device and install hp driver:
1. Run ‘hp-setup’ as root.
2. Click ‘Show Advanced Options’ button and check-enable ‘Manual Discovery’.
3. Enter IP address 192.168.59.63 in the ‘…device ID…’ entry and click ‘Next’ to install driver.

3). run system-config-printer as root, to add printer

# system-config-printer
click “Add”, Select Device -> Network Printer -> input the IP address of the printer such as “192.168.1.3″, click “Find” -> “Process”

If you encounter the following error:
File “/usr/share/system-config-printer/newprinter.py”, line 1531, in getNetworkPrinterMakeModel
debugprint (host + “: ” + args)
TypeError: cannot concatenate ‘str’ and ‘list’ objects

You should comment out the error line at 1531 of newprinter.py, and then continue to add the printer.

4). Finally, you can run hp-check to see what problems exists for the printer.

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

Could not chdir to home directory /home/USER: Permission denied

Filed under: linux,troubleshooting — euler @ 2:44 pm

We changed the home folder to /data/home/USER.

When I ssh to our centos server. It shows error “Could not chdir to home directory /home/USER: Permission denied”, however loggin ok. I must manually run cd ~ to go to the home directory.

Googled around, and found it is caused by selinux. The solution:

To disabling selinux or change it from enforcing to permissive.
vi /etc/sysconfig/selinux
change SELINUX from enforcing to permissive: SELINUX=permissive, then reboot.

For server could not be rebooted:
#setenforce permissive

You can check if set correctly
#getenforce

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

Open source libraries in chrome 17

Filed under: tech watch — euler @ 12:21 am February 19, 2012

Credits

David M. Gay’s floating point routines
homepage

dynamic annotations
homepage

Netscape Portable Runtime (NSPR)
homepage

Network Security Services (NSS)
homepage

purify headers
homepage

google-glog’s symbolization library
homepage

valgrind
homepage

xdg-mime
homepage

xdg-user-dirs
homepage

BSDiff
homepage

XZ Utils
homepage

google-jstemplate
homepage

Launchpad Translations
homepage

Mozilla Personal Security Manager
homepage

gssapi
homepage

Google Toolbox for Mac
homepage

WebKit
homepage

ActiveX Scripting SDK
homepage

Almost Native Graphics Layer Engine
homepage

Apple sample code
homepage

Darwin
homepage

bsdiff
homepage

bspatch
homepage

bzip2
homepage

Google Cache Invalidation API
homepage

Compact Language Detection
homepage

codesighs
homepage

expat
homepage

ffmpeg
homepage

OpenGL ES 2.0 Programming Guide
homepage

OpenGL ES 2.0 Conformance Tests
homepage

gpsd
homepage

Harfbuzz
homepage

hunspell
homepage

hunspell dictionaries
homepage

hyphen-2.6
homepage

IAccessible2 COM interfaces for accessibility
homepage

iccjpeg
homepage

ICU
homepage

Chinese and Japanese Word List
homepage

ISimpleDOM COM interfaces for accessibility
homepage

jemalloc
homepage

lcov
homepage

libevent
homepage

libjingle
homepage

libjpeg
homepage

libjpeg-turbo
homepage

libpng
homepage

libsrtp
homepage

libvpx
homepage

libwebpdecode
homepage

libxml
homepage

libxslt
homepage

LZMA SDK
homepage

MesaLib
homepage

modp base64 decoder
homepage

NSBezierPath additions from Sean Patrick O’Brien
homepage

Mongoose
homepage

Cocoa extension code from Camino
homepage

npapi
homepage

OCMock
homepage

OpenMAX IL
homepage

openssl
homepage

OTS (OpenType Sanitizer)
homepage

pdfsqueeze
homepage

ppapi
homepage

protobuf
homepage

pyftpdlib
homepage

pywebsocket
homepage

Google Safe Browsing
homepage

simplejson
homepage

skia
homepage

speex
homepage

SQLite
homepage

swig
homepage

talloc
homepage

tcmalloc
homepage

tlslite
homepage

undoview
homepage

Webdriver
homepage

WTL 8.0
homepage

xdg-utils
homepage

XUL Runner SDK
homepage

yasm
homepage

zlib
homepage

Strongtalk
homepage

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

fetchmail configuration file

Filed under: linux — euler @ 4:26 pm February 18, 2012

fetchmail is a mail-retrieval and forwarding utility; it fetches mail from remote mailservers and forwards it to
your local (client) machine’s delivery system. You can then handle the retrieved mail using normal mail user
agents such as mutt(1), elm(1) or Mail(1). The fetchmail utility can be run in a daemon mode to repeatedly poll
one or more systems at a specified interval.

Below is a sample configuration of $HOME/.fetchmailrc for gmail.

The user specifies email login name; password specifies the login password, and is ‘david’ means you login the linux system as user david. If the login name isn’t correct, “unkown user” error will show when retrieving email.

For certificates generation and promail configuration, see:

http://www.andrews-corner.org/mutt.html#fetchmail

poll pop.gmail.com 
with proto POP3 
user 'david.euler' 
there with password 'xxxxyyyy' 
is 'david' here 
mda "/usr/bin/procmail -d %T" 
options 
no keep 
ssl 
sslcertck 
sslcertpath /home/david/.certs/
Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

Pyramid FAQ: from the beginning to production deployment

Filed under: python,troubleshooting — Tags: — euler @ 10:07 pm February 12, 2012

I started to write pyramid applications 3 weeks ago, as a python and pyramid novice.  Now the application goes to production.  In the development process, many errors and problems happened to the application, although the pyramid documentation is very good, some latest changes are not included in the reference document.

I wrote down the common problems I came across ,  and the solution I chose against the problem here.

1. How to install pyramid?

For help getting Pyramid set up, try the official install guide.

Also you may be interested at my pyramid installation guide.

2. Is there a quick start?

Yes, there is. To get the feel of how a Pyramid web application is created, go to the single file tasks tutorial page on the official site.

But be careful, the single file application is just for learning. It is single threaded which is not suitable for most production deployment. And it is not good for maintenance when all content in a single file.

3. How could I create a project for production deployment?

A scaffold project could be created for production deployment. A scaffold project means a template project. You can create project from scaffold.  See the outdated official wiki tutorial for a general guide, the tutorial is for pyramid 1.2 which is different from 1.3. For current pyramid 1.3, you should read this document: Pyramid documentation 1.3 . I’ve written  A quick start for python pyramid for quick reference of pyramid project creation.

4.When I create project by the wiki tutorial of Pyramid and run paster create xxx, it shows command not found?

Paster may not be installed. And if you are using pyraid 1.3 paster is not needed any more, see official Pyramid documentation 1.3 for detail.

5. The wiki tutorial said run “paster serve development.ini” to start pyramid. But paster does not exist.

Paster serve is replaced by pserve in pyramid 1.3.

$ ../bin/pserve development.ini

6. When I run “pserve development.ini” to start pyramid by Pyramid documentation 1.3. It shows error: “pkg_resources.DistributionNotFound: waitress”.

In pyramid 1.3a7, a new dependency waitress is introduced. You can install the packed by easy_install or pip: pip install waitress

7.When I start pyramid by pserve, it allows visiting of only 1 concurrent user.

pserve is single threaded, and is for development purpose. For multithreaded production deployment, you should take one of these wsgi server.

8. Is there a simple way to deploy pyramid as multithreaded application?

Yes, you can Gunicorn to deploy pyramid as multithreaded application.

Firstly install gunicorn and eventlet:

$easy_install gunicorn eventlet

Then create file runapp.py as the following lines:

from paste.deploy import loadapp 

application = loadapp('config:production.ini', relative_to='.')

Then run the following command to start pyramid as multithreaded application:

$gunicorn --workers=2 --worker-class=eventlet --bind 127.0.0.1:80 runapp

the worker-class means to start gunicorn in async mode(multi threaded).

 

And for single threaded deployment, you can easily run Gunicorn with `pserve production.ini` if you’ve been using with Waitress. Just replace the [server:main] section within production.ini as shown in the document: http://gunicorn.org/run.html#paster-serve

http://gunicorn.org/design.html#contents

http://gunicorn.org/configure.html#worker-class

 

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

A quick start for python pyramid

Filed under: python — Tags: , — euler @ 11:30 pm February 10, 2012

Installation of pyramid

1.Install python 2.7/3.x

2.Install setuptools

For windows, please add C:\Python27\Scripts to PATH variable.

For ubuntu: apt-get install python-setuptools

3.Install virtualenv

$easy_install virtualenv

On windows, download virtualenv-1.7.tar.gz, decompress, and run : python setup.py install

create a virtual working directory mysite:
virtualenv mysite
cd mysite
source bin/activate  (Scripts\activate.bat)

Then variable $VIRTUAL_ENV will be set;  and virtualenv modified $PATH, appended bin/python,
also virtualenv installed pip at ENV/bin; so new python packages would be installed here, such as: pip install html5lib.

clear virtual env:
virtualenv –clear mysite

4. Install Pyramid and Preparation

Preparation, on unix/linux:

After you’ve got your env virtualenv installed, you may install Pyramid itself using the following commands from within the virtualenv (env) directory:

$ bin/easy_install pyramid

$bin/easy_install docutils nose coverage zope.sqlalchemy SQLAlchemy pyramid_tm waitress

Preparation, on Windows
Use your Python’s virtualenv to make a workspace:

c:\> c:\Python26\Scripts\virtualenv mysite
Switch to the pyramidtut directory:

c:\> cd mysite
Use easy_install to get Pyramid and its direct dependencies installed:

c:\mysite> Scripts\easy_install pyramid
Use easy_install to install various packages from PyPI.

c:\mysite> Scripts\easy_install docutils nose coverage zope.sqlalchemy SQLAlchemy pyramid_tm waitress
5. How to create project aimed at deployment
————————————
5.1. Making a Project
Your next step is to create a project. Pyramid supplies a variety of scaffolds to generate sample projects. We will use the pyramid_routesalchemy scaffold, which generates an application that uses SQLAlchemy and URL dispatch.

The below instructions assume your current working directory is the “virtualenv” named “pyramidtut”.

On UNIX, for pyramid 1.2:
$ bin/paster create -t pyramid_routesalchemy tutorial

On Windows, for pyramid 1.2:
c:\pyramidtut> Scripts\paster create -t pyramid_routesalchemy tutorial

On unix, for pyramid 1.3:
$ bin/pcreate –list-templates
$ bin/pcreate -s starter tutorial

On Windows, for pyramid 1.3:
scripts\pcreate –list-templates
scripts\pcreate -s starter tutorial

Note

If you are using Windows, the pyramid_routesalchemy scaffold may not deal gracefully with installation into a location that contains spaces in the path. If you experience startup problems, try putting both the virtualenv and the project into directories that do not contain spaces in their paths.

————————————–
5.2. Installing the Project in “Development Mode”
In order to do development on the project easily, you must “register” the project as a development egg in your workspace using the setup.py develop command. In order to do so, cd to the “tutorial” directory you created in Making a Project, and run the “setup.py develop” command using virtualenv Python interpreter.

On UNIX:

$ cd tutorial
$ ../bin/python setup.py develop

———————————
On Windows:

c:\pyramidtut> cd tutorial
c:\pyramidtut\tutorial> ..\Scripts\python setup.py develop

———————————
5.3 Running the Tests
After you’ve installed the project in development mode, you may run the tests for the project.

On UNIX:

$ ../bin/python setup.py test -q
On Windows:

c:\pyramidtut\tutorial> ..\Scripts\python setup.py test -q

————————–
5.4. install new project:

$ cd MyProject
$ ..\Scripts\python.exe setup.py develop

This will install adistributionrepresenting your project into the interpreter’s library set so it can be found by import statements and by other console scripts such aspserve, pshell, proutes and pviews

———————————-
5.5. Starting the Application
Start the application.

On UNIX:

$ ../bin/pserve development.ini
or
$ ../bin/pserve development.ini –reload

When –reload is passed topserve, changes to any Python module your project uses will cause the server to restart. This
typically makes development easier, as changes to Python code made within a Pyramid application is not
put into effect until the server restarts.

On Windows:

c:\pyramidtut\tutorial> ..\Scripts\pserve development.ini
———————–
5.6. Visit the Application in a Browser
In a browser, visit http://localhost:6543/. You will see the generated application’s default page.

One thing you’ll notice is the “debug toolbar” icon on right hand side of the page. You can read more about the purpose of the icon at The Debug Toolbar. It allows you to get information about your application while you develop.
———————–
5.7.Exposing Test Coverage Information

You can run the nosetests command to see test coverage information. This runs the tests in the same way that setup.py test does but provides additional “coverage” information, exposing which lines of your project are “covered” (or not covered) by the tests.

To get this functionality working, we’ll need to install a couple of other packages into our virtualenv: nose and coverage:

On UNIX:

$ ../bin/easy_install nose coverage
On Windows:

c:\pyramidtut\tutorial> ..\Scripts\easy_install nose coverage
Once nose and coverage are installed, we can actually run the coverage tests.

On UNIX:

$ ../bin/nosetests –cover-package=tutorial –cover-erase –with-coverage
On Windows:

c:\pyramidtut\tutorial> ..\Scripts\nosetests –cover-package=tutorial ^
–cover-erase –with-coverage
Looks like our package’s models module doesn’t quite have 100% test coverage.

 

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

The concurrent.futures module in python 3.2

Filed under: python — euler @ 12:55 pm February 7, 2012

A brief introduction on concurrent module in python 3.2 (from python docs)

Code for creating and managing concurrency is being collected in a new top-level namespace, concurrent. Its first member is a futures package which provides a uniform high-level interface for managing threads and processes.

The design for concurrent.futures was inspired by the java.util.concurrent package. In that model, a running call and its result are represented by a Future object that abstracts features common to threads, processes, and remote procedure calls. That object supports status checks (running or done), timeouts, cancellations, adding callbacks, and access to results or exceptions.

The primary offering of the new module is a pair of executor classes for launching and managing calls. The goal of the executors is to make it easier to use existing tools for making parallel calls. They save the effort needed to setup a pool of resources, launch the calls, create a results queue, add time-out handling, and limit the total number of threads, processes, or remote procedure calls.

Ideally, each application should share a single executor across multiple components so that process and thread limits can be centrally managed. This solves the design challenge that arises when each component has its own competing strategy for resource management.

Both classes share a common interface with three methods: submit() for scheduling a callable and returning a Future object; map() for scheduling many asynchronous calls at a time, and shutdown() for freeing resources. The class is a context manager and can be used in a with statement to assure that resources are automatically released when currently pending futures are done executing.

A simple of example of ThreadPoolExecutor is a launch of four parallel threads for copying files:

import concurrent.futures, shutil
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:
    e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
    e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
    e.submit(shutil.copy, 'src3.txt', 'dest4.txt')

See also

PEP 3148 – Futures – Execute Computations Asynchronously
PEP written by Brian Quinlan.

Code for Threaded Parallel URL reads, an example using threads to fetch multiple web pages in parallel.

Code for computing prime numbers in parallel, an example demonstrating ProcessPoolExecutor.

Using the New Python 3.2 Concurrent Programming Features

 

Digg This
Reddit This
Stumble Now!
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)
Older Posts »