emen2client.py
emen2client.py is the point-of-entry for EMEN2 clients. It has a number of actions available, including bulk upload and download from an EMEN2 database, and uploading EMAN/EMAN2 processing data (e.g. CTF parameters, box coordinates...)
The program is a relatively recent development, and replaces a number of older programs (uploadclient, downloadclient, libemen2client, etc.) that performed similar functions in the past.
Contents
Download emen2client.py
The easiest way to install emen2client, and other EMEN2 programs (database, server, emdash, etc.) is with Python easy_install:
sudo easy_install emdash
This will download the current EMEN2 release from our server, install it into your system's Python package directory, and place EMEN2 executable scripts in a $PATH directory.
Alternatively, you can visit the EMEN2 download page and select a release version, or the nightly build:
For source releases, download the archive, untar, and run "sudo python ./setup.py install"
I try to thoroughly test each version before publishing, but there is always the possibility of bugs, edge cases, or other issues. If you find any problems, please contact Ian Rees, ian.rees@bcm.edu, and include the command you ran, how it failed, and the complete program output.
emen2client.py performs a version check against the database each time it is run. If your version is significantly out of date, it will show a warning and suggest upgrading.
Usage
emen2client.py <action> Actions available: upload, download, sync For detailed help: ./emen2client.py <action> --help
When run from the command line, emen2client.py offers several utility "actions" that can be taken. You invoke an action by specifying it as the first command line argument.
The "upload" action is used for uploading data (ccd frames, volumes, etc.) to the database. Similarly, "download" transfers files from the database to your local machine. These two actions are documented below.
The "sync" action is new, and provides some support for uploading EMAN/EMAN2 image processing data to database records, e.g. CTF parameters. This is documented on a separate page: EMAN2 Integration
Download Usage
The "download" action is used to batch download files from the database. A Record ID is required; in most cases, this will be the Record ID of a grid_imaging session containing CCD frames. There are a number of options, detailed below, although in most cases the default options are fine. You can optionally specify a filename or pattern to filter the list of files that will be downloaded.
Usage: emen2client.py download [options] <recid>
Options:
-h, --help show this help message and exit
-U USERNAME, --username=USERNAME
Username
-P PASSWORD, --password=PASSWORD
Password (Note: specifying passwords in shell commands
is not secure)
-H HOST, --host=HOST Host endpoint URI
-C CTXID, --ctxid=CTXID
Valid Context ID
--recurse=RECURSE Recursion level
-o, --overwrite Overwrite existing files (default is to skip)
-r, --rename If a file already exists, save with format
'duplicate.recid:original_filename.dm3'
-s, --sidecar Include sidecar file with EMEN2 metadata in JSON
formatExample:
[irees@mbp ~/data]$ emen2client.py download 123456
Username: ianrees
Password:
1 of 1: 123456
Checking for items to download
Found 92 items in 93 records
1 of 92: 2009110400033
Downloading frame1.dm3.gz -> frame1.dm3...
2 of 92: 2009110400032
Downloading frame2.dm3.gz -> frame2.dm3...
3 of 92: 2009110400031
Downloading frame3.dm3.gz -> frame3.dm3...
.... snip ....
Upload Usage
The "upload" action will upload files into the database. A record type, record ID, and list of files must be specified. If you are uploading CCD frames, use "ccd", if you are uploading SerielEM tomography stacks, use "stack." Other types are also available, e.g. "volume" to upload a number of density maps.
Usage: emen2client.py upload [options] <record type> <recid> <files to upload>
Record type can be any valid database protocol.
Some record types have special, application-specific handlers, e.g.:
ccd CCD Frames
scan Scanned micrographs
stack Tomograms
Other values, e.g. "volume", will create child records of that type, with 1 file per child record.
Alternatively, you can use "none" for record type and the files will be attached directly to the specified record ID.
Options:
-h, --help show this help message and exit
-U USERNAME, --username=USERNAME
Username
-P PASSWORD, --password=PASSWORD
Password (Note: specifying passwords in shell commands
is not secure)
-H HOST, --host=HOST Host endpoint URIExample:
[irees@mbp ~/data]$ emen2client.py upload ccd 123456 *.dm3
Username: ianrees
Password:
3 Files to upload:
1.dm3
2.dm3
3.dm3
----- tem_magnification_set -----
Description: Magnification set on microscope
Units: K
Common values:
0) 60.0
1) 50.0
2) 40.0
3) 20.0
4) 80.0
5) 100.0
6) 15.0
7) 30.0
8) 25.0
10) None or N/A
11) Enter a different not listed above
Selection (0-11): 0
----- ctf_defocus_set -----
Description: Intended defocus while imaging, underfocus positive
Units: um
Common values:
0) 1.0
1) -2.0
2) 2.0
3) 1.5
4) 3.0
5) 2.5
6) -3.0
7) -1.5
8) -2.5
9) 0.8
10) None or N/A
11) Enter a different not listed above
Selection (0-11): 0
... snip: additional prompts ommitted for clarity ...
1 of 3: 1.dm3
Preparing for upload
Checking and committing records
1.dm3
Done. Uploaded 1.dm3 to record 417171 @ 15210.92 kb/sec
2 of 3: 2.dm3
Preparing for upload
Checking and committing records
2.dm3
Done. Uploaded 2.dm3 to record 417172 @ 15080.45 kb/sec
3 of 3: 3.dm3
Preparing for upload
Checking and committing records
3.dm3
Done. Uploaded 3.dm3 to record 417173 @ 14253.12 kb/sec
EMAN Synchronization
emen2client.py provides growing support for uploading processed image data back into the database. See EMAN2 Integration for details.