OpenMicroscopyEnvironment

Open Microscopy Environment (OME) Notes

Some notes about OME from a telephone conversation with Jason Swedlow, and subsequent explorations.

Contents:

  1. Random Notes
  2. OME installation notes
    1. RedHat notes
    2. Dependent packages
    3. Pre-installation
    4. Install script Q&A
    5. Problems with GD / libGD
    6. Problems with installation self-test
    7. Apache configuration problems
    8. PostgreSQL configuration and administration
    9. Accessing images on the VMware host filesystem via HGFS
    10. OME admin and test user setup
    11. Moving the OME environment to a different host
      1. General Instructions for Moving an OME database
      2. Other related postings to the OME users list
    12. Other post-installation notes
  3. OME Customization
    1. Create a new semantic type for DTGED image annotations
    2. Import the new semantic type into OME
    3. Refine the new semantic type definition
    4. Semantic type references
    5. Stuff - for later

1. Random Notes

2. OME installation notes

At the time of writing, I am using Scientific Linux 4.0 (a RedHat Enterprise Linux 4.0, or RHEL4, clone) on VMware 5.0 as the host operating system.

2.1. RedHat notes

See: [WWW] http://www.openmicroscopy.org/install/redhat.html

After installation, the first yum update can take a while, and the system appears to die. Be patient!

After Redhat has been installed, it may be that Emacs cannot be run from a console terminal window. This may be fixed by entering the commands: 

xhost local:root
xhost local:<user>

where <user> is the login name for the user for whom X Window access is to be permitted. I din't know why these options aren't default - maybe an Enterprise Linux thing?

2.2. Dependent packages

Use yum install <pkg> to find and install packages

Neither libjpeg nor libjpeg-devel are mentioned in Linux install notes for OME (this may be not true for later OME installation kits). libjpeg-devel is not part of the standard Linux installation:

The standard SL4 installation with Postgresql database does not include the postgres server. I also think that postgres-pl is required. This should fix these omissions:

This is needed for the GD library to be installed succesfully as described below.

2.3. Pre-installation

Apache must be running with MOD-PERL enabled.

Postgresql server must be installed and running (see note above about server modules missing from a standard SL4 installation).

I think it's probably best to pre-install the GD/libGD library (see below).

2.4. Install script Q&A

Unpack the OME distribution archive file (OME/stable.tar.gz) into a working directory. Make that the current directory. (I have used directory /store/kit/OME/OME/ on the host helenome.)

OME currently uses the same filename for all releases. Check that the correct distribution is being used. For version 2.4, released March 2005, I have stable.tar.gz with size 2869737 bytes, and having unpacked that archive: 

[root@delos OME]# ls --full-time
total 64
drwxr-xr-x   2 1002 users 4096 2005-03-09 11:39:31.000000000 +0000 conf
drwxr-xr-x   4 1002 users 4096 2005-03-09 11:39:31.000000000 +0000 doc
drwxr-xr-x   2 1002 users 4096 2005-03-09 11:39:31.000000000 +0000 images
-rwxr-xr-x   1 1002 users 8382 2005-01-27 17:27:51.000000000 +0000 install.pl
-rw-r--r--   1 1002 users  772 2004-04-02 15:05:33.000000000 +0100 README
drwxr-xr-x  11 1002 users 4096 2005-03-09 11:39:31.000000000 +0000 src
drwxr-xr-x  16 1002 users 4096 2005-03-09 11:39:31.000000000 +0000 update

The installation is initiated thus: 

[root@delos OME]# perl install.pl

This initiates a brief question-and-answer session, following which the installation proceeds. For the post part, default directories and usernames can be used. The installation may ask to install additional Perl modules: these should be allowed.

The installation log file is, by default, written to /var/tmp/OME/install/. The Perl source files for the running OME system are installed to /usr/lib/perl5/site_perl/5.8.5/OME/.... HTML template files are copied to /OME/html/Templates/. Look fornother files under /OME/.

To check C library dependencies prior to installation: 

perl install.pl -l

To check Perl library dependencies prior to installation: 

perl install.pl -c

To install a Perl library: 

perl -MCPAN -e 'install <full-module-name>'

First time, configuration is set up by a long-ish dialog. Defaults can be used for many options (just opress RETURN). See also: [WWW] http://search.cpan.org/~jhi/perl-5.8.0/lib/CPAN.pm

Sometimes, a self-test failure causes installation to fail. This can be overcome thus: 

perl -MCPAN -e shell
cpan> force install <full-module-name>

2.5. Problems with GD / libGD

See: [WWW] http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2004-September/000189.html

Needs: yum install xorg-x11-devel, then the instructions for installing GD from this message do indeed work. 

GD has unfortunately become a problem child.  Last I checked (several months ago), there was no perl RPM for GD.
There is a package for Fedora called something like (this is for a 64-bit OS):
  gd-2.0.21-2.x86_64.rpm
You might as well get these too (I think that's all of them)
  gd-progs-2.0.21-2.x86_64.rpm
  gd-devel-2.0.21-2.x86_64.rpm
download and install that using yum, red carpet or whatever (the version number doesn't matter very much - use "the latest").  Get the Perl bindings for it here:
  wget http://search.cpan.org/CPAN/authors/id/L/LD/LDS/GD-2.16.tar.gz
then:
  tar -zxvf GD-2.16.tar.gz; cd GD-2.16; perl Makefile.PL
answer the questions (just hit return), then
  make
  make test
(ignore any errors here)
and finally:
  sudo make install
(or su to become root then 'make install')

2.6. Problems with installation self-test

These problems seem to be related to a failure to install GD as described above, or are just Perl module self-tests whose failure doesn't affect OME. If loading Perl modules as part of the OME installation they cause the installation to fail, so it's probably best to load the perl modules by hand as described above. 

FAILURE TESTING MODULE -- OUTPUT: "PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
 t/caller..........FAILED test 5
        Failed 1/10 tests, 90.00% okay
 t/carp_default....FAILED tests 10-11
        Failed 2/11 tests, 81.82% okay
 t/carp_file.......FAILED tests 10-11
        Failed 2/11 tests, 81.82% okay
 t/carp_silent.....FAILED test 2
        Failed 1/2 tests, 50.00% okay
 t/default.........ok
 t/default_exp.....ok
 t/file............ok
 t/priority........ok
 t/tag_callback....skipped
         all skipped: no reason given
 t/tag_string......ok
 Failed 4/10 test scripts, 60.00% okay. 6/79 subtests failed, 92.41% okay.
 Failed Test      Stat Wstat Total Fail  Failed  List of Failed
 -------------------------------------------------------------------------------
 t/caller.t                     10    1  10.00%  5
 t/carp_default.t               11    2  18.18%  10-11
 t/carp_file.t                  11    2  18.18%  10-11
 t/carp_silent.t                 2    1  50.00%  2
 1 test skipped.
 make: *** [test_dynamic] Error 255

2.7. Apache configuration problems

If, during installation, the following report is seen: 

[Thu Jul 07 18:03:34 2005] [warn] The ScriptAlias directive in /OME/conf/httpd2.omeis.conf at line 2 will probably never match because it overlaps an earlier ScriptAlias.
Testing Apache configuration
  \__ mod_perl [FAILURE].


Errors executing task: Apache/mod_perl is not properly configured.
GATEWAY_INTERFACE is "CGI/1.1" in CGI test instead of "CGI-Perl/1.1".
CGI test: http://localhost/perl2/mod_perl_test.pl
See /var/tmp/OME/install/ApacheConfigTask.log for more details. at (eval 158) line 1

 at install.pl line 80
        main::run_tasks() called at install.pl line 249
# Looks like your test died before it could output anything.
[root@delos OME]#

It may be symptomatic of using an old version of OME (see installation Q&A section above). It seems that the GATEWAY_INTERFACE value has changed with later versions of MOD_PERL. Doing a direct retrieval of the test URI to a browser yields this response: 

Content-Type: text/plain

SCRIPT_NAME = /perl2/mod_perl_test.pl
SERVER_NAME = localhost
SERVER_ADMIN = root@localhost
HTTP_ACCEPT_ENCODING = gzip,deflate
HTTP_CONNECTION = keep-alive
REQUEST_METHOD = GET
HTTP_ACCEPT = text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
SCRIPT_FILENAME = /OME/perl2/mod_perl_test.pl
SERVER_SOFTWARE = Apache/2.0.52 (Red Hat)
HTTP_ACCEPT_CHARSET = ISO-8859-1,utf-8;q=0.7,*;q=0.7
QUERY_STRING = 
REMOTE_PORT = 32777
HTTP_USER_AGENT = Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.8) Gecko/20050523 Red Hat/1.0.4-1.4.1 Firefox/1.0.4
SERVER_PORT = 80
SERVER_SIGNATURE = <address>Apache/2.0.52 (Red Hat) Server at localhost Port 80</address>

HTTP_ACCEPT_LANGUAGE = en-us,en;q=0.5
REMOTE_ADDR = 127.0.0.1
HTTP_KEEP_ALIVE = 300
SERVER_PROTOCOL = HTTP/1.1
PATH = /sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin
REQUEST_URI = /perl2/mod_perl_test.pl
GATEWAY_INTERFACE = CGI/1.1
SERVER_ADDR = 127.0.0.1
DOCUMENT_ROOT = /var/www/html
HTTP_HOST = localhost
MOD_PERL = mod_perl/1.99_16

2.8. PostgreSQL configuration and administration

The OME installation creates a postgreSQL database called ome.

The default installation of postgreSQL, per Scientific Linux 4, has its data files at /var/lib/pgsql. IN particular, /var/lib/pgsql/data contains configuration files: 

[root@delos data]# ls -al /var/lib/pgsql/data/
total 100
drwx------  6 postgres postgres 4096 Jul  8 12:53 .
drwx------  4 postgres postgres 4096 Jul  8 12:53 ..
drwx------  5 postgres postgres 4096 Jul  8 13:21 base
drwx------  2 postgres postgres 4096 Jul  8 16:00 global
drwx------  2 postgres postgres 4096 Jul  8 12:53 pg_clog
-rw-------  1 postgres postgres 3396 Jul  8 12:53 pg_hba.conf
-rw-------  1 postgres postgres 1441 Jul  8 12:53 pg_ident.conf
-rw-------  1 postgres postgres    4 Jul  8 12:53 PG_VERSION
drwx------  2 postgres postgres 4096 Jul  8 13:38 pg_xlog
-rw-------  1 postgres postgres 7821 Jul  8 12:53 postgresql.conf
-rw-------  1 postgres postgres   59 Jul  8 12:53 postmaster.opts
-rw-------  1 postgres postgres   45 Jul  8 12:53 postmaster.pid

In order to manipulate the ome database, log in as user postgres, then issue the command 'psql ome'; e.g. 

[root@delos data]# su postgres
bash-3.00$ psql ome
Welcome to psql 7.4.8, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help on internal slash commands
       \g or terminate with semicolon to execute query
       \q to quit

ome=#

The ome utility can be run as user postgres to list, add and modify OME user details. Be aware that there are three levels of user identification here: Linux users, postgresql users and OME users, examplified by: 

less /etc/passwd                 # Linux users

psql ome
ome=# select * from pg_shadow ;  # postgresql users

ome admin users list             # OME users

2.9. Accessing images on the VMware host filesystem via HGFS

Under VMware, guest file system access to the host system Windows file system presents all files with owner root, group root, and owner access only. But the access controls are not enforced and any logged in user can read/write any file that the VMware host can access. cf. [WWW] http://www.vmware.com/community/thread.jspa?threadID=6079&tstart=0.

It appears this incorrect reporting of file access permissions this is sufficient to cause problems for OME. These problems can be overcome by patching the VMware HGFS driver.

A patch is reported here: [WWW] http://www.vmware.com/community/thread.jspa?forumID=19&threadID=6079&messageID=40291#40291.

I have successfully applied this patch using the following procedure. 

cd /usr/lib/vmware-tools/modules/source/
tax xvf vmhgfs.tar
cd vmhgfs-only
# edit driver.c to apply patch:
# find:
#   inode->i_mode |= (attr->permissions << 6); /* Only applies to owner. */
# replace with:
#   inode->i_mode |= (attr->permissions * 0111); /* apply to all. */
make
# new file /usr/lib/vmware-tools/modules/source/vmhgfs.o
uname -r                           # gives value for <kernelid>
cd /lib/modules/<kernelid>/misc/   # e.g. cd /lib/modules/2.6.9-11.EL/misc/
                                   # or   cd /lib/modules/`uname -r`/misc/
mv vmhgfs.o vmhgfs.o.save
cp -p /usr/lib/vmware-tools/modules/source/vmhgfs.o .

Now reboot the system (maybe there's an easier way, but reboot works).

This procedure may well need to be repeated if (when) the kernel is updated by an automatic patching service. In that event, the VMware tools will probably need re-installing. If VMware tools are re-installed, it is possible that the applied source code patch will be lost and will need to be re-applied.

2.10. OME admin and test user setup

su postgres
psql ome
ome=# create user omeadmin with createdb createuser ;
ome=# select * from pg_shadow ;  # confirm postgresql users
ome=# \q
ome admin users list             # OME users
exit # to original shell

su omeadmin
ome admin users list

At this point, it should be possible to log in as ometest, and import some test images into OME.

2.11. Moving the OME environment to a different host

Clone the VMware machine with installed OME, and tidy up the new system thus:

I seems that the host name is fixed in the OME database when the system is installed, so the database must be re-built when the software is moved to a new host. So far, the only way I have found to do this is to repeat the installation, after:

To delete the old OME database:

When the new installation is done, use the ome utility to re-create the omeadmin and ometest users as described previously. (Use the account created during OME installation for this purpose.)

It should now be possible to log in to OME as user ometest using a web browser.

2.11.1. General Instructions for Moving an OME database

(These instructions were provided by Chris Allan on the OME-Users list, at [WWW] http://lists.openmicroscopy.org.uk/pipermail/ome-users/2005-July/000084.html)

  1. Move your OME root directory over to the new machine.

  2. Dump the database and restore on the new machine using pg_dump and pg_restore.

  3. (using psql...?) UPDATE repositories SET image_server_url '[WWW] http://newhost/cgi-bin/omeis' WHERE attribute_id $id;

  4. Add OME Apache directives to httpd.conf. This of course assumes many things like the architecture of the two machines being the same, the GLIBC versions being the same, etc. If they're not you'll want to run the installer after Step 3.

2.11.2. Other related postings to the OME users list

2.12. Other post-installation notes

Other system configuration details to consider include:

3. OME Customization

For the DTGED project, we wish to capture the following information with each image:

  1. Date/time/user

  2. Strain of drosophila (wild type or identified mutant). Maybe two or more strains in an image.

  3. Gene identifier (target of hybridization probes)

  4. Slide identification (a unique identifier is required so that images from the same slide can be compared; for practical considerations, the slide preparation date and strain labels will be incorporated in the identification string.

  5. Developmental stages at which gene expression is observed (from an controlled vocabulary to be defined)

  6. Microscope magnification - objective (4x, 10x, 20x, 40x)

  7. Microscope magnification - optivar (1, 1.25, 1.6, 2)

Item 1 is captured automatically by OME, so no cutomization is needed there. The remaining information needs to be captured in one or more "semantic types", values for which are associated with each image.

The plan is to use a dataset to group a series of observations made from a single slide (possibly in a single session, possibly from multiple sessions). Thus, information that is common to all images in a dataset would usefully be entered just once for the dataset, even if it is stored with per-image information. Items 3 and 4 are per-slide values. A slide may contain multiple strains, but individual images may be of just one strain, so the range of values for item 2 is per-slide, but the.

Item 5 is strictly per-slide.

Item 6 is covered by the existing "Intsrument" semantic type, but item 7 is not. Instruments are described globally. I propose in the first instance to store items 6 and 7 as part of the per-image data. Then, I will look to extend the above list of items with a reference to an instrument. Later, I hope to find a way to associate an intrument and lens descriptions with the dataset and create per-image descriptions that use the per-dataset value as a starting default.

3.1. Create a new semantic type for DTGED image annotations

OME uses the term "semantic type" to mean a structured data type. It's not clear to me what the designation "semantic" actually means, since OME "semantic types" appear to have the same kinds of semantics as any programming language data type, or XML schema data type. I think a designation "application type" might more closely reflect their use, since they are used in constructing applications of the OME core engine. (An explanation I have received is that the semantics of the data is defined completely by the data type itself, which I think is more of a statement about how the data types are used than about any actual meaning inherent in the datatype.)

Directory src/xml/OME/ in the OME source distribution contains semantic type definitions. Two examples of these are src/xml/OME/Annotations/annotations.ome and src/xml/OME/Annotations/classification.ome. Using these examples as a starting point, create a new semantic type for DTGED image annotation:

File /home/omeadmin/DTGEDAnnotation.ome: 

<OME:OME
  xmlns:OME=
    "http://www.openmicroscopy.org/XMLschemas/OME/FC/ome.xsd"
  xmlns:STD=
    "http://www.openmicroscopy.org/XMLschemas/STD/RC2/STD.xsd"
  xmlns:xsi=
    "http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation = "
    http://www.openmicroscopy.org/XMLschemas/OME/FC/ome.xsd
      http://www.openmicroscopy.org/XMLschemas/OME/FC/ome.xsd
    http://www.openmicroscopy.org/XMLschemas/STD/RC2/STD.xsd
      http://www.openmicroscopy.org/XMLschemas/STD/RC2/STD.xsd">

<STD:SemanticTypeDefinitions>

  <STD:SemanticType
      Name = "DTGEDImageAnnotation"
      AppliesTo = "I">
    <STD:Description>
        Image annotation data for the DTGED project.
    </STD:Description>

    <STD:Element
        Name = "SlideID"
        DBLocation = "IMAGE_DTGED.SLIDE_ID"
        DataType = "string">
      <STD:Description>
          A string that uniquely identifies the slide from which the image
          was made.  
          Suggested form is yyyymmdd-s1[-s2][-x], where s1 and s2 are tags
          indicating drosophila strains used to make the slide.
          Preferably, the value for this field will be inherited from the
          containing dataset, since a dataset will be a series of images 
          from a single slide.
      </STD:Description>
    </STD:Element>

    <STD:Element
        Name = "Strain"
        DBLocation = "IMAGE_DTGED.D_STRAIN"
        DataType = "string">
      <STD:Description>
          Strain or strains of drosophila shown in image.
          (Wild-type or identified mutant.)
      </STD:Description>
    </STD:Element>

    <STD:Element
        Name = "Gene"
        DBLocation = "IMAGE_DTGED.D_GENE"
        DataType = "string">
      <STD:Description>
          Gene whose expression is exposed by the applied 
          in situ hybridization probes.
      </STD:Description>
    </STD:Element>

    <STD:Element
        Name = "DevStage"
        DBLocation = "IMAGE_DTGED.DEV_STAGE"
        DataType = "string">
      <STD:Description>
          Stages of development at which gene expression is observed.
          This value is determined by an experimenter studying the 
          image of in situ hybridization, and is a set of terms selected 
          from a defined vocabulary.
      </STD:Description>
    </STD:Element>

    <STD:Element
        Name = "ObjectiveMag"
        DBLocation = "IMAGE_DTGED.OBJECTIVE_MAG"
        DataType = "float">
      <STD:Description>
          Microscope objective lens magnification factor.
          This will, for preference, not be defined as a separate value 
          but the value defined by type "ImageInstrument" will be used.
      </STD:Description>
    </STD:Element>

    <STD:Element
        Name = "OptivarMag"
        DBLocation = "IMAGE_DTGED.OPTIVAR_MAG"
        DataType = "float">
      <STD:Description>
          Microscope optivar magnification factor.
          (The optivar is an additional lens that increases
          the objective magnification by a small factor:
          x1, x1.25, x1.6 or x2.)
          This value should probably be part of the 
          "ImageInstrument" type.
      </STD:Description>
    </STD:Element>

  </STD:SemanticType>

</STD:SemanticTypeDefinitions>

</OME:OME>

Notes:

3.2. Import the new semantic type into OME

Semantic types cannot be removed. Before adding any semantic types to OME, I suggest making a backup of the OME database (but not necessarily the image store), by issuing the following commands with root privileges: 

ome admin data backup -q /store/backup/ome_backup.tar.bz2

Any semantic types added (along with any new data entered) can be removed by restoring the backup using: 

ome admin data restore -q /store/backup/ome_backup.tar.bz2

Working as user omeadmin, the type definition file (as above) is created in the omeadmin home directory, as /home/omeadmin/DTGEDAnnotation.ome. Any of the available import mechanisms can be used to import the new semantic type definition (e.g. the web interface, but I think that would limit the directories from which the new definitions could be imported). I have chosen to use the "ome commander" utility, as follows: 

[omeadmin@helenome ~]$ ome import import DTGEDAnnotation.ome 
Importing files
         1/4: [IN PROGRESS] Starting import
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "image_dtged_pkey" for table "image_dtged"
         4/4: [FINISHED] Imported 0 images from 0 files. 1 scanned. 0 unknown format, 0 duplicates, 0 errors.
Exiting...
         4/4: [FINISHED] Imported 0 images from 0 files. 1 scanned. 0 unknown format, 0 duplicates, 0 errors.
Exiting...

Now use a web browser to access OME, and select an image. The custom annotation drop-down box contains an option DTGEDImageAnnotation; select this and then select "create" to display a new dialog box window that provides for entry of the various values described in the new Semantic Type definition.

There seem to be various reasons that the above sequence may fail. Here's a recipe that seems to work for me when updating the semantic type definitions. 

/etc/init.d/httpd stop
/etc/init.d/postgresql restart

ome admin data restore -q /store/backup/ome_backup.tar.bz2

/etc/init.d/postgresql restart

ome import import DTGEDAnnotation.ome

/etc/init.d/postgresql restart
/etc/init.d/httpd start

If, following the import, the web interface gives internal configuration errors, try shutting down and restarting httpd and postgres. Eventually, the system seems to sort itself out. 

/etc/init.d/httpd restart
/etc/init.d/postgresql restart

3.3. Refine the new semantic type definition

Note that the new fields are ordered randomly in the display. To force a display order, I tried incorporating numbers into the field display names and into the database field names, but that didn't work -- it just re-shuffled the fields.

3.4. Semantic type references

3.5. Stuff - for later

    <STD:Element
        Name = "xxxx"
        DBLocation = "IMAGE_DTGED.XXXX"
        DataType = "xxxx">
        DataType = "reference"
        RefersTo = "xxxx"
      <STD:Description>
          xxxx
      </STD:Description>
    </STD:Element>

-- GrahamKlyne 2008-11-18 03:21:47

last edited 2005-09-29 12:17:32 by GrahamKlyne