Python Internal Poller Example

What is the Python Internal Poller Example ?

OPUS is distributed with a sample pipeline which demonstrates some of the capabilities of the OPUS system. That pipeline includes both internal and external pollers as pipeline applications. Up until version 4.2, all internal pollers needed to be written in C++. Pipeline application writers can now take advantage of new OAPI capabilities to create applications in other languages, using CORBA to interact with OPUS as internal pollers. If desired, internally polling applications can now be written in any language for which a CORBA binding exists. Of course, externally polling applications can, as always, be written in any language.

This Python example is included to illustrate such an application. It is written in Python and runs on Solaris 2.8 using the free OmniORBpy CORBA ORB. There is some preliminary setup required, but hopefully that will be made less complicated by the following information.
What do I have to do to set this up ?

Because this process is going to use CORBA to communicate with OPUS, you will need a CORBA "ORB" for Python on the platform on which you are running. In this example we use the omniORBpy solution. It is free and runs on the Windows, Linux, Solaris, HPUX, and Irix platforms. This is merely the selected ORB for this example - you may use the ORB of your choice. This will require installation and configuration, and after that, the appropriate OPUS files need to be created. The following steps use a relatively recent version of omniORB. If you wish to use the older AT&T Labs version (which has the advantage of including pre-compiled binaries for Solaris), we are still providing the old instructions here though they are unsupported and will not run under Python 2.3.

Follow these steps to build, install, and use omniORBpy in OPUS:

  1. Download the core ORB package (omniORB) here. Select version 4.0.3 for Unix. Unzip and untar the bundle, and build it according to the included build and installation directions.

  2. Download the Python ORB package (omniORBpy) from the same location. Select version 2.3 for Unix. Unzip and untar the bundle, and build it according to the included build and installation directions.

    We will say for this discussion that you installed both packages under "/home/joe/omni" (e.g. you used configure with "--prefix=/home/joe/omni" and did a "make install"), and that we are using Python version 2.3.

  3. Compile the OPUS "IDL" files (found under opus/src/*.idl) in some directory (say "/home/joe/omni_opus"). To do this, first follow any instructions for setup that come with omniORBpy. In general however, that entails the following changes to your environment (e.g. put in your .cshrc file):

    Then, in a new shell, compile the IDL files via:

    The resulting Python files will be used by your Python internal poller.

  4. Install the process resource file. This has been included and is located in "opus/definitions/unix/pypoller.resource". Simply copy this into your OPUS_DEFINITIONS_DIR. If you are already setup to run the sample pipeline, then you may skip this step.

  5. Install the path file. You may create your own, or you may use the included "quick.path" path file that has been installed for use with the sample pipeline. If you wish to create your own, follow the format of "quick.path", set the directory locations to your needs, then simply copy it into your OPUS_DEFINITIONS_DIR. If you are already setup to run the sample pipeline, and you wish to use "quick.path", then you may skip this step.

  6. Install the Python script. This has been included and is located in "opus/bin/pyintpoller.py". Simply copy this into your PATH somewhere. Make sure that "python" itself is also in your PATH. If you are already setup to run the sample pipeline, then it has been put in your PATH, and you may skip this step.

Now that I have configured everything, how do I run it ?

Via the PMG.

Once the Python script has been created and is in your path, and the resource and path files are located in your OPUS_DEFINITIONS_DIR directory, simply start the OPUS servers and then run the PMG (as explained in detail in the OPUS PMG documentation). Find "pypoller" under the "Processes" tree in the PMG, listed under OPUS / Processes / SAMPLE / ALL. Drag it to the "quick" path tab on the right side of the PMG and select a node to run it on in the popup window. When it is listed as "pending", use the PMG menu to start the pending process.

If all goes well, it will eventually sit, listed as "idle", waiting for the correctly named files to appear in its input directory. At this point, you will be able to view the process log file ("View" menu) and verify that the Python process has started and, as its first task, has listed all the resource values it read from the resource file, via the OAPI.

Now you can trigger pypoller to process a dataset by placing an input file into the input directory. We will do this manually via the file system, for this example. Create an empty file named "workfile1.try" in the "input_data" directory, as listed in the path file (the input_data directory used in this example, as per quick.path, is "/home/sontag/opus_test/quick/input/"). This may be done via the unix "touch" command. Within 10 seconds (i.e., the default POLLING_TIME), the pypoller process will act on the dataset. You will be able to see the status change in the PMG and view the results in the log file. You have successfully processed a dataset though a Python internal poller using OPUS.

Note that if there is an error while processing a dataset (such as an exit from the script), the problem dataset will be renamed to "workfile1.try_bad" and moved to the "../error" directory, otherwise it will be renamed to "workfile1.try_done".

Now that I can run the example, how do I make my own ?

The most difficult part of this process was getting the example to run. If that was successful, then you can either write your own from scratch, or simply copy the example as a template. You may only want or need to edit the contents of the "processDataset()" method in pyintpoller.py, inserting the code that does your own specific processing. Or you may want to make some changes in the main loop to take more full advantage of the OAPI than is done by this example.

 

Back to the Top

Top of OPUS FAQ