[OPUS]

Frequently Asked Questions

Users may notice that this version of OPUS contains a sub-version letter (the 'b' in 5.4b). This is simply because this particular configured build of OPUS corresponds to an internal STScI software delivery at the 5.4b level - a rebuild of 5.4 with some enhancements. Release 5.4b is just as complete and well-tested a build as any other public OPUS release has been.

What's New in OPUS 5.4b?

What's New in OPUS 5.4b?

How does the OPUS release 5.4b differ from the last one?

Several enhancements, bug fixes, and repairs were made to the OPUS OAPI (OPUS applications programmers interface), the blackboard servers, and the Java Managers. These modifications have made the pipeline environment more robust, more configurable, and more maintainable.

From the user's perspective there are several modifications which have been suggested by people using OPUS around the world, and we hope the current release addresses the most important of those concerns.

This version of OPUS supports Solaris, AIX, and multiple Linux operating systems. Click here for specifics. Because there are many flavors of Linux currently competing for dominance in the marketplace, we have striven to support some of the most popular types. If you have questions or suggestions on this topic, please contact us at opushelp@stsci.edu.

What is a PR?

A PR is a Problem Report. The problem reporting system affects all components at the Space Telescope Science Institute and includes enhancements, change requests, as well as documentation updates. Not all of the 50,000+ problem reports have been filed against OPUS!

49555: Add the OSF trigger name as an ENV var

For external pollers, the OSF_EVENT environment variable has been made available, in case a process would like to know the OSF trigger name from the resource file.

49655: OMG needs enhanced View -> Dataset Log feature

The old Motif OMG had quite an elaborate system for viewing dataset-related text files from the GUI. When the OMG and PMG were ported to Java, only a small fraction of that capability was retained, even though the format of the omg_view.dat file remained unchanged. The Java OMG did not use the "x.x.x.name = " or the "x.x.x.filespec = " keywords from that file. It simply used the "x.x.x.dir = " line to find the correct directory, and then made some naming assumptions to find the *.trx files.

For this PR, the format of the file is simplified to reflect only what the OMG is actually doing, and the 'filespec' parameter is now again supported, though without wildcards. The existing Java OMG did not support either the 'filespec' parameter or wildcarding.

Dataset log files are now viewable again from the OMG GUI. Until now, the OMG log-viewing capabilities have been extremely limited. With this change, directory and filespec parameters are tunable. Wildcarding is not yet supported. This also allows the *.trx files of the Sample Pipeline to be viewed. To try this out, double-click on an OSF in the OMG.

For more information on this, click here.

49769: HEASARC fitsverify tool (CFITSIO) needs modification for HST

This PR entails modification of a CFITSIO tool (fitsverify) for use by HST pipeline applications. It does however affect the libfits_public.so library distributed with OPUS. The version of CFITSIO now being used with OPUS is 2.470 - 18 August 2003.

For more information on CFITSIO changes in OPUS, also see PR 51411.

50191: OMG/PMG tabbed-pane needs fix for drag-and-drop behavior

Drag and drop and tabbed panes are buggy in the Java VM and must be fixed to allow tabbed panes to have things dropped into them. Simple code changes can be made to do this but you have to extend the tabbed pane class and override some stuff in the OMG and perhaps the PMG.

Fixed. One side benefit that we have made is that the tabs no longer shift in the PMG when selected.

50263: Fix OPUS.make so that managers get correct build date

The current Makefile for the OPUS managers is not correctly handling the updating of the 'build date' property. This *should* be updated any time the managers are built, whether from a make-clean or not.

The result is managers in 5.0 have build dates of 12/2/2003 and 8/9/2001, both of which are wrong.

The correct version dates should now appear in the manager's Help->About dialogs. This will correspond to the last time the managers were built before the installer package was created.

50360: OPUS Sample Pipeline cleandata step is not cleaning up data

It looks like at some point (possibly during 44974), the cleandata code was changed and the Sample Pipeline wasn't checked. The fix is to simply add the 'GIF' class to a resource list inside the cleandata.resource file. Without this, the cleandata logic skips all the Sample Pipeline resource files since none of their CLASS values match the default.

It will now actually delete all of the files it is supposed to, and this is shown in detail in the process log file.

50465: pscluster.pl may fail on Linux

pscluster.pl fails if the location of 'id' is not /bin/id.

Since id exists as /usr/bin/id on Solaris,RH8,RH9,Fedora,AIX, and is likely to be the most common location, it was changed from /bin to /usr/bin.

50619: Make OMG and PMG status areas easier on the eyes

Two minor changes are needed to the OMG and PMG status areas:
1) Selection of the first PMG tab does not update its status area. Selection of all other tabs works correctly.
2) Change the format of the status message so it is easier to read.

The OMG and PMG status areas will now display status info in a new format which is hopefully more easy to read. One example is: 

   Status:    220 idle,   30 halted,   250 total

50653: 4.5 and 5.0 OPUS Mgrs do not install correctly on Win XP

During installation on XP (InstallAnywhere GUI), the "Full" option is not presented, only "Typical". The result is that the Windows C++ libraries are not installed, and when the user tries to run the OMG or PMG, the application complains that it cannot find these libraries.

This has been fixed. The installer should allow "Full" on both 2K and XP.

50665: OMG uses "red" backgrounds for non-trouble stage status values

The Ingest pipeline uses the TSTATUS value "d" in the RM (Request Manager) stage when it detects a duplication condition . This "trouble" value is correctly displayed in the OMG with a red background. The pipeline also uses an NSTATUS value "d" in the UP (Update) stage to indication that it is "done" with alarm signaling. This "non-trouble" value is incorrectly displayed in the OMG with a red background, and this is very misleading to users.

An analysis of the OMG code reveals that the OMG will display a status value with the trouble background color (red) if that value is labeled as TSTATUS for any stage in the pipeline.

The OMG code should be changed to display a status value with a red background only if that value has a TSTATUS label for the stage designated by the OMG column name.

Fixed. The revised OMG allows a stage status value used for a trouble state in one stage to be reused for a non-trouble state in a different stage, without the non-trouble state being erroneously highlighted by a red background.

50973: Enhance "OMG -> View -> Dataset Log" to do replace-all's

There are reserved words that are used in the dataset_views.dat file, which are necessary for dataset log viewing (e.g. "<dataset>", "<data_id>", and "<dcf_num>"). The first occurrence of each of these words in said files is replaced with their actual value. This should be changed so that ALL occurrences are replaced.

This has been fixed. The text dialog pop-up facility was also fixed so that copies to the system clipboard and text searches would work correctly during dataset log viewing.

For more information on this, click here.

49903: Ingest Project: HST OPUS pipeline (change to IDL on OAPI)

This PR is for an addition to the HST pipeline, but it included a change to the OAPI. The field 'eventname' was added to the OPUS.OPUS_API.OPUSUSER.Event IDL struct. Since this particularly affects Java and Python Internal Pollers, all OPUS users which have compiled the distributed IDL should now recompile it with their applications.

50605: Port OPUS to 64-bit Solaris

This PR changes the way the OPUS programs are compiled and linked so that they are now 64-bit executables and libraries on Solaris. 32-bit objects are still being created on all other supported operating systems.

OPUS users should pay special attention to the new settings in the delivered opus_login script. Also, users of Java programs on Solaris will have to switch to the 64-bit version of Java for any Java program which loads a native 64-bit OPUS C/C++ library. 32-bit Java may be used for non-OPUS programs.

There are no new Windows OMG/PMG necessary for this PR. The existing 32-bit windows Managers will communicate correctly with 64-bit Solaris servers.

Users on Solaris will notice that this removes the 256-process limit due to file descriptors. The tested limit on the maximum number of OPUS processes running on Solaris is now around 500 (not related to file descriptors) during nominal operations. This maximum is not yet known for Linux or AIX.

While we were at it, we fixed a few other items:

Users on a single node will also notice that the rsh/ssh utilities will no longer be used to start processes selected in the PMG. They are now spawned directly on the running node, when the servers and the process are running on the same node. This will slightly speed up process-starting, and will help reduce the number of process startup failures during large (e.g. 100) group submissions.

Users on AIX and Linux will also note that the extra "zombie processes" will no longer accumulate, waiting until the opus_env_server is cycled. In certain situations, this accumulation could have bumped up against the user's number-of-processes limit. This has been fixed - there should now be NO zombie (defunct) processes occurring as a result of the startup of OPUS processes from the PMG.

48524: starting from a pipeline file lower cases resource file names

I did a test to check on the report from an external user and sure enough, if you have a mixed case resource file name, eg. Cdb_Pol.resource and you try to start it from within a pipeline file: 
Cdb_Pol cdbs odocluster1
When you drop the pipeline file into the right-side pane of the PMG, the starting PSTAT appears with the process name all in lower case. When you try to start that process, the proc_stat changes to starting, but never gets any further. In the pmg log file you find an exception:

05/01.16:32:42-W-PMGStart: no password required OPUS.JOPUS.JOpusException: OPUS.JOPUS.JOpusEnv.getResourceValue: The server reports a bad argument(s): OpusEnv_impl::getKeyValue - An OAPI exception was thrown: An exception of type No_entry has occurred.

This has been fixed. Dragging a pipeline file across the PMG to start a group of processes will no longer lowercase its contents before trying to use it. Mixed case resource file names are allowed.

49165: Use of longs in calculations may be error prone

A survey of the code was performed, searching for improper use of long integers, or cases where calculations would cause overflows. Some potential problems were found, and fixed.

Some better exception handling was added to certain parts of the OPUS servers as well.

51503: improper constants can cause poor behavior during heavy processing

Running the servers during heavy loading conditions (processing data in the 300 to 500-process regime) has illuminated a problem with some OPUS source code. In at least one case, of multiple tests, this problem was found to be clearly at fault for erratic server behavior. This would be an intermittent problem, stable within one build, but not within another. It appears that we have been getting lucky with this for the most part. A short summary of the problem is that the runtime success of a C++ class constant actually having the correct value is NOT guaranteed unless certain steps are taken, such as by adding the re-declaration "const int X::RETRY_INTERVAL;" inside the definition code (.cpp file), or by changing the declaration to what is known as 'the enum hack'.

A survey of the OPUS code turned up one such case. It is a set of two constants responsible for controlling the timing of retry attempts when a call is made to find a CORBA server object (or start one if necessary). During some stress testing with a developer build, it was found that one such constant was set to integer garbage (a very high number), and thus a request for a CORBA blackboard object appeared to hang indefinitely.

A problem was also corrected where process startup errors were being hidden. All process startup errors should now either appear in the terminal window or the process log file.

51412: General resource file cleanup PR for OPUS 5.4 (and OmniORBPy change)

This was a PR for general HST resource file clean up issues, which affected the public version of OPUS.

The DISPLAY_ORDER keyword appeared in many if not all resource files, in both share and opus, and it is no longer used by anything. It was once important to the old Motif PMG. It has been removed.

This PR includes a change to OPUS to use a newer version of omniORBpy, which will run with Python 2.3. The current version in use (and tested) with OPUS is OmniORB 4.0.3 with OmniORBPy 2.3. Users will note that the Python Internal Poller example now describes how to work with this newer version.

49860: Update OPUS Windows Managers to use ACE 5.4 (and IDL change)

The Windows C++ libraries for the OMG/PMG have been rebuilt. A new InstallAnywhere installer has been created and must be used with this build. Old versions of the OMG/PMG will have problems with the new servers, and vice versa. 
*** Your old PC managers must be UNINSTALLED first.  Do NOT simply ***
*** install the new managers on top of the old ones!               ***
Users may also notice speedier OMG path-switching (for pre-opened paths) with some pipelines. See the ~11 second delay mentioned in PR 49254.

Some OPUS IDL member names were changed for compliance with standard CORBA IDL compilers, so any users who have created their own internal pollers will have to recompile the included OPUS IDL files for Python and Java. C++ internal pollers may simply take advantage of the changes as delivered in the new headers and liboapi.so.

AIX users will also notice that their OPUS distribution is reduced in size from OPUS 5.0. Do not be concerned, this is intentional. Version 5.0 used a static AIX OAPI library which was linked into every executable. This ballooned the total size. With this PR, this library is now being built as a shared object on AIX, and the distribution size is favorably reduced.

Also, a bug was fixed which caused class-loading errors during the startup of the OMG under Java 1.4.2_05 and higher.

51754: OPUS applications need easier access to their log file name

An external OPUS user brought up the need for an OPUS application to know its own log file name. His particular use is to email a copy of it to an operator in case of an error. The name of a log file can be determined from its component fields, all of which are available to the process either directly (as env vars) or indirectly through UNIX command-line calls (e.g. call "uname" to get the node name, and call ps twice with grep/awk/opus_id to get the hex pid value).

This has been made much easier by providing an env var to the process with the information, named OPUS_LOG_FILE.

49254: Solaris SunFire seems to respond more slowly to OMG/PMG

When a new path is opened in the OMG, there is a time delay to allow for that blackboard server to start. This is normal. But when a user is viewing a path and switches back to a pre-opened path (File -> Open path), the action should occur relatively quickly (less than a second if it only has a few OSFs) because the server is up and running already.

This was found to be causing an extra 11 second delay EVERY time a path was switched to in the OMG, whether pre-opened or not. Since this would only occur in configurations where a PC manager would connect to a Solaris server, it was assumed to be a network/CORBA difficulty.

This has been fixed as a side-effect of upgrading the version of ACE/TAO in PR 49860.

51977: osf_update should allow input criterion of timestamp

The "-x timestamp_in_hex" argument pair was added to osf_update.cpp to allow for further qualifying an OSF specification. This is the same as the osf_test syntax.


Top of What's New FAQ

Top of OPUS FAQ