![[OPUS]](opus.gif)
Note - The following was last updated in 1996. It is included here for reference. Some details concerning OPUS software are likely to be out of date.
The pipeline system at the Hubble Space Telescope Science Institute was designed in an era when monolithic mainframes were the norm, and tightly coupled, narrowly controlled systems were the result.
Changing requirements are no longer seen as an exception in the software evolution process. We have learned that change is now part of the process, and we must build systems that are adaptable. Monolithic special-purpose systems have a proven track record of failure to adapt.
OPUS, the second generation pipeline, is implemented as a fully concurrent blackboard system. As a result OPUS is now a flexible, extensible and robust system. By distributing the processing load across several workstations the system can make better, if not optimal, use of an institution's resources. By using the concurrency protection built into the operating system, the blackboard architecture ensures a robust solution as well as a cost effective one.
The Post Observation Data Processing System (PODPS) for the Hubble Space Telescope Science Operations Ground System was designed in the days of monolithic mainframes under the expectation that problems and bottlenecks would be resolved over time by bigger, faster computers. As either a coincidence or a consequence, PODPS processes are tightly coupled to other processes, and modifications to this rigid architecture are as difficult to implement as they are to maintain.
The PODPS ``pipeline" currently includes several major processing steps organized in a linear sequence: each step builds on the previous step. The current pipeline is designed for data concurrency: each step in the pipeline can be busy processing a different data set. In addition there are both variations and extensions to the sequence depending on the nature of the data set being processed.
With its monolithic architecture the PODPS pipeline continues to run on a single CPU despite the availability of significant resources on the cluster: over 30 workstations and many gigabytes of disk space. As the volume of data increases, as the amount of work done on these data increases, and as other reporting and monitoring requirements expand, it has become increasingly evident that the limitations imposed by the existing pipeline system are becoming a serious bottleneck.
We have proposed a solution to this bottleneck, opening up the architecture and taking advantage of the resources already available in the pipeline environment. Without touching the internal components of the system, OPUS, the second generation pipeline at the Institute, decouples the PODPS architecture, constructs a system with limited dependencies, and develops an architecture where global knowledge of the system's existence is unnecessary. Using a simple blackboard architecture such a (partially) object-oriented system allows multiple processes to be distributed on multiple nodes over multiple paths.
A key concept which helped in the development of a solution is well expressed in Penny Nii's introduction to Blackboard Architectures and Applications:
``...the blackboard approach provides freedom from message-passing constraints. The message-passing paradigm, although modular, requires a recipient of the message as well as a sender. Often, the recipient is not known, or the recipient might have been deleted. In the blackboard approach, the ``message" is placed on the blackboard, and the developer of the module is freed from worrying about other modules.'' [p.xxviii]
The coupling implied by message-passing systems exactly describes the difficulties we experience with the Institute's pipeline. The PODPS controller process has to know where to send all the messages, and if a recipient dies, control in the entire system enters an ambiguous state. Throughput in the system is impacted as much by keeping and recovering control as by the volume of observations to be processed. If blackboard architectures can assist in decoupling processes in only this way, then it could certainly serve as a model for the new system.
As she, and others, point out there are three components of a blackboard implementation:
Lawrence S. Baum (et al.) in presenting ``The Erasmus System" places three requirements on the implementation of the blackboard: communication efficiency, representational adequacy and organizational efficiency. Assuming that the blackboard is flexible enough to represent the problem domain at meaningful levels for that domain, it is clear that, as the exclusive communication channel between concurrent, parallel and distributed knowledge sources, the blackboard must be designed to optimize that communication.
The idea of communication efficiency is central and has to include criteria not only of speed, but also of robustness, reliability, flexibility and generality. In a distributed multiprocessing environment the blackboard must be easily visible from any place in the system, and must resolve deadlocks and other contentions which are implicit in a concurrent processing scenario.
The blackboard database is, of course, the central component of these systems, and Baum discusses several forms which have been used: lists, named structures, objects, and frames. One of the forms which has not been considered in the literature, and the one selected in the implementation of OPUS, is the file system.
A central component of every operating system is its file system, that core of routines which map and manage the physical device space: optical disk, magnetic disk, tape, even memory at times. This system is platform independent in the sense that every operating system has one; it is robust to the extent that the operating system is a mature one.
In today's environment of clusters or networks of workstations, file systems have had years of evolution where issues of efficiency and robustness have been succesfully addressed. In the VMS environment the HSC Controllers effectively link clusters of computers to common devices; in the UNIX world NFS (Network File System) controllers are now taken as a matter of course. Operating systems have addressed many of the same issues which are today being discussed in the domain of blackboard architectures. Our intent is to use the file system as an ``off-the-shelf" blackboard.
While he did not consider the file system as a potential blackboard candidate Baum suggests five criteria on which to evaluate blackboard schemes:
Knowledge sources (KS) in the literature are the most abstract of these components. In all cases these are the problem domain-dependent components of the system which actively enable the system to reach a solution. In the case of the OPUS system the ``knowledge sources" are the processes, the tasks, the executables which transform, analyze, and otherwise operate on the individual astronomical observations. The goal of the system is to convert spacecraft telemetry into scientifically useful information. Each process in the pipeline ``knows" how to do something useful to the data in order to achieve that goal.
As noted by Penny Nii (p.xxiii):
``One way to differentiate problem solving approaches is by the amount of task-specific knowledge needed or used to arrive at a solution. From this perspective, knowledge-poor systems must rely on search techniques for problem solving. Knowledge-rich systems use recognition techniques for problems solving.... At any particular computational state, instead of generating and evaluating the possible next states, a recognition system simply knows what the next state should be.''
For the most part OPUS is a knowledge-rich system in that sense: once an observation completes one stage of processing successfully, the next stage (or stages) is predetermined. The only searching required is finding the next observation on which a process is to operate. Currently, compared with the cost of processing, that search is a trivial matter.
Another way to distinguish systems of knowledge sources is the extent to which they are fine- or coarse-grained. Fine-grained systems, where the individual knowledge sources are highly decoupled and narrowly coherent, put a greater burden on the blackboard management components of the system. At this point OPUS is so large-grained as to approach boulder size. Instead of hundreds of independent, small, well defined tasks, OPUS has inherited a couple of dozen; and those are fairly massive objects. Thus communication with the blackboard is miniscule compared with the time and resources it takes to process the data.
If our objective was solely speed, our focus should be on the processes themselves and not on the communication and control aspects of the system. But our concerns are throughput and robustness; and it is the ability of blackboard systems to support a high degree of concurrency which makes that goal achievable. As discussed by Daniel Corkill [p.78] there are many levels of concurrency:
Vasudevan Jagannathan, in ``Realizing the Concurrent Blackboard Model", points out that one of the main mechanisms used in blackboarding to synchronize processing is through the use of ``trigger" conditions.
``This is an event driven blackboard system, and one obvious way to synchronize is by using appropriate trigger conditions. Trigger conditions are conditions that the blackboard process checks events against to determine whether a given knowledge source is applicable to a given situation. So if we want a knowledge source, A, to execute after another one, B, when B completes it simply generates an appropriate event which triggers A." (p.94)
The OPUS blackboard uses this trigger mechanism explicitly, but distributes control among the various knowledge sources. Specifically each knowledge source not only knows what its own trigger is, but also knows which triggers to set when it is complete. Distributing control in this fashion has its advantages and disadvantages. On the one hand a process only knows about its closest neighbors (in a control sense), and changes to one part of the system need not even be seen in another. By making the scope of control very local, we necessarily localize the impact of modifications.
On the other hand, it is easier in a system with distributed control to end up out of control. Particularly, if one knowledge source is absent from the system, it can break the entire chain of reasoning and cause a bottleneck. Even more insideous, because the scope of control is local, one might not be able to see the bigger logical picture, and this could lead to circular reasoning
(A --> B --> C --> D --> A)
Nevertheless, the extensibility and maintainability of systems with only local control is seen as an overwhelming advantage. The consistency, completeness, and integrity of the system can be assured by other means. In particular we elected to write a stand-alone compiler which is able to check, in advance, for most of the more damaging logical traps. Also an independent run-time monitor occasionally scans the systems for broken pipes.
This distribution of control is, indeed, consistent with the general blackboard approach. ``There is no control component specified in the blackboard model. The model merely specifies a general problem-solving behavior. The actual locus of control can be in the knowledge sources, on the blackboard, in a separate module, or in some combination of the three." [Nii 1986 p.39] Eliminating global control in OPUS not only resolves some major design and implementation constraints, but actually promotes modularity and extensibility in the pipeline.
The key to the OPUS system is the use of the file system as the blackboard database. It seems to be tailor-made for the problem at hand. It has a familiar hierarchical structure of directories which can be used to partition the problem domain if necessary. The file system can be ``seen" on a network or cluster environment from all nodes, and thus easily supports concurrency and parallel processing. Moreover, as a by-product of that support, the side issues of concurrency are largely resolved already by the file system.
One objection to the file system as a mechanism for anything which requires speed is the time it takes to open and close files, and the fine tuning which might be necessary in structuring those files for efficient access. Such a concern is a valid one, but one which OPUS avoids by never opening a file on the blackboard. OPUS only uses the capabilities of the file system to locate a file and to rename that file. All status files on the blackboard are empty.
When an observation enters the system the first pipeline process creates an ``observation status file" in the blackboard directory. The name of that file contains:
For example the following entries might be found on the OPUS blackboard:
19941105141608-CCCCCP_____.Y1GV0101T
19941105141628-CCCCCW_____.Y1GV0102T
19941105150000-CCW________.U0CK0101T
The first indicates that the observation Y1GV0101T (which started pipeline processing on 5 November 1994 14:16:08) is currently being processed by the sixth component of the pipeline. The second observation, Y1GV0102T, has completed the fifth step and is waiting to be processed by the sixth step. Similarly, observation U0CK0101T has completed the second step and is waiting to be analyzed by the third step.
Each process in the system polls the blackboard directory for files with a particular status. For example the third processing step will poll for files with names like *-CCW*.* which indicates the first two steps have been completed and the observation is waiting for the third process. Upon finding a candidate, the process will attempt to rename the observation status file, replacing the ``W" (waiting) with a ``P" (processing).
Because we can have multiple instances of this third process, the renaming can fail: another process might have renamed that observation status file already. This is not a problem, we just attempt to find another observation: better luck next time.
When processing for that step is complete, the observation status file is again renamed. If processing was successful, the ``P" is changed to a ``C" (complete), and the next process in the sequence of steps is flagged with a ``W'' enabling that next process to eventually recognize this observation as a candidate. If processing is unsuccessful, the ``P" is changed to an ``E" (error), and the error collector process will notice a candidate for its handling. Once all processing, including archiving, is complete for an observation the status file can finally be erased (deleted) from the blackboard.
It should be noted that using this structure OPUS can handle true data concurrency. On completion one knowledge source can set the triggers for more than one subsequent process:
19941105141642-CCCCCCW_WW_.Y1GV0103T
Here an observation has completed the sixth step in a linear sequence, and has ``triggered" three other processes, allowing all three to analyse the data so far generated simultaneously. Where those three processes are running on separate machines in the cluster, such parallelism results in real throughput improvements.
This parallel processing adds a minor wrinkle to the polling scenario. When one of the processes has completed, it will want at least to record the observation's completion status by again renaming the observation status file. But, because there were two (or more) knowledge sources active, one might have completed earlier, and already have renamed the status file. In that case another rename will fail since a rename must have the exact name of the file.
OPUS resolves this problem in a way that is consistent with the overall philosophy: find a file with a certain pattern and rename it. However, in this case the filename pattern includes the start time and observation name, and excludes (wildcards) the status. As with the other polling logic, a failure signals a retry, not a fault.
One can invert the approach to the entire problem and, instead of viewing the observations as the objects in the system which knowledge sources operate upon, consider the knowledge sources themselves as objects which are activated by the observations. At any time each of the processes is in some state: actively processing an observation, idly waiting for an observation to process, paused and not looking for any work, or absent. <[> To improve our ability to monitor what is going on in such a distributed system, each process creates and maintains its own process status file, another empty blackboard object. The information presented in the process status file name includes:
ENCORE-CP4WF2-1-BLUE_____-IDLE_____.199410141011-199410141013;1
where:
node: ENCORE
process: CP4WF2
instance: 1
path: BLUE
status IDLE
start time: 199410141011 = 14 OCT 1994 10:11
reinit time: 199410141013 = 14 OCT 1994 10:13
This feedback information about the state of the processes is independent of the domain problem-solving information contained in the observation status files. This level of information allows OPUS more global control of the processing environment: closing down a node, adding another instance of a knowledge source, opening a parallel pipeline.
To complete the cycle yet remain within the blackboard paradigm, each process is controlled in a global sense through the same type of blackboard messages. The process command files, which are also empty files, allow the system to pause, resume, reinitialize or terminate any of the processes in the system.
The information presented in the name of the process command file includes:
An example of a process command file might be:
19941014151602-HYPER_-CP30DE-1-MAUVE____.HALT or,
19941014151602-HYPER_-ALL___-A-ALL______.REINIT
The first example directs the first instance of the process CP30DE running
on the node HYPER, and processing observations in the MAUVE path to halt.
The latter example asks all processes running on HYPER to reinitialize.
Note again that these are just files in a directory, even though they are empty files. Because they are such familiar objects, controlling the system becomes a familiar exercise. To shut down the entire pipeline, for instance, one might just rename a set of terminate command files from his personal directory to the pipeline directory.
Because the essential control elements of the OPUS pipeline have been distributed first to each of the knowledge sources, and second to the native file system, there is no requirement for an additional control mechanism in the pipeline. That is, the blackboard pipeline can become operational without any additional features.
In descriptions of blackboarding systems, however, several authors point to capabilities which have been developed at the meta-level to manage the system. For example in his discussion of Schemer-II, Michael Fehling [p.212] presents four separate ``managers", and Hassan Laasri and Brigitte Maitre [p.313] discuss their ATOME system in terms of meta-level blackboard architectures. Similarly OPUS, in order to facilitate monitoring and managing such a heterogeneous and distributed system, would include a cluster of five Motif-based window managers to assist the operator:
Employing the use of the file system to implement the blackboard was a simple and effective solution. Decoupling the knowledge sources and distributing process control turned out to be equally simple. There is no longer any process which controls what is to be done next, by whom. Somewhat like Newell's demons, each instance of each process lurks behind some CPU on the network watching for a likely victim, and when one appears on the blackboard quickly renames it.
OPUS uses two mechanisms to implement the distributed control: an internal and an external polling process. The internal polling routines are designed for processes which have significant initialization overhead. They can perform this initialization once, and then start looking for observations to process. The external capability performs the polling and renaming functions on behalf of a non-OPUS task, and acts as a shell wrapped around that process.
This means that any task which runs on the computer, whether it is an operating system command, a command procedure, a script, or a special executable program can easily become part of the OPUS pipeline.
Furthermore, OPUS has made a major commitment to extensibility: it should be easy to add another process to the system without significant perturbing other parts of the system. Since the scope of control in terms of the sequence of processes is local, extending the pipeline, or even inserting a step is easily accomplished. Associated with each knowledge source is a simple text file which declares the trigger pattern to activate the process, and the completion patterns to schedule the subsequent process(es) for activation.
Using the blackboard architecture we have achieved a distributed parallel processing system with a high level of concurrency. The overall goals of flexibility, extensibility and throughput improvement were all made possible by adopting the blackboard approach.
While the goals of the Hubble Space Telescope Science Institute have been met by this implementation of the blackboard architecture, we believe that the solution is not specific to this problem domain. In fact this technique can be used in a wide variety of applications where the amount of processing stresses the capabilities of a single machine. Indeed ANY script which links independent (modular) applications in a linear or compound (branching) sequence is amenable to this treatment. Moreover, because of its Turing-like modifiable tape of instructions encoded in the observation status file names, the blackboard model can be used to implement a full range of language features as well as more complex conditional triggering logic.
Daniel Corkill [p.99] is concerned that ``the promise of multiprocessing and distributed blackboard architectures remains largely untapped", and Penny Nii [p.xxix] notes that the ``community of blackboard system builders, although rapidly growing, is still not very large." Because the implementation of blackboard architecture on the native file system is so easy, and because clusters of workstations on networks are becoming more commonplace, a growing number of applications of this technique in a variety of disciplines can be anticipated.
Baum, L.S., Dodhiawala, R.T., Jagannathan, V. (1989). ``The Erasmus System" in Blackboard Architectures and Applications, Jagannathan, V., Dodhiawala, R., Baum, L., editors, Academic Press, San Diego, CA, pp 347-370.
Corkill, D.D., ``Advanced Architectures: Concurrency and Parallelism" in Blackboard Architectures and Applications, Jagannathan, V., Dodhiawala, R., Baum, L., editors, Academic Press, San Diego, CA, pp 77-83.
Corkill, D.D., ``Design Alternatives for Parallel and Distributed Blackboard Systems"in Blackboard Architectures and Applications, Jagannathan, V., Dodhiawala, R., Baum, L., editors, Academic Press, San Diego, CA, pp. 99-136.
Fehling, M.R., Altman, A.M., Wilber, B.M., ``The Heuristic Control Virtual Machine: An Implementation of the Schemer Computational Model of Reflective, Real-Time Problem-Solving" in Blackboard Architectures and Applications, Jagannathan, V., Dodhiawala, R., Baum, L., editors, Academic Press, San Diego, CA, pp 191-218.
Jagannathan, V., ``Realizing the Concurrent Blackboard Model" in Blackboard Architectures and Applications, Jagannathan, V., Dodhiawala, R., Baum, L., editors, Academic Press, San Diego, CA, pp 85-97
Laasri, H., Maitre, B., ``Flexibility and Efficiency in Blackboard Systems: Studies and Achievements in ATOME" in Blackboard Architectures and Applications, Jagannathan, V., Dodhiawala, R., Baum, L., editors, Academic Press, San Diego, CA, pp 309-322
Nii, H.P., ``Blackboard Systems" AI Magazine, Vol 7, No. 2, p38-53.
Nii, H.P. (1989). ``Introduction" in Blackboard Architectures and Applications, Jagannathan, V., Dodhiawala, R., Baum, L., editors, Academic Press, San Diego, CA, pp xix-xxix.