Every lab engaged in cognitive psychology or similar research has its own methods for creating and running experiments. Ultimately, each one has to deal with experimental design, protocol creation, the logistics of running the experiment, handling failures, data management, and results analysis and interpretation.
MECMS (Multi-subject Control and Monitoring System) facilitates organizing, running and analysing multi-subject experiments. It is called MpcSusops at DCIEM, reflecting the fact that there it uses the local Susops (SUStained OPerationS) task battery.
To be more precise, MECMS refers to the core software. It understands things like consoles, workstations, dispatching, and protocols. MpcSusops represents a specific customization of MECMS. It understands what SUSOPS task command lines look like and how parameters (eg., subject IDs) are formatted and used. MECMS technology can be easily customized to accommodate different local task batteries.
The system consists of two types of components: workstations and consoles. There is one workstation for each subject participating in the experiment and a single console that operates as the experiment controls station. Each workstation is responsible for delivering a predefined series of tasks to the subject using it and reporting progress to the console. The console allows the experimenter to monitor subjects’ progress from a central location and provides some intervention mechanisms.
While MECMS can be used for short duration experiments, several features make it especially useful for protocols involving dozens, or even hundreds of tasks. It provides experimenters with a note-taking feature that assists in creating a formatted, legible record of events (eg., unscheduled breaks or medication) and measurements (eg., temperature, blood pressure). In addition, each workstation maintains its own log of task start times and error messages. These can be merged and sorted at the end of an experiment to produce an independently produced record of the protocol that was actually executed.
MECMS incorporates a sophisticated, but easy to use, restart and recovery mechanism. If a workstation fails during an experiment, it can be restarted and the console will restore its protocol and state information. This allows the workstation to continue from the task following the point of failure. Trial numbers are properly maintained. If the console fails, each workstation will continue dispatching tasks to its subject. When the console is restarted, it recovers state information automatically from the workstations, including the each workstation’s protocol, start times of all tasks, and scheduled modifications to the protocol that were made by the experimenter.
The following tutorial takes you through the basic steps in using the system.
Playing the role of the experimenter, start the system up on the console machine by double-clicking the "Mpcsusops console" icon on the desktop:
When the console starts up, it will ask for the URL of the console machine (in other words, the machine at which you are doing this), as well as the experiment ID. The default URL provided will be correct. The Experiment ID is simply a name for the experimenter’s log file. It defaults to the last one used. New entries will be appended to the log. This will be the desired action if the console is being restarted after a failure. You should change the name for each new experiment.
Clicking on the Make Console button will bring up the main console window:
There are four main menus on the console.
The Dispatcher menu handles interaction with the dispatcher that runs on each workstation. Menu items facilitate suspending, resuming and terminating dispatching. Recovery features are also accessed from here.
The Select menu provides alternate ways to select and deselect workstations.
The Notes menu provides access to the note-taking system.
The Help menu provides access to the HTML version of this document.
Left to its own devices, the console window will just sit there, blank. To go any further, you need to go start a dispatcher on one of the workstations. Go into one of the subject rooms, and double click the "Mpcsusops Dispatcher" icon on the desktop:
This will bring up an array of parameters for the dispatcher.
The fields of the property sheet will be blank if the workstation is being run for the first time in this machine. Otherwise, they will be filled in with values from when the workstation was last created or restored. Any fields including TITAN in the name need only be filled in if the TITAN Team Decision Making Task will be included in the protocol.
The fields should be filled in as follows:
Experiment ID – A string used to identify the workstation on the console display. Also passed to SUSOPS tasks as a parameter for result data identification.
Subject ID – A string used to identify the workstation on the console display. Also passed to SUSOPS tasks as a parameter for result data identification.
Subject Name – A string used to identify the workstation on the console display and as a memory aid for the experimenter.
Location – A string used to identify the workstation on the console display and as a memory aid for the experimenter.
Protocol File – The complete path and file name of the protocol file to be used for this experiment.
Result Directory – The path to the directory where (SUSOPS) tasks will write their result data. This is passed as a command line parameter to the tasks.
Exe Directory – The path to the directory containing the executable files for the task battery.
Log File – The complete path and file name that identifies this workstation’s log file. Status information is appended to this file.
Console URL – The URL of the console as displayed in the console’s startup screen.
Dispatcher URL – The URL of this dispatcher.
TITAN Station ID – The TITAN ID for this subject (eg., Alpha, Bravo, Charlie, Leader)
TITAN Leader PC Name – The URL of the workstation designated as the Leader for multi-player TITAN runs.
TITAN JVM – The full path and file name of the Java Virtual Machine to be used for TITAN runs.
TITAN Class Path – The class path identifying the locations of Java class components for TITAN.
Fill in the parameters appropriately and click "Create Workstation". A window entitled "java" will appear and start up the dispatcher. Wait until the java window says "Workstation ready", at which point the dispatcher is running in suspended mode. It will begin dispatching tasks as soon as the central console gives the resume command to all the workstations at once, at which point the experiment begins, with all the scheduled tasks on all the workstations beginning simultaneously.
The tasks are run entirely on the workstation. The central console simply observes the dispatcher’s progress, and allows the experimenter to suspend, resume and cancel tasks. At this point, you can return to the console. You should see that it has automatically detected the presence of a running dispatcher, and is displaying a window describing the status of that workstation. If there is a message box up that says "Creating workstation…", just wait a minute or so for the console to set up its view on the workstation. The following console shows a view on the dispatcher for subject Alpha, whose name is Emelie. Note that this is merely a remote view on the dispatcher, which is still running wholly on the workstation machine in the other room, not on the central console.
Once you have started dispatchers on all the workstations, the console should have one such window for each workstation. These can be moved around anywhere on the screen, or stacked on top of each other, as shown with the three below.
The image inside the window shows a picture of a computer (representing the workstation), connected to the queue of tasks, which is just the list of tasks for that dispatcher, in the order they are to be executed. The tasks are identified by their name, or "description", which is usually an abbreviation of three or so letters. Also listed for each task is its placement in the queue, current status and, for those tasks that have actually run, the time when they were started . For instance, task #1 in the queue shown above is the "QST" task. It is currently scheduled, so its state is "S", and it has no time, since it has not been run yet.
The numbers at the bottom of the workstation icon (ST22 and 22A in the above example) are the experiment ID and subject ID, respectively. Note that this workstation icon is connected to the task queue by a yellow connector line, with a yellow circle on it, which indicates the current state of the queue. If the dispatcher is suspended, then no tasks are running, and the pause symbol is displayed in the circle, which is the case above. The symbols used here are modelled after the standard symbols used for VCR and tape deck controls.
- The Select Menu
- The Dispatcher Menu
- Killing Dispatchers and Tasks
- The Queue Window
The screen shown above in the last section has one window selected (Bravo). You can tell it is selected because its title bar is highlighted in blue, while the other windows have grey title bars. Note that for each workstation represented in the console, there is a small icon in the top right of the window. A workstation can be selected by clicking on this icon, or by clicking the actual window of the workstation. To select more than one window at a time, hold the SHIFT key down while selecting the windows. To select all the workstations at once, choose "Select All" from the Select menu. To deselect all selected workstations, choose "Clear Selection".
In the example below, all three workstations are selected, as their title bars are all highlighted. Choosing "Clear Selection" at this point will deselect all the workstations, and the title bars will turn grey. Thus, the workstation windows do not work exactly like regular windows in the Windows operating system, since more than one workstation window can be selected at a time.
The dispatcher menu allows the experimenter to suspend, resume or remove a dispatcher. Since the dispatchers start out suspended, you need to resume them all at the same time at the start of an experiment. Select the workstation or workstations you want to resume, either by clicking on them, on their icons in the upper right corner, or by choosing "Select All" under the Select menu. Then, choose "Resume" under the Dispatcher menu to start the running of all tasks on all selected workstations. Alternatively, you can right-click on the yellow circle for the workstation you want, and choose "Resume" from the pop-up menu, as shown below.
The image of the computer in the little window will now show that the workstation is no longer "suspended", but is now running tasks. This is indicated by a change in the colour of the little computer’s monitor from white to green, and the display of the three-letter abbreviation for the task that is currently running. As well, the yellow circle now contains the run symbol (a left pointing arrow), instead of the pause symbol. The workstation icon also displays the time the task started, its estimated end time and approximate time left. The example below shows the Bravo workstation running the GTP task, which was task #2 on the queue. That leaves task #3 currently at the top of the queue, awaiting its turn. The start time is shown on the little computer screen as 11:33, along with an estimated end time of 11:37 and approximately two and one half minutes left to go.
The pop-up workstation menu also allows us to halt the currently running task by choosing "Kill task". We can also kill the entire dispatcher for that workstation with "Kill dispatcher". Killing the dispatcher is, of course, a drastic measure, but even just killing a task is also irreversible. This is not the same as canceling a program that has yet to run. Such tasks can be rescheduled, but killed tasks simply halt their execution and are gone. Therefore, the system will prompt you as to whether you are sure you want to perform the kill for the killing of both tasks and dispatchers.
So far, our view of the queue only shows tasks waiting in the queue. Double-clicking on the queue in the example above will give a larger view of the queue, as shown below. This is a scrollable list that includes completed tasks and the current running (or suspended) task.
Note that the currently running task is highlighted in green and its state is labelled "R" for "running". When a task is complete, its status is "D" for "Done", as in task #0 above. Task #7 is marked "C" for "Cancelled." The dispatcher will skip this task when it gets to it. Task #1 was cancelled and then bypassed, "B" by the dispatcher. Tasks that are cancelled, but not yet bypassed can be returned to the scheduled state. This facility is provided to allow the experimenter to respond to situations where subjects are slow, late returning to their stations, or too fatigued to handle the full work load.
The possible states of a task are:
S: Scheduled
R: Running
D: Done
C: Cancelled
B: Bypassed
F: Failed
A task is only marked as Failed if the workstation itself failed when the task was running.
A task can be selected by clicking on it in either the small queue in the workstation window or in the large queue window. Once selected you can cancel it by right clicking on it and selecting "Cancel" from the popup menu. Cancelled tasks can be rescheduled in the same way, but select "Schedule" from the popup.
More than one task can be selected in the queue at a time by using shift-click and control-click in the usual way.
Note that in the pop-up menu that, as well as canceling and scheduling, we can also choose to go to the start or end of the queue, to the currently running task, or to a particular task number within the queue. Choosing "Go to…" will bring up a prompt for the task number you wish to go to, and the queue will then display this task at the top of the displayed list, as shown below.
These queue operations can also be performed in the smaller version of the queue window. If you are just learning how to use the console, it is recommended that you play around with queue window at this point, and experiment with the different operations.
If any task, like the GTP task below goes over its estimated time, the workstation icon will turn yellow instead of green and will indicate a late time, rather than the usual time left.
Suspending the current task manually from the console is not the way to provide a pre-scheduled pause between tasks. That is done by scheduling pseudo-task called *HOLD task. Dispatchers treat the *HOLD task as a command to suspend dispatching. The HOLD has a nominal time associated with it, for example, the time scheduled for a meal break. Note that the VCR control automatically changes to a pause icon when a *HOLD is initiated. Dispatchers must be resumed manually. If the experimenter allows the hold to run longer than the scheduled time, the display will turn yellow and show the over-run time in the usual manner.
The note-taking system allows you to enter information about the experiment while it is running. It offers a number of advantages over notebooks, not the least of which are increased consistency and legibility.
A new note is started by selecting "Add" from the notes menu. It presents a form like the following:
The "Re:" field is used to identify a particular type of issue. Examples might be things like: software, general, medication, meals, etc. Each time you enter new text in this field, it is remembered and offered as a drop down choice next time. The "Title" field should be used to identify the specific event, for example "Gave Fred Tylenol." The "Author" field operates like "Re:" in that it remembers entered values and offers them as options. The "Date" field is filled in automatically based on information available when you selected the "Add" menu item. To complete the note simply fill in the fields and note body as in the following figure.
Click on the "Save" button to add it to the log, or the "Cancel" button to discard it.
The entire log can be viewed note by note, by selecting Notes->View from the menu. The most recent note is displayed. You can move back and forth in the log by clicking the "Previous" and "Next" buttons. Close the display by clicking "Done".
You can not change the text of a note once it has been saved, but you can add annotations to it. This is done by first viewing the desired note and then clicking on the "Annotate" button. The form will be changed as shown below. Enter the select/enter the author name and add the text of the annotation.
Thereafter, the note will be displayed as follows:
The internal note file is saved after each modification so that it is protected from system failures. This format can not be printed or modified. The entire log can be viewed at any time be selecting Notes->Preview from the menu. You will be presented with a display like this…
If you click on "Save As…" you will be able to save the log formatted as an HTML document. It can then be loaded into and printed by a variety of utilities and word processors.
You can leave this window visible all the time if you like. It will be automatically updated whenever a new note or modification is made.
We noted in the introduction that MECMS can survive a variety of failures. In designing this feature we wanted to meet the following goals:
As a first priority, the system will continue to dispatch work in an uninterrupted manner to all subjects not directly affected by the failure.
The system should be able to recover from either the failure of the experimenter’s console or one or more concurrent workstation failures.
The recovery mechanism should be easy to use and reliable.
The implementation provides the following features…
If the console fails, all workstations will continue dispatching tasks to subjects in the normal way. They will not be aware of any change in the environment. If they reach a *HOLD, they will remain held until the console is restarted.
No workstation failure will affect any other workstation or the operation of the console. An exception to this will arise if a task requires the cooperative actions of the subjects. An example is the TITAN team decision-making task. This task will simply fail to operate if all subjects are not available. The "Suspend", "Kill Task" and "Resume" features are used to get past this condition. The result is missing data points. In MECMS/MpcSusops, the dispatchers manage trial numbers, so trial numbering is properly maintained over failures.
If a workstation fails, it can be rebooted and restarted. All parameters will be restored to the workstation property sheet, and the experimenter simply has to click on the "Restore Workstation" button. Full MECM state information will be passed back to the dispatcher and it will resume in the protocol at the task following the one that was running at the time of failure. Note that MECM can not assume responsibility for information maintained internally or in files by the tasks themselves.
If the console fails, it can be rebooted and restarted. You can initiate recovery by selecting the Dispatcher->Reconnect menu item. The following window will be displayed…
The connection to each dispatcher is reestablished by clicking on the workstation in the top window and clicking on the "Connect" button. This is done for each in turn. The connection process takes place without disturbing dispatching activities on the running workstations. State information is passed back to the console and the workstation displays will appear as though there has been no failure. In particular, all entries will reflect correct task start times and the system will remember which upcoming tasks were manually cancelled by the experimenter.