1. Setting up ~/mreorgrc¶

(This is optional and covered in more detail below)

2. Creating a database¶

We first need to initialise an empty datadase, by running the command:

$ mreorg.curate builddb

If the database already exists, but you want to clear it out and start again, use the –rebuild flag:

$ mreorg.curate builddb --rebuild

This will wipe all existing results. By default the database is stored in ~/.mreorg/mreorg.sqlite

3. Launch the front-end¶

Launch the webserver with the command:

$ mreorg.curate runserver

By default mreorg.curate listens on 8000. Open a web browser and navigate to localhost:8000 . (If the mreorg.curate webserver crashes and blocks the port, you can relaunch on another port with mreorg.curate -p 8001). You should be greeted with an overview window as shown.

Every page has 5 navigation buttons on the right hand side (Overview, Output, Tracking, Queue, Failures). The start page is the ‘Overview’ page - this will contain a list of the simulations that are being tracked, and their current states. At the moment, no simulations are being tracked so the list is empty.

4. Track some Simulations¶

To use simulations, they need to be tracked by mreorg.curate. To do this, we first specify a set of ‘Monitored Directories’, which are directories that mreorg.curate is going to look for files to track.

Click the Tracking link in the top-right corner. This will bring you to a window similar to that shown:

In the section ‘Monitored Directories’, add the path of the directory where your simulation files are, and click add. This will add a new entry in the table. Check that the directory exists (Exists column), and click on the Rescan Filesystem button. (With lots of directories, rescaning the file-system can get quite slow, especially with recursive monitoring, so it is a manual command for now. If someone can improve this, that would be great)

This should populate the list of Untracked Simulations with all the .py files found in the monitored directories. In this case, we want to track example1.py so select the tickbox, and click Track Files directly underneath.

5. Queue up the Simulation¶

Now we have set up simulation tracking, we return to the Overview page, which should now look as follows:

The overview page contains a single table, one row for each simulation. The colour of each row represents the current state of the simulation (also written in the status column)

• Yellow: There are no results from running this simulation; either it has never been run, or it never calls import mreorg .
• Green: On the last submission, the simulation ran fine, and the file has not changed since then. (Note dependancies are not checked)
• Orange: The simulation has changed since the last run.
• Red: The last run of the simulation failed.

In this case; we have only tracked a single simulation, and it has not yet run, so we see a single row in yellow. However, the following screen shot shows a database tracking many simulations, in which the simulations are in different states:

In the table, clicking on the link for a simulation in the File column will bring up more information about the file. Clicking on the link in the Last-Run (shown as a date) will provide you with more information about that particular run; for example:

This page provides information for example about any exceptions thrown. From this pages it is possible to drop straight into an editor and terminal window by clicking on the hyperlink to the right of File. This editor can be customised in ~/.mreorgrc .

Returning to the Overview page; mreorg.curate implements a simple queue system for running simulations. Simulations can be added to the queue by clicking on Run in the first column of the row. When this happens, the text should change to Cancel and horizontal bars should appear in the box, to say the simulation is queued. Clicking this again will remove the simulations from the queue (unless they have already started). You can select multiple simulations to be queued.

6. Launch the backend¶

Although we have selected our simulation to run, currently nothing will happen; we need to create some backend-workers to actually run the simulations.

Returning to the command terminal, we create a process that will actually run the simulations:

$ mreorg.curate runbackend

This starts a process that will continually check for new simulations to be added to the queue. If it finds one, then it will execute it; update the database with the results; then look for another one; and so forth. It is possible to run many backends in parallel in different terminals; (they are smart enough not to run the same simulation). You can cancel a worker with Ctrl-C.

By default, a script that takes longer than 1800 secs (30 mins) will raise a TimeOutException. (I will make this configurable at some point).

When you run this command, the result should be a sucess, and after a short while, the yellow row on the overview page on the webpage should turn green automatically. (By default, the ‘Overview-Page’ updates every 60 seconds, but it is possible to force an update by reloading the entire page.)

7. Visualise the output¶

Now that the script has run; we want to see the output. One option is through the Last-Run link, but it is often more convieniant to see the output of all the simulations. This is available on the Output page. You should see something like:

The following shows the output from multiple simulation scripts.

Environmental Variable¶

mreorg recognises the following environmental variables. Thier contents is ignored, it is only important if they are set or not.

$ export MREORG_XYZ=''

$ unset MREORG_XYZ

$ env
# or
$ env | grep 'MREOG_XYZ'

$ cat Makefile
example1.py.out: example1.py
        export 'MREORG_XYZ'; python example1.py > example1.py.out
$

$ cat Makefile
MREORG_XYZ=
example1.py.out: example1.py
        python example1.py > example1.py.out
$

~/.mreorgrc¶

mreorg can be customised using a ~/.mreorgrc file. An example is given here (from mreorg/etc_sample/.mreorgrc).

# Where shall we track by default. Adding entries here makes it easier to wipe
# the results database and restart.



# default dictionary:  {recurse='False', should_exist=False, auto_track=False}
default_monitor_directories = [
    ( "/home/michael/hw_to_come/hw-results/src/",),
    ("/home/michael/hw_to_come/morphforge/src/morphforgeexamples/", {'recurse':True, 'should_exist':True, 'auto_track':True, 'filegroup':'examples'}),
    ("/home/michael/hw_to_come/hw-results/src/12-00/", {'recurse':True, 'should_exist':True , 'auto_track':False, 'filegroup':'blah'} ),
    ("/home/michael/hw_to_come/hw-results/src/simulations_for_paper1/", {'recurse':False, 'should_exist':True, 'auto_track':True, 'filegroup':'paper2' } )
]

#default_simulations = [
#    "/home/michael/hw_to_come/morphforge/src/morphforgeexamples/singlecell_simulation/*",
#    "/home/michael/hw_to_come/morphforge/src/morphforgeexamples/**",
#    "/home/michael/hw_to_come/hw-results/src/12-00/**",
#    "/home/michael/hw_to_come/hw-results/src/simulations_for_paper1/gen*.py",
#]


default_filegroups = {
    'dIN (No-EC)': ["/home/michael/hw_to_come/hw-results/src/12-00/C Multicompartmental dIN model analysis/C. Analysis of dIN model (Individual)/*.py"],
    'dIN (EC)':    ["/home/michael/hw_to_come/hw-results/src/12-00/C Multicompartmental dIN model analysis/D. Analysis of dIN model (In Electrically Coupled Network)/*.py"],
    'paper1':      ["/home/michael/hw_to_come/hw-results/src/simulations_for_paper1/gen*.py"],
    'all-results': ["/home/michael/hw_to_come/hw-results/src/**.py"],
}


# An environmental variable of None means it should be unset if it exists.
default_runconfigs ={
    'short':{
        'timeout': 10,
        'env_vars': { 'MF_QUICK':'' }
    },
    'std':{
        'timeout': 1800,
        'env_vars': { 'MF_QUICK': None, },
        'argv': [],
    },
    'full':{
        'timeout': None,
        'env_vars': { 'MF_QUICK': None, }
    }

}



FILENAME_EXCLUDES = [
    "py.py",
    "__*.py",
    "*parsetab*",
    '/home/michael/hw_to_come/morphforge/src/bin/SimulateBundle.py',
    "*/analysis_dins.py",
    "*/dINFiles.py",
    "preload_files.py",
    "*old*.py",
    "*OLD*.py",
    "*Old*.py",
]


SIMULATION_SQLLITE_FILENAME = '/home/michael/old_home/simmgr.sqlite'
SIMULATION_IMAGE_STOREDIR = '/home/michael/old_home/.mreorg/images/'

drop_into_editor_cmds = [
    'gnome-terminal &',
    'gvim "${full_filename}"',
]


COVERAGE_CONFIG_FILE ="/home/michael/hw_to_come/morphforge/etc/.coveragerc"
COVERAGE_OUTPUT_DIR = "/tmp/morphforge_coverage_output"