gibuu is hosted by Hepforge, IPPP Durham
GiBUU

Changes between Initial Version and Version 1 of HowToJobCardManagemen


Ignore:
Timestamp:
Apr 7, 2010, 5:27:58 PM (9 years ago)
Author:
Ivan
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • HowToJobCardManagemen

    v1 v1  
     1==  Use python scripts to automate development and job card submission ==
     2
     3Simulation is usually connected with large amounts of output but sometimes also with a great variety of input files. Both sides (input and output) need to be handled automatically, and usually depending on one another. Here a collection of tools is presented which can be found in SVN under ''workingCode/testRun/JobCardManagement''.
     4
     5=== Why python? ===
     6
     7Working on UNIX-like machines a natural choice would be using bash scripts to automate file handling. Such efforts have already been undertaken for GiBUU:
     8
     9* [http://gibuu.physik.uni-giessen.de/internalFiles More internal informations (only available within local 134.176.18.* network)]
     10
     11When coming to more complex applications the scripts become practically unmaintainable since the lack of language functionality (e.g. string handling) needs to be compensated by using chains of external programs with varying syntax and error handling. This results in lines like this
     12
     13{{{
     14free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f2-4 >> mem.stats
     15}}}
     16
     17With python being a mature interpreted language not only the high level (object-oriented) design patterns but also the interactive debugging possibilities are very promising.
     18
     19== A standard development cycle ==
     20
     21Here are some solutions to the daily GiBUU routine using bash scripts where the problems are very straight forward and python for the rest. All these scripts are bundled in the directory ''workingCode/testRun/JobCardManagement'' (general info in '''README_ODIC'''). The [custom] tags mean that prior to executing these script you should check the header of the script for custom PATHs.
     22
     23=== Write code and debug ===
     24
     25Compilation and execution of development versions of GiBUU (prior to submitting to SVN) should be done on powerful machines. A possibility is to use one of the ''tp'' workstations via ssh, in this fashion:
     26
     27 1) Create a ''debug'' folder in ''workingCode''
     28
     29 2) execute '''debug.sh''' [custom] to sync it to a folder on the nucleus file system
     30
     31   * this makes use of the script '''compile+test.sh''', which compiles the code and submits all the jobs to the local queue
     32   * submission is managed by '''submit_job.py''' [custom]
     33
     34=== Generate job cards ===
     35
     36Once your code runs with the job cards from ''debug'' you might want to modify them - to increase statistics or study further effects. Useful tools are:
     37
     38==== replace_strings.py ====
     39
     40To increase statistics in all job cards, you can set the ''numEnsembles'' variable to a higher number.
     41
     42{{{
     43replace_strings.py --pattern="numEnsembles=",500 *.job
     44}}}
     45
     46==== create_values.py ====
     47
     48If you want to create a set of job cards where only some parameters vary, e.g. ''energy_li'', try it in the following fashion:
     49
     50{{{
     51create_values.py --var=energy_li= --low=1. --up=2. --steps=11 *.job
     52}}}
     53
     54But other similar features like "fixed q" or "transversal analysis" are also implemented.
     55
     56=== Submit all job cards ===
     57
     58Once you have a set of job cards ready, you would like to have them computed on a cluster. One possibility is copying them by hand and writing a submit script for each, then submitting each, then collecting them by random output numbers separately. Another is to let '''send_jobs.py''' [custom] do all this in a fashion like
     59
     60{{{
     61send_jobs.py --machine=skylla --queue=serial --project=p1 *.job
     62}}}
     63
     64This simply means that all jobs will be copied to the ''skylla'' cluster, where they will be submitted to the ''serial'' queue and the links to the processes and the results collected in the folder ''p1''.
     65
     66{{{
     67send_jobs.py *.job
     68}}}
     69
     70will send all jobs to ''hadron'' and create a meaningful project title by default.
     71
     72Mostly this routine relies on '''submit_job.py''' [custom] and '''manage_jobs.py''' [custom], where jobs are submitted, linked and managed on the cluster side (necessitates python version >=2.5)
     73
     74=== Manage the output ===
     75
     76==== Folder structure ====
     77
     78If everything worked properly your project folder will contain only the files ''done'' and ''jobcards''. Within the folder ''done'' you will find the results of the different job cards in folders named according to them. These folders will NOT contain redundant files like 'GiBUU.x'. In addition if you specified a ''--target'', you will extract special files from the result directories which can be used for direct plotting.
     79
     80Most of this sorting is done by '''sort_output.py''' and '''harvest_output.py'''.
     81
     82==== Data file manipulation ====
     83
     84Of course there are many UNIX tools like ''awk'' to handle data files. Useful shorthands are however:
     85
     86==== collect_data.py ====
     87collect data from different output files into one
     88
     89==== extract_data.py ====
     90Extract columns from csv like data files to new files
     91
     92==== plot_dat.py ====
     93plot multiple files into one graph
     94
     95==== rename_files.py ====
     96rename files according to strings contained within them
     97
     98== Developing the scripts ==
     99
     100As most modules have a ''doctest'' routine you can check if your contributions didn't break everything by using
     101
     102{{{
     103module.py --doctest
     104}}}
     105
     106or '''doctest_all.sh'''.
     107
     108Also you can have a look at the other files in the directory which may not have a lot to do with GiBUU, or are still in alpha phase and thus not documented here. Do not hesitate to contact the author (Ivan.Lappo-Danilevski@theo.physik.uni-giessen.de).
     109
     110Here you can see an example of the [wiki:JobCardManagmentExample entire formalism in action]