Changes between Initial Version and Version 1 of WikiStart/OngoingWork/LogManagementSystem

02/16/14 14:28:19 (5 years ago)



  • WikiStart/OngoingWork/LogManagementSystem

    v1 v1  
     1Description of the requirements / ideas for a log management server in Rock 
     3== Use Case == 
     5Rock tooling / runscripts currently creates one ''dataset'' each time they are run. A dataset, in this sense, is a bunch of [ "pocolog logfiles"] stored in a given directory. In addition, it contains the text output of each Rock component and may contain a [ "syskit log file"]. The naming scheme for each of these files will be explained later. 
     7Once a dataset has been generated (e.g. somebody did an experiment), the whole folder gets saved in a standard project-specific location, following a naming scheme YYYYMMDD_textual_description that should allow to (1) retrieve the dataset by date or (2) retrieve it from its description. 
     9Moreover, a way to insert tags inside the logged data (for instance to delimit interesting parts of it) [ has been standardized] it is unfortunately seldom used these days. 
     11While the low-level support is there, the issue of finding data back within hundred of datasets. This is a common issue at the end of a multi-year project. Even more since very often only a person that actually did an experiment will remember what kind of data is usable in it (and even that person will easily forget after a year or so). 
     13Finally, it is hard to evaluate how useful a particular dataset is. While the general policy is to keep everything, this obviously does not scale very well. 
     15In other words, the log management system would have the task of: 
     17 - organizing all existing datasets, providing means to upload and download all or sections of them 
     18 - allowing to annotate the datasets and/or sections of a particular dataset with information that will allow to retrieve them more easily 
     19 - gather statistics that would allow to more easily decide which datasets can be removed (e.g. usage statistics) 
     21=== Dataset File Patterns === 
     23In a given dataset folder, one can find the following files: 
     24  - *.![0-9]+.log (Example: hokuyo.0.log). These are [ pocolog log files] which contain the data generated by the Rock components as well as the values of their properties. 
     25  - *-![0-9]+.txt and orocos.*-![0-9]+.txt. The first one is the text output of the component and the second the output of RTT's own logger (they currently cannot be redirected to the same file). The numeric part is the PID of the logged process (e.g. orocos-hokuyo.2805.txt and hokuyo.2805.txt) 
     26  - *-events.log: Roby log file.