wiki:WikiStart/OngoingWork/LogManagementSystem

Description of the requirements / ideas for a log management server in Rock

Use Case

Rock tooling / runscripts currently creates one dataset each time they are run. A dataset, in this sense, is a bunch of pocolog logfiles stored in a given directory. In addition, it contains the text output of each Rock component and may contain a syskit log file. The naming scheme for each of these files will be explained later.

Once a dataset has been generated (e.g. somebody did an experiment), the whole folder gets saved in a standard project-specific location, following a naming scheme YYYYMMDD_textual_description that should allow to (1) retrieve the dataset by date or (2) retrieve it from its description.

Moreover, a way to insert tags inside the logged data (for instance to delimit interesting parts of it) has been standardized it is unfortunately seldom used these days.

While the low-level support is there, the issue of finding data back within hundred of datasets. This is a common issue at the end of a multi-year project. Even more since very often only a person that actually did an experiment will remember what kind of data is usable in it (and even that person will easily forget after a year or so).

Finally, it is hard to evaluate how useful a particular dataset is. While the general policy is to keep everything, this obviously does not scale very well.

In other words, the log management system would have the task of:

  • organizing all existing datasets, providing means to upload and download all or sections of them
  • allowing to annotate the datasets and/or sections of a particular dataset with information that will allow to retrieve them more easily
  • gather statistics that would allow to more easily decide which datasets can be removed (e.g. usage statistics)

Dataset File Patterns

In a given dataset folder, one can find the following files:

  • *.[0-9]+.log (Example: hokuyo.0.log). These are pocolog log files which contain the data generated by the Rock components as well as the values of their properties.
  • *-[0-9]+.txt and orocos.*-[0-9]+.txt. The first one is the text output of the component and the second the output of RTT's own logger (they currently cannot be redirected to the same file). The numeric part is the PID of the logged process (e.g. orocos-hokuyo.2805.txt and hokuyo.2805.txt)
  • *-events.log: Roby log file.
Last modified 5 years ago Last modified on 02/16/14 14:28:19