Timeline

The Timeline is HSYCO's built-in event history. It keeps a time-stamped record of what happened in the system:

• I/O Status: the value changes of I/O data points, as reported by the I/O Servers
• I/O Commands: the commands sent to I/O Servers, from EVENTS, JavaScript, Java, user interface objects or the Manager
• Variables: the value changes of variables, both persistent and volatile

Data loggers aggregate numeric values into hourly, daily and monthly statistics. The Timeline is different: it stores every single change, whether numeric or text, together with its exact time and, for commands and variables, the origin of the change. Use it to answer questions such as "when did this door open last night?", "who switched this light on?" or "what happened in the few seconds before the alarm?".

The Timeline is always running. What it records is controlled by a list of rules, and nothing is recorded until at least one rule allows it. A new installation has no timeline.ini file, so nothing is recorded until you define the rules in Settings.

Recording rules

Rules are defined in the Timeline section of Settings and are stored in the timeline.ini file. Each rule has:

Field	Values	Description
Action	allow | deny	allow records the matching names; deny excludes them
Type	io (I/O Status) | cmd (I/O Command) | var (Variable) | * (all three)	The kind of event the rule applies to
Name Pattern	wildcard pattern	Matched against the full data point, command or variable name
Decimals	empty, or integer ≥ 0	allow rules only. Rounds numeric values to this number of decimals before they are stored
Min Delta	empty, or number ≥ 0	allow rules only. A numeric value is not stored when it differs from the last stored value by less than this amount
Enabled	on/off	Disabled rules are kept in the file as comments and are ignored

Rules are evaluated from top to bottom and the first matching rule wins. When a deny rule matches, the event is not recorded, even if a later allow rule would match it. Names that match no rule are not recorded. Put specific rules before general ones.

Name patterns

Patterns use Linux-style wildcards and must match the whole name:

Wildcard	Matches
*	any sequence of characters, including none
?	any single character
[abc], [a-z]	one character in the set or range
[!abc]	one character not in the set
\	escapes the next character, e.g. \*

The dot is a literal character, so knx.* matches every data point of the knx I/O Server.

I/O and command patterns are case-sensitive. Variable patterns are case-insensitive, because variable names are always upper case.

Persistent variables are the ones whose names end with !, so the pattern *! selects only persistent variables.

Decimals and Min Delta

Both options apply only to values that are numbers. Text values are always stored unchanged. A value is treated as a number when it is written in plain decimal notation, such as 21, -3.5 or 1.2E-4. Values that only look numeric, such as 0042 (leading zeros), +39021234567 (leading plus sign) or integers with more than 15 or 16 digits, are stored as text, exactly as received.

Decimals removes noise from analog readings, for example 21.4371 is stored as 21.4 with 1 decimal.

Min Delta reduces the number of stored records for values that fluctuate continuously. The comparison is made against the last value that was actually stored, so a slow drift is still recorded once it adds up to Min Delta. After a restart, the first value is always stored.

Examples

# action; type; name_pattern[; decimals[; min_delta]]

deny;  io;  knx.1/7/*
allow; io;  knx.*.temp; 1; 0.2
allow; io;  *
allow; cmd; *
allow; var; *!

With these rules:

• the knx.1/7/* group addresses are never recorded
• KNX temperatures are rounded to one decimal, and a change is stored only when it is at least 0.2 degrees
• all other data points and all commands are recorded
• only persistent variables are recorded

tip

Recording every volatile variable (allow; var; *) can produce a very large number of records if your programs use variables as counters or scratch values. Start from *! and add specific volatile variables when you need them.

In the same way, allow; io; * also records the system.* data points of the SYSTEM I/O Server, which change every 30 seconds. Add deny; io; system.* before it if you don't need their history.

Changes to the rules take effect within a few seconds, without restarting HSYCO. This includes saving from Settings and editing timeline.ini directly.

What is recorded, and when

• I/O Status: a record is written when a data point changes value, which is the same event that updates the time stamp in the Status Browser. Repeated identical values are not recorded. Updates that I/O Servers report without triggering events are recorded with origin !.
• I/O Commands: a record is written when a command is sent to an I/O Server. The origin is HTTP for commands sent from user interface objects and the Manager, EVENTS for commands sent from EVENTS, or the plugin class and method for commands sent from Java plugins.
• Variables: a record is written when a variable changes value. The origin is the same as for commands.

Timestamps are stored with microsecond resolution, which keeps events that happen in the same millisecond in their original order, and are shown with millisecond resolution. Records are written to disk in timestamp order about one second after they happen, so the most recent second of activity may not be visible yet.

Data retention

The Timeline Max Age setting defines how many days of history are kept. Timeline data is organized in UTC days, and it is deleted the same way: once a day, at midnight UTC, the days older than the limit are deleted. The same cleanup also runs when HSYCO starts, so days missed while HSYCO was stopped are removed too. When Timeline Max Age is 0 or not set, data is never deleted and the timeline folder keeps growing until the disk is full, at which point new events are lost. See Disk full.

Use the Statistics button in the Timeline section of Settings to check how much disk space the Timeline uses and which items generate the most records. These are usually the best candidates for a deny rule, or for Decimals or Min Delta.

Storage

Timeline data is stored in the timeline folder of HSYCO's main folder, in daily log files with one index per recorded item. The mapping between names and index files is kept in the HSYCO internal database, with a copy in the timeline folder that is used to restore it if the database is rolled back or recovered from a backup. The folder is managed automatically and should not be edited by hand.

The timeline folder is not synchronized between the master and the slave in High Availability configurations, so each server keeps its own history. It is included in the daily root backup, which can make the backup much larger.

tip

If you don't want Timeline data in the root backup, add ./timeline to RootBackupExclude. Setting RootBackupExclude replaces the default exclusions, so list ./logs and ./motion too, for example ./logs,./motion,./timeline.

Reliability and error handling

The Timeline is designed to keep recording, and to keep showing all the history that is still readable, when something goes wrong with the disk, the power or the clock. It never shows a value that was not recorded: a record that cannot be read back with certainty is left out.

Problems are not reported in the Manager. They are written to the HSYCO log, in lines that start with Timeline:. Messages that could repeat continuously are written at most once a minute.

Restart and power loss

When HSYCO is stopped normally, all pending events are written before it exits.

Events are passed to the operating system about one second after they happen, and are forced to disk once a minute. So:

• if HSYCO is killed or crashes, only about the last second of events is lost
• if the server loses power or the operating system crashes, up to about one minute of the most recent events can be lost

At the next start HSYCO checks the Timeline files and repairs by itself what an interrupted write may have left behind. The log lists what was repaired. No action is required.

The names of the recorded items are also restored at start if the HSYCO internal database was recovered from its backup, or was rolled back by the same power loss, so each history stays attached to the right data point, command or variable.

Damaged files

If part of a Timeline file is damaged, for example by a disk fault:

• the history of a single item, as shown in the Timeline popup, is missing only the records that were in the damaged area
• the Timeline tab is missing a limited portion of the day that contains the damaged area, and shows the other days in full
• everything else is shown normally, and recording is not affected

In both cases the list is simply shorter. The log tells that records were skipped:

Timeline: skipped 4 damaged index entries for type=0 name=knx.1/2/3
Timeline: error reading timeline_log_20261001.data, results may be incomplete - ...

Damaged records are not repaired, and are deleted with the rest of their day when it becomes older than Timeline Max Age.

Disk full

Events that cannot be written are lost; they are not kept in memory to be written later. The files remain consistent, and recording resumes by itself as soon as there is space again. The log reports how many events were lost:

Timeline: write error, 2470 events lost or not indexed since the last report (last error: writing daily log - java.io.IOException: No space left on device)

Set Timeline Max Age, or add deny rules for the items that generate the most records, to keep the timeline folder within the available space.

Events are also lost if they arrive faster than they can be written for a prolonged time, for example with a very slow disk. In this case the log reports Timeline: queue full.

System clock

The Timeline relies on the system clock, and requires it to be reliable: the server should have a working real-time clock and a network time source.

Records are always stored in increasing time order. An event whose time is not later than the last recorded one is stored with a time just after it, so no event is lost and the order is preserved. Small corrections of this kind are normal when many events happen at the same instant, and are reported in the log as ordinary messages. A clock change has more visible effects:

• The clock is moved back, or HSYCO starts with the clock behind the last recorded event. New events are stored with times just after the last recorded one, a few microseconds apart, until the real time passes it. The events of that period are all recorded and in the right order, but their times are not accurate. The log reports corrected ... out-of-order timestamps ... (possible clock change).
• The clock is wrong in the future for some time, and is then corrected. The events recorded in the meantime are stored at the future date. After the correction this is the same as the previous case, but it lasts until the real time reaches that future date, which can be very long. There is no automatic recovery from this condition.

Moving the clock forward to the correct time has no side effects.

Viewing the Timeline

• Status Browser: the timeline icon on a data point or variable opens its history, and the Timeline tab shows every recorded event in a date range. See Status Browser.
• Assistant: the Manager and Client Assistants can query the Timeline with the hsyco_timeline_* tools. See Assistant tools.