generating rrd graps for things other than weather

classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

generating rrd graps for things other than weather

dbemowsk
In MH I see support for rrd to graph weather data.  I am wondering if there is anything out there to graph things such as temperature data or electrical usage data?

Dan B.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

Lieven Hollevoet
Administrator
Hey Dan,

I’m using xpl-rrd to store temperature readings that are reported through xpl in an RRD database and then I use MisterHouse to run custom Perl scripts that generate the graphs from the RRD databases.

Links:
https://github.com/hollie/xpl-perl/blob/master/bin/xpl-rrd
https://code.google.com/p/hasy/source/browse/trunk/netnode/software/ta_uvr/create_graphs.pl

Example output:
http://lika.be/uushuus_stats/solar_days.png

Kind regards,
 Lieven.


Op 22-dec.-2013, om 18:58 heeft dbemowsk <[hidden email]> het volgende geschreven:

> In MH I see support for rrd to graph weather data.  I am wondering if there
> is anything out there to graph things such as temperature data or electrical
> usage data?
>
> Dan B.
>
>
>
> --
> View this message in context: http://misterhouse.10964.n7.nabble.com/generating-rrd-graps-for-things-other-than-weather-tp18979.html
> Sent from the Misterhouse - User mailing list archive at Nabble.com.
>
> ------------------------------------------------------------------------------
> Rapidly troubleshoot problems before they affect your business. Most IT
> organizations don't have a clear picture of how application performance
> affects their revenue. With AppDynamics, you get 100% visibility into your
> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
> ________________________________________________________
> To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365
>

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365


signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

Marc MERLIN-7
Administrator
Op 22-dec.-2013, om 18:58 heeft dbemowsk <[hidden email]> het volgende geschreven:

> In MH I see support for rrd to graph weather data.  I am wondering if there
> is anything out there to graph things such as temperature data or electrical
> usage data?

This is not mh anymore, but I use cacti for this:
http://marc.merlins.org/linux/cacti/

See also:
http://marc.merlins.org/perso/linuxha/post_2009-11-08_Brand-One-Powermeter_-Solar-and-Power-Monitoring-with-Cacti-and-Real-Time-PG_E-Time-of-Use-price-calculation.html
http://marc.merlins.org/perso/linuxha/post_2009-12-23_Temperature-monitoring-and-graphing-with-1wire-devices_-digitemp_-misterhouse_-and-cacti.html
http://marc.merlins.org/perso/linuxha/post_2010-08-06_Temperature_-moisture_-humidity_-and-UV-monitoring-and-graphing-with-1wire-devices_-owfs_-and-cacti.html
http://marc.merlins.org/perso/linuxha/post_2011-08-03_Weather-Monitoring-with-Oregon-Scientific-WMR968.html

For power see:
http://marc.merlins.org/perso/linuxha/post_2010-08-13_Fine-grained-house-wide-power-monitoring-with-Brultech-ECM1240_-ecmread_py-_with-net-metering-support_-and-graphing-with-cacti.html

Could of talks:
http://marc.merlins.org/perso/linuxha/post_2011-11-11_Recent-Power-Monitoring-and-Misterhouse-Talks.html

Hope this helps
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

dbemowsk
Thanks Lieven and Marc, THis is a lot of great information. I am sure I
can implement something from this.  I will look at it all over the next
few days.

I have to say that I really appreciate the help that you have given on
this and my many other issues.

Dan

On Mon, 2013-12-23 at 01:32 -0800, Marc MERLIN wrote:


------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

krkeegan
Administrator
In reply to this post by dbemowsk

Dan,

I have been tinkering with the same ideas.  I have tested out nimbits.com as an alternative to RRD.  I only have a very rudimentary system in place.  The following is the code I use, if you are interested, it should help get you started:

# Category = Charting

#@  Charts data points to nimbits cloud

use JSON;

$nimbits_proc = new Process_Item;
$house_wattage = new Generic_Item;

if ($Reload){
# Used to initialize an item whose state is numeric and should be tracked
# directly, such as a thermometer
nimbits_initialize($w_temp, "out_temp","value");
nimbits_initialize($upstairs_thermo_temp, "up_temp","value");
nimbits_initialize($garage_battery, "garage_battery","value");
nimbits_initialize($upstairs_thermo_humidity, "up_humidity","value");
#Dimming Object would look like:
nimbits_initialize($f_fm_lt_ma, "family_room","wattage", 57, 'house_wattage');
nimbits_initialize($f_kt_sink_lt_ma, "kitchen_sink", "wattage", 65, 'house_wattage');
nimbits_initialize($f_en_lt_ma, "entry_way", "wattage", 405, 'house_wattage');
nimbits_initialize($f_dn_ch_lt_ma, "dining_chandelier", "wattage", 100, 'house_wattage');
# initialize aggregate items last, so not setting the data_point for each
# new member
nimbits_initialize($house_wattage, "house_wattage","value");
}

# Initializes the nimbit data_point. Does many things ......

sub nimbits_initialize{
my ($object, $data_point, $tie_type, $max_watt, $aggregate_name) = @_;
#set the proper tie function type and initialize aggregate
my $curr_value;
if ($tie_type eq "value"){
$object->tie_event("nimbits_post_aggregate('$data_point',\$state,"
. "'$aggregate_name')");
$curr_value = $object->state;

elsif ($tie_type eq "wattage"){
$object->tie_event("nimbits_post_wattage('$data_point',\$state,"
."'$max_watt','$aggregate_name')");
$curr_value = nimbits_wattage($object->state, $max_watt);
}
elsif ($tie_type eq "network"){
$object->tie_event("nimbits_post_network('$data_point',\$state,"
."'$max_watt','$aggregate_name')");
$curr_value = nimbits_network($object->state, $max_watt);
}
elsif ($tie_type eq "whf"){
$object->tie_event("nimbits_post_whf('$data_point',\$state,"
."'$max_watt','$aggregate_name')");
$curr_value = nimbits_whf($object->state, $max_watt);
}
#Initialize aggregate item
if ($aggregate_name){
set_aggregate_item($aggregate_name, $data_point, $curr_value);
}
# Call nimbits post without a value to initialize the data point
nimbits_post_value($data_point);
}

# Posts a dimmable wattage to nimbits

sub nimbits_post_wattage {
my ($data_point, $p_state, $max_watt, $aggregate_name) = @_;
my $act_watt = nimbits_wattage($p_state, $max_watt);
nimbits_post_aggregate($data_point,$act_watt, $aggregate_name);
}

# Converts onlevel and maximum wattage into a current wattage value

sub nimbits_wattage {
my ($p_state, $max_watt) = @_;
my $level = 100;
if ($p_state eq 'off' or $p_state eq 'off_fast') {
$level = 0;
}
elsif ($p_state =~ /\d+%?/) {
$level = $p_state;
$level =~ /(\d+)%?/;
}
my $act_watt = $max_watt * ($level/100);
return $act_watt;
}

# Posts a network wattage to nimbits

sub nimbits_post_network {
my ($data_point, $p_state, $max_watt, $aggregate_name) = @_;
my $act_watt = nimbits_network($p_state, $max_watt);
nimbits_post_aggregate($data_point,$act_watt, $aggregate_name);
}

# Converts returns full wattage if up

sub nimbits_network {
my ($p_state, $max_watt) = @_;
my $level = $max_watt;
if ($p_state eq 'down') {
$level = 0;
}
return $level;
}

# Posts a whf wattage to nimbits

sub nimbits_post_whf {
my ($data_point, $p_state, $max_watt, $aggregate_name) = @_;
my $act_watt = nimbits_whf($p_state, $max_watt);
nimbits_post_aggregate($data_point,$act_watt, $aggregate_name);
}

# Converts high low wattage

sub nimbits_whf {
my ($p_state, $max_watt) = @_;
my $level = $max_watt;
if ($whf_main->state eq 'off') {
$level = 0;
}
elsif ($whf_speed->state eq 'on'){
$level = 197;
}
else {
$level = 44;
}
return $level;
}

# Track aggregate values, and post total value

sub nimbits_post_aggregate {
my ($data_point, $value, $aggregate_name) = @_;
#if aggregate item, call this function on the aggregate_class
if ($aggregate_name){
set_aggregate_item($aggregate_name, $data_point, $value);
}
#then call normal
nimbits_post_value($data_point, $value,);
}

# how to store the aggregate amount

sub set_aggregate_item {
my ($aggregate_name, $data_point, $value) = @_;
my $aggregate_item = get_object_by_name($aggregate_name);
$$aggregate_item{aggregate_members}{$data_point} = $value;
#Calculate new total value and call set
my $aggregate_total = 0.0;
my $on_items_string;
for (keys %{$$aggregate_item{aggregate_members}}) {
$aggregate_total += $$aggregate_item{aggregate_members}{$_};
if ($$aggregate_item{aggregate_members}{$_} > 0){
$on_items_string = $on_items_string . $_ . ", ";
}
}
if ($aggregate_total != $aggregate_item->state()){
$aggregate_item->set($aggregate_total, $on_items_string);
}
}

# Used to post a value to nimbits, uses an external process to post
# if no value is defined, it makes sure the data point exists

sub nimbits_post_value {
my ($data_point, $value, $epoch) = @_;
my $nim_url;
if (defined $value){
if (defined $epoch){
$nim_url = "get_url -quiet -post 'json={\"d\":$value, \"t\":$epoch}' ";
}
else {
$nim_url = "get_url -quiet -post 'json={\"d\":$value}' ";
}
$nim_url .= "'<a href="http://cloud.nimbits.com/service/v2/value?key=123456&amp;email=test\@example.com&amp;id=test\@example.com/$data_point">http://cloud.nimbits.com/service/v2/value?key=123456&email=test\@example.com&id=test\@example.com/$data_point' "
."/dev/null";

else {
#Create the data_point if it doesn't exist
$nim_url = "get_url -quiet -post 'json={\"highAlarm\":0.0,\"lowAlarm\":0.0,"
."\"deltaAlarm\":0.0,\"deltaSeconds\":0,\"expire\":10000,\"unit\":null,"
."\"highAlarmOn\":false,\"lowAlarmOn\":false,\"idleAlarmOn\":false,"
."\"deltaAlarmOn\":false,\"idleSeconds\":0,\"idleAlarmSent\":false,"
."\"filterType\":4,\"filterValue\":0.1,\"inferLocation\":false,"
."\"pointType\":0,\"values\":null,\"value\":null,\"name\":\"$data_point\","
."\"description\":null,\"entityType\":1,\"protectionLevel\":2,"
."\"alertType\":0,\"parent\":\"test\@example.com\","
."\"owner\":\"test\@gexamplemail.com\",\"readOnly\":false,\"children\":null,"
."\"instanceUrl\":null,\"isCached\":true}&action=createmissing' "
."'<a href="http://cloud.nimbits.com/service/v2/entity?key=123456&amp;email=test\@example.com">http://cloud.nimbits.com/service/v2/entity?key=123456&email=test\@example.com' "
."/dev/null";
}
if ($nimbits_proc->done != 0){
$nimbits_proc->set($nim_url);
$nimbits_proc->start();
} else {
$nimbits_proc->add($nim_url);
}
}

# Used to post a batch of values to nimbits, uses an external process to post

sub nimbits_post_batch {
my ($batch_hash_ref) = @_;
my $batch_json = encode_json $batch_hash_ref;
my $nim_url = "get_url -post 'json=$batch_json' "
."'<a href="http://cloud.nimbits.com/service/v2/batch?key=123456&amp;email=test\@example.com">http://cloud.nimbits.com/service/v2/batch?key=123456&email=test\@example.com' "
."/dev/null";
if ($nimbits_proc->done != 0){
$nimbits_proc->set($nim_url);
$nimbits_proc->start();
} else {
$nimbits_proc->add($nim_url);
}
}

In MH I see support for rrd to graph weather data.  I am wondering if there
is anything out there to graph things such as temperature data or electrical
usage data?

Dan B.



--
View this message in context: http://misterhouse.10964.n7.nabble.com/generating-rrd-graps-for-things-other-than-weather-tp18979.html
Sent from the Misterhouse - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365


------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

Lieven Hollevoet
Administrator
Hey all,

just for your information: if you want to share code, the github system allows you to do this easily (easier than sharing through email). Just navigate to github.com, click on ‘gist’ in the upper navigation bar, paste your code and add short description.

Kevin, I have taken the freedom to create a gist from your example here:

https://gist.github.com/hollie/8102864

The advantage is that people can easily view, clone or download such a gist and make changes and contribute changes back. Much easier than using email that possibly scrambles the formatting.

Kind regards,
 Lieven.

Op 23-dec.-2013, om 16:39 heeft Kevin Robert Keegan <[hidden email]> het volgende geschreven:

> # Category = Charting
>
> #@  Charts data points to nimbits cloud
>
> use JSON;
>
> $nimbits_proc = new Process_Item;
> $house_wattage = new Generic_Item;
>
> if ($Reload){
> # Used to initialize an item whose state is numeric and should be tracked
> # directly, such as a thermometer
> nimbits_initialize($w_temp, "out_temp","value");
> nimbits_initialize($upstairs_thermo_temp, "up_temp","value");
> nimbits_initialize($garage_battery, "garage_battery","value");
> nimbits_initialize($upstairs_thermo_humidity, "up_humidity","value");
> #Dimming Object would look like:
> nimbits_initialize($f_fm_lt_ma, "family_room","wattage", 57, 'house_wattage');
> nimbits_initialize($f_kt_sink_lt_ma, "kitchen_sink", "wattage", 65, 'house_wattage');
> nimbits_initialize($f_en_lt_ma, "entry_way", "wattage", 405, 'house_wattage');
> nimbits_initialize($f_dn_ch_lt_ma, "dining_chandelier", "wattage", 100, 'house_wattage');
> # initialize aggregate items last, so not setting the data_point for each
> # new member
> nimbits_initialize($house_wattage, "house_wattage","value");
> }
>
> # Initializes the nimbit data_point. Does many things ......
>
> sub nimbits_initialize{
> my ($object, $data_point, $tie_type, $max_watt, $aggregate_name) = @_;
> #set the proper tie function type and initialize aggregate
> my $curr_value;
> if ($tie_type eq "value"){
> $object->tie_event("nimbits_post_aggregate('$data_point',\$state,"
> . "'$aggregate_name')");
> $curr_value = $object->state;
> }
> elsif ($tie_type eq "wattage"){
> $object->tie_event("nimbits_post_wattage('$data_point',\$state,"
> ."'$max_watt','$aggregate_name')");
> $curr_value = nimbits_wattage($object->state, $max_watt);
> }
> elsif ($tie_type eq "network"){
> $object->tie_event("nimbits_post_network('$data_point',\$state,"
> ."'$max_watt','$aggregate_name')");
> $curr_value = nimbits_network($object->state, $max_watt);
> }
> elsif ($tie_type eq "whf"){
> $object->tie_event("nimbits_post_whf('$data_point',\$state,"
> ."'$max_watt','$aggregate_name')");
> $curr_value = nimbits_whf($object->state, $max_watt);
> }
> #Initialize aggregate item
> if ($aggregate_name){
> set_aggregate_item($aggregate_name, $data_point, $curr_value);
> }
> # Call nimbits post without a value to initialize the data point
> nimbits_post_value($data_point);
> }
>
> # Posts a dimmable wattage to nimbits
>
> sub nimbits_post_wattage {
> my ($data_point, $p_state, $max_watt, $aggregate_name) = @_;
> my $act_watt = nimbits_wattage($p_state, $max_watt);
> nimbits_post_aggregate($data_point,$act_watt, $aggregate_name);
> }
>
> # Converts onlevel and maximum wattage into a current wattage value
>
> sub nimbits_wattage {
> my ($p_state, $max_watt) = @_;
> my $level = 100;
> if ($p_state eq 'off' or $p_state eq 'off_fast') {
> $level = 0;
> }
> elsif ($p_state =~ /\d+%?/) {
> $level = $p_state;
> $level =~ /(\d+)%?/;
> }
> my $act_watt = $max_watt * ($level/100);
> return $act_watt;
> }
>
> # Posts a network wattage to nimbits
>
> sub nimbits_post_network {
> my ($data_point, $p_state, $max_watt, $aggregate_name) = @_;
> my $act_watt = nimbits_network($p_state, $max_watt);
> nimbits_post_aggregate($data_point,$act_watt, $aggregate_name);
> }
>
> # Converts returns full wattage if up
>
> sub nimbits_network {
> my ($p_state, $max_watt) = @_;
> my $level = $max_watt;
> if ($p_state eq 'down') {
> $level = 0;
> }
> return $level;
> }
>
> # Posts a whf wattage to nimbits
>
> sub nimbits_post_whf {
> my ($data_point, $p_state, $max_watt, $aggregate_name) = @_;
> my $act_watt = nimbits_whf($p_state, $max_watt);
> nimbits_post_aggregate($data_point,$act_watt, $aggregate_name);
> }
>
> # Converts high low wattage
>
> sub nimbits_whf {
> my ($p_state, $max_watt) = @_;
> my $level = $max_watt;
> if ($whf_main->state eq 'off') {
> $level = 0;
> }
> elsif ($whf_speed->state eq 'on'){
> $level = 197;
> }
> else {
> $level = 44;
> }
> return $level;
> }
>
> # Track aggregate values, and post total value
>
> sub nimbits_post_aggregate {
> my ($data_point, $value, $aggregate_name) = @_;
> #if aggregate item, call this function on the aggregate_class
> if ($aggregate_name){
> set_aggregate_item($aggregate_name, $data_point, $value);
> }
> #then call normal
> nimbits_post_value($data_point, $value,);
> }
>
> # how to store the aggregate amount
>
> sub set_aggregate_item {
> my ($aggregate_name, $data_point, $value) = @_;
> my $aggregate_item = get_object_by_name($aggregate_name);
> $$aggregate_item{aggregate_members}{$data_point} = $value;
> #Calculate new total value and call set
> my $aggregate_total = 0.0;
> my $on_items_string;
> for (keys %{$$aggregate_item{aggregate_members}}) {
> $aggregate_total += $$aggregate_item{aggregate_members}{$_};
> if ($$aggregate_item{aggregate_members}{$_} > 0){
> $on_items_string = $on_items_string . $_ . ", ";
> }
> }
> if ($aggregate_total != $aggregate_item->state()){
> $aggregate_item->set($aggregate_total, $on_items_string);
> }
> }
>
> # Used to post a value to nimbits, uses an external process to post
> # if no value is defined, it makes sure the data point exists
>
> sub nimbits_post_value {
> my ($data_point, $value, $epoch) = @_;
> my $nim_url;
> if (defined $value){
> if (defined $epoch){
> $nim_url = "get_url -quiet -post 'json={\"d\":$value, \"t\":$epoch}' ";
> }
> else {
> $nim_url = "get_url -quiet -post 'json={\"d\":$value}' ";
> }
> $nim_url .= "'http://cloud.nimbits.com/service/v2/value?key=123456&email=test\@example.com&id=test\@example.com/$data_point' "
> ."/dev/null";
> }
> else {
> #Create the data_point if it doesn't exist
> $nim_url = "get_url -quiet -post 'json={\"highAlarm\":0.0,\"lowAlarm\":0.0,"
> ."\"deltaAlarm\":0.0,\"deltaSeconds\":0,\"expire\":10000,\"unit\":null,"
> ."\"highAlarmOn\":false,\"lowAlarmOn\":false,\"idleAlarmOn\":false,"
> ."\"deltaAlarmOn\":false,\"idleSeconds\":0,\"idleAlarmSent\":false,"
> ."\"filterType\":4,\"filterValue\":0.1,\"inferLocation\":false,"
> ."\"pointType\":0,\"values\":null,\"value\":null,\"name\":\"$data_point\","
> ."\"description\":null,\"entityType\":1,\"protectionLevel\":2,"
> ."\"alertType\":0,\"parent\":\"test\@example.com\","
> ."\"owner\":\"test\@gexamplemail.com\",\"readOnly\":false,\"children\":null,"
> ."\"instanceUrl\":null,\"isCached\":true}&action=createmissing' "
> ."'http://cloud.nimbits.com/service/v2/entity?key=123456&email=test\@example.com' "
> ."/dev/null";
> }
> if ($nimbits_proc->done != 0){
> $nimbits_proc->set($nim_url);
> $nimbits_proc->start();
> } else {
> $nimbits_proc->add($nim_url);
> }
> }
>
> # Used to post a batch of values to nimbits, uses an external process to post
>
> sub nimbits_post_batch {
> my ($batch_hash_ref) = @_;
> my $batch_json = encode_json $batch_hash_ref;
> my $nim_url = "get_url -post 'json=$batch_json' "
> ."'http://cloud.nimbits.com/service/v2/batch?key=123456&email=test\@example.com' "
> ."/dev/null";
> if ($nimbits_proc->done != 0){
> $nimbits_proc->set($nim_url);
> $nimbits_proc->start();
> } else {
> $nimbits_proc->add($nim_url);
> }
> }
>

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365


signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

Neil Cherry-3
On 12/23/2013 02:17 PM, Lieven Hollevoet wrote:

> just for your information: if you want to share code, the github system allows you to
> do this easily (easier than sharing through email). Just navigate to github.com, click
> on ‘gist’ in the upper navigation bar, paste your code and add short description.

Liven, thanks :-) I've been looking for something like this.

--
Linux Home Automation         Neil Cherry       [hidden email]
http://www.linuxha.com/                         Main site
http://linuxha.blogspot.com/                    My HA Blog
Author of:     Linux Smart Homes For Dummies

------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

Michael Stovenour
Administrator
Gist is a nice interface.  It is even capable of tracking versions.  I think of it as
"repository lite".

-----Original Message-----
From: Neil Cherry [mailto:[hidden email]]
Sent: Monday, December 23, 2013 4:21 PM
To: The main list for the MisterHouse home automation program
Subject: Re: [mh] generating rrd graps for things other than weather

On 12/23/2013 02:17 PM, Lieven Hollevoet wrote:

> just for your information: if you want to share code, the github system allows you to
> do this easily (easier than sharing through email). Just navigate to github.com, click
> on 'gist' in the upper navigation bar, paste your code and add short description.

Liven, thanks :-) I've been looking for something like this.

--
Linux Home Automation         Neil Cherry       [hidden email]


------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

dbemowsk
In reply to this post by Lieven Hollevoet
Okay, I am trying to wrap my brain around RRD and have a few questions.  I have been reading the beginners guide to RRD here http://oss.oetiker.ch/rrdtool/tut/rrd-beginners.en.html.  In this article, they show the following example:

rrdtool create target.rrd \
         --start 1023654125 \
         --step 300 \
         DS:mem:GAUGE:600:0:671744 \
         RRA:AVERAGE:0.5:12:24 \
         RRA:AVERAGE:0.5:288:31

They go on to define the RRA lines as "RRA:CF:xff:step:rows".  I understand that the CF (Consolidation Function) defines the aggregate function that will be use on that data set ( AVERAGE, MINIMUM, MAXIMUM, or LAST). I understand that "step" is the number of data points that are aggregated together to form the displayed data point. I also understand that the "rows" is the number of "steps" that are saved.  What I don't see documented is the "xff" value.  In all of the RRD definitions that I have seen so far, that value is set to 0.5, but I don't know what that represents.

My next question in all of this has to do with data history.  I understand that the concept of RRD revolves around the whole round robin concept where the data works on a fixed length timeline where data overwrites data from the beginning once the saved points reach the end of the data set.  My question is, is there a way to archive data?  What I mean is, if you have an RRD that collects data for a 24 hour period, and when the next day starts effectively overwriting the first days data, is there a way to look back at a graph of data from a few days ago?  I see that the weather_rrd_update.pl file creates a CSV file.  Is this what the CSV is used for?  If so, I am guessing that it just keeps appending data to the end of the CSV, correct?

My last question has to do with Data Sources (DS).  In the example, they define one DS.  In looking at the weather_rrd_update.pl file, they define a number of them.  Does each DS represent a different line (color) on the graph?

Regards

Dan B.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

Lieven Hollevoet
Administrator
Hey Dan,

Op 25-dec.-2013, om 08:40 heeft dbemowsk <[hidden email]> het volgende geschreven:

> My next question in all of this has to do with data history.  I understand
> that the concept of RRD revolves around the whole round robin concept where
> the data works on a fixed length timeline where data overwrites data from
> the beginning once the saved points reach the end of the data set.  My
> question is, is there a way to archive data?  What I mean is, if you have an
> RRD that collects data for a 24 hour period, and when the next day starts
> effectively overwriting the first days data, is there a way to look back at
> a graph of data from a few days ago?  I see that the weather_rrd_update.pl
> file creates a CSV file.  Is this what the CSV is used for?  If so, I am
> guessing that it just keeps appending data to the end of the CSV, correct?
Nope, the idea of RRD is that you can use it stand alone. You define data sources (DS) that keep track of values. Then for every data source you create RRAs (round robin archives) with variable ‘timespan' per datapoint. E.g. suppose that you store a new temperature every 5 minutes, then your primary ‘resolution’ for the first RRA van be 5 minutes.
Then, for longer archiving you can define RRAs that keep track of your data over longer timespans. You then need to define the consolidation function for that datasource. Suppose you define secondary RRAs with a resolution of 1 hour per datapoint. The RRD framework can then take the values from the first high-resolution RRA, apply a consolidation function on those datapoints, and then calculate the value for the lower resolution dataset. The consolidation function can be e.g. average, minimum or maximum. This way you can keep track of values over longer time (e.g. a year) in 1 hour intervals. If you want to go even further you can then store values on a daily basis for 10’s of years.

>
> My last question has to do with Data Sources (DS).  In the example, they
> define one DS.  In looking at the weather_rrd_update.pl file, they define a
> number of them.  Does each DS represent a different line (color) on the
> graph?

Nope, a DS is a dataset. You can see it as a representation of a sensor that you read out and that you store the values in the database. To create the graphs you use the values from the RRAs or you calculate values based on the data in those RRAs.

I hope this clarifies it a bit further.

Kind regards,
 Lieven.



------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365


signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

dbemowsk
So, just to I have this straight I am going to use an abbreviated example taken from the weather_rrd_update.pl code that comes with MH.  In this, each DS defines a sensor value to show on the graph (correct?), then each RRA defines the graph resolutions that are possible for each MIN, AVERAGE and MAX aggregate storing data for a defined period (1 day, 2 days) at each resolution (2 min, 5 min)?  The primary resolution for the first RRA is at 6 hours using an AVERAGE aggregate of values every minute, correct?  I hope I said that right?

One last question I had in my post on this.  I am trying to understand the RRA's 0.5 value which is said to be "xff" by mr Oetiker in the beginners guide page that I listed.  What does the 0.5 represent or mean?  When I create my RRDs for graphing I can certainly use the 0.5 value that I see everyone else primarily using, but I like to know what it means.

  RRDs::create $RRD,
    '-b', $_[0], '-s', $RRD_STEP,
    "DS:temp:GAUGE:$RRD_HEARTBEAT:-150:150",
    "DS:humid:GAUGE:$RRD_HEARTBEAT:0:100",
    "DS:dew:GAUGE:$RRD_HEARTBEAT:-150:150",
    "DS:press:GAUGE:$RRD_HEARTBEAT:23:33",

    ~~~~~~~~~~~~~~~ abbreviated ~~~~~~~~~~~~~~~

    'RRA:AVERAGE:0.5:1:801', # details for 6 hours (agregate 1 minute)

    'RRA:MIN:0.5:2:801', # 1 day (agregate 2 minutes)
    'RRA:AVERAGE:0.5:2:801',
    'RRA:MAX:0.5:2:801',

    'RRA:MIN:0.5:5:641', # 2 day (agregate 5 minutes)
    'RRA:AVERAGE:0.5:5:641',
    'RRA:MAX:0.5:5:641',

    ~~~~~~~~~~~~~~~ abbreviated ~~~~~~~~~~~~~~~

    die "weather_rrd_update : unable to create $RRD: $err\n" if $err = RRDs::error;
}

Dan

On Fri, 2013-12-27 at 14:48 +0100, Lieven Hollevoet wrote:
Hey Dan,

Op 25-dec.-2013, om 08:40 heeft dbemowsk <[hidden email]> het volgende geschreven:

> My next question in all of this has to do with data history.  I understand
> that the concept of RRD revolves around the whole round robin concept where
> the data works on a fixed length timeline where data overwrites data from
> the beginning once the saved points reach the end of the data set.  My
> question is, is there a way to archive data?  What I mean is, if you have an
> RRD that collects data for a 24 hour period, and when the next day starts
> effectively overwriting the first days data, is there a way to look back at
> a graph of data from a few days ago?  I see that the weather_rrd_update.pl
> file creates a CSV file.  Is this what the CSV is used for?  If so, I am
> guessing that it just keeps appending data to the end of the CSV, correct?

Nope, the idea of RRD is that you can use it stand alone. You define data sources (DS) that keep track of values. Then for every data source you create RRAs (round robin archives) with variable ‘timespan' per datapoint. E.g. suppose that you store a new temperature every 5 minutes, then your primary ‘resolution’ for the first RRA van be 5 minutes.
Then, for longer archiving you can define RRAs that keep track of your data over longer timespans. You then need to define the consolidation function for that datasource. Suppose you define secondary RRAs with a resolution of 1 hour per datapoint. The RRD framework can then take the values from the first high-resolution RRA, apply a consolidation function on those datapoints, and then calculate the value for the lower resolution dataset. The consolidation function can be e.g. average, minimum or maximum. This way you can keep track of values over longer time (e.g. a year) in 1 hour intervals. If you want to go even further you can then store values on a daily basis for 10’s of years. 

> 
> My last question has to do with Data Sources (DS).  In the example, they
> define one DS.  In looking at the weather_rrd_update.pl file, they define a
> number of them.  Does each DS represent a different line (color) on the
> graph?

Nope, a DS is a dataset. You can see it as a representation of a sensor that you read out and that you store the values in the database. To create the graphs you use the values from the RRAs or you calculate values based on the data in those RRAs.

I hope this clarifies it a bit further.

Kind regards,
 Lieven.




------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

Lieven Hollevoet
Administrator
Hey Dan,

Op 27-dec.-2013, om 18:23 heeft Dan Bemowski <[hidden email]> het volgende geschreven:

So, just to I have this straight I am going to use an abbreviated example taken from the weather_rrd_update.pl code that comes with MH.  In this, each DS defines a sensor value to show on the graph (correct?)

Each DS does define a sensor value.

, then each RRA defines the graph resolutions that are possible for each MIN, AVERAGE and MAX aggregate storing data for a defined period (1 day, 2 days) at each resolution (2 min, 5 min)?  The primary resolution for the first RRA is at 6 hours using an AVERAGE aggregate of values every minute, correct?  I hope I said that right?

The primary resolution depends on $RRD_STEP in the code. For the MisterHouse weather code this value is set to 60 seconds if I’m correct. Basically if you say 

RRA:AVERAGE:0.5:1:801 

then you tell the tool to store a value in the database every $RRD_STEP seconds, and do this for the last 801 $RRD_STEP seconds.

Why do you need ‘AVERAGE’ then? Well, you can push values to the RRD at a higher speed than defined by $RRD_STEP and then RRD will store only the average of those values every $RRD_STEP seconds.

Now, back to the code. I think the comment that states ‘for 6 hours’ is not correct. Since the $RRD_STEP is 60 seconds in weather_rrd_update, 801 times step_size is 13 hours instead of 6 hours. This might be the source of the confusion you are experiencing. It is probably something nobody has noticed up to now because 13 hours > 6 hours and the graphs show to be correct. The only disadvantage is that the database is bigger than it is absolutely required.


One last question I had in my post on this.  I am trying to understand the RRA's 0.5 value which is said to be "xff" by mr Oetiker in the beginners guide page that I listed.  What does the 0.5 represent or mean?  When I create my RRDs for graphing I can certainly use the 0.5 value that I see everyone else primarily using, but I like to know what it means.

See http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

xff The xfiles factor defines what part of a consolidation interval may be made up from*UNKNOWN* data while the consolidated value is still regarded as known. It is given as the ratio of allowed *UNKNOWN* PDPs to the number of PDPs in the interval. Thus, it ranges from 0 to 1 (exclusive).

If you fail to feed data into the database for a certain amount of time, these points in time will be marked ‘unknown’. If you start to consolidate the rate (e.g. average data points) then the off factor determines from what point on the result of the consolidation function will also be unknown. If you set it to .5 then half of the values are allowed to be unknown before the consolidated results is also unknown.

Kind regards, 
 Lieven.


On Fri, 2013-12-27 at 14:48 +0100, Lieven Hollevoet wrote:
Hey Dan,

Op 25-dec.-2013, om 08:40 heeft dbemowsk <[hidden email]> het volgende geschreven:

> My next question in all of this has to do with data history.  I understand
> that the concept of RRD revolves around the whole round robin concept where
> the data works on a fixed length timeline where data overwrites data from
> the beginning once the saved points reach the end of the data set.  My
> question is, is there a way to archive data?  What I mean is, if you have an
> RRD that collects data for a 24 hour period, and when the next day starts
> effectively overwriting the first days data, is there a way to look back at
> a graph of data from a few days ago?  I see that the weather_rrd_update.pl
> file creates a CSV file.  Is this what the CSV is used for?  If so, I am
> guessing that it just keeps appending data to the end of the CSV, correct?

Nope, the idea of RRD is that you can use it stand alone. You define data sources (DS) that keep track of values. Then for every data source you create RRAs (round robin archives) with variable ‘timespan' per datapoint. E.g. suppose that you store a new temperature every 5 minutes, then your primary ‘resolution’ for the first RRA van be 5 minutes.
Then, for longer archiving you can define RRAs that keep track of your data over longer timespans. You then need to define the consolidation function for that datasource. Suppose you define secondary RRAs with a resolution of 1 hour per datapoint. The RRD framework can then take the values from the first high-resolution RRA, apply a consolidation function on those datapoints, and then calculate the value for the lower resolution dataset. The consolidation function can be e.g. average, minimum or maximum. This way you can keep track of values over longer time (e.g. a year) in 1 hour intervals. If you want to go even further you can then store values on a daily basis for 10’s of years. 

> 
> My last question has to do with Data Sources (DS).  In the example, they
> define one DS.  In looking at the weather_rrd_update.pl file, they define a
> number of them.  Does each DS represent a different line (color) on the
> graph?

Nope, a DS is a dataset. You can see it as a representation of a sensor that you read out and that you store the values in the database. To create the graphs you use the values from the RRAs or you calculate values based on the data in those RRAs.

I hope this clarifies it a bit further.

Kind regards,
 Lieven.





------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365


signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

Steve Switzer

What came of this thread? Do we have example code for graphing things like motion events in each room, door open/close, MH mode, alarm system status, etc? It would even be nice to see when lights are turned on over time, too.

Thank you!

Steve


On 01/05/2014 03:14 PM, Lieven Hollevoet wrote:
Hey Dan,

Op 27-dec.-2013, om 18:23 heeft Dan Bemowski <[hidden email]> het volgende geschreven:

So, just to I have this straight I am going to use an abbreviated example taken from the weather_rrd_update.pl code that comes with MH.  In this, each DS defines a sensor value to show on the graph (correct?)

Each DS does define a sensor value.

, then each RRA defines the graph resolutions that are possible for each MIN, AVERAGE and MAX aggregate storing data for a defined period (1 day, 2 days) at each resolution (2 min, 5 min)?  The primary resolution for the first RRA is at 6 hours using an AVERAGE aggregate of values every minute, correct?  I hope I said that right?

The primary resolution depends on $RRD_STEP in the code. For the MisterHouse weather code this value is set to 60 seconds if I’m correct. Basically if you say 

RRA:AVERAGE:0.5:1:801 

then you tell the tool to store a value in the database every $RRD_STEP seconds, and do this for the last 801 $RRD_STEP seconds.

Why do you need ‘AVERAGE’ then? Well, you can push values to the RRD at a higher speed than defined by $RRD_STEP and then RRD will store only the average of those values every $RRD_STEP seconds.

Now, back to the code. I think the comment that states ‘for 6 hours’ is not correct. Since the $RRD_STEP is 60 seconds in weather_rrd_update, 801 times step_size is 13 hours instead of 6 hours. This might be the source of the confusion you are experiencing. It is probably something nobody has noticed up to now because 13 hours > 6 hours and the graphs show to be correct. The only disadvantage is that the database is bigger than it is absolutely required.


One last question I had in my post on this.  I am trying to understand the RRA's 0.5 value which is said to be "xff" by mr Oetiker in the beginners guide page that I listed.  What does the 0.5 represent or mean?  When I create my RRDs for graphing I can certainly use the 0.5 value that I see everyone else primarily using, but I like to know what it means.

See http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

xff The xfiles factor defines what part of a consolidation interval may be made up from*UNKNOWN* data while the consolidated value is still regarded as known. It is given as the ratio of allowed *UNKNOWN* PDPs to the number of PDPs in the interval. Thus, it ranges from 0 to 1 (exclusive).

If you fail to feed data into the database for a certain amount of time, these points in time will be marked ‘unknown’. If you start to consolidate the rate (e.g. average data points) then the off factor determines from what point on the result of the consolidation function will also be unknown. If you set it to .5 then half of the values are allowed to be unknown before the consolidated results is also unknown.

Kind regards, 
 Lieven.


On Fri, 2013-12-27 at 14:48 +0100, Lieven Hollevoet wrote:
Hey Dan,

Op 25-dec.-2013, om 08:40 heeft dbemowsk <[hidden email]> het volgende geschreven:

> My next question in all of this has to do with data history.  I understand
> that the concept of RRD revolves around the whole round robin concept where
> the data works on a fixed length timeline where data overwrites data from
> the beginning once the saved points reach the end of the data set.  My
> question is, is there a way to archive data?  What I mean is, if you have an
> RRD that collects data for a 24 hour period, and when the next day starts
> effectively overwriting the first days data, is there a way to look back at
> a graph of data from a few days ago?  I see that the weather_rrd_update.pl
> file creates a CSV file.  Is this what the CSV is used for?  If so, I am
> guessing that it just keeps appending data to the end of the CSV, correct?

Nope, the idea of RRD is that you can use it stand alone. You define data sources (DS) that keep track of values. Then for every data source you create RRAs (round robin archives) with variable ‘timespan' per datapoint. E.g. suppose that you store a new temperature every 5 minutes, then your primary ‘resolution’ for the first RRA van be 5 minutes.
Then, for longer archiving you can define RRAs that keep track of your data over longer timespans. You then need to define the consolidation function for that datasource. Suppose you define secondary RRAs with a resolution of 1 hour per datapoint. The RRD framework can then take the values from the first high-resolution RRA, apply a consolidation function on those datapoints, and then calculate the value for the lower resolution dataset. The consolidation function can be e.g. average, minimum or maximum. This way you can keep track of values over longer time (e.g. a year) in 1 hour intervals. If you want to go even further you can then store values on a daily basis for 10’s of years. 

> 
> My last question has to do with Data Sources (DS).  In the example, they
> define one DS.  In looking at the weather_rrd_update.pl file, they define a
> number of them.  Does each DS represent a different line (color) on the
> graph?

Nope, a DS is a dataset. You can see it as a representation of a sensor that you read out and that you store the values in the database. To create the graphs you use the values from the RRAs or you calculate values based on the data in those RRAs.

I hope this clarifies it a bit further.

Kind regards,
 Lieven.






------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT 
organizations don't have a clear picture of how application performance 
affects their revenue. With AppDynamics, you get 100% visibility into your 
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk


________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365



------------------------------------------------------------------------------

________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

H Plato
IMO - RRD is a little fickle. I’ve had to expand my weather.rrd so that I can capture heat, cool, and humidy setpoints. I created a script to add a source:

#!/usr/local/bin/perl 
#rrd_add_source /path/to/rrd source_name type
use strict; 
use RRD::Simple (); 

my $rrd = RRD::Simple->new(); 
my $rrdfile=$ARGV[0]; 
my $source=$ARGV[1]; 
my $type=$ARGV[2]; 
chomp($type); 
$rrd->add_source($rrdfile, $source => $type); 

CAUTION: However, I think I had to modify the mh rrd scripts, since they are particular about the amount of DEFs that they expect to see in weather.rrd. Without changing other scripts, I think the RRD generation stopped.

Since RRD is lossy (ie if you want to see state changes for a given day 3 years ago, you’re out of luck), it might make more sense to create an event logger, or other type of object that things could be tied to. All that this object would have to do would be to write timestamp,object,state to a file. I actually started something like this a few years back but generated ical files so that the open/close, on/off ‘event’ would show up as a calendar item. 

  With the graphing capabilities of IA7, it would be relatively straightforward to use the current graphing setup in IA7 to point to another source.


On Aug 28, 2016, at 11:32 AM, Steve Switzer <[hidden email]> wrote:

What came of this thread? Do we have example code for graphing things like motion events in each room, door open/close, MH mode, alarm system status, etc? It would even be nice to see when lights are turned on over time, too.

Thank you!

Steve


On 01/05/2014 03:14 PM, Lieven Hollevoet wrote:
Hey Dan,

Op 27-dec.-2013, om 18:23 heeft Dan Bemowski <[hidden email]> het volgende geschreven:

So, just to I have this straight I am going to use an abbreviated example taken from the weather_rrd_update.pl code that comes with MH.  In this, each DS defines a sensor value to show on the graph (correct?)

Each DS does define a sensor value.

, then each RRA defines the graph resolutions that are possible for each MIN, AVERAGE and MAX aggregate storing data for a defined period (1 day, 2 days) at each resolution (2 min, 5 min)?  The primary resolution for the first RRA is at 6 hours using an AVERAGE aggregate of values every minute, correct?  I hope I said that right?

The primary resolution depends on $RRD_STEP in the code. For the MisterHouse weather code this value is set to 60 seconds if I’m correct. Basically if you say 

RRA:AVERAGE:0.5:1:801 

then you tell the tool to store a value in the database every $RRD_STEP seconds, and do this for the last 801 $RRD_STEP seconds.

Why do you need ‘AVERAGE’ then? Well, you can push values to the RRD at a higher speed than defined by $RRD_STEP and then RRD will store only the average of those values every $RRD_STEP seconds.

Now, back to the code. I think the comment that states ‘for 6 hours’ is not correct. Since the $RRD_STEP is 60 seconds in weather_rrd_update, 801 times step_size is 13 hours instead of 6 hours. This might be the source of the confusion you are experiencing. It is probably something nobody has noticed up to now because 13 hours > 6 hours and the graphs show to be correct. The only disadvantage is that the database is bigger than it is absolutely required.


One last question I had in my post on this.  I am trying to understand the RRA's 0.5 value which is said to be "xff" by mr Oetiker in the beginners guide page that I listed.  What does the 0.5 represent or mean?  When I create my RRDs for graphing I can certainly use the 0.5 value that I see everyone else primarily using, but I like to know what it means.

See http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

xff The xfiles factor defines what part of a consolidation interval may be made up from*UNKNOWN* data while the consolidated value is still regarded as known. It is given as the ratio of allowed *UNKNOWN* PDPs to the number of PDPs in the interval. Thus, it ranges from 0 to 1 (exclusive).

If you fail to feed data into the database for a certain amount of time, these points in time will be marked ‘unknown’. If you start to consolidate the rate (e.g. average data points) then the off factor determines from what point on the result of the consolidation function will also be unknown. If you set it to .5 then half of the values are allowed to be unknown before the consolidated results is also unknown.

Kind regards, 
 Lieven.


On Fri, 2013-12-27 at 14:48 +0100, Lieven Hollevoet wrote:
Hey Dan,

Op 25-dec.-2013, om 08:40 heeft dbemowsk <[hidden email]> het volgende geschreven:

> My next question in all of this has to do with data history.  I understand
> that the concept of RRD revolves around the whole round robin concept where
> the data works on a fixed length timeline where data overwrites data from
> the beginning once the saved points reach the end of the data set.  My
> question is, is there a way to archive data?  What I mean is, if you have an
> RRD that collects data for a 24 hour period, and when the next day starts
> effectively overwriting the first days data, is there a way to look back at
> a graph of data from a few days ago?  I see that the weather_rrd_update.pl
> file creates a CSV file.  Is this what the CSV is used for?  If so, I am
> guessing that it just keeps appending data to the end of the CSV, correct?

Nope, the idea of RRD is that you can use it stand alone. You define data sources (DS) that keep track of values. Then for every data source you create RRAs (round robin archives) with variable ‘timespan' per datapoint. E.g. suppose that you store a new temperature every 5 minutes, then your primary ‘resolution’ for the first RRA van be 5 minutes.
Then, for longer archiving you can define RRAs that keep track of your data over longer timespans. You then need to define the consolidation function for that datasource. Suppose you define secondary RRAs with a resolution of 1 hour per datapoint. The RRD framework can then take the values from the first high-resolution RRA, apply a consolidation function on those datapoints, and then calculate the value for the lower resolution dataset. The consolidation function can be e.g. average, minimum or maximum. This way you can keep track of values over longer time (e.g. a year) in 1 hour intervals. If you want to go even further you can then store values on a daily basis for 10’s of years. 

> 
> My last question has to do with Data Sources (DS).  In the example, they
> define one DS.  In looking at the weather_rrd_update.pl file, they define a
> number of them.  Does each DS represent a different line (color) on the
> graph?

Nope, a DS is a dataset. You can see it as a representation of a sensor that you read out and that you store the values in the database. To create the graphs you use the values from the RRAs or you calculate values based on the data in those RRAs.

I hope this clarifies it a bit further.

Kind regards,
 Lieven.






------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT 
organizations don't have a clear picture of how application performance 
affects their revenue. With AppDynamics, you get 100% visibility into your 
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk


________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365


------------------------------------------------------------------------------
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users



------------------------------------------------------------------------------

________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

Lieven Hollevoet
Administrator
Hey Howard,

just a small addition for the people who follow this thread:

> Since RRD is lossy (ie if you want to see state changes for a given day 3 years ago, you’re out of luck),

It is actually as lossy as you define it to be.

If you define your dataset to store data long enough then you can store data at minute resolution for 100 years. It is just that the default settings for a dataset are not defined that way, but you can go as wild as you want to.

If you need more help with this, give me a shout. I’m using RRD databases solely for logging all data in and around my house. Right now I’m logging and creating graphs with a few custom scripts that log xPL data into RRD databases and then I make graphs out of them to visualize the data. I just need to integrate these databases into MisterHouse when I find some time :-).

Kind regards,
 Lieven.

> Op 28 aug. 2016, om 20:30 heeft H Plato <[hidden email]> het volgende geschreven:
>
> IMO - RRD is a little fickle. I’ve had to expand my weather.rrd so that I can capture heat, cool, and humidy setpoints. I created a script to add a source:
>
> #!/usr/local/bin/perl
> #rrd_add_source /path/to/rrd source_name type
> use strict;
> use RRD::Simple ();
>
> my $rrd = RRD::Simple->new();
> my $rrdfile=$ARGV[0];
> my $source=$ARGV[1];
> my $type=$ARGV[2];
> chomp($type);
> $rrd->add_source($rrdfile, $source => $type);
>
> CAUTION: However, I think I had to modify the mh rrd scripts, since they are particular about the amount of DEFs that they expect to see in weather.rrd. Without changing other scripts, I think the RRD generation stopped.
>
> Since RRD is lossy (ie if you want to see state changes for a given day 3 years ago, you’re out of luck), it might make more sense to create an event logger, or other type of object that things could be tied to. All that this object would have to do would be to write timestamp,object,state to a file. I actually started something like this a few years back but generated ical files so that the open/close, on/off ‘event’ would show up as a calendar item.
>
>   With the graphing capabilities of IA7, it would be relatively straightforward to use the current graphing setup in IA7 to point to another source.
>
>
>> On Aug 28, 2016, at 11:32 AM, Steve Switzer <[hidden email]> wrote:
>>
>> What came of this thread? Do we have example code for graphing things like motion events in each room, door open/close, MH mode, alarm system status, etc? It would even be nice to see when lights are turned on over time, too.
>>
>> Thank you!
>>
>> Steve
>>
>> On 01/05/2014 03:14 PM, Lieven Hollevoet wrote:
>>> Hey Dan,
>>>
>>> Op 27-dec.-2013, om 18:23 heeft Dan Bemowski <[hidden email]> het volgende geschreven:
>>>
>>>> So, just to I have this straight I am going to use an abbreviated example taken from the weather_rrd_update.pl code that comes with MH.  In this, each DS defines a sensor value to show on the graph (correct?)
>>>
>>> Each DS does define a sensor value.
>>>
>>>> , then each RRA defines the graph resolutions that are possible for each MIN, AVERAGE and MAX aggregate storing data for a defined period (1 day, 2 days) at each resolution (2 min, 5 min)?  The primary resolution for the first RRA is at 6 hours using an AVERAGE aggregate of values every minute, correct?  I hope I said that right?
>>>
>>> The primary resolution depends on $RRD_STEP in the code. For the MisterHouse weather code this value is set to 60 seconds if I’m correct. Basically if you say
>>>
>>> RRA:AVERAGE:0.5:1:801
>>>
>>> then you tell the tool to store a value in the database every $RRD_STEP seconds, and do this for the last 801 $RRD_STEP seconds.
>>>
>>> Why do you need ‘AVERAGE’ then? Well, you can push values to the RRD at a higher speed than defined by $RRD_STEP and then RRD will store only the average of those values every $RRD_STEP seconds.
>>>
>>> Now, back to the code. I think the comment that states ‘for 6 hours’ is not correct. Since the $RRD_STEP is 60 seconds in weather_rrd_update, 801 times step_size is 13 hours instead of 6 hours. This might be the source of the confusion you are experiencing. It is probably something nobody has noticed up to now because 13 hours > 6 hours and the graphs show to be correct. The only disadvantage is that the database is bigger than it is absolutely required.
>>>
>>>>
>>>> One last question I had in my post on this.  I am trying to understand the RRA's 0.5 value which is said to be "xff" by mr Oetiker in the beginners guide page that I listed.  What does the 0.5 represent or mean?  When I create my RRDs for graphing I can certainly use the 0.5 value that I see everyone else primarily using, but I like to know what it means.
>>>
>>> See http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html
>>>
>>> xff The xfiles factor defines what part of a consolidation interval may be made up from*UNKNOWN* data while the consolidated value is still regarded as known. It is given as the ratio of allowed *UNKNOWN* PDPs to the number of PDPs in the interval. Thus, it ranges from 0 to 1 (exclusive).
>>>
>>> If you fail to feed data into the database for a certain amount of time, these points in time will be marked ‘unknown’. If you start to consolidate the rate (e.g. average data points) then the off factor determines from what point on the result of the consolidation function will also be unknown. If you set it to .5 then half of the values are allowed to be unknown before the consolidated results is also unknown.
>>>
>>> Kind regards,
>>>  Lieven.
>>>
>>>>
>>>> On Fri, 2013-12-27 at 14:48 +0100, Lieven Hollevoet wrote:
>>>>> Hey Dan,
>>>>>
>>>>> Op 25-dec.-2013, om 08:40 heeft dbemowsk <
>>>>> [hidden email]
>>>>> > het volgende geschreven:
>>>>>
>>>>>
>>>>> > My next question in all of this has to do with data history.  I understand
>>>>> > that the concept of RRD revolves around the whole round robin concept where
>>>>> > the data works on a fixed length timeline where data overwrites data from
>>>>> > the beginning once the saved points reach the end of the data set.  My
>>>>> > question is, is there a way to archive data?  What I mean is, if you have an
>>>>> > RRD that collects data for a 24 hour period, and when the next day starts
>>>>> > effectively overwriting the first days data, is there a way to look back at
>>>>> > a graph of data from a few days ago?  I see that the weather_rrd_update.pl
>>>>> > file creates a CSV file.  Is this what the CSV is used for?  If so, I am
>>>>> > guessing that it just keeps appending data to the end of the CSV, correct?
>>>>>
>>>>>
>>>>> Nope, the idea of RRD is that you can use it stand alone. You define data sources (DS) that keep track of values. Then for every data source you create RRAs (round robin archives) with variable ‘timespan' per datapoint. E.g. suppose that you store a new temperature every 5 minutes, then your primary ‘resolution’ for the first RRA van be 5 minutes.
>>>>> Then, for longer archiving you can define RRAs that keep track of your data over longer timespans. You then need to define the consolidation function for that datasource. Suppose you define secondary RRAs with a resolution of 1 hour per datapoint. The RRD framework can then take the values from the first high-resolution RRA, apply a consolidation function on those datapoints, and then calculate the value for the lower resolution dataset. The consolidation function can be e.g. average, minimum or maximum. This way you can keep track of values over longer time (e.g. a year) in 1 hour intervals. If you want to go even further you can then store values on a daily basis for 10’s of years.
>>>>>
>>>>>
>>>>> >
>>>>> > My last question has to do with Data Sources (DS).  In the example, they
>>>>> > define one DS.  In looking at the weather_rrd_update.pl file, they define a
>>>>> > number of them.  Does each DS represent a different line (color) on the
>>>>> > graph?
>>>>>
>>>>>
>>>>> Nope, a DS is a dataset. You can see it as a representation of a sensor that you read out and that you store the values in the database. To create the graphs you use the values from the RRAs or you calculate values based on the data in those RRAs.
>>>>>
>>>>> I hope this clarifies it a bit further.
>>>>>
>>>>> Kind regards,
>>>>>  Lieven.
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Rapidly troubleshoot problems before they affect your business. Most IT
>>> organizations don't have a clear picture of how application performance
>>> affects their revenue. With AppDynamics, you get 100% visibility into your
>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
>>>
>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
>>>
>>>
>>> ________________________________________________________
>>> To unsubscribe from this list, go to:
>>> http://sourceforge.net/mail/?group_id=1365
>>>
>>>
>>>
>>
>> ------------------------------------------------------------------------------
>> ________________________________________________________
>> To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>>
>
> ------------------------------------------------------------------------------
> ________________________________________________________
> To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>

------------------------------------------------------------------------------

________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users


signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

H Plato
That’s good to know. I thought the use case for RRD was when you needed less precision as time went on.

RRD is time driven, not even driven, right? Right now the RRD setup in MH takes the weather hash and dumps it into the rrd every minute. Since objects are event driven, there can be instances were there are events 10 seconds apart, and then nothing for a few hours. Keeping second precision for everything seems like overkill. Is there a way around this?

If you have a bit more time, look at json_server.pl and the rrd section. I hard code ‘weather.rrd’, and some weather specific manipulations, but you can see how the data is served up to the IA7 graphing piece. We might be able to make that more generic, so that by specifying any rrd file, it would grab the associated ia7_rrd_config file and then graph it.


> On Aug 28, 2016, at 1:13 PM, Lieven Hollevoet <[hidden email]> wrote:
>
> Hey Howard,
>
> just a small addition for the people who follow this thread:
>
>> Since RRD is lossy (ie if you want to see state changes for a given day 3 years ago, you’re out of luck),
>
> It is actually as lossy as you define it to be.
>
> If you define your dataset to store data long enough then you can store data at minute resolution for 100 years. It is just that the default settings for a dataset are not defined that way, but you can go as wild as you want to.
>
> If you need more help with this, give me a shout. I’m using RRD databases solely for logging all data in and around my house. Right now I’m logging and creating graphs with a few custom scripts that log xPL data into RRD databases and then I make graphs out of them to visualize the data. I just need to integrate these databases into MisterHouse when I find some time :-).
>
> Kind regards,
> Lieven.
>
>> Op 28 aug. 2016, om 20:30 heeft H Plato <[hidden email]> het volgende geschreven:
>>
>> IMO - RRD is a little fickle. I’ve had to expand my weather.rrd so that I can capture heat, cool, and humidy setpoints. I created a script to add a source:
>>
>> #!/usr/local/bin/perl
>> #rrd_add_source /path/to/rrd source_name type
>> use strict;
>> use RRD::Simple ();
>>
>> my $rrd = RRD::Simple->new();
>> my $rrdfile=$ARGV[0];
>> my $source=$ARGV[1];
>> my $type=$ARGV[2];
>> chomp($type);
>> $rrd->add_source($rrdfile, $source => $type);
>>
>> CAUTION: However, I think I had to modify the mh rrd scripts, since they are particular about the amount of DEFs that they expect to see in weather.rrd. Without changing other scripts, I think the RRD generation stopped.
>>
>> Since RRD is lossy (ie if you want to see state changes for a given day 3 years ago, you’re out of luck), it might make more sense to create an event logger, or other type of object that things could be tied to. All that this object would have to do would be to write timestamp,object,state to a file. I actually started something like this a few years back but generated ical files so that the open/close, on/off ‘event’ would show up as a calendar item.
>>
>>  With the graphing capabilities of IA7, it would be relatively straightforward to use the current graphing setup in IA7 to point to another source.
>>
>>
>>> On Aug 28, 2016, at 11:32 AM, Steve Switzer <[hidden email]> wrote:
>>>
>>> What came of this thread? Do we have example code for graphing things like motion events in each room, door open/close, MH mode, alarm system status, etc? It would even be nice to see when lights are turned on over time, too.
>>>
>>> Thank you!
>>>
>>> Steve
>>>
>>> On 01/05/2014 03:14 PM, Lieven Hollevoet wrote:
>>>> Hey Dan,
>>>>
>>>> Op 27-dec.-2013, om 18:23 heeft Dan Bemowski <[hidden email]> het volgende geschreven:
>>>>
>>>>> So, just to I have this straight I am going to use an abbreviated example taken from the weather_rrd_update.pl code that comes with MH.  In this, each DS defines a sensor value to show on the graph (correct?)
>>>>
>>>> Each DS does define a sensor value.
>>>>
>>>>> , then each RRA defines the graph resolutions that are possible for each MIN, AVERAGE and MAX aggregate storing data for a defined period (1 day, 2 days) at each resolution (2 min, 5 min)?  The primary resolution for the first RRA is at 6 hours using an AVERAGE aggregate of values every minute, correct?  I hope I said that right?
>>>>
>>>> The primary resolution depends on $RRD_STEP in the code. For the MisterHouse weather code this value is set to 60 seconds if I’m correct. Basically if you say
>>>>
>>>> RRA:AVERAGE:0.5:1:801
>>>>
>>>> then you tell the tool to store a value in the database every $RRD_STEP seconds, and do this for the last 801 $RRD_STEP seconds.
>>>>
>>>> Why do you need ‘AVERAGE’ then? Well, you can push values to the RRD at a higher speed than defined by $RRD_STEP and then RRD will store only the average of those values every $RRD_STEP seconds.
>>>>
>>>> Now, back to the code. I think the comment that states ‘for 6 hours’ is not correct. Since the $RRD_STEP is 60 seconds in weather_rrd_update, 801 times step_size is 13 hours instead of 6 hours. This might be the source of the confusion you are experiencing. It is probably something nobody has noticed up to now because 13 hours > 6 hours and the graphs show to be correct. The only disadvantage is that the database is bigger than it is absolutely required.
>>>>
>>>>>
>>>>> One last question I had in my post on this.  I am trying to understand the RRA's 0.5 value which is said to be "xff" by mr Oetiker in the beginners guide page that I listed.  What does the 0.5 represent or mean?  When I create my RRDs for graphing I can certainly use the 0.5 value that I see everyone else primarily using, but I like to know what it means.
>>>>
>>>> See http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html
>>>>
>>>> xff The xfiles factor defines what part of a consolidation interval may be made up from*UNKNOWN* data while the consolidated value is still regarded as known. It is given as the ratio of allowed *UNKNOWN* PDPs to the number of PDPs in the interval. Thus, it ranges from 0 to 1 (exclusive).
>>>>
>>>> If you fail to feed data into the database for a certain amount of time, these points in time will be marked ‘unknown’. If you start to consolidate the rate (e.g. average data points) then the off factor determines from what point on the result of the consolidation function will also be unknown. If you set it to .5 then half of the values are allowed to be unknown before the consolidated results is also unknown.
>>>>
>>>> Kind regards,
>>>> Lieven.
>>>>
>>>>>
>>>>> On Fri, 2013-12-27 at 14:48 +0100, Lieven Hollevoet wrote:
>>>>>> Hey Dan,
>>>>>>
>>>>>> Op 25-dec.-2013, om 08:40 heeft dbemowsk <
>>>>>> [hidden email]
>>>>>>> het volgende geschreven:
>>>>>>
>>>>>>
>>>>>>> My next question in all of this has to do with data history.  I understand
>>>>>>> that the concept of RRD revolves around the whole round robin concept where
>>>>>>> the data works on a fixed length timeline where data overwrites data from
>>>>>>> the beginning once the saved points reach the end of the data set.  My
>>>>>>> question is, is there a way to archive data?  What I mean is, if you have an
>>>>>>> RRD that collects data for a 24 hour period, and when the next day starts
>>>>>>> effectively overwriting the first days data, is there a way to look back at
>>>>>>> a graph of data from a few days ago?  I see that the weather_rrd_update.pl
>>>>>>> file creates a CSV file.  Is this what the CSV is used for?  If so, I am
>>>>>>> guessing that it just keeps appending data to the end of the CSV, correct?
>>>>>>
>>>>>>
>>>>>> Nope, the idea of RRD is that you can use it stand alone. You define data sources (DS) that keep track of values. Then for every data source you create RRAs (round robin archives) with variable ‘timespan' per datapoint. E.g. suppose that you store a new temperature every 5 minutes, then your primary ‘resolution’ for the first RRA van be 5 minutes.
>>>>>> Then, for longer archiving you can define RRAs that keep track of your data over longer timespans. You then need to define the consolidation function for that datasource. Suppose you define secondary RRAs with a resolution of 1 hour per datapoint. The RRD framework can then take the values from the first high-resolution RRA, apply a consolidation function on those datapoints, and then calculate the value for the lower resolution dataset. The consolidation function can be e.g. average, minimum or maximum. This way you can keep track of values over longer time (e.g. a year) in 1 hour intervals. If you want to go even further you can then store values on a daily basis for 10’s of years.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> My last question has to do with Data Sources (DS).  In the example, they
>>>>>>> define one DS.  In looking at the weather_rrd_update.pl file, they define a
>>>>>>> number of them.  Does each DS represent a different line (color) on the
>>>>>>> graph?
>>>>>>
>>>>>>
>>>>>> Nope, a DS is a dataset. You can see it as a representation of a sensor that you read out and that you store the values in the database. To create the graphs you use the values from the RRAs or you calculate values based on the data in those RRAs.
>>>>>>
>>>>>> I hope this clarifies it a bit further.
>>>>>>
>>>>>> Kind regards,
>>>>>> Lieven.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Rapidly troubleshoot problems before they affect your business. Most IT
>>>> organizations don't have a clear picture of how application performance
>>>> affects their revenue. With AppDynamics, you get 100% visibility into your
>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
>>>>
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
>>>>
>>>>
>>>> ________________________________________________________
>>>> To unsubscribe from this list, go to:
>>>> http://sourceforge.net/mail/?group_id=1365
>>>>
>>>>
>>>>
>>>
>>> ------------------------------------------------------------------------------
>>> ________________________________________________________
>>> To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>>>
>>
>> ------------------------------------------------------------------------------
>> ________________________________________________________
>> To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>>
>


------------------------------------------------------------------------------
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

Lieven Hollevoet
Administrator
Hello Howard,

the idea of an RRD database is indeed that you ‘push’ a value to the database at regular time intervals.

If you have events that are shorter than an interval then there is a chance they are not picked up if you only push data every minute. The only way I know around this is indeed to make the interval at which you feed data into the RRD shorter.

Best regards,
 Lieven.

> Op 28 aug. 2016, om 21:53 heeft H Plato <[hidden email]> het volgende geschreven:
>
> That’s good to know. I thought the use case for RRD was when you needed less precision as time went on.
>
> RRD is time driven, not even driven, right? Right now the RRD setup in MH takes the weather hash and dumps it into the rrd every minute. Since objects are event driven, there can be instances were there are events 10 seconds apart, and then nothing for a few hours. Keeping second precision for everything seems like overkill. Is there a way around this?
>
> If you have a bit more time, look at json_server.pl and the rrd section. I hard code ‘weather.rrd’, and some weather specific manipulations, but you can see how the data is served up to the IA7 graphing piece. We might be able to make that more generic, so that by specifying any rrd file, it would grab the associated ia7_rrd_config file and then graph it.
>
>
>> On Aug 28, 2016, at 1:13 PM, Lieven Hollevoet <[hidden email]> wrote:
>>
>> Hey Howard,
>>
>> just a small addition for the people who follow this thread:
>>
>>> Since RRD is lossy (ie if you want to see state changes for a given day 3 years ago, you’re out of luck),
>>
>> It is actually as lossy as you define it to be.
>>
>> If you define your dataset to store data long enough then you can store data at minute resolution for 100 years. It is just that the default settings for a dataset are not defined that way, but you can go as wild as you want to.
>>
>> If you need more help with this, give me a shout. I’m using RRD databases solely for logging all data in and around my house. Right now I’m logging and creating graphs with a few custom scripts that log xPL data into RRD databases and then I make graphs out of them to visualize the data. I just need to integrate these databases into MisterHouse when I find some time :-).
>>
>> Kind regards,
>> Lieven.
>>
>>> Op 28 aug. 2016, om 20:30 heeft H Plato <[hidden email]> het volgende geschreven:
>>>
>>> IMO - RRD is a little fickle. I’ve had to expand my weather.rrd so that I can capture heat, cool, and humidy setpoints. I created a script to add a source:
>>>
>>> #!/usr/local/bin/perl
>>> #rrd_add_source /path/to/rrd source_name type
>>> use strict;
>>> use RRD::Simple ();
>>>
>>> my $rrd = RRD::Simple->new();
>>> my $rrdfile=$ARGV[0];
>>> my $source=$ARGV[1];
>>> my $type=$ARGV[2];
>>> chomp($type);
>>> $rrd->add_source($rrdfile, $source => $type);
>>>
>>> CAUTION: However, I think I had to modify the mh rrd scripts, since they are particular about the amount of DEFs that they expect to see in weather.rrd. Without changing other scripts, I think the RRD generation stopped.
>>>
>>> Since RRD is lossy (ie if you want to see state changes for a given day 3 years ago, you’re out of luck), it might make more sense to create an event logger, or other type of object that things could be tied to. All that this object would have to do would be to write timestamp,object,state to a file. I actually started something like this a few years back but generated ical files so that the open/close, on/off ‘event’ would show up as a calendar item.
>>>
>>> With the graphing capabilities of IA7, it would be relatively straightforward to use the current graphing setup in IA7 to point to another source.
>>>
>>>
>>>> On Aug 28, 2016, at 11:32 AM, Steve Switzer <[hidden email]> wrote:
>>>>
>>>> What came of this thread? Do we have example code for graphing things like motion events in each room, door open/close, MH mode, alarm system status, etc? It would even be nice to see when lights are turned on over time, too.
>>>>
>>>> Thank you!
>>>>
>>>> Steve
>>>>
>>>> On 01/05/2014 03:14 PM, Lieven Hollevoet wrote:
>>>>> Hey Dan,
>>>>>
>>>>> Op 27-dec.-2013, om 18:23 heeft Dan Bemowski <[hidden email]> het volgende geschreven:
>>>>>
>>>>>> So, just to I have this straight I am going to use an abbreviated example taken from the weather_rrd_update.pl code that comes with MH.  In this, each DS defines a sensor value to show on the graph (correct?)
>>>>>
>>>>> Each DS does define a sensor value.
>>>>>
>>>>>> , then each RRA defines the graph resolutions that are possible for each MIN, AVERAGE and MAX aggregate storing data for a defined period (1 day, 2 days) at each resolution (2 min, 5 min)?  The primary resolution for the first RRA is at 6 hours using an AVERAGE aggregate of values every minute, correct?  I hope I said that right?
>>>>>
>>>>> The primary resolution depends on $RRD_STEP in the code. For the MisterHouse weather code this value is set to 60 seconds if I’m correct. Basically if you say
>>>>>
>>>>> RRA:AVERAGE:0.5:1:801
>>>>>
>>>>> then you tell the tool to store a value in the database every $RRD_STEP seconds, and do this for the last 801 $RRD_STEP seconds.
>>>>>
>>>>> Why do you need ‘AVERAGE’ then? Well, you can push values to the RRD at a higher speed than defined by $RRD_STEP and then RRD will store only the average of those values every $RRD_STEP seconds.
>>>>>
>>>>> Now, back to the code. I think the comment that states ‘for 6 hours’ is not correct. Since the $RRD_STEP is 60 seconds in weather_rrd_update, 801 times step_size is 13 hours instead of 6 hours. This might be the source of the confusion you are experiencing. It is probably something nobody has noticed up to now because 13 hours > 6 hours and the graphs show to be correct. The only disadvantage is that the database is bigger than it is absolutely required.
>>>>>
>>>>>>
>>>>>> One last question I had in my post on this.  I am trying to understand the RRA's 0.5 value which is said to be "xff" by mr Oetiker in the beginners guide page that I listed.  What does the 0.5 represent or mean?  When I create my RRDs for graphing I can certainly use the 0.5 value that I see everyone else primarily using, but I like to know what it means.
>>>>>
>>>>> See http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html
>>>>>
>>>>> xff The xfiles factor defines what part of a consolidation interval may be made up from*UNKNOWN* data while the consolidated value is still regarded as known. It is given as the ratio of allowed *UNKNOWN* PDPs to the number of PDPs in the interval. Thus, it ranges from 0 to 1 (exclusive).
>>>>>
>>>>> If you fail to feed data into the database for a certain amount of time, these points in time will be marked ‘unknown’. If you start to consolidate the rate (e.g. average data points) then the off factor determines from what point on the result of the consolidation function will also be unknown. If you set it to .5 then half of the values are allowed to be unknown before the consolidated results is also unknown.
>>>>>
>>>>> Kind regards,
>>>>> Lieven.
>>>>>
>>>>>>
>>>>>> On Fri, 2013-12-27 at 14:48 +0100, Lieven Hollevoet wrote:
>>>>>>> Hey Dan,
>>>>>>>
>>>>>>> Op 25-dec.-2013, om 08:40 heeft dbemowsk <
>>>>>>> [hidden email]
>>>>>>>> het volgende geschreven:
>>>>>>>
>>>>>>>
>>>>>>>> My next question in all of this has to do with data history.  I understand
>>>>>>>> that the concept of RRD revolves around the whole round robin concept where
>>>>>>>> the data works on a fixed length timeline where data overwrites data from
>>>>>>>> the beginning once the saved points reach the end of the data set.  My
>>>>>>>> question is, is there a way to archive data?  What I mean is, if you have an
>>>>>>>> RRD that collects data for a 24 hour period, and when the next day starts
>>>>>>>> effectively overwriting the first days data, is there a way to look back at
>>>>>>>> a graph of data from a few days ago?  I see that the weather_rrd_update.pl
>>>>>>>> file creates a CSV file.  Is this what the CSV is used for?  If so, I am
>>>>>>>> guessing that it just keeps appending data to the end of the CSV, correct?
>>>>>>>
>>>>>>>
>>>>>>> Nope, the idea of RRD is that you can use it stand alone. You define data sources (DS) that keep track of values. Then for every data source you create RRAs (round robin archives) with variable ‘timespan' per datapoint. E.g. suppose that you store a new temperature every 5 minutes, then your primary ‘resolution’ for the first RRA van be 5 minutes.
>>>>>>> Then, for longer archiving you can define RRAs that keep track of your data over longer timespans. You then need to define the consolidation function for that datasource. Suppose you define secondary RRAs with a resolution of 1 hour per datapoint. The RRD framework can then take the values from the first high-resolution RRA, apply a consolidation function on those datapoints, and then calculate the value for the lower resolution dataset. The consolidation function can be e.g. average, minimum or maximum. This way you can keep track of values over longer time (e.g. a year) in 1 hour intervals. If you want to go even further you can then store values on a daily basis for 10’s of years.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> My last question has to do with Data Sources (DS).  In the example, they
>>>>>>>> define one DS.  In looking at the weather_rrd_update.pl file, they define a
>>>>>>>> number of them.  Does each DS represent a different line (color) on the
>>>>>>>> graph?
>>>>>>>
>>>>>>>
>>>>>>> Nope, a DS is a dataset. You can see it as a representation of a sensor that you read out and that you store the values in the database. To create the graphs you use the values from the RRAs or you calculate values based on the data in those RRAs.
>>>>>>>
>>>>>>> I hope this clarifies it a bit further.
>>>>>>>
>>>>>>> Kind regards,
>>>>>>> Lieven.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Rapidly troubleshoot problems before they affect your business. Most IT
>>>>> organizations don't have a clear picture of how application performance
>>>>> affects their revenue. With AppDynamics, you get 100% visibility into your
>>>>> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
>>>>>
>>>>> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk
>>>>>
>>>>>
>>>>> ________________________________________________________
>>>>> To unsubscribe from this list, go to:
>>>>> http://sourceforge.net/mail/?group_id=1365
>>>>>
>>>>>
>>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> ________________________________________________________
>>>> To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>>>>
>>>
>>> ------------------------------------------------------------------------------
>>> ________________________________________________________
>>> To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>>>
>>
>

------------------------------------------------------------------------------

________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users


signature.asc (465 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

H Plato
In reply to this post by Steve Switzer
So I’ve been playing around with this a bit. I’ve extended the generic_item to add in a logger_enable method. If this method is added at $Startup, it will log state changes to an text log file $Data/object_log/<object>/YYYY/MM/DD.log 

I thought it might make sense to have this on an ‘opt-in’ basis, since this might create more processing overhead and more writes which is an issue for things like raspberry pi’s. It might make sense to ‘modularize’ this so that generic_item calls a library function — so that others could add in a mysql backend, or sqllite, or something else. Simple text files are good enough for my purpose.

Now, it’s integrated into IA7, so for those objects that have this method enabled, a graphic will show up on the modal:



Clicking on the green graph button will display points when objects are in various states:


I figured this would be a good improvement for an upcoming v4.2 release. Anyone have any comments, suggestions, or other ideas?

On Aug 28, 2016, at 11:32 AM, Steve Switzer <[hidden email]> wrote:

What came of this thread? Do we have example code for graphing things like motion events in each room, door open/close, MH mode, alarm system status, etc? It would even be nice to see when lights are turned on over time, too.

Thank you!

Steve


On 01/05/2014 03:14 PM, Lieven Hollevoet wrote:
Hey Dan,

Op 27-dec.-2013, om 18:23 heeft Dan Bemowski <[hidden email]> het volgende geschreven:

So, just to I have this straight I am going to use an abbreviated example taken from the weather_rrd_update.pl code that comes with MH.  In this, each DS defines a sensor value to show on the graph (correct?)

Each DS does define a sensor value.

, then each RRA defines the graph resolutions that are possible for each MIN, AVERAGE and MAX aggregate storing data for a defined period (1 day, 2 days) at each resolution (2 min, 5 min)?  The primary resolution for the first RRA is at 6 hours using an AVERAGE aggregate of values every minute, correct?  I hope I said that right?

The primary resolution depends on $RRD_STEP in the code. For the MisterHouse weather code this value is set to 60 seconds if I’m correct. Basically if you say 

RRA:AVERAGE:0.5:1:801 

then you tell the tool to store a value in the database every $RRD_STEP seconds, and do this for the last 801 $RRD_STEP seconds.

Why do you need ‘AVERAGE’ then? Well, you can push values to the RRD at a higher speed than defined by $RRD_STEP and then RRD will store only the average of those values every $RRD_STEP seconds.

Now, back to the code. I think the comment that states ‘for 6 hours’ is not correct. Since the $RRD_STEP is 60 seconds in weather_rrd_update, 801 times step_size is 13 hours instead of 6 hours. This might be the source of the confusion you are experiencing. It is probably something nobody has noticed up to now because 13 hours > 6 hours and the graphs show to be correct. The only disadvantage is that the database is bigger than it is absolutely required.


One last question I had in my post on this.  I am trying to understand the RRA's 0.5 value which is said to be "xff" by mr Oetiker in the beginners guide page that I listed.  What does the 0.5 represent or mean?  When I create my RRDs for graphing I can certainly use the 0.5 value that I see everyone else primarily using, but I like to know what it means.

See http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html

xff The xfiles factor defines what part of a consolidation interval may be made up from*UNKNOWN* data while the consolidated value is still regarded as known. It is given as the ratio of allowed *UNKNOWN* PDPs to the number of PDPs in the interval. Thus, it ranges from 0 to 1 (exclusive).

If you fail to feed data into the database for a certain amount of time, these points in time will be marked ‘unknown’. If you start to consolidate the rate (e.g. average data points) then the off factor determines from what point on the result of the consolidation function will also be unknown. If you set it to .5 then half of the values are allowed to be unknown before the consolidated results is also unknown.

Kind regards, 
 Lieven.


On Fri, 2013-12-27 at 14:48 +0100, Lieven Hollevoet wrote:
Hey Dan,

Op 25-dec.-2013, om 08:40 heeft dbemowsk <[hidden email]> het volgende geschreven:

> My next question in all of this has to do with data history.  I understand
> that the concept of RRD revolves around the whole round robin concept where
> the data works on a fixed length timeline where data overwrites data from
> the beginning once the saved points reach the end of the data set.  My
> question is, is there a way to archive data?  What I mean is, if you have an
> RRD that collects data for a 24 hour period, and when the next day starts
> effectively overwriting the first days data, is there a way to look back at
> a graph of data from a few days ago?  I see that the weather_rrd_update.pl
> file creates a CSV file.  Is this what the CSV is used for?  If so, I am
> guessing that it just keeps appending data to the end of the CSV, correct?

Nope, the idea of RRD is that you can use it stand alone. You define data sources (DS) that keep track of values. Then for every data source you create RRAs (round robin archives) with variable ‘timespan' per datapoint. E.g. suppose that you store a new temperature every 5 minutes, then your primary ‘resolution’ for the first RRA van be 5 minutes.
Then, for longer archiving you can define RRAs that keep track of your data over longer timespans. You then need to define the consolidation function for that datasource. Suppose you define secondary RRAs with a resolution of 1 hour per datapoint. The RRD framework can then take the values from the first high-resolution RRA, apply a consolidation function on those datapoints, and then calculate the value for the lower resolution dataset. The consolidation function can be e.g. average, minimum or maximum. This way you can keep track of values over longer time (e.g. a year) in 1 hour intervals. If you want to go even further you can then store values on a daily basis for 10’s of years. 

> 
> My last question has to do with Data Sources (DS).  In the example, they
> define one DS.  In looking at the weather_rrd_update.pl file, they define a
> number of them.  Does each DS represent a different line (color) on the
> graph?

Nope, a DS is a dataset. You can see it as a representation of a sensor that you read out and that you store the values in the database. To create the graphs you use the values from the RRAs or you calculate values based on the data in those RRAs.

I hope this clarifies it a bit further.

Kind regards,
 Lieven.






------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT 
organizations don't have a clear picture of how application performance 
affects their revenue. With AppDynamics, you get 100% visibility into your 
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk


________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365


------------------------------------------------------------------------------
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users



------------------------------------------------------------------------------

________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

Tobias Sachs
Hello,
I really love the new feature, i was thinking some time over how i
could get some sort of graphing functionality. And you just did it!
Thanks!

And since you asked for comments, suggestions and other ideas...
here we go:

What I do not like in its current state is the date format in the
textboxes above the graph, this is just plain wrong for almost anybody ;-)
It would be nice to change this to a more sane format or even better
have a configuration option.

It would be cool too have a summary of the selected timespan
below the graph to see the total time the item spent in each state.

For items that keep track of measured data like temperature it might
be good if there was a way to change graph to a curve instead of dots.

It would also be nice if one was able to generate a combined graph of several
items.

Just some ideas...

Tobi


H Plato <[hidden email]> wrote:

> So I’ve been playing around with this a bit. I’ve extended the generic_item to add in a logger_enable method. If this method is added at $Startup, it will log state changes to an text log file $Data/object_log/<object>/YYYY/MM/DD.log
>
> I thought it might make sense to have this on an ‘opt-in’ basis, since this might create more processing overhead and more writes which is an issue for things like raspberry pi’s. It might make sense to ‘modularize’ this so that generic_item calls a library function — so that others could add in a mysql backend, or sqllite, or something else. Simple text files are good enough for my purpose.
>
> Now, it’s integrated into IA7, so for those objects that have this method enabled, a graphic will show up on the modal:
>
>
>
>
> Clicking on the green graph button will display points when objects are in various states:
>
>
>
> I figured this would be a good improvement for an upcoming v4.2 release. Anyone have any comments, suggestions, or other ideas?
>
> > On Aug 28, 2016, at 11:32 AM, Steve Switzer <[hidden email]> wrote:
> >
> > What came of this thread? Do we have example code for graphing things like motion events in each room, door open/close, MH mode, alarm system status, etc? It would even be nice to see when lights are turned on over time, too.
> >
> > Thank you!
> >
> > Steve
> >
> > On 01/05/2014 03:14 PM, Lieven Hollevoet wrote:
> >> Hey Dan,
> >>
> >> Op 27-dec.-2013, om 18:23 heeft Dan Bemowski <[hidden email] <mailto:[hidden email]>> het volgende geschreven:
> >>
> >>> So, just to I have this straight I am going to use an abbreviated example taken from the weather_rrd_update.pl code that comes with MH.  In this, each DS defines a sensor value to show on the graph (correct?)
> >>
> >> Each DS does define a sensor value.
> >>
> >>> , then each RRA defines the graph resolutions that are possible for each MIN, AVERAGE and MAX aggregate storing data for a defined period (1 day, 2 days) at each resolution (2 min, 5 min)?  The primary resolution for the first RRA is at 6 hours using an AVERAGE aggregate of values every minute, correct?  I hope I said that right?
> >>
> >> The primary resolution depends on $RRD_STEP in the code. For the MisterHouse weather code this value is set to 60 seconds if I’m correct. Basically if you say
> >>
> >> RRA:AVERAGE:0.5:1:801
> >>
> >> then you tell the tool to store a value in the database every $RRD_STEP seconds, and do this for the last 801 $RRD_STEP seconds.
> >>
> >> Why do you need ‘AVERAGE’ then? Well, you can push values to the RRD at a higher speed than defined by $RRD_STEP and then RRD will store only the average of those values every $RRD_STEP seconds.
> >>
> >> Now, back to the code. I think the comment that states ‘for 6 hours’ is not correct. Since the $RRD_STEP is 60 seconds in weather_rrd_update, 801 times step_size is 13 hours instead of 6 hours. This might be the source of the confusion you are experiencing. It is probably something nobody has noticed up to now because 13 hours > 6 hours and the graphs show to be correct. The only disadvantage is that the database is bigger than it is absolutely required.
> >>
> >>>
> >>> One last question I had in my post on this.  I am trying to understand the RRA's 0.5 value which is said to be "xff" by mr Oetiker in the beginners guide page that I listed.  What does the 0.5 represent or mean?  When I create my RRDs for graphing I can certainly use the 0.5 value that I see everyone else primarily using, but I like to know what it means.
> >>
> >> See http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html <http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html>
> >>
> >> xff The xfiles factor defines what part of a consolidation interval may be made up from*UNKNOWN* data while the consolidated value is still regarded as known. It is given as the ratio of allowed *UNKNOWN* PDPs to the number of PDPs in the interval. Thus, it ranges from 0 to 1 (exclusive).
> >>
> >> If you fail to feed data into the database for a certain amount of time, these points in time will be marked ‘unknown’. If you start to consolidate the rate (e.g. average data points) then the off factor determines from what point on the result of the consolidation function will also be unknown. If you set it to .5 then half of the values are allowed to be unknown before the consolidated results is also unknown.
> >>
> >> Kind regards,
> >>  Lieven.
> >>
> >>>
> >>> On Fri, 2013-12-27 at 14:48 +0100, Lieven Hollevoet wrote:
> >>>>
> >>>> Hey Dan,
> >>>>
> >>>> Op 25-dec.-2013, om 08:40 heeft dbemowsk <[hidden email] <mailto:[hidden email]>> het volgende geschreven:
> >>>>
> >>>> > My next question in all of this has to do with data history.  I understand
> >>>> > that the concept of RRD revolves around the whole round robin concept where
> >>>> > the data works on a fixed length timeline where data overwrites data from
> >>>> > the beginning once the saved points reach the end of the data set.  My
> >>>> > question is, is there a way to archive data?  What I mean is, if you have an
> >>>> > RRD that collects data for a 24 hour period, and when the next day starts
> >>>> > effectively overwriting the first days data, is there a way to look back at
> >>>> > a graph of data from a few days ago?  I see that the weather_rrd_update.pl
> >>>> > file creates a CSV file.  Is this what the CSV is used for?  If so, I am
> >>>> > guessing that it just keeps appending data to the end of the CSV, correct?
> >>>>
> >>>> Nope, the idea of RRD is that you can use it stand alone. You define data sources (DS) that keep track of values. Then for every data source you create RRAs (round robin archives) with variable ‘timespan' per datapoint. E.g. suppose that you store a new temperature every 5 minutes, then your primary ‘resolution’ for the first RRA van be 5 minutes.
> >>>> Then, for longer archiving you can define RRAs that keep track of your data over longer timespans. You then need to define the consolidation function for that datasource. Suppose you define secondary RRAs with a resolution of 1 hour per datapoint. The RRD framework can then take the values from the first high-resolution RRA, apply a consolidation function on those datapoints, and then calculate the value for the lower resolution dataset. The consolidation function can be e.g. average, minimum or maximum. This way you can keep track of values over longer time (e.g. a year) in 1 hour intervals. If you want to go even further you can then store values on a daily basis for 10’s of years.
> >>>>
> >>>> >
> >>>> > My last question has to do with Data Sources (DS).  In the example, they
> >>>> > define one DS.  In looking at the weather_rrd_update.pl file, they define a
> >>>> > number of them.  Does each DS represent a different line (color) on the
> >>>> > graph?
> >>>>
> >>>> Nope, a DS is a dataset. You can see it as a representation of a sensor that you read out and that you store the values in the database. To create the graphs you use the values from the RRAs or you calculate values based on the data in those RRAs.
> >>>>
> >>>> I hope this clarifies it a bit further.
> >>>>
> >>>> Kind regards,
> >>>>  Lieven.
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >> ------------------------------------------------------------------------------
> >> Rapidly troubleshoot problems before they affect your business. Most IT
> >> organizations don't have a clear picture of how application performance
> >> affects their revenue. With AppDynamics, you get 100% visibility into your
> >> Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
> >> http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk <http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk>
> >>
> >> ________________________________________________________
> >> To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365 <http://sourceforge.net/mail/?group_id=1365>
> >>
> >
> > ------------------------------------------------------------------------------
> > ________________________________________________________
> > To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users
> >
>

> ------------------------------------------------------------------------------

> ________________________________________________________
> To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users
>


------------------------------------------------------------------------------
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: generating rrd graps for things other than weather

H Plato
What do you propose around the date picker. That's actually the piece I like best! It supports mobile, and with it on top makes it easy to change. Unfortunately the bootstrap datepicker has broken responsive tables, so that will need to be fixed. 

It supports combining objects (that's why there is a legend on the side), however I haven't thought about how that would be triggered. Adding a long press to a group adds in a bunch more complexity, and there's a lot of complexity in ia7 already...

I'm thinking about a graph/table button that would switch the layout. I'm not that happy with the graph layout, so maybe the table might work better for a first pass. 

Dots vs bars vs lines, thats a good idea. 

Sent from my mobile device. 

On Sep 13, 2016, at 1:03 PM, Tobias Sachs <[hidden email]> wrote:

Hello,
I really love the new feature, i was thinking some time over how i
could get some sort of graphing functionality. And you just did it!
Thanks!

And since you asked for comments, suggestions and other ideas...
here we go:

What I do not like in its current state is the date format in the
textboxes above the graph, this is just plain wrong for almost anybody ;-)
It would be nice to change this to a more sane format or even better
have a configuration option.

It would be cool too have a summary of the selected timespan
below the graph to see the total time the item spent in each state.

For items that keep track of measured data like temperature it might
be good if there was a way to change graph to a curve instead of dots.

It would also be nice if one was able to generate a combined graph of several
items.

Just some ideas...

Tobi


H Plato <[hidden email]> wrote:
So I’ve been playing around with this a bit. I’ve extended the generic_item to add in a logger_enable method. If this method is added at $Startup, it will log state changes to an text log file $Data/object_log/<object>/YYYY/MM/DD.log

I thought it might make sense to have this on an ‘opt-in’ basis, since this might create more processing overhead and more writes which is an issue for things like raspberry pi’s. It might make sense to ‘modularize’ this so that generic_item calls a library function — so that others could add in a mysql backend, or sqllite, or something else. Simple text files are good enough for my purpose.

Now, it’s integrated into IA7, so for those objects that have this method enabled, a graphic will show up on the modal:




Clicking on the green graph button will display points when objects are in various states:



I figured this would be a good improvement for an upcoming v4.2 release. Anyone have any comments, suggestions, or other ideas?

On Aug 28, 2016, at 11:32 AM, Steve Switzer <[hidden email]> wrote:

What came of this thread? Do we have example code for graphing things like motion events in each room, door open/close, MH mode, alarm system status, etc? It would even be nice to see when lights are turned on over time, too.

Thank you!

Steve

On 01/05/2014 03:14 PM, Lieven Hollevoet wrote:
Hey Dan,

Op 27-dec.-2013, om 18:23 heeft Dan Bemowski <[hidden email] <[hidden email]>> het volgende geschreven:

So, just to I have this straight I am going to use an abbreviated example taken from the weather_rrd_update.pl code that comes with MH.  In this, each DS defines a sensor value to show on the graph (correct?)

Each DS does define a sensor value.

, then each RRA defines the graph resolutions that are possible for each MIN, AVERAGE and MAX aggregate storing data for a defined period (1 day, 2 days) at each resolution (2 min, 5 min)?  The primary resolution for the first RRA is at 6 hours using an AVERAGE aggregate of values every minute, correct?  I hope I said that right?

The primary resolution depends on $RRD_STEP in the code. For the MisterHouse weather code this value is set to 60 seconds if I’m correct. Basically if you say

RRA:AVERAGE:0.5:1:801

then you tell the tool to store a value in the database every $RRD_STEP seconds, and do this for the last 801 $RRD_STEP seconds.

Why do you need ‘AVERAGE’ then? Well, you can push values to the RRD at a higher speed than defined by $RRD_STEP and then RRD will store only the average of those values every $RRD_STEP seconds.

Now, back to the code. I think the comment that states ‘for 6 hours’ is not correct. Since the $RRD_STEP is 60 seconds in weather_rrd_update, 801 times step_size is 13 hours instead of 6 hours. This might be the source of the confusion you are experiencing. It is probably something nobody has noticed up to now because 13 hours > 6 hours and the graphs show to be correct. The only disadvantage is that the database is bigger than it is absolutely required.


One last question I had in my post on this.  I am trying to understand the RRA's 0.5 value which is said to be "xff" by mr Oetiker in the beginners guide page that I listed.  What does the 0.5 represent or mean?  When I create my RRDs for graphing I can certainly use the 0.5 value that I see everyone else primarily using, but I like to know what it means.

See http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html <http://oss.oetiker.ch/rrdtool/doc/rrdcreate.en.html>

xff The xfiles factor defines what part of a consolidation interval may be made up from*UNKNOWN* data while the consolidated value is still regarded as known. It is given as the ratio of allowed *UNKNOWN* PDPs to the number of PDPs in the interval. Thus, it ranges from 0 to 1 (exclusive).

If you fail to feed data into the database for a certain amount of time, these points in time will be marked ‘unknown’. If you start to consolidate the rate (e.g. average data points) then the off factor determines from what point on the result of the consolidation function will also be unknown. If you set it to .5 then half of the values are allowed to be unknown before the consolidated results is also unknown.

Kind regards,
Lieven.


On Fri, 2013-12-27 at 14:48 +0100, Lieven Hollevoet wrote:

Hey Dan,

Op 25-dec.-2013, om 08:40 heeft dbemowsk <[hidden email] <[hidden email]>> het volgende geschreven:

My next question in all of this has to do with data history.  I understand
that the concept of RRD revolves around the whole round robin concept where
the data works on a fixed length timeline where data overwrites data from
the beginning once the saved points reach the end of the data set.  My
question is, is there a way to archive data?  What I mean is, if you have an
RRD that collects data for a 24 hour period, and when the next day starts
effectively overwriting the first days data, is there a way to look back at
a graph of data from a few days ago?  I see that the weather_rrd_update.pl
file creates a CSV file.  Is this what the CSV is used for?  If so, I am
guessing that it just keeps appending data to the end of the CSV, correct?

Nope, the idea of RRD is that you can use it stand alone. You define data sources (DS) that keep track of values. Then for every data source you create RRAs (round robin archives) with variable ‘timespan' per datapoint. E.g. suppose that you store a new temperature every 5 minutes, then your primary ‘resolution’ for the first RRA van be 5 minutes.
Then, for longer archiving you can define RRAs that keep track of your data over longer timespans. You then need to define the consolidation function for that datasource. Suppose you define secondary RRAs with a resolution of 1 hour per datapoint. The RRD framework can then take the values from the first high-resolution RRA, apply a consolidation function on those datapoints, and then calculate the value for the lower resolution dataset. The consolidation function can be e.g. average, minimum or maximum. This way you can keep track of values over longer time (e.g. a year) in 1 hour intervals. If you want to go even further you can then store values on a daily basis for 10’s of years.


My last question has to do with Data Sources (DS).  In the example, they
define one DS.  In looking at the weather_rrd_update.pl file, they define a
number of them.  Does each DS represent a different line (color) on the
graph?

Nope, a DS is a dataset. You can see it as a representation of a sensor that you read out and that you store the values in the database. To create the graphs you use the values from the RRAs or you calculate values based on the data in those RRAs.

I hope this clarifies it a bit further.

Kind regards,
Lieven.






------------------------------------------------------------------------------
Rapidly troubleshoot problems before they affect your business. Most IT
organizations don't have a clear picture of how application performance
affects their revenue. With AppDynamics, you get 100% visibility into your
Java,.NET, & PHP application. Start your 15-day FREE TRIAL of AppDynamics Pro!
http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk <http://pubads.g.doubleclick.net/gampad/clk?id=84349831&iu=/4140/ostg.clktrk>

________________________________________________________
To unsubscribe from this list, go to: http://sourceforge.net/mail/?group_id=1365 <http://sourceforge.net/mail/?group_id=1365>


------------------------------------------------------------------------------
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users



------------------------------------------------------------------------------

________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users



------------------------------------------------------------------------------

________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

12
Loading...