Quantcast

Uncompressed (raw) JSON?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Uncompressed (raw) JSON?

rickerdo
I'm not sure when it occurred, but it seems all JSON requests are now gzip compressed. This is creating a problem with third party interfaces like openremote which expect uncompressed JSON.

Do we have a way of requesting uncompressed JSON?

Example:
curl --compress 'http://banshee:8080/json/objects?items=Garage&fields=state'
{
   "data" : {
      "Garage" : {
         "state" : "off"
      }
   },
   "meta" : {
      "args" : {
         "fields" : [
            "state"
         ],
         "items" : [
            "Garage"
         ]
      },
      "client_ip" : "192.168.1.55",
      "path" : [
         "objects"
      ],
      "time" : 1477672883620.47
   }
}

curl 'http://banshee:8080/json/objects?items=Garage&fields=state'
D�Xm�
�0E�~E���#��@?��2�hS|����{I���aNr�Y#����V�9p�["GB�Ʀ����ic���Ńs�\-��]��㤧�d�8H�����4H���4T����䫋2�B\�<��τ���a|uE��>��ޟAdJI�&EV����_"���U

Thanks!
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uncompressed (raw) JSON?

Wayne Gatlin
The way it should work is if the client request header contains a blank Accept-Encoding: header, the server should not respond with a compressed response. Generally the server will not respond with compressed data when the client does not include a Accept-Encoding at all.


I think I found where this is happening in the code, I don't know how to fix it yet but the code needs to check the request header before compressing the response.

I think this is where the compression is happening:

/lib/json_server.pl


sub json_page {
    my ($json_raw) = @_;
    my $json;
    gzip \$json_raw => \$json;
    my $output = "HTTP/1.0 200 OK\r\n";
    $output .= "Server: MisterHouse\r\n";
    $output .= "Content-type: application/json\r\n";
    $output .= "Content-Encoding: gzip\r\n";
    $output .= "\r\n";
    $output .= $json;

    return $output;
}



I did some tests with curl and MH always compresses the json requests with gzip no matter what I put in the request header.



curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: none"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: none
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒▒▒▒▒▒▒R▒▒▒٭s▒u▒N:▒▒,▒▒H5:;▒7_1]▒▒e▒2▒▒▒▒▒▒▒
            -▒▒▒ך▒▒5▒▒▒▒T▒▒J▒R剌b▒n▒▒▒▒▒C%▒▒}+l▒▒▒▒▒,▒u▒/Q▒▒N▒,▒9▒4▒▒P^



 curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: deflate"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: deflate
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒kX▒g▒Ρ▒Y:▒phƳ▒6▒V▒>v▒▒▒▒gxH&▒/▒>▒▒^k▒▒З_▒▒?Py$U▒I▒_▒▒a▒▒#▒O~▒▒<▒▒▒▒~*N▒$▒▒▒/Z▒,▒▒*▒=▒^PuTTYPuTTY


_Wayne

On Fri, Oct 28, 2016 at 10:23 AM, rickerdo <[hidden email]> wrote:
I'm not sure when it occurred, but it seems all JSON requests are now gzip
compressed. This is creating a problem with third party interfaces like
openremote which expect uncompressed JSON.

Do we have a way of requesting uncompressed JSON?

Example:
curl --compress 'http://banshee:8080/json/objects?items=Garage&fields=state'
{
   "data" : {
      "Garage" : {
         "state" : "off"
      }
   },
   "meta" : {
      "args" : {
         "fields" : [
            "state"
         ],
         "items" : [
            "Garage"
         ]
      },
      "client_ip" : "192.168.1.55",
      "path" : [
         "objects"
      ],
      "time" : 1477672883620.47
   }
}

curl 'http://banshee:8080/json/objects?items=Garage&fields=state'
 D� X m�
�0 E�~E�� �#��@?��2�hS|����{I� ��aNr�Y#� ���V�9p� [
"G B� Ʀ����ic���Ńs�\-��]��㤧�d�8H �����4H���4T�� ��䫋2�B \�<��τ���a| uE��>��ޟAdJI�&EV����_"���U

Thanks!




--
View this message in context: http://misterhouse.10964.n7.nabble.com/Uncompressed-raw-JSON-tp21488.html
Sent from the Misterhouse - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users



------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uncompressed (raw) JSON?

H Plato
That’s the most elegant solution.

However, compressing the json was part of the original IA7 prototype that Kevin came up with, so I’d be a little reluctant to make this type of change.

That said, uncompressed can be useful, so maybe an option is to pass a uncompressed=1 type of argument. That would be straightforward to add and wouldn’t cause of bunch of regression testing in other apps/systems if they handle headers differently, or just assume compressed data.


On Oct 28, 2016, at 4:26 PM, Wayne Gatlin <[hidden email]> wrote:

The way it should work is if the client request header contains a blank Accept-Encoding: header, the server should not respond with a compressed response. Generally the server will not respond with compressed data when the client does not include a Accept-Encoding at all.


I think I found where this is happening in the code, I don't know how to fix it yet but the code needs to check the request header before compressing the response.

I think this is where the compression is happening:

/lib/json_server.pl


sub json_page {
    my ($json_raw) = @_;
    my $json;
    gzip \$json_raw => \$json;
    my $output = "HTTP/1.0 200 OK\r\n";
    $output .= "Server: MisterHouse\r\n";
    $output .= "Content-type: application/json\r\n";
    $output .= "Content-Encoding: gzip\r\n";
    $output .= "\r\n";
    $output .= $json;

    return $output;
}



I did some tests with curl and MH always compresses the json requests with gzip no matter what I put in the request header.



curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: none"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: none
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒▒▒▒▒▒▒R▒▒▒٭s▒u▒N:▒▒,▒▒H5:;▒7_1]▒▒e▒2▒▒▒▒▒▒▒
            -▒▒▒ך▒▒5▒▒▒▒T▒▒J▒R剌b▒n▒▒▒▒▒C%▒▒}+l▒▒▒▒▒,▒u▒/Q▒▒N▒,▒9▒4▒▒P^



 curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: deflate"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: deflate
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒kX▒g▒Ρ▒Y:▒phƳ▒6▒V▒>v▒▒▒▒gxH&▒/▒>▒▒^k▒▒З_▒▒?Py$U▒I▒_▒▒a▒▒#▒O~▒▒<▒▒▒▒~*N▒$▒▒▒/Z▒,▒▒*▒=▒^PuTTYPuTTY


_Wayne

On Fri, Oct 28, 2016 at 10:23 AM, rickerdo <[hidden email]> wrote:
I'm not sure when it occurred, but it seems all JSON requests are now gzip
compressed. This is creating a problem with third party interfaces like
openremote which expect uncompressed JSON.

Do we have a way of requesting uncompressed JSON?

Example:
curl --compress 'http://banshee:8080/json/objects?items=Garage&fields=state'
{
   "data" : {
      "Garage" : {
         "state" : "off"
      }
   },
   "meta" : {
      "args" : {
         "fields" : [
            "state"
         ],
         "items" : [
            "Garage"
         ]
      },
      "client_ip" : "192.168.1.55",
      "path" : [
         "objects"
      ],
      "time" : 1477672883620.47
   }
}

curl 'http://banshee:8080/json/objects?items=Garage&fields=state'
 D� X m�
�0 E�~E�� �#��@?��2�hS|����{I� ��aNr�Y#� ���V�9p� [
"G B� Ʀ����ic���Ńs�\-��]��㤧�d�8H �����4H���4T�� ��䫋2�B \�<��τ���a| uE��>��ޟAdJI�&EV����_"���U

Thanks!




--
View this message in context: http://misterhouse.10964.n7.nabble.com/Uncompressed-raw-JSON-tp21488.html
Sent from the Misterhouse - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users


------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users



------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uncompressed (raw) JSON?

rickerdo

Just my $.02 from having worked with several APIs over the years...

I've not seen an API that forces compression by default. IMHO Wayne's approach would be much more elegant and would allow for integration with other systems. At one point compression was not the default on MH.

I can appreciate the concerns about regression testing, but it really is a more universal approach. I would be happy to run some tests if someone can write the code to check for compression being requested as Wayne suggested.


On Oct 28, 2016 7:48 PM, "H Plato" <[hidden email]> wrote:
That’s the most elegant solution.

However, compressing the json was part of the original IA7 prototype that Kevin came up with, so I’d be a little reluctant to make this type of change.

That said, uncompressed can be useful, so maybe an option is to pass a uncompressed=1 type of argument. That would be straightforward to add and wouldn’t cause of bunch of regression testing in other apps/systems if they handle headers differently, or just assume compressed data.


On Oct 28, 2016, at 4:26 PM, Wayne Gatlin <[hidden email]> wrote:

The way it should work is if the client request header contains a blank Accept-Encoding: header, the server should not respond with a compressed response. Generally the server will not respond with compressed data when the client does not include a Accept-Encoding at all.


I think I found where this is happening in the code, I don't know how to fix it yet but the code needs to check the request header before compressing the response.

I think this is where the compression is happening:

/lib/json_server.pl


sub json_page {
    my ($json_raw) = @_;
    my $json;
    gzip \$json_raw => \$json;
    my $output = "HTTP/1.0 200 OK\r\n";
    $output .= "Server: MisterHouse\r\n";
    $output .= "Content-type: application/json\r\n";
    $output .= "Content-Encoding: gzip\r\n";
    $output .= "\r\n";
    $output .= $json;

    return $output;
}



I did some tests with curl and MH always compresses the json requests with gzip no matter what I put in the request header.



curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: none"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: none
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒▒▒▒▒▒▒R▒▒▒٭s▒u▒N:▒▒,▒▒H5:;▒7_1]▒▒e▒2▒▒▒▒▒▒▒
            -▒▒▒ך▒▒5▒▒▒▒T▒▒J▒R剌b▒n▒▒▒▒▒C%▒▒}+l▒▒▒▒▒,▒u▒/Q▒▒N▒,▒9▒4▒▒P^



 curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: deflate"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: deflate
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒kX▒g▒Ρ▒Y:▒phƳ▒6▒V▒>v▒▒▒▒gxH&▒/▒>▒▒^k▒▒З_▒▒?Py$U▒I▒_▒▒a▒▒#▒O~▒▒<▒▒▒▒~*N▒$▒▒▒/Z▒,▒▒*▒=▒^PuTTYPuTTY


_Wayne

On Fri, Oct 28, 2016 at 10:23 AM, rickerdo <[hidden email]> wrote:
I'm not sure when it occurred, but it seems all JSON requests are now gzip
compressed. This is creating a problem with third party interfaces like
openremote which expect uncompressed JSON.

Do we have a way of requesting uncompressed JSON?

Example:
curl --compress 'http://banshee:8080/json/objects?items=Garage&fields=state'
{
   "data" : {
      "Garage" : {
         "state" : "off"
      }
   },
   "meta" : {
      "args" : {
         "fields" : [
            "state"
         ],
         "items" : [
            "Garage"
         ]
      },
      "client_ip" : "192.168.1.55",
      "path" : [
         "objects"
      ],
      "time" : 1477672883620.47
   }
}

curl 'http://banshee:8080/json/objects?items=Garage&fields=state'
 D� X m�
�0 E�~E�� �#��@?��2�hS|����{I� ��aNr�Y#� ���V�9p� [
"G B� Ʀ����ic���Ńs�\-��]��㤧�d�8H �����4H���4T�� ��䫋2�B \�<��τ���a| uE��>��ޟAdJI�&EV����_"���U

Thanks!




--
View this message in context: http://misterhouse.10964.n7.nabble.com/Uncompressed-raw-JSON-tp21488.html
Sent from the Misterhouse - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users


------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users



------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uncompressed (raw) JSON?

rickerdo
As it turns out, the change wasn't too difficult to hack together. Using the latest release of master branch, here's a patch to enable the detection of a gzip'd request vs a non-gzip'd request by looking for an "Accept-Encoding" request header containing "gzip". This seems to work fine in my limited testing. 

I would submit for further testing and approval, but I'm not sure how to do so.

775d774
<     return &json_page($json_raw);
776a776,782
>     # Compression requested?
>     if ($Http{'Accept-Encoding'} =~ m/gzip/) {
>       return &json_page($json_raw, 'compress');
>     }
>     else {
>       return &json_page($json_raw);
>     }
1117c1123
<     my ($json_raw) = @_;
---
>     my ($json_raw, $compress_json) = @_;
1121c1127
<     gzip \$json_raw => \$json;
---
>     #gzip \$json_raw => \$json;
1125c1131,1141
<     $output .= "Content-Encoding: gzip\r\n";
---
>     #$output .= "Content-Encoding: gzip\r\n";
>     #$output .= "\r\n";
>     if ($compress_json) {
>       gzip \$json_raw => \$json;
>       $output .= "Content-Encoding: gzip\r\n";
>     }
>     else {
>       $json = $json_raw;
>     }
1128,1129d1143
< ##    $output .= $json_raw;


On Fri, Oct 28, 2016 at 8:22 PM, Rick Reed <[hidden email]> wrote:

Just my $.02 from having worked with several APIs over the years...

I've not seen an API that forces compression by default. IMHO Wayne's approach would be much more elegant and would allow for integration with other systems. At one point compression was not the default on MH.

I can appreciate the concerns about regression testing, but it really is a more universal approach. I would be happy to run some tests if someone can write the code to check for compression being requested as Wayne suggested.


On Oct 28, 2016 7:48 PM, "H Plato" <[hidden email]> wrote:
That’s the most elegant solution.

However, compressing the json was part of the original IA7 prototype that Kevin came up with, so I’d be a little reluctant to make this type of change.

That said, uncompressed can be useful, so maybe an option is to pass a uncompressed=1 type of argument. That would be straightforward to add and wouldn’t cause of bunch of regression testing in other apps/systems if they handle headers differently, or just assume compressed data.


On Oct 28, 2016, at 4:26 PM, Wayne Gatlin <[hidden email]> wrote:

The way it should work is if the client request header contains a blank Accept-Encoding: header, the server should not respond with a compressed response. Generally the server will not respond with compressed data when the client does not include a Accept-Encoding at all.


I think I found where this is happening in the code, I don't know how to fix it yet but the code needs to check the request header before compressing the response.

I think this is where the compression is happening:

/lib/json_server.pl


sub json_page {
    my ($json_raw) = @_;
    my $json;
    gzip \$json_raw => \$json;
    my $output = "HTTP/1.0 200 OK\r\n";
    $output .= "Server: MisterHouse\r\n";
    $output .= "Content-type: application/json\r\n";
    $output .= "Content-Encoding: gzip\r\n";
    $output .= "\r\n";
    $output .= $json;

    return $output;
}



I did some tests with curl and MH always compresses the json requests with gzip no matter what I put in the request header.



curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: none"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: none
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒▒▒▒▒▒▒R▒▒▒٭s▒u▒N:▒▒,▒▒H5:;▒7_1]▒▒e▒2▒▒▒▒▒▒▒
            -▒▒▒ך▒▒5▒▒▒▒T▒▒J▒R剌b▒n▒▒▒▒▒C%▒▒}+l▒▒▒▒▒,▒u▒/Q▒▒N▒,▒9▒4▒▒P^



 curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: deflate"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: deflate
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒kX▒g▒Ρ▒Y:▒phƳ▒6▒V▒>v▒▒▒▒gxH&▒/▒>▒▒^k▒▒З_▒▒?Py$U▒I▒_▒▒a▒▒#▒O~▒▒<▒▒▒▒~*N▒$▒▒▒/Z▒,▒▒*▒=▒^PuTTYPuTTY


_Wayne

On Fri, Oct 28, 2016 at 10:23 AM, rickerdo <[hidden email]> wrote:
I'm not sure when it occurred, but it seems all JSON requests are now gzip
compressed. This is creating a problem with third party interfaces like
openremote which expect uncompressed JSON.

Do we have a way of requesting uncompressed JSON?

Example:
curl --compress 'http://banshee:8080/json/objects?items=Garage&fields=state'
{
   "data" : {
      "Garage" : {
         "state" : "off"
      }
   },
   "meta" : {
      "args" : {
         "fields" : [
            "state"
         ],
         "items" : [
            "Garage"
         ]
      },
      "client_ip" : "192.168.1.55",
      "path" : [
         "objects"
      ],
      "time" : 1477672883620.47
   }
}

curl 'http://banshee:8080/json/objects?items=Garage&fields=state'
 D� X m�
�0 E�~E�� �#��@?��2�hS|����{I� ��aNr�Y#� ���V�9p� [
"G B� Ʀ����ic���Ńs�\-��]��㤧�d�8H �����4H���4T�� ��䫋2�B \�<��τ���a| uE��>��ޟAdJI�&EV����_"���U

Thanks!




--
View this message in context: http://misterhouse.10964.n7.nabble.com/Uncompressed-raw-JSON-tp21488.html
Sent from the Misterhouse - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users


------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users




------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uncompressed (raw) JSON?

H Plato
Great! I think this is a good idea.

The compression stuff was added in the original IA7 prototype by Kevin a year or so back. IA7 will work _without_ compression, since I was playing around with that while trying to figure out the high CPU utilization issue. (which I think is due to json_server processing every loop, but I don’t know how to fix)

While I haven’t noticed any significant performance differences between compressed and uncompressed in my very limited testing (I was more looking at cpu utilization), I think to incorporate this patch we’d want to ensure the current IA7 method remains unchanged.

I think to do this, all the $.ajax calls in javascript.js would have to be modified to include a beforeSend function to modify the header. See here: http://api.jquery.com/jquery.ajax/#jQuery-ajax-settings . Probably straightforward if you want to give it a try.

On Nov 4, 2016, at 3:48 PM, Rick Reed <[hidden email]> wrote:

As it turns out, the change wasn't too difficult to hack together. Using the latest release of master branch, here's a patch to enable the detection of a gzip'd request vs a non-gzip'd request by looking for an "Accept-Encoding" request header containing "gzip". This seems to work fine in my limited testing. 

I would submit for further testing and approval, but I'm not sure how to do so.

775d774
<     return &json_page($json_raw);
776a776,782
>     # Compression requested?
>     if ($Http{'Accept-Encoding'} =~ m/gzip/) {
>       return &json_page($json_raw, 'compress');
>     }
>     else {
>       return &json_page($json_raw);
>     }
1117c1123
<     my ($json_raw) = @_;
---
>     my ($json_raw, $compress_json) = @_;
1121c1127
<     gzip \$json_raw => \$json;
---
>     #gzip \$json_raw => \$json;
1125c1131,1141
<     $output .= "Content-Encoding: gzip\r\n";
---
>     #$output .= "Content-Encoding: gzip\r\n";
>     #$output .= "\r\n";
>     if ($compress_json) {
>       gzip \$json_raw => \$json;
>       $output .= "Content-Encoding: gzip\r\n";
>     }
>     else {
>       $json = $json_raw;
>     }
1128,1129d1143
< ##    $output .= $json_raw;


On Fri, Oct 28, 2016 at 8:22 PM, Rick Reed <[hidden email]> wrote:

Just my $.02 from having worked with several APIs over the years...

I've not seen an API that forces compression by default. IMHO Wayne's approach would be much more elegant and would allow for integration with other systems. At one point compression was not the default on MH.

I can appreciate the concerns about regression testing, but it really is a more universal approach. I would be happy to run some tests if someone can write the code to check for compression being requested as Wayne suggested.


On Oct 28, 2016 7:48 PM, "H Plato" <[hidden email]> wrote:
That’s the most elegant solution.

However, compressing the json was part of the original IA7 prototype that Kevin came up with, so I’d be a little reluctant to make this type of change.

That said, uncompressed can be useful, so maybe an option is to pass a uncompressed=1 type of argument. That would be straightforward to add and wouldn’t cause of bunch of regression testing in other apps/systems if they handle headers differently, or just assume compressed data.


On Oct 28, 2016, at 4:26 PM, Wayne Gatlin <[hidden email]> wrote:

The way it should work is if the client request header contains a blank Accept-Encoding: header, the server should not respond with a compressed response. Generally the server will not respond with compressed data when the client does not include a Accept-Encoding at all.


I think I found where this is happening in the code, I don't know how to fix it yet but the code needs to check the request header before compressing the response.

I think this is where the compression is happening:

/lib/json_server.pl


sub json_page {
    my ($json_raw) = @_;
    my $json;
    gzip \$json_raw => \$json;
    my $output = "HTTP/1.0 200 OK\r\n";
    $output .= "Server: MisterHouse\r\n";
    $output .= "Content-type: application/json\r\n";
    $output .= "Content-Encoding: gzip\r\n";
    $output .= "\r\n";
    $output .= $json;

    return $output;
}



I did some tests with curl and MH always compresses the json requests with gzip no matter what I put in the request header.



curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: none"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: none
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒▒▒▒▒▒▒R▒▒▒٭s▒u▒N:▒▒,▒▒H5:;▒7_1]▒▒e▒2▒▒▒▒▒▒▒
            -▒▒▒ך▒▒5▒▒▒▒T▒▒J▒R剌b▒n▒▒▒▒▒C%▒▒}+l▒▒▒▒▒,▒u▒/Q▒▒N▒,▒9▒4▒▒P^



 curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: deflate"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: deflate
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒kX▒g▒Ρ▒Y:▒phƳ▒6▒V▒>v▒▒▒▒gxH&▒/▒>▒▒^k▒▒З_▒▒?Py$U▒I▒_▒▒a▒▒#▒O~▒▒<▒▒▒▒~*N▒$▒▒▒/Z▒,▒▒*▒=▒^PuTTYPuTTY


_Wayne

On Fri, Oct 28, 2016 at 10:23 AM, rickerdo <[hidden email]> wrote:
I'm not sure when it occurred, but it seems all JSON requests are now gzip
compressed. This is creating a problem with third party interfaces like
openremote which expect uncompressed JSON.

Do we have a way of requesting uncompressed JSON?

Example:
curl --compress 'http://banshee:8080/json/objects?items=Garage&fields=state'
{
   "data" : {
      "Garage" : {
         "state" : "off"
      }
   },
   "meta" : {
      "args" : {
         "fields" : [
            "state"
         ],
         "items" : [
            "Garage"
         ]
      },
      "client_ip" : "192.168.1.55",
      "path" : [
         "objects"
      ],
      "time" : 1477672883620.47
   }
}

curl 'http://banshee:8080/json/objects?items=Garage&fields=state'
 D� X m�
�0 E�~E�� �#��@?��2�hS|����{I� ��aNr�Y#� ���V�9p� [
"G B� Ʀ����ic���Ńs�\-��]��㤧�d�8H �����4H���4T�� ��䫋2�B \�<��τ���a| uE��>��ޟAdJI�&EV����_"���U

Thanks!




--
View this message in context: http://misterhouse.10964.n7.nabble.com/Uncompressed-raw-JSON-tp21488.html
Sent from the Misterhouse - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users


------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uncompressed (raw) JSON?

rickerdo
Unless I'm missing something (wouldn't be the first time), the patch I submitted should not require any changes to IA7 at all. The patch only checks if the HTTP request contains a header named "Accept-Encoding" containing a value of "gzip". If yes, return compressed JSON, otherwise return uncompressed JSON.

Why would we want to make any ajax changes? Are you suggesting disabling JSON compression completely?



On Sat, Nov 5, 2016 at 9:46 AM, H Plato <[hidden email]> wrote:
Great! I think this is a good idea.

The compression stuff was added in the original IA7 prototype by Kevin a year or so back. IA7 will work _without_ compression, since I was playing around with that while trying to figure out the high CPU utilization issue. (which I think is due to json_server processing every loop, but I don’t know how to fix)

While I haven’t noticed any significant performance differences between compressed and uncompressed in my very limited testing (I was more looking at cpu utilization), I think to incorporate this patch we’d want to ensure the current IA7 method remains unchanged.

I think to do this, all the $.ajax calls in javascript.js would have to be modified to include a beforeSend function to modify the header. See here: http://api.jquery.com/jquery.ajax/#jQuery-ajax-settings . Probably straightforward if you want to give it a try.

On Nov 4, 2016, at 3:48 PM, Rick Reed <[hidden email]> wrote:

As it turns out, the change wasn't too difficult to hack together. Using the latest release of master branch, here's a patch to enable the detection of a gzip'd request vs a non-gzip'd request by looking for an "Accept-Encoding" request header containing "gzip". This seems to work fine in my limited testing. 

I would submit for further testing and approval, but I'm not sure how to do so.

775d774
<     return &json_page($json_raw);
776a776,782
>     # Compression requested?
>     if ($Http{'Accept-Encoding'} =~ m/gzip/) {
>       return &json_page($json_raw, 'compress');
>     }
>     else {
>       return &json_page($json_raw);
>     }
1117c1123
<     my ($json_raw) = @_;
---
>     my ($json_raw, $compress_json) = @_;
1121c1127
<     gzip \$json_raw => \$json;
---
>     #gzip \$json_raw => \$json;
1125c1131,1141
<     $output .= "Content-Encoding: gzip\r\n";
---
>     #$output .= "Content-Encoding: gzip\r\n";
>     #$output .= "\r\n";
>     if ($compress_json) {
>       gzip \$json_raw => \$json;
>       $output .= "Content-Encoding: gzip\r\n";
>     }
>     else {
>       $json = $json_raw;
>     }
1128,1129d1143
< ##    $output .= $json_raw;


On Fri, Oct 28, 2016 at 8:22 PM, Rick Reed <[hidden email]> wrote:

Just my $.02 from having worked with several APIs over the years...

I've not seen an API that forces compression by default. IMHO Wayne's approach would be much more elegant and would allow for integration with other systems. At one point compression was not the default on MH.

I can appreciate the concerns about regression testing, but it really is a more universal approach. I would be happy to run some tests if someone can write the code to check for compression being requested as Wayne suggested.


On Oct 28, 2016 7:48 PM, "H Plato" <[hidden email]> wrote:
That’s the most elegant solution.

However, compressing the json was part of the original IA7 prototype that Kevin came up with, so I’d be a little reluctant to make this type of change.

That said, uncompressed can be useful, so maybe an option is to pass a uncompressed=1 type of argument. That would be straightforward to add and wouldn’t cause of bunch of regression testing in other apps/systems if they handle headers differently, or just assume compressed data.


On Oct 28, 2016, at 4:26 PM, Wayne Gatlin <[hidden email]> wrote:

The way it should work is if the client request header contains a blank Accept-Encoding: header, the server should not respond with a compressed response. Generally the server will not respond with compressed data when the client does not include a Accept-Encoding at all.


I think I found where this is happening in the code, I don't know how to fix it yet but the code needs to check the request header before compressing the response.

I think this is where the compression is happening:

/lib/json_server.pl


sub json_page {
    my ($json_raw) = @_;
    my $json;
    gzip \$json_raw => \$json;
    my $output = "HTTP/1.0 200 OK\r\n";
    $output .= "Server: MisterHouse\r\n";
    $output .= "Content-type: application/json\r\n";
    $output .= "Content-Encoding: gzip\r\n";
    $output .= "\r\n";
    $output .= $json;

    return $output;
}



I did some tests with curl and MH always compresses the json requests with gzip no matter what I put in the request header.



curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: none"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: none
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒▒▒▒▒▒▒R▒▒▒٭s▒u▒N:▒▒,▒▒H5:;▒7_1]▒▒e▒2▒▒▒▒▒▒▒
            -▒▒▒ך▒▒5▒▒▒▒T▒▒J▒R剌b▒n▒▒▒▒▒C%▒▒}+l▒▒▒▒▒,▒u▒/Q▒▒N▒,▒9▒4▒▒P^



 curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: deflate"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: deflate
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒kX▒g▒Ρ▒Y:▒phƳ▒6▒V▒>v▒▒▒▒gxH&▒/▒>▒▒^k▒▒З_▒▒?Py$U▒I▒_▒▒a▒▒#▒O~▒▒<▒▒▒▒~*N▒$▒▒▒/Z▒,▒▒*▒=▒^PuTTYPuTTY


_Wayne

On Fri, Oct 28, 2016 at 10:23 AM, rickerdo <[hidden email]> wrote:
I'm not sure when it occurred, but it seems all JSON requests are now gzip
compressed. This is creating a problem with third party interfaces like
openremote which expect uncompressed JSON.

Do we have a way of requesting uncompressed JSON?

Example:
curl --compress 'http://banshee:8080/json/objects?items=Garage&fields=state'
{
   "data" : {
      "Garage" : {
         "state" : "off"
      }
   },
   "meta" : {
      "args" : {
         "fields" : [
            "state"
         ],
         "items" : [
            "Garage"
         ]
      },
      "client_ip" : "192.168.1.55",
      "path" : [
         "objects"
      ],
      "time" : 1477672883620.47
   }
}

curl 'http://banshee:8080/json/objects?items=Garage&fields=state'
 D� X m�
�0 E�~E�� �#��@?��2�hS|����{I� ��aNr�Y#� ���V�9p� [
"G B� Ʀ����ic���Ńs�\-��]��㤧�d�8H �����4H���4T�� ��䫋2�B \�<��τ���a| uE��>��ޟAdJI�&EV����_"���U

Thanks!




--
View this message in context: http://misterhouse.10964.n7.nabble.com/Uncompressed-raw-JSON-tp21488.html
Sent from the Misterhouse - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users


------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users




------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uncompressed (raw) JSON?

H Plato
No.

Currently compression is on by default, and IA7 doesn’t add a header to request compressed data. So that’s how it’s been working since prototype.

Adding this change to http_server.pl will switch compression off by default, meaning the way the ajax calls are structured, IA7 would now receive uncompressed data. Which may or may not cause issues.

I’m saying if we make this change, it would make sense to me to adjust IA7 to request compressed data again so that the change doesn’t effect the current design — especially with all the changes that have been made in IA7 since the 4.1 release.

Or you could add in a config.ini parameter to set-http-gzip-default = on|off. So your use case works, but for others that don’t use that function, things can continue as they are.

My personal preference is to wrap up all the current changes into master, start to build out the 4.2 release, and then incorporate this change as part of 4.3. 

On Nov 5, 2016, at 10:13 AM, Rick Reed <[hidden email]> wrote:

Unless I'm missing something (wouldn't be the first time), the patch I submitted should not require any changes to IA7 at all. The patch only checks if the HTTP request contains a header named "Accept-Encoding" containing a value of "gzip". If yes, return compressed JSON, otherwise return uncompressed JSON.

Why would we want to make any ajax changes? Are you suggesting disabling JSON compression completely?



On Sat, Nov 5, 2016 at 9:46 AM, H Plato <[hidden email]> wrote:
Great! I think this is a good idea.

The compression stuff was added in the original IA7 prototype by Kevin a year or so back. IA7 will work _without_ compression, since I was playing around with that while trying to figure out the high CPU utilization issue. (which I think is due to json_server processing every loop, but I don’t know how to fix)

While I haven’t noticed any significant performance differences between compressed and uncompressed in my very limited testing (I was more looking at cpu utilization), I think to incorporate this patch we’d want to ensure the current IA7 method remains unchanged.

I think to do this, all the $.ajax calls in javascript.js would have to be modified to include a beforeSend function to modify the header. See here: http://api.jquery.com/jquery.ajax/#jQuery-ajax-settings . Probably straightforward if you want to give it a try.

On Nov 4, 2016, at 3:48 PM, Rick Reed <[hidden email]> wrote:

As it turns out, the change wasn't too difficult to hack together. Using the latest release of master branch, here's a patch to enable the detection of a gzip'd request vs a non-gzip'd request by looking for an "Accept-Encoding" request header containing "gzip". This seems to work fine in my limited testing. 

I would submit for further testing and approval, but I'm not sure how to do so.

775d774
<     return &json_page($json_raw);
776a776,782
>     # Compression requested?
>     if ($Http{'Accept-Encoding'} =~ m/gzip/) {
>       return &json_page($json_raw, 'compress');
>     }
>     else {
>       return &json_page($json_raw);
>     }
1117c1123
<     my ($json_raw) = @_;
---
>     my ($json_raw, $compress_json) = @_;
1121c1127
<     gzip \$json_raw => \$json;
---
>     #gzip \$json_raw => \$json;
1125c1131,1141
<     $output .= "Content-Encoding: gzip\r\n";
---
>     #$output .= "Content-Encoding: gzip\r\n";
>     #$output .= "\r\n";
>     if ($compress_json) {
>       gzip \$json_raw => \$json;
>       $output .= "Content-Encoding: gzip\r\n";
>     }
>     else {
>       $json = $json_raw;
>     }
1128,1129d1143
< ##    $output .= $json_raw;


On Fri, Oct 28, 2016 at 8:22 PM, Rick Reed <[hidden email]> wrote:

Just my $.02 from having worked with several APIs over the years...

I've not seen an API that forces compression by default. IMHO Wayne's approach would be much more elegant and would allow for integration with other systems. At one point compression was not the default on MH.

I can appreciate the concerns about regression testing, but it really is a more universal approach. I would be happy to run some tests if someone can write the code to check for compression being requested as Wayne suggested.


On Oct 28, 2016 7:48 PM, "H Plato" <[hidden email]> wrote:
That’s the most elegant solution.

However, compressing the json was part of the original IA7 prototype that Kevin came up with, so I’d be a little reluctant to make this type of change.

That said, uncompressed can be useful, so maybe an option is to pass a uncompressed=1 type of argument. That would be straightforward to add and wouldn’t cause of bunch of regression testing in other apps/systems if they handle headers differently, or just assume compressed data.


On Oct 28, 2016, at 4:26 PM, Wayne Gatlin <[hidden email]> wrote:

The way it should work is if the client request header contains a blank Accept-Encoding: header, the server should not respond with a compressed response. Generally the server will not respond with compressed data when the client does not include a Accept-Encoding at all.


I think I found where this is happening in the code, I don't know how to fix it yet but the code needs to check the request header before compressing the response.

I think this is where the compression is happening:

/lib/json_server.pl


sub json_page {
    my ($json_raw) = @_;
    my $json;
    gzip \$json_raw => \$json;
    my $output = "HTTP/1.0 200 OK\r\n";
    $output .= "Server: MisterHouse\r\n";
    $output .= "Content-type: application/json\r\n";
    $output .= "Content-Encoding: gzip\r\n";
    $output .= "\r\n";
    $output .= $json;

    return $output;
}



I did some tests with curl and MH always compresses the json requests with gzip no matter what I put in the request header.



curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: none"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: none
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒▒▒▒▒▒▒R▒▒▒٭s▒u▒N:▒▒,▒▒H5:;▒7_1]▒▒e▒2▒▒▒▒▒▒▒
            -▒▒▒ך▒▒5▒▒▒▒T▒▒J▒R剌b▒n▒▒▒▒▒C%▒▒}+l▒▒▒▒▒,▒u▒/Q▒▒N▒,▒9▒4▒▒P^



 curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: deflate"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: deflate
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒kX▒g▒Ρ▒Y:▒phƳ▒6▒V▒>v▒▒▒▒gxH&▒/▒>▒▒^k▒▒З_▒▒?Py$U▒I▒_▒▒a▒▒#▒O~▒▒<▒▒▒▒~*N▒$▒▒▒/Z▒,▒▒*▒=▒^PuTTYPuTTY


_Wayne

On Fri, Oct 28, 2016 at 10:23 AM, rickerdo <[hidden email]> wrote:
I'm not sure when it occurred, but it seems all JSON requests are now gzip
compressed. This is creating a problem with third party interfaces like
openremote which expect uncompressed JSON.

Do we have a way of requesting uncompressed JSON?

Example:
curl --compress 'http://banshee:8080/json/objects?items=Garage&fields=state'
{
   "data" : {
      "Garage" : {
         "state" : "off"
      }
   },
   "meta" : {
      "args" : {
         "fields" : [
            "state"
         ],
         "items" : [
            "Garage"
         ]
      },
      "client_ip" : "192.168.1.55",
      "path" : [
         "objects"
      ],
      "time" : 1477672883620.47
   }
}

curl 'http://banshee:8080/json/objects?items=Garage&fields=state'
 D� X m�
�0 E�~E�� �#��@?��2�hS|����{I� ��aNr�Y#� ���V�9p� [
"G B� Ʀ����ic���Ńs�\-��]��㤧�d�8H �����4H���4T�� ��䫋2�B \�<��τ���a| uE��>��ޟAdJI�&EV����_"���U

Thanks!




--
View this message in context: http://misterhouse.10964.n7.nabble.com/Uncompressed-raw-JSON-tp21488.html
Sent from the Misterhouse - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users


------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users





------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Uncompressed (raw) JSON?

rickerdo
OK. Now I get it. Thanks for the explanation!

Digging a little deeper I see that it's the browser, not javascript.js, sending the "Accept-Encoding" header. So this would imply that the browser is decompressing the compressed JSON -before- ajax kicks in, correct?

My concern about not reading HTTP headers to determine whether or not to compress stems from integration with third party apps and standards. In other words, I believe one should not have to hard-code compression/non-compression in mh.ini. The web server should make that determination based on the capabilities of the web client making the request.
Said web clients could be: 
 - A browser that natively supports gzip compression
 - curl (optionally support gzip compression)
 - Any third party GUI app like: Open Remote, CommandFusion's iViewer, Home Remote (http://www.thehomeremote.com/) - my latest interface of choice for now.

None of the third-party apps I listed support gzip compression as far as I can tell. Nor should they have to since no other API forces compression by default.

Long winded way of saying that I would prefer MH stick to HTTP standards as much as possible. My goal is to use MH for it's powerful customization, rules, and to act as a JSON gateway for everything MH knows about. Speaking of which, I may have found a bug in the json_put function of json_server.pl. I'll create a separate topi for that one.


On Sat, Nov 5, 2016 at 11:24 AM, H Plato <[hidden email]> wrote:
No.

Currently compression is on by default, and IA7 doesn’t add a header to request compressed data. So that’s how it’s been working since prototype.

Adding this change to http_server.pl will switch compression off by default, meaning the way the ajax calls are structured, IA7 would now receive uncompressed data. Which may or may not cause issues.

I’m saying if we make this change, it would make sense to me to adjust IA7 to request compressed data again so that the change doesn’t effect the current design — especially with all the changes that have been made in IA7 since the 4.1 release.

Or you could add in a config.ini parameter to set-http-gzip-default = on|off. So your use case works, but for others that don’t use that function, things can continue as they are.

My personal preference is to wrap up all the current changes into master, start to build out the 4.2 release, and then incorporate this change as part of 4.3. 

On Nov 5, 2016, at 10:13 AM, Rick Reed <[hidden email]> wrote:

Unless I'm missing something (wouldn't be the first time), the patch I submitted should not require any changes to IA7 at all. The patch only checks if the HTTP request contains a header named "Accept-Encoding" containing a value of "gzip". If yes, return compressed JSON, otherwise return uncompressed JSON.

Why would we want to make any ajax changes? Are you suggesting disabling JSON compression completely?



On Sat, Nov 5, 2016 at 9:46 AM, H Plato <[hidden email]> wrote:
Great! I think this is a good idea.

The compression stuff was added in the original IA7 prototype by Kevin a year or so back. IA7 will work _without_ compression, since I was playing around with that while trying to figure out the high CPU utilization issue. (which I think is due to json_server processing every loop, but I don’t know how to fix)

While I haven’t noticed any significant performance differences between compressed and uncompressed in my very limited testing (I was more looking at cpu utilization), I think to incorporate this patch we’d want to ensure the current IA7 method remains unchanged.

I think to do this, all the $.ajax calls in javascript.js would have to be modified to include a beforeSend function to modify the header. See here: http://api.jquery.com/jquery.ajax/#jQuery-ajax-settings . Probably straightforward if you want to give it a try.

On Nov 4, 2016, at 3:48 PM, Rick Reed <[hidden email]> wrote:

As it turns out, the change wasn't too difficult to hack together. Using the latest release of master branch, here's a patch to enable the detection of a gzip'd request vs a non-gzip'd request by looking for an "Accept-Encoding" request header containing "gzip". This seems to work fine in my limited testing. 

I would submit for further testing and approval, but I'm not sure how to do so.

775d774
<     return &json_page($json_raw);
776a776,782
>     # Compression requested?
>     if ($Http{'Accept-Encoding'} =~ m/gzip/) {
>       return &json_page($json_raw, 'compress');
>     }
>     else {
>       return &json_page($json_raw);
>     }
1117c1123
<     my ($json_raw) = @_;
---
>     my ($json_raw, $compress_json) = @_;
1121c1127
<     gzip \$json_raw => \$json;
---
>     #gzip \$json_raw => \$json;
1125c1131,1141
<     $output .= "Content-Encoding: gzip\r\n";
---
>     #$output .= "Content-Encoding: gzip\r\n";
>     #$output .= "\r\n";
>     if ($compress_json) {
>       gzip \$json_raw => \$json;
>       $output .= "Content-Encoding: gzip\r\n";
>     }
>     else {
>       $json = $json_raw;
>     }
1128,1129d1143
< ##    $output .= $json_raw;


On Fri, Oct 28, 2016 at 8:22 PM, Rick Reed <[hidden email]> wrote:

Just my $.02 from having worked with several APIs over the years...

I've not seen an API that forces compression by default. IMHO Wayne's approach would be much more elegant and would allow for integration with other systems. At one point compression was not the default on MH.

I can appreciate the concerns about regression testing, but it really is a more universal approach. I would be happy to run some tests if someone can write the code to check for compression being requested as Wayne suggested.


On Oct 28, 2016 7:48 PM, "H Plato" <[hidden email]> wrote:
That’s the most elegant solution.

However, compressing the json was part of the original IA7 prototype that Kevin came up with, so I’d be a little reluctant to make this type of change.

That said, uncompressed can be useful, so maybe an option is to pass a uncompressed=1 type of argument. That would be straightforward to add and wouldn’t cause of bunch of regression testing in other apps/systems if they handle headers differently, or just assume compressed data.


On Oct 28, 2016, at 4:26 PM, Wayne Gatlin <[hidden email]> wrote:

The way it should work is if the client request header contains a blank Accept-Encoding: header, the server should not respond with a compressed response. Generally the server will not respond with compressed data when the client does not include a Accept-Encoding at all.


I think I found where this is happening in the code, I don't know how to fix it yet but the code needs to check the request header before compressing the response.

I think this is where the compression is happening:

/lib/json_server.pl


sub json_page {
    my ($json_raw) = @_;
    my $json;
    gzip \$json_raw => \$json;
    my $output = "HTTP/1.0 200 OK\r\n";
    $output .= "Server: MisterHouse\r\n";
    $output .= "Content-type: application/json\r\n";
    $output .= "Content-Encoding: gzip\r\n";
    $output .= "\r\n";
    $output .= $json;

    return $output;
}



I did some tests with curl and MH always compresses the json requests with gzip no matter what I put in the request header.



curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: none"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: none
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒▒▒▒▒▒▒R▒▒▒٭s▒u▒N:▒▒,▒▒H5:;▒7_1]▒▒e▒2▒▒▒▒▒▒▒
            -▒▒▒ך▒▒5▒▒▒▒T▒▒J▒R剌b▒n▒▒▒▒▒C%▒▒}+l▒▒▒▒▒,▒u▒/Q▒▒N▒,▒9▒4▒▒P^



 curl -v 'http://192.168.1.39:8080/json/objects?items=Garage&fields=state' -H "Accept-Encoding: deflate"
* Hostname was NOT found in DNS cache
*   Trying 192.168.1.39...
* Connected to 192.168.1.39 (192.168.1.39) port 8080 (#0)
> GET /json/objects?items=Garage&fields=state HTTP/1.1
> User-Agent: curl/7.35.0
> Host: 192.168.1.39:8080
> Accept: */*
> Accept-Encoding: deflate
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: MisterHouse
< Content-type: application/json
< Content-Encoding: gzip
<
* Closing connection 0
▒▒kX▒g▒Ρ▒Y:▒phƳ▒6▒V▒>v▒▒▒▒gxH&▒/▒>▒▒^k▒▒З_▒▒?Py$U▒I▒_▒▒a▒▒#▒O~▒▒<▒▒▒▒~*N▒$▒▒▒/Z▒,▒▒*▒=▒^PuTTYPuTTY


_Wayne

On Fri, Oct 28, 2016 at 10:23 AM, rickerdo <[hidden email]> wrote:
I'm not sure when it occurred, but it seems all JSON requests are now gzip
compressed. This is creating a problem with third party interfaces like
openremote which expect uncompressed JSON.

Do we have a way of requesting uncompressed JSON?

Example:
curl --compress 'http://banshee:8080/json/objects?items=Garage&fields=state'
{
   "data" : {
      "Garage" : {
         "state" : "off"
      }
   },
   "meta" : {
      "args" : {
         "fields" : [
            "state"
         ],
         "items" : [
            "Garage"
         ]
      },
      "client_ip" : "192.168.1.55",
      "path" : [
         "objects"
      ],
      "time" : 1477672883620.47
   }
}

curl 'http://banshee:8080/json/objects?items=Garage&fields=state'
 D� X m�
�0 E�~E�� �#��@?��2�hS|����{I� ��aNr�Y#� ���V�9p� [
"G B� Ʀ����ic���Ńs�\-��]��㤧�d�8H �����4H���4T�� ��䫋2�B \�<��τ���a| uE��>��ޟAdJI�&EV����_"���U

Thanks!




--
View this message in context: http://misterhouse.10964.n7.nabble.com/Uncompressed-raw-JSON-tp21488.html
Sent from the Misterhouse - User mailing list archive at Nabble.com.

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users


------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users






------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
________________________________________________________
To unsubscribe from this list, go to: https://lists.sourceforge.net/lists/listinfo/misterhouse-users

Loading...