Mirroring/intercepting SunPower Monitoring Traffic?

[robillard]

So I've been running with this sniffer for quite a while now, and while my production numbers are spot on, my consumption numbers are off. They tend to be less (sometimes significantly) from the SunPower website's numbers. Some of this I can see, due to the SunPower monitoring system rounding, but not to the extent that I'm seeing...

astroboy, which number are you using for net? I'm using this first entry from the 140 messages for net, since that appears to be a lifetime net...

[robillard]

i ran all day today using the "difference in total lifetime energy method" and PVOutputs shows 6W more than the old sunpower monitoring site. based on observations, i think the monitoring website is truncating the 3 decimal digits to 2 instead of rounding them, so the two values are probably exactly the same. going to declare victory.

Yes, I'm seeing similar results. I think the sunpower stuff is probably rounding inaccurately (or, to be cynical, maybe optimistically, to inflate production numbers?)...

also i looked at the return traffic from the monitoring server, and what i'm seeing is a 4-digit message number coming back (1002) with a UTC timestamp. this timestamp does not always match what goes to the server, so it's probably not used as a protocol level "ack". the MD5sum (or whatever) in message 102 does not come back either. interestingly in the return packet (which is an http response) they set a cookie with a session ID, but it's set to expire 0 or 1 seconds after the date in the http header. i suppose that all that's happening at a protocol level here is http: an HTTP request (containing the inverter data) is made, which then expects an HTTP response. if the supervisor/inverter doesn't get the response, i guess it just makes the request again. so there does not appear to be any other "custom" protocol at work here.

Interesting, I have been ignoring traffic coming back to the supervisor, and have been blissfully unaware that there was any...

also, i saw the supervisor repeat some 130 messages today. as long as there are not multiple outstanding messages this works out OK - the duplicate message causes my script to write PVOutput with the same data again, since the difference in the inverter total lifetime energy is 0. if multiple 130s are repeated out-of-order then my calculations will get screwed up.

Yeah, I saw this recently as well: a whole burst/flurry of past traffic being re-posted in one big lump to the monitoring server. I don't know what's causing this, though, since it would appear that the server is ACKing all the data. Perhaps something higher-level in the application protocol between the two is triggering this? Since my state-machine is also paying attention to the timestamp, this is not a problem for me, as I will simply ignore any messages that are for timestamps less then my expected next timestamp. I will have to deal properly with "holes" in the stream at some point...

i guess what happens when the supervisor has been out of contact with the monitoring website for a long time is still an open question, and one day i'll have to test that to make sure the script handles that corner case. will be interesting to see if multiple 130/131s go out with one 102 protecting them in a single HTTP request.

I think there are two things to deal with:
- when the server is down, but the supervisor still wants to send data
- when the power is out locally, and my supervisor is not sending data to the server

I'll have to observe the system over time, and figure out how to react to these...

[astroboy]

should have expected this but longer status messages are fragmented by TCP which makes capturing the traffic properly a little more difficult. i dunno if it's path MTU stuff or what but the fragmentation threshold is quite low, like 600 bytes.

[astroboy]

it gets better - i just realized that PVOutput will accept a lifetime cumulative energy value for v1 if you pass "1" as argument c1. it resets the cumulative daily value when it receives the first data point on a given day. so pretty much all you have to do is fish out the lifetime energy, convert to watt-hours, and post it. repeated posts of the same date/v1 pair should still be OK, i hope. we'll find out tomorrow...

[astroboy]

i ran all day today using the "difference in total lifetime energy method" and PVOutputs shows 6W more than the old sunpower monitoring site. based on observations, i think the monitoring website is truncating the 3 decimal digits to 2 instead of rounding them, so the two values are probably exactly the same. going to declare victory.

also i looked at the return traffic from the monitoring server, and what i'm seeing is a 4-digit message number coming back (1002) with a UTC timestamp. this timestamp does not always match what goes to the server, so it's probably not used as a protocol level "ack". the MD5sum (or whatever) in message 102 does not come back either. interestingly in the return packet (which is an http response) they set a cookie with a session ID, but it's set to expire 0 or 1 seconds after the date in the http header. i suppose that all that's happening at a protocol level here is http: an HTTP request (containing the inverter data) is made, which then expects an HTTP response. if the supervisor/inverter doesn't get the response, i guess it just makes the request again. so there does not appear to be any other "custom" protocol at work here.

also, i saw the supervisor repeat some 130 messages today. as long as there are not multiple outstanding messages this works out OK - the duplicate message causes my script to write PVOutput with the same data again, since the difference in the inverter total lifetime energy is 0. if multiple 130s are repeated out-of-order then my calculations will get screwed up.

i guess what happens when the supervisor has been out of contact with the monitoring website for a long time is still an open question, and one day i'll have to test that to make sure the script handles that corner case. will be interesting to see if multiple 130/131s go out with one 102 protecting them in a single HTTP request.

[astroboy]

alright, glad we got this sorted out... my script does not do any local logging, but i think what i'll do is just have it compute the very first interval the wrong way with the AC power (which at this point i'm assuming is probably peak), then switch to using the lifetime consumption. the error should be pretty tiny.

i guess your 3 vs 4 decimal digit difference could be down to different firmware revisions in the inverter or supervisor? anyway here's hoping we continue to have 1W resolution even after 9999kwh.

[robillard]

i was looking at the packets a little more while responding to you and realized that 130.1 seems to contain total lifetime energy expressed in kwh. because my system has not been up for that long, this number currently reads out as 1549.341 kwh - there are 3 decimal digits. looking at some of these fields, it seems that they might have a fixed number of digits and the decimal point just floats around, decreasing the accuracy as the number gets bigger. what's your lifetime production in 130.1 and does it still have 3 decimal digits?

Weird, two months ago in a capture I did (that I still have lying around), my 130.1 number was 4.4 digits. I just (mid last month) upgraded my solar install with more panels and a (new) larger inverter, so the 130.1 number restarted, and now I have 3.3 digits! Don't know why I lost that extra trailing digit of precision... (If it had been floating-decimal precision, then I should be getting 3.5!) We shall see how the number varies over time...

if the 3 decimal places holds even as you get to 10,000+ kwh then i think you can just derive the last interval's production from the difference between the prior lifetime and the current lifetime production...

Yes, I have this in my notes from a few months ago when last I looked at this:

TrueIntervalNet == "Net Meter Total Lifetime Energy(kWh)"[t0] - "Net Meter Total Lifetime Energy(kWh)"[t-1]
ProdInterval == "Inverter Cumulative Energy Produced(kWh)"[t0] - "Inverter Cumulative Energy Produced(kWh)"[t-1]

But I the "Inverter Cumulative Energy Produced" value appears to be synthetic, and it never occurred to me (don't know why) to just use the "Inverter Total Lifetime Energy(kWh)" value instead... And sure enough, that works! If I look at the (t0)-(t-5) values for 130.1 and compare them against the interval production values that I get from the SunPower monitoring site, they are on track (albeit SP supervisor or website is rounding). Here are some examples:

"Inverter Total Lifetime Energy(kWh)" (130.1) numbers from packet trace:
09:30: 2129.7991
09:35: 2129.9181
09:40: 2130.0261
09:45: 2130.1481

(t0)-(t-5) delta (kWh):
09:35: 0.1190
09:40: 0.1080
09:45: 0.1220

"System Interval Energy Produced(kWh)" from SunPower monitoring:
09:30: 0.09
09:35: 0.12
09:40: 0.11
09:45: 0.12

Cool beans, man, that works!

For the consumption monitoring, it's only slightly trickier: the "Net Meter Total Lifetime Energy(kWh)" value is net, i.e. production minus consumption, so it's pretty easy math to get the consumption value based on that.

Cool, I'll set that up and run with it for a bit, and see if the numbers agree over time.

It is unfortunate that I will have to prime the system with a single record of values before I can start producing meaningful results, but at least once I've done this once, and persisted the data, then I never need to do is again, because I can use the previously-persisted data when I start back up again (assuming no discontinuities)...

Cool, thanks for talking this through with me, man, much obliged! I owe you a beverage...

[astroboy]

yeah the 128 bits triggered the thought of md5sum in my mind. i did not notice the URL difference - that is interesting. my guess is that 100 is just a heartbeat since in a system with no consumption monitoring, the inverter stops reporting 130/131 when the sun goes down. having said that the 120 messages (which appear to be supervisor status) do continue 24 hours a day making message 100 somewhat redundant, but since it goes to a different URL i suppose it's purpose is different.

[astroboy]

i was looking at the packets a little more while responding to you and realized that 130.1 seems to contain total lifetime energy expressed in kwh. because my system has not been up for that long, this number currently reads out as 1549.341 kwh - there are 3 decimal digits. looking at some of these fields, it seems that they might have a fixed number of digits and the decimal point just floats around, decreasing the accuracy as the number gets bigger. what's your lifetime production in 130.1 and does it still have 3 decimal digits?

if the 3 decimal places holds even as you get to 10,000+ kwh then i think you can just derive the current interval's production from the difference between the prior lifetime and the current lifetime production...

also i don't have message 140 so i assume that contains consumption data?

[robillard]

What I see is that every 2 minutes, the SP supervisor sends a message to the server that contains only 100 and 102 sub-messages (for lack of a better term), then every 5 minutes it sends the full one. Those packets get properly ACKed by the server. In those packets, the 100 sub-message just contains identifying info (the serial number of my supervisor) and the date+time, then the 102 sub-message contains this 128-bit value. And this 128-bit value is unique to the message in question, i.e. it is not a repeat of a previous one. Given that the only variable data in the 100 sub-message is the date+time, if the 102 was merely a hash/md5, then what's the point of the entire message? Surely not a ping, since every 5 minutes we get a full message with all the system data...

On the other hand (after thinking about your theory for a bit more), it is rather significant that the size of the base64 data is exactly 128 bits, the size of an md5 or sha1 fingerprint...

Furthermore, on looking closer at the capture logs, I note that the "every 2 minutes" messages (that only contain 100 and 102 sub-messages) are actually sent to a different URL (/Command/SMS2DataCollector.aspx) than the full-data messages (/Data/SMS2DataCollector.aspx)...

So maybe it is simply a "hello" ping, and the actual data that the SP site shows is somehow derivable from the data being supplied. If so, I guess I just need to figure out how "Interval Energy Produced (kWh)" value is being computed...

[robillard]

interesting - when i found field 130.2 i started comparing it with "Inverter Avg AC Power(kW)" and found it to be identical. your response got me curious, so i asked the old sunpower interface to also plot Inverter Max AC Power and... the two datapoints appear to be identical, going back as far as i can see. do these differ in your account?

They do seem to be identical to each other, yes, but they are _not_ the same result as the System > Interval Energy Produced (kWh) (after the obvious power to energy conversion). They come close, but they do not correlate. In fact, the more intermittently-cloudy the day, the less they correlate. Some intervals come very close, others not so much:

0.813 max kW, 0.813 avg kW, 0.06 interval kWh

With the above, that last number is equivalent to having 0.72 kW average across the entire interval, which is nowhere near 0.813. Even if you assume that "0.06" is a rounded value, and we max out the pre-rounded value to 0.06499999 or 0.065, you're still looking at an average of 0.78 kW for the interval, which is still nowhere near 0.813...

I also find it highly suspicious that "max" would always be the same as "avg"... On a day where we have intermittent clouds, it's easy for the panels to get radically different UV exposure across the 5-minute interval, so the maximum should be very different than the average...

In fact, even the inverter data itself agrees with the System > Interval Energy Produced (kWh) data in the cases I've looked at. Specifically, for the internal I listed above, if I get the data for Inverter > Cumulative Energy Produced (kWh), and subtract the same value from the previous interval, it agrees with the 0.06 number, so something is off with the "Inverter Avg AC Power(kW)"... Or with my understanding of the relationship between average power and interval energy, which I admit could be the case...

actually, now i wonder if this is simply a hash/digest of the preceding messages... because 102 seems to accompany all outputs from the inverter. i have not looked at the responses from the monitoring server but i wonder if it sends back that hash to verify that it's received the monitoring data. that would be one way for the inverter to know if it needs to replay data.

Maybe? But, the checksum and ack/nack in TCP already handles the validity of the data nicely. Besides, it would make a lot more sense to include the hash/md5/digest/fingerprint with the actual message, rather than delivering it after the fact.

What I see is that every 2 minutes, the SP supervisor sends a message to the server that contains only 100 and 102 sub-messages (for lack of a better term), then every 5 minutes it sends the full one. Those packets get properly ACKed by the server. In those packets, the 100 sub-message just contains identifying info (the serial number of my supervisor) and the date+time, then the 102 sub-message contains this 128-bit value. And this 128-bit value is unique to the message in question, i.e. it is not a repeat of a previous one. Given that the only variable data in the 100 sub-message is the date+time, if the 102 was merely a hash/md5, then what's the point of the entire message? Surely not a ping, since every 5 minutes we get a full message with all the system data...

Besides, there are definitely values missing from the full message in the 130/131/140 sub-messages that are to be found in the full-on data that the (old-style flash) sunpowermonitor.com website has at its disposal (and will report via the UI-generated csv files). From the Consumption Metering, it's missing "Avg Real Power" and "Cumulative Energy", of which the most interesting is obviously the "Avg Real Power" (since presumably the cumulative can be derived from that if you keep state). From the Production Metering, it's missing "Inverter Interval Energy Produced(kWh)" and "Inverter Cumulative Energy Produced(kWh)". And in each of the sub-messages, there are data values missing: either blank (which is hard to spot in a tab-delimited file, but you can see the tab-runs if you look for them) or 0-valued. In my case, I'm always missing 130.0 (blank), 130.4 and 130.4 (blank), and 140.7 (0-valued).

i was watching this pretty carefully yesterday and it seems like as long as any sunlight is hitting the panels, you get data every 5 minutes. so that's good. i was worried about having to make the script stateful in case the interval changed. however, it turns out the script does have to be stateful because PVOutputs is expecting daily cumulative energy production in each update, rather than interval energy production. this kind of sucks as now i need to track and reset the daily cumulative at local midnight (or some other safe time.)

I thought you could supply just interval data to PVOutput, but maybe not. Yes, this is a bummer, I agree... Because I'm still downloading the data from the new-style sunpower site, and cross-posting that data (massaged slightly, IIRC) to PVOutput, I do already have that data, but if/when I can switch to just using my sniffed data, I would have preferred it be stateless as well...

i was concerned about this because a big burst could violate the pvoutput rate limit rules, and so i might have to handle this by aggregating some production data before posting it. i suppose one solution to that problem is just to donate to PVoutputs

Before I donated (and yes, I did after a couple weeks), I was throttling my posting, i.e. the cross-posting script would do two things to limit the number of post requests:
a) send batches (there's a batch-upload API) to the max size PVOutput allows, and
b) throttle the overall rate of posting, and if I hit it, sleep for the required quantum and start posting again

Since I started using PVOutput a few months after I had the solar installed, there was quite a bit of early throttled activity going on. Now, however, I never need to throttle because even if my cross-poster dies, as long as I start it back up again in a few hours, there is not sufficient data to hit the limit when using the batch API. Of course, now that I've donated, it doesn't matter, as that limit is significantly higher.

Thing is, now that I'm scraping the data locally (using a Raspberry-Pi), I just host it locally (on the same rPi) as well, so I no longer really need PVOutput, unless I want to compare my system to other systems, so the cross-posting is just kind of gravy at this point, and somewhat unnecessary. But I _do_ want all my data locally, not just the simplified data that the new SP site vends out... Hence this discussion...

[astroboy]

Not as far as I can tell... As far as I have been able to tell, 130.2(*) corresponds to "Inverter Max AC Power(kW)", but I believe what I want is the value that corresponds to "Inverter Avg AC Power(kW)", which the SunPower supervisor appears to be intentionally omitting (removing) from the data stream, and then sending it in the encrypted packet...

interesting - when i found field 130.2 i started comparing it with "Inverter Avg AC Power(kW)" and found it to be identical. your response got me curious, so i asked the old sunpower interface to also plot Inverter Max AC Power and... the two datapoints appear to be identical, going back as far as i can see. do these differ in your account?

It's cool that you're looking at the packets! Between us, maybe we can figure it all out. I worry if maybe SP sees this thread if they might either change their format or maybe hassle us? Not that there is anything wrong with this, after all, it's my data from my (owned) equipment that I'm trying to access...

i agree, same here, i own this equipment and i just want access to the data. my guess is that they have no way to upgrade the firmware in these inverters/monitors remotely and so i think what we've got is the way it's going to be. even if they could upgrade the firmware, with so many systems in the field i think it could be disastrous for them if they had a bad load. if they change anything they'd probably have to keep it all backward compatible. plus, they'd have to work with SMA on this so all in all it seems like a big hassle on their end to change anything.

Yup, I got that far. Decoding the base64 yields 16 bytes of binary data. I was hoping this might be 2 8-byte (double-precision) IEEE floating-point numbers, but it's not (I tried both BE and LE ordering). Neither is it 4 4-byte floats, nor does it appear to be binary integer data. It does appear to be encrypted or at least obfuscated somehow... Sucks that SP feels the need to hide data that I own from me...

actually, now i wonder if this is simply a hash/digest of the preceding messages... because 102 seems to accompany all outputs from the inverter. i have not looked at the responses from the monitoring server but i wonder if it sends back that hash to verify that it's received the monitoring data. that would be one way for the inverter to know if it needs to replay data.

Well, yes and no... It does for me, but I have consumption monitoring, so there is always data to be sent all the time...

But it would make sense if it does, because things like the "Inverter Avg Heat Sink Temperature" (130.8, in degrees C) is valid even if there is no solar production (it would likely equate to the ambient temperature).

I can check next time I remember to check the packet spew when the sun is not up...

i was watching this pretty carefully yesterday and it seems like as long as any sunlight is hitting the panels, you get data every 5 minutes. so that's good. i was worried about having to make the script stateful in case the interval changed. however, it turns out the script does have to be stateful because PVOutputs is expecting daily cumulative energy production in each update, rather than interval energy production. this kind of sucks as now i need to track and reset the daily cumulative at local midnight (or some other safe time.)

That's a good question, to which I do not know the answer. I haven't noticed any loss of connectivity (our net, internal and upstream, is pretty reliable). But it should be easy to test: pulling the enet cable coming from the supervisor (via the ethernet-over-power adapter) for a bit (20 minutes or so), and then watch what happens when you plug it back in. I'll try to remember to test this out when the sun is down some time...

i was concerned about this because a big burst could violate the pvoutput rate limit rules, and so i might have to handle this by aggregating some production data before posting it. i suppose one solution to that problem is just to donate to PVoutputs

[robillard]

maybe my system is different - i only have production monitoring - but doesn't message 130 contain the average AC power for the interval?

Not as far as I can tell... As far as I have been able to tell, 130.2(*) corresponds to "Inverter Max AC Power(kW)", but I believe what I want is the value that corresponds to "Inverter Avg AC Power(kW)", which the SunPower supervisor appears to be intentionally omitting (removing) from the data stream, and then sending it in the encrypted packet...

It's cool that you're looking at the packets! Between us, maybe we can figure it all out. I worry if maybe SP sees this thread if they might either change their format or maybe hassle us? Not that there is anything wrong with this, after all, it's my data from my (owned) equipment that I'm trying to access...

(* - I'm inventing this "dot notation" for the messages, with the fractional part being the (0-based) data item in the message, so 130.2 means the third data value from message 130.)

message 102 seems to be some kind of encrypted data, possibly in base64 format.

Yup, I got that far. Decoding the base64 yields 16 bytes of binary data. I was hoping this might be 2 8-byte (double-precision) IEEE floating-point numbers, but it's not (I tried both BE and LE ordering). Neither is it 4 4-byte floats, nor does it appear to be binary integer data. It does appear to be encrypted or at least obfuscated somehow... Sucks that SP feels the need to hide data that I own from me...

do you happen to know if the supervisor keeps outputting messages every 5 minutes even if production stops mid-day?

Well, yes and no... It does for me, but I have consumption monitoring, so there is always data to be sent all the time...

But it would make sense if it does, because things like the "Inverter Avg Heat Sink Temperature" (130.8, in degrees C) is valid even if there is no solar production (it would likely equate to the ambient temperature).

I can check next time I remember to check the packet spew when the sun is not up...

also what happens if it loses connectivity? do you get a bunch of 5-minute intervals in a burst when the connection is reestablished?

That's a good question, to which I do not know the answer. I haven't noticed any loss of connectivity (our net, internal and upstream, is pretty reliable). But it should be easy to test: pulling the enet cable coming from the supervisor (via the ethernet-over-power adapter) for a bit (20 minutes or so), and then watch what happens when you plug it back in. I'll try to remember to test this out when the sun is down some time...

[astroboy]

maybe my system is different - i only have production monitoring - but doesn't message 130 contain the average AC power for the interval? message 102 seems to be some kind of encrypted data, possibly in base64 format.

do you happen to know if the supervisor keeps outputting messages every 5 minutes even if production stops mid-day? also what happens if it loses connectivity? do you get a bunch of 5-minute intervals in a burst when the connection is reestablished?

[robillard]

I am trying to do something similar. Can you tell me what flags you used for the tcpdump command?

For my setup, my home router is running OpenWRT, an embedded Linux-based router software. This gives me a shell I can login to and it acts just like any other Linux terminal. I am able to run tcpdump and see the traffic flowing every few minutes to Supower's server but I haven't figured out how to view the data.

(Sorry, even though I had subscribed to this thread, I never got the notification that there was a subsequent post...)

I'm using perl's Net::Pcap package, and related packages for picking apart packets, i.e. NetPacket::Ethernet, NetPacket::IP, NetPacket::TCP, and NetPacket::UDP. So I'm not using tcpdump directly, as I don't think it would be up to the task of pretty-printing the data from the packets? My first approach was to capture the packets first, then dump the data (using a perl script) offline afterwards, and for that I used something like this:

[COLOR=#69253E][FONT=Andale Mono][SIZE=10px]sudo tcpdump -Z <local-user> -w ~/sp-pkts.pcap -U -i en0 'ip && tcp && ether src <sp-supervisor-mac-addr> && dst net <ip-addr-of-sunpowermonitor> && dst port <port>'[/SIZE][/FONT][/COLOR]

Where the values above were all gleaned from watching raw traffic filtered only with "ip && tcp".

I then wrote a simple pcap reader in perl that picked up that file and parsed the packets, displaying the contents of each packet (which are fortunately plain/text).

But since that was clumsy, I extended by perl script to do live capture from the enet interface (in promiscuous mode), and parse and dump the results as they come in to stdout. Once I understood the format of the data, I could pick that apart (it's a pretty straight-forward app-level protocol), but the data for the actual consumption and production values are, unfortunately, encrypted; all the other values are not. So I collect all the other values by sniffing this traffic, and then collect the consumption and production values by scraping the new-style monitoring site. I get all the data, in the end, but it's pretty clumsy, and I wish they were not encrypting the production and consumption data...

In your case, as for getting the data, I bet you can program your router to "echo" traffic going from the physical ethernet port to which your SP supervisor is connected, and send that to whatever ethernet port to which your computer is connected, essentially setting up a sniffer that is isolated to just the traffic from the SP supervisor to the server. At least then you're not restricted to just running what the shell on the router can provide, and you can run a perl script or other more compute-intensive stuff on the computer, rather than being limited to the computer power and tools installed on the router.