Mirroring/intercepting SunPower Monitoring Traffic?

[J.P.M.]

I've no idea how you got "upset" from the tone of my post, I was simply stating that posting a script that is specific to the hardware setup isn't going to be helpful in this case...

Sorry, Guess I should have used the sarcastic script.

[robillard]

I appreciate that, but I really cannot post my script, as it is wrapped up in 4 different workflows all tied into the same script(...)

For the curious (in case there are any), those 4 workflows are:
a) spawn the sniffer, and read and store the sniffed data
b) convert sniffed data to intermediate format for local storage (for use by web server)
c) download sunpower data from site and convert to intermediate format for local storage (for use by web server, alternate view for comparison purposes)
d) upload to PVOutput.org from locally-stored intermediate storage format

Yes, this is way more decomposed than it needs to be, but decoupling these functions allowed me to mix-and-match over time, and allows a certain amount of redundancy (for example when the sunpower site went down, or when PVoutput went down, it did not perturb the other functions).

So again, it's not really in a shareable form, and I don't have the time to put it into such a form. I truly am sorry, and I would be more than happy to help someone else with their own script.

[DanKegel]

Thanks for explaining. I'd be happy to sanitize the script for you, if you like (assuming you're willing to trust a stranger I'd keep it confidential, and just send you back the sanitized script. My site's http://kegel.com if you want to check me out.

[robillard]

That's ok. You can start off with a hardware-specific script. Someone else can generalize it. The point is just to get a proof-of-concept out there.

I appreciate that, but I really cannot post my script, as it is wrapped up in 4 different workflows all tied into the same script, and has my systemid and other stuff coded into it that I'm not willing to share (nor sanitize). I am happy to help others write a script, and share the knowledge I've gained in this process, but I cannot post my script anywhere.

Once you've got the stream of sniffed network data, filtering out the 140 and 130 entries is really easy, it's just a regex away:

/^(1[34]0)\t(20[0-9]{12})\t[0-9]+\t[^\s]+\t[^\s]*\t(-?[0-9]*\.[0-9]*)\t/

whichmsg = $1    # either 130 (for production) or 140 (for net metering)
utcdatetime = $2 # the date in UTC, use DateTime to convert to local time
currvalue = $3     # the data value (lifetime production if whichmsg == 130, lifetime net if whichmsg == 140)

production = (current production value) - (previous production value)
net = (current net value) - (previous net value)
consumption = production + net

then you're off to the races.

My script will assume no production if I do not receive a 130 (production) message before the next set of 130/140 messages and the datetime indicates that we're outside of daylight hours (by a conservative margin); in which case I assume 0 production. Otherwise, I wait a couple cycles to see if one comes in, and if it still hasn't, I assume zero production (which does mean that my latency varies over time, based on the data that gets produced). A missing 140 (net) message is, for me currently, a "die" condition, because if I get a 130 but no 140, that means something is wrong...

That's pretty much all there is to it.

[DanKegel]

I've no idea how you got "upset" from the tone of my post, I was simply stating that posting a script that is specific to the hardware setup isn't going to be helpful in this case...

He didn't really mean you were upset. He was just stating his opinion that I miss the point frequently.

In this case, I was making a different point -- sharing code is a good idea, even if the code isn't going to be useful as-is, it still can be illuminating and save other folks time. Yes, people could reinvent, but being able to look at your script while they do that might save them some time.

[robillard]

I wouldn't get upset about Dan missing the point. As I recall, he doesn't claim to know much but he does claim to like science, if I correctly interpret the sense of what he once wrote on his handle, take him at his word and, IMO only, as he seems to have demonstrated many times.

I've no idea how you got "upset" from the tone of my post, I was simply stating that posting a script that is specific to the hardware setup isn't going to be helpful in this case...

[DanKegel]

I think you're missing the point... The script will be naturally tailored to the workflow required by the hardware; mine certainly is...

That's ok. You can start off with a hardware-specific script. Someone else can generalize it. The point is just to get a proof-of-concept out there.

[robillard]

And yet it would be easy for you to check the script in to github.... maybe someone else would contribute polish to address those other issues. Give it a try, and be sure to credit that other project!

I think you're missing the point... The script will be naturally tailored to the workflow required by the hardware; mine certainly is...

[robillard]

I agree. The logic of the script is the easier part, setting up the system that is necessary to capture the packets and deal with the data is, while not exactly difficult, certainly more involved, and it's not something that can be captured on github...

Also, this thread serves as a pretty decent write-up, both of the approach to take for scripting, and for at least an approach to the hardware setup.

[robillard]

given all that, anyone who can build either one of those things probably also has the technical chops to just write the script from scratch...

I agree. The logic of the script is the easier part, setting up the system that is necessary to capture the packets and deal with the data is, while not exactly difficult, certainly more involved, and it's not something that can be captured on github...

[robillard]

the sunpower monitoring outage a couple of weekends ago was interesting - the supervisor seemed to keep retransmitting a whole load of data points for some number of hours, and then it gave up. when the monitoring came back on line it eventually transmitted everything that was pending, but it seems to have taken many hours of successful "live" transmissions for it to decide to replay the failed messages.

Indeed! Since my script _is_ stateful, and I serve up my own page with a graph, my stuff kept going despite the sunpower outage, just based on the data that the supervisor kept trying to send. Combine that with the finer resolution of the differential data (there's some heavy rounding going on on the sunpower site!), and I'm quite happy with the results!

my script is once again stateful; without checking the latest data point already submitted to PVOutputs, i pretty quickly overran the API request limit while the supervisor was sending the same stuff over and over again. also the lack of packet reassembly in Net::PcapUtils kind of sucks; with the big bursts of data the packets are fragmented. for whatever reason, under normal circumstances, if the supervisor sends a long packet, the 130 messages come early enough in the packet that they are not split by a fragmentation boundary. however when the supervisor was backed up there were lots of 130 messages in big, fragmented packets and so i missed some of them. Net::Pcap is kind of a pain in the butt so i'll probably leave well enough alone, it seems to work well enough.

I have two scripts: the first one simply handles the Pcap stuff and streams just the supervisor->sunpower traffic, extracted as text, to stdout, and the second acts on the data. The second script spawns the first, and since my workflow requires promiscuous mode on the network interface, which requires sudo, this has the added benefit that only the Pcap script needs to run sudo. The decoupling means I don't incur any buffer overflows in the Pcap kernel buffer (I've made that big enough anyways that it would be very unlikely to happen!) in the first script, and means that the second script only needs to deal with a stream of text lines, which is easy.

Also, turns out my net (and thus consumption) numbers were right after all (and track closely to the sunpower numbers, albeit without the massive rounding errors of the sunpower system), there was a flaw in my logic that summarizes each day's cumulative numbers. So the data was correct, just my processing sucks... I can fix that, I'm sure...

Thanks for the response, and for all your help working through this (and the revelation that the wonky 102 messages really are most likely simply a checksum), much obliged for the help!

[DanKegel]

anyone who can build either one of those things probably also has the technical chops to just write the script from scratch...

And yet it would be easy for you to check the script in to github.... maybe someone else would contribute polish to address those other issues. Give it a try, and be sure to credit that other project!

[astroboy]

Has anyone written something like https://github.com/jbuehl/solaredge but for sunpower? (hint, hint )

the section of my script that submits to PVO was ripped from that script!

i think the problem is that most people won't be able to make use of such a script. in order to run the script, you either have to build some kind of ethernet bridge out of a raspberry pi, or you have to have a home-made router which runs linux and can execute arbitrary scripts. probably even a D-link or similar running DDWRT or tomato would be a tall order, since as written my script relies on a whole lot of perl modules that have to be installed from CPAN. not sure how easy it is to install/compile all the perl support is on one of those machines.

given all that, anyone who can build either one of those things probably also has the technical chops to just write the script from scratch...

[DanKegel]

Has anyone written something like https://github.com/jbuehl/solaredge but for sunpower? (hint, hint )

[astroboy]

astroboy, which number are you using for net? I'm using this first entry from the 140 messages for net, since that appears to be a lifetime net...

my system does not have power monitoring - instead i am using an EAGLE, uploading that to wattvision, and then PVOutputs pulls that from WV. so basically i ignore every message except for 130... sorry i can't be of much help there.

the sunpower monitoring outage a couple of weekends ago was interesting - the supervisor seemed to keep retransmitting a whole load of data points for some number of hours, and then it gave up. when the monitoring came back on line it eventually transmitted everything that was pending, but it seems to have taken many hours of successful "live" transmissions for it to decide to replay the failed messages.

my script is once again stateful; without checking the latest data point already submitted to PVOutputs, i pretty quickly overran the API request limit while the supervisor was sending the same stuff over and over again. also the lack of packet reassembly in Net::PcapUtils kind of sucks; with the big bursts of data the packets are fragmented. for whatever reason, under normal circumstances, if the supervisor sends a long packet, the 130 messages come early enough in the packet that they are not split by a fragmentation boundary. however when the supervisor was backed up there were lots of 130 messages in big, fragmented packets and so i missed some of them. Net::Pcap is kind of a pain in the butt so i'll probably leave well enough alone, it seems to work well enough.