Mirroring/intercepting SunPower Monitoring Traffic?

[cebury]

I had a question about your regular expressions. I'm terrible at regex, it confuses me. I did have to modify your regex to look for 2017 in the timestamp, but I'm wondering if there's a better way to write the regex so it doesn't have to be modified yearly.

...
It looks like each of my panels are reporting on the 130 lines, so I can break those out into my own DB / web portal. I'm having a hard time deciphering the rest of the lines, though. I believe the 1st number (ie. "18.5466" in the first 130 line) is the Watts, the 2nd number (ie. "0.0417") is the kWh, but those are mostly guesses.

...
I get lost after that as to what the rest of the values are.

Not trying to be a troll here, but the first question is elementary. Not sure how you've gone this far without knowing about variables and date grabbing functions. It is even discussed at the top of page 5.


As for your second, the answer was provided by Rob above in his code itself:

if ($msg == 140) { # this is a net metering message, and $value is net metering value in (IIRC) W averaged over the 5-minute interval
# consumption = (corresponding production) + net
}
else { # this is a production message, and $value is a production valuein (IIRC) W averaged over the 5-minute interval
}

the hardest part is in what he wrote next

>>> (One of) the (many) trick(s) involved herein is how you deal with calculating consumption from net and production... They do not necessarily arrive in the same packet (although they often do) nor does production (130) always arrive before net (140).

and the following paragraph about handling the kwh value provided in the dump about net, production, and corresponding production row dump.

good luck, there is a lot of critical technical discussion in the last couple pages I order to make this script your own.

[ehampshire]

Not trying to be a troll here, but the first question is elementary. Not sure how you've gone this far without knowing about variables and date grabbing functions. It is even discussed at the top of page 5.

I understand variables and dealing with dates. I do program for my job, after all, but not in Perl. Always hated Perl, OO Perl is torture. Also, don't like regular expressions, which I stated. Never been good at them. I realize they can be super powerful, but I just don't get them. I've since switched the regex to not parse things for me. I'm writing the entire line to a log file and going to deal with the processing of the lines in a language I'm better at (Python or Java) and don't have to google for every little thing.

good luck, there is a lot of critical technical discussion in the last couple pages I order to make this script your own.

I very much AM doing just that. Thought I would ask for some help here and additional insight. This thread is the top google hit and the most informative for anyone looking for more detail than they can get in the existing SunPower portal. The general consensus of this thread is "read the thread and digest" (which I've done numerous times) and then mostly "figure it out for yourself". I don't understand the resistance to helping make it easier for others to follow in your tracks. Is it a legal fear?


Also, the main point of my post was asking if anyone had figured out what all those other columns are. I understand you guys are processing what is sent up to SunPower and getting more detailed info for the net and production values, but is there any use for the rest of the data?

[astroboy]

i am parsing 130 with the following code (kind of a mess, and i don't know what all the fields mean)

# message 130 seems to have missing fields when the sun
# goes down, hence the complexity of this match
# expression. it was created from a daytime and evening
# message 130 using regrxr.com.

my ($sm_time, $sm_tot_kwh, $peak_pwr, $dummy1,
$dummy2, $dummy3, $dc_volts, $dummy4,
$dummy5, $dummy6, $inv_temp_c, $dummy7, $sm_freq) =

$_ =~ m/130\s+(\d+)\s+\d+\s+SMA-SB-5000TL-US-22\s+(\d+\.\d+)\s((\d+.\d+)|\d)\s+((\d+\.\d+)|\d)\ s((\d+\.\d+)|\d)\s((\d+\.\d+)|\d)\s((\d+\.\d+)|\d+ )\s((\d+\.\d+)|\d)\s\d/;

if ((not defined $sm_time) or (not defined $peak_pwr) or
(not defined $sm_tot_kwh) or (not defined $dc_volts) or
(not defined $inv_temp_c)) {
# OK, give up...
$bad_parse = 1;
}

that's obviously tied to my particular inverter's model number so you'd have to change that based on what you see in your message 130.

[ehampshire]

Thanks, astroboy! That's actually pretty useful as you've figured out more fields than I certainly have. How did you know some what some of the values were, like the Temperature and DC Volts? Also, can you explain what $sm_freq and $peak_pwr is? An example line from your tcpdump would be helpful too (ie. the full line like below).

For instance, here's one of mine that just went by:

130 20170114185000 414051637006544 AC_Module_Type_C 20.9279 0.0491 244.7866 0.4237 0.0515 58.1278 0.8935 20.375 59.997 0

I know we have different inverters (looks like you have a Sunnyboy string inverter, I have micro inverters, so I get 24 of these 130 lines to match the number of panels I have). If I assume that the 3rd to the last is the temp (like yours), that would be 20C = 68F. Doesn't seem that likely as it's only 32F out there, unless the panels heat up a lot compared to ambient temperature (the sun is behind clouds, so again, doubtful).

[astroboy]

Thanks, astroboy! That's actually pretty useful as you've figured out more fields than I certainly have. How did you know some what some of the values were, like the Temperature and DC Volts? Also, can you explain what $sm_freq and $peak_pwr is? An example line from your tcpdump would be helpful too (ie. the full line like below).

For instance, here's one of mine that just went by:



I know we have different inverters (looks like you have a Sunnyboy string inverter, I have micro inverters, so I get 24 of these 130 lines to match the number of panels I have). If I assume that the 3rd to the last is the temp (like yours), that would be 20C = 68F. Doesn't seem that likely as it's only 32F out there, unless the panels heat up a lot compared to ambient temperature (the sun is behind clouds, so again, doubtful).

to figure these things out i went to the "old" sunpower monitoring site which can show everything the inverter reports, and painstakingly matched them up. all the "dummy" values are things that i just could not figure out, or seemed not worth tracking. i had a lot of trouble with it until i realized that there were fields missing when the sun had gone down yet the monitor was still producing 130s. a while after sunset, it stops sending 130s and only sends 100s. regrxr.com is really useful to test your match string, there's no way i could have come up with that string without it.

in my case the temperature is the inverter heatsink temperature, so 68F is probably a normal temperature for the heatsink at 32F ambient. i assume that they want to monitor this because once it reaches 80C+ it's probably time to shut down the inverter or at least drop the power output to avoid damage to the power transistors in the inverter. do your 130s show some variation in those temperatures? i would expect that since each inverter has its own heatsink...

sm_freq is apparently the inverter's AC waveform frequency. peak_pwr is the peak output power seen during the 5 minutes that the 130 sample covers. rather than trying to convert this to an energy, and since the monitor outputs the cumulative generated energy since commissioning and PVOutput accepts a cumulative energy value, i stopped trying to compute the interval energy and just report the cumulative power to PVO.

i'll find a 130 line later today, i need to step out soon.

[astroboy]

this is how i think my 130 message breaks down:

YYYYMMDDHHMMSS (UTC)

inverter serial #

inverter name

just a tab

total lifetime nrg (kwh)

peak AC power

just a tab

?? - sometimes just a tab

avg dc voltage

avg dc current

inverter heatsink, degC

frequency

? reactive power

130 20170115201000 XXXXXXXXXX SMA-SB-5000TL-US-22 9485.6914 2.713 1.3638 283.8639 4.8077 34.335 59.98 290

[intipower]

Hi everyone. For those of you who are using the network snooping method of collecting data would you mind telling me how many of you have PV Supervisor PVS5 equipment (SunPower). This is the equipment I have and I'm working on a solution which queries the pv supervisor directly to collect data (I want to collect data every 60 seconds) and I have a working solution now. I'm making a decision if my solution will be a hack job (as in ugly code) because I'm the only one interested in using it or if I'll make it more professional because others may wish to use it.
My solution is using a raspberry PI, Maria database and python code. Thanks

[ehampshire]

Yes, based on the serial of my system (PVS5M508095c) I'd say we have the same model supervisor. I'd be interested in knowing how you query it directly, that seems more reliable than the way the rest of us have set it up (by man-in-the-middling the traffic).

I was just about to come back to this thread and post my writeup of what I did. I just finished it yesterday, so it's a bit rough and I need to do some more formatting, but it does link to all the tools I developed for this little project.

http://2b1f625d-a29c-41a3-81d7-ca1904095e20-19686.remixer.website/

[intipower]

ehampshire: Oh... you are so very close the easy way to get the data, I'm surprised you missed the clue to it when you were changing your network connection on the pvs5. I checked out the link in your post. It's a nice set-up, I like the front end presentation you developed. I can write the back-end code but I'm not so good at the web stuff so I'm working on that part. My project isn't quite ready for general consumption but I'll give you the rough design and you and others can try it.

First it's necessary to know something about the pvs5 (pv supervisor.) There are two network ports. The one connected to your network just sends data to SunPower and this is the data that you are capturing on the 5 min interval. This network port acts as a DHCP client and gets it's IP address from your network. You can't do much with that port because it just sends the data. The other port (the installers network port) acts as a DHCP server so when the installer connects their laptop to the pvs5, it provides an IP address. I needed to have both ports on my network to get my solution to work. For my installation I didn't want to use wireless so I ran a cat-6 wire for the customer network connection to connect the pvs5 to my switch. This is how SunPower is collecting my data the same way as every one else. I ran another network wire from the pvs5 installers network port to my wiring cabinet. You can't connect the installers network port to your local network because then you would have two DHCP servers on your network and that would just mess things up. To isolate the installers network port I took a raspberry PI, added a second network port on it by attaching a USB-LAN adapter. One of the raspberry pi network ports goes to my LAN and the other goes to the pvs5 installers port. Both network adapters get their LAN address from the network they're attached to. The only other thing I did on the raspberry pi was to set routing between the two interfaces in /etc/ssctl.cnf and I made my LAN DHCP server provide a fixed address to my PI so I would always know what address it was on (in my case I'm giving it a fixed address of 192.168.1.100) Once this is set up you can log into the PI and see what addresses each interfaces got from the network. In my case the installers port network got an address from the 172.27.153.x network (the pvs5 turned out to have an address of 172.27.153.1) and on the other interface on my LAN it got a 192.168.1.100 address as I had set it up to do this in my DHCP. Now to test connectivity from my lan I went to my pc and added a static route (route add 172.27.153.0 mask 255.255.255.0 192.168.1.100) then try to ping 172.27.153.1. If it works then you may want to define that route to be permanent but in my case I didn't want to because of how my collection will ultimately be configured to work.

Now you have connectivity. My pvs5 being at address 172.27.153.1 I can go to my web browser and go to this URL:

http://172.27.153.1/cgi-bin/dl_cgi?Command=DeviceList

What I get back is a very nice JSON format listing of my systems information that looks like this below. I know this blog will probably mess up the nice formatting of the json I have pasted in, sorry. (I have taken out some things to prevent identification of my system and left in only one inverter data)

Now all you have to do is write the code to do the query automatically and store it where you want to. I'm using python to do that part and I'm not quite finished yet so can't share yet but I'll have something in the not too distant future. I also plan to put the whole process in a blog somewhere like the page you have so others can benefit. I hope you found this information interesting and would love to get any feedback.

{ "result":"succeed", "devices":[ { "ISDETAIL":"1", "SERIAL":"PVS5XXXXXXXX", "TYPE":"PVS5-METER-P", "STATE":"working", "STATEDESCR":"Working", "MODEL":"PVS5M0400p", "DESCR":"Power Meter PVS5XXXXXXX", "DEVICE_TYPE":"Power Meter", "SWVER":"4", "PORT":"", "DATATIME":"2017,03,29,21,15,37", "ct_scaler":"50", "energy_total":"0", "power_w":"0", "power_var":"0", "power_va":"0", "power_factor":"1", "freq":"60", "CAL0":"50", "CURTIME":"2017,03,29,21,15,38" }, { "ISDETAIL":"1", "SERIAL":"PVS5XXXXXXXX", "TYPE":"PVS5-METER-C", "STATE":"working", "STATEDESCR":"Working", "MODEL":"PVS5M0400c", "DESCR":"Power Meter PVS5XXXXXXXX", "DEVICE_TYPE":"Power Meter", "SWVER":"4", "PORT":"", "DATATIME":"2017,03,29,21,15,37", "ct_scaler":"100", "energy_total":"-245.07", "power_w":"-4.5647", "power_var":"0.4764", "power_va":"4.6007", "power_factor":"-0.9921", "freq":"60", "CAL0":"100", "CURTIME":"2017,03,29,21,15,38" }, { "ISDETAIL":"1", "SERIAL":"41405XXXXXXXXXX", "TYPE":"SOLARBRIDGE", "STATE":"working", "STATEDESCR":"Working", "MODEL":"AC_Module_Type_C", "DESCR":"Inverter 41405XXXXXXXXXX", "DEVICE_TYPE":"Inverter", "SWVER":"951007408", "PORT":"", "DATATIME":"2017,03,29,21,15,37", "energy_total":"95.1664", "ac_power":"0.2871", "ac_volt":"254.1222", "ac_curr":"1.1623", "dc_power":"0.299", "dc_volt":"52.2409", "dc_curr":"5.698", "heatsink_temp":"57.5", "freq":"59.9772", "CURTIME":"2017,03,29,21,15,38" },
{ "DETAIL":"detail", "STATE":"working", "STATEDESCR":"Working", "SERIAL":"ZT16258XXXXXXXXXXXX", "MODEL":"PV Supervisor PVS5", "HWVER":"3.3", "SWVER":"5.0.0, Build 313", "DEVICE_TYPE":"PVS", "DATATIME":"2017,03,29,21,15,00", "dl_err_count":"0", "dl_comm_err":"395", "dl_skipped_scans":"0", "dl_scan_time":"0", "dl_untransmitted":"0", "dl_uptime":"1354458", "dl_cpu_load":"0.08", "dl_mem_used":"29884", "dl_flash_avail":"12484", "CURTIME":"2017,03,29,21,15,40" } ] }

[ehampshire]

Ugh! Well, that is certainly easier. Also, best I can tell, you are correct, it updates every MINUTE! That's awesome, though, and probably much more reliable than my packet sniffing method. Time to move the BeagleBone Black out to the garage (where the PV monitor is). Right now it's just hooked up to my iMac in the garage. I guess I could enable IP forwarding on my mac, but, details of my network, bleh.... Connecting to that URL via a web browser pulls up the JSON data nicely. Success!

Ok, how did you figure out what "Command=" you could send? Are there others?

[ehampshire]

Oh, and thanks for the compliment on my front end. I took it as a personal challenge to just replicate what SunPower's portal does. There's a little "Details" link there that takes you to another page that does their fancy Power display, like this: https://imgur.com/a/WUMQl

[intipower]

While I was going through the screens on the installers port I was also "inspecting" the content provided by the web server within the pvs5. I got to a screen in the process where it displayed all the information and dug down into the javascript code until I found the link. I would guess that there probably are others but in the interest of not screwing up my configuration and since I already had all the info from that one link I didn't need to look further. Maybe you stopped in the process before you got to the screen which showed the data. It's that screen where it shows the data where you would have been so close but missed the clue to getting it directly. When I was going through the config screens I thought I messed up the configuration because my pvs5 stopped sending data for about 5 hours after I was done playing around so I thought I would have to beg the installers to dig me out of the hole I made for myself but it started up again to my relief. I wouldn't suggest to anyone to go through the screens unless they were willing to have to pay someone later to fix the configuration that they screwed up. I was willing to take the chance.

I need to clean up my python back end data processes and then get my front end together for the web page. I had planned something similar to your web page but not being a javascript programmer it's going to be a learning curve. I want my page to dynamically update as data comes in on each inverter (I have 22 of them) so I've got some work to do.

The solution works nicely because all you need is a $25 raspberry pi, a cheep usb-lan adapter and a bit of code. With the snooping method there are too many moving parts to worry about.

[cebury]

Nice work you guys and for sharing with others.

[JJNorcal]

Great thread. Thanks ehampshire and intipower! I haven

[JJNorcal]

OK. Short version now. I simply can't cut and paste from a real editor, even a rudimentary one. The preview looks fine but the end result is unreadable.

My goal is to report power generation to pvoutput. I've been using Efergy for some time, but it is inaccurate.

I played around with what I think is a reverse engineered SunPower api over at github.com/jeffkowalski/sunpower. I have confirmed that the curl queries are functional. I focused on GetCurrentPower, and while functional, it seems you can only get one sample every 15-30 minutes; repeated queries result in the same old data point, and then suddenly it moves forward by a big block of time. There are other methods that could be explored in the listed SP help links, but I would prefer to stay completely under the covers given that SP has not made the API public.

So I came back to this thread and was happy to see 2 new approaches for consideration. Thanks ehampshire and intipower!

Given that I'm chicken to drill a hole in my monitor to connect an enet cable to the mgmt rj45 port, and coupled with my minimal needs of pushing power data every 5 minutes, I decided to take the ehampshire approach for a spin.

I build a pi based 2-port enet switch, loaded nodejs onto it, and wrote a node app to parse tcpdump. So far so good. I should also say that I have correlated the 5th 130 parameter as power (either instantaneous or 5 min average).

The ehampshire approach results in a steady 5 minute stream of data, lagging maybe 10 minutes behind real time. I am in process of hooking it up to pvoutput.

Thanks again guys. Hope this post works...