Mirroring/intercepting SunPower Monitoring Traffic?

[sdold]

Thanks for some information here, I use the "devicelist" url to write a quick script to send data to pvoutput.org (using a cronjob every 5 mins):

https://pastebin.com/3aYeSH5u

And my system: https://pvoutput.org/list.jsp?userid=68865

Hi Faraway and welcome to solarpaneltalk.com. Once the admin approves you for posting, you'll be able to start your own thread. I left this one here since it seems to be related to the topic. Posting links it normally frowned on for first timers since spammers do it frequently, but I tried yours and they seem fine. I didn't know about pastebin.com, it looks useful.

[faraway]

Thanks for some information here, I use the "devicelist" url to write a quick script to send data to pvoutput.org (using a cronjob every 5 mins):

https://pastebin.com/3aYeSH5u

And my system: https://pvoutput.org/list.jsp?userid=68865

[JJWatMyself]

Great work by all on figuring out and discussing URLs. I created a new thread, basically as a landing page and place to discuss a project I'm working on for monitoring SunPower systems using PRTG. The thread is pending approval as I'm a noob on this forum. Here is a link to it. I'm guessing it will be approved in a day or so.

https://www.solarpaneltalk.com/forum/solar-panels-for-home/solar-panel-system-equipment/381469-project-monitoring-sunpower-using-prtg

[ehampshire]

Ok, my supervisor did update, but my code still works. Thought I'd update the thread to say that. I'm now on Version 2018.5, Build 2041

[ehampshire]

Looks like I'm still on Version 5.0.0, Build 325

I've complete forgot how I query the device for that, but browsing through the setup pages gives it to me. It also notes there's a firmware update available, downloads it, and "succeeds" on install. Of course, going back to that webpage shows it's still on the same version. So, I guess I'm lucky that it can't update past it? Sorry, maybe my updated scripts won't actually work for people past that firmware version.

[ehampshire]

Hi Eric. Because I am unable to intercept traffic from my supervisor, I decided to go the monitor.us.sunpower.com API route that Jeff Kowalski implemented. I have branched it and am doing a Python implementation of the capture part. I have the capture largely done, now working on the visualization / reporting. I will post repo link once done - hopefully later this week. I like your graphics.

I can't recall whether you said what firmware you are on. Did you say? I am on Version 2018.4, Build 1092

I won't have the per-panel data which is too bad but I don't think i have a micro-inverter (or do I, i have one? SMA 3800 TL). We only have 12 panels, small 3.9 kW system.

If you hook a Raspberry Pi (or any computer really) up to the installer port on your supervisor, you can use the same direct query method I use and skip the SunPower portal entirely. They are getting the individual panel data, they just aren't exposing it. I would love to see what you came up with, I wasn't sure what to do for visualization, so I kind of just copied the SunPower site as a challenge. Can slice/dice the data however I want, but my help page does show how to layout your panels like they are on your roof (and my page is laid out that way).

I forget how to get my supervisor version - remind me?

[cord_thomas]

Hi Eric. Because I am unable to intercept traffic from my supervisor, I decided to go the monitor.us.sunpower.com API route that Jeff Kowalski implemented. I have branched it and am doing a Python implementation of the capture part. I have the capture largely done, now working on the visualization / reporting. I will post repo link once done - hopefully later this week. I like your graphics.

I can't recall whether you said what firmware you are on. Did you say? I am on Version 2018.4, Build 1092

I won't have the per-panel data which is too bad but I don't think i have a micro-inverter (or do I, i have one? SMA 3800 TL). We only have 12 panels, small 3.9 kW system.

[Mike90250]

Hmmmmmm, so I wrote up this reply but it showed as awaiting approval. Maybe because I included an imgur link to a screenshot of my current code working? Anyways, wanted to let you guys know I updated my GitHub project with new, working code based on the direct querying of the SP supervisor.

posts with more than 1 link, get held for moderation, we released it.

[ehampshire]

Hmmmmmm, so I wrote up this reply but it showed as awaiting approval. Maybe because I included an imgur link to a screenshot of my current code working? Anyways, wanted to let you guys know I updated my GitHub project with new, working code based on the direct querying of the SP supervisor.

[ehampshire]

Hey everyone,
Over the last week I've been hacking on this and I got my math to work out finally. My capture script hasn't changed, although I did modify the "sp_raw2" table in my DB to include the insert_timestamp rather than just the src_timestamp from the panels/supervisor. Things are still slightly wonky as I'm using the supervisors total_energy reported to calculate whether I'm sending power to the grid or pulling from the grid. This, unfortunately, doesn't give me an entirely accurate view about how much power my house is CONSUMING. I will keep looking at that and see if I can't work out how to get a more accurate view. For now, though, check out my latest commits to the GitHub project here: https://github.com/ehampshire/sunpower/tree/dev . Mostly just updates to the web section (with index2.php) and a new solar_analysis3.py that fills in the sp_energy and sp_power tables appropriately. Here's a screenshot from a couple minutes ago of what it looks like for my system: https://imgur.com/a/3CdJzDh

Hope that helps, guys!

[robillard]

My PV Supervisor has two Ethernet jacks and they only use one for the powerline. In my comm box where they mounted only the PV super. they of course had to mount a standard elec. outlet to connect the powerline to, mine happens to be a dual outlet. Is there a reason I can't simply buy my own powerline and use the second PV super. port, (since their powerline is encrypted) and get the data that way?

The problem is not the powerline. I bypassed the powerline (disconnected it at the other end) and have it provisioned to use the Ethernet port.

the problem is that SP changed where and how they transmit the data to their server. They no longer send messages to SunPowers server in the clear. Instead they use an AWS-hosted service to receive the data, and they send the data encrypted to that server.

so no, changing from powerline to the hardwired Ethernet will make no difference.

[LLB]

robillard,

My system is the same as yours

https://www.solarpaneltalk.com/forum/solar-panels-for-home/solar-panel-system-equipment/19587-mirroring-intercepting-sunpower-monitoring-traffic?p=240342#post240342

No consumption monitoring though, but I'm assuming the same PV supervisor model, as we likely installed around the same time (as joining this forum).

My PV Supervisor has two Ethernet jacks and they only use one for the powerline. In my comm box where they mounted only the PV super. they of course had to mount a standard elec. outlet to connect the powerline to, mine happens to be a dual outlet. Is there a reason I can't simply buy my own powerline and use the second PV super. port, (since their powerline is encrypted) and get the data that way?

Thanks,
LLB

[cord_thomas]

Thank you robillard - that is what i thought I understood from the thread but wanted to be sure. I posted an issue to ehampshire's Github project and he said he's still able to pull data from the supervisor - maybe his firmware hasn't been updated? I hadn't even thought to get the per-panel generation data. I want data locally to blend it with my consumption data and occupancy information to understand how my generation/consumption match up to better inform some automation of how my various appliances operate, car charging time, etc. Also wanted to see how my panels' performance aligned with the solar situation: https://github.com/SunPower/SolarUtils.

I am looking for documentation then on their "API" - i found this https://github.com/jeffkowalski/sunpower project which points to some "help" files. What is not clear to me is whether you can get more than just System cumulative and interval energy production (which is all I really need) or the stats on the supervisor and inverter as well - all the data that is presented in the sonpowermonitor.com site.

Thanks again

[robillard]

To be clear: you can still use the older method of simply downloading the data from the sunpower site using their web API (well, it's not an API, but it's usable), and I continue to do that to cache the data locally, but for those of us with micro-inverter based systems, that data does not include the very useful per-panel data...

[robillard]

I am only seeing the 100/102 messages - i have been monitoring for an hour during daylight - overcast, but still generating some power.

100 SPMS 10 ZT163185000441C1526 20180525004318
102 AiF/Gs2xWTn4EpZdZF17

Yeah, unfortunately, around June 25th of last year (2017), Sunpower rolled out new firmware that uses a different site, and encrypted data traffic for their data uploads. That apparently applied to all micro-inverter-based installs, so our main house got hit by that, and I have no way to retrieve per-panel data now. On May 11th of this year (2018), this firmware update also hit our rental house (which uses DC from the panels to a single inverter), as well. Now the supervisor still pings the sunpower server using the old protocol, but that's just the 100/102 message (which appears to just be an "I'm alive and OK" message). All other traffic goes to a different server, and is encrypted. Very, very unfortunate. SunPower just does not realize that, as system owners, this is _our_ data, not theirs...