Mirroring/intercepting SunPower Monitoring Traffic?

[ehampshire]

Sweet! Your API docs helped me figure out the X,Y coordinates of my panels to lay out my webpage appropriately. I sincerely appreciate that! I started a PVO script to convert the SP dump into their format, but didn't get very far. Mostly because their API docs are kind of terrible and I wasn't having the best luck getting things to work.

No need to drill a hole, look on the bottom of your supervisor, there are 3 phillip head screws to remove and the bottom plate pops off. I've now got a Raspberry Pi hooked into it and bridging to my wireless network. I've got new tables for this dump, but my calculations for power consumption are off sampling this every minute. I haven't had time to tweak them properly, so my "every 5 minutes" tcpdump/sniffing method is still the better one for the moment. Once I figure out my power consumption miscalculations, I do plan on switching to it, because it's much more real time view of the power production (approx. every minute vs 5 minutes). I will post back here once I've had time to mess with it and will update my github with the new tables/scripts and write a formal, easy to follow post.

P.S. This collaboration is great!

[JJNorcal]

ehampshire, I realize I can take the bottom off, but I don't want to leave it off full time. I might if it were my only way forward, but your packet capture method works just fine for fishing out the one and only parameter I'm looking for. Plus pvoutput will only accept pushes every 5 minutes, so I don't really anything to get per minute output. And no need to leave bottom off (or drill hole), and all of my networking stuff gets to stay together in a closet.

A lesser consideration is that I would want to remember to put the monitor back together should SP need to perform some service.

WRT pvoutput, I have been exchanging emails with BB and I'm about to cut over. I think my posts are right, but they are being rejected with "invalid time"; might be because I haven't disabled Efergy pushes and I'm testing with a 4 hour old power data point. Expect BB will set me straight tonight. The process is pretty straightforward once you focus on the Add Status Service. BB confirmed timestamps should conform to your locale, and that retries should come 2 seconds apart.

-JJ

[JJNorcal]

I'm all wired up at pvoutput now. No more daily pvoutput rule tweaking for me!

I was wrongly putting POST parameters in http body; needed to be in the URL.

Thanks again for sharing.

[DanKegel]

Neat! Are your sources online anywhere?

[JJNorcal]

I will post procedures and node app once I finish writing them up.

[JJNorcal]

I have posted source and documentation to http://git@gitlab.com:JJNorcal/SpPvoConnector.git. The readme essentially represents my notes for the project from creation of a pi network appliance through installation and configuration of the node app. The main.js file represents the node source file. The readme may be terse, it may even contain some errors. I'm new to gitlab and not all that savvy on solarpaneltalk, but hopefully you can PM me with questions somehow. Good luck!

[Mike90250]

I have posted source and documentation to http://git@gitlab.com:JJNorcal/SpPvoConnector.git . The readme essentially represents my notes for the project from creation of a pi network appliance through installation and configuration of the node app. The main.js file represents the node source file. The readme may be terse, it may even contain some errors. I'm new to gitlab and not all that savvy on solarpaneltalk, but hopefully you can PM me with questions somehow. Good luck!

PM's not allowed here, and that's a pretty strange URL you have going.

[CharlieEscCA]

PM's not allowed here, and that's a pretty strange URL you have going.

The URL was likely:

http://gitlab.com/JJNorcal/SpPvoConnector

[JJNorcal]

CharlieEscA reported the correct browser URL. The URL I posted was for use with git command: "git clone git@gitlab.com:JJNorcal/SpPvoConnector.git".

I'm reluctant to put my email out there, so I guess use this thread if anybody digs in and has questions. Remember that this approach is laser focused on automating power pushes to pvoutput. ehampshire is working on a more thorough approach.

I will be updating main.js with some enhancements to streamline and hopefully take care of the 3 or 4 zeros at pvoutput I saw yesterday. The fixdatapoint script provides an example how to correct a single datapoint at pvoutput.

[JJNorcal]

Thought I would report back on some lessons learned with mirroring/monitoring approach. I ran into an anomaly today where packets were simply not showing up. After some frustration, I discovered that the powerline adapter SP installed got into a situation where it was unable to maintain a reliable connection. This became clear once I logged onto my home router and discovered that SP monitor was obtaining a DHCP address every couple of minutes. What I didn't know is that SP hooked into my wifi AND installed powerline. I was mystified when I unplugged the powerline and yet SP site continued to register data. Powerline appears to have priority, but when there is an issue, wifi kicks in. I' moved my snooper so that it was between my wifi and my home router, and packets once again materialized.

Parsing TCP byte stream is a bit tricky. I have made several adjustments. It would be easier if the parser/tcpdump focused exclusively on the SP monitoring IP address(es), but I'm trying to stay more general.

[DanKegel]

I am not surprised they are using two different methods at once, since both are unreliable. Does the SP unit have two network interfaces, though, or are they doing something kludgy?

BTW did you actually push code to that repo? I don't see any commits at the web version of that project...?

[JJNorcal]

Yes, they have two different network interfaces (three, I think, if you count cellular which I'm not sure is active). And yes, I pushed the code. You prob need to create a gitlab account. It's free. Let me know if you have an account and can't access. I'm new to gitlab...

[JJNorcal]

Some final thoughts.

I started by capturing all http (not just SP) packets to avoid dealing with specific IPs. This worked fine over powerline with two port pi (built-in enet + USB adapter). I added second USB adapter once I discovered that I need to deal with wifi. At this point I ran into tcpdump buffering issues (due to increased http traffic) which I was unable to overcome, so I reserved two DHCP IPs for the two SP macs, and the three port pi switch (built-in enet connected to powerline, USB1 to wifi, and USB2 to my home router) is performing fine.

I'm getting 60 Mbps throughput through the pi 3 (I get 225 Mbps over wire). My 802.11n wifi is somewhat faster than that (I can get over 100Mpbs on a good day in some rooms), but I can't tell the difference and my family is blissfully unaware of the new bottleneck. That said, burning wifi bandwidth without cause goes against my nature, so I'm likely going to take the port mirroring approach at some point, maybe when I upgrade my AP. But I'm leaving this as is for now.

Would have been better if I pushed SP to install enet cable.

[DanKegel]

And yes, I pushed the code. You prob need to create a gitlab account. It's free. Let me know if you have an account and can't access. I'm new to gitlab...

I have an account, and can't access gitlab.com/JJNorcal/SpPvoConnector... well, I can, but all it shows me is that there is a project.
gitlab.com/JJNorcal/SpPvoConnector/settings/members lists you as the only person with access.

You may have to open up access or grant me membership...

[JJNorcal]

DanKegel, just now seeing this. I must have missed a solarpaneltalk email indicating thread update. Sorry.

I just now changed "View and edit files in this project" to "Everyone with access" and enabled "Allow users to request access". Hopefully you can find a way in now (the repo has always had Public visibility) . Perhaps you have another suggestion where I can post current snapshot of files, but note that I have been checking in updates into this repo.