Failed to update device list (error code 3044)

Hello all! I am the maintainer of the Home Assistant integration. I can confirm that my personal system is back online (however I do not know if it will stay that way).

Later today I am meeting with a Security Architect from Wyze to discuss the API. I am optimistic about us finding a better long term solution between Wyze and this wonderful open-source community that has joined to offer extensions to their offerings.

10 Likes

So far Wyze hasn’t been forthcoming about what activity they were attempting to prevent. Some people seem to think it’s about cameras and camera feed streaming but plenty of people just wanting to control their smart bulbs & plugs through their own automations were affected. If you figure it out during your call, please share what activity they were attempting to block.

3 Likes

@mrvcdx @krayj

Just to be clear, we aren’t looking to block folks from using HA at all - you guys know us better than you might be trusting. We’re all for DIY and making our devices as accessible as possible, and we know in the absence of an official API at the moment that the open source community will find a way. We definitely aren’t strategizing ways to prevent people from using the devices they own in the ways that they want - otherwise TinyCam (and more recently Lumia Stream) wouldn’t have had longstanding Wyze device support. I’ve been a bit surprised by some of the conclusions being drawn in some of the responses in this thread haha.

Our intention is simply just to make sure that when these unauthorized and unofficial third-party extensions are in use that they don’t have an effect on the services for the rest of our users. This HA extension being used by a few thousand people has caused almost as much stress on some of our servers than the other 5 million+ users combined, and this load causes app and device issues across the board for all of our users that will only get worse as more people start using it. As far as error 3044 goes, we must have some kind of rate limiting in place to prevent things like this happening. Some users hit our servers with over a million requests in a single day, which is astronomical.

While I have a bit of a programming background, I think mulliken and our developers will be a better point of contact to share more specific insight/specifics after meeting with our team today. I just ask that you be patient with us and let them discuss how we can move forward. :slight_smile:

10 Likes

Thanks for the response, I appreciate it.

I’ve been a bit surprised by some of the conclusions being drawn in some of the responses in this thread haha.

There wasn’t really much for people to draw from as details about what exactly was going on was scant at best, so drawing their own conclusions was pretty much all there was left.

1 Like

Nice to see a response. Hopefully Wyze will allow for an integration that allows a good solution for Wyze and us users. Seems to me that a local API would be best to simply avoid any load on your servers. If it’s only a few thousand of us, should cause much of a worry about potential lost revenue!

Btw, would be nice if you guys would tackle rtsp on v3 while your working with us…that would really.make this whole situation a win for the community

We’ve been workin’ towards RTSP since we publicly promised it. Swear! This question has come up quite a bit in our AMAs, but basically from what I hear we’re hoping to have an update of some kind by the end of the year.

3 Likes

Thanks for the updates here and thank you as well to @mulliken for your work on the HA integration. Hopefully your meeting yields some positive results!

Updates from my conversation with Wyze

TLDR: Some functionality has to go away (motion sensors, contact sensors, and camera motion detection). We account for over 50% of their traffic while being a very small minority of their userbase. I would like to request some help fixing a bug with the auth endpoint → contact me here if you can help: joshua@mulliken.net

My conversation with the people at Wyze was very productive. We are planning a way forward to continue the integration with HA while being good citizens of their platform.

Based on the information I have from this repo and the limited information they were able to share, this integration is used by between 1,000-10,000 individual HA instances. This number is increasing steadily; however, it is still a tiny fraction of the multiple millions of people who use their services. Even though we are such a small percentage of their userbase, we account for over 50% of the traffic to their servers. This amounts to a tangible cost (monetarily for Wyze) and usability effect on the rest of their users.

Issues that need to be resolved

Point 1: Motion Sensors, Contact Sensors, and Camera Motion Detection

They want me to disable the binary sensor devices as these are absolutely pummeling their servers. I will release an update tonight that removes them entirely until we can find another way to implement them. If we don’t all upgrade to that version and stop requesting from their API for those devices, they have said they are considering shutting down all access from Home Assistant.

Point 2: Requests to the Auth Endpoint

They have also informed me that we have a bug that is causing requests to their authorization endpoint way more than is necessary (1 for every request) which I will be investigating ← and would appreciate any debugging assistance on this point from the community

Updates on Public APIs/Legit Home Assistant Integration

Some employees of Wyze use the integration :smile:!! They definitely want it to keep working! I asked about APIs or methods that I could use to reduce the load on their servers while still maintaining all functionality, and they were unable to speak to their future roadmap. They offered me an NDA, but since I am speaking to you now, it should be obvious that I declined.

They have promised to share information on how to implement the 2FA in line with their requirements going forward and did say that they will be requiring 2FA for all logins at some point in the near future. I will be working with them to ensure that this is working in the integration before that requirement is in place.

It is unclear when or if the devices that require up-to-date state information (motion sensors, contact sensors, and camera motion detection) will be available again.

For any additional discussion surrounding this issue I have created a Github Discussion forum here: Wyze Server Changes ¡ Discussion #232 ¡ JoshuaMulliken/ha-wyzeapi ¡ GitHub

5 Likes

Had you guys given any dialog or warning at all about the sudden decision to rate-limit and IP-block home-assistant integration users, this would have gone over much smoother. With no official warning or dialog from Wyze ahead of time, all we’re left with are our assumptions. You guys kind of did things in reverse order that logic would have dictated.

1 Like

Thanks, Jimmy! Really appreciate your transparency and support of the community. I also totally understand that when you guys discover a problem (e.g. 50% of your backend serving <<1% of your users) you need to move fast to remediate the problem. Communicating out the change can understandably come later. While it caused some of us a minor inconvenience, that’s life when using an unofficial, unsupported API. I’m thankful that you and Wyze are working with Josh to come up with a satisfactory solution for everyone.

1 Like

I appreciate the work but after being told that for years at this point (RTSP) I have to get off that ride and get something that supports things outright. As far as the Wyze Sense stuff goes I was hit by the battery dying issue that killed 3 or 4 of the 10 I have. And now losing motion sensing being used with the cameras it’s going to be a hard nope from me at this point. I get it on the API traffic being a problem but instead approaching it with the community (and you knew what community was using it) you instead not only broke the integration but you caused a major inconvenience to those customers who had to flock to the forums to figure out whats up with the supported app as well. Keep in mind most of this could have been avoided by allowing simple local control/access to the devices as well as so many other products offer or a paid api tier. Especially for lights and the like.

Thank you for taking the time to address it but this is end of the line for me.

2 Likes

@all please update to the latest version of HA-WyzeAPI. I will reiterate that if a significant number of users continue to use the previous version Wyze has said that they are likely to block access from ALL Home Assistant users.

Please respect the desires of Wyze in this regard and be good citizens of the platform.

4 Likes

Updated… WYZE Bulbs and plugs work again. (I Don’t use Wyze contact or Wyze Motion sensors with home assistant.)

Thanks

Chas

It’d be worth posting over in https://community.home-assistant.io/ if you haven’t already

I couldn’t disagree more … Wyze handled this issue VERY BADLY.

Well this really sucks. I just moved over to the HMS from Ring and I’m seriously considering moving back. At least with Ring I can have all the doors and motion sensors integrated. This needs to be resolved. We need local communication.

Oh please. If this was “VERY BAD”, what would banning us all and keeping us banned be? This could’ve gone wayyyy worse for us. They’re bending over backwards to support a very tiny fraction of their user base. And a tiny fraction that has been doubling their server costs for a year or more.

Our shit was broken for like 36 hours, during which time they communicated to the community that they were going to fix it AND they directly met with the lead developer and worked out a compliance path for the unsupported, unofficial, and unauthorized API. Then they unblocked everyone, despite the API not even being updated yet, let alone rolled out to a large fraction of users. Was this handled perfectly? Of course not. But it seems pretty damn far from handling this “VERY BADLY” to me.

Moreover, being home assistant users, we should all be used to dealing with unreliable software and unexpected reversions in functionality. There’s literally a long list of “breaking changes” with every HA release. If you wanted a perfectly stable, well supported solution that you never need to think about, replace all your stuff with Philips Hue or something.

We should REWARD developers who work with us, not huff at them even once we get what we want. Will I think twice about buying more server-locked Wyze devices? Sure. Do I have an overall positive impression of Wyze, and would I consider buying more of their non-server-IoT products? Definitely.

5 Likes

@WyzeJimmy

I am the actual author of GitHub - shauntarves/wyze-sdk: A modern Python client for controlling Wyze devices. and the person who spent months of effort reverse-engineering Wyze’s closed-off API and the complex request signing algorithms in use for controlling the “newer” devices (door locks, vacuums, etc.).

I was originally developing this code alongside Josh Mulliken because I wanted support for the Wyze robot vacuum in the Home Assistant plugin, and there was no ability in his code to work with any of the newer endpoints/devices.

After providing him access to my then-private repository where I was developing this code, when it came time to integrate it to the Home Assistant, he told me he wanted to “go in a different direction.” Once my code was finished, right before I open-sourced it, I again chatted with him about integrating all of the API support I had built and was again rebuked, but he assured me he would “properly attribute any code that comes form you or your projects.”

The ha-wyzeapi project that is now in use leverages his “private” (i.e., not published on GitHub) wyzeapy project from pypi.com. A simple download of this shows that he pretty much ripped off all of the work I had done and is now GETTING PAID by people for this library AND interfacing with WyzeLabs as if it was his intellectual property.

If you would like to converse about anything I can add to the wyze-sdk I built to help with your rate-limiting efforts, please reach out to me here or on github, or even by email.

3 Likes

If it’s been going on for so long, why wouldn’t they ever have opened a dialog about it? Why would they not give any warning ahead of time? Why go to all that effort to disrupt the operations of what is arguably their most engaged and heaviest users (biggest customers)? And on top of that, they didn’t just cut off access to the home assistant integration, they broke their own app for those users in the process.

Obviously something needed to change and the community would have been eager to work proactively with Wyze ahead of time. Instead, Wyze escalated straight to thermonuclear first strike. That’s not anything they deserve praise or apologist support for.

1 Like

This is the bed that they made. Theres been a request for an open API or local control for years now. We will reward them when they actually work with us.

2 Likes