Service Issue on 11/24 - Can not view live stream outside of local network


#1

Update: 11/29 1:00AM PST - After few days of monitoring our metrics, we are confident calling the issue resolved. Both our CS team and internal metrics show that we are back to normal business as of 11/28.

Root cause of the issue: Our service provider made a service upgrade intended to better support the US market. However, during the deployment there was a bug introduced into the system which caused camera login failures for Wyze Cams. That caused connection outside the home network to fail since the device showed as offline for streaming. Around midnight on 11/26, the issue was identified and fixed. However a device restart needs to happen for each camera to re-login. Given how many customers we have, it took about 2+ days for most cameras to be back online.

Again, we are very sorry for the service disruption! Thanks for your patience and continuous support. We have learned a lot from the two service issues last week and that’s definitely not where we want to set the bar. We are determined to give our customers a much better experience and we sincerely hope we can use our actions to earn your trust again.

If you are still having trouble connecting, this may be due to another issue and we’d love to work with you on it. In that case, please send us a PM or fill out this form to contact customer support:

https://support.wyzecam.com/hc/en-us/requests/new

Best,

Wyze

History:

Update 11/26 6:27PM PST - Still under investigation but there is improvement.

Many people are saying that their connections are working better but while the situation is improving our connection percentage isn’t quite to where it usually is. We’re still looking into this. If a reboot hasn’t resolved this problem, please let us know!

If you aren’t near your Wyze Cam, you may be able to use the in-app restart option. You can find this feature in Camera Settings.

Update: 11/26 8:25AM PST - Provider updated the issue resolved.

We received an update from our provider that they have identified the root cause and fixed the server issue on their side. You should NOT need to use our temporary solution anymore. If your device is not working, you need to either power cycle your camera or go to Camera Settings -> Restart Device. We would like to verify from our customers that the solution works now before we say the issue is resolved.

Update: 1:32PM PST - Temporary Solution.

We are still working on the root issue. Before we are able to get to the bottom of it, we are offering a temporary solution.

You can use this weblink https://user.wyzecam.com/upgradedevices to upgrade your camera firmware. (after log-in, click on user information -> upgrade devices). A manual firmware flash won’t work for this but the weblink should. We have a couple of other versions of the link if there ends up being a redirect issue or something but this is the most mobile friendly version. If you cannot see the menu, please try requesting the desktop version on your phone, using your phone in landscape mode, or switching to another browser (Chrome seems to be working pretty well for this but we’ve heard some reports of Safari having trouble).

Please note that this is a temporary solution. We are committed to find the root cause and implement a long-term solution. So, if you are not willing to use this temporary solution, we appreciate your patience while we working to fully resolve the issue.

For details about this temporary solution, please see below.


Our P2P (peer-to-peer) service provider uses global load balancing to optimize live stream connection reliability and speed.

After we launched Wyze Cam, we heard concerns from our community about traffic going outside of the United States due to the global load balancing. We then implemented “Server Restriction” to limit traffic to only US-based servers. Although we don’t think the server restriction is the root cause, it is related to this issue.
By removing the server restriction, a user should be able to view the live stream outside of their local network.

What does this mean?

  1. For the Live Stream, the P2P servers are still North America Only. So the live streaming video still stays inside North America.

The change is that in order to make device registration successful, a global list of servers (not only US-based) will be pinged, and the fastest server will respond for device registration. Before making this change, it would only use US servers.

  1. For Alert Videos there is NO CHANGE. Your alert video will still go to US-based AWS servers (Owned by Wyze) with the same encryption and security level.

Starting 11/24, we have noticed that some cameras can not be viewed live stream, outside of local network. The symptom is close to 11/19 issue, but the root cause has not been determined.
We are working on this issue, and will provide update as it progresses.

Sorry for the inconvenience.


V2 cam error 90
Cannot access when on external network
WyzeCam Connectivity Question - Remote Wifi and/or Cellular Connections
Live feed not working on over 1/2 my cameras
Pam Cam Flashing Blue Light
EDITED WITH UPDATE: Service issue on 11/19 - Cannot see live stream outside local network
Pan Remote Connectivity Issues
Off line security cameras on vacation house
#2

#3

You can also check here for updates:


#4

update at 9:00 AM PST, We are still working on the issue.


#5

Cant even view ON the local network. And no…not a router issue. 2 of 4 of my cameras work fine. Most of the time…


#6

Update at 11:55AM PST. We’re still working on the issue.


#7

Update: 1:32PM PST - Temporary Solution.

We are still working on the root issue. Before we are able to get to the bottom of it, we are offering a temporary solution.

You can use this weblink https://user.wyzecam.com/upgradedevices to upgrade your camera firmware. (after log-in, click on user information -> upgrade devices).

Please note that this is a temporary solution. We are committed to find the root cause and implement a long-term solution. So, if you are not willing to use this temporary solution, we appreciate your patience while we working to fully resolve the issue.

For details about this temporary solution, please see below.


Our P2P (peer-to-peer) service provider uses global load balancing to optimize live stream connection reliability and speed.

After we launched Wyze Cam, we heard concerns from our community about traffic going outside of the United States due to the global load balancing. We then implemented “Server Restriction” to limit traffic to only US-based servers. Although we don’t think the server restriction is the root cause, it is related to this issue.
By removing the server restriction, a user should be able to view the live stream outside of their local network.

What does this mean?

  1. For the Live Stream, the P2P servers are still North America Only. So the live streaming video still stays inside North America.

The change is that in order to make device registration successful, a global list of servers (not only US-based) will be pinged, and the fastest server will respond for device registration. Before making this change, it would only use US servers.

  1. For Alert Videos there is NO CHANGE. Your alert video will still go to US-based AWS servers (Owned by Wyze) with the same encryption and security level.

Global load balancing - what was wrong with this?
#8

Why am I asked if I want to leave Facebook when I press the link for the temporary upgrade?!


#9

… Because I copied the link from the Facebook post. :sweat_smile:


#10

Okay, I edited that part. Can you check to see if the Facebook thing is gone now? :slight_smile:


#11

Yes. Facebook is gone now.
When I log in and select my device to upgrade nothing happens. Says camera is offline? Am I not able to upgrade remotely?


#12

Thanks for the report!

Are you near your Wyze Cam or close enough you could check the status light (I know you said remote)? It may be offline due to something else and that would prevent this.


#13

Unfortunately I’m 2400 miles away.


#14

Well, crud. Um… Try again later, I guess? Sorry I don’t have a better solution for you. You SHOULD be able to do it remotely. :frowning:


#15

We’re so sorry about the inconvenience! If you don’t have access to you camera when it’s offline, unfortunately you have to wait for the solution. But we’re doing everything we can do fix the issue. We’ll keep you updated!


#16

I have 2 locations (Home and Lake House) that are experiencing this issue. Notifications were still pushing to the app, but could not view live. I was able to remotely update each camera and all are back online now.


#17

1 of my cameras did not come back online at all and has a solid yellow light. Tried power cycling and then the 10 second reset. It does not reset and the yellow light stays solid? Did these issues damage the camera?


#18

We’re sorry about that. Would you like the instructions for flashing firmware to see if you can bring it back? It’s completely optional but often works. As long as you’re in the US (due to our shipping limitations), we’d be happy to replace it for you if you can’t get it working again. We sometimes see solid, yellow lights after firmware upgrades so while the general server issue probably wouldn’t cause this (to my knowledge), it still may have happened due to this situation. We apologize.


#19

Yes, how do I flash the firmware?


#20

And on other 3 cameras cannot live view outside wifi. I tried the firmware link in your previous post. However it does not recognize my log in credentials. Just nothing working!