Why it takes so long to roll out the 22S41/BECM OTA recall update

[Bonehead]

I'm planning a road trip next week and didn't want to wait any longer to get the update, so I had the 22S41 and BECM updates performed at the dealer today.

I asked why it's taking so long to roll it out OTA, and the service manager explained that it's due to a limitation of server capacity on Ford's end. I guess it stands to reason that since it takes around 30 minutes to install it would greatly impact how many cars could get updated at any one time.

But I would think that Ford is getting a rude awakening as to how much server capacity they need to have access to in the future. As more MME's are purchased, this problem will obviously only get exacerbated.

Also, if you plan on taking the car into a dealer to do this update - don't forget your fob like I did if you have any plans to use your phone while they're working on it! It took me around 2 hours total to get the job done, and I had to leave my phone with the car.

Sponsored

 

[COACH CHRIS]

While I'm not impressed with Ford's handling of OTA updates, I'm skeptical that a service manager would know the state of Ford's server setup enough to declare it the sole problem. At the same time, that Ford has bitten off more than it can chew on the software front makes sense: None of their customer-facing software functions smoothly.

At the end of the day, I can forgo OTAs not working so long as dealers are apply updates directly. Of course, even this is inconsistent, which is super frustrating, especially when considering Jim Farley's answer to most feature requests is "coming soon via OTA!" I'm not sure Ford's right hand knows what the left is doing. Bless the dealerships and service centers that try to explain to customers why it's all so broken.

 

[Spooky]

The Ford CEO has admitted in interviews their OTAs are not working as well as they want, so at least they know it is an issue. Perhaps the update to the Lightning interface (assuming it is coming to existing MMEs) and other changes this Autumn will bring great strides.

As for their servers not being robust enough….does the car really need to stay in contact with the server for 30 minutes during the update? That seems odd/dangerous considering dropped connections and the like. So it’d just be downloading the data to the car then the car’s cpu(s) perform the update, then it could check back in with the server. Though I suppose it could be more complicated than just a download then apply.

 

[Bonehead]

The Ford CEO has admitted in interviews their OTAs are not working as well as they want, so at least they know it is an issue. Perhaps the update to the Lightning interface (assuming it is coming to existing MMEs) and other changes this Autumn will bring great strides.

As for their servers not being robust enough….does the car really need to stay in contact with the server for 30 minutes during the update? That seems odd/dangerous considering dropped connections and the like. So it’d just be downloading the data to the car then the car’s cpu(s) perform the update, then it could check back in with the server. Though I suppose it could be more complicated than just a download then apply.

As far as need to stay in contact with the server - it may be the case. I get the feeling it's not one file, but rather multiple files that need to be installed in sequence. This was the second time I've had a dealer do an update, and both times they warned me that if there was any loss of connection or power failure during the process, they'd have to start over from the beginning.

 

[Spooky]

Yikes

 

[Logal727]

That’s not how any of this works.

 

[Logal727]

As for their servers not being robust enough….does the car really need to stay in contact with the server for 30 minutes during the update?

Not at all

 

[COACH CHRIS]

so at least they know it is an issue.

How could they not? Half this forum are posts bemoaning OTAs (myself included. )

What's crazy to me is that Ford embarked on this EV strategy and seemingly failed to figure this out. To up the stakes, they're now using the OTA mechanism to "fix" and complete recall work. Insane.

I love my MME, and I feel joy every time I drive it, but car's core experience relies on solid software, and Ford just ain't cutting it. I do suspect my next EV will not be an MME, though admittedly I don't feel there's much in the way of competition yet (at least at the price point, specs, etc. that I'd need.)

 

[DevSecOps]

This thread will be entertaining . It started with a mechanic/service advisor and it will continue with people who think that because they setup a Unifi AP they understand the way all this works.

 

[Spooky]

How could they not? Half this forum are posts bemoaning OTAs (myself included. )

What's crazy to me is that Ford embarked on this EV strategy and seemingly failed to figure this out. To up the stakes, they're now using the OTA mechanism to "fix" and complete recall work. Insane.

There are a lot of issues I’d like to think somebody at Ford is aware of, following these forums. The head of the company pointing to OTA as something they need to do better is a good sign. A lot of times pressure from the top down gets things rolling.

On Ford not handling OTAs well….how many traditional car manufacturers are? While not an excuse for them it is a bit much to expect a car manufacturer to go from charging hundreds of dollars every year for 1-2 year old map updates to being Apple or Microsoft in pushing out updates. Hopefully it gets a lot better sooner rather than later though.

 

[MightyMike]

I seriously doubt that Ford designed their upgrade process to be so slow that they cannot upgrade 50,000 clients because of server congestion. Just think about it for a minute. Let's say each upgrade file is 100MB (a mach-e is a collection of small computers, so in reality this is probably 30 or 40 different upgrade files for the different smart-remotes throughout the system).

50k downloads would put a load on the server of about 5 petabytes of data transfer. Seems like a lot, but as soon as you start spreading it over time (7 days) , it becomes not so bad. Averages around 8MB/second (continuous).

Now 8MB continuous is unreasonable. But if you spread this over 10 servers, or use a service like Akamai, it becomes ridiculously easy to distribute a firmware upgrade.

Much more likely Ford is being very conservative. Upgrading a few 100 at a time, making sure that they don't brick people's cars. They may upgrade 100, then wait a week, to make sure it takes affect. Then another 100, and then wait a week. Eventually they will build confidence and start distributing to 500 a week , or 1000 a week.

Even large computer companies like Apple and Microsoft do something similar. They do not blast it out to 100M computers all at once and risk bricking the entire world.

MIke

 

[SnBGC]

I seriously doubt that Ford designed their upgrade process to be so slow that they cannot upgrade 50,000 clients because of server congestion. Just think about it for a minute. Let's say each upgrade file is 100MB (a mach-e is a collection of small computers, so in reality this is probably 30 or 40 different upgrade files for the different smart-remotes throughout the system).

50k downloads would put a load on the server of about 5 petabytes of data transfer. Seems like a lot, but as soon as you start spreading it over time (7 days) , it becomes not so bad. Averages around 8MB/second (continuous).

Now 8MB continuous is unreasonable. But if you spread this over 10 servers, or use a service like Akamai, it becomes ridiculously easy to distribute a firmware upgrade.

Much more likely Ford is being very conservative. Upgrading a few 100 at a time, making sure that they don't brick people's cars. They may upgrade 100, then wait a week, to make sure it takes affect. Then another 100, and then wait a week. Eventually they will build confidence and start distributing to 500 a week , or 1000 a week.

Even large computer companies like Apple and Microsoft do something similar. They do not blast it out to 100M computers all at once and risk bricking the entire world.

MIke

The idea a customer vehicle could be bricked (rendered unusable) by an OTA is terrifying. Whatever system Ford is using should be designed so it would be impossible to brick a vehicle under any unexpected situation. Unless that assurance exists, then Ford should not be attempting OTAs IMO...

 

[MightyMike]

Every device that takes an OTA update (or even a non OTA update) has the possibility of being bricked. Or having settings change in ways that are unexpected. Or introducing defects.

It's the nature of software. This is especially true when there are a variety of configurations (job 1, job2 ,etc), and definitely true in Covid when multiple configurations of electronics are almost certainly present in various electronics components to work around shortages.

Ford has never done this before. At my company when we did this the first time it was terrifying. But you get better, and eventually your testing improves and you end up with a much better product that has safe and effective upgrade.

 

[NoTechi]

A new theory of mine is that the OTA updates need certain conditions, such as a certain combination of PowerUp, Module and Sync versions as well as hardware combinations. Also other factors like the country (I am from Germany) could be part of the prerequisites. If the combination you have in your car does not match the prerequisites for an update, you will not be included in those updates.

Here might be a process behind how more combinations are added to the compatibility list, e.g. by Ford running tests with these combinations and then releasing them gradually. This is complicated by combinations made manually by dealers who updated only one part (e.g., PowerUp only), putting the vehicle in a state that was not planned for.

At least this would give me some explanation why I just received one minor OTA update so far (regarding the ford pass app showing the car being in deep sleep while it is not). I received some updates in the shop but I have no clue which ones. I know I am still on Sync 2118 and I am guessing I am on PowerUp 2.3.0 due to horizontal volume level bar but I am still missing SoC within Apple maps and the games coming with it.

Long story short: I think they screwed up the updates in general because they underestimated or simply didn't take into account the number of possible combinations and are now having trouble getting this back under control.

NoTechi

PS
I don't believe the update servers are a problem since nowadays it is not that you run some servers in some basement anymore you pay for scalable servers from companies which are specialized on it like Amazon, Microsoft or Google. At least I hope thats the case ...

 

[RickMachE]

Every device that takes an OTA update (or even a non OTA update) has the possibility of being bricked. Or having settings change in ways that are unexpected. Or introducing defects.

It's the nature of software. This is especially true when there are a variety of configurations (job 1, job2 ,etc), and definitely true in Covid when multiple configurations of electronics are almost certainly present in various electronics components to work around shortages.

Ford has never done this before. At my company when we did this the first time it was terrifying. But you get better, and eventually your testing improves and you end up with a much better product that has safe and effective upgrade.

A/B....

Sponsored