Firmware Over-the-air (FOTA) Is Hard

ChasingCoral

Well-Known Member
First Name
Mark
Joined
Feb 3, 2020
Threads
376
Messages
12,404
Reaction score
24,521
Location
Maryland
Vehicles
GB E4X FE, Leaf, Tacoma, F-150 Lightning ordered
Occupation
Retired oceanographer
Country flag
As PowerUp 1.7 continues to roll out slowly and 1.8 has yet to be deployed to any car, let’s recall that Firmware over-the-air (FOTA) Is hard. I predict that a year from now Ford will have this down but for now we need to be patient so we get the upgrades we want rather than bricks.

From https://electrek.co/2021/10/01/over-the-air-updates-how-does-each-ev-automaker-compare/

Firmware over-the-air (FOTA) updates
While SOTA updates have quickly become commonplace, updates for FOTA are not as easily accomplished. EV pioneers like Tesla and NIO have been able to crack the code so to speak, but other legacy automakers aren’t so eager to adapt.

As a newer concept, the flow of firmware updates over the air requires advanced technology not all automakers possess yet. Cybersecurity, speedy connectivity, and massive computing power all must be present in an EV to receive FOTA updates, not to mention a helluva lot of memory.
Sponsored

 

JamieGeek

Well-Known Member
Joined
Dec 29, 2019
Threads
82
Messages
3,556
Reaction score
6,746
Location
Southeastern Michigan
Website
spareelectrons.wordpress.com
Vehicles
Mach-E, old: Bolt, C-Max Energi, Focus Electric
Country flag
Interesting in that the article makes a distinction between software and firmware.

These days they are so close to each other to be indistinguishable.

(Not to mention the fact that the article doesn't really define what is software and what is firmware.)
 

newtruckinfl

Active Member
First Name
Dave
Joined
Sep 15, 2021
Threads
0
Messages
25
Reaction score
22
Location
Orlando
Vehicles
2021 F-150 Limited
Occupation
Real Estate
Country flag
I'm sure Ford took that into consideration when promising OTA capability and excited to see legacy automakers coming up to speed!
 


COACH CHRIS

Well-Known Member
First Name
C
Joined
Aug 12, 2021
Threads
16
Messages
201
Reaction score
256
Location
NJ
Vehicles
Mustang Mach-E AWD Select
Country flag
Software development is hard. And playing in the highly regulated automotive space is hard. Not to mention the level of safety requirements that need to be met both in the code and on the road.

And yet..

The only reason we have expectations that OTAs will be common and the app will work reliably is because Ford said they would. PAAK is bad. OTAs are sporadic and non-uniform. As a user, these things are fundamental to the ownership experience (for example, lots of us only got one key on the premise that PAAK would work. When it doesn't, and Ford won't provide a second key, what are we supposed to expect from the company?)

Also, to the point of Firmware vs. Software... that's not for customers to care about. They shouldn't have to, and car companies shouldn't be betting that customer sympathies will be higher for one type of update vs. another. It's Ford's job to figure this out, and communicate expectations that it can meet.

Finally (?), I'm super OK with waiting on these updates, and very happy to give Ford the runway to build those out. Of course, that's my personal preference - some folks want practically-Beta software asap. But it seems like there's a program for that - Ford should use it!

OK, stepping off the soapbox now. ✌?
 
OP
OP
ChasingCoral

ChasingCoral

Well-Known Member
First Name
Mark
Joined
Feb 3, 2020
Threads
376
Messages
12,404
Reaction score
24,521
Location
Maryland
Vehicles
GB E4X FE, Leaf, Tacoma, F-150 Lightning ordered
Occupation
Retired oceanographer
Country flag

jdmrc93

Well-Known Member
First Name
Joe
Joined
Jun 25, 2020
Threads
20
Messages
345
Reaction score
874
Location
Allen Park, MI, USA
Vehicles
2021 Mustang Mach-E First Edition in Grabber Blue
Occupation
IT Network Engineer
Country flag
I get that - but they could be a tad more transparent. Their website still says Q3 2021 for BlueCruise OTA availability. Last I checked, we're in Q4. A simple email saying "hey all we need some more time on this" would go a long way.
 

COACH CHRIS

Well-Known Member
First Name
C
Joined
Aug 12, 2021
Threads
16
Messages
201
Reaction score
256
Location
NJ
Vehicles
Mustang Mach-E AWD Select
Country flag
When did Ford promise the frequency of OTAs?
I read “regular” (as in “regular vehicle enhancements”) as “frequent”, or at the very least, well, regularly. There doesn’t seem to be a set cadence for Sync or FordPass updates.

Fair, but also some folks outside the program have gotten the update.

Regardless of your position on whether or not these dev practices are good (for Devs or customers or even Ford), as a Mach-E customer, I’m struggling to understand what I can expect from Ford, and when I can expect it by - if anything at all.
 

breeves002

Well-Known Member
First Name
Sam
Joined
Feb 21, 2021
Threads
90
Messages
1,731
Reaction score
3,548
Location
St. Louis, MO
Vehicles
2021 Mach-E GT PE
Country flag
This is the way I understand how Ford does things after 3+ years of working with doing software/firmware updates along with using unsupported firmwares, etc on various modules. My business revolves around this.

I don't think it is as hard as that article leads us to believe...I think Ford is just being extra safe to make sure they have all of their bases covered to avoid failed updates causing people to freak out or go to dealers prematurely. The gateway module acts like FDRS and a VCM. It downloads the update files and flashes them. I'm not saying this is simple, but I'm saying they're being EXTRA conservative.

The way Ford does modules you get the firmware and software in the same package in *most* modules. The only exception I know of for a fact is the APIM. It has a separate firmware and software package.

When a Ford module is updated it does the secondary boot loader first, followed by firmware and then software files. When a PMI is done after an update it configures the as built data, VIN, etc - which is generally stored separately with other configuration information in an EEPROM on the module.

For erase and replace modules if the update fails mid update it will be bricked....temporarily. All you have to do is do the firmware/software flash again and complete it successfully. If you put the wrong files on you can simply re-flash. The main boot loader is designed to allow you to always be able to flash new software/firmware even if it is corrupted.

For ABA swap modules if it fails it will not even try to boot that firmware and the module will still work correctly. These are not risky to try and update OTA since a failed flash wouldn't cause a problem. Both firmware AND software are on separate ROMs and can independently run the module.
 

Timelessblur

Well-Known Member
First Name
Tyler
Joined
Oct 6, 2020
Threads
1
Messages
392
Reaction score
448
Location
Austin
Vehicles
Mach E AWD Premium Extend Range RR
Occupation
Software Developer
Country flag
The issue is not that OTA is hard. It is the fact that ford is radio silent on the issue. Zero communication. They also have miss multiple deadlines they originally announced and us early adaptors are feeling really screwed.
 

Mirak

Well-Known Member
Joined
Oct 8, 2020
Threads
109
Messages
3,662
Reaction score
5,840
Location
Kansas
Vehicles
"Sonic" 2021 MME Grabber Blue First Edition
Country flag
When did Ford promise the frequency of OTAs?
Ok. Let’s start here.




And then I’ll give you two (or three) more words: “Q2 2021”

I’m sure there are more examples if I cared to dig back into the fawning puff pieces early on. But we all know Ford oversold this and has thus far WAY underdelivered.
 

Jimrpa

Well-Known Member
First Name
Jim
Joined
Sep 10, 2020
Threads
230
Messages
7,011
Reaction score
9,299
Location
Wayne, PA
Vehicles
2021 Infinite Blue Premium Mustang Mach E ER AWD
Occupation
Retied (formerly tried to herd highly technical, independent cats)
Country flag
Absolutely firmware over the air updates are VERY difficult. In a past life, I worked in an industry where we did firmware over the air updates that were, for all practical purposes, impossible to recover from, if they were very badly screwed up. The system was designed with a lot of redundancies and failsafes, and lots of testing of the updates were performed, to ensure that no update ever caused serious issues. Those solutions would be cost-prohibitive for the auto industry. From the descriptions, it sounds like ford had adopted some leading practices, such as A-B-A banks.
I think what would be reasonable would be the following:
  • More transparency into update status and progress, along with a clearer user-facing versioning system. I get that the different subsystems in the vehicle are at different version levels, but I find it hard to believe that there aren’t standard “collections” of versions. The apparent PowerUp scheme is definitely not intuitive to the casual user.
  • There is nothing wrong with ford saying, through FordPass, email, smoke signals from Dearborn, carrier pigeon, whatever, something like “Starting November 1st through December 15, we will be deploying PowerUp 1.9. Watch for it in your system update. If you do not receive PowerUp 1.9 by January 10, please contact a ford guide through FordPass for assistance.”
  • Once an update begins, system update should say something like “system update in progress. The update is 30% complete. You may continue to drive your car as usual. You will receive a notification on your Sync4A screen and through FordPass when the update is complete.”
  • Finally, and this is a hardware issue: we are in the 21st century. From what I’ve read, my understanding is that the internal bus in the car runs at a data rate of 512 Kbps. Seriously? Someone needs to let the genius who made that decision know that we can do a bit better on data rates now ?
 

BMT1071

Well-Known Member
First Name
Mike
Joined
Mar 21, 2021
Threads
61
Messages
3,078
Reaction score
4,223
Location
Glendale, AZ
Vehicles
21 MME SR RWD, 23 MME GTPE
Occupation
Machine Control Specialist
Country flag
  • Finally, and this is a hardware issue: we are in the 21st century. From what I’ve read, my understanding is that the internal bus in the car runs at a data rate of 512 Kbps. Seriously? Someone needs to let the genius who made that decision know that we can do a bit better on data rates now ?
I believe that to be the Ford VCM speed, not the in vehicle bus.
Sponsored

 
 




Top