machefan

Well-Known Member
Joined
Jan 14, 2020
Threads
25
Messages
1,709
Reaction score
2,433
Location
NJ
Vehicles
RR FE 21', Leaf 18' and Bronco BadLands 2022
Occupation
IT
Country flag
I can answer this two ways. We track down to the last nut every part on the vehicle as it left the factory. Ford, GM, and Toyota do as well and I assume at least every legacy OEM does as well. We're on the hook for recalls and being able to trace everything via our supply chains is critical to ensure we cover every possible affected by a bad part (and to minimize costs when recalls do occur).

As for the software, we also know exactly what software revision and hash check is on every ECU when it left the factory AND when ECUs are updated/swapped out at the dealer. Again, recalls. At least for FCA, the vehicle also reports in on every key on to check for updates to ensure it has the latest updates for everything.

Ford looks to be using Wind River as their OTA provider which is different that the one we use so I am not sure how they deal with updates. I imaging there are both check for update as well as query capabilities. It just makes sense when you need to manage a ton of devices, which these things are (2 ton devices capable of going 100+mph but devices nonetheless...).
Sure, I have been back to the dealer twice. Each time I here from them and others Ford tracks alerts, haven’t seen Ford get involved to help the dealer resolve my issues. Dealer wants me to return Monday so they can start the process over to engage an engineer to come down to fix the issues. So Monday they will start the process over again which means I’m waiting for an extended period. So much for QC checks for a month or so. Everyone here has been wonderful but when I hear tracking down to the nut and nothing about glitchy software I get worried. I rather have a miss aligned door over a unknown software / module issue. Miss aligned the dealer can fix, software they can’t so easily and it appears OTA is like updating my IPad but has zero impact on the firmware around it.

49CE0C59-83DC-4061-AFBD-40BB5343EE77.jpeg


BCDC5A0C-80C9-4391-9E9C-8E56AFFB34A3.png


FCFCE22C-CAEC-40E0-9BB9-D830123A080B.jpeg
 
OP
OP
ARK

ARK

Well-Known Member
Joined
Aug 21, 2020
Threads
42
Messages
2,732
Reaction score
3,986
Location
Los Angeles
Vehicles
Mustang Mach E
Country flag
Went to my local dealer today (which is not where I bought the car) - that are working on Monday (the holiday) - so made a 9 AM appointment - service writer was not sure how long the updates will take - as he noted "your's the first, so will know after we do yours." I'll post when done.
I dropped mine off at the end of the day yesterday and noticed they had plugged it in to charge as of early this afternoon. I picked it up right before they closed today.

So it doesn’t seem to take long at all, probably them going through their backlog of other Fords in for service is the longest part of the wait if I were to bet.
 

wmaney

Well-Known Member
First Name
Wade
Joined
Oct 12, 2020
Threads
44
Messages
271
Reaction score
398
Location
California
Vehicles
Mach-e GTPE, daughter drives Ford Focus Titanium
Country flag
Ark, did they say what was "fixed or changed." Or did you notice any differences in the car? Thanks
 
OP
OP
ARK

ARK

Well-Known Member
Joined
Aug 21, 2020
Threads
42
Messages
2,732
Reaction score
3,986
Location
Los Angeles
Vehicles
Mustang Mach E
Country flag
Ark, did they say what was "fixed or changed." Or did you notice any differences in the car? Thanks
My contact person wasn’t there, only a clerk at the service desk, so I am not certain. However, from that bulletin I posted yesterday, the issue seemed to be remote start not working after charging the vehicle.

Because I had never tried this before taking my MME in for service, I don’t know if it ever was an issue on my Mach-E. I otherwise noticed no differences this evening when I drove it.

Reposting the bulletin:

Ford Mustang Mach-E FSA Field Service Action No. 21B02 (Recall?) -- "Software Calibration Update" 65E10B73-9B6F-4546-A54C-ED791D6227B0
 

Shayne

Well-Known Member
Joined
Aug 9, 2020
Threads
18
Messages
3,341
Reaction score
2,489
Location
Northern Ontario Canada
Vehicles
2021 MME4x Prem
Occupation
Retired
Country flag
Sure, I have been back to the dealer twice. Each time I here from them and others Ford tracks alerts, haven’t seen Ford get involved to help the dealer resolve my issues. Dealer wants me to return Monday so they can start the process over to engage an engineer to come down to fix the issues. So Monday they will start the process over again which means I’m waiting for an extended period. So much for QC checks for a month or so. Everyone here has been wonderful but when I hear tracking down to the nut and nothing about glitchy software I get worried. I rather have a miss aligned door over a unknown software / module issue. Miss aligned the dealer can fix, software they can’t so easily and it appears OTA is like updating my IPad but has zero impact on the firmware around it.
My dealer is also not getting too much support. Think mine slipped by the speedway. Sounds like I should wait until there are more bulletins out by ford. Did you have the display warning and charging port light color at home or did they just happen when in at the dealer? Thing drives great but a couple of gremlins to scare out I think. Once you get a stable bios revision you normally do not need to update it like forever.

I got a big black box with a god awful orange border around it at 30% battery stating low battery. Too soon and too obtrusive for me. Electrical tape will not work it is too thin we will need duct tape to solve that bug. ;)
 


timbop

Well-Known Member
First Name
Tim
Joined
Jan 3, 2020
Threads
63
Messages
6,729
Reaction score
13,758
Location
New Jersey
Vehicles
Solar powered 2021 MME ER RWD & 2022 Corsair PHEV
Occupation
Software Engineer
Country flag
Miss aligned the dealer can fix, software they can’t so easily and it appears OTA is like updating my IPad but has zero impact on the firmware around it.
** I've been accused of being a rude jerk but what follows is NOT intended to be a dig but a perspective from 35 years in the software business **

There are a lot of assumptions in this and other threads, and the above just happens to be one from someone who's frustrated and worried things will "never be right" - I am quoting this post simply as an example. I get why he feels that way, and in time that feeling will go away. I also understand the emotion behind "well they should have".

I myself often think "well that's dumb, why'd they do it that way" - until I learn the nitty-gritty details, and then realize "ohhh, well that makes sense then". It's sometimes hard to suppress that urge to think "well they should", but you have to realize your understanding of a system as complex as a car, operating system, etc is far too incomplete to properly appreciate why things are done a certain way. One needs to recognize that people with a specific expertise are working on said items, and that they are in fact highly competent in their field.

The fact that Ford is having techs do the aforementioned update does not necessarily mean the process doesn't work or isn't capable of doing the update. Just maybe @RonTCat is correct that it is more a case of risk/reward. Since the MME is a brand new vehicle using a brand new process with brand new software, Ford may have just decided that having techs with specialized tools and techniques do the update is better than the very unlikely but very-embarassing-if-it-happens chance that the cars might become inoperable. That would be a huge PR nightmare and could prove fatal to the reputation of the car. Ford just may not be willing to risk it in their first OTA, even though all their testing has shown the process to work just fine.

Fundamentally computers do EXACTLY what the programming tells them to do, and sometimes programmers make simple mistakes. Sometimes that mistake is a typo (simple punctuation can completely change what a line of code does), and sometimes different pieces of seemingly unrelated code can interact in unexpected ways. There are likely tens of millions of lines of code in the MME, and each of those is an opportunity for a mistake. Sometimes those errors only manifest themselves once every hundred thousand times so it takes longer to realize they are there, and often it is really difficult to find the error.

That's why they've been testing the car for over a year and yet some things aren't quite right. It isn't that Ford is incompetent, EVERY computerized system faces the same challenges. It is impractical to test absolutely every possible circumstance, and the problem with android phones is a perfect example. Google made a change recently in their software that broke android auto - there's no way Ford could have known that was going to happen, nor could they have tested that change last july because the change hadn't happened yet.

The good news is that nothing is permanent with software. It is fundamentally malleable, and absolutely anything can be changed (although it may take a lot of time). What is much harder to change is hardware, and the good news is that the hardware appears to be thoroughly well tested and sound. Sure there may be individual instances of misassembly, but that kind of thing can get adjusted in the field.

What matters is that the hardware is fundamentally sound. Software fixes will continue over the life of the car.
 
Last edited:

wmaney

Well-Known Member
First Name
Wade
Joined
Oct 12, 2020
Threads
44
Messages
271
Reaction score
398
Location
California
Vehicles
Mach-e GTPE, daughter drives Ford Focus Titanium
Country flag
My contact person wasn’t there, only a clerk at the service desk, so I am not certain. However, from that bulletin I posted yesterday, the issue seemed to be remote start not working after charging the vehicle.

Because I had never tried this before taking my MME in for service, I don’t know if it ever was an issue on my Mach-E. I otherwise noticed no differences this evening when I drove it.

Reposting the bulletin:

Ford Mustang Mach-E FSA Field Service Action No. 21B02 (Recall?) -- "Software Calibration Update" 65E10B73-9B6F-4546-A54C-ED791D6227B0
Thanks ARK. I too had not tried the remote start. I will try after the update.
 

TheSteelRider

Well-Known Member
Joined
Dec 31, 2019
Threads
56
Messages
1,135
Reaction score
2,007
Location
United States
Vehicles
'21 Premium RWD
Country flag
I myself often think "well that's dumb, why'd they do it that way" - until I learn the nitty-gritty details, and then realize "ohhh, well that makes sense then". It's sometimes hard to suppress that urge to think "well they should", but you have to realize your understanding of a system as complex as a car, operating system, etc is far too incomplete to properly appreciate why things are done a certain way. One needs to recognize that people with a specific expertise are working on said items, and that they are in fact highly competent in their field.

...

Fundamentally computers do EXACTLY what the programming tells them to do, and sometimes programmers make simple mistakes. Sometimes that mistake is a typo (simple punctuation can completely change what a line of code does), and sometimes different pieces of seemingly unrelated code can interact in unexpected ways. There are likely tens of millions of lines of code in the MME, and each of those is an opportunity for a mistake. Sometimes those errors only manifest themselves once every hundred thousand times so it takes longer to realize they are there, and often it is really difficult to find the error.
These points well spoken. I've also been "in the biz" for little over 2 decades and concur. I will add on other bit of color and reality:

Software engineers write software. Business people run businesses. The sad truth is, all the companies I've worked for, the bean counters control products not the engineering staff. For example, business people say, "We need X feature complete by Y date so we can display it at Z trade show and start selling it to make quarter profits". Usually engineering staff are not consulted, or there advice is ignored.

This sounds terrible way to do business, but in a strange way it usually works out. Why? Engineers are passionate. I know the staff working on the Mach E have pride in their work. Issues will get fixed.

But, what if they don't you might ask? Simple solution ... Buy a different car. Money talks.
 

spp

Member
First Name
Shawn
Joined
Feb 1, 2021
Threads
0
Messages
20
Reaction score
29
Location
Michigan, USA
Vehicles
Jeep Cherokee, Pacifica PHEV
Occupation
Sr. Solution Architect
Country flag
Sure, I have been back to the dealer twice. Each time I here from them and others Ford tracks alerts, haven’t seen Ford get involved to help the dealer resolve my issues. Dealer wants me to return Monday so they can start the process over to engage an engineer to come down to fix the issues. So Monday they will start the process over again which means I’m waiting for an extended period. So much for QC checks for a month or so. Everyone here has been wonderful but when I hear tracking down to the nut and nothing about glitchy software I get worried. I rather have a miss aligned door over a unknown software / module issue. Miss aligned the dealer can fix, software they can’t so easily and it appears OTA is like updating my IPad but has zero impact on the firmware around it.

Ford Mustang Mach-E FSA Field Service Action No. 21B02 (Recall?) -- "Software Calibration Update" 65E10B73-9B6F-4546-A54C-ED791D6227B0
Let me be clear this is me speaking my thoughts and not anyone elses (or any other entities that may employ me).

Physical components are relatively easy to track down and figure out when things went wrong. As a fictional example, in batch 2021-3245-46B of blinker fluid bolts, we found 10% of the bolts fractured when exposed to weather below 0. We can track down which VINs had bolts with from that batch, order fixed bolts, and recall the affected VINs and replace the bolts. Door misaligned? Align it and report back to the mothership why it was misaligned so if we get enough reports, a service bulletin can be created.

With software (and a lot of electrical issues), many times we have to track down an amazing amount of parameters to figure out why the system is glitching. Sometimes we find bugs because the voltage sagged below a certain amount and for some reason it manifested itself as a left blinker turning on because the ECU got a bad canbus signal and the hash checks failed but the ECU still executed the command. Fix the ECU firmware to handle this edge case and try to figure out why the voltage sagged. Other times its because Bob in development forgot to comment out a line of code and it still escaped despite 6 months of validation testing.

Once it's pretty much determined nothing is physically wrong with the vehicle, that's where people like me have to take over and play Blind Sherlock Holmes (look at all the evidence remotely and try to piece together what went wrong. If something really went wrong, someone gets flown out). There isn't much for the dealer to do in this case except trying to replicate things the way engineering tells them to. I sincerely hope Ford is working this issue because its the right thing to do to ensure they don't have a bigger underlying issue that affects the rest of the fleet.

Half the reason I am on this forum is to learn about this vehicle because I want a BEV and my company isn't making an affordable one in the next year when I plan to buy one. I'd want them to be doing everything possible to make this vehicle as trouble free as possible as it literally is the future of the company (and every company right now long term it seems).
 

spp

Member
First Name
Shawn
Joined
Feb 1, 2021
Threads
0
Messages
20
Reaction score
29
Location
Michigan, USA
Vehicles
Jeep Cherokee, Pacifica PHEV
Occupation
Sr. Solution Architect
Country flag
** I've been accused of being a rude jerk but what follows is NOT intended to be a dig but a perspective from 35 years in the software business **

There are a lot of assumptions in this and other threads, and the above just happens to be one from someone who's frustrated and worried things will "never be right" - I am quoting this post simply as an example. I get why he feels that way, and in time that feeling will go away. I also understand the emotion behind "well they should have".

I myself often think "well that's dumb, why'd they do it that way" - until I learn the nitty-gritty details, and then realize "ohhh, well that makes sense then". It's sometimes hard to suppress that urge to think "well they should", but you have to realize your understanding of a system as complex as a car, operating system, etc is far too incomplete to properly appreciate why things are done a certain way. One needs to recognize that people with a specific expertise are working on said items, and that they are in fact highly competent in their field.

The fact that Ford is having techs do the aforementioned update does not necessarily mean the process doesn't work or isn't capable of doing the update. Just maybe @RonTCat is correct that it is more a case of risk/reward. Since the MME is a brand new vehicle using a brand new process with brand new software, Ford may have just decided that having techs with specialized tools and techniques do the update is better than the very unlikely but very-embarassing-if-it-happens chance that the cars might become inoperable. That would be a huge PR nightmare and could prove fatal to the reputation of the car. Ford just may not be willing to risk it in their first OTA, even though all their testing has shown the process to work just fine.

Fundamentally computers do EXACTLY what the programming tells them to do, and sometimes programmers make simple mistakes. Sometimes that mistake is a typo (simple punctuation can completely change what a line of code does), and sometimes different pieces of seemingly unrelated code can interact in unexpected ways. There are likely tens of millions of lines of code in the MME, and each of those is an opportunity for a mistake. Sometimes those errors only manifest themselves once every hundred thousand times so it takes longer to realize they are there, and often it is really difficult to find the error.

That's why they've been testing the car for over a year and yet some things aren't quite right. It isn't that Ford is incompetent, EVERY computerized system faces the same challenges. It is impractical to test absolutely every possible circumstance, and the problem with android phones is a perfect example. Google made a change recently in their software that broke android auto - there's no way Ford could have known that was going to happen, nor could they have tested that change last july because the change hadn't happened yet.

The good news is that nothing is permanent with software. It is fundamentally malleable, and absolutely anything can be changed (although it may take a lot of time). What is much harder to change is hardware, and the good news is that the hardware appears to be thoroughly well tested and sound. Sure there may be individual instances of misassembly, but that kind of thing can get adjusted in the field.

What matters is that the hardware is fundamentally sound. Software fixes will continue over the life of the car.
You said it better than I did. We have fast feedback vehicles go to executives and hope to hell that we caught all the stupid stuff ahead of time so we don't have someone with "president" in their title find it. We test, test, test, and then let other people test. We test right up to the day it gets released to the public to buy and then continue to test even more after the fact.
 

Shayne

Well-Known Member
Joined
Aug 9, 2020
Threads
18
Messages
3,341
Reaction score
2,489
Location
Northern Ontario Canada
Vehicles
2021 MME4x Prem
Occupation
Retired
Country flag
** I've been accused of being a rude jerk
Have no problems having the right tech doing the right job. All should understand under the current times as we are all in the same boat. It sure is not business as usual and a bit of tolerance will go along way. May be didn't needed to quote anyone for yours?

Are you involve with ford and the mme interface? Can you tell me how to manage EV, charging preference, and get my home location to repopulate back into previous locations so I can set it up again? I am hoping sitting and keeping it charged to 100% at -18F is no problem for my new beauty, love this vehicle so far. But not for the OS yet; it drives nice. I am off on a loop today to plug in to a free 40 KW like Jamie mentioned. I hope it is working (powered up) at -5F and resets my home location when I get back. It will be my first DCFC so I hope it takes it easy on me being my first time ;).

@machefan Still would like to know when all the idiot lights popped up.
 

timbop

Well-Known Member
First Name
Tim
Joined
Jan 3, 2020
Threads
63
Messages
6,729
Reaction score
13,758
Location
New Jersey
Vehicles
Solar powered 2021 MME ER RWD & 2022 Corsair PHEV
Occupation
Software Engineer
Country flag
Half the reason I am on this forum is to learn about this vehicle because I want a BEV and my company isn't making an affordable one in the next year when I plan to buy one. I'd want them to be doing everything possible to make this vehicle as trouble free as possible as it literally is the future of the company (and every company right now long term it seems).
My wife's uncle worked for Chrysler corp for 40+ years, and my wife has been a loyal Dodge buyer for decades. Her uncle actually worked in the Durango plant for a time so it is likely he assembled part of one of her 3 Durangos. It is really disappointing to see them spending so much of their limited resources putting "bigger and badder" engines in yet more of their products rather than recognizing what a dead end such moves are. It reminds me of the path GM was on just before the crash in late 2008. GM was lucky to get a bailout to save them from their own decision making, but I don't think FCA/Stellantis USA is going to have the same opportunity. I really hope there is a change of direction soon, before it is too late.
 

Shayne

Well-Known Member
Joined
Aug 9, 2020
Threads
18
Messages
3,341
Reaction score
2,489
Location
Northern Ontario Canada
Vehicles
2021 MME4x Prem
Occupation
Retired
Country flag
Physical components are relatively easy to track down and figure out when things went wrong. As a fictional example, in batch 2021-3245-46B of blinker fluid bolts, we found 10% of the bolts fractured when exposed to weather below 0.
Not a good example at all. Would that be due to road stresses or just pre-torque tension? What automaker do you work for ;)? I think all understand that loading software to a vehicle has a certain duty of care greater than dropping bugs onto your dumb phone. We are early adopters what the heck did we expect. There are problems on every project, I never remember exactly what the problems were, just how others behaved.
 

Clownbaby

New Member
First Name
Michael
Joined
Feb 11, 2021
Threads
0
Messages
2
Reaction score
10
Location
California
Vehicles
First Edition Carbonized Grey
Occupation
Healthcare
Country flag
Thanks to @SnBGC I got the service bulletin. It is posted below. Looks like it was a remote start issue.

Ford Mustang Mach-E FSA Field Service Action No. 21B02 (Recall?) -- "Software Calibration Update" 65E10B73-9B6F-4546-A54C-ED791D6227B0
Thanks ARK, or is this just some diabolical scheme to make us second guess not buying that extended warranty?
 

spp

Member
First Name
Shawn
Joined
Feb 1, 2021
Threads
0
Messages
20
Reaction score
29
Location
Michigan, USA
Vehicles
Jeep Cherokee, Pacifica PHEV
Occupation
Sr. Solution Architect
Country flag
My wife's uncle worked for Chrysler corp for 40+ years, and my wife has been a loyal Dodge buyer for decades. Her uncle actually worked in the Durango plant for a time so it is likely he assembled part of one of her 3 Durangos. It is really disappointing to see them spending so much of their limited resources putting "bigger and badder" engines in yet more of their products rather than recognizing what a dead end such moves are. It reminds me of the path GM was on just before the crash in late 2008. GM was lucky to get a bailout to save them from their own decision making, but I don't think FCA/Stellantis USA is going to have the same opportunity. I really hope there is a change of direction soon, before it is too late.
We have BEVs currently (Fiat 500 in EMEA and Maserati) and I am pretty sure we'll have more as the years roll by. We've been told a reason for the PSA merger was their electrification tech complements ours. I have all the confidence in the world we will have affordable EVs here at some point, but not when I need to buy a car this year. Next time though :D
Sponsored

 
 




Top