Ford's Software Release Rhythm

[TheSeg]

As a software developer and growing up with @Ford Motor Company, I'm concerned Ford's current approach to software development is not actually conservative. At best Ford's current approach is ignorant. At worst, irresponsible.

A lot of the talk of Ford vs. Tesla OTAs is a conversation of the speed of releases. Not enough time for testing before release absolutely reflects on final product quality. However, the ability to consistently maintain and release software - the release rhythm - is equally important to software development and product support. I believe Ford's software release rhythm is not a result of the rigorous testing and reliability, but an inability to release software in a consistent and timely fashion. Hiding problems within the process, rather than identifying and addressing issues.


Release Rhythm
While interconnected, the time to develop a release is different from release rhythm. Rhythm is how often a release is regularly distributed to the product. The less frequent the rhythm, the more disruptive a release is to the entire support apparatus for a product.

To explain, I'll use Semantic Versioning terms as they apply to iOS development. This can apply to most professionally developed software, but I'm picking an OS tied to a hardware device to better align with car development.

Semantic expresses a version number as MAJOR.MINOR.PATCH. The current release version of iOS is 14.7.1. MAJOR (14) and MINOR (7) are scheduled releases. MAJOR happens once a year, MINOR roughly every two months. This builds into the entire company the expectation of releases happening at these intervals. Not just releasing the software, but other areas of the business from MarCom to the most important: Product support.

PATCH is reserved for the unexpected releases. Bug fixing for cosmetic, quality of life, or most importantly security. Their very nature is unplanned as it's fixing issues discovered after release. The smarter managers schedule teams to be ready to react to these problems. Then one can cut the patch release and deploy that fix as soon as safely possible. While the release is unplanned, the regularity of releases makes deployment normalized and easier.


Ford's release rhythm isn't a rhythm. Nor is it semantic in any conceivable way.
What software versions are running in you personal Mach-E right now?
Do you know if there's updates available for the car? If so, are you even allowed to update?

If you received "Power Up 1.6" followed by "Power Up 1.4", you know the numbers in update notes are useless in explaining the current state of your Mach-E.

Why is the development and approval of software completely separated from the process of release? If the company is confident to release via service centers, why is it not considered ready for OTA? When an OTA is released, why is it a rollout over cars taking weeks? If it's not ready for all cars, why only let some out at a time?

I don't have insight to what's going on at Ford, but for the life of me I can't think of a valid reason to have two competing release channels for the same software release package. Each path with vastly different states within the channel.


Support Problems
The only user accessible status is the Sync software build number, which is not actually any status of the car or the components. There's the shortcut to the diag screen, but the data isn't a complete picture and only if you're "in the know" of the feature.

When you roll up to a service center, it may take the tech hours to update due to the required battery procedure for updating a car. A lack of locally cached update data or high-speed Internet further complicates the problem. All of which disincentivises the tech from standardizing the car to the latest release. Resulting in known fixable problems and available features not delivered to the customer.

The "conservative" software development in practice means Ford is in the dark of identifying and solving software problems. Nor is Ford in the position to deploy a patch for P0 problems in a reasonable amount of time. The result is a wide assortment of possible combinations of production software versions that no one can track or standardize. The software stack Ford is testing will never be the conditions your car will be, present or future. The 1.7 update of today may not reach you till after you get 1.10.

Imagine how easier it would be for service centers if basic updates were the same as OTAs? More so if local file updates (download to a USB stick, then update car) were possible. The security needed are solved problems that make professional support folks focus on real technical service. Not busy work bogging down the shop, reducing capacity for servicing more cars, which reduces income for service centers. All because OTAs don't work consistently.

This week, Apple will announce iPhone 13 and (possibly) Apple Watch. The next iOS (and watchOS) will be out just before supporting devices from the past six years (iPhone 6s) and four years (Watch gen 3) respectively.

Ford released to customers the GT with software destined for the rest of the Mach-E line... at some point. As if only iPhone 13 could get iOS 15 for months ahead of every other device. If you think a car shouldn't be expected to have this type of long tail support line, then an iPhone's six year lifespan is longer than our cars.


Solving the Problems
The problems I highlight are not rested at the feet of any individual developer at Ford. In fact, my concerns are in part for a better work environment for the Ford team as a whole. What I point out are ultimately management issues. While I'm hopeful the recent hiring of Doug Field will help guide the company, one person alone can not change the tide. The intersection of the advances of software development can benefit not only the @Ford Motor Company, but the dealerships, the service centers, suppliers, and most importantly the customers.

I love how my Mach-E drives. It's the best car I have ever experienced with Ford in my entire life. It falls flat when I see folks around me choose Tesla above Ford when I can't reasonably demonstrate a better total experience than what Tesla provides them. A lot of good people worked on this car, and I want their work to shine. But we're not there yet. And I worry not enough folks in the Glasshouse realize this. Or don't listen to the folks inside the house saying this.


About Me
My software development work crosses education, entertainment, marketing/communication (MarCom), and retail sectors. My father worked for Ford though out my childhood in Dearborn (and a few years with Ford Aerospace in Newport Beach, CA). I gained a lot of personal prosperity though Ford, and I want to see the company succeed in bringing that prosperity to the current Ford team by creating quality experiences to Ford customers.

This started as a reply to a post, but then developed so large that I decided to make a new thread. I... have opinions...

Sponsored

 

[ncaadam]

this is the most sensical thing on this forum I’ve read regarding the MME’s software. Thanks for taking the time to write it up. Hopefully people here take the time to read it and begin to understand that the issues some of us see with the MME’s software are fundamental and systemic in nature, not just lack of feature X and Y.

 

[TheSeg]

this is the most sensical thing on this forum I’ve read regarding the MME’s software.

Thank you! I've had these thoughts floating around for a while and shared bits and pieces on posts. Finally got pushed to write a single piece.

Which isn't to say I have a feature request list of my own, but we can't get there when the pipeline is this broken.

 

[macchiaz-o]

A lot of the talk of Ford vs. Tesla OTAs is a conversation of the speed of releases. Not enough time for testing before release absolutely reflects on final product quality. However, the ability to consistently maintain and release software - the release rhythm - is equally important to software development and product support. I believe Ford's software release rhythm is not a result of the rigorous testing and reliability, but an inability to release software in a consistent and timely fashion. Hiding problems within the process, rather than identifying and addressing issues.

Agreed.

Though I would say that you're describing Utopia. A company that overstaffs software engineering and related disciplines, to include test, requirements analysts, designers, trainers, field support, hw/sw system integrators, user communities, and so on. And then gives them the freedom and just the right amount of organization and management for the best work to rise to the top.

This is very few companies. Even in the software world, it's still a tiny percentage. I wish Ford were there, but they're obviously not at this time.

What software versions are running in you personal Mach-E right now

I don't know. It doesn't really matter to me.

Do you know if there's updates available for the car?

I don't know. But since I left my vehicle at its default setting to automatically install updates, it doesn't need to continuously or even periodically occupy my thoughts.

If so, are you even allowed to update?

I don't see why not.

Why is the development and approval of software completely separated from the process of release? If the company is confident to release via service centers, why is it not considered ready for OTA

This is purely my speculation, but I think there's an unresolved issue blocking the full functioning of OTA updates in our vehicles.

Otherwise, I cannot see why Ford would publish updates that only dealers and factories can install.

This is costing Ford a lot of money. They have to pay dealers to install these updates, and to try to make whole any upset or tangibly impacted customers.

If OTA updates were working fully properly, Ford could deliver them at a tiny fraction of the warranty costs they're currently forced to pay out to those who seek remedy via dealerships. I guarantee you that Ford knows this. This is why I speculate OTA is in some way still "broken."

When an OTA is released, why is it a rollout over cars taking weeks? If it's not ready for all cars, why only let some out at a time?

Good question and one I've wondered myself. Part of it may be anxiety over unexpectedly escaped issues that reach many customers. I can't think of any other reason so far.

Imagine how easier it would be for service centers if basic updates were the same as OTAs? More so if local file updates (download to a USB stick, then update car) were possible. The security needed are solved problems that make professional support folks focus on real technical service. Not busy work bogging down the shop, reducing capacity for servicing more cars, which reduces income for service centers. All because OTAs don't work consistently.

Right. But perhaps the issues blocking significant OTA updates -- again, I'm speculating -- are also preventing USB or other high speed update options.

 

[curiousurick]

I have to say that this post is amazing. I work on a large consumer software system that impacts millions of customers and even our deployment system had a lot of growing pains. But in our darkest moments of “what the fuck are we doing”, we were never as bad as Ford seems to be.

There are so many resources for every level of management and producer that the Ford software team could have taken from to be ready for this car’s release. They made so many huge promises and so far, have completely failed to follow through. I’ve yet to have any major bugs interrupt my usage of the car but I really do envy the Tesla customers who get regular updates and who have their delivery rep tell them honestly “today, this is the worst your car will ever be”.

 

[garfieldorlando]

We have 20 days left for the Q3, when the Q3 OTA was promised. I seriously doubt it is coming soon, maybe some very limit of cars will get it in Q3 but majority will not see it till some time in Q4. Anyway this supposes to be the first real OTA, not the previous gateway updates, we can see how it play out. If Mach E fails eventually, it will be the software killing it. I hope this won't happen.

 

[macchiaz-o]

They made so many huge promises and so far, have completely failed to follow through. I’ve yet to have any major bugs interrupt my usage of the car

Ford completely failed, yet you've not had any major bugs interrupt your usage of the car? This doesn't sound like a complete failure to me.

Yes, some things were promised for early 2021 and are not yet delivered. Such as electronic opening of the frunk. ?

I really do envy the Tesla customers who get regular updates and who have their delivery rep tell them honestly “today, this is the worst your car will ever be”.

Surely some of the updates lessened the ownership experience? Such as removing use of radar sensors and firmware bugs that cause windows to roll back down when you're trying to close them?

 

[Mirak]

What @TheSeg fails to account for is #SoftwareIsHard.

 

[kltye]

When you roll up to a service center, it may take the tech hours to update due to the required battery procedure for updating a car. A lack of locally cached update data or high-speed Internet further complicates the problem. All of which disincentivises the tech from standardizing the car to the latest release. Resulting in known fixable problems and available features not delivered to the customer.

Imagine how easier it would be for service centers if basic updates were the same as OTAs? More so if local file updates (download to a USB stick, then update car) were possible. The security needed are solved problems that make professional support folks focus on real technical service. Not busy work bogging down the shop, reducing capacity for servicing more cars, which reduces income for service centers. All because OTAs don't work consistently.

Just wanted to correct a couple points made here. The length of updates isn't due to the "required battery procedure". (Actually, it's the length of the updates that require hooking up the car to an external source) But, as mentioned lots of times in these forums, a) only the IPMA update takes a couple hours, and b) the reason it takes so long is because the HS-CAN standard only goes up to 512kbps. I've performed all other updates without needing to hook up my car to external power.

As far as file update sizes go, it's not very large - I performed the updates while tethering my laptop to my cellphone (since I don't have a garage), and the actual file downloads took a couple minutes at most.

 

[woody]

As a software developer and growing up with @Ford Motor Company, I'm concerned Ford's current approach to software development is not actually conservative. At best Ford's current approach is ignorant. At worst, irresponsible.

A lot of the talk of Ford vs. Tesla OTAs is a conversation of the speed of releases. Not enough time for testing before release absolutely reflects on final product quality. However, the ability to consistently maintain and release software - the release rhythm - is equally important to software development and product support. I believe Ford's software release rhythm is not a result of the rigorous testing and reliability, but an inability to release software in a consistent and timely fashion. Hiding problems within the process, rather than identifying and addressing issues.


Release Rhythm
While interconnected, the time to develop a release is different from release rhythm. Rhythm is how often a release is regularly distributed to the product. The less frequent the rhythm, the more disruptive a release is to the entire support apparatus for a product.

To explain, I'll use Semantic Versioning terms as they apply to iOS development. This can apply to most professionally developed software, but I'm picking an OS tied to a hardware device to better align with car development.

Semantic expresses a version number as MAJOR.MINOR.PATCH. The current release version of iOS is 14.7.1. MAJOR (14) and MINOR (7) are scheduled releases. MAJOR happens once a year, MINOR roughly every two months. This builds into the entire company the expectation of releases happening at these intervals. Not just releasing the software, but other areas of the business from MarCom to the most important: Product support.

PATCH is reserved for the unexpected releases. Bug fixing for cosmetic, quality of life, or most importantly security. Their very nature is unplanned as it's fixing issues discovered after release. The smarter managers schedule teams to be ready to react to these problems. Then one can cut the patch release and deploy that fix as soon as safely possible. While the release is unplanned, the regularity of releases makes deployment normalized and easier.


Ford's release rhythm isn't a rhythm. Nor is it semantic in any conceivable way.
What software versions are running in you personal Mach-E right now?
Do you know if there's updates available for the car? If so, are you even allowed to update?

If you received "Power Up 1.6" followed by "Power Up 1.4", you know the numbers in update notes are useless in explaining the current state of your Mach-E.

Why is the development and approval of software completely separated from the process of release? If the company is confident to release via service centers, why is it not considered ready for OTA? When an OTA is released, why is it a rollout over cars taking weeks? If it's not ready for all cars, why only let some out at a time?

I don't have insight to what's going on at Ford, but for the life of me I can't think of a valid reason to have two competing release channels for the same software release package. Each path with vastly different states within the channel.


Support Problems
The only user accessible status is the Sync software build number, which is not actually any status of the car or the components. There's the shortcut to the diag screen, but the data isn't a complete picture and only if you're "in the know" of the feature.

When you roll up to a service center, it may take the tech hours to update due to the required battery procedure for updating a car. A lack of locally cached update data or high-speed Internet further complicates the problem. All of which disincentivises the tech from standardizing the car to the latest release. Resulting in known fixable problems and available features not delivered to the customer.

The "conservative" software development in practice means Ford is in the dark of identifying and solving software problems. Nor is Ford in the position to deploy a patch for P0 problems in a reasonable amount of time. The result is a wide assortment of possible combinations of production software versions that no one can track or standardize. The software stack Ford is testing will never be the conditions your car will be, present or future. The 1.7 update of today may not reach you till after you get 1.10.

Imagine how easier it would be for service centers if basic updates were the same as OTAs? More so if local file updates (download to a USB stick, then update car) were possible. The security needed are solved problems that make professional support folks focus on real technical service. Not busy work bogging down the shop, reducing capacity for servicing more cars, which reduces income for service centers. All because OTAs don't work consistently.

This week, Apple will announce iPhone 13 and (possibly) Apple Watch. The next iOS (and watchOS) will be out just before supporting devices from the past six years (iPhone 6s) and four years (Watch gen 3) respectively.

Ford released to customers the GT with software destined for the rest of the Mach-E line... at some point. As if only iPhone 13 could get iOS 15 for months ahead of every other device. If you think a car shouldn't be expected to have this type of long tail support line, then an iPhone's six year lifespan is longer than our cars.


Solving the Problems
The problems I highlight are not rested at the feet of any individual developer at Ford. In fact, my concerns are in part for a better work environment for the Ford team as a whole. What I point out are ultimately management issues. While I'm hopeful the recent hiring of Doug Field will help guide the company, one person alone can not change the tide. The intersection of the advances of software development can benefit not only the @Ford Motor Company, but the dealerships, the service centers, suppliers, and most importantly the customers.

I love how my Mach-E drives. It's the best car I have ever experienced with Ford in my entire life. It falls flat when I see folks around me choose Tesla above Ford when I can't reasonably demonstrate a better total experience than what Tesla provides them. A lot of good people worked on this car, and I want their work to shine. But we're not there yet. And I worry not enough folks in the Glasshouse realize this. Or don't listen to the folks inside the house saying this.


About Me
My software development work crosses education, entertainment, marketing/communication (MarCom), and retail sectors. My father worked for Ford though out my childhood in Dearborn (and a few years with Ford Aerospace in Newport Beach, CA). I gained a lot of personal prosperity though Ford, and I want to see the company succeed in bringing that prosperity to the current Ford team by creating quality experiences to Ford customers.

This started as a reply to a post, but then developed so large that I decided to make a new thread. I... have opinions...

Thank you.
Those of us less knowledgeable assumed as much in simpler more basic terms.
almost 6 months now for me.
Still no functional FordPass (took five and one half months to get vehicle screen changed from Ford Edge to MachE; no plug and charge; no automatic updates [had to follow Forum to be aware of any updates, then forced them from Play Store]; no Power ups; no Securialert; etc.
I have finally persuaded dealer to look into the problem leading to a telematics control unit module install/replace? Sometime (promised 3-5 days last week). May fix it? Wait and see and wait some more (6 months does not seem reasonable to me for something that should have been fixed at the factory or at the very least at the dealership before I took delivery).
But what do I know?
Thanks again.

 

[SashaLondon]

I agree with the opening post. Hopefully, that is why they hired the ex-Tesla Doug Field. He will bring experience in software release planning to the MME.
I think they can see how much their customers value software updates when they said that they were surprised by how many people updated the cars plus all the talk on here about OTAs (that they monitor)!

 

[Marcel]

Something that doesn't help is the differences in regulation/software for US-EU-UK.

 

[Jeffvisor]

It is well written and logical. The one aspect that he mentions that is very true and likely a nightmare for the new guy coming in is that our Mach-e's are running different versions of software. I recently went to Ford service to update the module that quiets the charging 21-2288 and asked that they update all the software they didn't only that module. The GT's all seem to be on more advanced software updates. Some say they never got 1.4 and 1.6. Hopefully with the Q3 big update they bring all Mach-e's to the newest software. I don't have faith in that since I have a update for the charging noise that Ford said will not come to over the air update until Q1 2022. If the software is available at the dealer in September why do we need to wait until 5-7 months for it to come over the air? Especially when that update is so much better for the customer and car.

 

[Finkerton]

I recently went to Ford service to update the module that quiets the charging 21-2288 and asked that they update all the software they didn't only that module.

And in my case the dealer refused, saying that they would update whatever the system said they should related to the TSB and wouldn't update anything else because Ford didn't tell them to (and they were afraid it'd cause some other problem they'd have to deal with).

 

[sfabris]

Ford completely failed, yet you've not had any major bugs interrupt your usage of the car? This doesn't sound like a complete failure to me.

Sorry,

I had to bring my car to the dealer in order to correct issue:
- Wrench icon showing up in various occasions.
- Fan noise too loud for me to sleep while charging (my car is sitting next to my doorbed windows).

So bugs interrupted my usage of the car (and like me many others). Everything was solved with modules update.

Sponsored