Ford's Software Release Rhythm

Dictate

Member
First Name
Benjamin
Joined
Jun 17, 2021
Threads
1
Messages
10
Reaction score
10
Location
Los Angeles, California
Vehicles
Ford Mustang Mach-E
Occupation
Retried
Country flag
PAK still only works 25% of the time. How about the Frunk release, still not available. Why not use a minimum viable product approach and release updates faster?
Sponsored

 

jdmrc93

Well-Known Member
First Name
Joe
Joined
Jun 25, 2020
Threads
20
Messages
345
Reaction score
874
Location
Allen Park, MI, USA
Vehicles
2021 Mustang Mach-E First Edition in Grabber Blue
Occupation
IT Network Engineer
Country flag
From one engineer to another, THANK YOU for saying it. I've had the same thoughts but you wrote them out and hit every nail on the head. I hope they listen.
 

agoldman

Well-Known Member
Joined
Jan 23, 2020
Threads
49
Messages
2,889
Reaction score
2,397
Location
Charleston, SC
Vehicles
GT Cyber Orange
Country flag
A big old company like Ford, and yet they are over their own heads in this new age, and chasing a young company like Tesla. How this all plays out remains to be seen.
 

Jimrpa

Well-Known Member
First Name
Jim
Joined
Sep 10, 2020
Threads
230
Messages
7,005
Reaction score
9,298
Location
Wayne, PA
Vehicles
2021 Infinite Blue Premium Mustang Mach E ER AWD
Occupation
Retied (formerly tried to herd highly technical, independent cats)
Country flag
I appreciate the insight. For those who criticize “large established companies” not being able to change and adapt, I think there are many examples that disprove that. It may take a bit to get the ship turned, but they can turn and do a great job doing so, if they have the right leadership and commitment.
I’d still like to know why, in a car with a largely new electronics architecture, nobody thought to put in a modern, high-speed connection (such as Ethernet or USB-C to update the software. A 512KBbps interface in the 21st century is stupid. Frankly, I’d Ford developed an optional dealer-installed upgrade interface, I’d pay for it!
 

OrchidMania

Well-Known Member
Joined
Feb 26, 2021
Threads
10
Messages
677
Reaction score
436
Location
Miami Beach
Vehicles
Grabber Blue Premium ER
Occupation
Retired IT, School Math Volunteer
Country flag
As a software developer and growing up with @Ford Motor Company, I'm concerned Ford's current approach to software development is not actually conservative. At best Ford's current approach is ignorant. At worst, irresponsible.

A lot of the talk of Ford vs. Tesla OTAs is a conversation of the speed of releases. Not enough time for testing before release absolutely reflects on final product quality. However, the ability to consistently maintain and release software - the release rhythm - is equally important to software development and product support. I believe Ford's software release rhythm is not a result of the rigorous testing and reliability, but an inability to release software in a consistent and timely fashion. Hiding problems within the process, rather than identifying and addressing issues.


Release Rhythm
While interconnected, the time to develop a release is different from release rhythm. Rhythm is how often a release is regularly distributed to the product. The less frequent the rhythm, the more disruptive a release is to the entire support apparatus for a product.

To explain, I'll use Semantic Versioning terms as they apply to iOS development. This can apply to most professionally developed software, but I'm picking an OS tied to a hardware device to better align with car development.

Semantic expresses a version number as MAJOR.MINOR.PATCH. The current release version of iOS is 14.7.1. MAJOR (14) and MINOR (7) are scheduled releases. MAJOR happens once a year, MINOR roughly every two months. This builds into the entire company the expectation of releases happening at these intervals. Not just releasing the software, but other areas of the business from MarCom to the most important: Product support.

PATCH is reserved for the unexpected releases. Bug fixing for cosmetic, quality of life, or most importantly security. Their very nature is unplanned as it's fixing issues discovered after release. The smarter managers schedule teams to be ready to react to these problems. Then one can cut the patch release and deploy that fix as soon as safely possible. While the release is unplanned, the regularity of releases makes deployment normalized and easier.


Ford's release rhythm isn't a rhythm. Nor is it semantic in any conceivable way.
What software versions are running in you personal Mach-E right now?
Do you know if there's updates available for the car? If so, are you even allowed to update?

If you received "Power Up 1.6" followed by "Power Up 1.4", you know the numbers in update notes are useless in explaining the current state of your Mach-E.

Why is the development and approval of software completely separated from the process of release? If the company is confident to release via service centers, why is it not considered ready for OTA? When an OTA is released, why is it a rollout over cars taking weeks? If it's not ready for all cars, why only let some out at a time?

I don't have insight to what's going on at Ford, but for the life of me I can't think of a valid reason to have two competing release channels for the same software release package. Each path with vastly different states within the channel.


Support Problems
The only user accessible status is the Sync software build number, which is not actually any status of the car or the components. There's the shortcut to the diag screen, but the data isn't a complete picture and only if you're "in the know" of the feature.

When you roll up to a service center, it may take the tech hours to update due to the required battery procedure for updating a car. A lack of locally cached update data or high-speed Internet further complicates the problem. All of which disincentivises the tech from standardizing the car to the latest release. Resulting in known fixable problems and available features not delivered to the customer.

The "conservative" software development in practice means Ford is in the dark of identifying and solving software problems. Nor is Ford in the position to deploy a patch for P0 problems in a reasonable amount of time. The result is a wide assortment of possible combinations of production software versions that no one can track or standardize. The software stack Ford is testing will never be the conditions your car will be, present or future. The 1.7 update of today may not reach you till after you get 1.10.

Imagine how easier it would be for service centers if basic updates were the same as OTAs? More so if local file updates (download to a USB stick, then update car) were possible. The security needed are solved problems that make professional support folks focus on real technical service. Not busy work bogging down the shop, reducing capacity for servicing more cars, which reduces income for service centers. All because OTAs don't work consistently.

This week, Apple will announce iPhone 13 and (possibly) Apple Watch. The next iOS (and watchOS) will be out just before supporting devices from the past six years (iPhone 6s) and four years (Watch gen 3) respectively.

Ford released to customers the GT with software destined for the rest of the Mach-E line... at some point. As if only iPhone 13 could get iOS 15 for months ahead of every other device. If you think a car shouldn't be expected to have this type of long tail support line, then an iPhone's six year lifespan is longer than our cars.


Solving the Problems
The problems I highlight are not rested at the feet of any individual developer at Ford. In fact, my concerns are in part for a better work environment for the Ford team as a whole. What I point out are ultimately management issues. While I'm hopeful the recent hiring of Doug Field will help guide the company, one person alone can not change the tide. The intersection of the advances of software development can benefit not only the @Ford Motor Company, but the dealerships, the service centers, suppliers, and most importantly the customers.

I love how my Mach-E drives. It's the best car I have ever experienced with Ford in my entire life. It falls flat when I see folks around me choose Tesla above Ford when I can't reasonably demonstrate a better total experience than what Tesla provides them. A lot of good people worked on this car, and I want their work to shine. But we're not there yet. And I worry not enough folks in the Glasshouse realize this. Or don't listen to the folks inside the house saying this.


About Me
My software development work crosses education, entertainment, marketing/communication (MarCom), and retail sectors. My father worked for Ford though out my childhood in Dearborn (and a few years with Ford Aerospace in Newport Beach, CA). I gained a lot of personal prosperity though Ford, and I want to see the company succeed in bringing that prosperity to the current Ford team by creating quality experiences to Ford customers.

This started as a reply to a post, but then developed so large that I decided to make a new thread. I... have opinions...
I agree. I retired from a very large computer company who struggled with software (other than operating systems). They had all the processes in place to have a strict software development. But they weren't flexible enough to make changes quickly as needs changed. Thus the dilemma .... speed or quality. Can Ford have both. An aside my god child , a young 24 year old, works in IT in the AI arena. He said it would be fun to work for Telsa but not Ford..
Please I hope the new hire from Apple brings excitement, investments, processes, and the cool factor.
 


Cobra427

Well-Known Member
First Name
John
Joined
Jun 1, 2021
Threads
43
Messages
319
Reaction score
350
Location
Ohio
Vehicles
2021 F-150 Lariat Powerboost, 2022 Mustang Mach E
Occupation
Retired
Country flag
Wondering if a solution might be to not outsource all the software? Since we are using Apple analogies since Ford hired an Apple guy, Apple became successful writing it's own software.
 

TruWrecks

Well-Known Member
First Name
Doug
Joined
Jun 3, 2021
Threads
22
Messages
860
Reaction score
917
Location
Oregon
Vehicles
Mach-E Star White Premium AWD Ext (Ghost Stallion)
Occupation
Retired
Country flag
I think even Ford forgot that it built the sexiest super computer on wheels. Now it needs to get the software tuned up.
 

alexgorod

Well-Known Member
First Name
Alex
Joined
Jul 27, 2021
Threads
14
Messages
1,071
Reaction score
1,134
Location
Michigan
Vehicles
21 Job 2 MME Premium SR AWD (21 CR-V - wife's)
Occupation
DBA
Country flag
Wondering if a solution might be to not outsource all the software?
Do they outsource it now? I was interviewed there abut 8 years ago, and at that time they had internal IT with a lot of contractors imbedded in. Either way, it's up to Ford to set up processes and timelines, and it definitely looks like they struggle with both.
 

Cobra427

Well-Known Member
First Name
John
Joined
Jun 1, 2021
Threads
43
Messages
319
Reaction score
350
Location
Ohio
Vehicles
2021 F-150 Lariat Powerboost, 2022 Mustang Mach E
Occupation
Retired
Country flag
Do they outsource it now? I was interviewed there abut 8 years ago, and at that time they had internal IT with a lot of contractors imbedded in. Either way, it's up to Ford to set up processes and timelines, and it definitely looks like they struggle with both.
Well, they have. The first edition of SYNC was written by Microsoft, of all people.

Every time I see a "PAAK doesn't work" post, I wonder, "Who wrote PAAK, anyway?"
 

SpacePony

Well-Known Member
Joined
Mar 29, 2021
Threads
5
Messages
206
Reaction score
363
Location
Kansas City Metro
Vehicles
2021 Mach-e Preimum AWD Ext. Battery
Country flag
Well, they have. The first edition of SYNC was written by Microsoft, of all people.
I frequently hear this about SYNC, but although it was written on top of Windows CE (embedded Windows OS) for automotive, the SYNC product (the features and interface) itself didn't actually start with Microsoft at all, but it did finish with it. Development and function was such a mess that basically Microsoft stepped in before release (and hence was "co-developed") to clean it up enough to make it mostly-functional and not ruin their embedded reputation entirely.

Although it was pretty revolutionary at the time (especially for non-luxury cars), and won awards, it had a lot of issues, although much more in MyFord Touch than the original Sync. The original Sync was actually pretty solid, although it was remembered for having issues with voice control. Funny that SYNC3 is a complete ground-up effort that bears no relation to the original Sync and MyFord Touch, and run on QNX instead of Windows CE.
 
Last edited:

jmccauley40

Member
First Name
Jim
Joined
Aug 16, 2021
Threads
2
Messages
8
Reaction score
3
Location
97035
Vehicles
2021 Mach-E California Tour trim
Country flag
Thank you certified smart person. Don't understand why there is such a fricking problem with improvements to the software. My Mach-E spent 4 days in the shop because the on-board, Ford navigation system went to crap. Could not locate my car, did not know what elevation I was at and the interface with the third-party navigation at best, was intermittent. My understanding with the local ford service was that the tech was on some hotline trying to trouble-shoot the issue directly with Ford. No success.. Love the car, but get the damn software fixed!!
 

unhandled

Well-Known Member
First Name
Mathieu
Joined
Jul 28, 2021
Threads
13
Messages
176
Reaction score
173
Location
Stoke, Quebec, Canada
Vehicles
Mustang Mach-E Premium AWD ER
Country flag
There a lot to unpack in your post, so here we go!

A lot of the talk of Ford vs. Tesla OTAs is a conversation of the speed of releases. Not enough time for testing before release absolutely reflects on final product quality. However, the ability to consistently maintain and release software - the release rhythm - is equally important to software development and product support. I believe Ford's software release rhythm is not a result of the rigorous testing and reliability, but an inability to release software in a consistent and timely fashion. Hiding problems within the process, rather than identifying and addressing issues.

Release Rhythm
While interconnected, the time to develop a release is different from release rhythm. Rhythm is how often a release is regularly distributed to the product. The less frequent the rhythm, the more disruptive a release is to the entire support apparatus for a product.
I'd be curious to understand why their release pace is so slow. A few thoughts come to mind:

- If capturing issues is painful, do they have the right telemetry infrastructure to capture and rationalize all that data?
- If quality assurance is the bottleneck, what's the ratio of manual vs automated test? If it takes them weeks/months to test new releases, that's a major issue. Compare that with organizations that ship multiple releases per day.
- Once that's done and the release quality is good for rollout, what's their release/deployment process?

I'm looking at this from a DevOps/DevSecOps (you name the new flavor) point of view where the following key metrics are often measured:

  1. Lead time (from code committed to code deployed)
  2. Deployment frequency (to production)
  3. Change fail percentage (for production deployments)
  4. Mean time to restore (from a production failure)

Those touches on a few of your points. i.e.

Lead time: It takes seemingly a long time before we can see the fixes in our cars.
Deployment frequency: Since the updates are infrequent, they are potentially large, disruptive updates. Probably raising the anxiety level of release managers at Ford. Smaller rocks thrown in the pond are less scary than giant meteorites.
Change fail percentage: Hopefully their release process doesn't fail a lot, causing issues when the updates are applied to the cars.
MTTR: When things go wrong, how fast can they go through the whole process to fix a bad situation.


Ford's release rhythm isn't a rhythm. Nor is it semantic in any conceivable way.
What software versions are running in you personal Mach-E right now?
Do you know if there's updates available for the car? If so, are you even allowed to update?

If you received "Power Up 1.6" followed by "Power Up 1.4", you know the numbers in update notes are useless in explaining the current state of your Mach-E.
I agree with this. It's rather counterintuitive from a release point of view to ship 1.4 after 1.6 unless there are Power Ups for different modules/components and each of those have their own Power-Up versioning. On the outside it looks odd but there's probably an logical explanation for that.


Why is the development and approval of software completely separated from the process of release? If the company is confident to release via service centers, why is it not considered ready for OTA? When an OTA is released, why is it a rollout over cars taking weeks? If it's not ready for all cars, why only let some out at a time?

I don't have insight to what's going on at Ford, but for the life of me I can't think of a valid reason to have two competing release channels for the same software release package. Each path with vastly different states within the channel.
This part I'm debating a bit. If the updates installed at the dealer would follow the same release/deployment process as the general public OTA updates, I could see that channel being an inner release ring. i.e. Rollout of new release could go something like this (similar to Windows Insider)

1) Internal Ford testers
2) Ford employees
3) Dealers (to address issues faced by customers who are bringing their cars to be serviced while being in a more controlled environment/tighter communication channel with Ford engineers)
4) Early access subscribers (probably a representation of different trims/geographies)
5) General public

If you follow a release model like this, I think by the time it reaches the general public, you are fairly confident about the update. How quickly you can/want to go though this is another thing.

The result is a wide assortment of possible combinations of production software versions that no one can track or standardize. The software stack Ford is testing will never be the conditions your car will be, present or future. The 1.7 update of today may not reach you till after you get 1.10.
From an issue tracking/resolution perspective, version fragmentation is a real pain. Ford doesn't want to be in a position where they need to do regression testing for every possible combination of module versions currently running in the field. That's why operating systems like Windows/iOS moved aways from individual component patches as much as possible into single OS versions. They basically test the full OS with all the latest versions of each components included in that release.


About Me
I've worked in different software engineering/infrastructure/architecture roles for the last 15 years or so. I've also started tracking a good chunk of the issues faced by Mach-E owners on a publicly viewable board to help raising visibility for both the community and Ford.
 

fokkerlit

Well-Known Member
First Name
Rob
Joined
Aug 9, 2021
Threads
5
Messages
65
Reaction score
58
Location
Massachusetts
Vehicles
2021 Mach-e
Country flag
I'm a software developer as well and the entire rollout has rubbed me the wrong way. Ford needs to understand that they aren't just a car company anymore, they're a software company and act accordingly. Bug reports and issue tracking for starters, there is a lot that they aren't doing that would help reception from the community.
Sponsored

 
 




Top