Ford's Software Release Rhythm

[unhandled]

Many organizations have adopted DevOps for embedded systems, something seemingly impossible based on previous posts. Here are a few references I found that suggest otherwise:

The Scene of DevOps in the Automotive Industry - DZone DevOps
Embedded DevOps: Implementing CI/CD in Embedded Development (windriver.com)

Tooling for car manufacturers to do DevOps
DevOps for Automotive Companies and OEMs With the JFrog Platform

It's also strange if Daimler is looking to hire DevOps engineers for their embedded systems. Maybe we should reach out to them to let them know it can't be done. Would save them a bunch of money!

DevOps Engineer for Autonomous Driving platform (all genders) - in Berlin| Daimler > Career > Job Search > Job Postings

Side note, I also happened to have found that Ford uses that manage OTA:

Customer Success: FORD MOTOR COMPANY | Wind River

There are no such thing as "independent" releases in real life multicomponent multiteam multivendor projects. Not to repeat myself some, but when Windows Update used to ship 10 "independent" patches a month (an average number to make a point), that was 1,024 possible combinations, which became 1,048,576 after two months, and so on. Do you think all those combinations were tested? Of course not. Microsoft has said so publicly. And that was just the OS, not the app layer or all the hardware drivers or... Even in a perfect DevOps pipeline world (which nobody had because nobody develops without outside OSS nowadays and that stuff is never fully tested as independent components nor properly updated), there are still component dependencies. (This is I admit somewhat pedantic because I am focusing on one word perhaps too much.)

I agree with the above point, which is why I was mentioning that Microsoft and Apple have simplified the testing process for new OS updates by reducing the number of permutations that need to be tested for at least the code they are shipping. All that while each individual components included in that release is continually unit and integrated tested in the overall OS build. External dependencies are something are like any dependency, they are part of the code to ship, so they need to be integration tested at a minimum.

To prove that all execution branches in a piece of software have been exercised of that a program logic is sound is difficult for sure but it's definitely a topic of active research. Languages have been developed to tackle that particular challenge, recently in the world of blockchain (i.e. the Reach language) as you could understand that you don't want a smart contract logic to be faulty when it comes to handling people/organization money.

All that to say we shouldn't be talking in absolutes in one side or the other. It's a tough challenge for sure but those are not problems without solutions. Many organizations have overcome those already as of today. It may take time for Ford to ramp-up to that level across the board but seeing some movement on some user facing components would probably go a long way in demonstrating their abilities and willingness to continually improve. I'm sure some people over there are working their tails off to get there but organizational changes take time when the sense or urgency to change is not fully realized by the majority.

Sponsored

 

[unhandled]

I’ve read everything you’ve written very avidly. All I can say is that it all seems spot-on and in agreement with my experience. This final quote here completely aligns with what my intuition from too many years of experience is whispering in my ear, may be some of the major issues the program is wrestling with right now, and why Ford is being a bit quiet and a bit slow with us early adopters.

My biggest fear is that they may have discovered that some component/subsystem ended up not being fully BlueCruise capable and their production engineers and service engineers are trying to figure out how to cost-effectively get the upgraded hardware in. Ugh. I really hope I’m wrong.

Hopefully BlueCruise doesn't hold up all the other changes that need to be released!

 

[unhandled]

Interesting anecdote, my boss was telling me about his niece who works at NASA JPL on Perseverance/Ingenuity. She was telling him that the biggest thing was to land the rover but after that, they had to release a bunch of updates to patch stuff that wasn’t working as expected. If we can update a robot’s navigation logic on Mars, pretty sure we can get FordPass to show the proper charge level on the home screen! ?

 

[Woeo]

I really do envy the Tesla customers who get regular updates and who have their delivery rep tell them honestly “today, this is the worst your car will ever be”.

Plenty of Tesla owners would tell you and many have made their opinions known on the web, including certain YouTubers who self identify as fanboys, that the path forward for Tesla at times includes regressions.

 

[Luke]

Interesting anecdote, my boss was telling me about his niece who works at NASA JPL on Perseverance/Ingenuity. She was telling him that the biggest thing was to land the rover but after that, they had to release a bunch of updates to patch stuff that wasn’t working as expected. If we can update a robot’s navigation logic on Mars, pretty sure we can get FordPass to show the proper charge level on the home screen! ?

The main difference between both companies/agencies would be the budget. ?

 

[unhandled]

The main difference between both companies/agencies would be the budget. ?

That’s why I scaled back from BlueCruise to just showing the same numbers on two different screens! ?

 

[burtonhen]

Wonder what happened with the person in the other thread who said they were told Friday or Monday for 1.7…

 

[TruWrecks]

Wonder what happened with the person in the other thread who said they were told Friday or Monday for 1.7…

Maybe they got drunk with @Ford Motor Company ?

 

[mpshizzle]

Wonder what happened with the person in the other thread who said they were told Friday or Monday for 1.7…

I'm here and feeling very betrayed by the guy that spewed those lies at me ????

 

[TheSeg]

Sorry for my delay in response. Life happens, you know? I'm not going to direct quote folks as that becomes a jumbled mess of replies and there's no need to respond to personal attacks.

I know folks here and in the car biz claim real-time computing is a different threat model than other forms of computing. That somehow this programming paradigm makes a car immune to potential serious faults. That by fact that it hasn't happened before, introducing connectivity will still make it safe. I don't subscribe to this thinking.

Perhaps I should started this thread differently:
If an exploitable flaw with easy penetration (like this Jeep in 2015) was disclosed to Ford and the fix was QA approved at 5pm today, how soon could it get the fix to all Mach-Es in the fleet?

Monday's release of iOS 14.8 and the rest of the OS platform highlighted a similar nightmare scenario. On Tuesday Sept 7th, a zero-click flaw was disclosed to Apple. The use and ramifications of the flaw causes direct and actionable harm to customers in various ways (domestic and state-sponsored ways). Six days later: iOS, iPadOS, watchOS, and macOS had the patch distributed to the entire actively supported fleet of devices.

The current fastest from of releasing software to Mach-Es approved for release to customers is from the factory or at a service center. The long lines that would amass would be quicker than it's taking to patch known issues.

Again, part of the reason I'm pointing out these problems is to improve the quality of life in the Ford family. Flaws happen and just as I wouldn't want to be judged, I don't judge developers for a flaw happening in the first place. Software is hard. What matters is addressing and deploying the fix in a timely manner.

What keeps me up at night is when the bad thing happens, a lot of people are going to be overburdened with deploying software fixes because OTA deployment is seemingly not up to the task. Right now it's all lack of promised features and stop safely now situations where we wait for deployment to happen or convince a service center to update our car if we know the right codes. What happens when it's a worse problem that needs software to be patched ASAP?

I'm a bit of an Orpheus. I try to show others how the world could be, in spite of what it is. To simply turn around and give up because it's hard degrades the talent of the folks at Ford and it's partners. This is a hard problem, but a solvable problem that will makes everyone's quality of life improve.

To note: @unhandled link post is good reading.

 

[bp99]

The current fastest from of releasing software to Mach-Es approved for release to customers is from the factory or at a service center. The long lines that would amass would be quicker than it's taking to patch known issues.

There have been OTA releases. We know it's an ability they have. You're conflating their current process with what their capabilities are.

It's a fairly standard process to roll out changes to widening audiences. Start with internal testers. Next move on to a targeted group in a controlled environment (dealer service departments against cars exhibiting the problem being resolved). Then you might start waves of roll outs. Sometimes by region, by build date, random until a certain count, etc. Once confident no harm is being done, another, larger wave. Repeat until you open it up for everyone.

There's no reason to believe that they can't push out a critical update much faster OTA if the need exists. So far, there has not been an issue that merits such a need.

Would I like to see some of the infotainment quirks pushed out faster? Yes I would. Are any of them critical? No. I really want to see PAAK working better, but I also believe the problems lies in the phone app, not the car (my car has the latest PAAK update and it's still a crap shoot if it works). Not something quicker OTAs would help with. Although I also believe it's never going to work great as architected. Using Bluetooth was a mistake and a technology dead end. It needed to be UWB, even though that would have initially limited them to newer iPhones and Samsung phones. PAAK as we know it will be orphaned within two years.

Do I want to see them pushing out system modules OTA quickly? No I don't. I only want them mucking with critical car modules once they're confident they're not making things worse. Our cars are not broken. There are things which can use improvement that we've seen signs of being worked on. I'm capable of being patient.

The bigger question becomes what is Ford's intent? Tesla has made people look at a car as an evolving piece of technology. They roll out new features to existing cars. The legacy manufacturers are in the business of enticing owners to buy new cars for new features. Tesla has a relatively small owner base. They see their growth coming from luring ice drivers to electric. That's a lot of potential customers. They don't see existing owners as lost sales if they keep getting updates. Legacy car makers look for brand loyalty and existing owners as the next buyers. Giving those owners reason to stay in their current car becomes a lost sale. It will be interesting to see how the legacy manufacturers adapt to the direction Tesla is pushing the industry. BMW has given hints of renting features/charging for software updates. That's one model to monetize evolving software that comes at the cost of new car sales.

 

[Blanked]

And in my case the dealer refused, saying that they would update whatever the system said they should related to the TSB and wouldn't update anything else because Ford didn't tell them to (and they were afraid it'd cause some other problem they'd have to deal with).

same... So i've only gotten the PAAK update from the dealership. We need to be able to maintain a "known GOOD baseline"...

 

[ncaadam]

The bigger question becomes what is Ford's intent? Tesla has made people look at a car as an evolving piece of technology. They roll out new features to existing cars. The legacy manufacturers are in the business of enticing owners to buy new cars for new features. Tesla has a relatively small owner base. They see their growth coming from luring ice drivers to electric. That's a lot of potential customers. They don't see existing owners as lost sales if they keep getting updates. Legacy car makers look for brand loyalty and existing owners as the next buyers. Giving those owners reason to stay in their current car becomes a lost sale. It will be interesting to see how the legacy manufacturers adapt to the direction Tesla is pushing the industry. BMW has given hints of renting features/charging for software updates. That's one model to monetize evolving software that comes at the cost of new car sales.

This is a very interesting topic ive thought about myself, but have not discussed with really anyone else. I guess I am one of the customers with the Tesla mindset you’re mentioning.

I have literally no brand loyalty to Ford. Have never even owned a Ford. The car being a Ford had practically zero input into me buying the car except for the fact that they have a long-standing reputation (and they “didn’t take the bail out” ?.) The features of the car and the car itself is why I bought it. Had I known the obviously minor infotainment bugs wouldn’t get fixed in the first half year of it being out, would I have still bought it, probably. But it would’ve had an impact on my decision, for sure.

it is just an interesting topic you bring up and I’ve often wondered how the big OEMs are going to adapt (or even if they’ll feel the need to)

Sponsored