Programmers - why so many bugs?

RWG

Well-Known Member
First Name
Randal
Joined
Jan 16, 2022
Threads
0
Messages
104
Reaction score
166
Location
Chaska Minnesota
Vehicles
2021 Mach E
Occupation
Retired
Country flag
I suspect some of the bug issues are because FordPass, and associated software/hardware, are hooked up to literally millions of vehicles. This means, MME software updates must be thoroughly scrutinized so the changes don't mess up other shared vehicle features. This means, that any software updates on a vehicle have to be compatible with FordPass software, and it means that software updates on FordPass need to be compatible on other vehicles as well. It is kinda like playing Waca Mole, i.e. fix something over there and then some thing pops up broken over here. Let's not forget, every software upgrade also gets scrutinized for security issues / threats too, which adds more complications.

With all of the chip technology in the cars, it just makes the software process more tedious/complex/confusing, couple that with the consumer tendencies to want "cool" features, just adds to more complexity. The car companies are all scrambling to offer more and more "cool" features, driven by software. It is the sign of the times.
Sponsored

 

Blue highway

Well-Known Member
First Name
Steve
Joined
Oct 15, 2021
Threads
4
Messages
2,104
Reaction score
3,236
Location
Oregon
Vehicles
Mach E Premium SR RWD
Country flag
In my 20+ years in the business, it ultimately comes back to priorities - where do the resources go, and what are they working on.

Development teams could spend 100% of their time fixing bugs and ship zero new features (not usually a popular choice) or 100% of their time delivering new features to a system that gets increasingly unstable, and at some point will probably fall over completely.

Where you are in that continuum all depends on the priorities of Management / Product Owners. The vast majority of the time it is a delicate balance between the two.

For armchair quarterbacks out there it seems like an obvious thing to do, but these systems are, to paraphrase South Boston, wicked complex.

Personally I'm pretty happy with the Ford software - maybe I cut them way more slack then the average person, but I think given the task at hand, they are pulling it off.

Consider the Audi Q4 e-tron, which has been out for two years, is due for a dealer-only software update to allow for...wait for it...OTA updates. Other new BEVs out there with OTA capabilities have received only a handful of updates in the past year. Tesla of course has nailed it, but they've had time to sort it out.

All in all, Ford ain't doin' too badly.
two things are true in my mind...

1) Ford is making strides in OTA... a year ago they couldn't really do them at all, now they mostly work. Nobody but Tesla is in this league at the minute.

2)Ford is still king of the pigs... leading in a race of really abysmal performers. https://electrek.co/2022/06/07/over-the-air-updates-how-does-each-ev-automaker-compare/


The 4.2x OTA series of updates seem to be bug fixes... no new features, just fixes. This is good (overdue, but still good)
 

Hammered

Well-Known Member
Joined
Oct 4, 2022
Threads
26
Messages
1,228
Reaction score
1,093
Location
SE US
Vehicles
2022 PB F150, MME GTPE
Country flag
My dream: As a consumer, I could download the software of my car onto a USB stick, plug the car into a Level 2 charger, and do a full car reinstall from the USB stick...
You can flash new software on to just about anything in the car with FDRS.
 

TheSeg

Well-Known Member
First Name
Seg
Joined
Oct 5, 2020
Threads
9
Messages
284
Reaction score
646
Location
San Francisco Bay Area, CA
Website
seg.fyi
Vehicles
2021 Mach-E Premium EB-RWD (Built: Feb 2021)
Country flag
You can flash new software on to just about anything in the car with FDRS.
Yes, I guess, but you need:
  • A J2534 Interface ($550+)
  • A laptop running Windows ($250+)
  • A 12 Volt Power Supply ($100+)
  • An FDRS License ($50 - $900)
  • A USB Stick ($22+)
  • The knowledge to run FDRS.
  • Removing the frunk panels to connect the batter to the power supply.
  • The hours of time it takes to watch the status and initiate each step of each update.
What I want is the simplicity of a user-serviceable approach. No extra software to buy, no extra hardware to buy.
  • Download a file (albeit it very large) from any type of computer, Windows, macOS, or *nix.
  • Copy file to a USB drive ($22+).
  • Plug the car into a Level 2 charger.
  • Put the USB stick into the existing console port.
  • Initiating a full reinstall of all installable components (likely from the car's settings panel).
From there the car does a fresh install without the user babysitting to keep hitting buttons for the next component. It would still take a while, but it will do it all once executing the update.
 


Hammered

Well-Known Member
Joined
Oct 4, 2022
Threads
26
Messages
1,228
Reaction score
1,093
Location
SE US
Vehicles
2022 PB F150, MME GTPE
Country flag
Yes, I guess, but you need:
  • A J2534 Interface ($550+)
  • A laptop running Windows ($250+)
  • A 12 Volt Power Supply ($100+)
  • An FDRS License ($50 - $900)
  • A USB Stick ($22+)
  • The knowledge to run FDRS.
  • Removing the frunk panels to connect the batter to the power supply.
  • The hours of time it takes to watch the status and initiate each step of each update.
What I want is the simplicity of a user-serviceable approach. No extra software to buy, no extra hardware to buy.
  • Download a file (albeit it very large) from any type of computer, Windows, macOS, or *nix.
  • Copy file to a USB drive ($22+).
  • Plug the car into a Level 2 charger.
  • Put the USB stick into the existing console port.
  • Initiating a full reinstall of all installable components (likely from the car's settings panel).
From there the car does a fresh install without the user babysitting to keep hitting buttons for the next component. It would still take a while, but it will do it all once executing the update.
I'm a user and have those 🤷‍♂️
 

Shayne

Well-Known Member
Joined
Aug 9, 2020
Threads
18
Messages
3,342
Reaction score
2,489
Location
Northern Ontario Canada
Vehicles
2021 MME4x Prem
Occupation
Retired
Country flag
I'm a user and have those 🤷‍♂️
Have what? Have a car that can OTA update? A car you can download the install check the SHA copy it to a MBR formatted usb and install it sitting in the car? Or a very expensive annual subscription? Spending 1000's to update a car purchased to be OTA updatable does not make much sense to some. Just would not sit right and just because we can install does not mean we want to pay to do it. Not as understood at purchase anyway. 🤷‍♂️ I still have hope in OTA but since much appears to be a usb thru FDRS I would go with just getting rid of the scam FDRS.
 

MellowJohnny

Well-Known Member
First Name
Christian
Joined
Nov 16, 2021
Threads
69
Messages
1,258
Reaction score
2,001
Location
YYZ
Vehicles
2022 Mach-E Premium AWD
Occupation
Solution Architect
Country flag
Thanks for that link - great article. I was going to make a similar point - not all OTA capabilities are created equally - I've seen lots of BEV OTA updates right now limited to things "Infotainment" systems and maps (Hyundai), not more useful, 2nd level stuff like updates to core systems, like braking or HVAC etc. Maybe it's coming with the other guys, but Ford pushing updates to improve blended braking, charging curve, HVAC & ADAS (crossing fingers) is to me the coolest part of OTA.
 
Last edited:

Fremont Kid

Well-Known Member
First Name
Steve
Joined
Oct 8, 2022
Threads
8
Messages
443
Reaction score
298
Location
Colorado Springs, CO
Vehicles
2022 Mustang Mach e Premium AWD
Occupation
Retired IT
Country flag
I worked in software my whole working life, and I find a lot of the Ford user facing software frequently annoying. One of the basic tenets of user facing software is that, when you give users the opportunity to 'customize' their settings, the data that supports those settings is NOT YOURS TO PLAY WITH!!!!!!

I don't know how may times I have gotten into the car and found that my radio presets are totally different than they were when I parked it. Or the seat temperature. Or the drive mode. Or the charge schedule. (that one is particularly irksome because a few weeks ago I was taking a longer drive than usual and after coming home with 50% charge, the next morning instead of finding my specified 90% charge, I found it had done nothing!) Need I go on?

This is called user experience. And like it or not, it is soon going to be conflated with driving characteristics as a criterion for purchase.

When you design the software, the data that defines the user's preferences should be separate from any upgrades you perform! I suspect that the programmers that were recruited to this project came from teams that managed car performance management, which does not have to consider the user. But they need to take the user into account. Right now, while I am fairly happy with my Mach-E, I would not recommend it to friends without major caveats. Caveats that - as a former software engineer - are bewildering!
My only qualification is characterizing the software engineers' experience. I was in the systems department of a very large company. We used 'architecture' rules to define software features and behavior. Security architecture. Network architecture. Development architecture. Data architecture - to your exact and accurate point. Did not prevent all problems, but certainly helped.

Your conclusions may be valid, just that Ford's development environment may not be as mature as needed - which then appears to us as software developer issues.
 

Fremont Kid

Well-Known Member
First Name
Steve
Joined
Oct 8, 2022
Threads
8
Messages
443
Reaction score
298
Location
Colorado Springs, CO
Vehicles
2022 Mustang Mach e Premium AWD
Occupation
Retired IT
Country flag
I suspect some of the bug issues are because FordPass, and associated software/hardware, are hooked up to literally millions of vehicles. This means, MME software updates must be thoroughly scrutinized so the changes don't mess up other shared vehicle features. This means, that any software updates on a vehicle have to be compatible with FordPass software, and it means that software updates on FordPass need to be compatible on other vehicles as well. It is kinda like playing Waca Mole, i.e. fix something over there and then some thing pops up broken over here. Let's not forget, every software upgrade also gets scrutinized for security issues / threats too, which adds more complications.

With all of the chip technology in the cars, it just makes the software process more tedious/complex/confusing, couple that with the consumer tendencies to want "cool" features, just adds to more complexity. The car companies are all scrambling to offer more and more "cool" features, driven by software. It is the sign of the times.
To your point, this is why software must be architected so that essential functions are either decoupled or loosely coupled from other functions. These issues reflect the software architecture. Ford simply is not disclosing enough information that allows owners to understand where their software is strong and stable.

I really think Ford's EV software is architected better than we know, but still not good enough. Think about your exact point. If the software did not have a minimally coherent underlying architecture, all the software functions and updates across millions of vehicles would be chaotic and unmanageable - probably perform much worse than we experience.
 

Fremont Kid

Well-Known Member
First Name
Steve
Joined
Oct 8, 2022
Threads
8
Messages
443
Reaction score
298
Location
Colorado Springs, CO
Vehicles
2022 Mustang Mach e Premium AWD
Occupation
Retired IT
Country flag
Thanks for that link - great article. I was going to make a similar point - not all OTA capabilities are created equally - I've seen lots of BEV OTA updates right now limited to things "Infotainment" systems and maps (Hyundai), not more useful, 2nd level stuff like updates to core systems, like braking or HVAC etc. Maybe it's coming with the other guys, but Ford pushing updates to improve blended braking, charging curve, HVAC & ADAS (crossing fingers) is to me the coolest part of OTA.
Thanks for the link. The reliance on software/firmware and their updates I think will require oversight at some point. Consider wrecks that can be attributed to software/firmware, and legal cases to demonstrate that liability resides with the software/firmware. I think at some point automakers and their software development partners will need to file architectures with regulators. Regulators will probably be requested by Congress to publish architectures and best practices that can guide companies.

Astute company liability lawyers and insurance company liability lawyers hopefully are already preparing for this regulatory environment.
 

bbulkow

Well-Known Member
First Name
Brian
Joined
Aug 30, 2022
Threads
13
Messages
325
Reaction score
199
Location
menlo park, california
Vehicles
Honda CRV, Porsche Cayman S
Country flag
Hi. Professional with long industry experience here.

"why do things that work end up broken later"

The SYNC software in the car is a pretty complex beast. It must have real-time capabilities, it has to exist in a limited hardware footprint (cost reduction), it has to drive a lot of pixels, it has to co-exist with both Android Auto and Apple CarPlay.

When things that used to work now break, there are two fundamental reasons: code got "refactored" at an underlying layer to add a feature or fix another bug, or a new bit of code took more resources (memory / CPU) which changed the operating environment.

Testing solves only so many problems, and in my professional experience, teams over rely on testing. Testing is critical, but it breeds complacence. Most tests are functional, and don't test the CPU used or memory used by a particular function call, as one of a large number of examples. Unit tests aren't done in parallel, so the change in a lock structure to fix one bug causes a deadlock in another existing bit of code.

For reasons of reliability, car software and vehicle software generally is a breed different from most other software (medical device software being the other case). The number of tools (eg, languages) which are built for this level of reliability is low. Android itself is not built for this level of reliability (just reboot your phone). Companies that focused on this capability have fallen on hard times (eg, VxWorks). The rarity of this kind of software engineering makes it a difficult skill to hire, and also makes it underappreciated by executives. The second factor tends to be more important than the first.

"why do things take so long"

We've learned that the companies who build software for high reliability environments can build bad internal culture (or destroy a good internal culture that has been built through the painful lessons of the market). They can be influenced by greater software engineering trends (eg, unreliable internet), they can believe they can apply cost reduction. A common model (not saying it's happening at Ford) is a company investing in quality, building generational quality, then having a change in leadership (essentially) looting that long-term value by removing the personnel who built quality. Quality takes a few years to decrease, then for the market to notice it, and management can pocket money during this cycle (essentially, this is the private equity playbook, looking for under appreciated value that can be monetized in the short term), and We've certainly read this in the public press about Boeing, where they had a strong culture of reliability, then then broke that culture for cost savings.

The nature of reliability is building highly reliable software requires time and investment. It requires tool-building as much as it requires feature-building. It requires training the management staff regarding planning - the cost of high reliability engineering and toolsmithing is not usually built in correctly.

Where is Ford in this journey? Over its 100+ year history, it seems Ford has had strong periods and weak periods in terms of quality generally. The quality of SYNC software feels pretty good to me, where "pretty good" isn't great but isn't terrible. I'm not afraid to be in this car (obviously) and I'm willing to BlueCruise.
 

woody

Well-Known Member
Joined
Sep 27, 2020
Threads
0
Messages
778
Reaction score
675
Location
CO
Vehicles
Mustang Mach E, Chevy Bolt
Country flag
If you limit the junk on your phone that you use/install, you eliminate many problems.
I do not worship the master(only recently forced to obtain one because of technology in car, solar, heat pumps, etc.) anyway. That makes it easier on me. When something on the app. does not work w/ ease, I do not use it. Simple. And simple works for me.
When the car gets me from point A to point B safely and without a hitch I am content.
If a gimmick works flawlessly, I would consider employing it.
My car is not a hobby. I use it to get there. Reliably. Not a new idea. I live in the land of the automobile. I now own a couple of them that finally make sense.
I realize for some of us their car(s) are a hobby and I think that is great. Especially the old ones of yesteryore. Memories.
Driving in my car, turn on the radio (do they still have that old technology? They do! How about non-subscription radio stations that actually play music?) play music on the master! It's "free" w/ my carrier's plan.
Have fun
Sponsored

 
 




Top