Programmers - why so many bugs?

[azerik]

The vehicle effectively has 50 playstations

These Playstations:

Not this Playstation:

 

[Blue highway]

You said just about everything I would say, including starting with this as external visibility. Some notes to add:

* There's no dead-simple way to make a Mach-E match a known configuration.

WIth most electronics, one can to a clean install of the OS and software. You may loose settings and other personalizations, but it will standardize the device to a known configuration. Without going deep in the details, there's no way one can service a Mach-E in this way. It's a process that's arcane and hard for even Ford service techs to understand. I personally had to bring documentation on how a service tech improperly attempted to update my Mach-E. Many others are not as knowledgeable or have the resources to do that (and they shouldn't have to!).

This means that very few cars on the road match whatever testing Ford conducts on the software or firmware. Whatever efforts are being made in continuous integration (CI) or testing, it doesn't begin to match what's on the road currently. Bugs that can't be reproduced in a lab setting. The configurations are too vast and varied to be testable.


My dream: As a consumer, I could download the software of my car onto a USB stick, plug the car into a Level 2 charger, and do a full car reinstall from the USB stick. Just like the process for a PlayStation. This is different from an OTA update; this is checking and reinstalling everything that's updatable to the latest as tested and released by Ford. Sure, it may take an hour or three, but it would be comprehensive or identify exactly what hardware faults are happening.

This would advantage service centers by making the process of updating the car much less resource intensive and quicker to diagnose issues. First step would be updating to known release version of the software, then see if the problem is fixed. If it's fixed, then the tech didn't babysit a car updating for 1 - 8+ hours and was able to work on more cars that day. If not, then the real diagnostics can begin as the technician can compare to a known configuration.

I agree that getting to a known configuration of all software seems like an obvious step forward... yes there are lots of modules, but instead of being baffled by bugs that arise from edge cases, why not do this? I'll bet all the code in the car could fit in a 250GB flash and if not, it would certainly fit on my PC.

 

[Blue highway]

The "dream" will likely be achievable in the next generation vehicle architecture. The one in the Mach-E has far too much "technical hardware debt" to ever make this possible. The vehicle effectively has 50 playstations (sans the graphics power) trying to work together. If each has only one edge case bug... well, you see why we are where we are.

Other posts lists bullet points about how to develop and test software. All of that exists within Ford... but with the technical debt of this vehicle, you can throw, say, Meta/Facebook amounts of heads at this car, and you would not see an appreciable difference in the amount of bugs in the final product. There are more lines of code in this vehicle than exist in ALL of Meta/Facebooks products.

I'm sure this this is true... but I hold out hope there is some noticeable progress on bug fix before all focus shifts to the new architecture... which has to be pretty soon. 2024 at the latest?

 

[bp99]

Totally, but it does increase the file size regardless which sucks. Just way more optimal to get out of the commenting habit. It’s a pet peeve of mine when I see code.

There should never be commented out code. That's what source control is for. Leaving a short comment about removing code if there's concern it might need to be put back is fine.

For non-compiled code, strip all the comments as part of release packaging. There's no reason for comments to be in production.

 

[Rt1AWD]

Do they actually have a bug reporting system? I've never seen one. Has anybody reported a bug officially and got the ticket number assigned?

 

[leehinde]

Who do think is beta testing for them outside of the team? Could be all the coders are driving 'real' mustangs.

 

[Fremont Kid]

As someone who works in software but not a programmer it is not their fault.
Programmers: We can have all the bugs fixed in the new release which doesnt add any features, will take us 6 months.
Some Uniformed executive: Ok, how about we push the release up to 1 month from now, no one cares about bugs, but just put something new so there is excitement
Programmers: Please let us just do our job, can you hear us....
Some Uniformed executive: Perfect, just tweeted out about the release, we ship in 3 weeks, just cut out the testing timeline.....

Certainly a Dilbert perspective.

 

[ThatGuyLando]

Do they actually have a bug reporting system? I've never seen one. Has anybody reported a bug officially and got the ticket number assigned?

No they just take your Vin and info an promise to get back to you...

 

[Rt1AWD]

Who do think is beta testing for them outside of the team? Could be all the coders are driving 'real' mustangs.

I'm not sure they are collecting and analyzing the driving data from the users. I haven't seen any evidence of that. That is why I'm asking.

 

[ThatGuyLando]

I'm sure this this is true... but I hold out hope there is some noticeable progress on bug fix before all focus shifts to the new architecture... which has to be pretty soon. 2024 at the latest?

I doubt they will make the hardware any better than the bare minimum to get things running and looking good. They seem okay with having the most recalls of any brand as long as their sales numbers are up.

 

[Fremont Kid]

I have been writing software systems for decades. With millions of lines of code running behind the scenes, think it is a miracle that there is software with so few bugs The complexity is mind-boggling. When a few packages are glued together via APIs it is very hard to cover all the corner cases.

Compared to my previous EVs, the MME does seem better with fewer random crashes and more reliable CarPlay. My main complaint is very unreliable PAAK, which is likely a Bluetooth hardware/firmware issue.

Compared to the GM app, the FordPass app is clumsy but it does work quite reliably. When PAAK fails, clicking unlock on the Apple Watch app is the fastest way to open the car.

My Ford Pass app experience is not as positive. One problem is that app settings seem to synchronize out to a centralized server, rather than local settings being managed just between the app and the car. Creates many synchronization problems.

 

[Fremont Kid]

You said just about everything I would say, including starting with this as external visibility. Some notes to add:

* There's no dead-simple way to make a Mach-E match a known configuration.

WIth most electronics, one can to a clean install of the OS and software. You may loose settings and other personalizations, but it will standardize the device to a known configuration. Without going deep in the details, there's no way one can service a Mach-E in this way. It's a process that's arcane and hard for even Ford service techs to understand. I personally had to bring documentation on how a service tech improperly attempted to update my Mach-E. Many others are not as knowledgeable or have the resources to do that (and they shouldn't have to!).

This means that very few cars on the road match whatever testing Ford conducts on the software or firmware. Whatever efforts are being made in continuous integration (CI) or testing, it doesn't begin to match what's on the road currently. Bugs that can't be reproduced in a lab setting. The configurations are too vast and varied to be testable.


My dream: As a consumer, I could download the software of my car onto a USB stick, plug the car into a Level 2 charger, and do a full car reinstall from the USB stick. Just like the process for a PlayStation. This is different from an OTA update; this is checking and reinstalling everything that's updatable to the latest as tested and released by Ford. Sure, it may take an hour or three, but it would be comprehensive or identify exactly what hardware faults are happening.

This would advantage service centers by making the process of updating the car much less resource intensive and quicker to diagnose issues. First step would be updating to known release version of the software, then see if the problem is fixed. If it's fixed, then the tech didn't babysit a car updating for 1 - 8+ hours and was able to work on more cars that day. If not, then the real diagnostics can begin as the technician can compare to a known configuration.

I suspect the code is far too large to store on a USB stick.

 

[Fremont Kid]

I'm sure this this is true... but I hold out hope there is some noticeable progress on bug fix before all focus shifts to the new architecture... which has to be pretty soon. 2024 at the latest?

I like thinking that the architecture may be revised.

 

[Rt1AWD]

No they just take your Vin and info an promise to get back to you...

So how can they fix the bugs if they don't even have a way of collecting info about them.

 

[DevSecOps]

For non-compiled code, strip all the comments as part of release packaging. There's no reason for comments to be in production.

There's obviously some wordpress designers mixed in here that have never seen code compiled. Comments should not be in compiled code. Additionally, the size of a compiled package is far less than the raw code. 100% agree with you here.

You can pick and pull parts of this thread to assume what's going on at Ford and I'm sure some of it is spot on. There's some of us on this forum who have worked for very large companies (I know a couple people from Meta here and others who I worked with at Oracle). I'm sure they would likely agree that without guessing at all, there's a saying in the industry "Big companies move slow", which is why they buy startups who can push stuff out at a rapid pace. I know this because I founded a software company (2016) that was purchased by a big slow company (2019), and I was an active developer and product owner (NodeJS/Angular). There are exceptions, mostly when they start new divisions within large companies, they often act like a startup.

The level of complexity, not in the code, but in the bloat of people, meeting the quotas for diversity or based on degrees instead of hiring people based on skill, lead to more management of people and less people doing what they are good at doing. My best developers were extremely motivated talent out of highschool over college grads who just picked software because they didn't know what they wanted to do. Work for a big tech company and you'll see how hard it is to get anything shipped. Work at a software startup and you'll see a product owner yell at a developer to fix something, then it's immediately sent to the QA team, tested and integrated within minutes/hours. I've seen both sides and it's night and day. These difference can also be correlated to the ideologies of waterfall vs agile.

So, we could sit here and armchair testing, integration, developers, lack of bug reporting capabilities, poor feedback channels, cultural issues, but what we know for sure is that big companies just move slow (and break shit in the process). If Ford saw an issue on this forum and we had a fix the next day, we would all be happy. There's always bugs, as other's have said, they just aren't nimble enough to satisfy us.

Sponsored