Programmers - why so many bugs?

[azerik]

#bestshoring
Also known as where to get as many lines of code as cheap as possible. Been going on since the 2000's.

Sponsored

 

[TheSeg]

Disclaimer: this is external visibility, there's lots of things that could be true internally, but these are best guesses based on my time in hardware/software platform development.

You said just about everything I would say, including starting with this as external visibility. Some notes to add:

* There's no dead-simple way to make a Mach-E match a known configuration.

WIth most electronics, one can to a clean install of the OS and software. You may loose settings and other personalizations, but it will standardize the device to a known configuration. Without going deep in the details, there's no way one can service a Mach-E in this way. It's a process that's arcane and hard for even Ford service techs to understand. I personally had to bring documentation on how a service tech improperly attempted to update my Mach-E. Many others are not as knowledgeable or have the resources to do that (and they shouldn't have to!).

This means that very few cars on the road match whatever testing Ford conducts on the software or firmware. Whatever efforts are being made in continuous integration (CI) or testing, it doesn't begin to match what's on the road currently. Bugs that can't be reproduced in a lab setting. The configurations are too vast and varied to be testable.


My dream: As a consumer, I could download the software of my car onto a USB stick, plug the car into a Level 2 charger, and do a full car reinstall from the USB stick. Just like the process for a PlayStation. This is different from an OTA update; this is checking and reinstalling everything that's updatable to the latest as tested and released by Ford. Sure, it may take an hour or three, but it would be comprehensive or identify exactly what hardware faults are happening.

This would advantage service centers by making the process of updating the car much less resource intensive and quicker to diagnose issues. First step would be updating to known release version of the software, then see if the problem is fixed. If it's fixed, then the tech didn't babysit a car updating for 1 - 8+ hours and was able to work on more cars that day. If not, then the real diagnostics can begin as the technician can compare to a known configuration.

 

[MachLovin]

As someone who works in software but not a programmer it is not their fault.
Programmers: We can have all the bugs fixed in the new release which doesnt add any features, will take us 6 months.
Some Uniformed executive: Ok, how about we push the release up to 1 month from now, no one cares about bugs, but just put something new so there is excitement
Programmers: Please let us just do our job, can you hear us....
Some Uniformed executive: Perfect, just tweeted out about the release, we ship in 3 weeks, just cut out the testing timeline.....

ACCURATE!

 

[bp99]

I totally get the explanations that certain things are higher priority. Just seems like after two years even lower priority stuff should get some attention.

I totally get the explanation that some things we view as bugs, the engineers might view as a feature. It's just tough to envision why the engineers would intend for...
- the car to always remote start to guest profile

The remote start profile issue is fixed in newer cars. My '21 Job 1 had the issue. My late '22 does not. But I also have a new issue that I suspect is related to the fix.

My guess on many of the issues is that they're integrating components and systems from multiple vendors. Many of these components likely have different timings which are inconsistent. When events come from different components at different times, they might not be handled the same. Especially if a different component which depends on events from another switches order.

For the profile issue, it seemed like the startup sequence was likely checking for a fob/phone in the car too early (before the driver was in the car) rather than when the Start button was physically pressed. In my newer car, it's getting the profile. However, I'm frequently getting an error of "cannot set profile while car is moving' (something along those lines) when I press Start after a remote start despite the seat having already moved. This is with the car in park and my foot on the brake. Moving one part of the sequence (profile checking/setting) later to make it work, changed the timing with something else (making sure the seat does not automatically move when the car is being driven).

If they're following modern development practices, they likely have a slew of unit tests which they depend on to ensure the code works. The problem with relying on unit tests is it does not account for integration between multiple systems. It fakes the interactions to make sure code works as defined. It cannot handle unexpected/missing responses from other components. It cannot handle timing issues. You need much more robust tests to handle timing issues.

It's also very likely none of this code is tested on a complete system (an actual car) where all the systems are running and firing events at random times. It's either all stubbed components or some integration in an emulator (where many events still must be faked). It becomes more complex when a vendor (or different team) updates their component and the system needs to have integration regression testing done. Emulators and stubbed unit testing can only test so much when dependent components change.

The issue here is time and cost. For critical systems like medical devices and space craft, the consequences are greater and more time and money is put in to ensure a near flawless system. For non-critical things like infotainment in a car, the risk/reward curve is much different and the budget is lower.

 

[azerik]

Yeah this stuff is surely emulated through automated testing. That's part of the problem. They can't actually recreate a pair of phones walking up at different speeds, with different profiles in a purse and in a pocket. Passenger reaches door first pushed button is PaaK opens door. Driver shows up second later, door opens, profile ignored. Range from phone, first activated anything etc all comes into play. They should. And I say Should very lightly as privacy is 'key' now days even though it might be required for a feature to work, should be collecting these logs from the cars and using the cars as the debuggers. At least post release. Don't base features off a beta test of pushing something out then finding out how it works. Netflix Apple etc do this, BUT they RELIABLY push it out and collect, sift and fix things based on those findings. Ford is many a step behind being able to even try to do this.

 

[mdolan92869]

As someone who works in software but not a programmer it is not their fault.
Programmers: We can have all the bugs fixed in the new release which doesnt add any features, will take us 6 months.
Some Uniformed executive: Ok, how about we push the release up to 1 month from now, no one cares about bugs, but just put something new so there is excitement
Programmers: Please let us just do our job, can you hear us....
Some Uniformed executive: Perfect, just tweeted out about the release, we ship in 3 weeks, just cut out the testing timeline.....

I know what you meant, but when I read Uniformed (not uninformed like I think you meant), I pictured General Patton standing there, all dressed up lecturing all the engineers.

You also forgot
Salesperson: I just promised feature X to a customer for next quarter. How long will it take you to get them a beta? What do you mean you want a detailed description of feature X? Just do your job and implement it. Don't make me do all the work!

 

[Mirak]

The remote start profile issue is fixed in newer cars. My '21 Job 1 had the issue. My late '22 does not. But I also have a new issue that I suspect is related to the fix.

I did not know that had been fixed with more recent models. It would make sense that part of the problem is different builds / hardware. But that doesn’t make me super optimistic as a 2021 owner. Ford hasn’t even enabled the electronic frunk release for us yet. I don’t really care much about the feature, but I worry about lack of support and feature parity.

 

[AhardFSU]

You also forgot
Salesperson: I just promised feature X to a customer for next quarter. How long will it take you to get them a beta? What do you mean you want a detailed description of feature X? Just do your job and implement it. Don't make me do all the work!

Yeah in a company the size of Ford, some salesperson isn't going to have any influence on which software items are tested or included based on some sort of promise being made to some rando customer. That's not how any of this works.

 

[AlanJ]

i agree that excessive commenting out of code makes the code hard to read, but if they are using compiled languages the commented code is not in the binary and the car’s computer does not have to (and simply can not) load it or execute it. the compiler ignores those lines and no machine instructions are emitted for the commented code.

interpreted languages are a different story - the interpreter sees those lines at runtime but in theory should do nothing when they are encountered.

Totally, but it does increase the file size regardless which sucks. Just way more optimal to get out of the commenting habit. It’s a pet peeve of mine when I see code.

 

[astrorob]

Yeah in a company the size of Ford, some salesperson isn't going to have any influence on which software items are tested or included based on some sort of promise being made to some rando customer. That's not how any of this works.

yeah i doubt this happens at ford. in some businesses though marketing/salespeople regularly do this and then go harangue engineers for that feature. in small (desperate) companies this is rampant. in larger companies with more well defined management structures the best these salespeople can do is go ask an engineering manager for the feature.

 

[MacherAWD]

I know what you meant, but when I read Uniformed (not uninformed like I think you meant), I pictured General Patton standing there, all dressed up lecturing all the engineers.

You also forgot
Salesperson: I just promised feature X to a customer for next quarter. How long will it take you to get them a beta? What do you mean you want a detailed description of feature X? Just do your job and implement it. Don't make me do all the work!

Haha, salesperson you are right! But sometimes they are the ones who drive innovation, well they sold it, let's figure out if it's possible.

 

[MacherAWD]

yeah i doubt this happens at ford. in some businesses though marketing/salespeople regularly do this and then go harangue engineers for that feature. in small (desperate) companies this is rampant. in larger companies with more well defined management structures the best these salespeople can do is go ask an engineering manager for the feature.

Software implementation I always start, ok so what do you think you bought? And what do you actually need?

 

[woody]

Because if there weren’t any bugs at release the programmer wouldn’t have a job tomorrow.

Y2K bug all over and over again. Was driving then, and my programmer passengers would brag (cab drivers are stupid so you can say anything in the back seat of the taxi cab) about the 2000 bugs they deliberately installed and how much they would make on the "fix" - as in very comfortable immediate retirement $$$.
The beat goes on.

 

[Chuck]

I know I am probably going to regret the avalanche of nerdery that is about to ensue, but can anybody explain to a non-programmer why Ford is having so many bugs with SYNC, and why it is so slow to fix those bugs?

There are countless threads about (1) bugs that still exist after two years, and (2) how new releases not only fail to fix old bugs but introduce new ones.

I guess the first question is, is SYNC inordinately buggy, or is this to be expected and accepted?

What do you think is the reason for the bugginess? To my feeble mind, once you create a stable platform, you should be able to add features without effing up the stuff that already works, but that doesn’t seem to be the case with Ford. Yes? No?

And then there are the bugs that Ford has known about for years but hasn’t fixed. Why the delay? Is it really so hard to fix bugs that these cannot be prioritized?

Is it possible that part of the problem is outsourcing portions of software development to different countries / languages? I don’t even know that Ford does that, but I assume that they do. I remember when NASA crashed that probe into Mars because they failed to convert imperial to metric, for example.

Or is this a “you get what you pay for” situation and Ford hasn’t adequately invested in software dev?

Any insight for laymen would be appreciated.

I've been a manager in development for decades. My theory is that any program that has more than 3 lines of code in it has a bug in it. We just haven't found it yet. We can have some code in our corporate systems that has been running perfectly for decades and then a weird combination of occurrences can occur to expose a bug.

There's no such thing as bug free code. Microsoft used to have a rule that they wouldn't release versions of Windows until they got the known major bugs to under 500.

 

[RonTCat]

You said just about everything I would say, including starting with this as external visibility. Some notes to add:

* There's no dead-simple way to make a Mach-E match a known configuration.

WIth most electronics, one can to a clean install of the OS and software. You may loose settings and other personalizations, but it will standardize the device to a known configuration. Without going deep in the details, there's no way one can service a Mach-E in this way. It's a process that's arcane and hard for even Ford service techs to understand. I personally had to bring documentation on how a service tech improperly attempted to update my Mach-E. Many others are not as knowledgeable or have the resources to do that (and they shouldn't have to!).

This means that very few cars on the road match whatever testing Ford conducts on the software or firmware. Whatever efforts are being made in continuous integration (CI) or testing, it doesn't begin to match what's on the road currently. Bugs that can't be reproduced in a lab setting. The configurations are too vast and varied to be testable.


My dream: As a consumer, I could download the software of my car onto a USB stick, plug the car into a Level 2 charger, and do a full car reinstall from the USB stick. Just like the process for a PlayStation. This is different from an OTA update; this is checking and reinstalling everything that's updatable to the latest as tested and released by Ford. Sure, it may take an hour or three, but it would be comprehensive or identify exactly what hardware faults are happening.

This would advantage service centers by making the process of updating the car much less resource intensive and quicker to diagnose issues. First step would be updating to known release version of the software, then see if the problem is fixed. If it's fixed, then the tech didn't babysit a car updating for 1 - 8+ hours and was able to work on more cars that day. If not, then the real diagnostics can begin as the technician can compare to a known configuration.

The "dream" will likely be achievable in the next generation vehicle architecture. The one in the Mach-E has far too much "technical hardware debt" to ever make this possible. The vehicle effectively has 50 playstations (sans the graphics power) trying to work together. If each has only one edge case bug... well, you see why we are where we are.

Other posts lists bullet points about how to develop and test software. All of that exists within Ford... but with the technical debt of this vehicle, you can throw, say, Meta/Facebook amounts of heads at this car, and you would not see an appreciable difference in the amount of bugs in the final product. There are more lines of code in this vehicle than exist in ALL of Meta/Facebooks products.

Sponsored