Programmers - why so many bugs?

[Fremont Kid]

I know I am probably going to regret the avalanche of nerdery that is about to ensue, but can anybody explain to a non-programmer why Ford is having so many bugs with SYNC, and why it is so slow to fix those bugs?

There are countless threads about (1) bugs that still exist after two years, and (2) how new releases not only fail to fix old bugs but introduce new ones.

I guess the first question is, is SYNC inordinately buggy, or is this to be expected and accepted?

What do you think is the reason for the bugginess? To my feeble mind, once you create a stable platform, you should be able to add features without effing up the stuff that already works, but that doesn’t seem to be the case with Ford. Yes? No?

And then there are the bugs that Ford has known about for years but hasn’t fixed. Why the delay? Is it really so hard to fix bugs that these cannot be prioritized?

Is it possible that part of the problem is outsourcing portions of software development to different countries / languages? I don’t even know that Ford does that, but I assume that they do. I remember when NASA crashed that probe into Mars because they failed to convert imperial to metric, for example.

Or is this a “you get what you pay for” situation and Ford hasn’t adequately invested in software dev?

Any insight for laymen would be appreciated.

I read through all of the posts in hopes that I won't duplicate anyone. I hope this helps for lay people who are not programmers. Ford's main problem is that they have not shared aspects of their system architecture with customers, i.e. how it is intended to work. In the old days the automotive companies shared a lot of information. Examples: Which cylinder is really the primary ignition cylinder - often not cylinder 1. How many degrees above or below top dead center the distributor should be synched to provide optimal ignition. Camshaft angles, electrical resistances, and many more. I am completely baffled that Ford will not share even rudimentary parts of their system architecture. Perhaps intellectual property protection?!

Consequently, we as users are all reverse engineering the software based on our experiences. I SUSPECT WE ARE ALL CORRECT TO A DEGREE AND ARE ALL WRONG TO A DEGREE! Not casting blame. Just that without even basic information from Ford we are trying to describe an elephant while blindfolded without knowing it is an elephant.

For lay people: A fundamental aspect of software engineering is to determine critical functions and protect them by not allowing other parts of the software to affect those critical functions. Any part of the software that controls the motors which drive the vehicle should be considered critical. Yet Ford has created 3 different drive modes, so immediately three different optional settings could impact how the motors propel the vehicle. Each drive mode sets colors that are presented on the dash display and the ambient lighting. I could describe how things like variables and parameters work to do this, but I'm trying to keep this description for lay people.

Now extrapolate this example to all of the different functions such as charging settings and departure time settings - are both of these considered charging just because they pull electricity from a charger? If so, they should be part of one critical function. If not, they should be separated as two functions. Other functions include profiles, PAAK, walk away locking, navigation, finding charging stations, etc. The system architecture can become extremely complicated. My perspective is to develop the system architecture as basic as possible for the critical functions, then test, test, test. Second iteration adds a discrete set of additional features, then test, test, test. Especially what is termed regression testing to ensure the already developed critical systems have not been affected negatively. Do this iteratively until all of the functionality of the vehicle is in place.

Now, synchronizing settings between the Ford Pass app, the vehicle and some central server is another issue. Ford seems to want these settings to be sent to and controlled by some authoritative server, which creates contention between which settings are now controlling charging, when charging occurs, etc. I wish Ford would stop this and allow settings between the Ford Pass app and the vehicle to be authoritative. Use a remote server for logging only.

I agree with those who state that software will always have errors, i.e. bugs. However, the initial architecture may also have flaws. Ever wish your house could be rebuilt with a slightly different floor plan or that the water or electrical lines were installed differently? Once built, changing a house is impossible or quite costly. If the architecture is revised then much of the software must be reworked, tested, tested, tested, etc. You probably get the point.

I apologize if this explanation is way too long. Just trying to provide the requested perspective.
All the best.

Sponsored

 

[Deleted member 16048]

There should never be commented out code. That's what source control is for. Leaving a short comment about removing code if there's concern it might need to be put back is fine.

For non-compiled code, strip all the comments as part of release packaging. There's no reason for comments to be in production.

Glad you have that mindset too, sadly that isn't very common elsewhere. :-/

 

[Rt1AWD]

Glad you have that mindset too, sadly that isn't very common elsewhere. :-/

(facepalm) comments are ignored by the compiler

 

[Deleted member 16048]

(facepalm) comments are ignored by the compiler

It doesn't matter, it causes larger file sizes and they shouldn't be in prod environment. That's the attitude that makes code look like a mess. You also missed the quoted section of the post. Clean code = more efficient for the whole system.

 

[azerik]

I get the feeling that most don’t like comments in code. Which is correct as no end user should be reading the production code. I leave all my comments in my commits.

- hated this code in the firstplace
- removed other bits that never worked
- typo’s this shit to piss off Tim
- sending it and getting a sammich

 

[Peugfan]

Honestly a lot of these bugs are inconsistent so it makes it really hard to have steps to reproduce. Along with this, clear logs after reproducing it are not really something they get from end users.

You can never have software be 100% perfect, things crop up a lot. The CarPlay "bug" for example. You're working with a system that is half if not mostly controlled by Apple. The "bug" could be iOS itself.

Now for issues they are aware of, it can come down to how many users are affected and does it pose a security risk. If not, it's going to be deprioritized. On top of that, they need to figure out if it's on specific configureations only or on any. Then continue to isolate the issue out. This takes a lot of time. There are bugs in iOS that have been there for years an Apple has no plans to address them.

Typically everyone blames the car first and not the phone side for any of those phone related functions. My PAAK worked nearly perfect from day 1 in April 2021 with my 5 year old iPhone SE always updated to the latest iOS. 10 months later the door locking/opening functions stopped working. The car never had a NO KEY DETECTED problem. 4 months ago I updated to an IPhone 14. My PAAK resumed working perfectly. I can't say for sure, but it appears the problem was not my Mach E, but my phone.

 

[woody]

1) American management has been a failure for decades and continues to fail.
2) Designers/manufacturers/installers whose product fails must share the blame for the failure.
3) If you cannot or will not do it right DON"T DO IT
4) dumping the blame for failure is a copout.
5) claiming "lay people" are stupid and are to blame is is irresponsible - they paid for it and it does not work. Simple: the customer was cheated. It may be the way, but it is still wrong.

Since the Chinese can steal our technology and do it right.........

 

[Rt1AWD]

It doesn't matter, it causes larger file sizes and they shouldn't be in prod environment. That's the attitude that makes code look like a mess. You also missed the quoted section of the post. Clean code = more efficient for the whole system.

Clean code is more about reuse then comments. "Dirty code" is stupid/duplicate code and stupid comments

 

[Blue highway]

I like thinking that the architecture may be revised.

changing the architecture of the car does nothing for current owners... if anything it hurts us.

This is because resources will shift to working on the Android Automotive architecture and the current Blackberry architecture will be largely left as is.

Ford confirmed to some tech journalists that current cars can not be migrated to the new operating system

 

[bwr1338]

I know I am probably going to regret the avalanche of nerdery that is about to ensue, but can anybody explain to a non-programmer why Ford is having so many bugs with SYNC, and why it is so slow to fix those bugs?

There are countless threads about (1) bugs that still exist after two years, and (2) how new releases not only fail to fix old bugs but introduce new ones.

I guess the first question is, is SYNC inordinately buggy, or is this to be expected and accepted?

What do you think is the reason for the bugginess? To my feeble mind, once you create a stable platform, you should be able to add features without effing up the stuff that already works, but that doesn’t seem to be the case with Ford. Yes? No?

And then there are the bugs that Ford has known about for years but hasn’t fixed. Why the delay? Is it really so hard to fix bugs that these cannot be prioritized?

Is it possible that part of the problem is outsourcing portions of software development to different countries / languages? I don’t even know that Ford does that, but I assume that they do. I remember when NASA crashed that probe into Mars because they failed to convert imperial to metric, for example.

Or is this a “you get what you pay for” situation and Ford hasn’t adequately invested in software dev?

Any insight for laymen would be appreciated.

I worked my entire life in the the technology industry. I was not a programmer but worked closely with them. I bought a First Edition the day it was released and knew I was going to be an early adopter and would have to deal with problems as they learned how to make an electric car.

The problem Ford and other old car companies have is that they are not software companies that are use to developing complex software systems and not just software for a module or infotainment system.

The Mach E and all electric cars are computers with wheels. The software controls everything and there in lies the challenge. Some of the variables they need to account for are:
1. Operating system versions
2. Firmware versions of every component or module in the car
3. Every version of every circuit board in the car and on the road. Each hardware change impacts the firmware for that board. This problem grows over time.
4. Updates to purchased software used in the car
5. All the possible combinations of the above

Add to that the companies testing processes across all the variables and the problem grows.

I also think that Ford is making their problems worse by releasing only a few updates with lots of changes and not including release notes that tell us what was changed.

All of us would like to see frequent updates with good release notes and a simple way for users to report back to Ford with any issues they find. The forums are really the only way to report back to Ford and the poor release notes means we all are guessing at what changed and what it impacted. More frequent updates would also increase the users perception of the car as it keeps getting better and the key feature OTA is of real value.

I love my car and driving it even with all the recalls and software issues. I plan to replace this EV in about three years and will see who has the best EV at that time. I hope Ford will be in the running.

 

[Deleted member 16048]

Typically everyone blames the car first and not the phone side for any of those phone related functions. My PAAK worked nearly perfect from day 1 in April 2021 with my 5 year old iPhone SE always updated to the latest iOS. 10 months later the door locking/opening functions stopped working. The car never had a NO KEY DETECTED problem. 4 months ago I updated to an IPhone 14. My PAAK resumed working perfectly. I can't say for sure, but it appears the problem was not my Mach E, but my phone.

Exactly, so many moving parts it's easy to say it's one sides fault. Personally I have always told people when moving to a new iPhone, so start clean and not restore from backup. It brings bad data over from the old device that could cause new issues.

But yes, some issues people are seeing here may not be the car at all. It's likely iOS.

 

[Fremont Kid]

changing the architecture of the car does nothing for current owners... if anything it hurts us.

This is because resources will shift to working on the Android Automotive architecture and the current Blackberry architecture will be largely left as is.

Ford confirmed to some tech journalists that current cars can not be migrated to the new operating system

I have to ask what empirical support leads to this conclusion, and to stating that the architecture is from Blackberry? You may be correct, but I've never heard this before.

Also, allocating resources from one task/project to another has nothing to do with the architecture itself. Your statement seems to conflate factors that are not related. If two people are allocated to working on the architecture while 200 are allocated to working on something else, only affects how fast the work could complete. It does not change the actual structure of the architecture.

Also, from a computer science perspective, as long as the vehicle hardware can process the software, any number of architectures can be used. The Linux operating system can be used on personal computers, rather than Windows. Now operating systems are not software architecture, but are even more basic and fundamental - communicating with the CPU, modems, display functions, etc.

Do you know which publications these tech journalists published these Ford disclosures?

I'll accept your answers then not reply because I'm very far down a path not related to the initial thread question.

 

[agoldman]

I am sure a few more OTAs will really help. The MME is still new for Ford. We just have to give them time.

Did you copy this pots from 2021?

 

[AhardFSU]

I have to ask what empirical support leads to this conclusion, and to stating that the architecture is from Blackberry? You may be correct, but I've never heard this before.

Also, allocating resources from one task/project to another has nothing to do with the architecture itself. Your statement seems to conflate factors that are not related. If two people are allocated to working on the architecture while 200 are allocated to working on something else, only affects how fast the work could complete. It does not change the actual structure of the architecture.

Also, from a computer science perspective, as long as the vehicle hardware can process the software, any number of architectures can be used. The Linux operating system can be used on personal computers, rather than Windows. Now operating systems are not software architecture, but are even more basic and fundamental - communicating with the CPU, modems, display functions, etc.

Do you know which publications these tech journalists published these Ford disclosures?

I'll accept your answers then not reply because I'm very far down a path not related to the initial thread question.

I think that it was in The Verge and a couple of other sites. Ford was very clear that current Mach E’s and Lightings would not get the Android Automative system. Not sure if there was a explanation for why.

 

[Blue highway]

I have to ask what empirical support leads to this conclusion, and to stating that the architecture is from Blackberry? You may be correct, but I've never heard this before.

Also, allocating resources from one task/project to another has nothing to do with the architecture itself. Your statement seems to conflate factors that are not related. If two people are allocated to working on the architecture while 200 are allocated to working on something else, only affects how fast the work could complete. It does not change the actual structure of the architecture.

Also, from a computer science perspective, as long as the vehicle hardware can process the software, any number of architectures can be used. The Linux operating system can be used on personal computers, rather than Windows. Now operating systems are not software architecture, but are even more basic and fundamental - communicating with the CPU, modems, display functions, etc.

Do you know which publications these tech journalists published these Ford disclosures?

I'll accept your answers then not reply because I'm very far down a path not related to the initial thread question.

This is widely known ... here is an article in the popular press. The current system used by Ford is Blackberry QNX, which replaced a Microsoft OS in 2014.

https://9to5google.com/2022/04/26/f...le's biggest partners,delayed by a few months.

Sponsored