Perfection is cost prohibitive. Agreed. But the current "acceptable" rate of failure is clearly costing Ford a significantly higher amount given warrantee-covered repairs vs simply having spent marginally more for hardware that was properly spec'ed given realistic performance demands.
Hi I can clear a couple points here.TLDR: Probably not
Longer version:
Thanks to John Kelly's excellent video we can lift the mystery of the operation of the High Voltage Junction Box (HVJB) in the MME. We can also figure out the likely problem that triggered the recall for the contactor. How? Buckle up and let's get technical!
What is in the HVBJB?
Here is the full schematic that I reverse-engineered from this and several other sources:
Edit: This teardown post also provides additional detail on the components.
The main task of the HVBJB is to distribute the high voltage battery energy and maximize safety. It contains the following major parts:
What is the problem?
- Main Contactor+ which connects the positive battery terminal. All current flows through this contactor.
- A Pre-charge contactor + resistor to prevent sparks during startup. Since Main Contactor+ is the last one to close there will be 380Volt across the 2 pins. The pre-charge contactor closes first, charging the terminal slowly through a 24 Ohm resistor. After the voltage is equalized Main Contactor+ can be closed spark-free. Without the precharge circuit, a huge inrush current would flow. The high current and sparks could potentially weld the contacts, or damage the contacts. All EVs apply this trick.
- Main Contactor- connecting the negative battery terminal. This switches both motors and the HVAC parts.
- A 630A Rear motor + DCFC fuse. This supplies either the powerful rear motor or the negative terminal of the DCFC port.
- A smaller Front Motor Fuse.
- A Cabin HVAC Fuse that protects both the air conditioner compressor and the electric PTC heater. Both these parts heat or cool the cabin as well as the battery. See my earlier post here for details on how that is plumbed.
- A charger contactor with a charger fuse that connects to both the onboard 240VAC->380VDCcharger and the 380DC->12DC charger. This is switched separately, allowing for L2 charging without powering up both motor inverters. More on that later.
- Two beefy DCFC contactors isolate the high voltage pins of the charge ports. They are the same type as the Main Contactors.
The subject of the recall is a malfunction of one of the 4 large contractors. According to Ford: "Direct Current (“DC”) fast charging and repeated wide open pedal events can cause the high voltage battery main contactors to overheat. This overheating may lead to arcing and deformation of the electrical contact surfaces, which can result in an electric relay switch remaining open or a relay switch that welds close from heat. An overheated relay switch that opens while driving can result in a loss of motive power, which can increase the risk of an accident."
What does the contactor look like?
A contactor (AKA a relay) is just an electrically controlled switch. When a 12V low voltage is applied to the primary input, the main contacts close which can conduct a very large current. Cars contain dozens of smaller contactors. The ones that switch the high voltage battery need to be particularly strong. The 4 large jampot-shaped contactors in the HVJB are made in Mexico by TE. This is the datasheet:
Notice that the maximum rated continuous current is 500Amps . We'll come back to that later.
Contactors can fail. With high currents, the heat and sparking may 'weld' the contacts closed. It is called 'stuck closed' and this is what 'welded contacts' inside of this type of TE contactor look like (source):
High heat might deform the plastic of the housing and make the contact 'stuck open'. Either of the failures is monitored by the MME. There is no temperature sensor in the HVJB to avoid overheating.
How does the HVJB operate during normal driving?
Before entering the car all HVJB contactors are open (that is, not conducting). The MME is powered by the 12V battery only. When you press the start button in the MME the following sequence of events will likely be commanded by the computer:
After this, the HVJB will be switched like this, with both main contactors conducting:
- The Main Contactor- (right side) closes. No current can flow yet because Main Contactor+ on the left is still open. The computer commands the inverters, compressor, and heater to be off.
- The computer verifies the voltages to check whether Main Contactor- is indeed closed (0 V across) and Main Contactor+ is open (~380V across). It also verifies that DCFC contactors are indeed open (and not weld shut).
- The pre-charge contactor closes to charge the left bus up to 380V.
- The computer verifies that the voltage on the left bus is high (~380V).
- After that Main Contactor+ closes.
- The pre-charge contactor opens, as its job is done.
- The computer verifies that the voltage on the left bus remains high (~380V) to verify that the contactor is closed. Likely it briefly commands a little load on the bus to check whether voltages remain stable. If not, this could be a contactor issue and it will throw an error.
- The front and rear motor inverters probably run a self-check.
- If all is OK, we are ready to drive!
How much current flows while driving?
Normal steady driving commands about 25kW. This means that a current of (25,000/380=) 66Amps through the two contactors. Maximum regenerative braking is about 70kW, resulting in a reverse current flow of about 185Amps. During pedal-to-the-metal acceleration the peak current through the contactors will be ~900Amps in the GT. In the dual-motor MME it will be about 690A peak.
How does the HVJB operate during DC Fast Charging?
The likely chain of events is as follows after the car is plugged into DCFC:
After that, the HVJB will be switched like this:
- The Main Contactor - (right side) closes. No current flows yet because Main Contactor+ on the left is still open. The computer commands the inverters, compressor, and heater to be off.
- The computer verifies the voltages to check whether Main Contactor- is indeed closed (0 V across) and Main Contactor+ is open (~380V across). It also verifies that DCFC contactors are indeed open (and not weld shut). It will error out if that is not the case. and the car needs to be towed.
- Both DCFC contactors close.
- The computer verifies the proper voltages, verifying that the contactors are indeed closed and that the DCFC charger does not deliver power.
- The pre-charge contactor closes to charge the left bus up to 380V.
- The computer verifies that the voltage on the left bus is high (~380V). If not, the car needs to be towed.
- Main Contactor+ closes.
- The pre-charge contactor opens
- The computer verifies that the voltage on the left bus remains high (~380V).
- The computer commands the right amount of charge voltage and current from the DCFC charger. It constantly verifies that the voltages are as expected.
The DCFC charge current flows through all 4 contactors. If needed, the AC compressor is run to cool the battery, or the PTC heater is run to (pre-) heat the battery.
The battery cooling is needed because of the internal losses in the battery. The parasitic resistance of the battery can be calculated from the voltage drop under load. When flooring the pedal I see a from 380V to 350V with a current of 250A. So the internal resistance is (30/350=0.09Ohm). That is approx. 2kW of heat in the battery during charging that needs to be sucked out.
Do the contactors get hot?
It seems so! The contactor is rated for 500A continuous and 1000A for 1 minute at a hot 85C. Hence the limitations in the GT. According to the datasheet, the internal resistance of the contact is (0.1V/200A =) 0.0005 Ohm maximum. That seems tiny, but at the rated 500A continuous current that would generate (500*500*0.0005=) 125Watt of heat. During DCFC charging at 150kW the current will be about 400A, so the 4 contactors will generate up to 80Watts of heat each. That is pushing it, as all that heat will need to go somewhere. On top is that the primary coil in the contactor also gets quite warm: 650mA minimum hold current at 12V is another 8 Watts of heat.
The 80 Watt heat is not insignificant: it will get as hot as an incandescent light bulb. It does not look like there is any dedicated cooling in the plastic HVJB assembly. The only way it is cooled is indirectly from the battery pack behind it. Hopefully, the actual contact resistance is lower than that.
So it is possible that during DCFC and/or frequent heavy accelerations the contactors get so hot that they break down or age prematurely. At 85C (hot coffee) the rating is 1000W for 1 minute. But given the likely poor heat conductivity, 80W continuous heat could make the contactor much hotter than 85C. So it could get out-of-spec. Most contactors will survive some abuse, but some unlucky ones will melt plastic and break open or weld themselves shut.
What could a software fix do to prevent the contactors from blowing open or welding shut?
There is a tiny possibility that a bug in the software causes the pre-charging contactor to open too late (or not at all). That would degrade Main Contactor+ quickly. The fix for that would be simple. This scenario is quite unlikely given the information we have from the recall.
The software could reduce the system current when the contactors are hot. This inevitably means some performance degradation or longer charge times. The problem is that there is no temperature sensor in the HVJB or anywhere near the contactors. Ford's software would have to guess the heat based on the average current and the battery temperature. I don't think that can be done accurately without being conservative, causing a noticeable performance degradation.
The software might attempt to detect whether a contactor is about to go bad. The problem with that the voltage drop of a loaded contactor is small (<0.1V) compared to the voltage drop of the battery under load (~30V). The voltages on the battery side of the Main Contactors likely cannot be measured accurately enough to detect contact degradation. With any increase of resistance due to a dirty contact, the contactor will get much hotter very quickly due to the high currents. It will blow or weld shut very rapidly. So, I don't think this is feasible.
I could be missing something, but I don't see a way that a software fix can avoid the hardware problem of overheating without some noticeable degradation of the specs.
What could the software fix do?
It does not seem likely that the OTA software fix prevents the problem from occurring. So it probably just mitigates the problem. Let's analyze what could be done:
What about a hardware fix?
- Case 1: One of the Main Contactors is stuck open. No current can flow so the car will be bricked. No way around that.
- Case 2: One of the Main Contactors welded itself shut. This is detected in the startup sequence. Rather than bricking, the MME could be put in a limp-home mode to significantly reduce contactor current. That is likely what Ford's fix does. Since the other Main Contactor can still open there should be no significant safety risk.
- Case 3: One of the Main Contactors opened itself while driving due to overheating, or some discontinuity is detected while driving. This will result in a 'stop safely now' event. Rather than bricking the car in the middle of the road, the software could attempt to restart after a brief cool-down period l. The it can switch to the reduced power limp-home mode. This is likely what happens.
- Case 4: One of DCFC contactors is open. In that case, DCFC is INOP, but the car should otherwise operate normally.
- Case 5: One of the DCFC contactors is welded itself shut. This is a slight safety issue but should not have to result in the car going into limp-home mode.
Any hardware fix would be very expensive for Ford because removing and opening the battery is significant labor and requires skilled people. It would probably run at $3K-$5K per vehicle. I suspect that Ford is monitoring the statistics. They decided that the problem is still rare enough to fix a HVJB on a case-by-case basis. There is no significant safety risk, especially after the software fix.
A permanent solution would be a complete re-design of the HVJB with better cooling and temperature monitoring.
Ford could also replace just the contactors in the HVJB with a better version. This is what the repair seems to do. I suspect that significantly better contactors in the same form factor are not available. Therefore this does not guarantee the problem from re-occurring. It does not seem to be a real manufacturing defect, but rather a structural issue in the design of the HVJB.
How does the MME operate with AC L2 charging?
Both the DC-DC and AC-DC chargers are on a separate (lower-power) contactor. This allows the 12V battery to be re-charged without switching on the entire car. This saves energy because the motor inverters will be off. L2 AC charging is also possible without switching on the entire car:
In that case, only the Main Connector+ and the charger connector need to close. The cooling of the DC-DC converter and the Charger runs entirely of the 12V battery so it does not need HV to be active.
As soon as the battery needs to be heated or cooled the rest of the circuit needs to wake up. In moderate temperatures that is not needed because L2 charging produces little heat.
Hope this helps!
Disclaimer: The above is my speculation based on my research. It might be inaccurate or contain errors.
His description of the precharge is correct. It only charges the bus for matching to close the main contactor. The mains cannot be closed otherwise. Also, the Precharge fhas a timed function to stop to save the resistor from overheating. Should a precharge fail, the controller tries again after a delay for cooling (Around 1-3 minutes, I think 1 min). After a defined number of tries, the battery is bricked.
Thanks for the insights. So if I understand correctly, if any of the contactors weld shut or are stuck open the MME will always be bricked.
My first failure of the HVBJB was a stuck open condition. When cold it would not close, error out and kill the 12V. I would put a charger on it, recharge the 12V and the car would start and run like normal. This happened over and over again each time with the code of contact stuck open and each time charge the 12V and it would go. I would not call that as bricked but dysfunctional. The second time it was stuck closed and was sent it into a limp mode with a bunch of error messages as by that time they knew the design error. Still I would not say bricked.
That is interesting. The symptom seems to indicate a broken charger contactor, which is independent of the big main contactors. With that charger contactor stuck open, it would seem that you cannot use the AC charging as well. But if this is indeed the case, the 12V battery would drain very quickly while driving as well, because only the DC/DC converter charges the battery and supplies all 12V power.
Yes. If the accessory one gets stuck open I think the car will still drive, but the 12V battery will go dead pretty fast.
The accessory one (negative) would make sense but each time the code read positive contact stuck open in which there is just one. ?That is interesting. The symptom seems to indicate a broken charger contactor, which is independent of the big main contactors. With that charger contactor stuck open, it would seem that you cannot use the AC charging as well. But if this is indeed the case, the 12V battery would drain very quickly while driving as well, because only the DC/DC converter charges the battery and supplies all 12V power.
it is a 36Ah 12V battery, and at idle the power draw on the 12V system is ~40A. When running it is more. So that is 1 hour tops, but in practice likely way less. But enough to limp to the dealer...
That is weird indeed. It seems to indicate that the positive main contactor + does close when the 12V battery is fully charged, but not when its voltage is too low. After all, when you were driving the car it must have been closed. That could be a bad 12V battery or a crusty contactor.The accessory one (negative) would make sense but each time the code read positive contact stuck open in which there is just one. ?
If it was stuck open it closed as I drove 100's of miles after recharging the 12V and I would think it would have never lasted for 4 or 5 hour drive it if it was not charging it during the drive. It was a random stuck open and -20C would do it for sure.
If you figure most batteries have less capacity than new (maybe 30 Ah), aren’t fully charged (maybe 70%), and the Peukert effect from a 1.5C discharge rate, I’d say you have about 15 minutes tops.
Yes the 12V Battery will indeed run dead since there is no maintenance source of power.
So from what I am reading, apparently there is a general thought to name the BEC as the HVJB. Interesting. I can absolutely tell though that the array of Main and Secondary (such as charging contactors) contactors is in Fords terms called the BEC (Battery Electrical Center).That is interesting. The symptom seems to indicate a broken charger contactor, which is independent of the big main contactors. With that charger contactor stuck open, it would seem that you cannot use the AC charging as well. But if this is indeed the case, the 12V battery would drain very quickly while driving as well, because only the DC/DC converter charges the battery and supplies all 12V power.
