HVJB Deep Dive: Is there any way that software fixes the problem of overheating contactors?

[Shayne]

From my understanding that happens because the contractors become molten while being driven, then when cooling after power off they weld in an open state.

Open contractors amounts to maybe 10% of the cases I've seen. Closed is much more prominent.

They are all stuck close condition? So they never open when powered off and have been glued together during driving? Not my problem mine is a stuck open in the cold and I only know about it as it does not fire up the dc to dc convert and my 12V battery dies. Charge the 12V battery they appear to unstick as the car operates fine. This is in -28C and nothing getting too hot. Totally different problem just like these failures may not all be due to the same problem? Ford is pointing at the hvjb contacts for a heat problem I think but document a stuck open and closed positive contact for a hvjb replacement? Not sure if I am wearing out contacts but it happen the first month I owned it without too much wear and tear on it unless you can break them really easy?

Sponsored

 

[AZBill]

Obviously we don't drive at WOT for extended periods but it's very possible to have heavy regen for extended periods. Such as Phx-SD for example. Coming down out of the mountains heading into El Cajon is mostly all regen. The SOC actually increases for a good 15-20 miles during that descent. Then DCFC in El Cajon might add in more heat. Just thinking out loud though....
I will be making that trip next month. Might capture some data during that descent and see what it looks like.

I just did that trip. In fact I did my charging stop in El Centro and pulled 153kw maximum, charged for 18 minutes to get to 60% to make it to Carlsbad. Drove up the steep climb right after that then regen down the hill into SD. Car worked just fine. When I got to my destination, Carlsbad, I then did a 15% to 90% charge, which peaked at 133kw.

 

[SnBGC]

Problem when driving downhill: Electric Mustangs run downhill in rows (newsfounded.com)

it's not clear to me what exactly Ford updated in the software.

Not clear to many (or any) of us either.

 

[SnBGC]

I just did that trip. In fact I did my charging stop in El Centro and pulled 153kw maximum, charged for 18 minutes to get to 60% to make it to Carlsbad. Drove up the steep climb right after that then regen down the hill into SD. Car worked just fine. When I got to my destination, Carlsbad, I then did a 15% to 90% charge, which peaked at 133kw.

I would do the same. Charge in El Centro instead of El Cajon. Sounds like that might be less stress on the vehicle so if given a choice that might be preferred. Could also be 6 of 1 or half dozen of the other kinda of thing as well.

 

[AZBill]

I would do the same. Charge in El Centro instead of El Cajon. Sounds like that might be less stress on the vehicle so if given a choice that might be preferred. Could also be 6 of 1 or half dozen of the other kinda of thing as well.

I bet you will find the max regen downhill to be 20-30kw range. That is roughly what I get in my Bolt, which has a nice display for power.

 

[AZBill]

Even if the front and rear motors are not running, supplying 380V to their inverters will lose some energy. I presume that the entire reason that a separate Charger Contactor exists is to avoid that energy loss. It would have been cheaper and simpler to switch it all with the Main Contactors. Instead, Ford engineers added this extra complication.

The reason for the extra contactors is to eliminate the battery voltage from being present at the CCS connector when the car is powered on. The charging contactors will not engage until the charger tells the car it is connected and ready.

 

[Shayne]

WeberAuto

Thank you, yes the black contactor bypasses the main negative contactor and precharge is not needed.

So the dc/dc not getting power would be limited to the main + and the black box contact? Pre-charge not involved. When I put the car in accessory mode (no brake) is it running through the black box?

WeberAuto
I believe that is correct

An educated maybe and therefore not a for sure.

 

[Flowell]

TLDR: Probably not

Longer version:
Thanks to John Kelly's excellent video we can lift the mystery of the operation of the High Voltage Junction Box (HVJB) in the MME. We can also figure out the likely problem that triggered the recall for the contactor. How? Buckle up and let's get technical!

What is in the HVBJB?
Here is the full schematic that I reverse-engineered from this and several other sources:
Edit: This teardown post also provides additional detail on the components.

The main task of the HVBJB is to distribute the high voltage battery energy and maximize safety. It contains the following major parts:

• Main Contactor+ which connects the positive battery terminal. All current flows through this contactor.
• A Pre-charge contactor + resistor to prevent sparks during startup. Since Main Contactor+ is the last one to close there will be 380Volt across the 2 pins. The pre-charge contactor closes first, charging the terminal slowly through a 24 Ohm resistor. After the voltage is equalized Main Contactor+ can be closed spark-free. Without the precharge circuit, a huge inrush current would flow. The high current and sparks could potentially weld the contacts, or damage the contacts. All EVs apply this trick.
• Main Contactor- connecting the negative battery terminal. This switches both motors and the HVAC parts.
• A 630A Rear motor + DCFC fuse. This supplies either the powerful rear motor or the negative terminal of the DCFC port.
• A smaller Front Motor Fuse.
• A Cabin HVAC Fuse that protects both the air conditioner compressor and the electric PTC heater. Both these parts heat or cool the cabin as well as the battery. See my earlier post here for details on how that is plumbed.
• A charger contactor with a charger fuse that connects to both the onboard 240VAC->380VDCcharger and the 380DC->12DC charger. This is switched separately, allowing for L2 charging without powering up both motor inverters. More on that later.
• Two beefy DCFC contactors isolate the high voltage pins of the charge ports. They are the same type as the Main Contactors.

What is the problem?
The subject of the recall is a malfunction of one of the 4 large contractors. According to Ford: "Direct Current (“DC”) fast charging and repeated wide open pedal events can cause the high voltage battery main contactors to overheat. This overheating may lead to arcing and deformation of the electrical contact surfaces, which can result in an electric relay switch remaining open or a relay switch that welds close from heat. An overheated relay switch that opens while driving can result in a loss of motive power, which can increase the risk of an accident."

What does the contactor look like?
A contactor (AKA a relay) is just an electrically controlled switch. When a 12V low voltage is applied to the primary input, the main contacts close which can conduct a very large current. Cars contain dozens of smaller contactors. The ones that switch the high voltage battery need to be particularly strong. The 4 large jampot-shaped contactors in the HVJB are made in Mexico by TE. This is the datasheet:

Notice that the maximum rated continuous current is 500Amps . We'll come back to that later.

Contactors can fail. With high currents, the heat and sparking may 'weld' the contacts closed. It is called 'stuck closed' and this is what 'welded contacts' inside of this type of TE contactor look like (source):

High heat might deform the plastic of the housing and make the contact 'stuck open'. Either of the failures is monitored by the MME. There is no temperature sensor in the HVJB to avoid overheating.

How does the HVJB operate during normal driving?
Before entering the car all HVJB contactors are open (that is, not conducting). The MME is powered by the 12V battery only. When you press the start button in the MME the following sequence of events will likely be commanded by the computer:

1. The Main Contactor- (right side) closes. No current can flow yet because Main Contactor+ on the left is still open. The computer commands the inverters, compressor, and heater to be off.
2. The computer verifies the voltages to check whether Main Contactor- is indeed closed (0 V across) and Main Contactor+ is open (~380V across). It also verifies that DCFC contactors are indeed open (and not weld shut).
3. The pre-charge contactor closes to charge the left bus up to 380V.
4. The computer verifies that the voltage on the left bus is high (~380V).
5. After that Main Contactor+ closes.
6. The pre-charge contactor opens, as its job is done.
7. The computer verifies that the voltage on the left bus remains high (~380V) to verify that the contactor is closed. Likely it briefly commands a little load on the bus to check whether voltages remain stable. If not, this could be a contactor issue and it will throw an error.
8. The front and rear motor inverters probably run a self-check.
9. If all is OK, we are ready to drive!

After this, the HVJB will be switched like this, with both main contactors conducting:

How much current flows while driving?
Normal steady driving commands about 25kW. This means that a current of (25,000/380=) 66Amps through the two contactors. Maximum regenerative braking is about 70kW, resulting in a reverse current flow of about 185Amps. During pedal-to-the-metal acceleration the peak current through the contactors will be ~900Amps in the GT. In the dual-motor MME it will be about 690A peak.

How does the HVJB operate during DC Fast Charging?
The likely chain of events is as follows after the car is plugged into DCFC:

1. The Main Contactor - (right side) closes. No current flows yet because Main Contactor+ on the left is still open. The computer commands the inverters, compressor, and heater to be off.
2. The computer verifies the voltages to check whether Main Contactor- is indeed closed (0 V across) and Main Contactor+ is open (~380V across). It also verifies that DCFC contactors are indeed open (and not weld shut). It will error out if that is not the case. and the car needs to be towed.
3. Both DCFC contactors close.
4. The computer verifies the proper voltages, verifying that the contactors are indeed closed and that the DCFC charger does not deliver power.
5. The pre-charge contactor closes to charge the left bus up to 380V.
6. The computer verifies that the voltage on the left bus is high (~380V). If not, the car needs to be towed.
7. Main Contactor+ closes.
8. The pre-charge contactor opens
9. The computer verifies that the voltage on the left bus remains high (~380V).
10. The computer commands the right amount of charge voltage and current from the DCFC charger. It constantly verifies that the voltages are as expected.

After that, the HVJB will be switched like this:

The DCFC charge current flows through all 4 contactors. If needed, the AC compressor is run to cool the battery, or the PTC heater is run to (pre-) heat the battery.
The battery cooling is needed because of the internal losses in the battery. The parasitic resistance of the battery can be calculated from the voltage drop under load. When flooring the pedal I see a from 380V to 350V with a current of 250A. So the internal resistance is (30/350=0.09Ohm). That is approx. 2kW of heat in the battery during charging that needs to be sucked out.

Do the contactors get hot?
It seems so! The contactor is rated for 500A continuous and 1000A for 1 minute at a hot 85C. Hence the limitations in the GT. According to the datasheet, the internal resistance of the contact is (0.1V/200A =) 0.0005 Ohm maximum. That seems tiny, but at the rated 500A continuous current that would generate (500*500*0.0005=) 125Watt of heat. During DCFC charging at 150kW the current will be about 400A, so the 4 contactors will generate up to 80Watts of heat each. That is pushing it, as all that heat will need to go somewhere. On top is that the primary coil in the contactor also gets quite warm: 650mA minimum hold current at 12V is another 8 Watts of heat.

The 80 Watt heat is not insignificant: it will get as hot as an incandescent light bulb. It does not look like there is any dedicated cooling in the plastic HVJB assembly. The only way it is cooled is indirectly from the battery pack behind it. Hopefully, the actual contact resistance is lower than that.

So it is possible that during DCFC and/or frequent heavy accelerations the contactors get so hot that they break down or age prematurely. At 85C (hot coffee) the rating is 1000W for 1 minute. But given the likely poor heat conductivity, 80W continuous heat could make the contactor much hotter than 85C. So it could get out-of-spec. Most contactors will survive some abuse, but some unlucky ones will melt plastic and break open or weld themselves shut.

What could a software fix do to prevent the contactors from blowing open or welding shut?
There is a tiny possibility that a bug in the software causes the pre-charging contactor to open too late (or not at all). That would degrade Main Contactor+ quickly. The fix for that would be simple. This scenario is quite unlikely given the information we have from the recall.

The software could reduce the system current when the contactors are hot. This inevitably means some performance degradation or longer charge times. The problem is that there is no temperature sensor in the HVJB or anywhere near the contactors. Ford's software would have to guess the heat based on the average current and the battery temperature. I don't think that can be done accurately without being conservative, causing a noticeable performance degradation.

The software might attempt to detect whether a contactor is about to go bad. The problem with that the voltage drop of a loaded contactor is small (<0.1V) compared to the voltage drop of the battery under load (~30V). The voltages on the battery side of the Main Contactors likely cannot be measured accurately enough to detect contact degradation. With any increase of resistance due to a dirty contact, the contactor will get much hotter very quickly due to the high currents. It will blow or weld shut very rapidly. So, I don't think this is feasible.

I could be missing something, but I don't see a way that a software fix can avoid the hardware problem of overheating without some noticeable degradation of the specs.

What could the software fix do?
It does not seem likely that the OTA software fix prevents the problem from occurring. So it probably just mitigates the problem. Let's analyze what could be done:

• Case 1: One of the Main Contactors is stuck open. No current can flow so the car will be bricked. No way around that.
• Case 2: One of the Main Contactors welded itself shut. This is detected in the startup sequence. Rather than bricking, the MME could be put in a limp-home mode to significantly reduce contactor current. That is likely what Ford's fix does. Since the other Main Contactor can still open there should be no significant safety risk.
• Case 3: One of the Main Contactors opened itself while driving due to overheating, or some discontinuity is detected while driving. This will result in a 'stop safely now' event. Rather than bricking the car in the middle of the road, the software could attempt to restart after a brief cool-down period l. The it can switch to the reduced power limp-home mode. This is likely what happens.
• Case 4: One of DCFC contactors is open. In that case, DCFC is INOP, but the car should otherwise operate normally.
• Case 5: One of the DCFC contactors is welded itself shut. This is a slight safety issue but should not have to result in the car going into limp-home mode.

What about a hardware fix?
Any hardware fix would be very expensive for Ford because removing and opening the battery is significant labor and requires skilled people. It would probably run at $3K-$5K per vehicle. I suspect that Ford is monitoring the statistics. They decided that the problem is still rare enough to fix a HVJB on a case-by-case basis. There is no significant safety risk, especially after the software fix.

A permanent solution would be a complete re-design of the HVJB with better cooling and temperature monitoring.
Ford could also replace just the contactors in the HVJB with a better version. This is what the repair seems to do. I suspect that significantly better contactors in the same form factor are not available. Therefore this does not guarantee the problem from re-occurring. It does not seem to be a real manufacturing defect, but rather a structural issue in the design of the HVJB.

How does the MME operate with AC L2 charging?
Both the DC-DC and AC-DC chargers are on a separate (lower-power) contactor. This allows the 12V battery to be re-charged without switching on the entire car. This saves energy because the motor inverters will be off. L2 AC charging is also possible without switching on the entire car:

In that case, only the Main Connector+ and the charger connector need to close. The cooling of the DC-DC converter and the Charger runs entirely of the 12V battery so it does not need HV to be active.

As soon as the battery needs to be heated or cooled the rest of the circuit needs to wake up. In moderate temperatures that is not needed because L2 charging produces little heat.

Hope this helps!
Disclaimer: The above is my speculation based on my research. It might be inaccurate or contain errors.

Great stuff here. I would be really curious if you could find the equivalent parts for a Tesla mode 3 or Y and be able to look at the differences and see why they generally don’t have the problem. Seems they are also 400V architecture and charge faster then mach-e and have more powerful motor than mach-e. Both of which seemingly would contribute to a similar problem? I haven’t been able to find such a good video of a Teala beakdown as the one referenced here.

 

[Howard]

Great write up and explanation of HVBJB. I was looking at data sheet for contactor on page 1 of thread. Something caught my eye that surprised me on coil section of data sheet. It lists the duty cycle of the contactor coil at 20-30%. I would have expected the coil to be rated for "continuous" duty. If you aren't familiar with duty cycle rating, here is definition:

Duty cycle is the proportion of time during which a component, device, or system is operated. The duty cycle can be expressed as a ratio or as a percentage.

When I received my electrical training, a duty cycle of 30% would mean that the component could be energized for 18 minutes out of a 60 minute period of time. I am curious if this could be contributing to the excessive heat generation within the contactors. In the Navy, I worked on HV AC and DC electrical distribution systems which included circuit breakers, motors, contactors, motor generators, and turbine generators. Most of the time when we saw contactors and even breakers with contact damage it was due to excessive arcing during closing and opening sequences. However, there were times that we found the coils were weakened allowing the contactors to chatter (rapid opening and then immediately shutting sequences). Every time an arc is produced, pitting occurs on the contact's conducting surface. This pitting damages the surface which reduces the conduction properties of the material and changes the current density in the undamaged areas of the contact surface. Same amount of current will still flow but it will now be through a smaller surface area which will increase heat in those areas. Over time, this will lead to stuck shut contactors. I don't know if the coils in this contactor are solid state or electro-mechanical; but I have seen coils "swell" on electro-coils to the point that the contacts would not fully close when operated which will quickly cause material to start melting. Some of the equipment I worked on had currents in excess of 10,000 amps going through them. Whether the coil on contactor is solid state or electro-mechanical, a duty cycle rating of 20-30% seems unusual on something that will be potentially continuously energized for hours.

I would be interested if anyone, including OP, has any thoughts on this. Sorry for long post.

 

[i8iridium]

Very informative, but in the end, the Mach-e is UNDER ENGINEERED, like most domestic manufactures. The Electro-mechanical part is NOT robust enough! FORD's solution, NEUTOR the cars power just like they have they gutted maximum acceleration of the GT due to the battery thermal bottle-neck issue. Yeah FORD, just keep screwing us with your inadequate software sleight of hand (not fixes).

There is NO evidence that the recall software update has neutered power or DCFC rates at all. It only reduces power when it detects an eminent issue with the HVBJB to allow you to safely drive the vehicle to service. At that point, it’s a brand new HVBJB.

But that’s fine, people will regurgitate that shit all day long. I watch it on FB constantly.

 

[Mach-Lee]

When I received my electrical training, a duty cycle of 30% would mean that the component could be energized for 18 minutes out of a 60 minute period of time. I am curious if this could be contributing to the excessive heat generation within the contactors. In the Navy, I worked on HV AC and DC electrical distribution systems which included circuit breakers, motors, contactors, motor generators, and turbine generators. Most of the time when we saw contactors and even breakers with contact damage it was due to excessive arcing during closing and opening sequences. However, there were times that we found the coils were weakened allowing the contactors to chatter (rapid opening and then immediately shutting sequences). Every time an arc is produced, pitting occurs on the contact's conducting surface. This pitting damages the surface which reduces the conduction properties of the material and changes the current density in the undamaged areas of the contact surface. Same amount of current will still flow but it will now be through a smaller surface area which will increase heat in those areas. Over time, this will lead to stuck shut contactors. I don't know if the coils in this contactor are solid state or electro-mechanical; but I have seen coils "swell" on electro-coils to the point that the contacts would not fully close when operated which will quickly cause material to start melting. Some of the equipment I worked on had currents in excess of 10,000 amps going through them. Whether the coil on contactor is solid state or electro-mechanical, a duty cycle rating of 20-30% seems unusual on something that will be potentially continuously energized for hours.

The "coil power" is switched with solid state electronics many times per second, probably 1kHz or faster. It's switched faster than the magnetic field in the coil can decay, so the contacts stay closed and aren't chattering. You only need full power on a contactor to close the contacts, it doesn't take nearly as much force to keep them closed so thats why you can economize the holding current with PWM. If you ran full 12VDC continuously the coils would overheat with the current.

 

[Howard]

Thanks for info on coil power. I had done some digging and found where a similar part number had the option for the "coil economizer" but wasn't sure if one on HVBJB had that feature.

 

[Mach-Lee]

Thanks for info on coil power. I had done some digging and found where a similar part number had the option for the "coil economizer" but wasn't sure if one on HVBJB had that feature.

No, the coil is externally economized by the BECM, rather than it being built into the contactor.

 

[DR.J56]

Ok so my ADD makes me skip from page 1 to here. I did however watch the vid references in post #1, very interesting and 100% clear. His other vids are awesome. This is better than Monroe (for us consumers) who only cares about building cars.

My question is, and sorry if this was answered previously, what has Ford changed/fixed in the new HVJB?

 

[Mach-Lee]

Ok so my ADD makes me skip from page 1 to here. I did however watch the vid references in post #1, very interesting and 100% clear. His other vids are awesome. This is better than Monroe (for us consumers) who only cares about building cars.

My question is, and sorry if this was answered previously, what has Ford changed/fixed in the new HVJB?

New contactors. We aren't sure what changed internally, could be a different coating or metal on the contacts or something.

Sponsored