My winter Mustang Mach-E project

[Jiji]

The last few years I find a winter project, two years ago was logging and visualizing my solar arrays (I have three inverters and I wasn't happy seeing one at time).

Last winter it was some whole house load monitoring using current transformers on each load:

This winter I decided to dig into my Mustang Mach-E. I don't have any visualizations yet but I have posted my first release of some Python code I have been using to explore the MME via the OBDII diagnostic port. The idea is I want to record high fidelity data using a state machine approach, data collected on a trip is very much different from the data needed from a charging session so by detecting what the car is doing (sleeping, charging, driving, etc) I can vary the data requested and how often it is requested.

Right now I have basic data collection looking at the battery and vehicle dynamics in motion and AC charging at home. It has been a challenge since I started with no knowledge of OBDII diagnostic port and vehicle standards like ISO-TP and UDS so I just assumed I could plug in and capture data. First setback was why couldn't I capture any CAN bus data? Turns out there is no CAN bus traffic on the OBDII connector, you have to request it so I had to get a OBDII dongle to make the buses come alive (obviously that Ford doesn't want us messing around via the OBDII port).

The code is on GitHub at MME-Record and I will update this post with news as I make some progress on the data collection, processing, and visualization. My original goal was to replace my beloved TeslaFi web app, I was never happy with what FordPass provides so lets see what I can learn before the days get longer and I can go for ice cream and BBQ again.

 

[Gimme_my_MME]

Mmmm BBQ

 

[Jiji]

Here is an brief overview of the project:

Hardware
I used a Raspberry Pi 4 that I had laying around and an SK Pang PiCAN2 Duo CAN Bus Board which draws power directly from the always on ODBII connector and provides connections to the HSCAN and MSCAN buses. This is a requirement if you wish to see the entire car since the Gateway Module has some modules on the HSCAN bus and others on the MSCAN bus (which is the same 500K bit speed as the HSCAN bus).

On the right is the new $15 Raspberry Pi Zero 2 W which I may migrate to at some point when I start to address the standby power draw on the battery.

Development tools
- MacBook Pro/MacOS (host)
- Raspberry Pi 4/Ubuntu 20.04 LTS (target)
- Python 3.10
- VS Code development environment
- InfluxDB database
- Tailscale WireGuard service
- Grafana visualization software

VS Code runs on my MacBook Pro and part on the Raspberry Pi target in a remote connection mode making it possible to test and debug on the target. But since the target hardware may not have a routable IP address when using a phone or vehicle hotspot, a VPN is needed to connect everything together so SSH and VS Code can connect to the hardware. The newer WireGuard protocol is fast and simple and Tailscale connects everything together with fixed IP addresses. Both InfluxDB and Grafana run in Docker containers on a Raspberry Pi server on my home network.

Record
The Record utility is a state-based utility to read the vehicle sensors and save to a file or a database. To keep file sizes and network bandwidth to a minimum it only writes state changes. Each state has a list of vehicle data to read and can be read as fast or slow as required. Configuration is done with a YAML file and vehicle modules, sensors, and devices are defined in JSON files so it is easy to update without making code changes. I'll provide a deep dive into this in a future post since I think it has had advantages over using software like FORScan Lite (which is what I use since I lack a PC or Android devices).

Playback
The Playback utility reads the state change output files from Record and emulates a Mustang Mach-E by responding to requests with the current state data. In my basement 'lab' I have the two CAN buses hooked up in a loopback mode and Playback can run in the background while simultaneously running Record making it possible to work without connection to a physical vehicle (which happens to be my wife's car). Features include a speed up mode so you can run at 100x speed to quickly test Record while making changes to the state detection code or adding new sensors.

 

[Jiji]

Today was spent working on the Record utility state machine to determine what sort of data be collected. Here are my current state definitions:

Unknown = auto() # initial state until another is determined
Sleeping = auto() # limited responses to ReadDID requests
Off = auto() # the vehicle was turned off
On = auto() # the vehicle was turned on
Trip = auto() # the vehicle is in a gear other than Park
Preconditioning = auto() # the vehicle is preconditioning (remote start)
PluggedIn = auto() # the vehicle has plugged in
Charging_AC = auto() # the vehicle is AC charging
Charging_DCFC = auto() # the vehicle is DC fast charging

When Record is started it doesn't know the state of the MME so we query the vehicle to get us into a known state. I chose JSON files to allow changes on the fly, here is the file used for the Unknown state:

[
...{
......"module": "GWM",
......"arbitration_id": 1814,
......"period": 5,
......"dids": [
.........{
............"did_name": "KeyState",
............"did_id": 16671,
.........}
......]
...}
]

What this record defines is reading a list DIDs (Data Identifiers in ISO-speak) from the Gateway module (GWM) at address of 1814. This list has only one member with identifier '16671' which will be read every 5 seconds ('period') until a new state is detected.

The Gateway module is special as it always responds to requests so is a good place to start. We use the DID identifier 16671 (411Fh) and the Gateway module returns the state of the Start button on the dashboard so we now know if the car is sleeping or running and can transition to a new state. The new state will load a new state file that varies the data reads for what makes the most sense, battery and vehicle dynamics for a trip or the details of a charging session. This differs from the typical ODBII software which uses a fixed list of data points to be read without any means of adjusting the frequency - the displayed SOC changes slowly so why update every few seconds jst because you want to track it?

If you made it this far you must be a techy-type of MME owner, next I'll look more at the state machine logic since once you have defined the states it is easy to collect the specific data associated with the state.

 

[Jiji]

Got the state machine work mostly done, but Ford doesn't make it easy.

Seems there are three type of modules in the Mach-E when the vehicle is sleeping:
1. Always on
2. Can be woken up
3. Out for the count

The Gateway Module (GWM) is the first type and while I haven't seen any others there could be some lurking around waiting to be discovered.

I stumbled on to the second type by accident. While the Gateway module always responds to a read of the Start button state, the state is not very good as it doesn't distinguish between Accessory and Run modes and a remote start completely bypasses it. Turns out the Body Control Module (BCM) has some state variables that do but when I tried reading them there was no response - bummer. But somehow, I noticed I received a reply if I sent two requests within a few seconds, the first would time out but it seems it woke up the module so it would reply to the second.

The rest are the third type, perhaps powered down until an event wakes them up. Determining the start of a charging session falls in this category, the Battery Energy Control Module (BECM) has the charging status and the reads time out until something happens and the module is needed to manage the charging session.

Since the program's state machine reliably moves from Off, Accessory, On, Trip, Plugged In, and AC Charging (gave up on Preconditioning for now as it is vague and I need to sort out the states to be read), I decided to add the Charging and Trip events to the Record program since I want to see results without having to pull them from the database. Hope to share some results in the next update.

 

[DrSteveBrule]

Following. I do some work in observability.

 

[Jiji]

Been sidetracked but tonight I am a recording a charging session so I can use it to finish up the database integration. Since I am unsure of how it will be organized in the database (no background in this), I figured I would share some of the details of the process while I wait for it to complete.

First up, I plug into the OBDII port and it starts the Record app.

2022-01-20 23:35:23 state_manager INFO Vehicle state set to 'Unknown'
2022-01-20 23:35:24 record_statemgr DEBUG 0726/41B9: Start engine normal=0, remote=0, disable=1
2022-01-20 23:35:24 record_statemgr DEBUG 0726/417D: Inferred key state: Key Out
2022-01-20 23:35:24 state_manager INFO Vehicle state changed from 'Unknown' to 'Off'

Since there is no (virtual) key and not in remote start we deduce that the vehicle is off and the Off state is selected. In the Off state Record looks for the Start button to be pressed or the charger plug to be connected.

2022-01-20 23:36:45 record_statemgr DEBUG The request from 07E2 request timed out
2022-01-20 23:39:15 record_statemgr DEBUG 07E2/4843: charge_plug_connected=True
2022-01-20 23:39:15 state_manager INFO Vehicle state changed from 'Off' to 'PluggedIn'
2022-01-20 23:39:15 record_statemgr DEBUG 07E2/4843: charge_plug_connected=True
2022-01-20 23:39:16 record_statemgr DEBUG 07E4/4851: EVSE type: BasAC
2022-01-20 23:39:16 record_statemgr DEBUG 07E4/484D: Charger status: Ready
2022-01-20 23:39:26 record_statemgr DEBUG 07E4/484D: Charger status: Charging
2022-01-20 23:39:26 state_manager INFO Vehicle state changed from 'PluggedIn' to 'Charging_AC'

At first the charge plug connected reads time out since the BECM module is off, when I plug in the charger plug at 23:39:15 it changes to the PluggedIn state and uses the EVSE type and charger status to determine that a Level 2 AC charge is taking place. Now the Charging_AC state takes over and records the data useful for a charging analysis:

2022-01-20 23:39:26 record_statemgr DEBUG 07E4/480D: HVB voltage is 357.01 V
2022-01-20 23:39:26 state_manager DEBUG FFFF/8000: HVB power is 0 W (calculated)
2022-01-20 23:39:26 record_statemgr DEBUG 07E4/48F9: HVB current is 1.7 A
2022-01-20 23:39:26 state_manager DEBUG FFFF/8000: HVB power is 607 W (calculated)
2022-01-20 23:39:27 record_statemgr DEBUG 07E2/4850: AC charger output current is 1.0 A
2022-01-20 23:39:27 record_statemgr DEBUG 07E2/484A: AC charger output voltage is 357.2 V
2022-01-20 23:39:27 record_statemgr DEBUG 07E2/485F: AC charger input current is 2 A
2022-01-20 23:39:27 record_statemgr DEBUG 07E2/485E: AC charger input voltage is 246.5 V
2022-01-20 23:39:27 record_statemgr DEBUG 07E4/4801: Internal SOC is 63.282%
2022-01-20 23:39:27 record_statemgr DEBUG 07E4/4845: Displayed SOC is 68% (67.5%)
2022-01-20 23:39:27 record_statemgr DEBUG 07E4/4848: HVB energy to empty is 48.672 kWh
2022-01-20 23:39:36 record_statemgr DEBUG 07E4/480D: HVB voltage is 357.25 V
2022-01-20 23:39:36 state_manager DEBUG FFFF/8000: HVB power is 607 W (calculated)
2022-01-20 23:39:36 record_statemgr DEBUG 07E4/48F9: HVB current is -3.9 A
2022-01-20 23:39:36 state_manager DEBUG FFFF/8000: HVB power is -1393 W (calculated)

At the start the HVB is discharging but the charger slowly ramps up the output current until it draws the full 48A my wall charger can deliver. In this log, the AC charger is in the MME, the input is the wall charger output AC and the output is the DC used to charge the battery.

2022-01-21 00:28:19 state_manager DEBUG FFFF/8000: HVB power is -10063 W (calculated)
2022-01-21 00:28:19 record_statemgr DEBUG 07E2/4850: AC charger output current is 28.2 A
2022-01-21 00:28:19 record_statemgr DEBUG 07E2/484A: AC charger output voltage is 369.2 V
2022-01-21 00:28:19 record_statemgr DEBUG 07E2/485E: AC charger input voltage is 238.2 V

Using the power at the AC charger input, output and at the HVB we can get an idea of the efficiency of each step. When fully charging and drawing 47A from the wall charger, the power is 238V * 47A or 11186W, while the power at the HVB is -10063W (negative indicates it is charging), so the charging efficiency is 10063/11186 or ~90%. Is this good, bad, or typical - I have no idea.

The charge just finished so I have this last bit to share:

2022-01-21 00:44:20 record_statemgr DEBUG 07E2/4850: AC charger output current is 27.6 A
2022-01-21 00:44:20 record_statemgr DEBUG 07E2/484A: AC charger output voltage is 372.0 V
2022-01-21 00:44:20 record_statemgr DEBUG 07E2/485F: AC charger input current is 46 A
2022-01-21 00:44:20 record_statemgr DEBUG 07E2/485E: AC charger input voltage is 237.9 V
2022-01-21 00:44:25 record_statemgr DEBUG 07E4/484D: Charger status: Not Ready
2022-01-21 00:44:25 state_manager INFO Vehicle state changed from 'Charging_AC' to 'PluggedIn'
2022-01-21 00:44:25 state_manager INFO Charging session:
'start': '2022-01-20 23:39', 'duration': 3898,
'starting_soc': 67.5, 'ending_soc': 79.5, 'kwh_added': 9.796}

For an AC charge to 80% SOC Ford uses the maximum current right up to the end. I also see a minor problem caused by sampling. I would guess the ending SOC hit 80% and the charge ended, something I can easily fix by sampling more or reading the SOC one last time before reporting results.

And I can confirm the efficiency, here is grab of the power draw by my wall charger measured at the service panel:

Charged for 64 minutes at 11.3kW so less than 87% efficiency measured this way. Should be a sunny (but cold) day tomorrow so I hope to earn some of that back.

 

[Jiji]

Why you might not want to trust in the displayed SOC in your Mach-E.

Since you may not know, there are three ways to view battery data on the Mach-E:
- DID read 4845 (Displayed SOC)
- DID read 4801 (Internal SOC)
- DID read 4848 (Energy to Empty)

The Displayed SOC returns a value that you multiply by 0.5 to get the SOC, this is what you see on the dash display but I show both in the log. The Internal SOC returns a value that takes into account the reserves Ford uses to manage charging or discharging at the extremes (my guess, please correct me if I am wrong). The Energy to Empty is the number of kilowatt-hours your MME has available before you need to pull over.

So I was finishing up my custom logging application and plugged in at 14:23:14 and charging commenced at 14:23:29:

In just 10 seconds, the Displayed SOC has gone from 49.0% to 51.5% with just a tiny movement in the Internal SOC and Energy to Empty values. I'm sure that there are a number of inputs used in the calculation but the only changes I see in the 10 seconds is the charging current being stepped up what eventually will be ~48A.

I report the start and ending values along with the date, time, duration, and location of the charging session but thought it usual for the Displayed SOC to move so quickly. There must be some serious mental gymnastics behind computing the Displayed SOC, just wondering how it gets done.

 

[Jiji]

Had to reorganize some code and got side tracked by feature creep but my project can now record AC charging sessions and provide extra insight that Ford doesn't provide.

I only have one charge in the database and I need to figure out some visualization issues but here is my first stab at it:

This is from a Grafana query asking for the charges in the last seven days, it displays the charge session time and the bar height is the energy added in the session. I need to figure out a way to show the height as the energy gained, 35 kWh in this example, and the width being the charge start and stop times but for now this will have to do.

As I mentioned there is.quite a bit of data collected for a charge session. One sidetrack was adding the ability to integrate the power (in W) over time to get the energy used (in Wh or kWh), this allows calculation of the charging efficiency, defined as the change in energy gained divided by the energy delivered to the charging plug.

The AC charger in the Mach-E records the voltage and current delivered by the wall or mobile charger and also the change in the HVB energy but has no other power or energy sensors. It is left to the Record application to calculate the power (in Watts) and energy (in Watt-hours) for the various measurement points, and in this session we observe that just over 9% of the energy was lost to charging inefficiencies.

 

[Jiji]

Had to reorganize the database schema to make queries work , luckily I can just playback recorded charing sessions when I have to dump the database. So here is what I have for a simple display of charging sessions and energy consumed:

The height of the bar is the energy (green to the battery, red from the grid) and the width is how long it took (these were all charges to 80% SoC). Grafana (the open source visualization software) has a time picker so it displays the sessions and energy totals for whatever period you select, above is the 'Last 24 hours', here is a 'Last 90 days' view:

'Energy Added' is increase to your GOM and the 'Energy Used' times your $/kWh is the contribution to your electric bill - be sure to use the red number when calculating your cost per mile (km).

I think this is good enough (until it isn't) so I will start working on getting some trips logged.

 

[Jiji]

A heads up for those looking at external temperature via the OBDII port.

A cold day today, in the single digits and the MME display says it is -14℃ (6℉) but the car reports a much warmer temperature from the OBDII port. It is sunny so maybe it was that but to sort this out it is time to dig out the CANDUMP utility and see what the different modules that respond to external temperature are reporting.

One returns an error, six report (0x7F or 0xFF) so we can throw those out, the rest report 0x25 or 0x19 and I had selected one that is reporting 0x25. These are temps in ℃ so we are looking at a difference of 12℃ in the reported external temperature - not insignificant.

Temperature is encoded to fit in a single unsigned byte but is offset to allow for negative temperatures, so subtract 40 (0x28) to get the actual temperature in ℃. So the MME reports either 0x19 - 0x28 which is -15℃ (5℉) or it reports 0x25 - 0x28 which is -3℃ (27℉).

A big difference, choose wisely if you wish to report an accurate external temperature. I switched to using the external temperature from the Secondary OBD Control Module C (SOBDMC) shown as 07EE which jibes with the MME value.

When I last checked the new A Better Route Planner (ABRP) integration is using the Secondary OBD Control Module (SOBDM) shown as 07EA so their range model may be less accurate than it could be.

 

[Jiji]

CAN bus sniffing is dead.

Prior to getting my MME, I had only used an OBDII dongle once to see why I was getting a check engine light. Those were the good old days when you you plugged in could see data on one or more CAN buses.

With the MME, and likely most new cars, this is no longer possible. The number of buses, CAN or some other variety, has exploded in number so it is no longer possible to see everything. And with all the remote connectivity, car manufacturers don't want you to see everything to keep your ex from tracking your location or stealing your car.

But I have a minor dilemma, I want to be able to determine if my MME is plugged into a charger. Easy enough if you plug in after you start watching as the BECM (Battery Energy Control Module) will wake up and you can get the charge plug status. But with the MME in idle (just sitting there), I haven't found a way to determine if the MME is plugged in without waking it up so I decided to sniff the BCM (Body Control Module) to see if it would help.

The BCM has in my counting, 421 different responses so there are a lot things you can ask for and it reliably responds when the MME is idled, maybe not the first time but it will respond if you send back-to-back requests. But there is no CAN bus traffic to sniff so I have create the traffic to look for changes and that means sending 421 requests and looking for the messages that change when you plug in or out.

Processing 421 requests/replies on a 500K bus isn't going to be fast but Ford likes to encode status in a 32-bit word or just in a single byte so by winnowing down the requests to those with a reply of 4 or fewer bytes we can save time and hope what are looking for isn't in a longer message. There is even one reply that goes on for 1520 bytes, gotta wonder if that is everything in the BCM in one simple request.

I was hoping the charge port door would show up as there is a warning if it left open and it would be a useful proxy for being plugged in. And the BCM has all the door ajar switches so maybe it is in with them. But alas no, opening and closing the charge port door did not show up so it must be in some other module.

I did see some activity on a couple of BCM replies when I plugged in and out but they would also change value at random times leaving me with nothing to use so far. Right now I am collecting some long-term data on the 12V battery, nice to know that it can draw less than 50W when in idle:

 

[Jiji]

Now I have my first LVB (low voltage battery) results. I know there are others on the forum looking at similar data but I wanted to test out features of my custom software. I don't know if this true for other OBDII devices but mine require for the car to be powered up before it will work, even though the OBDII connector has continuous power and the GWM (Gateway Module) will respond to requests.

So what I wanted to see was the LVB at idle, where idle is the state where the MME is turned off and does not have any charging device connected. Using the commands to read the LVB voltage, current, and SoC we see the LVB is slowly discharging because of the small load of the GWM, BCM (Body Control Module), and other modules needed to unlock the car and perform any internal or remote commands.

Not shown is a load of -4A (only changes are logged and it is further back in the log) so the power draw is about 50W and we are losing 1% of the LVB SoC every 4 minutes.

In the next screen grab we see that the MME turns on the DCDC (DCDC Converter Module) and starts charging the LVB, initially pushing 93A into the battery at 15V:

The LVB charging continues using a tapering charge current until 100% SoC when the DCDC module disengages and the load transfers to the LVB which begins a new discharge cycle. Here is the LVB charging current curve:

According to the HVB Energy to Empty sensor, the complete charge-discharge cycle of ~5.5 hours consumed 676Wh (0.676 kWh) of the HVB charge which is an average load of 123W. This should be on the high side as the Raspberry Pi hardware could be as high as a 15W sink and there is cost of making the various MME modules respond to the high level of data requests.

Next up is the same analysis but with the car plugged in the wall charger.

 

[hawkeye3point1]

initially pushing 93A

Nice work on this, had no idea the in-rush would be that high. Not sure that is good for the LVB and probably why the software waits so long to fire up the DC-DC.

 

[Jiji]

Let's dive into the MME sitting in idle connected to a wall charger after having completed charging the HVB to 80% SoC.

At this point the MME will be slowly discharging the LVB, losing about 1% SoC every 4 minutes:

The battery starts at 12.8V which is good for what I assume is an AGM lead-acid battery. Initially the load is 100W but within a few minutes it drops to 50W as the car shuts down unnecessary equipment. This continues until a few minutes after 40% SoC is hit and the DC-DC module is re-enganged:

Unlike my previous post where the recharging was from the HVB, there is a power source connected to the charge plug. We can see that it starts out drawing 10A from the wall charger. Charging starts at over 100A, peaks at 110A but will quickly taper down, just 8 minutes later it is half the peak current and the LVB SoC will be nearly 80%.

It takes over an over to bring the LVB back to 100% SoC, in this example it wasn't until 9:30 that the AC charger disconnected and the LVB started a new discharge cycle.

I don't claim to know the details of how the AC charger and DC-DC modules work together but this should give you an idea why Ford nags you to plug in when you shut down the car to keep the HVB at peak condition and let the wall charger do the work of refreshing the LVB and preconditioning the car.