Creating an internal data log descriptor.

[seank]

Currently any software that wishes to make any sense of the data stream coming from the ECU, must have an external or internal hard-coded definition file of some sort.

What we need:

A simple method to send a request to the ECU so that it will at a minimum, return descriptor information.


Requirements:

1. Everyone more or less agrees with a proposed solution.
2. It needs to be generated at compile time, one way or another, so that no one needs to maintain it manually.

At a glance, I thought of simply utilizing the C preprocessor to generate offsets based on the existing data struct. We would need to add look up names or codes that developers have a reference.

Once something like that is in place it becomes much harder to break applications, as the application immediately becomes aware of new or missing data and would no longer be dependent on data order.

Let me know your thoughts guys. Remember we have TONS of extra flash space.

-sean

[malcom2073]

Are you intending to have everything included in the ECU, or just the datalog format?

You could just have the format itself available by request, which would tell you which values were in which order coming back through the datalog packet, or you could have everything available, field name, scalars, offset, size, etc. The latter would be significantly more difficult to generate and parse, but entirely eliminate the need for a datalog definition file on the tuner side.

I think the main concern is with the ability to change the datalog format, so just having the order of fields coming back would be incredibly useful. I'm not against the tuner having to have a generic definition file that lets it know information about what the different fields are, and then the ECU reports what order the fields are coming back in.

[dandruczyk]

Are you intending to have everything included in the ECU, or just the datalog format?

You could just have the format itself available by request, which would tell you which values were in which order coming back through the datalog packet, or you could have everything available, field name, scalars, offset, size, etc. The latter would be significantly more difficult to generate and parse, but entirely eliminate the need for a datalog definition file on the tuner side.

I think the main concern is with the ability to change the datalog format, so just having the order of fields coming back would be incredibly useful. I'm not against the tuner having to have a generic definition file that lets it know information about what the different fields are, and then the ECU reports what order the fields are coming back in.

I'm partial to the latter approach, i.e. field name, scalars/conversions, size. As the former approach STILL requires the tuning side to have fore-knowledge in order to be able to parse, and that can change with FW over time. I envision a "master table" of all possible dataloggable vars stored in flash, this table contains the following fields (to be adjusted as needed) that is compiled into the firmware (.i.e. generated as a json string as part of the build process?)

1. ID (numeric, 16 bit, must be unique)
2. Name (charactor, 16 chars?)
3. size (uchar 8 bit enumeration) U08, S08, U16, S16, U32, S32, U64, S64
4. multiplier
5. adder
6. Description. (char, freeform up to 255 chars?)
... ?

Fields 4 and 5 are for conversion of "ECU units" to "real world units". In the cases where those are one in the same, multiplier would be 1, adder would be zero. Notice the lack of an "offset". this is because you can request the values returned in any particular order below.

Several default definitions would be defined that return a subset of the master list, with ID's in a reserved block (let's say ID's 1-10 are "fixed" (compiled it?) lists, and id's 11-99 are user settable lists). I propose API calls to return the master list above (with all details), the current datalog table ID (an ID number), followed by a "detailed datalog table request" that returns a list of each of the 6 fields above for the requested table, and the ability to set datalog "tables", by feeding in a LIST of ID's in the order that you want them returned (i.e. setdataLogID<DatalogTableID><numberofvalues><val1><val2><valN><cksum>. Thus you have maximum flexibility, on the tuning side to say. I only want ID 2, 16, 18,21, and 36, assign that to a datalog table ID X i.e. 42, tell the ECU that's the default datalog table ID to be used and on each datalog request it returns that set.

You would need api calls to do the following:
Return the (compiled in master list), so you know what values are which loggable ID
Return the currently set datalog list ID
Return the list of values in that datalog list ID
Set a list for new loggable ID values for a specific datalog list ID
Set the default datalog list ID

In the UI I envision this is a drag and drop table, left has the master list, right has the way you want them, you drag and drop to suite, then save and you're done.

One big advantage of this method is you can get MUCH faster datalog speed as you're only getting sent what you want to see and nothing that you don't. Teh datalog request SHOULD ALWAYS return the datalog table ID as one of its first parameters so the tuning side knows immediately how to parse it., thus allowing interleaving of multiple datalog request types in the datastream .( i.e. default log type will be streamed continually, but it should be possible to request a different stream type as a "one shot" request to update controls/data sources that don't need ultra fast refresh rates.



Thoughts?
-- David

[malcom2073]

I think that sounds good, especially the part about being able to "save" datalog formats so you can quickly and easily switch between different datalog styles.

This could also work as a lead-in for having multiple dataformats going out over multiple paths (serial0, serial1, CAN0, CAN1, etc), because you could potentially say you want datalog format 0x00 going over one, and 0x01 going over another.

I guess if the ECU internally has a map of what ram address each datalog field ID points to, then there would be no need of an offset. Just request a particular datalog table ID from the ECU, then tell the ECU to use that ID and you're set.

I also think we should leave the datalog format alone by default, and add this stuff as features on top. Make the current format ID 1, and default to that on a new firmware load. It would make integrating much easier.

[seank]

Thanks Dave, you propose a pretty comprehensive solution.

I think the master table is a must.

The log table ID is taking it a step further than I was thinking, sounds neat.

To fully understand our options, I think we should investigate how to create some sort of a def file at compile time.

I'm not sure we need to support out of order data, but implementing it may not bring additional overhead.

One big advantage of this method is you can get MUCH faster datalog speed as you're only getting sent what you want to see and nothing that you don't.

And this can be a critical debug tool.

I need to dwell on some ways to efficiently get only the data your requesting, maybe a linked list? The good news is that currently sending data-log packets is a bit inefficient, so it will likely not get any worse.


Thanks guys,
Sean

[admin]

I created a redmine issue #14 detailing the idea..
https://redmine.libreems.org/issues/14

As for getting the data we want, a simple array should work, it would just require a little magic in the code. i.e. I request a log of ID's 1 3 7 and 14. it would look at the flash master list which should contain a pointer to that value's "location" thus you'd need a array of pointers and when its time to run the log, just iterate the array returning each value in turn. IT would help if EVERY dataloggable variable is the same size (i.e. all 16 bit) it makes the pointer math easy, if the vars are mixed size, it might get uglier (but not undoable, just much more bug prone).

Are all loggable variable the same size? (i.e. 16 bit) , if so it makes it easier, and removes the need for the "size" enumeration in the master array of structs.

[rusefi]

So in the extreme case you are talking about auto-generation the [OutputChannels] section of TunerStudio configuration? (I am referring to TS just as an example)

[malcom2073]

So in the extreme case you are talking about auto-generation the [OutputChannels] section of TunerStudio configuration? (I am referring to TS just as an example)

Essentially yes. The ECU would respond with the equivalent of that section, detailing exactly how to parse the incoming data. How much, what kind, and the format of the information is up in the air at the moment

[rusefi]

This might sounds as an over-kill, but how about XML? We are talking about meta-data targeted at self-describing the binary stream. The unified format for meta-info would be XML.

[malcom2073]

XML is a possible format, as is JSON and YAML. The idea being that when the firmware is compiled, the definitions are automatically generated and stored in the image to be uploaded. I think the first step is getting customizable datalogs working with a couple "hard-coded" packet formats, and then we can move forward from there and work on getting the actual meta-information automatically generated at compile time and put into the S19.