Another FLYXXX.DAT converter

[BudWalker]

I run below from the window 7 command line prompt. The shell window displays numbers, lots of them, and after about 20 seconds stops and beeps(seems like it crashes). What am I doing wrong? Shouldn't the dump be saved to a txt file?
java -jar FlyDump.jar -f <FLYXXX.DAT filename>

FlyDump un-encodes the payload for each record and then presents all possible number types for each location in the payload. I.e., character, hex, 1 byte decimal, short Int, Unsigned short int, int, unsigned int, long int, float, and double. You're right it's huge. You could re-direct the output into a file and then view that with an editor.

I see that I need to add some command line arguments to reduce the output. E.g. to just produce the output between 2 tick#, or between 2 file positions. Also, to just produce output for selected payload type(s).

I'm guessing that you thought there should have been a .txt produced because you thought that FlyDump converted the input to a .txt that would have been produced by the DJI GO App?

 

[Luap]

FlyDump un-encodes the payload for each record and then presents all possible number types for each location in the payload. I.e., character, hex, 1 byte decimal, short Int, Unsigned short int, int, unsigned int, long int, float, and double. You're right it's huge. You could re-direct the output into a file and then view that with an editor.?

Yes that's what I thought the dump would do and save to a txt file or other to view in an editor. Otherwise how can the average Joe (me) try to figure something out ?
q
I'm guessing that you thought there should have been a .txt produced because you thought that FlyDump converted the input to a .txt that would have been produced by the DJI GO App?
uq
No you would have it figured out if it had been same format as the DJI GO APP

 

[BudWalker]

Yes that's what I thought the dump would do and save to a txt file or other to view in an editor. Otherwise how can the average Joe (me) try to figure something out ?
q
I'm guessing that you thought there should have been a .txt produced because you thought that FlyDump converted the input to a .txt that would have been produced by the DJI GO App?
uq
No you would have it figured out if it had been same format as the DJI GO APP

I was going to post to you again when I saw this post. It's not my place to tell others what they are capable of. But, not saying something would be unfair to you. Maybe you don't have the right skill set to do something like this. The last time I did was over 45 years ago. I thought it'd be fun to try it again using modern day techniques and tools. It was fun, but it still requires an usual skill set.

I have a lot of respect for you. You took this thing on directly. I wasn't expecting that, and I apologize for that.

 

[Luap]

No problem - I know I'm tackling something over my skills - but you always learn from trying.
If you have time to provide a converter (full or more complete than the comprehensive one) for the DJI GO App .txt would be great as well.

 

[BudWalker]

I installed a new version yesterday. It doesn't seem to have any problems. It's a little early to announce this but I'll be referring to some of the new capabilities in another thread later this morning.

1) a new section called "Offline Tools" has been added. It describes DatConverter and DatDump and how to download them to your machine. These are offline versions; i.e. you don't have to upload to get your .DAT converted. It also describes how to use them.
2) the converter now comes as standard the inclusion of Record68 which includes battery data

 

[FredzMaxxUAV]

I installed a new version yesterday. It doesn't seem to have any problems. It's a little early to announce this but I'll be referring to some of the new capabilities in another thread later this morning.

1) a new section called "Offline Tools" has been added. It describes DatConverter and DatDump and how to download them to your machine. These are offline versions; i.e. you don't have to upload to get you .DAT converted. It also describes how to use them.
2) the converter now comes as standard the inclusion of Record68 which includes battery data

Very nice! Battery data, wow! Any flight mode data found?

 

[Luap]

to Bud: on your site you write:
The Record68 contains battery voltages and the battery current. It also contains 4 short integers whose meaning is unknown.
The 4 short integers:
REC68:I0= rated battery capacity = 4480mAh (remains constant through dat file - I suppose for the Inspire 1 it will be different)
REC68:I1= charged capacity of battery = is a constant
REC68:I2= remaining capacity - value decreases throughout flight
For info REC68:I2 divided by REC68:I0 = remainPowerPercent(thats what Litchi csv logs)
REC68:I3= currentVoltage (thats column in Litchi) I checked that sum of cells 1 to 4 usually add up to this but not always - probably a further sensor used somewhere else in the circuit

Is there command line argument to save for example every 20 ticks starting at that tick to have a more portable file?
Dashware does not appear to have any issues with these massive files but still would be great to have less redundant data.
And thanks for this great little converter -

 

[BudWalker]

The code for the converter can be found at rowlandjohnson/FlyDat · GitHub This is a snapshot taken a few days ago. For those so inclined using/modifying the code can be an effective means to find data items within the .DAT files.

The deployment of the offline version, DatConverter, seems to have gone well. At least, I haven't received any bug reports. DatConverter has more options and flexibility than does the online version, which basically has no options or flexibility. Given a choice between uploading a 200 MB file, and then downloading the .csv , or, using DatConverter on your computer I gotta think you'd rather do the latter. From my perspective DatConverter is much easier to maintain, develop and deploy. Unless there are some serious objections I plan to stop development on the online version. The online version will still be available. The website will take on the function of providing the means to 1) transmit .DAT to me in the event that I need to see your .DAT file to deal with a bug, and 2) transmit .DAT files between people working on a particular .DAT file.

 

[ferraript]

Thank you, Rowland, you have done great job. It's unbelievable how much information is stored in those DAT files. And it's great you managed to decode it
I have got my Phantom 3 for only a week now, so I don't have many of DAT files to try your tools. But I used it to get CSV file from DAT file of my last flight and I found 2 problems (or bugs?) :

1.
first line of CSV file starts with Tick#,time,Longitude,
second line of CSV file starts with 20365,33,942,21.258565015486017,
so I think that one column name is missing in first line

2.
altitude value is incorrect, looks like shifted:
during my last flight, Phantom flew in altitudes from 210 to 411 and back to 210 meters
in CSV file there are altitude values from 89 to 290 (GPS value is 294) and back to 89 meters
maybe it has something to do with geoid model?

 

[BudWalker]

Thank you, Rowland, you have done great job. It's unbelievable how much information is stored in those DAT files. And it's great you managed to decode it
I have got my Phantom 3 for only a week now, so I don't have many of DAT files to try your tools. But I used it to get CSV file from DAT file of my last flight and I found 2 problems (or bugs?) :

1.
first line of CSV file starts with Tick#,time,Longitude,
second line of CSV file starts with 20365,33,942,21.258565015486017,
so I think that one column name is missing in first line

2.
altitude value is incorrect, looks like shifted:
during my last flight, Phantom flew in altitudes from 210 to 411 and back to 210 meters
in CSV file there are altitude values from 89 to 290 (GPS value is 294) and back to 89 meters
maybe it has something to do with geoid model?

Could you try this instead and then we can take a look. What is geoid model?

 

[BudWalker]

Could you try this instead and then we can take a look. What is geoid model?

And, please submit that .DAT to FLYXXX.DAT and tell me the file name so that I can retrieve it.

 

[ferraript]

Could you try this instead and then we can take a look. What is geoid model?

GUI version: ok, I will let you know in that topic
Geoid: Geoid - Wikipedia, the free encyclopedia
- if GPS says, that altitude is X, in fact it is not X, because Earth is not a sphere, so altitude must be recalculated according to geoid height in that specific location
- but I found, that geoid height in my location is only ~ 40 meters and that is still not enough, because difference is 210 - 89 = 121 m
- this site gives me same wrong altitude values as your tool does
- but this site, using TXT logs from mobile phone, gives me correct altitude values

And, please submit that .DAT to FLYXXX.DAT and tell me the file name so that I can retrieve it.

ok, I will do that

 

[BudWalker]

.........
1.
first line of CSV file starts with Tick#,time,Longitude,
second line of CSV file starts with 20365,33,942,21.258565015486017,
so I think that one column name is missing in first line
..........

This is the part I'm most interested in. Please upload the .DAT so I can take a look.
Thank you.

 

[ferraript]

This is the part I'm most interested in. Please upload the .DAT so I can take a look.
Thank you.

on 4.2.2016 at 20:54 UTC I uploaded file FLY033.DAT

I downloaded CSV file from that site and it is OK.
I found out, where the problem is with your datconverter.exe: in second column (time), you are using comma instead of dot.
It looks like this: 20365,33,942,21.258565015486017,
And it should look like this: 20365,33.942,21.258565015486017,

Or maybe I have old version of your tool? I have version 1.0.1

 

[BudWalker]

Sorry I didn't realize that this is a known problem. It has to do with localized formatting. To make it even more embarrassing a special version, called DatGerman, and distributed to pilots on a German forum. There shouldn't be any problem with the DatCon or submitting .DATs to flylog.info.

 

[ferraript]

Sorry I didn't realize that this is a known problem. It has to do with localized formatting. To make it even more embarrassing a special version, called DatGerman, and distributed to pilots on a German forum. There shouldn't be any problem with the DatCon or submitting .DATs to flylog.info.

Wouldn't it be enough just to replace all commas for dots for every csvBuffers[i] ? For example in csvLine() function?

 

[FredzMaxxUAV]

Sorry I didn't realize that this is a known problem. It has to do with localized formatting. To make it even more embarrassing a special version, called DatGerman, and distributed to pilots on a German forum. There shouldn't be any problem with the DatCon or submitting .DATs to flylog.info.

Interesting. What forum? Kopterforum?

 

[BudWalker]

Interesting. What forum? Kopterforum?

Yes, @Luap is the one who is on that forum and communicates with its members.

 

[BudWalker]

Wouldn't it be enough just to replace all commas for dots for every csvBuffers[i] ? For example in csvLine() function?

Yes, that would work for the fields that use the csvBuffers[],csvTermTypes, etc system. But, that system isn't used anymore. Its purpose is to make it easy and error proof to get a new .DAT field into the .csv. But, it won't work for columns that are derived from .DAT fields, e.g. latitude which is in radians and needs to be converted to degrees. Mixing the the csvBuffers[],csvTermTypes, etc system and hard coding the columns is difficult and very error prone.