[TOOL][WIN] Offline TXT FlightRecord to CSV Converter

Joined
Jan 11, 2017
Messages
7
Likes
0
Age
65
it took me a little while to think over the reply to your comment, so here we go:

in this case I wasn't complaining about DJI, they can mostly do good job
but they still deserve complaining, at least for two things:
- DJI GO (at least Android's version) is still very buggy and it looks like they are making things worse only
- with all those new prohibitions and conditions, it's like they are punishing users (for example it's totally inacceptable to tell users when they can fly or force them to update firmware or DJI GO)

and now to the code and decoding
with your attitude you are exactly the same as dozens of people asking us to share the decoding algorithm
it's sad that nobody has come with something like "hey, I want to help you, I found something in DJI GO / firmware that you haven't, I want to make your tool better"

for a while I was thinking about some kind of poll, where users of my tool could be able to vote if I'm (or exactly we are) to share the algorithm or not
but then I realized few things:
1. I believe DJI encrypted the logs, because they wanted to have some kind of insurance, that the logs were valid, not faked.
Because if somebody has encryption/decryption algorithm, he can abuse it and alter the logs.
I can easily see webpages offering "did you crash because of your fault? send us your TXT log, 50 bucks, and we will change the log so it will look like it was hardware fault and you can ask DJI for free repair"
It has been 19 months since DJI came up with encryption and it's 18 months since we broke it.
DJI hasn't changed the algorithm since then so I believe they are satisfied with things as they are now.
2. You say "You're not the only smart person out there; please let the rest of us help you with this".
Well, where are all those smart people? Because until now, as far as I know, there are no new services offering to read the encrypted logs, except all those old ones - Airdata, djilogs, phantomhelp and my converter.
Sadly that also means that if DJI changes the encryption in the future, there will be nobody to break it (maybe Airdata again).

So I decided I don't want to risk to lose the current win-win situation, I don't want to give DJI any reason to change the algorithm.
I don't care of what people will think of me - I'm trying to do what I think is the best for users of my tool.
I will never agree to publish the algorithm.

Anybody smart enough has to agree with me. BudWalker, the guy who invented decoding algorithm for my tool, supports me with this too.
End of story.
 
Joined
Feb 3, 2016
Messages
384
Likes
184
Location
Slovakia/Kosice
I bought Mavic Pro (Platinum), did a few TXT conversions and found out that some fixes have to be done, so new version is out

batterySN values in both RECOVER and DETAILS section were fine for older types of the drones, but reversed for the newer ones - it should be fine now

on my very first flight I had compass error with situation when Mavic switched to Atti and didn't use GPS signals
looking in the logs I realized few things, for example, that GPS values are still there and they look legit,
but also that although the Mavic was moving at the time, CUSTOM.hSpeed showed 0 (or very low values),
which means that this field represents only what Mavic thinks at the current circumstances

so there is new column CALC.hSpeed, calculated from the GPS coordinates (sometimes the values are little bit off, because GPS coordinates don't change precisely, but in general it will give you good values)
CALC.hSpeed.running_max is now based on this new CALC.hSpeed column
new column CUSTOM.hSpeed.running_max is based on "official" CUSTOM.hSpeed column
 
Joined
Feb 1, 2014
Messages
4,305
Likes
935
Location
Indy, USA
Website
multicopterpilots.com
I bought Mavic Pro (Platinum), did a few TXT conversions and found out that some fixes have to be done, so new version is out

batterySN values in both RECOVER and DETAILS section were fine for older types of the drones, but reversed for the newer ones - it should be fine now

on my very first flight I had compass error with situation when Mavic switched to Atti and didn't use GPS signals
looking in the logs I realized few things, for example, that GPS values are still there and they look legit,
but also that although the Mavic was moving at the time, CUSTOM.hSpeed showed 0 (or very low values),
which means that this field represents only what Mavic thinks at the current circumstances

so there is new column CALC.hSpeed, calculated from the GPS coordinates (sometimes the values are little bit off, because GPS coordinates don't change precisely, but in general it will give you good values)
CALC.hSpeed.running_max is now based on this new CALC.hSpeed column
new column CUSTOM.hSpeed.running_max is based on "official" CUSTOM.hSpeed column
@ferraript
have you by chance checked values using "mi/h" anytime lately? I have updated to the current most recent version(2018/04/21) of the tool and seem to have ran into issues in all hSpeed areas. In testing the data from a Mavic Pro settings are set to display in "feet" and "mph". None of the D column or CUSTOM.hSpeed [mi/h] display correctly. The H column appears to display some of the values double the amount. I have not yet tried using metric as imperial setting are the usual for me. Has anyone else mentioned such issues that you are aware of? And, is there a fix for this? Thanks.

*Edit: The metric displays in metric, but also does not display the values correctly as well as almost double amount in some cases.
 
Last edited:
Joined
Feb 3, 2016
Messages
384
Likes
184
Location
Slovakia/Kosice
@ferraript
have you by chance checked values using "mi/h" anytime lately? I have updated to the current most recent version(2018/04/21) of the tool and seem to have ran into issues in all hSpeed areas. In testing the data from a Mavic Pro settings are set to display in "feet" and "mph". None of the D column or CUSTOM.hSpeed [mi/h] display correctly. The H column appears to display some of the values double the amount. I have not yet tried using metric as imperial setting are the usual for me. Has anyone else mentioned such issues that you are aware of? And, is there a fix for this? Thanks.

*Edit: The metric displays in metric, but also does not display the values correctly as well as almost double amount in some cases.
long time no see :)
if you can, attach some txt log that gives you bad values
and also write here at least one example of what values exactly you don't like and what do you expect them to be so I can evaluate it
 
Joined
May 20, 2018
Messages
3
Likes
1
Age
47
Hi, I ended up here looking for a way to add a telemetry overlay to my P3S videos. Thank you very much for the tool to do the conversion offline!

Now my only problem is which data profile to choose in DashWare (which works fine for me since I use Win7). In an older video on YT, the guy selects Flytrex. But for me this generates an error:

"Error locating required column(s): You must map a data column to the "<Required> / Data Running Time Seconds" data type."

So which one of the many profiles would work with the current output of the tool? Thanks in advance.
 
Likes: embayweather
Joined
Jun 3, 2018
Messages
1
Likes
0
Age
55
it took me a little while to think over the reply to your comment, so here we go:

in this case I wasn't complaining about DJI, they can mostly do good job
but they still deserve complaining, at least for two things:
- DJI GO (at least Android's version) is still very buggy and it looks like they are making things worse only
- with all those new prohibitions and conditions, it's like they are punishing users (for example it's totally inacceptable to tell users when they can fly or force them to update firmware or DJI GO)

and now to the code and decoding
with your attitude you are exactly the same as dozens of people asking us to share the decoding algorithm
it's sad that nobody has come with something like "hey, I want to help you, I found something in DJI GO / firmware that you haven't, I want to make your tool better"

for a while I was thinking about some kind of poll, where users of my tool could be able to vote if I'm (or exactly we are) to share the algorithm or not
but then I realized few things:
1. I believe DJI encrypted the logs, because they wanted to have some kind of insurance, that the logs were valid, not faked.
Because if somebody has encryption/decryption algorithm, he can abuse it and alter the logs.
I can easily see webpages offering "did you crash because of your fault? send us your TXT log, 50 bucks, and we will change the log so it will look like it was hardware fault and you can ask DJI for free repair"
It has been 19 months since DJI came up with encryption and it's 18 months since we broke it.
DJI hasn't changed the algorithm since then so I believe they are satisfied with things as they are now.
2. You say "You're not the only smart person out there; please let the rest of us help you with this".
Well, where are all those smart people? Because until now, as far as I know, there are no new services offering to read the encrypted logs, except all those old ones - Airdata, djilogs, phantomhelp and my converter.
Sadly that also means that if DJI changes the encryption in the future, there will be nobody to break it (maybe Airdata again).

So I decided I don't want to risk to lose the current win-win situation, I don't want to give DJI any reason to change the algorithm.
I don't care of what people will think of me - I'm trying to do what I think is the best for users of my tool.
I will never agree to publish the algorithm.

Anybody smart enough has to agree with me. BudWalker, the guy who invented decoding algorithm for my tool, supports me with this too.
End of story.

Given that you won't publish the algorithm can you at least add an option to output the data in a .GPX format? That's the format I ultimately need it in and have written an app to convert the .CSV file to .GPX. I'd much rather have this as a one step process.
 
Joined
Feb 1, 2014
Messages
4,305
Likes
935
Location
Indy, USA
Website
multicopterpilots.com
long time no see :)
if you can, attach some txt log that gives you bad values
and also write here at least one example of what values exactly you don't like and what do you expect them to be so I can evaluate it
Good to see you too. I apologize for not responding back before now. I have been working on so much other stuff. As you might remember I have always been a freak about researching the data of each of my flights. And since discovering the issue I mentioned above I simply stopped flying since the data was not displaying correctly. Let me get back into this and refresh my memory with and I will post again soon. :smiley:
 
Joined
Feb 3, 2016
Messages
384
Likes
184
Location
Slovakia/Kosice
can you at least add an option to output the data in a .GPX format?
it should be doable, let me think about it
by the way, I also create GPX files for myself, but I use CSVs generated by DatCon (from DAT logs) as the source, because TXT logs often have large gaps

Let me get back into this and refresh my memory with and I will post again soon.
good luck :D
 
Joined
Feb 1, 2014
Messages
4,305
Likes
935
Location
Indy, USA
Website
multicopterpilots.com
Good to see you too. I apologize for not responding back before now. I have been working on so much other stuff. As you might remember I have always been a freak about researching the data of each of my flights. And since discovering the issue I mentioned above I simply stopped flying since the data was not displaying correctly. Let me get back into this and refresh my memory with and I will post again soon. :smiley:
incorrect2.jpg

********************************************************************************************
incorrect.jpg



Ok, so I did not look at all the categories for errors. There are 3 errors above that I would call common as this same issue has come up before from previous firmware updates to the GO4 app. I don't know why DJI has to make changes to these type of files but it probably explains why they have some of the issues they do from time to time. If I were to predict when these changes took place I would guess it changed when the Platinum came onto the scene.

@ferraript , thanks for looking into this!

*EDIT: I just found the Battery Cell #1, #2, #3, data amounts are off and need attention.
Also, the SMART_BATTERY.volumeConsume (EE category) appear to be way off.

I'm sorry for being all over the place with this info. But I'm at this point I hope there is an easy fix like there was in the past. I will edit more if I happen to stumble upon any additional.
 
Last edited:
Joined
Feb 3, 2016
Messages
384
Likes
184
Location
Slovakia/Kosice
I'm sorry for being all over the place with this info. But I'm at this point I hope there is an easy fix like there was in the past
it's no problem at all to report problems/errors
fixing is easy part, but I need to distinguish which logs need to be adjusted

SMART_BATTERY.volumeConsume ... I don't even know what this is, so I can't tell if the values are ok or not - if I look into logs from my P3A or Mavic, they don't make any sense to me
Battery Cell #1, #2, #3 ... these values look fine for logs I've got

CALC values were added just recently and I explained that 1. they are for reference only, 2. they are calculated from GPS coordinates, so they won't be precise all the time
so there is nothing I can/will do about them
maybe I will remove CALC.hSpeed.running_max column as it is really useless

with CUSTOM.hSpeed (or any other potential columns) it's the problem I started to explain
for example for my logs these values are fine
and I suppose these values were also fine for your logs, but in one moment they started to be off
so if you want me to fix this, I will need cooperation of you and other users to find this moment, so I can distinguish, whether I read/write the values just like I did before or I need to use new method (in this specific case to multiply them by 10)
some of the possible detection signs could be logVersion (11th byte of TXT log), OSD.flycVersion or DJI GO version
and of course I need to know if it's only iOS relevant or it's valid for Androids too

if it's too complicated for you, then easier way could be to send me 2 consecutive logs, where first log's values are still fine, but second log's values are off
and I will try to adjust my tool according to information I'll get from them
 
Joined
Feb 1, 2014
Messages
4,305
Likes
935
Location
Indy, USA
Website
multicopterpilots.com
If you can provide an email address I can send you a good(old) & not good(new) files.

I spent a little more time looking throughout the converted version of my most recent flight and there are other red flags popping up, some columns no longer have data, and most columns appear to have NO problems. Because of this is why I should send the files to you.

What bothers me about this is why has nobody else mentioned this before I discovered it? Is others just not having any issues, or not paying attention and noticing a revision is needed?

Btw, this is on Android. My good(old) file is from app version 4.1.8. The not good(new) file is from app version 4.1.22.
 
Joined
Feb 3, 2016
Messages
384
Likes
184
Location
Slovakia/Kosice
I can send you a good(old) & not good(new) files
PM sent

>why has nobody else mentioned this before I discovered it?
no idea, maybe most of the people use DatCon and CsvView as they provide full flight data (I also prefer DatCon as TXT logs have large gaps, especially when AC is far away) and luckily DJI is not changing the structure/format of DAT logs like they are doing in DJI GO
 
Joined
Jun 24, 2015
Messages
1,294
Likes
471
Age
70
Location
Santa Margarita, Ca
PM sent

>why has nobody else mentioned this before I discovered it?
no idea, maybe most of the people use DatCon and CsvView as they provide full flight data (I also prefer DatCon as TXT logs have large gaps, especially when AC is far away) and luckily DJI is not changing the structure/format of DAT logs like they are doing in DJI GO
Another reason is that the columns most people use don't have problems. An exception is the recent problem with the the individual battery cell voltages. There were a lot of people that noticed that which lead to the fix you did.

TXTlogToCSSVtool is widely used. Especially for the lost drone scenario.
 
Likes: sar104
Joined
Feb 3, 2016
Messages
384
Likes
184
Location
Slovakia/Kosice
new (major) version is out
- CALC.hSpeed.running_max removed as it was useless
- the output filename is now automatically assigned according to the input filename
- problem reported by flyNfrank should be solved now
- added possibility to export GPX file (also in command line version - parameter G) as requested by JohnF
- new group of records - CAMERA_INFO displays information about what camera is doing, SD card info, etc...
- new group of records - MC_PARAM displays DJI GO settings of failSafe and Obstacle Avoidance (2 notices: 1. this record is present in the TXT log on every 100th row so the real delay can be more than 10 seconds; 2. I tested it myself and sometimes it doesn't report correct data, so we can't completely rely on it)

notice: we are now at 274 columns so if there is someone like me who used to convert CSV to XLS, well, this is now not possible as XLS can hold 255 columns only :(

BudWalker: FYI
 
Last edited:
Joined
Sep 6, 2017
Messages
3
Likes
3
Website
www.framestamp.io
new (major) version is out
- CALC.hSpeed.running_max removed as it was useless
- the output filename is now automatically assigned according to the input filename
- problem reported by flyNfrank should be solved now
- added possibility to export GPX file (also in command line version - parameter G) as requested by JohnF
- new group of records - CAMERA_INFO displays information about what camera is doing, SD card info, etc...
- new group of records - MC_PARAM displays DJI GO settings of failSafe and Obstacle Avoidance (2 notices: 1. this record is present in the TXT log on every 100th row so the real delay can be more than 10 seconds; 2. I tested it myself and sometimes it doesn't report correct data, so we can't completely rely on it)

notice: we are now at 274 columns so if there is someone like me who used to convert CSV to XLS, well, this is now not possible as XLS can hold 255 columns only :(

BudWalker: FYI
Thanks for all the work you put into this tool !

Is there any chance the stored video file name / ID would be exposed as a column ?
Also, is there an option to output 9 decimal precision (instead of the default 6) using the command line ?

Thanks.
 
Joined
May 20, 2018
Messages
3
Likes
1
Age
47
Hi, I ended up here looking for a way to add a telemetry overlay to my P3S videos. Thank you very much for the tool to do the conversion offline!

Now my only problem is which data profile to choose in DashWare (which works fine for me since I use Win7). In an older video on YT, the guy selects Flytrex. But for me this generates an error:

"Error locating required column(s): You must map a data column to the "<Required> / Data Running Time Seconds" data type."

So which one of the many profiles would work with the current output of the tool? Thanks in advance.
Quoting myself, since I still haven't found the solution. If no replies mean that it simply doesn't work with DashWare, please inform me, if there is a better way - thanks a lot!
 
Joined
Feb 3, 2016
Messages
384
Likes
184
Location
Slovakia/Kosice
Is there any chance the stored video file name / ID would be exposed as a column ?
no, this information is not present in the logs

is there an option to output 9 decimal precision (instead of the default 6) using the command line ?
what for? you are pushing it...
I chose 6 decimal digits because it gives precision 16 cm (6")
even GPS is not so precise, so there is absolutely no sense of using more decimal digits

Quoting myself, since I still haven't found the solution
I didn't answer your question because I hoped somebody else would
I don't use Dashware, can't help you, sorry
I think somebody found a way, it's somewhere in this thread, so keep looking...
 

New Threads

Forum statistics

Threads
129,755
Messages
1,347,463
Members
94,558
Latest member
Bbsvenom