Welcome to PhantomPilots.com

Sign up for a weekly email of the latest drone news & information

[TOOL][WIN] Offline TXT FlightRecord to CSV Converter

Discussion in 'Pro/Adv Discussion' started by ferraript, Mar 6, 2016.

  1. msinger

    Joined:
    Oct 30, 2014
    Messages:
    29,658
    Likes Received:
    9,495
    Location:
    Harrisburg, PA (US)
    Which link?
     
  2. Hans 75

    Joined:
    Apr 15, 2016
    Messages:
    23
    Likes Received:
    1
    Location:
    Berlin
    Please, have a look to #538. I can´t download the new version of Ferraripts program.
    May be, you have a good idea?
    H.
     
  3. msinger

    Joined:
    Oct 30, 2014
    Messages:
    29,658
    Likes Received:
    9,495
    Location:
    Harrisburg, PA (US)
    You can download it from the bottom of post #1.
     
  4. BudWalker

    Joined:
    Jun 24, 2015
    Messages:
    1,282
    Likes Received:
    469
    Location:
    Santa Margarita, Ca
    From here
    upload_2017-12-28_5-3-51.png
     
    Digdat0 likes this.
  5. Hans 75

    Joined:
    Apr 15, 2016
    Messages:
    23
    Likes Received:
    1
    Location:
    Berlin
    Thank you, that I know almost from the beginning of the program. But my Firefox opens for a moment a new tab and that´s all.
    H.
    Oh, sorry, the downloading is so quick. I have it!
     
    #545 Hans 75, Dec 28, 2017
    Last edited: Dec 28, 2017
  6. unocifra

    Joined:
    Jan 12, 2018
    Messages:
    1
    Likes Received:
    0
    @Ferraripts Is it possible to get the source code? Would like to have a Linux version also.
     
  7. ferraript

    Joined:
    Feb 3, 2016
    Messages:
    381
    Likes Received:
    185
    Location:
    Slovakia/Kosice
    nope
    use Windows emulator, you won't be the first
     
  8. ferraript

    Joined:
    Feb 3, 2016
    Messages:
    381
    Likes Received:
    185
    Location:
    Slovakia/Kosice
    new version is out
    it fixes one small bug
    in rare cases text fields (like APP_SER_WARN.warn) contained new line character - that led to broken CSVs
    so new line character is now replaced with semicolon (or comma, if you use semicolon as delimiter)
    thanks, BudWalker, for pointing that out
     
    Hans 75, speleo, sar104 and 1 other person like this.
  9. BudWalker

    Joined:
    Jun 24, 2015
    Messages:
    1,282
    Likes Received:
    469
    Location:
    Santa Margarita, Ca
    Thank you
     
  10. sar104

    Joined:
    Mar 22, 2014
    Messages:
    4,085
    Likes Received:
    1,833
    Location:
    Los Alamos, NM.
    Yes - thanks for all your work on this - invaluable.
     
  11. ferraript

    Joined:
    Feb 3, 2016
    Messages:
    381
    Likes Received:
    185
    Location:
    Slovakia/Kosice
    fool's day brings you another new version :)

    but this time there is some bad news too

    changes:
    - added better support for Mavic Air - especially correct cell voltages reporting (note: BudWalker did let me know about this problem and I had 1 source TXT only so I just hope it will work fine for all of you)
    - added some battery info for Mavics - if exists in the log
    - added a few new text constants for flycCommand and other fields

    and now a little bit bad news :neutral:
    it's been a long time since I decompiled DJI GO apk, my main source of information
    and few days ago I tried it again with newest version (4.2.8) and I found out two things:
    1. Since 4.1.4, DJI encoded all text constants, so it's harder to read the code. Fortunately some clever guys found the decryption algorithm so it's possible to decode the sources.
    2. Since 4.1.10, DJI must have added some protection to APKs, because decompilation is now basically not helpful at all - part of the code is decompiled fine, but part of the code is not decompiled at all and part of the code is totally missing.

    conclusion: at the moment, the tool is based on the sources from DJI GO 4.1.9
    in the future, if DJI changes important parts of the DJI GO, it will be very hard (maybe impossible) to adapt my tool for it
    lets just hope they don't do that anytime soon...
     
  12. finlayson

    Joined:
    Jun 30, 2014
    Messages:
    130
    Likes Received:
    14
    Don't you find it at least a bit ironic that you're complaining about DJI keeping their code (and their encoding mechanism) secret - yet you yourself refuse to release the source code for your decoding application, or describe in detail what you've learned about DJI's encoding (which would allow other people to help you keep up with DJI's apparent continued obfuscation).

    So please, stop being so secretive. You're not the only smart person out there; please let the rest of us help you with this. Please release your source code, or at least a detailed description of what you've learned about how DJI's TXT logs are currently encoded.
     
  13. ferraript

    Joined:
    Feb 3, 2016
    Messages:
    381
    Likes Received:
    185
    Location:
    Slovakia/Kosice
    it took me a little while to think over the reply to your comment, so here we go:

    in this case I wasn't complaining about DJI, they can mostly do good job
    but they still deserve complaining, at least for two things:
    - DJI GO (at least Android's version) is still very buggy and it looks like they are making things worse only
    - with all those new prohibitions and conditions, it's like they are punishing users (for example it's totally inacceptable to tell users when they can fly or force them to update firmware or DJI GO)

    and now to the code and decoding
    with your attitude you are exactly the same as dozens of people asking us to share the decoding algorithm
    it's sad that nobody has come with something like "hey, I want to help you, I found something in DJI GO / firmware that you haven't, I want to make your tool better"

    for a while I was thinking about some kind of poll, where users of my tool could be able to vote if I'm (or exactly we are) to share the algorithm or not
    but then I realized few things:
    1. I believe DJI encrypted the logs, because they wanted to have some kind of insurance, that the logs were valid, not faked.
    Because if somebody has encryption/decryption algorithm, he can abuse it and alter the logs.
    I can easily see webpages offering "did you crash because of your fault? send us your TXT log, 50 bucks, and we will change the log so it will look like it was hardware fault and you can ask DJI for free repair"
    It has been 19 months since DJI came up with encryption and it's 18 months since we broke it.
    DJI hasn't changed the algorithm since then so I believe they are satisfied with things as they are now.
    2. You say "You're not the only smart person out there; please let the rest of us help you with this".
    Well, where are all those smart people? Because until now, as far as I know, there are no new services offering to read the encrypted logs, except all those old ones - Airdata, djilogs, phantomhelp and my converter.
    Sadly that also means that if DJI changes the encryption in the future, there will be nobody to break it (maybe Airdata again).

    So I decided I don't want to risk to lose the current win-win situation, I don't want to give DJI any reason to change the algorithm.
    I don't care of what people will think of me - I'm trying to do what I think is the best for users of my tool.
    I will never agree to publish the algorithm.

    Anybody smart enough has to agree with me. BudWalker, the guy who invented decoding algorithm for my tool, supports me with this too.
    End of story.
     
    schach, phlo, Dronason and 3 others like this.
  14. finlayson

    Joined:
    Jun 30, 2014
    Messages:
    130
    Likes Received:
    14
    You just don't get it. The reason nobody wants to help you "make your tool better" is because you insist on keeping your tool (and its algorithm) to yourself.

    But anyway - your loss. The Internet will 'route around' you, rendering you irrelevant.
     
  15. ferraript

    Joined:
    Feb 3, 2016
    Messages:
    381
    Likes Received:
    185
    Location:
    Slovakia/Kosice
    my loss? sorry, I can see no loss for me in this :)

    yes, once in the future, internet will find the way
    but for 18 months it hasn't, so we are the ones who decide the conditions
     
    speleo likes this.
  16. quaddamage

    Joined:
    Jun 30, 2016
    Messages:
    627
    Likes Received:
    176
    Location:
    PL
    Looks like now we have Software Justice Warriors to add to the pack...

    I hope you will go write the same thing to owners of all the online converters, especially paid ones. Otherwise you're picking the person who is the most open and provides the most information about the log format.

    If a good programmer is really interested in the source, he will reverse-engineer the tool with no problem. A basic one who can't do this, is unlikely to be of any help anyway.

    (I get triggered by demanding attitude towards people who provide their work for free; sorry)
     
  17. finlayson

    Joined:
    Jun 30, 2014
    Messages:
    130
    Likes Received:
    14
    Really? What is the URL of the web page that describes the log format?

    This is the one thing that would be most useful to people. If no such page currently exists, then I guess I'll have to write one myself. Stay tuned...
     
  18. quaddamage

    Joined:
    Jun 30, 2016
    Messages:
    627
    Likes Received:
    176
    Location:
    PL
    Oh. Ok, I did not expected that.

    Here is a Wireshark dissector which can read the DAT logs within the drone, and DUML packets from internal communication between modules within the drone:
    o-gs/dji-firmware-tools

    The dissector is provided as a series of LUA scripts, which are relatively easy to understand.

    It is likely that the TXT format will contain packets similar to the DUML format (or just plain dumps of the DUML packets).
     
  19. ferraript

    Joined:
    Feb 3, 2016
    Messages:
    381
    Likes Received:
    185
    Location:
    Slovakia/Kosice
    sorry, I made a mistake there
    the correct version of the citation in my post should be "hey, I want to help other people too..."
    because the tool is for other users, not for me
    the version sufficient for me was completed 2 years ago, I personally need nothing of these new fields

    it's funny, I had the same thing on my mind :)
    and don't forget about DJI!!!

    there was Excel document attached that described it in detailed way
    but I removed it
    read Notice 2 in the first post of this thread

    anyway, the discussion moved far away from the topic and I won't be wasting any more time with explaining or arguing, because in the end it's useless, it brings nothing good
     
    Dronason, BudWalker and speleo like this.
  20. ferraript

    Joined:
    Feb 3, 2016
    Messages:
    381
    Likes Received:
    185
    Location:
    Slovakia/Kosice
    new version is out
    last version maybe fixed some Mavics' cell voltages, but it also broke some others'
    this time I did it (finding the offset of voltages) by another method, it should be working fine for all logs

    if not, don't hesitate to let me know!
     
    sar104 and speleo like this.
Loading...
Similar Threads - [TOOL][WIN] Offline TXT Forum Date
offline Litchi Waypoint upload Software Jul 29, 2018
Offline maps on P4+ - since update Phantom 4 Help Jul 23, 2018
Mapbox - Offline satelite images? Phantom 4 Discussion Jul 8, 2018
DJI Go 4.2.16 and MAPBOX - Offline Control (Go, Litchi, etc..) Jul 5, 2018
DJI Go mod offline waypoint mission (Including spark) Control (Go, Litchi, etc..) May 20, 2018