Ultrasonic Systems Error

[sar104]

The data from this flight provides a one of a kind opportunity to observe how GPS data is used. Not really pertinent to see what happened, just interesting - to me at least.

The Lat,Long coords used by the FC don't come directly from the GPS module. Rather, the Lat,Long coords are derived from the GPS data and IMU data. By fusing these data it's possible to obtain higher accuracy coords (both temporally and position) than can be obtained from either GPS data or IMU data. This flight allows us to observe the fused coords with GPS data held constant. Here the red plot is GPS longitude data (held constant) and the blue plot is the fused longitude data.
View attachment 96113

Presumably the changes in the fused longitude are due to just IMU data shown here. The pink background are where the FC has decided the GPS data can't be used and coincides with abrupt changes in IMU data.

View attachment 96114

I think the reason for the saw-toothed lat/long data is that the GPS health is still listed as 3, and so the FC is still trying to use the GPS lat/long (now constant) via the Kalman filter to correct for IMU drift and bias. The IMU data indicate that the aircraft is still moving, but the GPS says that it is stationary, and the GPS data are causing it to reset its estimate of position much more aggressively than it should in that situation.

What I don't understand is why the process has not been abandoned in the same way that when magnetic yaw and IMU yaw disagree the FC stops the sensor fusion process in terms of computing yaw. If the GPS totally failed that would happen, but it appears that the code does not have a contingency to cope with spurious GPS data that are being reported as good.

 

[BudWalker]

I think the reason for the saw-toothed lat/long data is that the GPS health is still listed as 3, and so the FC is still trying to use the GPS lat/long (now constant) via the Kalman filter to correct for IMU drift and bias. The IMU data indicate that the aircraft is still moving, but the GPS says that it is stationary, and the GPS data are causing it to reset its estimate of position much more aggressively than it should in that situation.

What I don't understand is why the process has not been abandoned in the same way that when magnetic yaw and IMU yaw disagree the FC stops the sensor fusion process in terms of computing yaw. If the GPS totally failed that would happen, but it appears that the code does not have a contingency to cope with spurious GPS data that are being reported as good.

I think that's right. Just spit balling here. In the case of the Yaw/magYaw separation it seems there is a threshold that has to be reached before gpsHealth drops. And, probably the same is true for the GPS coords vs IMU coords error. Maybe the P3 wasn't moving enough to make this error large enough. Except in those intervals where gpsHealth dropped and GPS data wasn't being used. Sorry, I should've included the gpsHealth signals.

It is a bit puzzling though. Outside of those intervals gpsHealth == 3. I thought that the threshold was gpshealth >= 4 for GPS to be used. But, there have been a few incidents where it seems that if gpsHealth had ever been >=4 then the threshold becomes gpsHealth >= 3.

 

[Sir Nobby]

How about at about 13 min 18 sec when it veers to the right, goes way over the dyke and over the country club, but, that's not reflected in the path from the kml file is it ? That was the turn that wasn't commanded. Also, I'm referring to the time stamp on the video, I'm not certain how that matches up to the flight record.

Hi, totally agree with what you say, video shows you most definitely went over golf course but isn't reflected in the data. The crazy gimbal angles too! Looks as if you were fighting for control.
The reason I say you were over the country club was because you can clearly see the golf course when you are trying to get back towards the ponds.

 

[BudWalker]

Hi, totally agree with what you say, video shows you most definitely went over golf course but isn't reflected in the data. The crazy gimbal angles too! Looks as if you were fighting for control.
The reason I say you were over the country club was because you can clearly see the golf course when you are trying to get back towards the ponds.

The Gimbal pitch was -10.2° during this part of the video; i.e. almost horizontal.

 

[solentlife]

"Elbert Hill" :

Regardless of all the techno ..... I am still trying to figure out why you didn't just land out and manually retrieve ?

Nigel

 

[Sir Nobby]

The Gimbal pitch was -10.2° during this part of the video; i.e. almost horizontal.
View attachment 96126

Hi @BudWalker and @solentlife - you both have excellent skills with this kind of data and I do have a lot of sympathy for the OP as I too have had a very bad experience with my P3A which was a week old at the time (can't give you any data on that as I deleted everything regarding it) as it too took on a life of its own and began drifting away from me despite being in GPS mode. My P3A was giving me compass errors and flipping in and out of GPS mode and ATTI mode every few seconds but most frighteningly began climbing to 400 metres! The battery gave out at this point and it was over 500 ft away from me (darkness was descending) but managed to freewheel it back to within 60 ft using the local village streetlights as a makeshift runway. Found it 3 days later having managed to land it in a hedge hanging by its two front arms, not even a scratch on it! Still fly it as if nothing ever happened but I've never flown from that spot since...

 

[Elbert Hill]

I really do appreciate the extensive analysis of my flight data. Fortunately, I have a back up P3P to use for now.

Looking at YouTube today I found this video that seems to describe these exact same problems and behaviours. Here's the link

This video also states that I guess he sent it into DJI and they sent it back repaired.

This P3P is long out of warranty so I'm inclined to try and figure it out on my own. With that being said, I've got an extra Ultrasoni8c module which I will change out, in the hope that may have had something to do with GPS failure, along with the GPS antenna inside the top of the shell. A couple of question

• Please correct me if I'm wrong but isn't the GPS built into the main ESC board ? I'm pretty sure it is, but It's brand new board pn 96, well almost, had it in for a month or two with no issues.
  
• May be a stupid question, are compass errors and GPS errors the same or similar ?

In the mean time I'm not going to fly this drone very far away, short test flights only as you suggested, for quite some time until I can be confident of it's reliable performance

 

[Elbert Hill]

"Elbert Hill" :

Regardless of all the techno ..... I am still trying to figure out why you didn't just land out and manually retrieve ?

Nigel

Actually, I did do as you mention, I landed about a mile from "home" and manually retrieved. The reasons I didn't do it earlier was:

• I had flown this course many many times with no problems
• by the time I realized I had serious problems, was when the craft veered to the right, away from the straight path it was on to RTH
• it was fighting my attempts to turn away to get back on course to home
  
• and by now it was over other side of dyke and well over into the country club where I DID NOT want it to go down

As soon as I was able to get back on the other side of the dyke I initiated descent, but then it wanted to land in some desert percolation ponds, so that took me another 20 seconds or so before I managed to veer it off of that course, and then finally land.

Then, I hiked about a mile from where "home" was at , to the area where I finally landed it , and in about an hour, I was able to find it.And remarkably, with no damage other than a broken prop. Now it's all about trying to figure out why.

But hindsight is 20/20, if I had known ahead of time just how bad these problems were going to cascade into, then yes I would have put it down much sooner.

 

[BudWalker]

I really do appreciate the extensive analysis of my flight data. Fortunately, I have a back up P3P to use for now.

Looking at YouTube today I found this video that seems to describe these exact same problems and behaviours. Here's the link

This video also states that I guess he sent it into DJI and they sent it back repaired.

This P3P is long out of warranty so I'm inclined to try and figure it out on my own. With that being said, I've got an extra Ultrasoni8c module which I will change out, in the hope that may have had something to do with GPS failure, along with the GPS antenna inside the top of the shell. A couple of question

• Please correct me if I'm wrong but isn't the GPS built into the main ESC board ? I'm pretty sure it is, but It's brand new board pn 96, well almost, had it in for a month or two with no issues.
  
• May be a stupid question, are compass errors and GPS errors the same or similar ?

In the mean time I'm not going to fly this drone very far away, short test flights only as you suggested, for quite some time until I can be confident of it's reliable performance

The GPS problems here are HW errors and not related to compass errors.

I believe the GPS module is separate from the main board. I gotta ask. Is the GPS module connected properly to the main board? Replacing the main board would have required the GPS module be reconnected. The GPS module data has all the earmarks of a disconnected module.

 

[Elbert Hill]

Bud
I took it for a couple of short flights yesterday and everything was great, no errors. That certainly doesn't mean I trust i for a second though.

I did replace the Ultrasonic module though.

I looked into it and found the GPS module is attached to the GPS antennae inside the top half of the clamshell. I've got at least one, maybe two others as spares, so I'll change that out and see if that seems to make a difference in the long term. Also, on that same note, in the past I have had a problem or two with that connector from the GPS module to the board came loose. There's a reason DJI glues those darn plugs in place.

 

[BudWalker]

Bud
I took it for a couple of short flights yesterday and everything was great, no errors. That certainly doesn't mean I trust i for a second though.

I did replace the Ultrasonic module though.

I looked into it and found the GPS module is attached to the GPS antennae inside the top half of the clamshell. I've got at least one, maybe two others as spares, so I'll change that out and see if that seems to make a difference in the long term. Also, on that same note, in the past I have had a problem or two with that connector from the GPS module to the board came loose. There's a reason DJI glues those darn plugs in place.

Just to make sure I understand. The GPS module and antenna are both underneath the foil in the top half shell The cable coming out from under that foil is from the module not the antenna. And, it's the connector on that cable that plugs into the main board. Since you've had trouble with that connector before I would think it very likely that is the issue here.

 

[sar104]

Just out of curiosity I used the roll, pitch and yaw data (assuming the usual Tait-Bryan Euler angle representation) to transform the raw accelerometer data into the earth frame of reference. That allows it to be integrated once w.r.t. time to get velocity and twice to get displacement, for comparison with the recorded data. It's effectively the pure IMU-computed motion prediction without the Kalman filter GPS element. The results are somewhat strange, even just in terms of velocity:

I expected the values to disagree after the the GPS failure since the IMU velocities were being modified by the incorrect GPS data, but things start to go wrong much earlier, around 500 s. I'm not sure what to make of that, but it makes me wonder if the GPS data were already going bad at that point.

Anyone have any thoughts on that?

 

[sar104]

I forgot to point out the other obvious feature of the accelerometer data - that after around 860 s the integrated accelerometer data clearly becomes non-physical. The velocity east values show the same problem. I don't see how that can be due to bad GPS data, which just makes the results more confusing, but is coincident with the start of rather strange yaw data.

 

[BudWalker]

Just out of curiosity I used the roll, pitch and yaw data (assuming the usual Tait-Bryan Euler angle representation) to transform the raw accelerometer data into the earth frame of reference. That allows it to be integrated once w.r.t. time to get velocity and twice to get displacement, for comparison with the recorded data. It's effectively the pure IMU-computed motion prediction without the Kalman filter GPS element. The results are somewhat strange, even just in terms of velocity:

View attachment 96220

I expected the values to disagree after the the GPS failure since the IMU velocities were being modified by the incorrect GPS data, but things start to go wrong much earlier, around 500 s. I'm not sure what to make of that, but it makes me wonder if the GPS data were already going bad at that point.

Anyone have any thoughts on that?

I looked at the GPS:Time signal in the [518, 538] interval with the sampling rate at maximum. I was looking to see if there was any variation in the number of samples that corresponded to a particular value of GPS:Time. If so, then that would indicate that the GPS module was reporting values intermittently. I didn't see any of this - each GPS:Time value had exactly 255 samples. This isn't a conclusive test, but out of 20 GPS:Time values no anomalies were seen.

Regarding the accelerometer data. I'm assuming that you used the Vel:N signal as your Velocity:N (FC fusion computed). I had assumed (like you I think) this to be dependent via a Kalman filter on GPS data. For a while I was thinking that it may not be dependent on GPS data, but convinced myself that it is. Don't know what to say about this. Are you using the highest sampling rate? Not sure that would make a difference. Although it might help with the non-physical issue in your next post.

An ancillary issue is that regarding the possibility raised by @Elbert Hill and @Sir Nobby that the P3 got as far east as the Golf course. I.e. location data is erroneous and does not report the true position over the golf course. Using the video and the fact that the P3 camera has a 66° vertical FOV I was able to determine that the P3 did not get that far east.

 

[sar104]

I looked at the GPS:Time signal in the [518, 538] interval with the sampling rate at maximum. I was looking to see if there was any variation in the number of samples that corresponded to a particular value of GPS:Time. If so, then that would indicate that the GPS module was reporting values intermittently. I didn't see any of this - each GPS:Time value had exactly 255 samples. This isn't a conclusive test, but out of 20 GPS:Time values no anomalies were seen.

Regarding the accelerometer data. I'm assuming that you used the Vel:N signal as your Velocity:N (FC fusion computed). I had assumed (like you I think) this to be dependent via a Kalman filter on GPS data. For a while I was thinking that it may not be dependent on GPS data, but convinced myself that it is. Don't know what to say about this. Are you using the highest sampling rate? Not sure that would make a difference. Although it might help with the non-physical issue in your next post.

An ancillary issue is that regarding the possibility raised by @Elbert Hill and @Sir Nobby that the P3 got as far east as the Golf course. I.e. location data is erroneous and does not report the true position over the golf course. Using the video and the fact that the P3 camera has a 66° vertical FOV I was able to determine that the P3 did not get that far east.

Yes - Vel:N were the data (red trace above) that I was comparing to. I directly integrated the raw accelerometer data after transforming the frame of reference, to get the green trace. I checked to see if it was sample-rate sensitive - the answer is yes but only slightly. The non-physical velocities later in the flight are approximately the same whether I use 10 Hz or 200 Hz. Having watched the video, I think the problem may be that the yaw data are wrong during that period.

The velocity traces in this case are a pretty direct example showing that there is quite significant weighting applied to the GPS data in calculating velocity as well as position.

 

[Elbert Hill]

Bud and Sar, you guys a speaking of technical stuff I am not really familiar with.Bur Sar, what exactly did you mean by "significant weighting applied to the GPS data"if you could put it into laymans terms.

Does to perhaps mean the craft was somehow overreacting to data being fed to it ?

On another note, I took the faulty bird out there yesterday, to see if there were any external influences/interference. As I approached.....slowly I should add, I noticed I started getting a sketch signal, momentary loss of video feed, I turned the craft around and hi tailed it back home. THEN,,,Today, I took my other P3Pro out, and flew same path as yesterday, and I didn't get any anomilies at all, flew a smooth semi circle sort of, no problem.

I ordered a new GPS module tonite, found a new one at a good price. I really hope thats it.

But please let me know the answer to the above statement if you could Sar

Thanks

 

[Elbert Hill]

Just to make sure I understand. The GPS module and antenna are both underneath the foil in the top half shell The cable coming out from under that foil is from the module not the antenna. And, it's the connector on that cable that plugs into the main board. Since you've had trouble with that connector before I would think it very likely that is the issue here.

Bud
Yes, that is correct ! I thought it was just the antennae but it turns out theres a little processor board on it, plus I'm sure your aware of that "github" website where they have schematics and pictures of all the components for the P3P among others.I was also able to confirm that was the GPS Module.
As it regards whether or not it came unplugged from the main board or not. When I separated the shell it was still plugged in so I don't think that had anything to do with it. Plus remember that I flew it again for a few minutes after retrieving it, and everything seemed ok. You said you saw that flight data too. Maybe....perhaps..it may have been a little loose in the connector, but it didn't seem to be.

Thanks loads for all the analysis everybody has done on this, especially Bud and Sar....even if I don't understand everything you discussed with Bud.

Thanks agin'

 

[sar104]

Bud and Sar, you guys a speaking of technical stuff I am not really familiar with.Bur Sar, what exactly did you mean by "significant weighting applied to the GPS data"if you could put it into laymans terms.

Does to perhaps mean the craft was somehow overreacting to data being fed to it ?

Effectively, yes - overreacting to the spurious data from the GPS module. The FC primarily uses accelerometer data, integrated with respect to time, to calculate its movements, with the GPS data as a sort of sanity check to correct for the slight drift and bias in the accelerometer values.

If you look at the green trace in the graph above - that is the north component of velocity calculated purely from the accelerometer, rate gyros and compass data - I did not use any GPS information to derive that. For the first part of the flight it follows the FC's calculated north velocity, which is corrected with GPS data, pretty well. After the GPS stops updating the FC's corrected velocity becomes nonsense because the GPS data are telling it that it is stationary, while the IMU data tell it that it is moving. What surprised me was how aggressively it used the spurious GPS data to correct the velocity data.

The other surprise is that shortly after the FC's velocity calculation goes bad (due to the the GPS data issue), my computed velocity data also go unphysical - it was clearly never heading south at over 100 m/s. Either I'm missing something basic here or it was not just the GPS module that stopped working. The fact that you have flown it since with GPS working could also be argued to point to something other than a simple physical connection issue with the GPS module, but I have no idea what might account for that. This is a new one to me and may be determined by so many unknown details about the control system that we have no realistic prospect of figuring it out.

 

[Elbert Hill]

Effectively, yes - overreacting to the spurious data from the GPS module. The FC primarily uses accelerometer data, integrated with respect to time, to calculate its movements, with the GPS data as a sort of sanity check to correct for the slight drift and bias in the accelerometer values.

If you look at the green trace in the graph above - that is the north component of velocity calculated purely from the accelerometer, rate gyros and compass data - I did not use any GPS information to derive that. For the first part of the flight it follows the FC's calculated north velocity, which is corrected with GPS data, pretty well. After the GPS stops updating the FC's corrected velocity becomes nonsense because the GPS data are telling it that it is stationary, while the IMU data tell it that it is moving. What surprised me was how aggressively it used the spurious GPS data to correct the velocity data.

The other surprise is that shortly after the FC's velocity calculation goes bad (due to the the GPS data issue), my computed velocity data also go unphysical - it was clearly never heading south at over 100 m/s. Either I'm missing something basic here or it was not just the GPS module that stopped working. The fact that you have flown it since with GPS working could also be argued to point to something other than a simple physical connection issue with the GPS module, but I have no idea what might account for that. This is a new one to me and may be determined by so many unknown details about the control system that we have no realistic prospect of figuring it out.

Rightly or wrongly I've always thought that usually when some kind of electronics "goes out" it usually just goes out like a burnt out light bulb, and usually doesn't come and goor slowly fade away. Is that line of thought correct or not.....or is it a "depends" ?

And gosh, I simply hate the prospect that we may never figure it out completely as to what went wrong. I hate mysteries !

I guess the only hope I have at this point is that the new GPS module I ordered will prevent any recurrence of this in the long run.

 

[solentlife]

I hope you haven't jumped the gun and bought a module for nothing ...

The DJI series are seriously integrated in their systems data flows and 'actions' based on. That is why I said very early on in my opinion due to information received by Flight Controller - it basically got 'confused' and AC was in state of no real outcome except bad. The information rec'd by the FC is compared / integrated / rejected / accepted - resulting in commands being created and sent out to motors to adjust etc. It can be one item in error or more ...

With all the respect to others who have gone to great technical length ... I still feel that the only people who can really say what is wrong are DJI themselves or third party repair services who have the means to connect directly to your machine and do full diagnostics. I do not believe that DJI have programmed for ALL info to be available via dat files and logs that we can look at.

I also over significant time come to conclusion that DIY repair is questionable. So many threads about people replacing parts and still not working ...
I recently could have rebuilt my P3P .... but decided that giving to DJI even though an expensive action - gave me warranty and also a machine that would work when sent back. No guessing ... no sifting through dat / log files ... no hoping that new gimbal etc. would work ...

It came back in totally good order.

Sorry if I sound -ve ... not my intention ... I just hope things do sort ..

Nigel