Autonomous camera aiming & tracking module

[xtracrispy]

Hello,

So I've been playing around with drone videography lately, and quickly realized that it's really tough to aim the camera while flying a quad in anything but a straight line.
I envisioned this one sweeping pan shot of an object in a field, but just couldn't do it with the quadrotor rolling around due to my flight control.

Anyway, I'm a roboticist and hacked together a working prototype that uses the video stream to track anything I want, using computer vision algorithms.

Here's a quick youtube video to show what I mean:
https://www.youtube.com/watch?v=QJW9ReKU-0I&list=UUy6SYEB_Gk40j9NOy8RI5aQ

So far, I programmed in a smooth-motion and the ability to track anything I select in a box. In the future I am going to make an android/ipad app so that you can just multi-touch define a subject with 2 fingers.
After looking into the DJI D-bus, I'm fairly sure I can make the Zenmuse/Ronin gimbals follow my commands, in addition to commanding the flight paths of the quad itself to follow something (ie. a bike, or a car)

Is anyone out there also interested in this tech? It's easy enough for me to make and share, and any feedback on cool ideas to implement would be great
Cheers

 

[ProfessorStein]

That sounds ultra smart! We've seen drones that autonomously track objects by locking onto a supplemental transceiver (like the Air Dog)... but if you're tracking purely in the video stream, that's a huge step!

 

[noiseboy72]

Brilliant! A real market for this I would say. Are you suggesting this would be an add on module for the quadcopter, with WiFi link back to a smart phone to define targets?

 

[xtracrispy]

Brilliant! A real market for this I would say. Are you suggesting this would be an add on module for the quadcopter, with WiFi link back to a smart phone to define targets?

Yes, what I'm currently thinking is a small block that lifts off with your quadrotor so you get high-speed control without lag.
A little wifi transmitter can stream a video or picture feed, so that you can tell the system what to look at.

After that, just fly around in circles and get perfect video!

 

[noiseboy72]

Get it built! I am first in the queue, cash in hand

 

[DKDarkness]

Hello,

So I've been playing around with drone videography lately, and quickly realized that it's really tough to aim the camera while flying a quad in anything but a straight line.
I envisioned this one sweeping pan shot of an object in a field, but just couldn't do it with the quadrotor rolling around due to my flight control.

Anyway, I'm a roboticist and hacked together a working prototype that uses the video stream to track anything I want, using computer vision algorithms.

Here's a quick youtube video to show what I mean:
https://www.youtube.com/watch?v=QJW9ReKU-0I&list=UUy6SYEB_Gk40j9NOy8RI5aQ

So far, I programmed in a smooth-motion and the ability to track anything I select in a box. In the future I am going to make an android/ipad app so that you can just multi-touch define a subject with 2 fingers.
After looking into the DJI D-bus, I'm fairly sure I can make the Zenmuse/Ronin gimbals follow my commands, in addition to commanding the flight paths of the quad itself to follow something (ie. a bike, or a car)

Is anyone out there also interested in this tech? It's easy enough for me to make and share, and any feedback on cool ideas to implement would be great
Cheers

Looks really promising! Keep up the good work

 

[ramirez]

Looks great.

 

[Buckaye]

wow! That's awesome! Great work!

Now... I have to ask... you know not to make robots stronger than us right?

 

[Peter Patricelli]

Where you are going with this is the inevitable future of drone videography. Someone has got to be the first, and stay out front on this development, might as well be you.

But if I might suggest.....

"it's really tough to aim the camera while flying a quad in anything but a straight line.
I envisioned this one sweeping pan shot of an object in a field, but just couldn't do it with the quadrotor rolling around due to my flight control"

You are correct....for normal flying modes. The ORIENTATION of the bird determines where the camera is pointing and for the pilot keeping track of that orientation...determined by the required camera direction....and making the required mental adjustments and switches to keep the bird flying in the desired direction INDEPENDENT of the nose orientation....is VERY difficult.

Which is EXACTLY why you should look at IOC, specifically Home Lock or Course Lock flying...while doing videography. Suddenly the right hand stick consistently and directly controls the birds 2D movement, direction and speed, completely independent of bird....and camera.... orientation and filming requirement. At the same time the left hand stick is free to direct the camera....and bird orientation...completely independent of the birds flying requirements.

Seems to me that what IOC flying achieves for videography solves a LOT of the problems you will face. Without it you would need an independent 360 degree 3D gimbal.......and a whole new generation/design of drones.

Realizing what IOC could do, yesterday I EASILY pulled off a full 360 degree sweeping pan of myself standing in the center (Home Lock) position, something that would have been difficult, and ugly (for me anyway) in regular flying mode. I bet in IOC you could pull off the pan you envisioned in your opening statement.

Using IOC, you would only have to link your tracking algorithm to the left hand stick YAW control...and you have GOT IT!

Peter Patricelli

 

[xtracrispy]

Hi Peter,

Thanks for testing that and providing the very good suggestion! Definitely a good "keep it simple" approach to solving the problem.
Can you think of a way to get a double-dynamic shot with our current technology also?

For example, your signature's website has some windsurfing shoots where people are moving over the water and doing jumps. If the subject is moving also, I wonder if there's a way to track it other than via our ground-based eyeballs.

 

[Happyflyer]

wow! That's awesome! Great work!

Now... I have to ask... you know not to make robots stronger than us right?

You got that right.
.
Great! Keep working.

 

[The Editor]

For example, your signature's website has some windsurfing shoots where people are moving over the water and doing jumps. If the subject is moving also, I wonder if there's a way to track it other than via our ground-based eyeballs.

FPV and a litttle practice or move to a two man rig and a 360 degree gimbal - simples.

 

[Peter Patricelli]

The programming technology of lock-on auto-focus exists, since my D800 camera in video mode will lock on thefocus of a 600mm lens, very critical, to a moving windsurder against a background of moving waves. From there to centering the locked-on subject ought to be relatively simple, programming-wise.

I did a test flight, actually filming, of panning independently while flying in HC/Home lock, this morning. My FPV monitor is currently broken, awaiting a replacement being shipped, so this was strictly visual flying. Other than a need to slow down the panning speed, a perpetual problem, the results were very good for a first time, out of the box test. Without HC/Home Lock I could not have come close to the same results.

Try it!!

Peter Patricelli

 

[MadMitch88]

God **** you, bro --- get this thing perfected for the Phantom Vision+ and upload it to the Google Play store and then take my money !!! :mrgreen:

But seriously --- I'm glad so many smart fellas like yourself are getting interested in the UAV hobby because great forward progress like motion tracking video capture will soon be a reality. I can only imagine how many cool uses this will have for everyone who flies and wants to get great shots of a particular object during flight. I know some big game hunters who would love to have an easy way to keep animals in the center of the frame while the UAV buzzes around. Your invention would do that.

Keep up the good work and I hope to see your hard work pay off with cold, hard cash

 

[xtracrispy]

Thanks for the compliments guys! It's really motivating me to try new things now :lol:

Keep up the good work and I hope to see your hard work pay off with cold, hard cash

I wonder if that could be a possibility one day. What would you guys think this to be worth, given that the electronics will probably cost a few hundred for parts?

 

[fastsmiles]

This is awesome technology, keep us posted on progress. There is a great demand for this ability. What kind of price do you project?

 

[yusiang]

Thanks for the compliments guys! It's really motivating me to try new things now :lol:

Keep up the good work and I hope to see your hard work pay off with cold, hard cash

I wonder if that could be a possibility one day. What would you guys think this to be worth, given that the electronics will probably cost a few hundred for parts?

I think a similar setup(non-DJI) has done it with some off-the-shelf hardware under $100...
http://diydrones.com/profiles/blogs/red-balloon-finder

 

[xtracrispy]

This is awesome technology, keep us posted on progress. There is a great demand for this ability. What kind of price do you project?

I think that the sum total of a decent camera, powerful with gpu embedded computer, and wireless transmitter will be a few hundred dollars.
Do you guys think a price somewhere between 500-1000 is reasonable, given the parts involved? That would place it about equivalent to the cost of a non-active gimbal.

I think a similar setup(non-DJI) has done it with some off-the-shelf hardware under $100...

Cool! I looked into this as well. So it appears to do pure color tracking, which is computationally easier.

One problem with this however, is that lighting conditions can change, the color signature of an object can get lost.
Last but not least, it'd require the object to have a set color, which can lead to problems if the background also has the same color. (Imagine trying to track a green tshirt in a park!)

This in part can be solved by "color coding", which tracks a sequence of colors to reject outliers.
Here's another experiment I did with a system that costs $60 combined!
https://www.youtube.com/watch?v=8SbWyUAWq9M&list=UUy6SYEB_Gk40j9NOy8RI5aQ

 

[Yaw Master]

Plus one. This is precisely what we need. Personally, I had expectations that DJI would have inconcluded this in their automated waypoint upgrade. Go for it, please.

 

[DBeard]

What camera are you using in that example youtube?