Processing Images from the DJI Matrice 100 and Zenmuse X3 (without the Manifold)

A few months ago, we had to test some tracking algorithms on the Matrice and came to the realization that it was not possible to directly access images from the X3 without the Manifold. While it was possible to acquire images from the stereo cameras of the guidance over usb (see tutorial), no such option existed officially for the X3 – if you did not buy the Manifold.

We had a few days to pull this off, and unfortunately, the Manifold was no longer listed on the DJI store (As of writing, it has been restored). After searching for alternative methods of accessing video streams directly from the camera, I came across a few interesting posts – something to consider if you don’t have access to the Manifold or if the method described later in this post is inadequate.

  • This thread from RC groups has a lot of information on the control and acquisition of video from the camera including the X3’s pinouts.
  • This post covered a similar project with the DJI Osmo. The main difference is the fact that the OSMO has wifi, which makes access to the video stream a bit easier.
  • These two commercial projects provide third party control of the cameras.
  • Another attempt at video acquisition was made here, on the Phantom 3, which also had wifi.

In the end, we came up with a two options for accessing the video, which did not involve low level access.

  • Writing an app based on the Mobile SDK sample and either running the tracker on the phone/tablet, or steaming it over to a desktop via RTMP for processing. We noticed the DJI Go app had a custom streaming option, so we considered that instead.
  • Connecting the remote controller’s HDMI output, to a USB converter, and accessing it like a webcam.

We tested both with good results. The second however seemed more promising and had lower latency.

Streaming from DJI Go

First, you would need to setup an RTMP Server to receive the stream from the app. You could setup an Nginx server or simply download and install MonaServer.

With MonaServer, you would have to check if the www folder has a subfolder in it. This folder is by default called “live”.

After running MonaServer, you should see the ports on which the server is running. In this case, 1935.monaserver

Ensure your mobile device and the server are on the same wifi network.

Enter the IP address of the server, followed by the port and the subfolder under the www folder, in the form

rtmp://ipaddress:port/subfolder

For example,

rtmp://192.168.24.3:1935/live

The IP of the server could be found via typing ipconfig in command prompt on windows or ifconfig in terminal on linux.

livestream
Settings menu > Choose livestreaming Platform > Custom

 

You can confirm the stream works with VLC media player.

Open VLC > Media > Open network stream

Type the rtmp url listed above as the network url and click play.

Once that works without issue, you could consider accessing it with OpenCV. Your code will have to be robust against malformed frames especially when the network quality drops. During tests on the university’s wifi, the frames per second ranged from 0.5 to values exceeding 50 fps.

You may want to confirm access to the stream using a try/while/if statement in your code before beginning processing.

Edit the video capture line of your OpenCV code to include,
C++

cv::VideoCapture cap('rtmp://ipaddress:port/subfolder'); 
if(!cap.isOpened())
{
    std::cout << "Could not open RTMP stream" <<std::endl;
    return -1;
}

Python

cap = cv2.VideoCapture('rtmp://ipaddress:port/subfolder')
if not cap.isOpened():
    print "Could not open RTMP stream"

 

Using the remote controller’s HDMI output

Note: This approach was not completely tested, so no product recommendations can be made. Explanations at the end.

The DJI remote controller for the Matrice has an mini HDMI output connection. Other consumer DJI drones may not have that and need to purchase an add-on. This HDMI output connection can be connected to an HDMI to USB capture card. I found this online. It was the smallest one with linux support, not necessarily the cheapest.

You could search amazon for similar offerings. You would also need to purchase an mini-HDMI cable to connect from the Remote controller to your capture card.

We tested the HDMI connection from the remote controller and not much else. We tried the Razer Ripsaw, which is windows only and tested it with Open Broadcaster Studio (OBS). You’re probably wondering why – and in hindsight, I also wonder why I did not test it with OpenCV to clear that up. In OBS, it appeared under cameras, so it is quite likely it would work with OpenCV; we just did not test it.

In using this approach, you would need to go to the remote controller’s settings on DJI Go and turn off the On Screen Display (OSD) on the HDMI output to ensure you only have raw images with no text overlay

Screenshot_20170705-214928 (2)
Menu > HD > Display OSD on HDMI output

This method is more robust and recommended. If all the options stated are inadequate, get a Manifold and run this example.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s