Coral Reef Monitoring with Drones: Mapping and AI Detection

[00:00:00]uh The Nature Conservancy is a non-profit organization that works around the world on habitats and uh we have been using drones for probably six years now six or seven years um actually since May 2014 so I guess almost 10 years now and we're really interested in in leveraging this technology to and being able to build capacity with the partners that we work with our goal is to help countries protect their habitats and Preserve life upon preserve these habitats upon which all life depends and we've been able to leverage this technology specifically in our choral monitoring work we spend a lot of time growing Corals in nurseries and out planting those coral and we're interested in identifying how that restoration work is succeeding and so I'm going to walk you through uh kind of the history of how we've been using drones over the years and specifically show you some of the results from our deep learning algorithms where we feed this imagery into a deep learning environment and try to detect uh the occurrence of these these Coral colonies so you know back in May 2014 we started with this drone which which we built uh it was built on a budget of two thousand dollars and we used a GoPro and had a custom 3D model frame with a uh all those Electronics inside of a a tupperware that you see there and our goal is to as we survey we wanted to get into areas where we couldn't access with the boat that were that were shallow and we were creating a lot of benthic habitat maps from both satellite imagery and uh and drone imagery and this this was kind of our first prototype and then we moved over the next year into 3dr solo platform where we we used modified GoPros to be able to capture both RGB and infrared uh so we were we were trying to find the latest and greatest platforms to do this work we did move over to a Sony qx1 camera which provided us better imagery acquiring 20 meter megapixel Stills and and then we we moved over to DJI products and we're currently using a mavic and a three multi-spectral and we also use a wingtra uh Gen 2 drone uh which has a Sony RX1 R2 camera which really uh has been incredibly helpful I know David you know he he talked a little bit about fixed swing drones in the beginning and we found that these are really helping us cover larger areas we can fly higher and and this drone is able to map uh up to 50 to 60 Minutes depending on weather conditions and with that 42 megapixel camera we can we could fly higher and inquire better resolution imagery for the coral reef monitoring work that we do we fly at about 65 meters and that gives us about a one centimeter pixel and we've done uh this throughout many different areas of the world uh here's the team that I worked with in April collecting imagery in Maria an island in Tahiti we had two drones uh mapping a third of that island and I'll show you some of the results we originally looked at world view 2 imagery to be able to map these these Coral colonies the reason that we were mapping this area is because there's a group called Coral gardeners that works to restore some of the coral that's been lost in Tahiti and around the world and they were really interested in the question of can we map live Coral and that's a very difficult thing to do we especially with with satellites it requires you know hyperspectral imagery and a lot of field data and we wanted to ask the question well if we get the re the spatial resolution fine enough can we identify those individual live Coral colonies and so here's here's an example of what the imagery from the worldview 2 satellite looks like and we tried to classify that but in our initial attempts we realized that uh it wasn't possible with the limited uh bands and so I want to show you some of this this drone imagery that was acquired with a winkra uh flying at 65 meters with a one centimeter spatial resolution you can see here the the coral nurseries that are where they have these tables laid out where they're growing coral and then they take that coral and outplant it onto the reef and you can see as we zoom in the spatial resolution where the color indeed represents live Coral and so we wanted to take advantage of this high spatial resolution High contextual data set to see if we could identify individual Coral species and so we worked with the team there the experts in the identifying the different species and and these are the ones that we could identify from the imagery and so we built a machine learning model around this trying to identify if these individual Coral colonies and so this is the imagery we segmented the imagery which groups similar pixel reflectance values and then we we would classify those and we're in the process of finishing this work up and it's trying to assign these individual classes to Coral but the color really drives the identification of these these corals and also the the texture that's found within each of the images I want to show an example also in the U.S Virgin Islands this is in Saint Croix an area around Buck Island and also a linear Reef called lose Reef that we worked on WE collected a lot of field data we used underwater camera that was attached to a uh uh and rtk gnss and then we PPK corrected that uh and and got our accuracy down to about two to three centimeters that every photo and we needed that accuracy to be able to really identify you could see some of the underwater photos that were that were taken so when we flew the Drone imagery we collected the RGB data uh very similar to what David was talking about we ran a python script to extract bathymetry and we used both of those uh the the RGB data sets in the bathymetry to feed that into the machine learning um and I want to show you some examples here so here's a coral calling in it's Cropper palmata or the common name is Elkhorn coral and this is a very important shallow Reef building Coral that experienced a lot of die off and it's actually an endangered species and so we're we're trying to regrow colonies of of these and and establish the the population and look at genetics we know we hear a lot about bleaching and disease and so we're really interested in in growing genotypes that seem to be more resilient to some of the the threats that corals are facing and so we want to monitor their growth and and collect uh uh fragments from this species and and we micro frag those we we cut them with a bandsaw which stimulates the growth we put those in nurseries and then that that those those fragments grow at an accelerated speed and so here we could see what that what that Colony that exact same Colony looks like from above and you know if we take different data sets this is looking at from using a phantom 4 at a low altitude this is from a mavic 2 uh at a higher uh resolution this is the data set from the wingtra flying at 75 meters we want to understand different environmental conditions this was partly cloudy in the afternoon this is clear in the morning so we were collecting a lot of different data sets at different times of day in different conditions to see how that Colony changed over time and you can see if we raise that flying head up to 90 meters what it looks like and then going to 120 meters uh during cloudy conditions so we evaluated what these corals were look what they were what they appeared like at different conditions at different times of day at different altitudes to collect a library that we could train the machine Learning Network on and so here you can see some training data that was fed into the the Deep learning algorithm we looked at different algorithms and and tried to assess their accuracy and we're working on a paper right now to report out the accuracy I'm working with Dr George raber at the University of Southern Mississippi and he's leading the Deep learning work but we're we're finding very promising results and being able to identify these uh individual Coral colonies and be able to quantify live coral and this is really exciting for us because we're doing a lot of out planning of coral and we want to measure our success and some of the initial findings show the density of this crop or a palmatus species across that area and lose Reef in Saint Croix we have some preliminary models we flew it just three months ago and and we're comparing that data from the the data set we collected two years ago to look at growth now we know that Coral grows very slowly but uh Cropper palmata this particular species is one of the faster growing species and so we're having success identifying those areas that have increased in size we're looking at different things like is there a difference between using a JPEG compression versus a raw and you can kind of see the differences going from JPEG to Raw jpeg is a much smaller file size but the raw gives us a little more detail we're looking at different post-processing algorithms that we use for example drone deploy or meta shape and how those orthomosaics appear and how they function in these deep learning algorithms we're doing a lot of work uh mapping areas where these governments are experiencing uh beach erosion and a lot of that beach erosion is due to an erosion of the coral the coral is important because it's it's protecting that Shoreline it's attenuating the wave energy uh that comes in and so many governments are very interested in restoring their coral and this also this involves Coral nurseries as well as hybrid Solutions where we work with Shoreline Engineers to to develop artificial reefs and then there's also hybrid Solutions and this information is really important we can take drone imagery and we can classify where the live Coral is where the rubble and hard bottom the sand the seagrass and the turf algal substrates and and so we see this as a promising area in particular for local scale studies uh here's some work that was published uh that I was involved with in Belize where we were looking at success of out planning efforts in an area called The Laughing bird key National Park and we demonstrated that we could we could identify a 17 percent increase in a crop or a server cornice over time which is the the species that and for this particular project they're really interested in so real quick uh some of the Lessons Learned we experimented with different flying Heights and we identified that flying around 65 meters was sufficient at a one centimeter pixel to map the individual Coral colonies now with that flying height uh when your Coral colonies are less than 50 centimeters on a side a half a meter are our accuracy goes down so once it gets above that threshold our accuracy exceeds 80 percent these smaller colonies are a lot more difficult to detect because they get confused with with other features the the sea fans and other species of coral that are similar that have a similar color and we also determined that we can detect about a five percent change in the coral once it reaches that 50 centimeter threshold and it's very useful to acquire higher resolution mosaics for monitoring these small areas because it really helps with the selection of the training data and also you know the field data is very very important so some of our research areas that we are exploring is the what are the ideal times and these are some of the things we're addressing in the paper that we're working on uh looking at cloud cover Sun angle water motion these data sets have to be acquired at low Sun angles typically early morning or late afternoon to avoid things like sun glint we're looking at different camera parameters under different Sky conditions looking at shutter speeds we want to avoid image blur and and we want to also capture that information at its full content and contextual information and so that that means experimenting with different ISO levels we want to look at different post-processing softwares what are the pros and cons of each and what produces the best output for deep learning how does depth or distance from Shore relate to Coral fragment growth over time is another question where we're trying to answer and and the flying lap and Mission parameters that we set how does that affect the ability to stitch these order Ortho mosaics and how can we use color correction to approve detection performance so those are those are some of the the questions and and ideas that we're researching you know we're we're talking about drones but I want to take you underwater and and show you some of the the work that we're doing with close proximity photogrammetry so here's an example of a camera system that's set up that's that's on a boogie board where we're collecting stereo photo we have a snorkeler that's pushing this along and a lawnmower pattern and we're collecting stereo data just like a drone and we can also do this uh via scuba and we can get two meters from The Reef uh and we can collect stereo images uh very very detailed stereo images of the reef and this captures information that we can't answer from the the Drone imagery because we know from drone imagery our ability to detect features that exceed about two to three meters is is difficult because the refraction of the water and sometimes the the turbidity or vision the visibility of the water column so we collect these data sets much like we do with a drone uh we're we're doing this in typically 10 by 10 meter monitoring plots we put down scale bars and and we we put in markers permanent markers into the coral that that allow us to be able to align these data sets over time and do very detailed change detection we also do a lot of pre-processing color correction because we know that the different colors of wavelengths are absorbed the deeper you go and so we try to introduce there's there's methods to introduce that those Reds and those greens back into the the image data sets which helps uh with the uh with the identification of these species here's an example of some out plants on a coral this is in Bahamas where where we we have server cornice a proper server cornice growing or this Staghorn Coral you can see and this information really helps us identify a lot of indicators that traditionally scuba divers have collected but we can do this with much detail at a finer resolution we can do it for these monitoring plots and and calculate things like spatial ecology the structural complexity the demographics of coral colonies over time very very very powerful uh way to monitor these coral reefs we can bring it into software such as tag lab which is an open source software to be able to label and identify live Coral the presence of disease presence of predation and track these over time and see how the these reefs are changing so this is something that we're doing with the Perry Marine Institute in areas across the Bahamas and some of the challenge are the you know getting accurate enough uh geo-referencing information to be able to align those pixels over time dealing with data gaps and having sufficient overlap analyzing 2D versus 3D models and trying to get that side information of the coral when you've got higher regosity looking at water quality issues back scatter sunlight refraction is all issues that we have to deal with in the water column the shear data volume processing power and then and then the capacity of being able to extract the information that we need and build that skill set with the partners that we're working with that's that's another challenge that we have so uh with that I'm going to go ahead and end