David Knox - Principal, Data Science
Lynker Analytics have been in the office for just over a week now and today marks the official launch of our website and presence on social media. This seems like a good time for my first blog post.
I’m David Knox, the Principal Data Scientist with Lynker Analytics and I’m very excited to introduce one of our first projects: building tools to automate the understanding of underwater video to help us survey and understand the marine environment.
Before I describe our project, we need some background. The Marine, Ocean and Coastal Science and Information Group at Lynker (our partner) conducts environmental monitoring around Monterey Bay on behalf of NOAA – to understand anthropogenic impacts on the underwater environment. That monitoring includes the collection of hours and hours of underwater video.
Presently, human experts analyse and annotate the underwater video. Wouldn’t it be great if we had tools to automate object detection and location within the video, thus freeing up our human experts for other tasks or giving them a head start on finding the most interesting sections of video for further analysis?
We are analysing video from many sources such as diver held cameras, towed camera sleds and Remotely Operated underwater Vehicles (ROVs). Some of this video is annotated and some isn’t.
We are teaching our neural networks to identify and locate objects in the underwater video from identifying and locating human structures, such as pipeline infrastructure or old gumboots to identifying natural objects and marine life.
Underwater video can be hard to work with, with low light, sediment and bubbles occluding and obscuring the scene. Also, our many sources of video come in a variety of resolutions, image quality and from a variety of locations and depths.
As an early part of this project, we’re building a pre-processing step to ingest video and clarify the image using multiple frame smoothing and image filtering techniques. In a future post, I will elaborate on how we are optimizing our pre-processing for image classification and object localisation.
Here’s an example showing an image from a video surveying an oil pipeline. The left image shows the raw frame and the right shows the results of our early pre-processing.
Photo Credit: Lynker LLC
Data pre-processing is an essential part of this project but we are producing more than prettier videos.
Next, we move on to image classification using deep convolutional neural networks. Our first task is to label our training data. We tackle label propagation and multiple cycles of human QA and correction in a cycle of model improvement.
Informed by our image classification models we then build further models to locate objects within images and segment the moving video. I will explore each of these steps in future blog posts.
The output of these steps is a video processing pipeline that can provide information for underwater surveys of human structures and marine life and ultimately help our partners and customers in their work: understanding the marine environment and the human impacts on it.
I’m super excited about this project (and others we have on the go!) and look forward to sharing our progress in future blog posts.