Deep learning: new neural nets could model continuous processes

An image of deep learning, AI, Deep learning: new neural nets could model continuous processes

In deep learning, neural nets use specific hidden layers to deliver defined results. AI researcher David Duvenaud is questioning all of that with ODE nets.

Deep learning is incredible: truly, it is. Being able to map human-like brain power onto a computer, so that it learns as we do, should never be taken for granted. It is one of the most astonishing scientific breakthroughs in the history of our species, however, deep learning is not beyond improvement.

At the heart of a deep learning model lies a neural net. This is the brain, if you like: a combination of stacked layers of simple nodes that work to try and find the patterns in data. The net then assigns values to data that it processes, filtering this data through different layers to come to a final conclusion.

Now, scientists are questioning how the values are assigned to data and whether there’s a more efficient way to run deep learning algorithms.

David Duvenaud, an AI researcher at the University of Toronto, set out to build a medical deep learning model that would predict a patient’s health over a period of time. Traditional neural networks thrive when they learn from data with defined observation stages: basically, the hidden layers within a deep learning model. This is difficult to align with healthcare.

Health is a continuous topic to assess. It does not rely on binary questions as it contains so many variables. So how can a neural net pick up on continuous data?

Can neural nets be improved?

Think of a deep learning model as being similar to a game of classic board game, Guess Who. In the game, each player has a selection of characters in front of them, all with a different appearance: some have facial hair, glasses, blue eyes, brown eyes and each of them unique.

One player of Guess Who asks the other binary questions to discount characters from their investigation, until they are left with the final chosen character through this process of elimination: this is the output layer.

This is similar to how a neural network works. It processes its data through different stages, eliminating more and more of the dataset until it’s left with the correct answers available. This is the technology that is used in face recognition software, for example.

Software 2.0: How neural networks work
Basic neural network model

David Duvenaud saw an opportunity. He sought to break from the binary for a more fluid form of deep learning.

Traditionally, the answer is to simply add more layers to a neural net to reach a more accurate endpoint. This is not always sensible though. Why, for example, should you have to define the number of layers within a neural network, train the data and then wait to see how accurate it is? Duvenaud’s neural net lets you specify the accuracy first, then it finds the most efficient way to train itself within that margin of error.

This is what researchers describe as an “ODE net”, short for “ordinary differential equations”.

How can an ODE be solved?

Solving an ODE numerically can be done by integration. This is a computationally intensive task and there have been methods suggested in the past to reduce the hidden stages within deep learning.

Duvenaud worked with a number of researchers on a paper that proposed a simpler method to solve an ODE. The method relies on solving a second, augmented ODE backwards and doesn’t take up too much memory. The gradient computation algorithm works by introducing an “ODEsolve” operation as an operator later on in the process.


The ODE poses interesting questions about what the most efficient methods of deep learning truly are.


This operator relies on the initial state, the function, the initial time, the end time and the searched parameters from the ODE. The presented paper provided Python code to easily compute the derivatives of the ODE solver.

The paper suggested that supervised learning – particularly MNIST written digit classification – was one application in which the ODESolve method can perform compared to a residual network with much fewer parameters.

Will ODEs revolutionise deep learning?

The ODE is not the only way to run a deep learning model. There could be any number of reasons that a scientist would want to define the number of stages for the AI that they run. Either way, “it’s not ready for prime time yet,” Duvenaud claims.

However, the ODE poses interesting questions for deep learning moving forward about how we build neural nets and what the most efficient methods of deep learning truly are. This is not a particularly new idea, but this is a breakthrough of kinds. Whether this approach works for a range of models remains to be seen.

An image of deep learning, AI, Deep learning: new neural nets could model continuous processes

Luke Conrad

Technology & Marketing Enthusiast

The critical role of data integrity in generative AI

Anjan Kundavaram • 23rd November 2023

The quest to harness the full potential of generative AI relies on finding trustworthy data to achieve outstanding results for diverse use cases. With the continued growth and transformative impact of generative AI, business leaders need to ensure that the data being fed into it has integrity.

Navigating a CTO-as-a-Service arrangement

Cyril Samovskiy • 21st November 2023

Attracting a top-tier Chief Technology Officer (CTO) can be challenging at the best of times, but for tech startups – who often have limited resources, a yet-to-be-proven product-market fit, and financial instability – it can be even more so. Add tech’s ongoing talent shortage to the mix, and it’s easy to see why CTO-aaS is...

The Importance of SBOM and CVE in Medical

Diego Buffa • 18th November 2023

This article explores the critical landscape of medical device cybersecurity, focusing on the IMDRF’s “Principles and Practices for Medical Device Cybersecurity.” It advocates for a holistic approach throughout the product life cycle, with particular emphasis on the vital role of the Software Bill of Materials (SBOM). The article addresses the FDA’s stringent postmarket vulnerability reporting...

AI powered fused spurs unveiled by measurable.energy

Diana Kamkina • 15th November 2023

measurable.energy, experts in eliminating wasted energy, are proud to announce the launch of their latest innovation – fused spurs. This highly anticipated addition to their product line is set to transform the landscape of energy management in construction and commercial buildings.

AI powered fused spurs unveiled by measurable.energy

Diana Kamkina • 15th November 2023

measurable.energy, experts in eliminating wasted energy, are proud to announce the launch of their latest innovation – fused spurs. This highly anticipated addition to their product line is set to transform the landscape of energy management in construction and commercial buildings.

Technology for a Sustainable Tomorrow

Mark Robison • 09th November 2023

We currently face the critical challenge of reducing carbon emissions in an effort to reach net zero targets. This is the challenge of our lifetime and for many more generations to come. Fortunately, this challenge has ushered in a new era of innovation, where technology plays a leading role in creating a sustainable future.

Preparing UK Businesses for the Coming PSTN Switch Off

Chris Wade • 01st November 2023

The PSTN Switch Off will require a robust framework of action as all business sectors will be impacted. In order to stay ahead of this significant change, businesses must start considering new, digital alternatives such as VoIP based communication technology.

Dark Fibre’s Role in Supercharging Edge Data Centers

Sean Lowry • 18th October 2023

In response to Proximity Data Centre’s e-book, Glide’s CTO, Sean Lowry explores the impact of low latency on gaming, the Metaverse, and AI. He explains how dark fibre and Glide’s “Fibre Cities” are primed to support the evolving needs of edge data centres and seamless connectivity.