Can machine learning and operations research inspire each other?

Will deep learning really be able to do everything?

Opinions differ about the true potential of deep learning. Geoffrey Hinton, recognized for pioneering deep learning, is not entirely unbiased, but others, including Hinton’s deep learning collaborator Yoshua Bengio, are attempting to infuse deep learning with elements of an area that’s still under the radar: operations research or an analytical problem-solving and decision-making method used in the management of organizations.

Machine learning and its deep learning variant are now practically a household name. There is a lot of hype surrounding deep learning, as well as a growing number of applications using it. However, their limitations are also being better understood. This is probably why Bengio turned to operations research.

In 2020, Bengio and his collaborators examined recent attempts from both the machine learning community and operational research to use machine learning to solve combinatorial optimization problems. They advocate further advancing the integration of machine learning and combinatorial optimization and describe a methodology.

However, to date there has been no publicly visible operations research renaissance, and commercial applications are few compared to machine learning.

Nikolaj van Omme and Funartech want to change that.

Operations Research uses domain knowledge for optimization

While operations research (OR) typically originated during World War II, its mathematical roots can go even further back to the 19th century.

In OR, problems are broken down into basic components and then solved in defined steps through mathematical analysis. Van Omme describes himself as a mathematician and computer scientist. After his postgraduate studies, he noticed the similarity and complementarity between machine learning and OP. After not getting the attention he was looking for to continue exploring this potential synergy, he founded Funartech in 2017 to make it happen himself.

For van Omme, there were several reasons why combining machine learning and OR seemed like a good idea. First, machine learning is data hungry, and in the real world there are cases where there isn’t enough data.

It’s also a matter of philosophy: “If you’re just using data, you hope that your algorithms will pull some patterns out of the data,” said van Omme. “They hope to glean some limitations and insights from the data. But actually you’re not sure if you’ll be able to.”

In OR, he added, knowledge can be modeled. “You can talk to the engineers and they can tell you what they’re doing, what they’re thinking and how they’re doing it,” he explained. “You can turn that into mathematical equations so you have that knowledge and you can use it. If you combine both data and domain knowledge, you can go even further.”

The OP is all about optimization and using it can yield 20% to 40% optimized results, according to van Omme. Like Bengio, he was referring to the Traveling Salesman Problem (TSP) – a reference problem in computer science. In TSP, the goal is to find the optimal route to visit all cities in a traveling salesman’s assigned district once.

According to van Omme, if you approach the TSP with OR, you can create exact solutions for 100,000 cities. On the other hand, using machine learning, the best you can do for an exact solution is to solve the same problem with 100 cities. This is an order of magnitude of difference, so the question becomes: Why isn’t OR used more often?

For van Omme, the answer is multifaceted: “Machine learning was considered a subset of OR a few years ago, so I wouldn’t say OR isn’t being applied, although today people tend to put machine learning on one side and OR on the other others,” he said. “There are some areas where OP is really heavily used – transport for example or manufacturing.”

However, machine learning has been so successful in some areas that it has eclipsed all other approaches, he explained.

3 ways to combine operations research and machine learning

1. Van Omme isn’t out to bash machine learning. What he advocates is an approach that combines machine learning and OR to have the best of both worlds. Typically, van Omme said, you first use machine learning to get some estimates, and then use those estimates as input to your OR algorithm for optimization.
2. Machine learning and OR can also be used together to help each other. Machine learning can be used to improve OR algorithms and OR can be used to improve machine learning algorithms. OR is mostly rules-based, and when the rules apply, that’s hard to beat, van Omme noted.
3. Construct new algorithms. Once you have a basic understanding of the strengths and weaknesses of machine learning and OR, there are ways to combine the two so that the weaknesses of one are offset by the strengths of the other. Van Omme mentioned graph neural networks as an example of this approach.

Disadvantages

OR is not without problems and van Omme recognizes that. The problem, he says, is that “most of the time the rules don’t apply. You don’t know exactly how to use them. And there is a certain probability that if you go one way or the other, you will get completely different results.”

This is aptly illustrated in one of Funartech’s most well-known use cases: its collaboration with Aisin Group, a major Japanese supplier of automotive parts and systems and a Fortune Global 500 company. Aisin wanted to streamline the movement of parts between depots and warehouses.

This cannot be addressed in a ‘traditional’ way with a model that can solve the entire problem because it is a very complex problem on a large scale, van Omme noted. After four months of work on it, Funartech was able to achieve a 53% optimization. However, it turned out that they didn’t have the right data for some parts of the problem.

So when Funartech tried to figure out whether or not their solution made sense, they quickly realized that some estimates were actually not very good for the data they didn’t have. When the right data was provided, the optimization dropped to 30%.

“The thing is, our algorithms are so tailored to the instance that they stopped working when they gave us the right data,” he said. “They couldn’t produce anything. So we had to go back and simplify our approach a bit. And since the project was over, we didn’t want to invest that much time.”

Scaling operations research

Van Omme also explained that Funartech spends a lot of time with customers to develop a tailor-made approach to each problem. That seems to be both a blessing and a curse. At this point, although van Omme mentioned that Funartech is working on developing a platform, it’s hard to imagine how this service-oriented approach could scale.

Part of what has made the machine learning approach so successful is the fact that there are algorithms and platforms that people can use without having to build everything from scratch. On the other hand, van Omme pointed out that Funartech has a 100% success rate, while 85% of machine learning and 87% of data science projects fail.

But there’s another, perhaps unexpected, obstacle that surgical practitioners face, according to van Omme: learning to get along. The “no Ph.D. required for this to work” was a key part of machine learning’s push into the mainstream. In OR things are not there yet.

The fact that OR practitioners are highly skilled also means, according to van Omme, that they tend to be very opinionated. Social skills, such as learning to listen and compromise, are therefore essential.

All in all, OR – and the many possible combinations with machine learning – seems like a double-edged sword. It has the potential to produce highly optimized results, but currently also looks brittle, resource and skill intensive, and difficult to apply.

On the other hand, you could probably say the same thing about machine learning a few years ago. Perhaps the cross-fertilization of the two disciplines with techniques and insights gained could help lift both.