You can find the competition rules and answers to commonly asked questions on this website.

News

Important Dates

The following dates use the anywhere on earth (AoE) time zone:

Event Date
Competition kickoff. The registration is opened and participants can download the starterkit and the training/validation datasets. July 1st, 2022
Submission available. The leaderboard and forum are opened, and the submissions are accepted. July 15th, 2022
Deadline for submission. October 15th, 2022
Winners notification. Winning teams are notified and instructed to provide information that will be included in the workshop report. October 31st, 2022
Report submission deadline. November 20th, 2022
NeurIPS competition workshop. December 2022

Introduction

The Natural Language for Optimization (NL4Opt) NeurIPS 2022 competition aims to improve the accessibility and usability of optimization solvers. The task of converting text description of a problem into proper formulations for optimization solvers can often be time-consuming and require an optimizations research expert. The participants will investigate methods of automating this process to be more efficient and accessible to non-experts. This competition aims to answer the following question: can learning-based natural language interfaces be leveraged to convert linear programming (LP) word problems into a format that solvers can understand?

This competition presents two challenges for ML: (1) detect problem entities from the problem description and (2) generate a precise meaning representation of the optimization formulation.

Challenges

The competition is split into two main tasks that are related, but tackled independently. Participants can compete in any subset of these two challenges and the 5 best winning submissions of each task will be awarded (see the Prizes Section below).

The two inter-related tasks are to find an intelligent solution to:

  1. detect problem entities from the problem statement,

  2. generate a precise meaning representation of the optimization formulation.

Sub-task 1 - named entity recognition

The goal of this task is to recognize the label the semantic entities that correspond to the components of the optimization problem. The solutions of the task will take as input an optimization description as a word problem that describes the decision variables, the objective, and a set of constraints. The multi-sentence word problem input exhibits a high level of ambiguity due to the variability of the linguistic patterns, problem domains, and problem structures. This first task aims to reduce the ambiguity by detecting and tagging the entities of the optimization problems such as the objective name, decision variable names, or the constraint limits. This is a preliminary step to simplify the second sub-task and can be seen as a preprocessing task.

Metric: F1 score

Relevant resources:

Sub-task 2 - generating the precise meaning representation

The goal of this task is to take as input the problem description, the labeled semantic entities, and the order mapping of variable mentions and formulate the precise meaning representation. This meaning representation will be converted into a format that solvers can understand. The solutions will be evaluated on the canonical form and conversion scripts from our pilot study has been released as part of the starter kit. We welcome you to create your own meaning representation or use the representation and conversion scripts provided in the starter kit.

Metric: Declaration-level mapping accuracy

Relevant resources:

For more information regarding the type of data in each sub-task and the resources provided to help you get started, visit the Getting Started page of our website.

Metrics

Sub-task 1 (named entity recognition): The solutions will be evaluated on their achieved micro-averaged F1 score:

\[\text{F1} ={2\times P \times R \over P+R},\]

where \(P\) and \(R\) are the average precision and average recall averaged over all entity types, respectively.

Sub-task 2 (generation): The solutions will be evaluated using an application-specific metric since the task is motivated by the need to precisely formulate the optimization problem. The models will be benchmarked based on the declaration-level mapping accuracy given by:

\[\text{Acc} = 1-\frac{\sum_{i=1}^N\text{FP}_{i} + \text{FN}_i}{\sum_{i=1}^{N}D_{i}},\]

where \(N\) is the number of LP problems in the test set. For each LP problem \(i\), \(D_{i}\) is the number of ground-truth declarations. The false positive \(\text{FP}_{i}\) is the number of non-matched predicted declarations whereas the false negative \(\text{FN}_{i}\) denotes the number of ground-truth declarations without a match.

Prizes

A total monetary prize of $22,000 USD will be awarded. The 5 best winning submissions of each task will be awarded the following prizes:

All participants will receive a certificate of participation. Winners will be invited to give talks at the workshop.