NLP
ULSE
A platform for unbiased, low-code, sustainable and expedient research in natural language processing
Short
Description
The project focuses on a technical solution to two major challenges of empirical NLP research: the low reproducibility and comparability of quantitative experiment results. In the relevant literature, this is also referred to as a “reproduction crisis”. Experiment results are typically produced when comparing newly developed NLP techniques with existing techniques or in pure comparative studies and thus represent both the performance of the techniques and a classification in the current state of research. However, at least six core issues are encountered when preparing and conducting the corresponding experiments, including different experimental setups in the underlying scientific publications, insufficient resources, considerable manual effort, incomprehensible deviations in the experimental results, high computational requirements and often also a high level of very specific expertise.
A GlObal
ISSUE
We asked 804 people from all around the world about their experiences working with natural language processing. Most of them saw great potential in NLP, but also great challenges that have to be solved in the coming years.
Find out more about the respondents of our survey, learn about the fields NLP is used in, how old pracitioners are and what they use NLP for.
View the results of our survey and learn about the needs of the NLP community.
Learn how NLPulse will help solving these needs and how its collaborative features will improve how people work on new and existing NLP methods.
Community
Voices
”
Looking forward for having help as non-research time at work is just too long and I simply have no time for testing ideas that come to my mind almost daily.
”
I am very glad to hear you want to create such a platform, with it many researchers could do their work better.
”
Evaluation is an extremely crucial topic for any decent work in NLP, but, at the same time, is an extremely costly and time consuming activity when done properly. Collaborative efforts are thus utterly crucial.
Main
Goals
LOWER EFFORT & EXPERTISE
NLPulse aims to enable the evaluation and use of NLP techniques with lower manual effort and less specific expertise than conventional work with author-supplied source code, specially created datasets, and potentially laborious or incomplete setup instructions. The decentralized architecture of NLPulse should further enable the distribution of computational effort across different infrastructures so that users with limited computational resources can contribute effectively to the community.
ESTABLISHING TRUSTWORTHINESS
NLPulse should make it possible to create the results of experiments in a reproducible, comparable, and reliably citable manner and to guarantee their resilience through the resulting traceability. This should reduce the need to repeat experiments and promote collaboration between NLP researchers.
RESEARCH PROCESS SUPPORT
NLPulse should work in a process-oriented manner and specifically support the typical processes of developing and testing new techniques, the realization of comparative studies and also the implementation of peer reviews within the NLP research community. In this context, the establishment of cooperation with research groups, conferences (e.g. EMNLP, ACL) and/or journals is also an objective of the project.