Positions: Master's Candidate
Created: 2023-10-22 Deadline:
Location: Warsaw University of Technology, Poland
Transformers are the foundation for many well performing neural language processing models. Unfortunately, they require a lot of computational resources which results in slow inference. In this project we aim to leverage conditional computation methods to speed up inference along three axes: depth-wise sparsity (early exits), width-wise sparsity (mixture of experts) and input-wise sparsity (dynamic sequence pruning). Additionally, we would like examine the hypothesis that some data points are easier to process for neural networks. For that purpose, among others, we would like to implement dynamic variant of mixture of experts (MoE) that enables MoE layers to use less resources for easy data points and compare it with difficulty rating extracted from early exit models.
Contact: Tomasz Trzcinski (filip.szatkowski [ at ] pw.edu.pl)
Computer Vision Lab is a research group at Warsaw University of Technology that gathers faculty and students working together on topics at the intersection of computer vision, machine learning and perception.
See lab's page