• The goal is to systematically investigate the importance of the vision aspect (e.g., screenshots) for the "Large Action Model" (LAM). The key question is whether a purely text-based approach (e.g., prompt + UI elements from the DOM) is sufficient or if incorporating screenshots provides significant advantages.
• This research will explore different methods of incorporating visual information and evaluate their contribution to model performance. Ablation studies should be conducted to assess the impact of vision on fine-tuning for next-action prediction.
• The findings of this thesis can significantly influence the development of models for automated UI interactions and control agents. A better understanding of the role of vision input can contribute to the creation of more efficient models, either based purely on text or leveraging visual information to improve prediction accuracy and UI understanding.

• You are a student (f/m/d) at a university or a university of applied sciences. We’re looking for someone who takes initiative, perseveres, and stays curious. You like to work on meaningful innovative projects and are energized by lifelong learning.

Desired skills / experience to be successful in this role:

• Knowledge of machine learning and neural networks (must-have)
• Experience with LLMs and vision-based models (must-have)
• Experience in fine-tuning models (e.g., PyTorch, Hugging Face) (must-have)
• Proficiency in Python programming (must-have)
• Experience with processing UI data (DOM structures, bounding boxes, screenshots)
• Experience with cloud platforms, especially Azure
• Master student in Computer Science / Business Informatics
• Language requirements: English fluent