Internship | Investigate vision-language models for specialized fine-grained classification

Modern image classification has made tremendous progress thanks to big data. Deep learning models like CLIP, trained on over 400 million images, can now easily distinguish between different cat species with impressive accuracy. However, in specialized domains such as defense, safety, and security; the scarcity of available training data severely limits model performance. Few-shot learning techniques offer some improvements, but the fundamental challenge of limited data remains. However, substantial domain knowledge exists in alternative forms, such as detailed technical handbooks and expert documentation that describe distinguishing features of military objects. Current Vision-Language Models (VLMs) are constrained by the knowledge embedded during pre-training, leaving valuable external domain expertise untapped. This project will explore novel methods for incorporating external expert knowledge into VLMs to enhance fine-grained classification capabilities in data-scarce domains.

During this internship, you will investigate novel approaches to integrate domain-specific knowledge from textual descriptions into existing classification models and VLMs. You will start by understanding the current state-of-the-art in VLMs and exploring how to utilize external textual information for image-based classification models. You will then compare existing fine-grained classification models in data-scarce scenarios to determine if and where textual information can have a positive impact on these models.

Optionally, you can write a scientific paper on your findings. Your research could make a direct impact on object recognition systems for defense, safety, and security; potentially improving accuracy and reliability in critical identification tasks. The proposed solutions could enhance the capability of AI systems to leverage domain expertise without requiring extensive image datasets.

We are looking for a highly motivated student to join our cutting-edge research team working on computer vision. You should be passionate about AI, and bridging computer vision and natural language processing.

You are in the final stages of your master's degree in artificial intelligence, computer science, mathematics, (bio)medical engineering, or a related field. You have experience with computer programming (e.g. Python) and deep learning frameworks (e.g. PyTorch). Prior experience with computer vision and natural language processing concepts, and familiarity with transformer architectures and vision-language models is appreciated. The duration of the master project or internship is between 6 and 12 months.

Your supervision team will consist of two experienced TNO-colleagues who are actively involved in ongoing research projects at TNO. They have experience both in state-of-the-art image analysis research, as well as supervising BSc/MSc/PhD students in internships and graduate projects.

You want an internship opportunity on the precursor of your career; an internship gives you an opportunity to take a good look at your prospective future employer. TNO goes a step further. It’s not just looking that interests us; you and your knowledge are essential to our innovation. That’s why we attach a great deal of value to your personal and professional development. You will, of course, be properly supervised during your work placement and be given the scope for you to get the best out of yourself. Furthermore, we provide:

• A highly professional, innovative internship environment, within a team of top experts.
• A suitable internship allowance (615 euro for wo-, hbo- and mbo-students, for a full-time internship).
• Possibility of eight hours of free leave per internship month (for a full-time internship).
• A free membership of Jong TNO, where you can meet other TNO professionals and join several activities, such as sports activities, (work-related) courses or the yearly ski-trip.
• Use of a laptop.
• An allowance for travel expenses in case you don’t receive an OV-card.

At TNO, we innovate for a healthier, safer and more sustainable life. And for a strong economy. Since 1932, we have been making knowledge and technology available for the common good. We find each other in wonder and ingenuity. We are driven to push boundaries. There is all the space and support for your talent and ambition. You work with people who will challenge you: who inspire you and want to learn from you. Our state-of-the-art facilities are there to realize your vision. What you do at TNO matters: impact makes the difference. Because with every innovation you contribute to tomorrow’s world. Read more about TNO as an employer.

At TNO we encourage an inclusive work environment, where you can be yourself. Whatever your story and whatever unique qualities you bring to the table. It is by combining our unique strengths and perspectives that we are able to develop innovations that make a real difference in society. Want to know more? Read what steps we are taking in the area of diversity and inclusion.

After the first CV selection, the application process will be conducted by the concerning department. TNO will provide a suitable internship agreement. If you have any questions about this vacancy, you can contact the contact person mentioned below.

For this internship vacancy it is required that the AIVD issues a security clearance (VGB) after conducting a security screening. Take into account that this process may take about 8 weeks. If you have been abroad for more than 6 consecutive months, or if you do not have the Dutch nationality, it may take longer. Read more about security screening on the AIVD website.

Important to be aware of before applying:

• Before the start of the internship, the internship agreement from TNO needs to be signed. For students at a college or university based in the Netherlands, TNO uses the UNL-template (supplemented with a number of specific agreements from TNO). For students of foreign and MBO educational institutions, the TNO internship agreement applies. TNO does not sign any other internship agreements.
• Before the start of the internship, the educational institution will need to confirm in writing that:
  • 1) you are enrolled at the educational institution during the internship, and;
  • 2) the internship takes place as part of the programme of the study.
• The confirmation of educational institution takes place by signing the UNL template or forms prepared by TNO.
• Interns at TNO must be in possession of a Dutch residential address at the start of the internship. Performance of internship activities from abroad is not possible.