Location – Bengaluru, India
Join Databricks and Shape the Future of AI-Powered Search
Are you passionate about building large-scale AI systems that transform how users discover and interact with data? Databricks is looking for an experienced Staff Engineer – Applied AI Search & Discovery to lead the development of next-generation search technologies powered by Machine Learning (ML), Natural Language Processing (NLP), and Large Language Models (LLMs).
As part of the Applied AI team, you will work on innovative search and discovery solutions that help thousands of organizations efficiently find critical assets across the Databricks ecosystem. This is a unique opportunity to influence the future of AI-driven search experiences at one of the world’s leading Data and AI companies.
About the Role
The Applied AI team at Databricks is dedicated to advancing intelligent search and discovery capabilities across the platform. Databricks customers generate vast amounts of content, including data tables, notebooks, dashboards, SQL queries, machine learning models, pipelines, data rooms, and other digital assets. Many enterprise customers manage hundreds of millions of assets, making effective search a mission-critical capability.
As a Staff Engineer, you will play a key role in improving search quality, enhancing ranking algorithms, advancing query understanding, expanding asset coverage, and building scalable evaluation frameworks. Your expertise will directly impact how users discover information and accelerate productivity across the Databricks platform.
Key Responsibilities
Lead AI-Powered Search Innovation
- Design, develop, and deploy advanced machine learning-based search relevance systems.
- Build scalable search and discovery solutions integrated into Databricks products and services.
- Drive innovation in search ranking, retrieval systems, and semantic search technologies.
Develop Intelligent ML and NLP Pipelines
- Create automated machine learning workflows for search optimization.
- Implement data preprocessing, query understanding, query rewriting, ranking, retrieval, and evaluation pipelines.
- Enable rapid experimentation and continuous model improvement.
Apply Large Language Models (LLMs)
- Leverage cutting-edge LLM technologies to enhance search relevance and user experience.
- Improve semantic understanding of user intent and content relationships.
- Develop intelligent retrieval strategies for large-scale enterprise environments.
Collaborate Across Teams
- Partner with Product Managers, AI Researchers, Data Scientists, and Engineering teams.
- Contribute to technology-driven product roadmaps and business initiatives.
- Drive strategic decisions that improve search and discovery experiences for customers.
Build Search Evaluation Frameworks
- Develop robust offline and online evaluation methodologies.
- Measure ranking effectiveness and search quality improvements.
- Establish metrics and experimentation frameworks to guide product development.
Required Qualifications
Education
- Bachelor’s degree in Computer Science, Artificial Intelligence, Machine Learning, or a related technical field.
- Master’s degree or PhD preferred.
Experience
- 10+ years of experience building and deploying large-scale search relevance systems.
- Proven track record in production environments or high-impact research organizations.
- Hands-on experience applying Large Language Models (LLMs) to search and retrieval challenges.
- Strong expertise in machine learning, information retrieval, and AI-driven search technologies.
Technical Skills
Experience in one or more of the following areas:
- Query Understanding
- Natural Language Processing (NLP)
- Text Mining
- Recommendation Systems
- Personalization Algorithms
- Search & Discovery Platforms
- Conversational AI
- Semantic Search
- Information Retrieval
- Machine Learning Engineering
Additional Qualifications
- Strong foundation in computer science fundamentals, algorithms, and distributed systems.
- Experience contributing to widely adopted open-source software projects.
- Excellent problem-solving, communication, and technical leadership skills.
Why Join Databricks?
At Databricks, you’ll work on some of the most challenging and impactful AI problems in the industry. Our team is building state-of-the-art technologies that redefine how organizations interact with data, analytics, and artificial intelligence.
What You’ll Gain
- Opportunity to build cutting-edge AI and search technologies.
- Work alongside world-class AI, ML, and data engineering experts.
- Influence products used by thousands of organizations worldwide.
- Solve large-scale search and discovery challenges involving millions of assets.
- Accelerate your career in one of the fastest-growing AI companies globally.
If you’re excited about pushing the boundaries of AI, Machine Learning, Search Relevance, and Large Language Models, we encourage you to apply and become part of our mission.
About Databricks
Databricks is the Data and AI company trusted by more than 10,000 organizations worldwide, including leading enterprises and over half of the Fortune 500. The Databricks Data Intelligence Platform helps organizations unify data, analytics, and artificial intelligence to drive innovation and business growth.
Headquartered in San Francisco, Databricks was founded by the original creators of Apache Spark™, Delta Lake, MLflow, and the Lakehouse architecture. Today, Databricks continues to lead the industry in modern data and AI solutions.
Employee Benefits
Databricks offers a comprehensive and competitive benefits package designed to support employees’ health, financial well-being, professional development, and work-life balance. Benefits may vary by location and region.
Diversity, Equity & Inclusion
Databricks is committed to creating an inclusive workplace where individuals from all backgrounds can thrive. We celebrate diversity and ensure equal employment opportunities for all qualified applicants regardless of race, ethnicity, gender identity, sexual orientation, disability, age, religion, veteran status, or any other protected characteristic.
Equal Opportunity Employer
Databricks is proud to be an Equal Opportunity Employer and is committed to fair and inclusive hiring practices throughout the recruitment process.
Compliance Notice
Certain positions may require access to export-controlled technology or source code. Where applicable, employment may be subject to obtaining necessary government authorizations and approvals. Databricks reserves the right to determine eligibility based on applicable legal and regulatory requirements.
Take the next step in your career and help build the future of AI-powered search, machine learning innovation, and intelligent data discovery at Databricks.