Generative AI Application Evaluation and Governance
This is your introduction to evaluating and governing generative AI systems. First, you’ll explore the meaning behind and motivation for building evaluation and governance/security systems. Next, we’ll connect evaluation and governance systems to the Databricks Data Intelligence Platform. Third, we’ll teach you about a variety of evaluation techniques for specific components and types of applications. Finally, the course will conclude with an analysis of evaluating entire AI systems with respect to performance and cost.
- Familiarity with natural language processing concepts
- Familiarity with prompt engineering/prompt engineering best practices
- Familiarity with the Databricks Data Intelligence Platform
- Familiarity with RAG (preparing data, building a RAG architecture, concepts like embedding, vectors, vector databases, etc.)
- Experience with building LLM applications using multi-stage reasoning LLM chains and agents
Outline
Importance of Evaluating GenAI Applications
- Why to Evaluate GenAI Applications
- Exploring Licensing of Datasets
- Prompts and Guardrails Basics
- Implementing and Testing Guardrails for LLMs
Securing and Governing GenAI Applications
- AI System Security
- Implementing AI Guardrails
GenAI Evaluation Techniques
- Evaluation Techniques
- Benchmark Evaluation
- LLM-as-a-Judge
- Domain-Specific Evaluation
End-to-end Application Evaluation
- AI System Architecture
- Custom Metrics
- Offline vs. Online Evaluation
Upcoming Public Classes
Date | Time | Language | Price |
---|---|---|---|
Jul 23 | 02 PM - 06 PM (America/Los_Angeles) | English | $750.00 |
Jul 29 | 12 PM - 04 PM (Asia/Kolkata) | English | $750.00 |
Jul 29 | 02 PM - 06 PM (Europe/London) | English | $750.00 |
Public Class Registration
If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.
Private Class Request
If your company is interested in private training, please submit a request.
Registration options
Databricks has a delivery method for wherever you are on your learning journey
Self-Paced
Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos
Register nowInstructor-Led
Public and private courses taught by expert instructors across half-day to two-day courses
Register nowBlended Learning
Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase
Purchase nowSkills@Scale
Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details
Upcoming Public Classes
Career Workshop/
March 20