Skip to main content

Generative AI Application Evaluation and Governance

This is your introduction to evaluating and governing generative AI systems. First, you’ll explore the meaning behind and motivation for building evaluation and governance/security systems. Next, we’ll connect evaluation and governance systems to the Databricks Data Intelligence Platform. Third, we’ll teach you about a variety of evaluation techniques for specific components and types of applications. Finally, the course will conclude with an analysis of evaluating entire AI systems with respect to performance and cost.


Skill Level
Associate
Duration
4h
Prerequisites
  • Familiarity with natural language processing concepts
  • Familiarity with prompt engineering/prompt engineering best practices 
  • Familiarity with the Databricks Data Intelligence Platform
  • Familiarity with RAG  (preparing data, building a RAG architecture, concepts like embedding, vectors, vector databases, etc.)
  • Experience with building LLM applications using multi-stage reasoning LLM chains and agents



Outline

Importance of Evaluating GenAI Applications

  • Why to Evaluate GenAI Applications
  • Exploring Licensing of Datasets
  • Prompts and Guardrails Basics
  • Implementing and Testing Guardrails for LLMs


Securing and Governing GenAI Applications

  • AI System Security
  • Implementing AI Guardrails


GenAI Evaluation Techniques

  • Evaluation Techniques
  • Benchmark Evaluation
  • LLM-as-a-Judge
  • Domain-Specific Evaluation


End-to-end Application Evaluation

  • AI System Architecture
  • Custom Metrics
  • Offline vs. Online Evaluation

Upcoming Public Classes

Date
Time
Language
Price
Jul 23
02 PM - 06 PM (America/Los_Angeles)
English
$750.00
Jul 29
12 PM - 04 PM (Asia/Kolkata)
English
$750.00
Jul 29
02 PM - 06 PM (Europe/London)
English
$750.00

Public Class Registration

If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.

Private Class Request

If your company is interested in private training, please submit a request.

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Runtime

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Register now

Instructors

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Register now

Learning

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Scale

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Career Workshop

Career Workshop/

March 20

Careers at Databricks

We're on a mission to help data teams solve the world's toughest problems. Will you join us?
Advance my career now

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.