Skip to main content

Generative AI Application Evaluation and Governance

This is your introduction to evaluating and governing generative AI systems. First, you’ll explore the meaning behind and motivation for building evaluation and governance/security systems. Next, we’ll connect evaluation and governance systems to the Databricks Data Intelligence Platform. Third, we’ll teach you about a variety of evaluation techniques for specific components and types of applications. Finally, the course will conclude with an analysis of evaluating entire AI systems with respect to performance and cost.


Languages Available: English | 日本語 | Português BR | 한국어

Skill Level
Associate
Duration
4h
Prerequisites
  • Familiarity with natural language processing concepts
  • Familiarity with prompt engineering/prompt engineering best practices 
  • Familiarity with the Databricks Data Intelligence Platform
  • Familiarity with RAG  (preparing data, building a RAG architecture, concepts like embedding, vectors, vector databases, etc.)
  • Experience with building LLM applications using multi-stage reasoning LLM chains and agents



Outline

Importance of Evaluating GenAI Applications

  • Why to Evaluate GenAI Applications
  • Exploring Licensing of Datasets
  • Prompts and Guardrails Basics
  • Implementing and Testing Guardrails for LLMs


Securing and Governing GenAI Applications

  • AI System Security
  • Implementing AI Guardrails


GenAI Evaluation Techniques

  • Evaluation Techniques
  • Benchmark Evaluation
  • LLM-as-a-Judge
  • Domain-Specific Evaluation


End-to-end Application Evaluation

  • AI System Architecture
  • Custom Metrics
  • Offline vs. Online Evaluation

Upcoming Public Classes

Date
Time
Language
Price
Mar 18
09 AM - 01 PM (Europe/London)
English
$750.00
Mar 19
08 AM - 12 PM (America/Los_Angeles)
English
$750.00
Apr 15
09 AM - 01 PM (America/New_York)
English
$750.00
Apr 17
09 AM - 01 PM (Asia/Singapore)
English
$750.00
Apr 18
01 PM - 05 PM (Europe/London)
English
$750.00
May 13
01 PM - 05 PM (Australia/Sydney)
English
$750.00
May 14
09 AM - 01 PM (Europe/London)
English
$750.00
May 14
01 PM - 05 PM (America/New_York)
English
$750.00
Jun 16
01 PM - 05 PM (Europe/London)
English
$750.00
Jun 17
09 AM - 01 PM (America/New_York)
English
$750.00
Jun 18
01 PM - 05 PM (America/Sao_Paulo)
Portuguese - BR
$750.00
Jun 20
09 AM - 01 PM (Australia/Sydney)
English
$750.00
Jul 14
01 PM - 05 PM (America/New_York)
English
$750.00
Jul 16
09 AM - 01 PM (Asia/Kolkata)
English
$750.00
Jul 17
01 PM - 05 PM (Europe/London)
English
$750.00

Public Class Registration

If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.

Private Class Request

If your company is interested in private training, please submit a request.

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Runtime

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Register now

Instructors

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Register now

Learning

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Scale

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Data Engineer

Automated Deployment with Databricks Asset Bundles

This course provides a comprehensive review of DevOps principles and their application to Databricks projects. It begins with an overview of core DevOps, DataOps, continuous integration (CI), continuous deployment (CD), and testing, and explores how these principles can be applied to data engineering pipelines.

The course then focuses on continuous deployment within the CI/CD process, examining tools like the Databricks REST API, SDK, and CLI for project deployment. You will learn about Databricks Asset Bundles (DABs) and how they fit into the CI/CD process. You’ll dive into their key components, folder structure, and how they streamline deployment across various target environments in Databricks. You will also learn how to add variables, modify, validate, deploy, and execute Databricks Asset Bundles for multiple environments with different configurations using the Databricks CLI.

Finally, the course introduces Visual Studio Code as an Interactive Development Environment (IDE) for building, testing, and deploying Databricks Asset Bundles locally, optimizing your development process. The course concludes with an introduction to automating deployment pipelines using GitHub Actions to enhance the CI/CD workflow with Databricks Asset Bundles.

By the end of this course, you will be equipped to automate Databricks project deployments with Databricks Asset Bundles, improving efficiency through DevOps practices.

Paid
4h
Lab
Professional

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.