MST 508Mastery · 90 days3 US credit hours·Mastery course · Upper-division / professional elective·v1 · updated 3 Oct 2026

LLM Engineering & LLMOps Mastery

University / Board Instructor Student Outcomes & Skills

Course outcomes

CO1

Demonstrate Foundations & Theory competence: complete the Day 1–15 builds, explain the underlying theory, and defend the phase artefact under live questioning.

C2 Understand

Foundation model mechanics Model strategy: prompting vs RAG vs fine-tuning LLM cost & latency estimation Running open models locally (Ollama)Production LLM client engineering (streaming, retries, cancellation)Structured outputs & schema validation Team delivery & technical communication

CO2

Demonstrate Core LLM Engineering competence: complete the Day 16–30 builds, explain the underlying theory, and defend the phase artefact under live questioning.

C3 Apply

Production LLM client engineering (streaming, retries, cancellation)Context engineering Prompt registry: versioning, A/B routing, rollback Structured outputs & schema validation Tool calling & MCP integration DSPy: metric-driven prompt optimisation Model selection, routing, fallback and provider portability Agent memory, budgets & guardrails Team delivery & technical communication

CO3

Demonstrate Retrieval, RAG & Agents competence: complete the Day 31–45 builds, explain the underlying theory, and defend the phase artefact under live questioning.

C4 Analyse

Tool calling & MCP integration RAG pipeline design Hybrid search, chunking & reranking Citations, groundedness and the ten-cause RAG diagnosis Agentic & self-correcting retrieval (routing, Self-RAG, CRAG, multi-hop)Multi-agent orchestration Team delivery & technical communication

CO4

Demonstrate Evaluation & Observability competence: complete the Day 46–60 builds, explain the underlying theory, and defend the phase artefact under live questioning.

C4 Analyse

LLM cost & latency estimation Prompt registry: versioning, A/B routing, rollback Evaluation engineering with golden sets LLM-as-judge calibration CI regression gates for AI LLMOps observability & SLOs A/B experiment design for LLM features Team delivery & technical communication

CO5

Demonstrate Serving, Fine-tuning & Performance competence: complete the Day 61–75 builds, explain the underlying theory, and defend the phase artefact under live questioning.

C5 Evaluate

Running open models locally (Ollama)Structured outputs & schema validation Model selection, routing, fallback and provider portability Inference math, capacity planning & load testing vLLM serving, PagedAttention & speculative decoding Quantization: GPTQ, AWQ, GGUF and measured quality cost Fine-tuning: dataset curation, LoRA/QLoRA, SFT, DPO Model registry, adapter serving & staged promotion Team delivery & technical communication

CO6

Demonstrate Production LLMOps, Security & Mastery competence: complete the Day 76–90 builds, explain the underlying theory, and defend the phase artefact under live questioning.

C6 Create

Model strategy: prompting vs RAG vs fine-tuning LLM cost & latency estimation Production LLM client engineering (streaming, retries, cancellation)Prompt registry: versioning, A/B routing, rollback Evaluation engineering with golden sets AI security & prompt-injection defence AI red-teaming Agent memory, budgets & guardrails PII redaction, retention & LLM data governance Agent environments & sandboxing Responsible AI & regulatory compliance Failure taxonomy, improvement flywheel, labelling operations and incident response Internal LLM platform: golden paths & self-service Team delivery & technical communication

Phases & skills

Phase	Topic	Outcomes	Deliverable	Skills
1	Foundations & Theory Days 1–15 · C2	CO1	CAPSTONE (Phase 1) — the LLM systems primer	Foundation model mechanics Model strategy: prompting vs RAG vs fine-tuning LLM cost & latency estimation Running open models locally (Ollama)Production LLM client engineering (streaming, retries, cancellation)Structured outputs & schema validation Team delivery & technical communication
2	Core LLM Engineering Days 16–30 · C3	CO2	CAPSTONE (Phase 2) — the tool-using structured assistant	Production LLM client engineering (streaming, retries, cancellation)Context engineering Prompt registry: versioning, A/B routing, rollback Structured outputs & schema validation Tool calling & MCP integration DSPy: metric-driven prompt optimisation Model selection, routing, fallback and provider portability Agent memory, budgets & guardrails Team delivery & technical communication
3	Retrieval, RAG & Agents Days 31–45 · C4	CO3	CAPSTONE (Phase 3) — the grounded, cited RAG agent	Tool calling & MCP integration RAG pipeline design Hybrid search, chunking & reranking Citations, groundedness and the ten-cause RAG diagnosis Agentic & self-correcting retrieval (routing, Self-RAG, CRAG, multi-hop)Multi-agent orchestration Team delivery & technical communication
4	Evaluation & Observability Days 46–60 · C4	CO4	CAPSTONE (Phase 4) — the evaluation & observability harness	LLM cost & latency estimation Prompt registry: versioning, A/B routing, rollback Evaluation engineering with golden sets LLM-as-judge calibration CI regression gates for AI LLMOps observability & SLOs A/B experiment design for LLM features Team delivery & technical communication
5	Serving, Fine-tuning & Performance Days 61–75 · C5	CO5	CAPSTONE (Phase 5) — fine-tune, quantize, serve, benchmark	Running open models locally (Ollama)Structured outputs & schema validation Model selection, routing, fallback and provider portability Inference math, capacity planning & load testing vLLM serving, PagedAttention & speculative decoding Quantization: GPTQ, AWQ, GGUF and measured quality cost Fine-tuning: dataset curation, LoRA/QLoRA, SFT, DPO Model registry, adapter serving & staged promotion Team delivery & technical communication
6	Production LLMOps, Security & Mastery Days 76–90 · C6	CO6	MASTERY CAPSTONE — ship a production-grade LLM platform slice	Model strategy: prompting vs RAG vs fine-tuning LLM cost & latency estimation Production LLM client engineering (streaming, retries, cancellation)Prompt registry: versioning, A/B routing, rollback Evaluation engineering with golden sets AI security & prompt-injection defence AI red-teaming Agent memory, budgets & guardrails PII redaction, retention & LLM data governance Agent environments & sandboxing Responsible AI & regulatory compliance Failure taxonomy, improvement flywheel, labelling operations and incident response Internal LLM platform: golden paths & self-service Team delivery & technical communication