Meta tags:
description=Official course website for Stanford CS336: Language Modeling from Scratch (Spring 2026), including logistics, schedule, assignments, and course materials.;
author=Stanford CS336 Staff;

Headings (most frequently used words):

course, coursework, cs336, language, modeling, from, scratch, staff, logistics, content, schedule, youtube, playlist, what, is, this, about, prerequisites, assignments, gpu, compute, for, self, study, honor, code, submitting, late, days, regrade, requests, sponsor,

Text of the page (most frequently used words):
the (36), and (25), assignment (20), you (18), for (16), code (12), percy (11), will (11), language (11), wed (10), mon (10), model (10), may (9), tatsu (9), april (9), pdf (8), #course (8), training (7), are (7), your (7), with (7), data (6), all (6), this (6), #assignments (6), students (6), gpu (6), pricing (6), hour (6), from (6), learning (6), gates (6), due (5), lecture (5), class (5), preview (5), out (5), scaling (5), please (5), compute (5), that (5), coursework (5), should (5), models (5), 30pm (5), spring (5), like (4), stanford (4), days (4), late (4), not (4), own (4), implementation (4), questions (4), systems (4), cs336 (4), alignment (3), part (3), gpus (3), memory (3), schedule (3), other (3), classes (3), student (3), modal (3), staff (3), submit (3), can (3), deadline (3), any (3), ask (3), slack (3), implement (3), when (3), about (3), using (3), autocomplete (3), have (3), study (3), their (3), train (3), transformer (3), basics (3), machine (3), deep (3), understanding (3), python (3), scratch (3), guest (2), june (2), day (2), optional (2), post (2), filtering (2), deduplication (2), evaluation (2), laws (2), parallelism (2), triton (2), attention (2), version (2), pytorch (2), march (2), deadlines (2), youtube (2), playlist (2), following (2), sponsor (2), made (2), regrade (2), gradescope (2), use (2), per (2), each (2), hours (2), work (2), than (2), submitting (2), many (2), only (2), contact (2), via (2), email (2), handouts (2), self (2), existing (2), such (2), level (2), but (2), completing (2), function (2), names (2), makes (2), content (2), policy (2), tools (2), understand (2), complete (2), group (2), collaboration (2), honor (2), operations (2), together (2), nebius (2), runpod (2), lambda (2), public (2), single (2), 2026 (2), apply (2), math (2), minimal (2), standard (2), comfortable (2), know (2), basic (2), nlp (2), system (2), through (2), entire (2), operating (2), steven (2), cao (2), herman (2), brunborg (2), wednesdays (2), fridays (2), marcel (2), rød (2), tuesdays (2), hashimoto (2), liang (2), instructor (2), previous (2), offerings (2), 2024 (2), 2025 (2), modeling (2), dan, daniel, selsam, lecture_17, multimodality, memorial, lecture_16, rlvr, lecture_15, mid, sft, rlhf, lecture_14, mixing, synthetic, lecture_13, sources, datasets, lecture_12, lecture_11, lecture_10, inference, lecture_09, lecture_08, lecture_07, lecture_06, kernels, tpus, alternatives, mixture, experts, architectures, hyperparameters, recording, lecture_02, einops, resource, accounting, flops, arithmetic, intensity, lecture_01, overview, tokenization, materials, description, date, currently, tentative, listed, take, seriously, respect, policies, would, thank, sponsoring, believe, objective, error, grading, request, within, after, grades, released, requests, has, extends, partial, better, times, until, grade, last, submission, anything, goes, wrong, question, assistant, submitted, implementations, things, exist, online, give, contained, need, consult, third, party, producing, thus, look, unless, otherwise, specified, prompting, llms, chatgpt, permitted, low, programming, high, conceptual, directly, solve, problem, prohibited, strongly, encourage, disable, cursor, tab, github, copilot, ide, though, non, autocompleting, totally, fine, found, much, harder, engage, deeply, see, inspired, groups, allowed, must, hand, one, worked, put, members, top, convenience, save, money, recommend, debugging, correctness, cpu, first, then, count, recommended, runs, benchmarking, minimum, cheaper, longer, commitments, preemptible, labs, offers, free, monthly, charged, actual, idle, resources, switching, between, local, dev, large, scale, experiments, simple, here, few, options, b200, along, home, access, cloud, provider, safety, methods, dpo, supervised, finetuning, reinforcement, lms, reason, solving, problems, reasoning, perform, improve, performance, convert, raw, common, crawl, dumps, into, usable, pretraining, query, api, fit, law, project, component, build, efficient, distributed, profile, benchmark, layers, advanced, optimize, flashattention2, components, tokenizer, architecture, optimizer, necessary, note, unit, very, heavy, allocate, enough, time, cs221, cs229, cs230, cs124, cs224n, 109, equivalent, probabilities, gaussian, distributions, mean, deviation, etc, probability, statistics, cme, 100, matrix, vector, notation, college, calculus, linear, algebra, significant, involve, making, neural, run, quickly, efficiently, across, multiple, machines, expect, able, strong, familiarity, concepts, hierarchy, experience, optimization, majority, unlike, most, given, scaffolding, amount, write, least, order, magnitude, greater, therefore, being, proficient, software, engineering, paramount, proficiency, prerequisites, serve, cornerstone, modern, natural, processing, applications, open, new, paradigm, having, general, purpose, address, range, downstream, tasks, field, artificial, intelligence, continues, grow, possessing, becomes, essential, scientists, engineers, alike, designed, provide, comprehensive, walking, them, process, developing, drawing, inspiration, courses, create, lead, every, aspect, creation, including, collection, cleaning, pre, construction, before, deployment, what, related, channels, announcements, also, personal, matters, spr2526, lists, edu, mondays, thursdays, 30am, 200, location, 392, 498, 415, 12am, 364, 11am, 12pm, 366, office, recordings, monday, wednesday, 20pm, skilling, auditorium, lectures, logistics, tatsunori,

Text of the page (random words):
stanford cs336 language modeling from scratch cs336 coursework schedule cs336 language modeling from scratch stanford spring 2026 previous offerings spring 2025 spring 2024 previous offerings spring 2025 spring 2024 course staff tatsunori hashimoto instructor percy liang instructor herman brunborg ca marcel rød ca steven cao ca logistics lectures monday wednesday 3 00 4 20pm in skilling auditorium recordings youtube playlist office hours percy liang fridays 11am 12pm in gates 366 tatsu hashimoto tuesdays 11 12am in gates 364 marcel rød tuesdays 4 30 5 30pm in gates 498 wednesdays 4 30 5 30pm in gates 415 herman brunborg wednesdays 1 30 2 30pm fridays 1 30 2 30pm location gates 392 steven cao mondays 4 30 5 30pm thursdays 9 30 10 30am gates 200 contact students should ask all course related questions in public slack channels all announcements will also be made in slack for personal matters email cs336 spr2526 staff lists stanford edu content what is this course about language models serve as the cornerstone of modern natural language processing nlp applications and open up a new paradigm of having a single general purpose system address a range of downstream tasks as the field of artificial intelligence ai machine learning ml and nlp continues to grow possessing a deep understanding of language models becomes essential for scientists and engineers alike this course is designed to provide students with a comprehensive understanding of language models by walking them through the entire process of developing their own drawing inspiration from operating systems courses that create an entire operating system from scratch we will lead students through every aspect of language model creation including data collection and cleaning for pre training transformer model construction model training and evaluation before deployment prerequisites proficiency in python the majority of class assignments will be in python unlike most other ai classes students will be given minimal scaffolding the amount of code you will write will be at least an order of magnitude greater than for other classes therefore being proficient in python and software engineering is paramount experience with deep learning and systems optimization a significant part of the course will involve making neural language models run quickly and efficiently on gpus across multiple machines we expect students to be able to have a strong familiarity with pytorch and know basic systems concepts like the memory hierarchy college calculus linear algebra e g math 51 cme 100 you should be comfortable understanding matrix vector notation and operations basic probability and statistics e g cs 109 or equivalent you should know the basics of probabilities gaussian distributions mean standard deviation etc machine learning e g cs221 cs229 cs230 cs124 cs224n you should be comfortable with the basics of machine learning and deep learning note that this is a 5 unit class this is a very implementation heavy class so please allocate enough time for it coursework assignments assignment 1 basics implement all of the components tokenizer model architecture optimizer necessary to train a standard transformer language model train a minimal language model assignment 2 systems profile and benchmark the model and layers from assignment 1 using advanced tools optimize attention with your own triton implementation of flashattention2 build a memory efficient distributed version of the assignment 1 model training code assignment 3 scaling understand the function of each component of the transformer query a training api to fit a scaling law to project model scaling assignment 4 data convert raw common crawl dumps into usable pretraining data perform filtering and deduplication to improve model performance assignment 5 alignment and reasoning rl apply supervised finetuning and reinforcement learning to train lms to reason when solving math problems optional part 2 implement and apply safety alignment methods such as dpo all currently tentative deadlines are listed in the schedule gpu compute for self study if you are following along at home you can access gpu compute from a cloud provider to complete the assignments here are a few options public pricing for a single b200 gpu on march 28 2026 modal sponsor 6 25 hour offers 30 of free monthly compute you are only charged for actual compute no idle resources and their ux makes switching between local dev and large scale gpu experiments simple modal pricing lambda labs 6 69 hour lambda pricing runpod 4 99 hour runpod pricing nebius 5 50 hour 3 05 hour preemptible nebius pricing together 7 49 hour minimum 8 gpus cheaper for longer commitments together pricing for convenience and to save money we recommend debugging correctness of your implementation on cpu first and then using gpu s with the count recommended in the assignments for completing training runs a1 a4 a5 or benchmarking gpu operations a2 honor code like all other classes at stanford we take the student honor code seriously please respect the following policies collaboration study groups are allowed but students must understand and complete their own assignments and hand in one assignment per student if you worked in a group please put the names of the members of your study group at the top of your assignment please ask if you have any questions about the collaboration policy ai tools prompting llms such as chatgpt is permitted for low level programming questions or high level conceptual questions about language models but using it directly to solve the problem is prohibited we strongly encourage you to disable ai autocomplete e g cursor tab github copilot in your ide when completing assignments though non ai autocomplete e g autocompleting function names is totally fine we have found that ai autocomplete makes it much harder to engage deeply with the content see the ai policy inspired by this existing code implementations for many of the things you will implement exist online the handouts we ll give will be self contained so that you will not need to consult third party code for producing your own implementation thus you should not look at any existing code unless when otherwise specified in the handouts submitting coursework all coursework are submitted via gradescope by the deadline do not submit your coursework via email if anything goes wrong please ask a question in slack or contact a course assistant you can submit as many times as you d like until the deadline we will only grade the last submission partial work is better than not submitting any work late days each student has 6 late days to use a late day extends the deadline by 24 hours you can use up to 3 late days per assignment regrade requests if you believe that the course staff made an objective error in grading you may submit a regrade request on gradescope within 3 days after the grades are released sponsor we would like to thank modal for sponsoring compute for this class schedule youtube playlist date description course materials deadlines 1 mon march 30 overview tokenization percy lecture_01 py assignment 1 out code preview 2 wed april 1 pytorch einops resource accounting flops memory arithmetic intensity percy lecture_02 py recording version 3 mon april 6 architectures hyperparameters tatsu lecture 3 pdf 4 wed april 8 attention alternatives and mixture of experts tatsu lecture 4 pdf 5 mon april 13 gpus tpus tatsu lecture 5 pdf 6 wed april 15 kernels triton percy lecture_06 py assignment 1 due assignment 2 out code preview 7 mon april 20 parallelism percy lecture_07 py 8 wed april 22 parallelism tatsu lecture_08 pdf 9 mon april 27 scaling laws tatsu lecture_09 pdf 10 wed april 29 inference percy lecture_10 py assignment 2 due assignment 3 out code preview 11 mon may 4 scaling laws tatsu lecture_11 pdf 12 wed may 6 evaluation percy lecture_12 py assignment 3 due assignment 4 out code preview 13 mon may 11 data sources datasets percy lecture_13 py 14 wed may 13 data filtering deduplication mixing synthetic data percy lecture_14 py 15 mon may 18 mid post training sft rlhf tatsu lecture_15 pdf 16 wed may 20 post training rlvr tatsu lecture_16 pdf assignment 4 due assignment 5 out code preview optional part 2 mon may 25 no class memorial day 17 wed may 27 alignment multimodality percy lecture_17 py 18 mon june 1 guest lecture daniel selsam 19 wed june 3 guest lecture dan fu assignment 5 due