Meta tags:
description=Course materials for STAT 4830;

Headings (most frequently used words):

and, optimization, on, numerical, machine, learning, course, project, deliverable, software, in, stat, 4830, for, data, science, brief, historical, perspective, table, of, contents, high, level, thoughts, the, official, syllabus, pre, requisites, assessment, final, due, week, textbooks, readings, libraries, timeline, milestones, format, feedback, loop, rules, ideas, tutorials, general, background, convex, python, cvxpy, pytorch, jax, auto, differentiation, scalable, linear, algebra, transformers, diffusion, models, version, control, git, github, training, visualization, puzzles,

Text of the page (most frequently used words):
and (68), the (59), optimization (47), for (34), course (31), #project (26), will (20), your (20), https (20), you (19), methods (18), this (17), #github (16), problems (15), with (15), week (15), learning (14), that (14), research (14), pytorch (11), software (11), com (11), code (11), numerical (10), use (10), report (10), draft (10), data (9), have (9), are (9), final (9), convex (8), models (8), puzzles (8), machine (8), more (7), google (7), self (7), deliverable (7), such (7), based (6), deep (6), language (6), linear (6), slides (6), critique (6), level (6), science (6), cvxpy (5), jax (5), often (5), how (5), training (5), diffusion (5), model (5), algorithms (5), some (5), should (5), due (5), apr (5), stat (5), high (4), modern (4), large (4), work (4), solve (4), useful (4), algorithmic (4), differentiation (4), implementation (4), library (4), org (4), tuning (4), but (4), libraries (4), development (4), each (4), include (4), colab (4), results (4), can (4), from (4), them (4), not (4), feedback (4), something (4), students (4), mar (4), feb (4), 4830 (4), solver (3), well (3), nonconvex (3), build (3), polished (3), get (3), gradient (3), what (3), tools (3), solving (3), let (3), scale (3), srush (3), docs (3), resources (3), algebra (3), educational (3), optimizers (3), python (3), benchmarking (3), benchopt (3), benchmarks (3), edu (3), theory (3), material (3), draw (3), must (3), llm (3), formulate (3), math (3), develop (3), existing (3), improve (3), component (3), discuss (3), detailed (3), environment (3), multiple (3), apply (3), help (3), instructions (3), presentation (3), team (3), end (3), class (3), assessment (3), basic (3), techniques (3), tentative (3), who (3), overview (3), table (3), contents (3), appreciate (2), classical (2), via (2), frameworks (2), driven (2), iterate (2), approach (2), instead (2), form (2), functions (2), iterative (2), other (2), neural (2), like (2), matlab (2), made (2), problem (2), could (2), 20th (2), programming (2), industrial (2), discipline (2), brief (2), historical (2), perspective (2), autodiff (2), gpu (2), tensor (2), gpus (2), automatic (2), hello (2), world (2), started (2), control (2), repository (2), annotated (2), mingpt (2), gpt (2), karpathy (2), cola (2), scalable (2), structure (2), micrograd (2), auto (2), html (2), www (2), press (2), commons (2), ucla (2), introduction (2), online (2), foundations (2), applications (2), there (2), official (2), several (2), added (2), throughout (2), textbooks (2), readings (2), ideas (2), student (2), interests (2), first (2), scope (2), automatically (2), tune (2), lhs (2), rhs (2), one (2), way (2), than (2), would (2), these (2), all (2), related (2), instructor (2), versions (2), improving (2), example (2), repo (2), penn (2), find (2), align (2), extensive (2), write (2), fine (2), choose (2), best (2), developing (2), projects (2), also (2), coding (2), group (2), address (2), format (2), written (2), key (2), become (2), break (2), why (2), peer (2), pre (2), requisites (2), mathematical (2), formulations (2), sequential (2), decision (2), making (2), which (2), numerically (2), see (2), schedule (2), know (2), production (2), focus (2), goal (2), meant (2), their (2), about (2), thoughts (2), both, sides, approaches, structured, flexible, powered, tasks, era, tensorflow, shifted, emphasis, specifying, our, hands, dirty, direct, custom, loss, exploration, precisely, enabled, explosion, llms, powerful, architectures, 2000s, cvx, formulating, standard, accessible, broad, audience, specify, friendly, handle, 1960s, 1990s, grew, important, interior, point, mid, century, emerged, critical, tool, operations, fueled, george, dantzig, simplex, method, was, pivotal, logistics, military, planning, resource, allocation, evolved, over, last, seventy, years, sasha, rush, has, luckily, series, open, source, helping, understand, weights, biases, wandb, site, visualization, start, journey, version, git, containing, papers, diff, usion, awesome, huggingface, blog, transformers, framework, exploits, found, wilson, labs, tiny, stable, optim, full, everyone, tutorials, schmidt, 2021, descending, through, crowded, valley, proceedings, mlr, v139, schmidt21a, moreau, 2022, reproducible, efficient, collaborative, 2023, network, mlcommons, efficiency, tree, main, arxiv, abs, 2306, 07179, playbook, tuning_playbook, ece236c, systems, seas, vandenbe, ee236c, hazan, 2016, trends, 157, 325, boyd, vandenberghe, 2004, cambridge, university, nocedal, wright, 1999, springer, beck, 2014, nonlinear, society, applied, mathematics, general, background, contained, outside, prerequisites, textbook, recent, published, academic, conferences, journals, mature, below, listed, may, post, adapted, weeks, equivalent, walkthrough, claims, approve, autoopt, combined, given, task, need, prove, disprove, inequalities, leq, variable, minimize, value, less, zero, disproved, inequality, highly, difficult, tricks, relaxations, thereof, really, cool, illustrating, advanced, concepts, covered, comfortable, including, future, little, modification, new, benchmark, recently, been, proposed, independent, researchers, attempting, train, small, fast, possible, potential, own, set, initial, records, algoperf, modded, nanogpt, augmenting, say, already, working, faculty, includes, likely, bit, ensure, enough, credit, reproducibility, attempt, reproduce, falsify, literature, novel, prime, intellect, hub, demonstrate, performing, leads, desired, behavior, strategy, vision, describe, failures, successes, rabbit, holes, illustrate, compare, performance, skills, learned, steps, receive, times, better, chance, creating, valuable, shape, content, areas, interest, ensuring, lectures, assignments, goals, writing, brainstorming, verify, outputs, critically, don, rely, blindly, usage, fosters, collaboration, while, keeping, workload, balanced, size, rules, additional, questions, welcome, anytime, meet, semi, weakly, direction, reflect, progress, challenges, check, before, loop, creative, mind, just, run, default, short, conference, style, paper, pages, supplementary, provide, runnable, demonstration, overall, create, minimal, notebook, shows, core, functionality, executable, demo, sure, clear, readme, reproducing, centralize, materials, single, rushed, into, regular, drafts, cycles, substantial, refined, checkpoints, opportunities, revision, piece, showcase, worthy, portfolio, internship, note, semester, presentations, spring, lightning, talks, friday, jan, email, names, jiahao, jiahaoai, wharton, upenn, deliverables, fridays, timeline, milestones, very, available, following, link, base, midterm, process, involvement, critiques, participation, attendance, starting, thur, march, experience, scripting, comfort, iteration, probability, 4300, calculus, 2400, intelligent, consumer, principles, underlying, practices, popular, stochastic, quasi, newton, extensions, constrained, regularized, distributed, preserve, privacy, sensitive, tour, viewed, expect, considerations, convexity, smoothness, saddle, points, stochasticity, quadratic, semidefinite, programs, solvers, statistical, estimation, inverse, predictive, generative, bandit, reinforcement, modeling, formulated, solved, teach, mathematically, appropriate, implement, topics, syllabus, local, free, remote, cursor, ide, aware, scratch, adapt, similar, courses, universities, mainly, notebooks, focuses, intricacies, taking, separate, skill, able, fields, intelligently, experimental, undergraduate, roughly, junior, senior, year, beneficial, phd, wish, want, fundamental, maximally, portable, between, virtually, every, field, study, quote, openai, joshua, achiam, works, everything, learn, career, whether, scientist, researcher, engineer, any, kind, understanding, actually, real, excited, tuesdays, office, hours, lecture, notes, aka, view, damek, davis,

Text of the page (random words):
course overview stat 4830 numerical optimization for data science and machine learning stat 4830 numerical optimization for data science and machine learning damek davis table of contents course overview view on github stat 4830 numerical optimization for data science and machine learning aka optimization in pytorch for lecture notes see table of contents office hours tuesdays 12 30 1 30 pm table of contents high level thoughts on the course overview pre requisites deliverable a final project due on week 4 textbooks readings and software libraries brief historical perspective on optimization high level thoughts on the course why am i excited about this course optimization works everything you learn about solving optimization problems numerically will be useful in your career whether you re a data scientist ai researcher or an engineer of any kind understanding how optimization methods work and how to use them will actually help you solve real problems to quote joshua achiam openai if you want to know something deep fundamental and maximally portable between virtually every field study mathematical optimization who is this course meant for this is an experimental undergraduate course meant for students who are roughly in their junior or senior year the course should also be beneficial to phd students who wish to apply optimization techniques in their research what is the goal of the course by the end of the course my goal is for you to be able to intelligently apply numerical optimization techniques in a research project in machine learning data science or related fields we will discuss optimization theory in the course but that will not be the focus production vs research level code we will mainly be coding python notebooks developing research level code that focuses on the intricacies of optimization methods taking research level code to production is a separate skill that we will not focus on in this course are there similar courses at penn or other universities not that i am aware of if you find some let me know i will develop this course from scratch and adapt it to student interests as we go which ide should i use for local development use cursor it s free for students with a edu address for remote development use google colab official syllabus see tentative course schedule for the tentative course schedule optimization is the modeling language in which modern data science machine learning and sequential decision making problems are formulated and solved numerically this course will teach you how to formulate these problems mathematically choose appropriate algorithms to solve them and implement and tune the algorithms in pytorch tentative topics include optimization based formulations of statistical estimation and inverse problems in data science predictive and generative models in machine learning and control bandit and reinforcement learning problems in sequential decision making a high level tour of the foundations of mathematical optimization viewed as an algorithmic discipline and what to expect from theory key considerations such as convexity smoothness saddle points and stochasticity classical formulations such as linear quadratic and semidefinite programs numerical solvers such as cvxpy popular optimization methods such as online and stochastic gradient methods quasi newton methods algorithmic extensions to constrained regularized and distributed problems as well as optimization methods that preserve privacy of sensitive data modern software libraries such as pytorch and jax and the principles underlying automatic differentiation techniques best practices in tuning optimization methods e g in deep learning problems by the end of this course you will become an intelligent consumer of numerical methods and software for solving modern optimization problems pre requisites basic calculus and linear algebra math 2400 basic probability stat 4300 programming experience in a scripting language such as python basic comfort in iteration and functions assessment in class participation and attendance starting thur march 20th 5 final project report implementation and development process 65 implementation 25 report self critiques 25 team peer assessment 5 instructor assessment of involvement in project 10 midterm project presentation 10 final project presentation 20 final project deliverable due on week 4 very detailed instructions on how to get started with your project are available at the following link stat 4830 project base timeline and milestones deliverables due fridays week 2 jan 23 email project team names to ai jiahao jiahaoai wharton upenn edu week 4 feb 6 report draft 1 code self critique week 5 feb 13 slides draft 1 week 6 feb 20 report draft 2 code self critique week 7 feb 27 slides draft 2 week 8 lightning talks in class mar 3 5 report draft 3 due friday spring break mar 7 15 week 9 mar 20 slides draft 3 week 10 mar 27 report draft 4 code self critique week 11 apr 3 slides draft 4 week 12 apr 10 report draft 5 code self critique week 13 apr 17 slides draft 5 week 14 apr 21 23 final presentations in class week 15 apr 28 final report code self critique note instructions for peer feedback will be added throughout the semester for each deliverable why this approach final projects often become a single rushed deliverable we ll break the project into regular drafts and feedback cycles so your team can iterate improve and build something more substantial and refined you ll have multiple checkpoints each with opportunities for critique and revision by the end you ll have a polished piece of work you can showcase something worthy of your portfolio or internship applications deliverable format github repository centralize all materials your written report code and presentation slides be sure to include a clear readme with instructions for reproducing results executable demo provide a runnable demonstration in google colab if your overall code is extensive create a minimal colab notebook that shows core functionality or key results written report by default structure it like a short conference style paper e g 8 pages supplementary if you have a more creative format in mind just run it by me first feedback loop i will meet with each group semi weakly to discuss project scope and direction reflect on progress and address challenges and check in before deliverable are due additional questions are welcome anytime rules group size 3 4 students this fosters collaboration while keeping the workload balanced llm usage you should use large language models in your work to help with writing brainstorming and coding but verify outputs critically don t rely on them blindly by developing your projects in iterative steps you will receive feedback multiple times and have a better chance of creating something valuable this also help me shape the course content based on your areas of interest ensuring that lectures and assignments align well with your goals course project ideas in the course project you will apply the optimization skills you learned in the course for example your final project could include training fine tuning a model a detailed write up of your strategy for training or fine tuning a model e g language code math or vision model describe failures successes and rabbit holes illustrate the use of the final model use gpus to scale up compare multiple optimization methods and choose one with the best performance build a useful novel rl environment on prime intellect s environment hub demonstrate that performing rl on the environment leads to desired behavior reproducibility a detailed write up of your attempt to reproduce falsify or improve numerical results from the optimization literature augmenting an existing research project let s say you re already working on a research project with penn faculty that includes an optimization component we can likely find a way to align your project in the course with your research project by improving the optimization component of your project we must discuss a bit to ensure the optimization component is extensive enough for course credit improving optimizers on an existing new benchmark several benchmarks have recently been proposed for optimization methods in ml for example 1 in modded nanogpt repo some independent researchers are attempting to train small versions of gpt as fast as possible 2 in the algoperf repo the ml commons project is benchmarking optimization methods for training deep learning models a potential project is to improve existing methods on such benchmarks or develop your own and set the initial records educational tools and software educational resources illustrating advanced concepts related to but not covered in the course material the project should be polished the instructor should be comfortable including the deliverable in future versions of the course with little to no modification ai for math software in research we often need to prove or disprove inequalities of the form lhs x leq rhs x for some variable x one way to do this is to use optimization software to minimize rhs x lhs x if the value is less than zero than we ve disproved the inequality it would be highly useful to develop software that can do this automatically with optimization methods often these are difficult nonconvex optimization problems so you ll have to use all the tricks from the course to solve them use an llm to formulate the problem or relaxations thereof this would be really cool to have autoopt tools combined with an llm automatically formulate and tune optimization methods for a given task i will post more project ideas adapted to student interests in the course s first weeks each project must include a google colab or equivalent walkthrough of your results and claims i must approve the scope of the project textbooks readings and software libraries this course material will be self contained outside of the prerequisites but there will be no official textbook some material will draw on recent research published in academic conferences and journals some will draw on software libraries that are mature or in development below i have instead listed several resources that we may draw more will be added throughout the course optimization general background beck a 2014 introduction to nonlinear optimization theory algorithms and applications with matlab society for industrial and applied mathematics nocedal j wright s j 1999 numerical optimization springer boyd s vandenberghe l 2004 convex optimization cambridge university press hazan e 2016 introduction to online convex optimization foundations and trends in optimization 2 3 4 157 325 ucla course ece236c optimization methods for large scale systems https www seas ucla edu vandenbe ee236c html numerical optimization in machine learning google research n d deep learning tuning playbook https github com google research tuning_playbook google research 2023 benchmarking neural network training algorithms and the ml commons library https arxiv org abs 2306 07179 https github com mlcommons algorithmic efficiency tree main moreau t et al 2022 benchopt reproducible efficient and collaborative optimization benchmarks https github com benchopt benchopt schmidt r et al 2021 descending through a crowded valley benchmarking deep learning optimizers https proceedings mlr press v139 schmidt21a software and tutorials convex optimization in python cvxpy cvxpy convex optimization for everyone https www cvxpy org pytorch jax and auto differentiation pytorch the full library https pytorch org numerical implementation of optimizers in pytorch https pytorch org docs stable optim html algorithms jax library https github com google jax micrograd a tiny educational auto differentiation library https github com karpathy micrograd scalable linear algebra cola a framework for scalable linear algebra that exploits structure often found in machine learning problems https github com wilson labs cola transformers and diffusion models mingpt a pytorch re implementation of gpt https github com karpathy mingpt the annotated diffusion model https huggingface co blog annotated diffusion repository containing resources and papers for diffusion models https diff usion github io awesome diffusion models version control git and github github docs hello world https docs github com en get started start your journey hello world training visualization weights and biases https wandb ai site puzzles sasha rush s has luckily made a series of open source puzzles for helping you understand gpus automatic differentiation and optimization tensor puzzles https github com srush tensor puzzles gpu puzzles https github com srush gpu puzzles autodiff puzzles https github com srush autodiff puzzles brief historical perspective on optimization it s useful to appreciate how optimization evolved as an algorithmic discipline over the last seventy years mid 20th century linear programming emerged as a critical tool in operations research fueled by george dantzig s simplex method this was pivotal for industrial logistics military planning and resource allocation 1960s 1990s convex optimization grew more important driven by work on gradient based methods interior point methods and software that could solve large scale linear and convex problems 2000s tools like cvx matlab based made formulating and solving standard convex problems more accessible to a broad audience i e specify your problem in a solver friendly language and let the solver handle it modern era deep learning frameworks e g pytorch tensorflow jax have shifted the emphasis to a build and iterate approach instead of specifying problems in a polished convex form we often get our hands dirty with nonconvex models direct gradient based methods and custom loss functions this iterative exploration is precisely what enabled the explosion of large language models llms and other powerful neural architectures in this course we will appreciate both sides solver based approaches for classical well structured problems via cvxpy and more flexible high powered frameworks via pytorch for data driven nonconvex tasks