Meta tags:
description=Prof. Darren Cosker - Principal Scientist Manager at Microsoft, Professor at the University of Bath, working across multimodal AI, post-training, proactive agents, computer vision, graphics, motion, and digital humans.;

Headings (most frequently used words):

and, ai, previous, impact, from, to, research, 11, prof, darren, cosker, navigate, leadership, delivery, product, multimodal, immersive, experiences, digital, humans, biomechanics, human, perception, funding, awards, public, data, alumni, university, personal, microsoft, mesh, camera, memories, retold, is, anna, ok, cosmos, within, us, rgbd, dog, d3dfacs, shadow, removal,

Text of the page (most frequently used words):
and (118), the (55), for (27), camera (18), motion (18), #research (18), with (17), work (16), that (16), systems (16), capture (15), from (14), real (14), microsoft (13), time (12), teams (12), human (12), digital (11), multimodal (11), have (10), facial (10), computer (10), vision (10), experiences (10), this (10), more (9), data (9), animation (9), product (9), impact (9), humans (8), immersive (8), training (8), into (8), evaluation (7), fec (7), building (7), through (7), mesh (7), also (6), our (6), can (6), performance (6), perception (6), communication (6), where (6), has (6), post (6), previous (6), one (5), bath (5), director (5), including (5), applications (5), 2020 (5), generation (5), royal (5), funding (5), understanding (5), products (5), was (5), team (5), models (5), delivered (5), collaboration (5), you (4), professor (4), technical (4), university (4), across (4), dynamic (4), page (4), creating (4), reality (4), value (4), studio (4), centre (4), analysis (4), entertainment (4), below (4), well (4), pose (4), estimation (4), these (4), technologies (4), central (4), presence (4), biomechanics (4), useful (4), video (4), avatars (4), feedback (4), driven (4), model (4), proactive (4), agents (4), reinforcement (4), learning (4), tools (4), based (4), people (4), helping (4), leadership (4), darren (3), cosker (3), working (3), together (3), are (3), personal (3), engineers (3), shadow (3), removal (3), dataset (3), d3dfacs (3), modelling (3), rgbd (3), fellowship (3), 2012 (3), society (3), industry (3), 2016 (3), epsrc (3), total (3), project (3), innovate (3), 2018 (3), 2019 (3), 2021 (3), major (3), such (3), projects (3), avatar (3), full (3), body (3), mixed (3), signals (3), created (3), used (3), facing (3), graphics (3), synthetic (3), generative (3), theme (3), interaction (3), built (3), face (3), been (3), recently (3), studios (3), using (3), house (3), turning (3), meeting (3), intelligence (3), lab (3), live (3), improve (3), how (3), conversational (3), interested (3), part (3), multiple (3), scientist (3), foundation (2), all (2), but (2), number (2), life (2), family (2), even (2), giving (2), apple (2), lead (2), scientists (2), alumni (2), website (2), available (2), contains (2), over (2), facs (2), images (2), paper (2), dog (2), next (2), academy (2), engineering (2), 2007 (2), partner (2), contributions (2), ahrc (2), 2015 (2), environments (2), 2022 (2), horizon (2), myworld (2), 45m (2), core (2), awards (2), devices (2), egocentric (2), embodiment (2), sport (2), markerless (2), could (2), pipelines (2), games (2), ideas (2), applied (2), same (2), modern (2), areas (2), early (2), speech (2), today (2), broader (2), drive (2), spans (2), end (2), became (2), foundations (2), use (2), google (2), scholar (2), within (2), aardman (2), experience (2), production (2), ship (2), fortunate (2), winning (2), themes (2), partnerships (2), around (2), health (2), environment (2), meetings (2), focused (2), support (2), collaborative (2), bringing (2), problems (2), included (2), shipping (2), quest (2), tracking (2), audio (2), new (2), agentic (2), context (2), chat (2), agent (2), examples (2), azure (2), connected (2), services (2), settings (2), strategy (2), toward (2), move (2), industrial (2), world (2), kind (2), which (2), visual (2), effects (2), fellow (2), before (2), founded (2), principal (2), manager (2), design, updated, original, templated, rights, reserved, love, thing, anything, having, motivates, desire, make, sure, best, forces, efficient, productive, appreciate, martin, parsons, murray, evans, yiguo, qiao, living, ruh, innovateuk, jack, saunders, george, fletcher, jake, deane, kyle, reed, cubic, jose, serra, domain, ilm, anamaria, ciucanu, mmu, pedro, mendes, shridhar, ravikumar, amazon, alastair, barber, foundry, wenbin, han, gong, charalampos, koniaris, disney, daniel, beale, sinan, mutlu, framestore, nicholas, swafford, nadejda, roubtsova, sinead, kearney, maryam, naghizadeh, catherine, taylor, marshmallow, laser, feast, talented, former, had, privilege, outstanding, students, researchers, staff, collaborators, ground, truth, provides, benchmark, single, image, enabling, open, quantitative, comparison, challenging, range, cases, here, 500, coded, sequences, individuals, meshes, stereo, maps, colour, calibration, files, information, iccv, 2011, valid, action, unit, database, morphable, multiview, rgb, dogs, performing, different, actions, get, code, view, cvpr, github, public, exploiting, techniques, 460, 640, 100, 887, 998, 728, additional, harpc, hmc, augmented, 119, 025, 517, 616, dove, deformable, objects, virtual, 128, 746, 562, 559, 2017, bristol, creative, cluster, rheumatoid, arthritis, flare, profiler, 165, 126, 663, 290, tool, reveal, individual, differences, 402, 113, medical, council, innovation, 901, 391, 151, 614, 2025, ukri, 2026, activities, ambitious, programmes, selection, supported, large, scale, centres, targeted, explored, head, mounted, viewpoints, supports, convincing, cameras, hmds, motivations, apply, state, art, elite, worked, closely, coaches, athletes, british, skeleton, create, provide, biomechanical, related, continue, form, embodied, render, realistic, labels, animal, principles, now, extend, naturally, loops, quality, touched, several, during, phd, recent, personalised, connects, combine, behaviour, shape, monocular, retargeting, rigged, automated, transfer, later, flame, max, planck, reflects, spanning, faces, bodies, animals, always, measuring, generating, expanded, list, publications, found, adaptation, satore, doubles, performers, animated, them, cannes, lion, winner, cosmos, bbc, rigging, solutions, anna, bandai, namco, game, bafta, nominated, memories, retold, framework, deploying, delivering, clients, technology, practice, helped, small, snapshot, enough, contribute, award, capability, photogrammetry, multi, disciplinary, three, converging, synthesis, enhancement, rehabilitation, each, partnered, routes, allowing, translate, enables, spaces, control, richer, established, expertise, solve, hololens, meta, reconstruction, sparse, headset, increasingly, comes, advances, communicate, coordinate, behaviours, includes, aimed, keep, track, structured, assistance, follow, shared, facilitator, reflect, hands, millions, customers, tooling, required, evaluate, pulled, further, question, become, reliably, once, they, leave, demo, stage, enter, every, day, especially, must, under, latency, constraints, help, forward, rather, than, only, reacting, prompt, pattern, academic, partners, raising, substantial, hiring, growing, interdisciplinary, infrastructure, something, both, strong, remains, clearest, institution, long, most, enjoy, meant, shift, setting, direction, testing, trained, not, just, algorithms, defining, establishing, mentoring, experimentation, workflows, organizations, prototypes, robust, customer, delivery, career, far, interests, spanned, heart, solving, greatest, possible, outside, grounded, intersection, many, years, digitisation, started, deeper, self, play, directly, scenarios, particularly, science, joined, still, hold, position, between, held, dneg, previously, double, negative, oscar, london, vfx, initial, person, facilities, first, secure, investment, south, west, 20m, led, remote, contributed, shipped, compelling, windows, orgs, cambridge, currently, involves, proactively, interact, voice, learn, adapt, user, preferences, improvement, practical, challenges, making, style, optimization, overview, navigate, darrencosker, hotmail, com, twitter, linkedin, contact, prof, home,

Text of the page (random words):
darren cosker s home page prof darren cosker principal scientist manager director microsoft professor university of bath royal society industry fellow google scholar page contact linkedin or x twitter darrencosker hotmail com personal navigate overview leadership previous product impact previous research impact funding data alumni personal i am a principal scientist manager director at microsoft based in cambridge uk where i currently lead work on post training multimodal conversational ai evaluation systems and meeting intelligence for microsoft teams this involves training systems to proactively interact with people in real time through chat video and voice and to learn and adapt to user preferences using reinforcement learning style optimization and feedback driven improvement as well as the practical challenges of making agentic systems useful in real products before this i was part of the microsoft mesh and mixed reality orgs where i founded and led the motion lab team working on real time understanding and generation of human motion for remote communication in 2d and mixed reality our work contributed to shipped experiences across teams microsoft mesh and azure connected services including audio driven avatars face tracking body pose estimation and technologies for more compelling presence in immersive communication including shipping on quest and windows up to 2021 i was a full time professor of computer science at the university of bath where i joined as a royal academy of engineering epsrc research fellow in 2007 and where i still hold a part time position between 2012 and 2016 i also held a royal society fellowship in collaboration with dneg previously double negative visual effects a multiple oscar winning london based vfx house in 2015 i founded and became director of the centre for the analysis of motion entertainment research and applications camera building it from an initial vision into a 50 person centre with major facilities industrial partnerships a first of it s kind research to impact model and over 20m in funding and partner contributions before helping secure the 45m myworld investment in the south west of the uk in my career so far my interests have spanned multiple areas an applied scientist at heart i m interested in solving problems with real value and which have the greatest impact possible in the world outside the lab grounded at the intersection of ai computer vision and computer graphics i focused for many years on multimodal human understanding with applications in visual effects human digitisation and motion capture more recently i have started to go deeper into reinforcement learning interested in how self play and human feedback can directly improve models and proactive agents in multiple scenarios particularly real time multimodal ai post training real time communication digital humans computer vision leadership and ai delivery a major part of my work today is not just creating new models or algorithms but building the environment in which ai can be delivered well defining product strategy establishing teams mentoring scientists and engineers creating evaluation and experimentation workflows and helping organizations move from prototypes to robust customer facing systems in teams this has meant helping drive the shift toward multimodal agents post trained ai systems and proactive collaboration experiences setting technical direction for conversational ai and building the foundations for synthetic data generation model evaluation and live testing using human and ai feedback the same leadership pattern was central to camera creating the vision bringing together academic and industrial partners raising substantial funding hiring and growing an interdisciplinary team building studios and technical infrastructure and turning the centre into something that delivered both strong research and real world impact that remains one of the clearest examples of the kind of institution building and long horizon technical leadership i most enjoy it has also pulled my work further toward reinforcement learning post training and the broader question of how modern ai systems become reliably useful once they leave the demo stage and enter products that people use every day i am especially interested in real time settings where models must work under latency context and interaction constraints and in proactive agents that can help move collaboration forward rather than only reacting to a prompt previous product impact from multimodal ai to immersive experiences the examples below reflect previous product work i have been fortunate to ship into the hands of millions of customers through microsoft teams microsoft mesh and azure connected services more recently this has included multimodal conversational ai post training strategy agent systems and the tooling required to evaluate and improve these systems in live product settings within teams this includes work around collaborative ai experiences such as facilitator a chat based agent experience aimed at helping people keep meetings and collaboration on track through structured assistance follow up and shared context at microsoft teams the product impact increasingly comes from turning advances in ai into tools that improve how people communicate and coordinate that spans meeting intelligence multimodal interaction and new agentic behaviours that support real time collaboration microsoft mesh i established the motion lab team bringing together expertise in computer vision generative ai graphics and animation to solve problems in real time human understanding this included shipping live audio driven avatars real time face tracking and technologies for full body reconstruction from sparse headset based signals for devices such as hololens 2 and meta quest microsoft mesh enables meetings in 2d through teams and in immersive 3d spaces through vr my work across mesh and teams has focused on the ai systems that support presence communication avatar control meeting intelligence and richer collaborative experiences camera at camera i created a multi disciplinary research and production environment with three themes converging on human perception analysis and synthesis entertainment human performance enhancement and health and rehabilitation each theme was partnered with industry giving us real routes to impact and allowing us to translate research into tools studios and delivered projects as director of camera i built research themes partnerships teams and studio capability around motion capture photogrammetry digital humans and ai for entertainment sport and health applications at camera i also created a framework for turning research into production tools with a team of engineers deploying these in our studio and delivering projects to clients that used the technology in practice this helped ship a number of video games and award winning immersive experiences below is a small snapshot of products and experiences i have been fortunate enough to contribute to 11 11 memories retold with aardman and bandai namco bafta nominated we delivered motion capture for the video game at the camera studio is anna ok with bbc and aardman an immersive experience delivered with our in house facial rigging animation and motion capture solutions cosmos within us with satore studios cannes lion winner we built digital doubles for performers and animated them using our in house tools previous research impact from digital humans to biomechanics and human perception the research below reflects previous work spanning digital humans motion capture of faces bodies and animals animation computer vision biomechanics and human perception a central theme has always been motion understanding it measuring it generating it and building systems that use it well more recently this work expanded into multimodal ai systems synthetic data evaluation foundation model adaptation reinforcement learning and proactive agents for communication and immersive experiences a list of publications can be found on my google scholar page facial performance capture and animation this spans early 4d facial capture monocular facial capture and performance retargeting from motion capture we built end to end pipelines for rigged digital facial models automated performance transfer and dynamic face modelling the d3dfacs dataset later became one of the foundations of the flame model from max planck generative models of shape and motion my work has touched several areas of generative ai from early speech driven facial animation during my phd through to recent personalised animation and multimodal avatar work today this broader theme also connects to product facing work on multimodal ai post training and systems that combine speech vision and interaction signals to drive useful behaviour in real products synthetic data for ai and computer vision computer graphics can be used to render realistic images and labels for training ai systems we have applied these ideas across pose estimation digital humans and animal motion and the same principles now extend naturally to modern multimodal systems where data generation evaluation and feedback loops are central to product quality digital humans and avatars animation embodiment and perception in camera we created pipelines for digital humans that could be used for video games vr research and immersive experiences at microsoft related ideas continue in more product facing form through avatars presence technologies and systems for embodied communication in teams and mesh markerless motion capture and analysis for biomechanics one of the core motivations of camera was to apply state of the art computer vision and ai to biomechanics and elite sport we worked closely with coaches and athletes including the british skeleton team to create markerless systems that could provide useful biomechanical signals in real training environments human pose estimation from egocentric cameras and hmds at microsoft we have explored 3d human pose estimation from head mounted devices and egocentric viewpoints including work that supports avatar embodiment and full body understanding in immersive products these technologies are central to convincing presence experiences in mixed reality research funding and awards as a professor one of the core activities in building ambitious research programmes is funding below is a selection of the major awards that supported large scale centres such as camera as well as more targeted projects in ai computer vision digital humans performance capture and perception 2021 2026 myworld 45m fec ukri pi university of bath 2020 2025 camera 2 0 centre for the analysis of motion entertainment research and applications 4 151 614 fec epsrc 2019 2021 camera motion capture innovation studio 901 391 horizon 2020 2019 2022 a tool to reveal individual differences in facial perception 402 113 medical research council 2018 2020 rheumatoid arthritis flare profiler 165 126 total project value 663 290 innovate uk 2018 2022 bristol and bath creative cluster 4m ahrc 2017 2019 dove deformable objects for virtual environments 128 746 total project value 562 559 fec innovate uk 2016 2018 harpc hmc for augmented reality performance capture 119 025 total project value 517 616 fec innovate uk 2015 2020 camera 4 998 728 fec 5m additional partner contributions epsrc ahrc 2012 2016 next generation facial capture and animation 100 887 fec royal society industry fellowship 2007 2012 exploiting 4d data for creating next generation facial modelling and animation techniques 460 640 fec royal academy of engineering research fellowship public data rgbd dog rgbd dog contains motion capture and multiview rgb and rgbd data for dogs performing different actions you can get the data code to view it and the cvpr 2020 paper from our github page d3dfacs d3dfacs contains over 500 facs coded dynamic 3d sequences from 10 individuals including meshes stereo uv maps colour images and calibration files more information is available in our iccv 2011 paper a facs valid 3d dynamic action unit database with applications to 3d dynamic morphable facial modelling and on the dataset website shadow removal shadow removal ground truth and evaluation provides a benchmark and dataset for single image shadow removal enabling open quantitative comparison across a challenging range of cases the evaluation website is available here alumni university at microsoft i lead teams of talented scientists and engineers as a professor and former director of camera i have also had the privilege of working with outstanding students researchers technical staff and collaborators martin parsons camera murray evans camera yiguo qiao living with ruh innovateuk jack saunders george fletcher jake deane kyle reed cubic motion jose serra digital domain ilm anamaria ciucanu mmu pedro mendes shridhar ravikumar amazon apple alastair barber the foundry wenbin li bath han gong apple charalampos koniaris disney research daniel beale sinan mutlu framestore nicholas swafford nadejda roubtsova camera sinead kearney camera maryam naghizadeh catherine taylor marshmallow laser feast personal i love my work but the number one thing in my life is my family if anything having a family motivates me even more in my work giving me the desire to make sure we all have the best life it also forces you to be efficient and productive with the time that you are working and to appreciate the time you have together even more darren cosker all rights reserved design updated from the original templated foundation