Meta tags:

Headings (most frequently used words):

learning, self, supervised, contrastive, contents, pseudo, labels, types, comparison, with, other, forms, of, machine, examples, references, further, reading, external, links, autoassociative, non, joint, embedding, and, predictive, architectures,

Text of the page (most frequently used words):
#learning (119), the (93), #supervised (57), self (56), and (43), data (32), neural (27), that (25), for (25), model (21), machine (20), from (18), network (18), doi (18), arxiv (17), with (16), training (16), this (15), edit (15), ieee (14), conference (14), contrastive (14), input (14), are (13), iccv (13), representation (12), used (12), vision (11), deep (11), task (11), computer (10), models (10), s2cid (10), international (10), 2019 (10), using (9), lecun (9), language (9), image (9), artificial (9), latent (9), pseudo (9), examples (9), labels (9), wikipedia (8), text (8), visual (8), yann (8), intelligence (8), embedding (8), proceedings (8), unsupervised (8), 978 (8), isbn (8), 1109 (8), joint (8), can (8), networks (8), example (8), other (8), positive (8), retrieved (7), based (7), systems (7), recognition (7), context (7), images (7), processing (7), predictive (7), semi (7), learn (7), where (7), may (6), non (6), autoencoder (6), architectures (6), speech (6), human (6), loss (6), 2015 (6), 2017 (6), 2021 (6), analysis (6), which (6), not (6), negative (6), classification (6), displaystyle (6), left (6), right (6), search (5), page (5), generative (5), architecture (5), andrew (5), reasoning (5), automated (5), approach (5), gradient (5), reinforcement (5), method (5), regression (5), 2018 (5), pmid (5), issn (5), doersch (5), carl (5), october (5), 2023 (5), 2020 (5), june (5), representations (5), jepa (5), noise (5), estimation (5), random (5), autoassociative (5), machines (5), its (5), ncssl (5), ssl (5), sample (5), pairs (5), contents (4), policy (4), use (4), articles (4), technical (4), july (4), convolutional (4), state (4), term (4), john (4), knowledge (4), large (4), world (4), diffusion (4), autonomous (4), word (4), engineering (4), association (4), computational (4), 2022 (4), wang (4), april (4), micron (4), prediction (4), links (4), bardes (4), adrien (4), own (4), new (4), main (4), shot (4), transfer (4), path (4), statistical (4), facebook (4), concept (4), uses (4), labeled (4), encoder (4), create (4), process (4), output (4), such (4), between (4), types (4), log (4), vector (4), hide (4), move (4), sidebar (4), toggle (3), view (3), about (3), privacy (3), september (3), january (3), too (3), 2025 (3), short (3), wikidata (3), art (3), video (3), recurrent (3), transformer (3), alex (3), fei (3), semantic (3), list (3), audio (3), general (3), agent (3), algorithm (3), symbolic (3), open (3), datasets (3), descent (3), clustering (3), variance (3), history (3), linguistics (3), november (3), methods (3), robust (3), 4673 (3), multi (3), quentin (3), program (3), repair (3), diagnostics (3), 1038 (3), jean (3), emnlp (3), natural (3), supervision (3), few (3), via (3), 2016 (3), cvpr (3), pattern (3), features (3), multimodal (3), reduction (3), 2013 (3), correlation (3), detection (3), drift (3), through (3), humans (3), unlabeled (3), trained (3), instead (3), small (3), learns (3), two (3), only (3), way (3), itself (3), however (3), pretext (3), unlike (3), one (3), requiring (3), does (3), these (3), supervisory (3), signals (3), then (3), autoencoders (3), space (3), rather (3), than (3), has (3), while (3), without (3), would (3), samples (3), function (3), pair (3), when (3), help (3), related (3), field (3), article (3), tools (3), languages (2), table (2), statistics (2), safety (2), contact (2), under (2), terms (2), was (2), last (2), 2026 (2), hidden (2), categories (2), 2024 (2), description (2), different (2), category (2), impact (2), applications (2), principle (2), act (2), gan (2), mamba (2), rnn (2), perceptron (2), gru (2), long (2), lstm (2), turing (2), daniel (2), jan (2), david (2), oriol (2), vinyals (2), yoshua (2), bengio (2), geoffrey (2), hinton (2), oliver (2), rule (2), openai (2), ibm (2), gpt (2), synthesis (2), alexnet (2), protocol (2), neuro (2), source (2), coding (2), improvement (2), rlhf (2), sarsa (2), augmentation (2), regularization (2), bias (2), tradeoff (2), software (2), projects (2), algorithms (2), glossary (2), yarowsky (2), sense (2), zheng (2), xin (2), yong (2), guoyou (2), liu (2), jianguo (2), 3796689 (2), 29425969 (2), 0968 (2), 4328 (2), 1016 (2), 010 (2), 107 (2), fast (2), segmentation (2), white (2), blood (2), cell (2), gupta (2), abhinav (2), efros (2), alexei (2), december (2), 1430 (2), 9062671 (2), 8391 (2), 167 (2), 1505 (2), 05192 (2), 1422 (2), zisserman (2), 2079 (2), 473729 (2), 5386 (2), 1032 (2), 226 (2), 1708 (2), 07860 (2), 2070 (2), external (2), mark (2), florian (2), garrido (2), pierre (2), further (2), reading (2), martin (2), 1145 (2), acm (2), selfapr (2), test (2), execution (2), pmc (2), efficient (2), genomics (2), bootstrap (2), your (2), 2010 (2), empirical (2), google (2), bert (2), pre (2), vincent (2), abstract (2), aaai (2), ponce (2), motion (2), content (2), assran (2), mahmoud (2), ballas (2), nicolas (2), misra (2), ishan (2), michael (2), towards (2), nature (2), variable (2), modelling (2), vicreg (2), invariance (2), covariance (2), pmlr (2), barlow (2), twins (2), canonical (2), jmlr (2), workshop (2), 1476 (2), 355 (2), bibcode (2), com (2), pdf (2), system (2), computing (2), adaptation (2), icml (2), label (2), wav2vec (2), perez (2), 7281 (2), 4803 (2), cvf (2), boosting (2), what (2), like (2), references (2), across (2), diverse (2), fields (2), leverage (2), application (2), domains (2), applied (2), version (2), predictor (2), being (2), given (2), produced (2), results (2), understand (2), particularly (2), each (2), most (2), important (2), information (2), reconstruction (2), current (2), often (2), tasks (2), involves (2), designed (2), require (2), inherent (2), structures (2), generate (2), same (2), time (2), extracted (2), comparison (2), forms (2), jepas (2), structure (2), decision (2), aims (2), they (2), class (2), maximizing (2), agreement (2), constraints (2), cca (2), enforce (2), local (2), zero (2), binary (2), classify (2), requires (2), online (2), side (2), target (2), encoding (2), dimensional (2), into (2), classifier (2), birds (2), distance (2), reconstruct (2), closely (2), during (2), typically (2), difference (2), original (2), error (2), meaningful (2), mean (2), essential (2), predictions (2), risk (2), components (2), manual (2), labeling (2), instance (2), predicted (2), ground (2), truth (2), make (2), recent (2), paradigm (2), relationships (2), theory (2), curve (2), net (2), forest (2), factor (2), anomaly (2), bayes (2), structured (2), dimensionality (2), feature (2), appearance (2), upload (2), file (2), changes (2), read (2), account (2), donate (2), menu (2), add, topic, mobile, cookie, statement, developers, code, conduct, legal, contacts, disclaimers, available, additional, apply, site, you, agree, registered, trademark, profit, organization, wikimedia, foundation, inc, creative, commons, attribution, sharealike, license, rendered, parsoid, edited, utc, dmy, dates, all, https, org, index, php, title, supervised_learning, oldid, 1375949950, workplace, warfare, military, games, marketing, chatbot, psychosis, healthcare, fiction, education, engine, optimization, explainable, environmental, competition, arms, race, anthropomorphism, winter, slop, literacy, infrastructure, effect, center, bubble, boom, social, economic, opposition, centers, propaganda, virtual, politician, regulation, precautionary, nationalism, ethics, elections, takeover, alignment, government, cold, war, political, graph, gnn, adversarial, variational, vae, highway, residual, cnn, multilayer, mlp, echo, gated, unit, memory, vit, differentiable, françois, chollet, kokotajlo, leike, mustafa, suleyman, schulman, aidan, gomez, noam, shazeer, ashish, vaswani, andrej, karpathy, silver, demis, hassabis, ian, goodfellow, quoc, ilya, sutskever, krizhevsky, james, goodnight, graves, stephen, grossberg, lotfi, zadeh, jürgen, schmidhuber, hopfield, paul, werbos, seppo, linnainmaa, seymour, papert, joseph, weizenbaum, bernard, widrow, frank, rosenblatt, selfridge, herbert, simon, cliff, shaw, allen, newell, nathaniel, rochester, mccarthy, marvin, minsky, takeo, kanade, kunihiko, fukushima, shun, ichi, amari, claude, shannon, christopher, manning, von, neumann, walter, pitts, warren, sturgis, mcculloch, alan, people, yago, dbpedia, conceptnet, bases, opencog, lida, clarion, soar, cognitive, reasoners, procedural, logic, programs, inference, engines, expert, deductive, classifiers, robot, control, autogpt, action, selection, muzero, driving, car, five, alphazero, alphago, decisional, watsonx, watson, project, debater, oasis, genie, udio, suno, riffusion, music, generation, veo, seedance, sora, kling, hailuo, runway, gen, dream, stable, recraft, midjourney, imagen, ideogram, flux, firefly, dall, aurora, alphafold, facial, whisper, elevenlabs, ocr, hwr, wavenet, implementations, physical, agent2agent, hypothetical, superintelligence, asi, agi, weak, lethal, weapons, laws, humanity, exam, companion, intelligent, nmt, game, playing, theorem, proving, actor, critic, situated, sovereign, blended, vibe, hallucination, recursive, reflection, llm, post, uncanny, valley, rag, adversary, autoregression, imitation, prompt, weight, initialization, gating, rectifier, sigmoid, softmax, activation, batchnorm, normalization, convolution, attention, backpropagation, conjugate, quasi, newton, sgd, overfitting, double, functions, hyperparameter, parameter, constraint, satisfaction, planning, concepts, lists, proprietary, institutions, companies, timeline, 1995, cambridge, 196, 3115, 981658, 981684, 189, 33rd, annual, meeting, disambiguation, rivaling, balestriero, randall, ibrahim, sobal, vlad, morcos, ari, shekhar, shashank, goldstein, tom, bordes, mialon, gregoire, tian, yuandong, schwarzschild, avi, wilson, gordon, geiping, jonas, fernandez, cookbook, 2304, 12210, martinez, matias, luo, xiapu, zhang, tao, monperrus, 4503, 9475, 3551349, 3556926, 37th, gündüz, hüseyin, anil, binder, xiao, yin, mreches, rené, bischl, bernd, mchardy, alice, münch, philipp, rezaei, mina, 928, 37696966, 10495322, 2399, 3642, s42003, 023, 05310, communications, biology, grill, bastien, strub, altché, florent, tallec, corentin, richemond, buchatskaya, elena, pires, bernardo, avila, guo, zhaohan, azar, mohammad, gheshlaghi, piot, bilal, 2006, 07733, wilcox, ethan, qian, peng, futrell, richard, kohita, ryosuke, levy, roger, ballesteros, miguel, stroudsburg, usa, 4652, 222291675, 18653, 375, 05725, 4640, structural, improves, syntactic, generalization, blog, sourcing, francois, lavet, precup, doina, pineau, joelle, combined, 1809, 04506, littwin, etai, wolf, lior, 3966, 6517610, 8851, 429, 1511, 09033, 3957, multiverse, 2307, 12698, zhengcong, fan, mingyuan, huang, junshi, listen, 2311, 15830, najman, laurent, leveraging, 2403, 00504, duval, bojanowski, piotr, pascal, rabbat, 2301, 08243, openreview, ing, andrades, alvaro, cosenza, marco, raffaele, korbel, 1075, 40709098, 12283373, 2522, 5839, s42256, 025, 01052, 1053, integrating, cancer, 2105, 04906v3, zbontar, jure, jing, deny, stephane, 12320, 12310, 38th, redundancy, galen, arora, raman, bilmes, jeff, livescu, karen, 1255, 1247, 30th, gutmann, hyvärinen, aapo, march, 304, 297, thirteenth, unnormalized, oord, aaron, van, den, yazhe, 1807, 03748, becker, suzanna, 1992, 6356, 163, 1729650, 4687, 355161a0, 1992natur, 161b, 161, organizing, discovers, surfaces, dot, stereograms, demystifying, key, technique, kramer, 1991, 243, 1002, aic, 690370209, 1991aiche, 233k, 233, aiche, journal, nonlinear, principal, component, deng, chen, european, symposium, security, euros, eurosp, 00014, evolving, android, malware, gama, joão, 2014, 2523813, surveys, survey, lee, dong, hyun, challenges, wrepl, simple, gidaris, spyros, bursuc, andrei, komodakis, nikos, patrick, cord, matthieu, 8067, 186206588, 00815, 1906, 05186, 8058, beyer, lucas, zhai, xiaohua, avital, kolesnikov, alexander, 1485, 167209887, 00156, 1905, 03670, s4l, bouchard, louis, medium, will, ever, able, continues, gain, prominence, ability, effectively, opens, possibilities, advancement, especially, driven, perturbations, previous, synthesize, patches, genomenet, directpred, directly, sets, weights, typical, number, predict, point, polysemous, byol, excellent, benchmarks, imagenet, autoregressive, translate, texts, answer, questions, among, things, better, queries, bidirectional, transformers, suitable, developed, perform, build, supervising, combination, losses, kept, compressed, intrinsically, constitutes, because, needs, become, optimal, jargon, refers, setup, design, case, fully, contained, reused, combines, portion, similar, done, belongs, insofar, goal, classified, explicit, correlations, metadata, embedded, present, implicitly, autonomously, domain, operate, entirely, avoiding, pixel, level, focus, just, predicting, masked, visible, been, introduced, step, making, including, founded, included, enable, replicate, intellect, providing, exist, major, moves, beyond, views, preventing, collapse, rooted, includes, jea, invariant, sampling, dlvpm, generalizes, orthogonality, counterintuitively, converges, useful, minimum, reaching, trivial, solution, trivially, effective, extra, back, propagate, mathcal, mathrm, mathbb, frac, sum, infonce, optimize, jointly, nce, set, containing, proposal, distribution, minimizes, following, mid, ldots, clip, allows, pretraining, matching, have, span, angle, having, cosine, similarity, early, maximize, their, divided, those, match, identify, include, contain, both, minimize, presenting, possible, penalizes, reconstructed, minimizing, squared, comes, fact, essentially, associating, achieved, type, consist, maps, lower, reconstructs, decoder, specific, reproduce, words, tasked, captures, allowing, regenerate, many, pipelines, chosen, produces, sufficiently, confident, reducing, propagating, errors, instances, incorporated, refresh, evolve, understanding, emerging, patterns, existing, show, signs, aging, due, distributional, shifts, strategy, reduces, reliance, helping, maintain, performance, adaptive, also, plays, role, must, adapt, properties, change, over, scenarios, detect, incoming, deviates, previously, learned, behavior, generates, result, updating, retraining, becoming, outdated, enables, continuous, dynamic, environments, annotation, automatically, generated, assigns, widely, annotations, limited, unavailable, treating, surrogate, quantities, promising, years, found, practical, others, steps, first, solved, auxiliary, initialize, next, actual, performed, parameters, relying, externally, provided, within, solving, them, capturing, augmented, transformed, creates, serves, formulate, signal, involve, introducing, cropping, rotation, transformations, more, imitates, objects, outline, research, ijcai, iclr, neurips, ecml, pkdd, eccv, journals, conferences, topological, pac, occam, minimization, kernel, mathematical, foundations, roc, confusion, matrix, coefficient, determination, mechanistic, interpretability, loop, crowdsourcing, active, play, temporal, ecram, electrochemical, ram, memtransistor, spiking, physics, informed, radiance, deepdream, lenet, som, restricted, boltzmann, reservoir, esn, feedforward, isolation, outlier, ransac, markov, conditional, graphical, sdl, sne, pgd, pca, nmf, lda, ica, exploratory, shift, optics, dbscan, expectation, maximization, fuzzy, means, hierarchical, cure, birch, support, svm, relevance, rvm, logistic, naive, linear, bagging, ensembles, trees, apprenticeship, ontology, grammar, induction, rank, rules, automl, cleaning, density, modeling, problems, quantum, neuromorphic, curriculum, batch, meta, paradigms, mining, part, series, how, remove, message, please, removing, details, understandable, experts, improve, readers, confused, free, encyclopedia, item, printable, download, print, export, switch, legacy, parser, get, shortened, url, cite, permanent, link, here, actions, english, talk, tiếng, việt, українська, ไทย, português, polski, 한국어, 日本語, italiano, bahasa, indonesia, galego, français, فارسی, español, ελληνικά, deutsch, català, bosanski, العربية, subsection, top, personal, special, pages, community, portal, contribute, events, navigation, jump,

Text of the page (random words):
ticle about wikipedia contact us contribute help learn to edit community portal recent changes upload file special pages search search appearance donate create account log in personal tools donate create account log in contents move to sidebar hide top 1 pseudo labels 2 types toggle types subsection 2 1 autoassociative self supervised learning 2 2 contrastive self supervised learning 2 3 non contrastive self supervised learning 2 4 joint embedding and predictive architectures 3 comparison with other forms of machine learning 4 examples 5 references 6 further reading 7 external links toggle the table of contents self supervised learning 20 languages العربية bosanski català deutsch ελληνικά español فارسی français galego bahasa indonesia italiano 日本語 한국어 polski português ไทย українська tiếng việt 粵語 中文 edit links article talk english read edit view history tools tools move to sidebar hide actions read edit view history general what links here related changes upload file permanent link page information cite this page get shortened url switch to legacy parser print export download as pdf printable version in other projects wikidata item appearance move to sidebar hide from wikipedia the free encyclopedia machine learning paradigm not to be confused with semi supervised learning this article may be too technical for most readers to understand please help improve it to make it understandable to non experts without removing the technical details july 2025 learn how and when to remove this message part of a series on machine learning and data mining paradigms supervised learning unsupervised learning semi supervised learning self supervised learning reinforcement learning transfer learning meta learning few shot learning zero shot learning online learning batch learning curriculum learning rule based learning neuro symbolic ai neuromorphic engineering quantum machine learning problems classification generative modeling regression clustering dimensionality reduction density estimation anomaly detection data cleaning automl association rules semantic analysis structured prediction feature engineering feature learning learning to rank grammar induction ontology learning multimodal learning supervised learning classification regression apprenticeship learning decision trees ensembles bagging boosting random forest k nn linear regression naive bayes artificial neural networks logistic regression perceptron relevance vector machine rvm support vector machine svm clustering birch cure hierarchical k means fuzzy expectation maximization em dbscan optics mean shift dimensionality reduction factor analysis exploratory cca ica lda nmf pca pgd t sne sdl structured prediction graphical models bayes net conditional random field hidden markov anomaly detection ransac k nn local outlier factor isolation forest neural networks autoencoder deep learning feedforward neural network recurrent neural network lstm gru esn reservoir computing boltzmann machine restricted gan diffusion model som convolutional neural network u net lenet alexnet deepdream neural field neural radiance field physics informed neural networks transformer vision mamba spiking neural network memtransistor electrochemical ram ecram reinforcement learning q learning policy gradient sarsa temporal difference td multi agent self play learning with humans active learning crowdsourcing human in the loop mechanistic interpretability rlhf model diagnostics coefficient of determination confusion matrix learning curve roc curve mathematical foundations kernel machines bias variance tradeoff computational learning theory empirical risk minimization occam learning pac learning statistical learning vc theory topological deep learning journals and conferences aaai cvpr eccv ecml pkdd emnlp iccv neurips icml iclr ijcai ml jmlr related articles glossary of artificial intelligence list of datasets for machine learning research list of datasets in computer vision and image processing outline of machine learning v t e self supervised learning ssl is a paradigm in machine learning where a model is trained on a task using the data itself to generate supervisory signals rather than relying on externally provided labels in the context of neural networks self supervised learning aims to leverage inherent structures or relationships within the input data to create meaningful training signals ssl tasks are designed so that solving them requires capturing essential features or relationships in the data the input data is typically augmented or transformed in a way that creates pairs of related samples where one sample serves as the input and the other is used to formulate the supervisory signal this augmentation can involve introducing noise cropping rotation or other transformations self supervised learning more closely imitates the way humans learn to classify objects 1 during ssl the model learns in two steps first the task is solved based on an auxiliary or pretext classification task using pseudo labels which help to initialize the model parameters 2 3 next the actual task is performed with supervised or unsupervised learning 4 5 6 self supervised learning has produced promising results in recent years and has found practical application in fields such as audio processing and is being used by facebook and others for speech recognition 7 pseudo labels edit pseudo labels are automatically generated labels that a model assigns to unlabeled data based on its own predictions they are widely used in self supervised and semi supervised learning where ground truth annotations are limited or unavailable by treating predicted labels as surrogate ground truth learning algorithms can make use of large quantities of unlabeled data in the training process 8 pseudo labeling also plays an important role in systems that must adapt to concept drift where the statistical properties of the data change over time in these scenarios the model may detect that an incoming instance deviates from previously learned behavior the system then generates a classification result for that instance and this predicted class is used as a pseudo label for updating or retraining model components that are becoming outdated this approach enables continuous adaptation in dynamic environments without requiring manual annotation 9 10 in many adaptive learning pipelines pseudo labels are chosen when the classifier produces sufficiently confident predictions reducing the risk of propagating errors these pseudo labeled instances are then incorporated into training to refresh or evolve the model s understanding of emerging data patterns particularly when existing components show signs of aging due to drift or distributional shifts this strategy reduces reliance on manual labeling while helping maintain long term model performance types edit autoassociative self supervised learning edit autoassociative self supervised learning is a specific category of self supervised learning where a neural network is trained to reproduce or reconstruct its own input data 11 in other words the model is tasked with learning a representation of the data that captures its essential features or structure allowing it to regenerate the original input the term autoassociative comes from the fact that the model is essentially associating the input data with itself this is often achieved using autoencoders which are a type of neural network architecture used for representation learning autoencoders consist of an encoder network that maps the input data to a lower dimensional representation latent space and a decoder network that reconstructs the input from this representation the training process involves presenting the model with input data and requiring it to reconstruct the same data as closely as possible the loss function used during training typically penalizes the difference between the original input and the reconstructed output e g mean squared error by minimizing this reconstruction error the autoencoder learns a meaningful representation of the data in its latent space contrastive self supervised learning edit for a binary classification task training data can be divided into positive examples and negative examples positive examples are those that match the target for example if training a classifier to identify birds the positive training data would include images that contain birds negative examples would be images that do not 12 contrastive self supervised learning uses both positive and negative examples the loss function in contrastive learning is used to minimize the distance between positive sample pairs while maximizing the distance between negative sample pairs 12 an early example uses a pair of 1 dimensional convolutional neural networks to process a pair of images and maximize their agreement 13 contrastive language image pre training clip allows joint pretraining of a text encoder and an image encoder such that a matching image text pair have image encoding vector and text encoding vector that span a small angle having a large cosine similarity infonce noise contrastive estimation 14 is a method to optimize two models jointly based on noise contrastive estimation nce 15 given a set x x 1 x n displaystyle x left x_ 1 ldots x_ n right of n displaystyle n random samples containing one positive sample from p x t k c t displaystyle p left x_ t k mid c_ t right and n 1 displaystyle n 1 negative samples from the proposal distribution p x t k displaystyle p left x_ t k right it minimizes the following loss function l n e x log f k x t k c t x j x f k x j c t displaystyle mathcal l _ mathrm n mathbb e _ x left log frac f_ k left x_ t k c_ t right sum _ x_ j in x f_ k left x_ j c_ t right right non contrastive self supervised learning edit non contrastive self supervised learning ncssl uses only positive examples counterintuitively ncssl converges on a useful local minimum rather than reaching a trivial solution with zero loss for the example of binary classification it would trivially learn to classify each example as positive effective ncssl requires an extra predictor on the online side that does not back propagate on the target side 12 joint embedding and predictive architectures edit a major class of self supervised learning moves beyond contrastive pairs instead maximizing the agreement between views while preventing collapse through statistical constraints rooted in deep canonical correlation analysis deep cca 16 this approach includes joint embedding architectures jea like barlow twins 17 and vicreg 18 which enforce covariance constraints to learn invariant representations without negative sampling deep latent variable path modelling dlvpm 19 generalizes this to multimodal systems using path models to enforce correlation and orthogonality across diverse data types in 2022 yann lecun introduced joint embedding predictive architectures jepa as a step towards decision making reasoning and autonomous human intelligence in machines including self improvement through autonomous learning founded in representation learning lecun included the concept of a world model in jepa which aims to enable machines to replicate human intellect by providing machines with a concept for the world in which they exist 20 unlike autoencoders jepas operate entirely in latent space avoiding pixel level noise to focus on semantic structure rather than just learning invariance jepas learn by predicting masked latent representations from visible context 21 jepa has been applied to domains such as image analysis 22 audio processing 23 and motion in images and video 24 comparison with other forms of machine learning edit ssl belongs to supervised learning methods insofar as the goal is to generate a classified output from the input at the same time however it does not require the explicit use of labeled input output pairs instead correlations metadata embedded in the data or domain knowledge present in the input are implicitly and autonomously extracted from the data these supervisory signals extracted from the data can then be used for training 1 ssl is similar to unsupervised learning in that it does not require labels in the sample data unlike unsupervised learning however learning is not done using inherent data structures semi supervised learning combines supervised and unsupervised learning requiring only a small portion of the learning data be labeled 3 in transfer learning a model designed for one task is reused on a different task 25 training an autoencoder intrinsically constitutes a self supervised process because the output pattern needs to become an optimal reconstruction of the input pattern itself however in current jargon the term self supervised often refers to tasks based on a pretext task training setup this involves the human design of such pretext task s unlike the case of fully self contained autoencoder training 11 in reinforcement learning self supervising learning from a combination of losses can create abstract representations where only the most important information about the state are kept in a compressed way 26 examples edit self supervised learning is particularly suitable for speech recognition for example facebook developed wav2vec a self supervised algorithm to perform speech recognition using two deep convolutional neural networks that build on each other 7 google s bidirectional encoder representations from transformers bert model is used to better understand the context of search queries 27 openai s gpt 3 is an autoregressive language model that can be used in language processing it can be used to translate texts or answer questions among other things 28 bootstrap your own latent byol is a ncssl that produced excellent results on imagenet and on transfer and semi supervised benchmarks 29 the yarowsky algorithm is an example of self supervised learning in natural language processing from a small number of labeled examples it learns to predict which word sense of a polysemous word is being used at a given point in text directpred is a ncssl that directly sets the predictor weights instead of learning it via typical gradient descent 12 self genomenet is an example of self supervised learning in genomics 30 selfapr is an example of self supervised learning applied to automated program repair where a neural model is trained on perturbations of a previous version of the program under repair and uses test execution diagnostics to synthesize patches 31 self supervised learning continues to gain prominence as a new approach across diverse fields its ability to leverage unlabeled data effectively opens new possibilities for advancement in machine learning especially in data driven application domains references edit 1 2 bouchard louis 25 november 2020 what is self supervised learning will machines ever be able to learn like humans medium retrieved 9 june 2021 doersch carl zisserman andrew october 2017 multi task self supervised visual learning 2017 ...