Meta tags:

Headings (most frequently used words):

work, coding, in, paradise, cloudless, open, source, deep, learning, pipeline, for, orbital, satellite, data, introduction, motivation, related, approach, results, future, conclusion, bibliography, what, is, planet, labs, trained, model, limitations, back, to, codinginparadise, org,

Text of the page (most frequently used words):
the (131), and (74), for (42), this (35), from (33), data (29), with (29), that (25), #planet (23), labs (23), are (22), #cloudless (22), image (22), cloud (20), detection (19), not (18), learning (18), satellite (18), training (18), can (17), deep (17), what (17), bounding (16), boxes (15), figure (15), network (14), location (14), model (13), clouds (13), pipeline (13), trained (13), into (13), example (13), imagery (12), hand (12), were (12), work (11), orbital (11), other (11), annotation (11), which (11), images (11), non (10), was (10), project (10), selective (9), search (9), accuracy (9), tool (9), approach (9), you (9), week (8), open (8), currently (8), localization (8), final (8), has (8), more (8), time (8), neural (8), about (8), then (8), these (7), visual (7), such (7), detailed (7), source (7), day (7), its (7), some (7), fairly (7), two (7), get (7), iterations (7), over (7), object (6), caffe (6), pixel (6), help (6), during (6), hack (6), one (6), future (6), computer (6), tasks (6), use (6), focused (6), would (6), done (6), validation (6), once (6), only (6), when (6), fed (6), satellites (6), see (6), details (6), much (6), them (6), san (6), francisco (6), bay (6), area (6), detected (6), around (6), stretchtext (6), note (5), their (5), api (5), vision (5), three (5), could (5), pipelines (5), run (5), will (5), have (5), how (5), well (5), different (5), etc (5), bands (5), tuning (5), mask (5), below (5), single (5), via (5), all (5), however (5), constant (5), showing (5), motion (5), blur (5), based (5), results (5), python (5), draw (5), dropbox (4), should (4), just (4), than (4), approaches (4), box (4), but (4), bootstrap (4), step (4), labelled (4), greater (4), used (4), cars (4), addition (4), each (4), same (4), important (4), also (4), hyperparameter (4), learn (4), given (4), annotated (4), generated (4), few (4), fine (4), interesting (4), difference (4), takes (4), years (4), they (4), doves (4), launched (4), orbit (4), colearning (4), blog (3), those (3), new (3), deconvolution (3), brad (3), neuberg (3), automated (3), space (3), page (3), here (3), above (3), feature (3), beyond (3), problem (3), post (3), system (3), extended (3), generally (3), finally (3), itself (3), probably (3), earlier (3), groups (3), resolution (3), class (3), sets (3), examples (3), input (3), information (3), white (3), patch (3), while (3), being (3), where (3), having (3), items (3), spectrum (3), side (3), infrared (3), thermal (3), good (3), eliminate (3), convolutional (3), output (3), classification (3), might (3), number (3), gpu (3), uses (3), been (3), though (3), performance (3), table (3), like (3), allowing (3), days (3), changes (3), rapideye (3), after (3), named (3), yellow (3), shown (3), alexnet (3), quickly (3), graph (3), downstream (3), classifier (3), candidate (3), within (3), think (3), through (3), detect (3), any (3), doing (3), daily (3), earth (3), created (3), structure (3), community (3), group (3), back (2), feed (2), segmentation (2), zoo (2), recognition (2), plcompositor (2), johann (2), hauswald (2), max (2), nova (2), making (2), github (2), strong (2), end (2), most (2), oriented (2), rather (2), increase (2), engineering (2), wide (2), range (2), part (2), competitive (2), tuned (2), even (2), free (2), train (2), gauge (2), users (2), multi (2), roads (2), able (2), themselves (2), told (2), came (2), snow (2), very (2), sure (2), looking (2), natural (2), give (2), quite (2), allow (2), take (2), out (2), make (2), general (2), clear (2), techniques (2), visible (2), spectral (2), user (2), raw (2), whether (2), thresholds (2), level (2), unfortunately (2), depends (2), always (2), yet (2), ideas (2), section (2), chosen (2), processing (2), took (2), imaged (2), task (2), does (2), reported (2), experiments (2), attempt (2), built (2), four (2), lower (2), best (2), larger (2), engineered (2), related (2), without (2), rolled (2), getting (2), clearer (2), months (2), fleet (2), false (2), positives (2), low (2), necessarily (2), amounts (2), converged (2), rest (2), present (2), runs (2), restricted (2), later (2), possible (2), overlaid (2), before (2), implementation (2), library (2), pursued (2), stacked (2), represents (2), unchanging (2), elements (2), comes (2), compare (2), changed (2), passed (2), infer (2), perhaps (2), way (2), helping (2), attention (2), know (2), spend (2), focusing (2), early (2), pieces (2), ec2 (2), pre (2), browser (2), using (2), cnns (2), recent (2), therefore (2), involves (2), many (2), surface (2), makes (2), first (2), multispectral (2), means (2), elimination (2), problems (2), detecting (2), company (2), counting (2), deforestation (2), five (2), available (2), iss (2), nanosats (2), online (2), coworking (2), ago (2), weekly (2), study (2), release (2), paper (2), inline (2), subject (2), codinginparadise, org, please, personal, views, expressed, pages, mine, alone, employer, stay, date, posts, follow, twitter, subscribe, rss, hao, crowdsourcing, annotations, hyeonwoo, noh, semantic, fork, koen, van, sande, iccv, 2011, uijlings, ijcv, 2013, frank, warmerdam, lapse, compositor, gary, jedlovec, nasa, marshall, flight, center, bibliography, special, thanks, graciously, giving, access, found, aspect, proposed, enhancements, manual, investments, improve, introduced, provides, tools, component, conclusion, mechanical, turk, necessary, taken, parameters, labellers, quality, power, lines, ocean, tankers, buildings, biomes, latitude, longitude, altitude, year, position, sun, clearly, human, identify, something, north, pole, classify, unsure, equator, tropical, forest, assign, high, probability, blury, weren, nyc, answers, knew, winter, summer, seems, features, decide, dealing, boundary, cases, essentially, bayesian, figuring, informed, guess, inferring, presence, annotates, channel, saved, integrates, become, option, computational, slowness, crafted, equation, possibility, turn, feeding, value, desired, appropriate, eliminating, kind, require, providing, generating, hyperparameters, generic, across, requires, hands, values, motivates, scheme, computationally, intensive, macbook, pro, laptop, ghz, intel, core, nvidia, minutes, terrible, parallelized, easily, since, night, tested, regions, cause, issues, schemes, limitations, augmentation, clipping, mirroring, transformations, manually, rotating, preparation, degrees, directions, aided, counterintuitively, actually, solution, confusion, matrix, steps, added, stronger, precision, recall, difficult, say, exactly, perform, dependent, threshold, unlike, solutions, variety, totalling, 4000, affect, looks, cloudy, generalize, format, started, purchased, better, involved, things, despite, still, realm, plausibility, caused, 700, discovered, small, extensive, affecting, rapidly, shows, converging, hundred, stabilizing, 000, bvlc, layers, frozen, fully, connected, layer, imagenet, softmax, reduced, thousand, classes, indicating, had, standard, split, between, hold, epochs, state, california, setups, decent, major, bolded, outputs, json, file, gives, coordinates, consumers, order, compatible, ported, increasing, loss, decreasing, portion, chops, representing, either, splits, trains, binary, version, includes, scripts, instances, amazon, explore, settings, parallel, download, query, understand, went, graphs, animated, gif, action, drawing, straightforward, draws, normalizes, processes, presents, web, objects, attempts, case, tune, generates, statistics, relate, performs, allows, nets, surprisingly, seem, fit, cnn, supervised, large, readily, exist, requiring, ourselves, consists, second, taking, stack, averaging, together, definition, remove, leaving, consistent, cities, behind, stacking, removes, stripping, away, compelling, extraction, extract, relevant, various, sometimes, season, specific, complicated, universal, except, gained, acquisition, near, depend, survey, solved, there, broad, entire, focuses, needed, tweaking, applied, ignore, nanosat, want, appearing, forests, disappearing, change, often, lead, reason, hence, name, pun, fact, originated, who, says, funny, applications, numbers, walmart, parking, lots, correlate, consumer, demand, illegal, logging, operations, meters, great, enough, publicly, forestry, suitable, full, deployed, result, flood, person, efficiently, process, computers, instead, motivation, startup, fleets, size, shoebox, eighty, called, both, international, station, secondary, payloads, commercially, hardware, placed, resulting, prices, per, rapid, iteration, known, agile, aerospace, contrast, traditional, geospatial, imaging, cost, billions, dollars, ten, develop, geosynchronous, because, wanted, thing, physical, freedom, self, education, chance, people, progress, course, writers, adults, similar, worked, hours, several, tradition, engineers, invited, guests, matter, blue, sky, born, august, 2015, contributions, myself, continued, working, bring, long, ted, nelson, screens, aren, move, simply, mimicing, crack, going, deeper, article, optionally, clicked, opened, dashed, bordered, proud, announce, powered, under, covers, contains, technical, report, introduction, coding, paradise,

Text of the page (random words):
p learning can t be funny detecting clouds in orbital satellite imagery to ignore or eliminate them is an important pre processing step to doing interesting work with nanosat imagery as we want to detect changes over time such as cars appearing forests disappearing etc being able to first detect and eliminate clouds which change often and could lead to false positives is therefore important note that even though the cloudless pipeline is currently focused on cloud detection and localization the entire pipeline focuses on what would be needed for any visual orbital detection and localization task and with tweaking can be applied to other problems related work cloud detection and elimination are by no means solved problems however there are currently two broad non deep learning approaches for cloud detection and elimination that help with the problem the first involves detailed hand rolled feature extraction pipelines that attempt to extract relevant pixel level details from infrared thermal multispectral bands etc and then use various thresholds that are sometimes location day and season specific they are complicated and not necessarily universal in addition the planet labs satellites only currently work in the visual spectrum except for a few satellites gained through a recent acquisition named rapideye which can detect in the near infrared which means the multispectral and thermal bands some of these techniques depend on are not present a good survey of these hand engineered approaches is in 1 the second approach involves taking a stack of satellite imagery of the same location over many days and averaging them together this will by definition remove the clouds from the image leaving consistent details such as cities roads surface details etc behind unfortunately it also removes much of the changes over time information stripping away what makes planet labs imagery so compelling a good example of the stacking approach is in an open source library from planet labs themselves named plcompositor 2 as a side note not pursued in this project an interesting approach might be to use such stacked images to get a constant image that represents unchanging elements of a location then when a new image from this location comes in compare it to the constant image to get a difference mask to see what has changed within this location this difference mask could then be passed downstream to other deep or non deep learning pipelines to infer what should be focused on in the image for object detection and localization perhaps one way to think of it is as helping a neural network attention model to know what to spend its time focusing on approach results convolutional neural nets cnns have shown surprisingly strong results in image classification tasks in recent years and seem to be a natural fit for this problem once trained a cnn can be used to do object localization to draw bounding boxes around candidate detected items however supervised training of cnns depends on large amounts of training data which does not readily exist for orbital satellite data requiring us to bootstrap it ourselves cloudless therefore consists of three pieces an annotation tool that takes data from the planet labs api and allows users to draw bounding boxes around clouds to bootstrap training data a training pipeline that takes annotated data runs it on ec2 on gpu boxes to fine tune a neural network model using caffe and then generates validation statistics to relate how well the trained model performs a bounding box system that takes the trained cloud classifier and attempts to draw bounding boxes on orbital satellite data in this case clouds the annotation tool is fairly straightforward it draws imagery from the planet labs api normalizes and pre processes it and then presents it to a user through a web browser to draw bounding boxes over candidate objects figure 5 animated gif showing annotation tool in action drawing bounding boxes around clouds the training portion takes the annotated images chops them into pieces representing images that either have clouds or not splits them into 80 training and 20 validation sets and trains a binary classifier on caffe the model itself is a fine tuned version of alexnet the open source project includes scripts to train these on gpu instances on amazon ec2 to explore different hyperparameter settings in parallel quickly and to download and query the trained results to understand how well training went on the validation data see two examples graphs generated from the training pipeline figure 3 example graph generated by pipeline from early training run showing loss decreasing over time figure 4 example graph generated by pipeline from early training showing accuracy increasing over time as a side note not pursued in this project an interesting approach might be to use such stacked images to get a constant image that represents unchanging elements of a location then when a new image from this location comes in compare it to the constant image to get a difference mask to see what has changed within this location this difference mask could then be passed downstream to other deep or non deep learning pipelines to infer what should be focused on in the image for object detection and localization perhaps one way to think of it is as helping a neural network attention model to know what to spend its time focusing on finally the trained classifier is fed into a python based implementation of selective search 3 4 in order to draw candidate bounding boxes around clouds as part of the work on this project the python selective search library was back ported from python 3 to python 2 7 to be compatible with caffe and the rest of the python 2 7 based cloudless pipeline here are example before and after output showing detected clouds on the final trained model detailed more below in the trained model section detected clouds are overlaid with yellow boxes below figure 6 satellite image before bounding boxes shown figure 7 satellite image with detected bounding boxes from cloudless overlaid in yellow the bounding box system also outputs a json file that gives detected cloud coordinates for later use by possible downstream computer vision consumers trained model it took quite a number of iterations to get a model with decent results the major approaches are detailed in table 1 with the final best approach bolded table 1 performance results for different model setups detailed later all iterations were based on a bvlc alexnet caffe zoo trained model 6 with fine tuning done on cloudless annotation data convolutional layers were frozen during fine tuning with training done only on the final fully connected layer the alexnet imagenet output softmax was reduced from one thousand classes to two indicating whether a cloud is present in a given image or not all iterations had a standard 80 20 split between training and hold out validation data the number of training epochs for most runs converged fairly quickly as you can see in the graph below finally all iterations used location imagery restricted to the san francisco bay area as we were restricted via the planet labs api to the state of california figure 8 training converged very rapidly this shows accuracy converging after a few hundred iterations and then stabilizing for the rest of the 20 000 iterations 700 images from the planet labs api were hand labelled via the annotation tool and trained the final accuracy was fairly low only 62 5 this was discovered to not necessarily be from small amounts of data but rather from fairly extensive motion blur affecting some satellite imagery an example is shown below figure 9 motion blur in planet labs imagery despite this the bounding boxes generated were still in the realm of plausibility though with some false positives caused by the motion blur an example from the earlier image figure 10 planet labs motion blur image with detected cloud bounding boxes in yellow getting to a better accuracy involved three things 1 about two months after cloudless was started planet labs purchased a fleet of satellites named rapideye the data from these are clearer than the planet labs satellites currently are an example image figure 11 example rapideye imagery showing clearer resolution without motion blur 2 the same location but over 65 days were annotated and fed into the network allowing the network to learn what is constant in a location and what changes here s four days of the larger format imagery that fed into the annotation tool as an example figure 12 a single day from the san francisco bay area figure 13 a single day from the san francisco bay area figure 14 a single day from the san francisco bay area figure 15 a single day from the san francisco bay area in affect the network was getting examples of what the san francisco bay area looks like when it s clear and when it s cloudy allowing it to generalize from this 3 more images were hand labelled with the annotation tool totalling about 4000 images these three steps added up to a much larger final accuracy of 89 69 on the validation data with much stronger precision and recall than the earlier run while its difficult to say exactly how the hand engineered non deep learning cloud detection pipelines detailed in related work above and in 1 perform as its fairly dependent on location and hand chosen threshold general performance accuracy is reported to be in the 80 to 90 range making cloudless generally competitive with them unlike those other solutions however cloudless could be fed with multi class annotated data and trained for a wide variety of tasks without a hand rolled feature engineering pipeline focused just on clouds for the final best solution see the final confusion matrix and details on the raw data fed into neural network training pipeline experiments were done to attempt to get beyond the 89 69 accuracy via greater data augmentation all iterations used caffe s built in clipping and mirroring transformations during training experiments were done however with manually rotating all labelled images via data preparation 90 degrees in four directions to see if this aided training counterintuitively though performance was actually lower as detailed in table 1 above limitations since cloudless uses only visible spectrum imagery it does not work with night time images in addition cloudless has not been trained or tested on regions with snow which has been reported to cause issues with other cloud detection schemes in addition the selective search bounding box scheme chosen is fairly computationally intensive on my macbook pro laptop with a 2 5 ghz intel core i7 and an nvidia gpu for example processing a single image took about two minutes this is not terrible for planet labs as a given location would only be imaged once a day and this task can be parallelized fairly easily unfortunately generating the bounding boxes depends on a number of hyperparameters for selective search that are not always generic across input images and requires some hand tuning for a given location it s not always hands free yet example hyperparameter values are given on the github cloudless page this motivates some of the ideas in the future work section below future work a good future step is to eliminate selective search s computational slowness and hand crafted hyperparameter tuning from the equation one possibility is to turn the convolutional neural network currently in cloudless into a deconvolution neural network 7 feeding in a raw image and having the output be a pixel mask where each value is the class of that pixel whether its a cloud or some other desired classification and localization this might allow the network itself to learn what hyperparameter thresholds are appropriate for different input images eliminating the kind of hand tuning the selective search bounding boxes currently require as well as providing pixel level classification to increase accuracy in general it s clear from non deep learning cloud detection techniques that non visible spectral bands can be important for inferring the presence of clouds or other items when a user annotates a satellite image in the annotation tool in the visual spectrum other side channel non visual bands can be saved and also fed into the network such as infrared thermal etc as planet labs integrates greater spectral bands into their satellites this will become more of an option in addition we should be able to feed in the latitude longitude altitude day and time of the year and the position of the sun of each pixel as input into the network this is clearly information that a human themselves would use to identify what something is for example if you were told that an image came from the north pole you would probably classify a white patch as snow if you were unsure while if you were told it came from the equator in a tropical forest you d assign a very high probability to a blury white patch as being a cloud if you weren t sure what a white patch was while looking at an orbital image of nyc you d probably have different answers if you knew it was winter or summer it seems only natural to give these same features to a neural network to help it decide when its dealing with boundary cases where its not quite sure having this information would allow it to essentially take a bayesian approach to figuring out what it is looking at to make an informed guess finally the annotation tool itself can be extended to run on mechanical turk to bootstrap even more data which would probably be necessary if the deconvolution approach earlier is taken as it has more free parameters to train if this is done the annotation tool will have to be extended with a training step to gauge how well labellers do as well as a validation step to run labelled data by other groups of users to gauge their quality as in 8 once done and once planet labs has satellite data with a greater resolution this can be used to bootstrap multi class data sets with examples of cars roads power lines ocean tankers buildings different biomes etc conclusion in this blog post i ve introduced cloudless a three part open source orbital satellite pipeline that provides annotation tools a deep learning component and a bounding box system it is currently focused on cloud detection and localization but could be extended for other non cloud tasks in the future the final trained model has an accuracy of 89 69 on cloud detection generally competitive with hand tuned non deep learning cloud detection pipelines one strong aspect of deep learning is that future proposed enhancements such as those detailed in future work above should help with end to end learning of most computer vision oriented orbital satellite data tasks not just cloud detection tasks rather than just increase cloud detection such as with other non deep learning oriented approaches that use manual feature engineering investments in deep learning can help improve...