Meta tags:
description=Scala, Java, Typesafe, Play Framework, Akka, Spark, Kafka, Elasticsearch, Node.js, MongoDB, PowerBI;

Headings (most frequently used words):

to, and, power, bi, data, spark, in, datasets, from, creating, loading, saving, file, dataset, navigation, mongo, store, it, read, kafka, mongodb, tutorial, overview, of, elasticsearch, recent, sparksession, rklick, solutions, llc, service, connect, desktop, new, features, push, streaming, by, microsoft, flow, report, view, updates, get, again, db, stream, how, load, save, different, source, quick, core, functionality, character, filter, introduction, post, category, tags, posts, comments, archives, meta, scala, java, typesafe, play, framework, akka, node, js, powerbi, dataframe, rdd, api, points, be, discussed, using, session, dataframes, csv, json, txt,

Text of the page (most frequently used words):
the (156), spark (115), and (93), for (69), api (57), data (55), #dataset (47), val (45), from (42), dataframe (42), can (42), this (34), create (33), will (32), mongodb (31), new (30), rdd (30), #sparksession (30), kafka (29), name (27), now (26), you (26), with (25), power (25), streaming (23), use (23), have (22), collection (22), which (21), using (20), read (20), then (19), example (19), that (19), json (18), content (17), rklick (17), get (16), datasets (16), are (16), following (16), 2016 (15), mongo (15), here (15), com (14), solutions (14), filter (14), select (14), database (14), also (14), localhost (14), llc (13), into (13), main (13), schema (13), string (13), option (13), scala (12), session (12), table (12), first (12), like (11), how (11), save (11), store (11), start (11), master (11), local (11), all (11), list (11), object (11), value (11), file (11), csv (11), write (10), tutorial (10), would (10), builder (10), getorcreate (10), but (10), was (10), entry (10), point (10), need (10), map (10), text (10), some (10), format (10), comment (9), april (9), 2017 (9), functionality (9), any (9), appname (9), catalog (9), type (9), where (9), step (9), true (9), resources (9), based (9), method (9), writeconfig (9), readconfig (9), elasticsearch (8), character (8), context (8), different (8), abstraction (8), topic (8), testindex (8), query (8), your (8), key (8), stream (8), deserializer (8), load (7), flow (7), uncategorized (7), core (7), search (7), apache (7), spark2 (7), more (7), access (7), fetch (7), sqlcontext (7), note (7), dataframes (7), testtype (7), analyzer (7), gramanalyzer (7), want (7), src (7), consumer (7), server (7), command (7), info (7), free (6), singh (6), features (6), service (6), typesafe (6), look (6), add (6), code (6), show (6), has (6), created (6), objects (6), used (6), named (6), curl (6), 9200 (6), see (6), below (6), our (6), txt (6), creates (6), header (6), document (6), price (6), leave (6), messages (6), config (6), bin (6), uses (6), action (6), tweet (6), wordpress (5), comments (5), blog (5), november (5), ved (5), prakash (5), desktop (5), sparksql (5), play (5), adding (5), feel (5), hivecontext (5), streamingcontext (5), standard (5), about (5), there (5), domain (5), not (5), class (5), two (5), array (5), custom (5), karra (5), john (5), http (5), index (5), set (5), println (5), group (5), itemno (5), kph (5), options (5), sparkcontext (5), zookeeper (5), documents (5), savetomongodb (5), mongospark (5), axis (5), formatting (5), reports (5), sentiment (5), loading (4), view (4), introduction (4), quick (4), again (4), microsoft (4), navigation (4), connect (4), bootstrap (4), provides (4), basic (4), make (4), together (4), suggestion (4), suggest (4), stay (4), tuned (4), available (4), earlier (4), versions (4), other (4), needed (4), sql (4), build (4), definition (4), part (4), interface (4), able (4), rdds (4), one (4), work (4), back (4), match (4), out (4), settings (4), properties (4), rklickdata (4), rklickwords (4), java (4), creating (4), team (4), version (4), services (4), web (4), enable (4), selecteddata (4), age (4), producer (4), run (4), ssc (4), 27017 (4), org (4), indicates (4), uri (4), customer (4), customerinfo (4), conditional (4), visual (4), down (4), slicer (4), workspace (4), powerbi (4), dashboards (4), workspaces (4), report (3), log (3), sign (3), subscribe (3), already (3), october (3), may (3), august (3), overview (3), recent (3), push (3), application (3), schemardd (3), framework (3), demonstrates (3), next (3), grow (3), send (3), full (3), accumulator (3), old (3), contexts (3), manipulated (3), hive (3), becoming (3), called (3), source (3), single (3), replacing (3), typed (3), optimized (3), major (3), become (3), library (3), structured (3), cover (3), most (3), these (3), distributed (3), flatmap (3), time (3), existing (3), each (3), cluster (3), total (3), xget (3), _search (3), term (3), karraandjohn (3), above (3), convert (3), going (3), tokenizer (3), steps (3), tutorials (3), split (3), count (3), cars_price (3), applications (3), describe (3), def (3), args (3), uuid (3), randomuuid (3), generate (3), saving (3), include (3), enter (3), produce (3), mongodbformat (3), mode (3), kafkautils (3), colourcode (3), fields (3), colour (3), specify (3), host (3), mongodbdatabase (3), mongodbcollection (3), mongodbconfig (3), stratio (3), datasource (3), 9092 (3), https (3), console (3), line (3), take (3), topics (3), its (3), sparkconf (3), info_updated (3), loadfrommongodb (3), connector (3), color (3), menu (3), selecting (3), hierarchical (3), drop (3), dashboard (3), preview (3), akka (3), started (2), site (2), website (2), required (2), subscribed (2), account (2), feed (2), updates (2), posts (2), lightbend (2), activator (2), rethinkdb (2), elastic (2), category (2), post (2), week (2), onwards (2), working (2), databases (2), introduced (2), well (2), hierarchy (2), types (2), backward (2), compatibility (2), future (2), those (2), internally (2), actual (2), computation (2), every (2), them (2), both (2), talks (2), special (2), compilation (2), checks (2), makes (2), specific (2), signifies (2), always (2), strongly (2), difference (2), better (2), performance (2), though (2), runtime (2), development (2), row (2), methods (2), groupbykey (2), combined (2), stable (2), should (2), over (2), benefits (2), constructed (2), many (2), issues (2), frame (2), sources (2), structure (2), partitions (2), false (2), successful (2), hits (2), seen (2), check (2), containing (2), characters (2), before (2), html (2), analysis (2), char_filter (2), quotes (2), mapping (2), mappings (2), whitespace (2), lowercase (2), search_analyzer (2), useful (2), import (2), implicits (2), rklickgroupedwords (2), rklickwordcount (2), similar (2), processing (2), help (2), easily (2), discuss (2), tried (2), basics (2), technology (2), company (2), providing (2), business (2), dynamic (2), latest (2), 5000000 (2), location (2), mumbai (2), inserted (2), project (2), peopledf (2), mongodboptiops (2), covert (2), createdirectstream (2), topic1 (2), kafkaparams (2), pass (2), structtype (2), schemastring (2), fieldname (2), default (2), parameters (2), servers (2), connection (2), implements (2), classof (2), stringdeserializer (2), auto (2), seconds (2), mongodb_2 (2), input (2), separate (2), 2181 (2), case (2), configuration (2), implicit (2), column (2), apply (2), label (2), title (2), labels (2), matrix (2), turn (2), field (2), off (2), experience (2), pane (2), box (2), when (2), realtimedata (2), pipe (2), detection (2), right (2), screen (2), cognitive (2), tweets (2), number (2), twitter (2), posted (2), their (2), click (2), feature (2), node (2), design, email, collapse, bar, manage, subscriptions, reader, privacy, entries, meta, 2015, february, march, june, july, archives, malti, yadav, supriya, srivastava, asynchronous, async, tags, webjars, crud, bootswatch, architecture, older, modules, changes, listtables, listdatabases, metadata, designed, simpler, support, specialization, primitive, been, deprecated, retained, essentially, combination, replaces, users, common, confusion, subsumes, demonstrated, notebook, still, kept, superset, released, brings, flexibility, platform, compared, releases, long, layer, low, level, mostly, user, land, written, against, subset, starting, just, alias, untyped, groupby, points, discussed, cases, developer, advised, embracing, added, strong, typing, ability, powerful, lambda, functions, execution, engine, jvm, functional, transformations, etc, solved, good, enough, replacement, compile, safety, held, people, everywhere, were, fill, gap, organized, columns, conceptually, equivalent, relational, python, richer, optimizations, under, hood, wide, such, files, tables, external, brought, memory, management, generation, greatly, improved, last, year, improvements, went, due, facing, issue, related, advanced, optimization, move, fundamental, immutable, divided, logical, computed, nodes, resilient, learned, took, timed_out, _shards, failed, max_score, _index, _type, _id, _score, _source, result, pre, filtered, indexing, _analyze, xpost, filters, preprocess, passed, strip, markup, word, xput, whitespaceanalyzer, _all, include_in_all, explain, done, suppose, tolowercase, alpha, however, instead, serialization, kryo, they, specialized, encoder, serialize, transmitting, network, textfile, looks, exactly, reading, replace, once, previous, know, savetxt, creative, digital, focused, helping, clients, mobile, mature, solution, oriented, product, customized, consulting, since, 2007, dedicated, techno, whiz, engaged, outstanding, diverse, entities, across, north, america, known, professional, technologists, customers, stack, deliver, quality, cost, effective, products, ferrari, 52000000, jaguar, 15000000, mercedes, 10000000, audi, lamborghini, examples, mahendra, dhoni, ranchi, virat, kohli, delhi, shikhar, dhawan, rohit, sharma, stuart, binny, chennai, kafkamongotest1, people1, same, produced, irrespective, fomat, taken, than, null, consume, directory, reside, awaittermination, end, elementdstream, foreachrdd, savemode, append, dstream, preferconsistent, pair, parameter, structfield, stringtype, nullable, incoming, filed, scehma, lets, assume, getting, mentioned, stored, coloursinfo, refer, documentation, newconsumerconfigs, port, pairs, establishing, initial, unique, identifies, belongs, use_a_separate_group_id_for_each_stream, offset, reset, commit, lang, boolean, values, passing, mvnrepository, maven, repository, 10_2, writing, program, depedecies, buid, sbt, beginning, dump, output, broker, few, comes, client, sent, message, replication, factor, colourcoe, names, test, severs, machine, standalone, hexadecimal, put, call, address, defined, without, arguments, specified, helper, overwrite, accepts, withcolumn, current_age, operation, previously, made, customerinfo_new, multiply, whole, english, calles, new_info, modification, operations, readconf, element, records, printschema, representing, argument, mention, refere, docs, classes, respectively, integration, programming, diffentely, shown, connector_2, dependency, libraries, insert, exist, after, control, separately, between, lowest, middle, highest, giving, opening, measure, turning, concatenation, drill, detail, understanding, particaular, details, slicers, lots, locate, tile, finally, titled, actions, entered, information, side, analytics, detect, understand, positivity, takes, outputs, negative, positive, whenever, contains, flows, blank, sentiments, datetime, necessary, remember, real, pushing, rest, endpoint, having, update, visuals, radio, button, try, tab, choose, general, gear, icon, top, corner, choosing, switch, clicking, arrow, current, appears, shows, area, tabs, workbooks, shared, others, easy, quickly, recently, accessed, expand, navigate, restart, order, effect, live, allow, allows, published, graphx, microserives, scheduler, home, skip,

Text of the page (random words):
p v v value foreachrdd rdd val peopledf spark read schema schema json rdd peopledf write format mongodbformat mode savemode append options mongodboptiops save 15 at the end start spark context ssc start ssc awaittermination 16 now go to the directory where project reside and run it 17 produce some messages in producer for consumer to consume note here as we have taken data in json format enter the data in json format if you enter data other than json format it will show null in the mongo db table in consumer you will get the same messages produced by consumer irrespective of the fomat 18 now you can see the this data inserted in the mongo db table here name of our database is kafkamongotest1 and collection name is people1 so here we are able to see the data inserted in the mongo db table leave a comment tutorial how to load and save data from different data source in spark 2 0 2 november 26 2016 november 26 2016 ved prakash singh dataframe dataframe api dataset rdd scala schemardd spark spark 2 0 spark session spark2 0 sparksql in this blog we will discuss about spark 2 0 2 it demonstrates the basic functionality of spark 2 0 2 we also describe how to load and save data in spark2 0 2 we have tried to cover basics of spark 2 0 2 core functionality like read and write data from different source csv json txt loading and saving csv file as an example the following creates a dataframe based on the content of a csv file read a csv document named team csv with the following content and generate a table based on the schema in the csv document id name age location 1 mahendra singh dhoni 38 ranchi 2 virat kohli 25 delhi 3 shikhar dhawan 25 mumbai 4 rohit sharma 33 mumbai 5 stuart binny 22 chennai def main args array string val spark sparksession builder master local appname spark2 0 getorcreate val df spark read option header true csv resources team csv val selecteddata df select name age selecteddata write option header true save s src main resources uuid randomuuid println ok loading and saving json file here we include some basic examples of structured data processing using dataframes as an example the following creates a dataframe based on the content of a json file read a json document named cars_price json with the following content and generate a table based on the schema in the json document itemno 1 name ferrari price 52000000 kph 6 1 itemno 2 name jaguar price 15000000 kph 3 4 itemno 3 name mercedes price 10000000 kph 3 itemno 4 name audi price 5000000 kph 3 6 itemno 5 name lamborghini price 5000000 kph 2 9 def main args array string val spark sparksession builder master local appname spark2 0 getorcreate val df spark read option header true json resources cars_price json val selecteddata df select name price selecteddata write option header true save s src main resources uuid randomuuid println ok loading and saving txt file as an example the following creates a data frame based on the content of a text file read a text document named rklick txt with the following content and generate a table based on the schema in the text document rklick creative technology company providing key digital services focused on helping our clients to build a successful business on web and mobile we are mature and dynamic solution oriented it full service product development and customized consulting services company since 2007 we re dedicated team of techno whiz engaged in providing outstanding solutions to diverse group of business entities across north america known as professional technologists we help our customers enable latest technology stack or deliver quality cost effective products and applications def main args array string val spark sparksession builder master local appname spark2 0 getorcreate import spark implicits _ val rklickdata spark read text src main resources rklick txt as string val rklickwords rklickdata flatmap value value split s val savetxt rklickwords write text s src main resources uuid randomuuid println ok we would look at how we can create more useful tutorials to grow it then we would be adding more content to it together if you have any suggestion feel free to suggest us stay tuned 1 comment tutorial quick overview of spark 2 0 1 core functionality november 4 2016 november 4 2016 ved prakash singh dataframe dataframe api dataset rdd scala spark spark 1 6 spark 2 0 spark core spark session sparksql typesafe in this blog we will discuss about spark 2 0 1 core functionality it demonstrates the basic functionality of spark 2 0 1 we also describe sparksession spark sql and dataframe api functionality we have tried to cover basics of spark 2 0 1 core functionality and sparksession sparksession sparksession is new entry point of spark in previous version 1 6 x of spark spark context was entry point for spark and in spark 2 0 1 sparksession is entry point of spark spark session internally has a spark context for actual computation as we know rdd was main api it was created and manipulated using context api s for every other api we needed to use different contexts for streaming we needed streamingcontext for sql sqlcontext and for hive hivecontext but as dataset and dataframe api s are becoming new standard api s we need an entry point build for them so in spark 2 0 1 we have a new entry point for dataset and dataframe api s called as spark session creating sparksession here we describe how to create sparksession val spark sparksession builder master local appname spark example getorcreate once we have created spark session then we can use it to read the data read data using spark session it looks like exactly like reading using sqlcontext you can easily replace all your code of sqlcontext with sparksession now val spark sparksession builder master local appname spark example getorcreate creating dataframes how to create dataframes with the help of sparksession applications can create dataframes from an existing rdd from a from data sources as an example the following creates a dataframe based on the content of a csv file val dataframe spark read option header true csv src main resources team csv creates a dataframe based on the content of a json file val dataframe spark read option header true json src main resources cars_price json dataframe show creating datasets dataset is new abstraction in spark introduced as alpha api in spark 1 6 it s becoming stable api in spark 2 0 1 datasets are similar to rdds however instead of using java serialization or kryo they use a specialized encoder to serialize the objects for processing or transmitting over the network the major difference is dataset is collection of domain specific objects where as rdd is collection of any object domain object part of definition signifies the schema part of dataset so dataset api is always strongly typed and optimized using schema where rdd is not dataset definition also talks about dataframes api dataframe is special dataset where there is no compilation checks for schema so this makes dataset new single abstraction replacing rdd from earlier versions of spark we read data using read text api which is similar to textfile api of rdd the following creates a dataframe based on the content of a txt file import spark implicits _ val rklickdata spark read text src main resources rklick txt as string val rklickwords rklickdata flatmap value value split s val rklickgroupedwords rklickwords groupbykey _ tolowercase val rklickwordcount rklickgroupedwords count rklickwordcount show we would look at how we can create more useful tutorials to grow it then we would be adding more content to it together if you have any suggestion feel free to suggest us stay tuned 1 comment elasticsearch character filter october 25 2016 rklick solutions llc elastic search es elasticsearch es in this post i am going to explain how elasticsearch character filter work so there are following steps to done this step 1 set mapping for your index suppose our index name is testindex and type is testtype now we are going to set analyzer and filter curl xput localhost 9200 testindex d settings analysis char_filter quotes type mapping mappings and analyzer gramanalyzer type custom tokenizer whitespace char_filter quotes filter lowercase whitespaceanalyzer type custom tokenizer whitespace filter lowercase mappings testtype _all analyzer gramanalyzer search_analyzer gramanalyzer properties name type string include_in_all true analyzer gramanalyzer search_analyzer gramanalyzer store true character filters are used to preprocess the string of characters before it is passed to the tokenizer a character filter may be used to strip out html markup or to convert characters to the word and as you seen above we have set a filter to convert to and step 2 index your data now we are going to index data containing character but we want to fetch data from and curl xpost localhost 9200 testindex testtype 1 d name karra john step 3 check how analyzer work for your index data curl http localhost 9200 testindex _analyze analyzer gramanalyzer d karra john step 4 fetch data from match query now i want to fetch data from match query as you have seen above indexing data is karra john but now we would fetch those data from karra john see below query curl xget http localhost 9200 testindex testtype _search d query match name query karraandjohn analyzer gramanalyzer you can also do like this curl xget http localhost 9200 testindex testtype _search d query match name query karraandjohn step 5 fetch data from filter pre curl xget http localhost 9200 testindex testtype _search d query filtered filter term name karraandjohn result would like this took 1 timed_out false _shards total 5 successful 5 failed 0 hits total 1 max_score 1 0 hits _index testindex _type testtype _id 1 _score 1 0 _source name karra john so we have learned how character filter work this is the start of elasticsearch from next week onwards we would be working on new topic if you have any suggestion feel free to suggest us stay tuned 2 comments introduction to spark 2 0 august 11 2016 august 11 2016 rklick solutions llc application dataframe dataframe api dataset rdd scala schemardd spark spark 2 0 spark core spark streaming sparksql spark session overview of dataset dataframe and rdd api resilient distributed datasets rdd is a fundamental data structure of spark it is an immutable distributed collection of objects each dataset in rdd is divided into logical partitions which may be computed on different nodes of the cluster but due to facing issue related to advanced optimization move to dataframe dataframe brought custom memory management and runtime code generation which greatly improved performance so in last year most of the improvements went into dataframe api a dataframe is a dataset organized into named columns it is conceptually equivalent to a table in a relational database or a data frame in r python but with richer optimizations under the hood dataframes can be constructed from a wide array of sources such as structured data files tables in hive external databases or existing rdds though dataframe api solved many issues it was not a good enough replacement for rdd api one of the major issues with dataframe api was no compile time safety and not able to work with domain objects so this held back people using dataframe api everywhere but with introduction of dataset api in 1 6 we were able to fill the gap a dataset is a distributed collection of data dataset is a new interface added in spark 1 6 that provides the benefits of rdds strong typing ability to use powerful lambda functions with the benefits of spark sql s optimized execution engine a dataset can be constructed from jvm objects and then manipulated using functional transformations map flatmap filter etc so in spark 2 0 dataset api will be become a stable api so dataset api combined with dataframe api should able to cover most of the use cases where rdd was used earlier so as a spark developer it is advised to start embracing these two api s over rdd api from spark 2 0 points to be discussed datasets starting in spark 2 0 dataframe is just a type alias for dataset of row both the typed methods e g map filter groupbykey and the untyped methods e g select groupby are available on the dataset class also this new combined dataset interface is the abstraction used for structured streaming for long rdd was the standard abstraction of spark but from spark 2 0 dataset will become the new abstraction layer for spark though rdd api will be available it will become low level api used mostly for runtime and library development all user land code will be written against the dataset abstraction and it s subset dataframe api dataset is a superset of dataframe api which is released in spark 1 3 dataset together with dataframe api brings better performance and flexibility to the platform compared to rdd api dataset will be also replacing rdd as an abstraction for streaming in future releases the major difference is dataset is collection of domain specific objects where as rdd is collection of any object domain object part of definition signifies the schema part of dataset so dataset api is always strongly typed and optimized using schema where rdd is not dataset definition also talks about dataframes api dataframe is special dataset where there is no compilation checks for schema so this makes dataset new single abstraction replacing rdd from earlier versions of spark sparksession a new entry point that replaces the old sqlcontext and hivecontext for users of the dataframe api a common source of confusion for spark is which context to use now you can use sparksession which subsumes both as a single entry point as demonstrated in this notebook note that the old sqlcontext and hivecontext are still kept for backward compatibility in earlier versions of spark spark context was entry point for spark as rdd was main api it was created and manipulated using context api s for every other api we needed to use different contexts for streaming we needed streamingcontext for sql sqlcontext and for hive hivecontext but as dataset and dataframe api s are becoming new standard api s we need an entry point build for them so in spark 2 0 we have a new entry point for dataset and dataframe api s called as spark session sparksession is essentially combination of sqlcontext hivecontext and future streamingcontext all the api s available on those contexts are available on spark session also spark session internally has a spark context for actual computation val sparksession sparksession builder master local appname spark session example getorcreate new accumulator api we have designed a new accumulator api that has a simpler type hierarchy and support specialization for primitive types the old accumulator api has been deprecated but retained for backward compatibility catalog api in spark2 0 a new catalog api is introduced to access metadata you can fetch all database list as well as table list using t...

Images from subpage: "rklicksolutions.wordpress.com/category/spark/" Verify

Images from subpage: "rklicksolutions.wordpress.com/category/spark-2-0/" Verify

Images from subpage: "rklicksolutions.wordpress.com/category/spark-session/... " Verify

Images from subpage: "rklicksolutions.wordpress.com/category/spark2-0/" Verify

Images from subpage: "rklicksolutions.wordpress.com/category/sparksql/" Verify