Meta tags:
description=Scala, Java, Typesafe, Play Framework, Akka, Spark, Kafka, Elasticsearch, Node.js, MongoDB, PowerBI;

Headings (most frequently used words):

to, and, power, bi, data, spark, in, datasets, from, creating, loading, saving, file, dataset, navigation, mongo, store, it, read, kafka, mongodb, tutorial, overview, of, elasticsearch, recent, sparksession, rklick, solutions, llc, service, connect, desktop, new, features, push, streaming, by, microsoft, flow, report, view, updates, get, again, db, stream, how, load, save, different, source, quick, core, functionality, character, filter, introduction, post, category, tags, posts, comments, archives, meta, scala, java, typesafe, play, framework, akka, node, js, powerbi, dataframe, rdd, api, points, be, discussed, using, session, dataframes, csv, json, txt,

Text of the page (most frequently used words):
the (156), spark (115), and (93), for (69), api (57), data (55), #dataset (47), val (45), from (42), dataframe (42), can (42), this (34), create (33), will (32), mongodb (31), new (30), rdd (30), #sparksession (30), kafka (29), name (27), now (26), you (26), with (25), power (25), streaming (23), use (23), have (22), collection (22), which (21), using (20), read (20), then (19), example (19), that (19), json (18), content (17), rklick (17), get (16), datasets (16), are (16), following (16), 2016 (15), mongo (15), here (15), com (14), solutions (14), filter (14), select (14), database (14), also (14), localhost (14), llc (13), into (13), main (13), schema (13), string (13), option (13), scala (12), session (12), table (12), first (12), like (11), how (11), save (11), store (11), start (11), master (11), local (11), all (11), list (11), object (11), value (11), file (11), csv (11), write (10), tutorial (10), would (10), builder (10), getorcreate (10), but (10), was (10), entry (10), point (10), need (10), map (10), text (10), some (10), format (10), comment (9), april (9), 2017 (9), functionality (9), any (9), appname (9), catalog (9), type (9), where (9), step (9), true (9), resources (9), based (9), method (9), writeconfig (9), readconfig (9), elasticsearch (8), character (8), context (8), different (8), abstraction (8), topic (8), testindex (8), query (8), your (8), key (8), stream (8), deserializer (8), load (7), flow (7), uncategorized (7), core (7), search (7), apache (7), spark2 (7), more (7), access (7), fetch (7), sqlcontext (7), note (7), dataframes (7), testtype (7), analyzer (7), gramanalyzer (7), want (7), src (7), consumer (7), server (7), command (7), info (7), free (6), singh (6), features (6), service (6), typesafe (6), look (6), add (6), code (6), show (6), has (6), created (6), objects (6), used (6), named (6), curl (6), 9200 (6), see (6), below (6), our (6), txt (6), creates (6), header (6), document (6), price (6), leave (6), messages (6), config (6), bin (6), uses (6), action (6), tweet (6), wordpress (5), comments (5), blog (5), november (5), ved (5), prakash (5), desktop (5), sparksql (5), play (5), adding (5), feel (5), hivecontext (5), streamingcontext (5), standard (5), about (5), there (5), domain (5), not (5), class (5), two (5), array (5), custom (5), karra (5), john (5), http (5), index (5), set (5), println (5), group (5), itemno (5), kph (5), options (5), sparkcontext (5), zookeeper (5), documents (5), savetomongodb (5), mongospark (5), axis (5), formatting (5), reports (5), sentiment (5), loading (4), view (4), introduction (4), quick (4), again (4), microsoft (4), navigation (4), connect (4), bootstrap (4), provides (4), basic (4), make (4), together (4), suggestion (4), suggest (4), stay (4), tuned (4), available (4), earlier (4), versions (4), other (4), needed (4), sql (4), build (4), definition (4), part (4), interface (4), able (4), rdds (4), one (4), work (4), back (4), match (4), out (4), settings (4), properties (4), rklickdata (4), rklickwords (4), java (4), creating (4), team (4), version (4), services (4), web (4), enable (4), selecteddata (4), age (4), producer (4), run (4), ssc (4), 27017 (4), org (4), indicates (4), uri (4), customer (4), customerinfo (4), conditional (4), visual (4), down (4), slicer (4), workspace (4), powerbi (4), dashboards (4), workspaces (4), report (3), log (3), sign (3), subscribe (3), already (3), october (3), may (3), august (3), overview (3), recent (3), push (3), application (3), schemardd (3), framework (3), demonstrates (3), next (3), grow (3), send (3), full (3), accumulator (3), old (3), contexts (3), manipulated (3), hive (3), becoming (3), called (3), source (3), single (3), replacing (3), typed (3), optimized (3), major (3), become (3), library (3), structured (3), cover (3), most (3), these (3), distributed (3), flatmap (3), time (3), existing (3), each (3), cluster (3), total (3), xget (3), _search (3), term (3), karraandjohn (3), above (3), convert (3), going (3), tokenizer (3), steps (3), tutorials (3), split (3), count (3), cars_price (3), applications (3), describe (3), def (3), args (3), uuid (3), randomuuid (3), generate (3), saving (3), include (3), enter (3), produce (3), mongodbformat (3), mode (3), kafkautils (3), colourcode (3), fields (3), colour (3), specify (3), host (3), mongodbdatabase (3), mongodbcollection (3), mongodbconfig (3), stratio (3), datasource (3), 9092 (3), https (3), console (3), line (3), take (3), topics (3), its (3), sparkconf (3), info_updated (3), loadfrommongodb (3), connector (3), color (3), menu (3), selecting (3), hierarchical (3), drop (3), dashboard (3), preview (3), akka (3), started (2), site (2), website (2), required (2), subscribed (2), account (2), feed (2), updates (2), posts (2), lightbend (2), activator (2), rethinkdb (2), elastic (2), category (2), post (2), week (2), onwards (2), working (2), databases (2), introduced (2), well (2), hierarchy (2), types (2), backward (2), compatibility (2), future (2), those (2), internally (2), actual (2), computation (2), every (2), them (2), both (2), talks (2), special (2), compilation (2), checks (2), makes (2), specific (2), signifies (2), always (2), strongly (2), difference (2), better (2), performance (2), though (2), runtime (2), development (2), row (2), methods (2), groupbykey (2), combined (2), stable (2), should (2), over (2), benefits (2), constructed (2), many (2), issues (2), frame (2), sources (2), structure (2), partitions (2), false (2), successful (2), hits (2), seen (2), check (2), containing (2), characters (2), before (2), html (2), analysis (2), char_filter (2), quotes (2), mapping (2), mappings (2), whitespace (2), lowercase (2), search_analyzer (2), useful (2), import (2), implicits (2), rklickgroupedwords (2), rklickwordcount (2), similar (2), processing (2), help (2), easily (2), discuss (2), tried (2), basics (2), technology (2), company (2), providing (2), business (2), dynamic (2), latest (2), 5000000 (2), location (2), mumbai (2), inserted (2), project (2), peopledf (2), mongodboptiops (2), covert (2), createdirectstream (2), topic1 (2), kafkaparams (2), pass (2), structtype (2), schemastring (2), fieldname (2), default (2), parameters (2), servers (2), connection (2), implements (2), classof (2), stringdeserializer (2), auto (2), seconds (2), mongodb_2 (2), input (2), separate (2), 2181 (2), case (2), configuration (2), implicit (2), column (2), apply (2), label (2), title (2), labels (2), matrix (2), turn (2), field (2), off (2), experience (2), pane (2), box (2), when (2), realtimedata (2), pipe (2), detection (2), right (2), screen (2), cognitive (2), tweets (2), number (2), twitter (2), posted (2), their (2), click (2), feature (2), node (2), design, email, collapse, bar, manage, subscriptions, reader, privacy, entries, meta, 2015, february, march, june, july, archives, malti, yadav, supriya, srivastava, asynchronous, async, tags, webjars, crud, bootswatch, architecture, older, modules, changes, listtables, listdatabases, metadata, designed, simpler, support, specialization, primitive, been, deprecated, retained, essentially, combination, replaces, users, common, confusion, subsumes, demonstrated, notebook, still, kept, superset, released, brings, flexibility, platform, compared, releases, long, layer, low, level, mostly, user, land, written, against, subset, starting, just, alias, untyped, groupby, points, discussed, cases, developer, advised, embracing, added, strong, typing, ability, powerful, lambda, functions, execution, engine, jvm, functional, transformations, etc, solved, good, enough, replacement, compile, safety, held, people, everywhere, were, fill, gap, organized, columns, conceptually, equivalent, relational, python, richer, optimizations, under, hood, wide, such, files, tables, external, brought, memory, management, generation, greatly, improved, last, year, improvements, went, due, facing, issue, related, advanced, optimization, move, fundamental, immutable, divided, logical, computed, nodes, resilient, learned, took, timed_out, _shards, failed, max_score, _index, _type, _id, _score, _source, result, pre, filtered, indexing, _analyze, xpost, filters, preprocess, passed, strip, markup, word, xput, whitespaceanalyzer, _all, include_in_all, explain, done, suppose, tolowercase, alpha, however, instead, serialization, kryo, they, specialized, encoder, serialize, transmitting, network, textfile, looks, exactly, reading, replace, once, previous, know, savetxt, creative, digital, focused, helping, clients, mobile, mature, solution, oriented, product, customized, consulting, since, 2007, dedicated, techno, whiz, engaged, outstanding, diverse, entities, across, north, america, known, professional, technologists, customers, stack, deliver, quality, cost, effective, products, ferrari, 52000000, jaguar, 15000000, mercedes, 10000000, audi, lamborghini, examples, mahendra, dhoni, ranchi, virat, kohli, delhi, shikhar, dhawan, rohit, sharma, stuart, binny, chennai, kafkamongotest1, people1, same, produced, irrespective, fomat, taken, than, null, consume, directory, reside, awaittermination, end, elementdstream, foreachrdd, savemode, append, dstream, preferconsistent, pair, parameter, structfield, stringtype, nullable, incoming, filed, scehma, lets, assume, getting, mentioned, stored, coloursinfo, refer, documentation, newconsumerconfigs, port, pairs, establishing, initial, unique, identifies, belongs, use_a_separate_group_id_for_each_stream, offset, reset, commit, lang, boolean, values, passing, mvnrepository, maven, repository, 10_2, writing, program, depedecies, buid, sbt, beginning, dump, output, broker, few, comes, client, sent, message, replication, factor, colourcoe, names, test, severs, machine, standalone, hexadecimal, put, call, address, defined, without, arguments, specified, helper, overwrite, accepts, withcolumn, current_age, operation, previously, made, customerinfo_new, multiply, whole, english, calles, new_info, modification, operations, readconf, element, records, printschema, representing, argument, mention, refere, docs, classes, respectively, integration, programming, diffentely, shown, connector_2, dependency, libraries, insert, exist, after, control, separately, between, lowest, middle, highest, giving, opening, measure, turning, concatenation, drill, detail, understanding, particaular, details, slicers, lots, locate, tile, finally, titled, actions, entered, information, side, analytics, detect, understand, positivity, takes, outputs, negative, positive, whenever, contains, flows, blank, sentiments, datetime, necessary, remember, real, pushing, rest, endpoint, having, update, visuals, radio, button, try, tab, choose, general, gear, icon, top, corner, choosing, switch, clicking, arrow, current, appears, shows, area, tabs, workbooks, shared, others, easy, quickly, recently, accessed, expand, navigate, restart, order, effect, live, allow, allows, published, graphx, microserives, scheduler, home, skip,

Text of the page (random words):
ds have an implicit helper method savetomongodb to write data to mongodb for example the following uses the documents rdd defined above and uses its savetomongodb method without any arguments to save the documents to the collection specified in the sparkconf documents savetomongodb uses the sparkconf for configuration call savetomongodb with a writeconfig object to specify a different mongodb server address database and collection documents savetomongodb writeconfig map uri mongodb example com customer customerinfo uses the writeconfig leave a comment read data from kafka stream and store it in to mongodb april 4 2017 april 4 2017 rklick solutions llc apache kafka kafka mongodb uncategorized mongodb read data from kafka stream and store it in to mongodb use case in this tutorial we will create a topic in kafka and then using producer we will produce some data in json format which we will store to mongodb for example here we will pass colour and its hexadecimal code in json in kafka and put it in the mongodb table version which we are using kafka 0 10 0 spark 2 0 2 scala 2 11 8 steps 1 start the zookeeper run the command below bin zookeeper server start sh config zookeeper properties 2 start the kafka severs on your local machine in standalone mode run the below command bin kafka server start sh config server properties 3 create a topic in kafka server from which you want to produce messages for example we want to create a topic names test bin kafka topics sh create zookeeper localhost 2181 replication factor 1 partitions 1 topic colourcoe note you can check the list of topics using following command bin kafka topics sh list zookeeper localhost 2181 4 kafka comes with a command line client that will take input from a file or from standard input and send it out as messages to the kafka cluster by default each line will be sent as a separate message run the producer and then type a few messages into the console to send to the server bin kafka console producer sh broker list localhost 9092 topic colourcode 5 kafka also has a command line consumer that will dump out messages to standard output bin kafka console consumer sh bootstrap server localhost 9092 topic colourcode from beginning 6 now before writing program we need to include following depedecies in buid sbt org apache spark spark streaming kafka 0 10_2 11 2 1 0 org apache spark spark streaming 2 1 0 com stratio datasource spark mongodb_2 11 0 12 0 you can get this from maven repository https mvnrepository com 7 first create a sparksession and set master val spark sparksession builder master local config spark some config option some value getorcreate 8 now create a streamingcontext which is for kafka streaming val ssc new streamingcontext spark sparkcontext seconds 2 here we are passing sparkcontext as spark sparkcontext 9 now create one val containing all the values for kafka parameters val kafkaparams map string object bootstrap servers localhost 9092 key deserializer classof stringdeserializer value deserializer classof stringdeserializer group id use_a_separate_group_id_for_each_stream auto offset reset latest enable auto commit false java lang boolean here bootstrap servers indicates a list of host port pairs to use for establishing the initial connection to the kafka cluster key deserializer indicates deserializer class for key that implements the deserializer interface value deserializer indicates deserializer class for key that implements the deserializer interface group id indicates a unique string that identifies the consumer group this consumer belongs to note also you can refer to so many other properties here http kafka apache org documentation html newconsumerconfigs 10 now create val to set the parameters for mongodb for example val mongodbformat com stratio datasource mongodb val mongodbdatabase coloursinfo val mongodbcollection colour val mongodboptiops map mongodbconfig host localhost 27017 mongodbconfig database mongodbdatabase mongodbconfig collection mongodbcollection where mongodbformat is the format in which we store in mongodb using this library mongodbdatabase is the database in mongodb in which we want to save that collection mongodbcollection is the collection of database in which data will be stored by default host for mongodb is 27017 which is mentioned in th options 11 now here we can get data in any format from kafka stream lets assume we are getting data in json format we can specify schema structure for the data using structtype for example in incoming data if we have two filed name and age we can specify scehma like val schemastring colour code val fields schemastring split map fieldname structfield fieldname stringtype nullable true val schema structtype fields 12 we need to pass the topic name as a parameter in the kafkautils method so val topic1 array colourcode 13 kafka library provides us kafkautils class of which createdirectstream method we can use to fetch the kafka streaming data and get in format of key value pair val stream kafkautils createdirectstream string string ssc preferconsistent subscribe string string topic1 kafkaparams 14 we get dstream in the stream val we will covert that stream into dataframe for that first we convert it into rdd of stream which we will covert into dataframe that dataframe we will store into mongodb val elementdstream stream map v v value foreachrdd rdd val peopledf spark read schema schema json rdd peopledf write format mongodbformat mode savemode append options mongodboptiops save 15 at the end start spark context ssc start ssc awaittermination 16 now go to the directory where project reside and run it 17 produce some messages in producer for consumer to consume note here as we have taken data in json format enter the data in json format if you enter data other than json format it will show null in the mongo db table in consumer you will get the same messages produced by consumer irrespective of the fomat 18 now you can see the this data inserted in the mongo db table here name of our database is kafkamongotest1 and collection name is people1 so here we are able to see the data inserted in the mongo db table leave a comment tutorial how to load and save data from different data source in spark 2 0 2 november 26 2016 november 26 2016 ved prakash singh dataframe dataframe api dataset rdd scala schemardd spark spark 2 0 spark session spark2 0 sparksql in this blog we will discuss about spark 2 0 2 it demonstrates the basic functionality of spark 2 0 2 we also describe how to load and save data in spark2 0 2 we have tried to cover basics of spark 2 0 2 core functionality like read and write data from different source csv json txt loading and saving csv file as an example the following creates a dataframe based on the content of a csv file read a csv document named team csv with the following content and generate a table based on the schema in the csv document id name age location 1 mahendra singh dhoni 38 ranchi 2 virat kohli 25 delhi 3 shikhar dhawan 25 mumbai 4 rohit sharma 33 mumbai 5 stuart binny 22 chennai def main args array string val spark sparksession builder master local appname spark2 0 getorcreate val df spark read option header true csv resources team csv val selecteddata df select name age selecteddata write option header true save s src main resources uuid randomuuid println ok loading and saving json file here we include some basic examples of structured data processing using dataframes as an example the following creates a dataframe based on the content of a json file read a json document named cars_price json with the following content and generate a table based on the schema in the json document itemno 1 name ferrari price 52000000 kph 6 1 itemno 2 name jaguar price 15000000 kph 3 4 itemno 3 name mercedes price 10000000 kph 3 itemno 4 name audi price 5000000 kph 3 6 itemno 5 name lamborghini price 5000000 kph 2 9 def main args array string val spark sparksession builder master local appname spark2 0 getorcreate val df spark read option header true json resources cars_price json val selecteddata df select name price selecteddata write option header true save s src main resources uuid randomuuid println ok loading and saving txt file as an example the following creates a data frame based on the content of a text file read a text document named rklick txt with the following content and generate a table based on the schema in the text document rklick creative technology company providing key digital services focused on helping our clients to build a successful business on web and mobile we are mature and dynamic solution oriented it full service product development and customized consulting services company since 2007 we re dedicated team of techno whiz engaged in providing outstanding solutions to diverse group of business entities across north america known as professional technologists we help our customers enable latest technology stack or deliver quality cost effective products and applications def main args array string val spark sparksession builder master local appname spark2 0 getorcreate import spark implicits _ val rklickdata spark read text src main resources rklick txt as string val rklickwords rklickdata flatmap value value split s val savetxt rklickwords write text s src main resources uuid randomuuid println ok we would look at how we can create more useful tutorials to grow it then we would be adding more content to it together if you have any suggestion feel free to suggest us stay tuned 1 comment tutorial quick overview of spark 2 0 1 core functionality november 4 2016 november 4 2016 ved prakash singh dataframe dataframe api dataset rdd scala spark spark 1 6 spark 2 0 spark core spark session sparksql typesafe in this blog we will discuss about spark 2 0 1 core functionality it demonstrates the basic functionality of spark 2 0 1 we also describe sparksession spark sql and dataframe api functionality we have tried to cover basics of spark 2 0 1 core functionality and sparksession sparksession sparksession is new entry point of spark in previous version 1 6 x of spark spark context was entry point for spark and in spark 2 0 1 sparksession is entry point of spark spark session internally has a spark context for actual computation as we know rdd was main api it was created and manipulated using context api s for every other api we needed to use different contexts for streaming we needed streamingcontext for sql sqlcontext and for hive hivecontext but as dataset and dataframe api s are becoming new standard api s we need an entry point build for them so in spark 2 0 1 we have a new entry point for dataset and dataframe api s called as spark session creating sparksession here we describe how to create sparksession val spark sparksession builder master local appname spark example getorcreate once we have created spark session then we can use it to read the data read data using spark session it looks like exactly like reading using sqlcontext you can easily replace all your code of sqlcontext with sparksession now val spark sparksession builder master local appname spark example getorcreate creating dataframes how to create dataframes with the help of sparksession applications can create dataframes from an existing rdd from a from data sources as an example the following creates a dataframe based on the content of a csv file val dataframe spark read option header true csv src main resources team csv creates a dataframe based on the content of a json file val dataframe spark read option header true json src main resources cars_price json dataframe show creating datasets dataset is new abstraction in spark introduced as alpha api in spark 1 6 it s becoming stable api in spark 2 0 1 datasets are similar to rdds however instead of using java serialization or kryo they use a specialized encoder to serialize the objects for processing or transmitting over the network the major difference is dataset is collection of domain specific objects where as rdd is collection of any object domain object part of definition signifies the schema part of dataset so dataset api is always strongly typed and optimized using schema where rdd is not dataset definition also talks about dataframes api dataframe is special dataset where there is no compilation checks for schema so this makes dataset new single abstraction replacing rdd from earlier versions of spark we read data using read text api which is similar to textfile api of rdd the following creates a dataframe based on the content of a txt file import spark implicits _ val rklickdata spark read text src main resources rklick txt as string val rklickwords rklickdata flatmap value value split s val rklickgroupedwords rklickwords groupbykey _ tolowercase val rklickwordcount rklickgroupedwords count rklickwordcount show we would look at how we can create more useful tutorials to grow it then we would be adding more content to it together if you have any suggestion feel free to suggest us stay tuned 1 comment elasticsearch character filter october 25 2016 rklick solutions llc elastic search es elasticsearch es in this post i am going to explain how elasticsearch character filter work so there are following steps to done this step 1 set mapping for your index suppose our index name is testindex and type is testtype now we are going to set analyzer and filter curl xput localhost 9200 testindex d settings analysis char_filter quotes type mapping mappings and analyzer gramanalyzer type custom tokenizer whitespace char_filter quotes filter lowercase whitespaceanalyzer type custom tokenizer whitespace filter lowercase mappings testtype _all analyzer gramanalyzer search_analyzer gramanalyzer properties name type string include_in_all true analyzer gramanalyzer search_analyzer gramanalyzer store true character filters are used to preprocess the string of characters before it is passed to the tokenizer a character filter may be used to strip out html markup or to convert characters to the word and as you seen above we have set a filter to convert to and step 2 index your data now we are going to index data containing character but we want to fetch data from and curl xpost localhost 9200 testindex testtype 1 d name karra john step 3 check how analyzer work for your index data curl http localhost 9200 testindex _analyze analyzer gramanalyzer d karra john step 4 fetch data from match query now i want to fetch data from match query as you have seen above indexing data is karra john but now we would fetch those data from karra john see below query curl xget http localhost 9200 testindex testtype _search d query match name query karraandjohn analyzer gramanalyzer you can also do like this curl xget http localhost 9200 testindex testtype _search d query match name query karraandjohn step 5 fetch data from filter pre curl xget http localhost 9200 testindex testtype _search d query filtered filter term name karraandjohn result would like this took 1...