Meta tags:

Headings (most frequently used words):

to, 2016, how, tuesday, may, 2015, blog, all, test, ip, for, spark, my, note, on, solutions, pages, monday, october, 24, wednesday, august, 10, sunday, july, 31, 17, december, 15, thursday, 21, search, this, blogs, archive, pacemaker, create, virtual, lb, purpose, nodes, locate, magento, credit, card, leakge, group, established, connections, by, target, window, function, failure, union, expected, but, found, run, testing, application, in, your, fav, python, ide, elasticsearch, geolocation, support, http, chunked, transfer,

Text of the page (most frequently used words):
the (91), and (36), this (23), you (23), share (21), all (18), node1 (17), can (16), for (15), may (11), spark (11), using (10), see (9), here (9), from (9), file (9), how (8), #create (8), apache (8), then (8), use (8), sql (8), team (8), nodes (8), corosync (8), december (7), test (7), pinterest (7), facebook (7), blogthis (7), email (7), comments (7), posted (7), are (7), http (7), some (7), setup (7), data (7), print (7), pyspark_submit_args (7), sort (7), april (6), august (6), october (6), once (6), have (6), run (6), spark_home (6), scala (6), awk (6), just (6), now (6), july (5), pacemaker (5), 2016 (5), ryan (5), call (5), after (5), try (5), out (5), org (5), there (5), like (5), want (5), environ (5), path (5), sqlcontext (5), division (5), score (5), one (5), locate (5), credit (5), card (5), node2 (5), 192 (5), 168 (5), etc (5), june (4), september (4), november (4), january (4), february (4), march (4), windows (4), solr (4), node (4), end (4), get (4), response (4), will (4), chunked (4), check (4), with (4), within (4), found (4), error (4), your (4), python (4), pyspark (4), shell (4), very (4), rank (4), grep (4), hacker (4), special (4), post (4), hosts (4), they (4), request (4), gif (4), purpose (4), vagrant (4), node3 (4), enable (4), systemctl (4), pcs (4), 2015 (3), source (3), ssh (3), net (3), smart (3), traffic (3), transfer (3), about (3), encoding (3), find (3), quick (3), index (3), rest (3), query (3), looks (3), mapping (3), client (3), copy (3), put (3), parse (3), following (3), need (3), spark_release_file (3), sys (3), not (3), order (3), got (3), west (3), whether (3), connection (3), server (3), netstat (3), est (3), target (3), uniq (3), any (3), files (3), change (3), time (3), config (3), always (3), nodeone (3), bash (3), done (3), disable (3), cluster (3), vip (3), virtual (2), blog (2), wcf (2), tutorial (2), nutch (2), soapui (2), script (2), php (2), performance (2), open (2), linux (2), java (2), asp (2), archiva (2), search (2), posts (2), home (2), tell (2), seconds (2), delay (2), inspect (2), between (2), simple (2), code (2), rfc2616 (2), sec3 (2), content (2), earthquake (2), api (2), couple (2), return (2), earthquakes (2), count (2), aggregated (2), range (2), basically (2), indexing (2), define (2), sense (2), when (2), bulk (2), called (2), csv (2), tuesday (2), application (2), ide (2), essentially (2), load (2), module (2), folder (2), take (2), example (2), lib (2), setupspark (2), parallelize (2), good (2), insert (2), bin (2), testing (2), window (2), syntax (2), failure (2), union (2), expected (2), but (2), select (2), over (2), partition (2), desc (2), package (2), catalyst (2), abstractsparksqlparser (2), anonfun (2), apply (2), 211 (2), east (2), that (2), cross (2), endpoints (2), established (2), column (2), command (2), tcp (2), only (2), customer (2), turns (2), analysis (2), certificate (2), which (2), 3rd (2), party (2), nothing (2), most (2), more (2), must (2), intercept (2), store (2), did (2), magento (2), tab (2), footer_bg (2), was (2), minutes (2), log (2), backdoor (2), capture (2), hostname (2), configure (2), box (2), box_url (2), 101 (2), true (2), deploy (2), value (2), entry (2), echo (2), install (2), pcsd (2), start (2), firewalld (2), conf (2), haclusteruser (2), auth (2), crm_mon (2), status (2), stonith (2), again (2), note (2), solutions (2), skip (2), 2010, 2011, 2012, 2013, 2014, archive, xml, wordpress, wmic, phone, management, azure, windbg, asmx, varnish, utility, troubleshotting, tomcat, tika, streaminsight, sqlserver, 2008, soapbox, silverlight, service, security, searching, repo, quickstart, postgresql, patch, oracle, coherence, opencart, solution, trigger, mvn, mvc, mod_jk, maven, macbook, pro, lucene, log4j, putty, jmx, jconsole, j2ee, j233, iptables, ipad, iis, hadoop, greenplum, git, free, tools, firewall, dead, loop, controller, cloud, computing, cassandra, camel, camle, android, bit, programming, remoting, blogs, subscribe, atom, older, wireshark, enough, frames, invloved, req, res, trailer, okay4, 2nd, returns2, 1st, returns, okay1, wirelevel, bits, pull, wget, notice, okay3, write, full, spec, www, protocols, html, zipped, resources, things, assure, length, header, ins, also, thursday, assign, poi, miles, those, happened, near, san, diego, 300kms, results, showing, individual, buckets, grad, usgs, closed, first, used, postman, best, other, plugin, chome, give, paste, convered, importer, new, document, doc, itself, feed, small, json, format, tool, elasticsearch, geolocation, support, step, dependency, environment, context, grab, under, standard, distribution, project, bootstrap, care, environments, import, exists, read, join, sparkcontext, py4j, src, zip, release, master, local, users, development, hadoop2, self, def, object, class, fav, weird, ranking, perspective, fix, please, make, sure, hivecontext, instead, protocol, py4jjavaerror, occurred, while, calling, o36, lang, runtimeexception, defaultparserdialect, parserdialect, errors, complaining, createdataframe, map, row, registerastable, lambda, raptors, sas, mia, heat, golden, state, lakers, function, common, case, even, distributed, system, side, send, requests, basic, idea, show, 4th, 5th, many, each, fnr, top, rows, hint, group, connections, sunday, chance, help, leaking, issues, who, complaints, leak, made, purchase, steps, issue, tcpdump, sending, smtp, port, stand, https, tls, represents, evil, site, shipping, rate, calculation, fedex, integration, url, has, huge, hit, same, handy, aggregation, requestlog, hits, crawlers, google, feel, little, big, frustrated, somewhere, definitly, where, she, root, directly, spect, mintues, placed, dummy, mmin, changed, ago, should, chagne, complicated, image, since, stored, encrypted, access, lot, week, coming, containing, global, encrypt, his, her, rsa, public, key, nobody, decrypt, ride, housekeeping, inject, dum, watch, hours, leakge, wednesday, sometimes, than, lbs, floatip, virual, pacepaker, corosyncd, easily, booted, machine, well, figured, nodetwo, nodethree, vagrantfile_api_version, network, private_network, synced_folder, disabled, bash_env, profile, exec, provision, provider, virtualbox, customize, modifyvm, memory, 1300, name, gui, 102, 103, vms, ready, vagran, dependencies, yum, daemon, automatically, topology, quorum, settings, dfault, mcast, maintain, membership, user, named, among, you_special_compliceted_password, passwd, stdin, authorize, eacy, host, shared, credential, floating, resource, vip2, ipaddr2, 100, cidr_netmask, nic, enp0s8, monitor, interval, 30s, might, stopped, property, set, enabled, false, bound, shutdown, assigned, monday, contact, tip, pages, sidebar, main,

Text of the page (random words):
my note on solutions skip to main skip to sidebar pages home contact tip me my note on solutions monday october 24 2016 how to test pacemaker to create a virtual ip for lb purpose for 3 nodes sometimes you need have more than one lbs for ha purpose and we always want to have one floatip virual ip cross all lb nodes and we can use pacepaker corosyncd to do this very easily for the testing purpose i will use vagrant to create 3 nodes node1 node2 node3 once booted up all 3 machine are well figured for hostname nodeone node1 nodetwo node2 nodethree node3 vagrant configure vagrantfile_api_version do config node 1 config vm define node1 do node1 node1 vm box box node1 vm box_url box_url node1 vm network private_network ip 192 168 2 101 node1 vm hostname nodeone node1 vm synced_folder vagrant disabled true node1 ssh shell bash c bash_env etc profile exec bash node1 vm provision shell path post deploy sh run always node1 vm provider virtualbox do v v customize modifyvm id memory 1300 v name nodeone v gui true end end and post deploy sh bin bash value grep ic entry etc hosts if value eq 0 then echo hosts entry 192 168 2 101 node1 192 168 2 102 node2 192 168 2 103 node3 etc hosts fi once done you can call vagrant up node1 node2 node3 to have 3 vms ready now vagran ssh all nodes to install the dependencies yum install y pcsd pacemaker corosync after done enable all daemon to start automatically systemctl enable pcsd systemctl enable pacemaker systemctl enable corosync disable firewalld systemctl disable firewalld now time to configure the corosync for nodes topology you can just copy the etc corosync corosync conf example to etc corosync corosync conf just change your ip and quorum settings by dfault it use mcast to maintain the cluster membership then create a user named haclusteruser among 3 nodes echo you_special_compliceted_password passwd stdin haclusteruser once done you can call pcs cluster auth to authorize eacy host using the shared credential pcs cluster auth node1 after this start the corosync and pacemaker on all nodes run crm_mon to see all the nodes now we can create a floating vip to ha purpose pcs resource create vip2 ipaddr2 ip 192 168 2 100 cidr_netmask 32 nic enp0s8 op monitor interval 30s after that you might see the status is always stopped we need to disable the stonith pcs property set stonith enabled false try the check again for the status the vip is bound to one node node1 here to test the ha shutdown node1 and run crm_mon again the vip was assigned on node2 posted by ryan at 11 58 am 15 comments email this blogthis share to x share to facebook share to pinterest wednesday august 10 2016 how to locate magento credit card leakge just got a chance to help locate the leaking issues of a customer who got complaints from customer about credit card leak after made purchase and it turns out to be a very smart hacker here is the steps i try to locate the issue i use tcpdump to capture all traffic for a couple hours and do a quick analysis to see whether there are some special like sending out credit card using smtp with the port 25 or just use stand http post if it https you can sort the tls certificate to see any special certificate which represents some evil 3rd party hosts nothing found special for me most just traffic to this site and some 3rd party api call like shipping rate calculation fedex integration then do a analysis about the request log t o see any url which has huge hit from the same ip you can use some handy linux command to do the aggregation awk print 8 2 requestlog sort uniq c sort r n more nothing special most hits are from crawlers ip with google feel a little big frustrated now this must be a smart hacker then i watch for file change within 5 minutes if they hacker intercept the request and store somewhere i will definitly find out where did he she store the file go the root directly of the magento i spect all files change within 5 mintues i placed a order using a dummy credit card find mmin 5 ls there we go one file called db tab footer_bg gif was changed minutes ago there should not be any chagne for gif files this turns out to be a complicated image file since it s all stored with encrypted data it must be credit card there if i check the access log not a lot request to this file only once a week coming from the hacker s ip now time to locate how can they capture the data and dum p to this file just search all php files containing db tab footer_bg gif it s in the global config file here is the content essentially they intercept all post request and encrypt using his her rsa public key and put to the gif file nobody can decrypt it now time to get ride of the backdoor and do the housekeeping to locate how did they inject the backdoor on the server posted by ryan at 11 47 am 9 comments email this blogthis share to x share to facebook share to pinterest sunday july 31 2016 group all established connections by target ip here is a very common case that you want to check whether you have even connection for a distributed system on the server side or whether the client send the requests cross all server endpoints basic idea netstat an grep est will show you all the established connection so the 4th column is the client endpoints and 5th column is the target one if we want to see how many connection for each target as a ip you can do the following command netstat an p tcp grep est awk print 5 awk f print 1 2 3 4 sort uniq c sort n r and if we only want to see the top 5 we can put rows index for the awk hint netstat an p tcp grep est awk print 5 awk f fnr 6 print 1 2 3 4 sort uniq c sort n r posted by ryan at 2 26 pm 1 comments email this blogthis share to x share to facebook share to pinterest tuesday may 17 2016 spark window function failure union expected but found this a very weird error when i try to run a simple window ranking all looks good from the syntax perspective team lakers west 29 golden state west 89 mia heat east 79 sas west 9 raptors east 29 sql createdataframe sc parallelize team map lambda x row team x 0 division x 1 score x 2 registerastable team print sql sql select team division score rank over partition by division order by score desc as rank from team take 10 and i got this errors complaining the syntax 4j protocol py4jjavaerror an error occurred while calling o36 sql java lang runtimeexception 1 43 failure union expected but found select team division score rank over partition by division order by score desc as rank from team at scala sys package error package scala 27 at org apache spark sql catalyst abstractsparksqlparser parse abstractsparksqlparser scala 36 at org apache spark sql catalyst defaultparserdialect parse parserdialect scala 67 at org apache spark sql sqlcontext anonfun 2 apply sqlcontext scala 211 at org apache spark sql sqlcontext anonfun 2 apply sqlcontext scala 211 to fix this please make sure you are using hivecontext instead of sqlcontext posted by x y z at 5 18 pm 7 comments email this blogthis share to x share to facebook share to pinterest how to run spark testing application in your fav python ide here is a quick step to run and test your spark application using python ide essentially we need load the dependency module setup the environment and load the context 1 copy and grab pyspark folder under the standard spark distribution to your project folder 2 setup some bootstrap to take care the environments using the following code i use 1 6 1 as an example and you may create this as a module class setup object def setupspark self os environ spark_home users and development spark spark 1 6 1 bin hadoop2 6 os environ pyspark_submit_args master local 2 spark_home os environ get spark_home spark_release_file spark_home release if os path exists spark_release_file and spark 1 6 1 in open spark_release_file read pyspark_submit_args os environ get pyspark_submit_args if not pyspark shell in pyspark_submit_args pyspark_submit_args pyspark shell os environ pyspark_submit_args pyspark_submit_args sys path insert 0 spark_home python sys path insert 0 os path join spark_home python lib py4j 0 9 src zip return pyspark sparkcontext 3 you are good to go from lib setup import setup sc setup setupspark print sc parallelize range 1 10 count posted by x y z at 5 11 pm 8 comments email this blogthis share to x share to facebook share to pinterest tuesday december 15 2015 how to test elasticsearch geolocation support here is a quick tutorial to setup es 2 1 and index some earthquake data by using the rest api and then do by query by assign the poi within couple miles and return those earthquakes count aggregated by range the query looks like this basically query all the earthquake happened near san diego within 300kms and the results showing the individual earthquakes and the buckets aggregated you can grad the data source from usgs i closed the csv file here to do the indexing first we define the mapping i used postman to do the rest call my mapping looks like this once we have the mapping and index we can feed the data to es using the sense tool i put a small script to parse the csv data to json following the bulk format like this and i found sense is the best client when you do the bulk indexing all other rest plugin in chome give me the encoding error then copy and paste the data we convered basically tell the importer we want to create a new document called eq then the doc itself posted by ryan at 11 07 am 0 comments email this blogthis share to x share to facebook share to pinterest thursday may 21 2015 http 1 1 chunked transfer if you inspect the http response with some zipped resources you may find 2 things i assure we are on http 1 1 there is no content length header ins the response also the transfer encoding is chunked and http 1 1 have a full spec about the chunked encoding check it here http www w3 org protocols rfc2616 rfc2616 sec3 html sec3 6 if you use node js we can write a simple test to try out the chunked transfer here is the source code and try pull the response using wget you will notice the 3 seconds delay between okay3 and 4 and inspect the traffic in between you can see the wirelevel bits 1st response returns okay1 then the 2nd returns2 and 3 after 3 seconds delay we get okay4 once we call the end you can see the end trailer wireshark are smart enough to tell you 4 frames invloved in this http req res posted by ryan at 12 28 pm 21 comments email this blogthis share to x share to facebook share to pinterest older posts home subscribe to posts atom search this blog all blogs net remoting 1 32 64 bit programming 1 android 11 apache 2 apache archiva 1 apache camle 2 apache solr 1 archiva 1 asp net 1 asp net 4 0 2 camel 2 cassandra 1 cloud computing 1 controller 1 dead loop 1 firewall 1 free tools 4 git 1 greenplum 1 hadoop 1 iis 7 7 5 2 ipad 1 iptables 1 j233 2 j2ee 2 java performance jmx jconsole 6 linux putty ssh 1 log4j 1 lucene solr 1 macbook pro 1 maven 4 mod_jk 2 mvc 1 mvn 1 node js trigger 1 nutch 1 open source solution 8 opencart 1 oracle coherence 9 patch 1 performance 1 php 1 postgresql 2 quickstart 2 repo 1 script 1 searching 1 security 2 service 1 silverlight 1 soapui 2 soapui soapbox 2 solr 3 solr nutch 1 sqlserver 2008 r2 1 streaminsight 2 tika 1 tomcat 5 troubleshotting 7 tutorial 5 utility 6 varnish 1 wcf 3 wcf asmx test 2 windbg 3 windows 7 2 windows azure 1 windows management 1 windows phone 7 2 wmic 1 wordpress 1 xml 1 blog archive 2016 5 october 1 how to test pacemaker to create a virtual ip for l august 1 july 1 may 2 2015 14 december 1 may 4 april 2 march 2 february 3 january 2 2014 21 december 4 november 1 october 4 september 3 may 1 april 2 march 6 2013 25 december 2 september 6 august 3 june 1 may 4 april 3 february 4 january 2 2012 56 december 3 november 3 october 3 september 5 august 5 july 4 june 8 may 8 april 1 march 5 february 9 january 2 2011 61 december 4 november 3 october 4 august 5 july 2 june 4 may 7 april 5 march 8 february 13 january 6 2010 94 december 6 november 9 october 12 september 17 august 14 july 15 june 6 may 13 april 2