Meta tags:

Headings (most frequently used words):

kong, and, aws, 2017, security, to, ec2, groups, with, the, api, using, install, thursday, friday, blog, for, modifying, via, lambda, functions, adding, an, rate, plugin, in, logs, elasticsearch, apache, example, ip, create, start, dashboard, agile, testing, january, 11, 2018, december, 29, november, 24, monday, july, 31, june, 01, search, this, about, me, archive, labels, articles, tutorials, creative, commons, license, passion, healing, followers, experiences, gateway, installing, on, ubuntu, 16, 04, limiting, putting, load, balancer, front, of, syslog, cloudwatch, log, aggregation, visualization, authentication, whitelisting, scenarios, ssl, termination, http, caching, haproxy, varnish, report, abuse, checking, dynamic, dns, replacing, it, group, new, cidr, block, several, from, deb, package, postgresql, database, user, modify, configuration, run, migrations, job, increase, open, files, limit, systemd, service, basic, auth, verifying, that, our, endpoint, is, limited, sending, kibana,

Text of the page (most frequently used words):
the (143), kong (84), and (58), for (55), log (35), this (33), aws (33), logs (33), http (32), you (30), group (30), api (30), port (28), etc (28), that (26), com (24), haproxy (23), cloudwatch (23), #varnish (21), ec2 (20), #lambda (20), with (20), ssl (20), client (19), permission (19), role (18), name (17), share (16), december (16), january (16), security (16), service (16), function (16), november (15), https (15), install (15), june (14), july (14), 2017 (14), here (14), from (14), can (14), allow (14), access (14), dashboard (14), minute (14), stage (14), mydomain (14), groups (13), use (13), default (13), are (13), apache (13), elasticsearch (13), february (12), march (12), april (12), october (12), via (12), server (12), conf (12), user (12), ratelimit (12), may (11), august (11), all (11), list (11), var (11), content (11), create (11), rate (11), groupid (11), september (10), one (10), then (10), which (10), iam (10), agent (10), new (10), to_add (10), result (10), start (9), file (9), added (9), version (9), kong1 (9), str (9), import (9), ubuntu (8), using (8), configure (8), type (8), authentication (8), require (8), kibana (8), created (8), policy (8), index (8), limiting (8), post (8), json (8), endpoint (8), limit (8), cidr_blocks (8), will (7), adding (7), cidr (7), web (7), traffic (7), set (7), errorfile (7), errors (7), your (7), more (7), admin (7), directory (7), cluster (7), awslogs (7), query (7), remaining (7), return (7), def (7), len (7), ippermissions (7), cidr_block (7), load (6), blog (6), scenario (6), also (6), add (6), 443 (6), backend (6), rsyslog (6), restart (6), systemctl (6), syslog (6), mysite (6), www (6), see (6), key (6), get (6), have (6), basic (6), called (6), filters (6), arn (6), step (6), cwl (6), gateway (6), want (6), actual (6), was (6), response (6), plugin (6), connection (6), keep (6), alive (6), target (6), url (6), print (6), append (6), update_security_group (6), new_ip_address (6), home (5), example (5), our (5), source (5), pinterest (5), facebook (5), blogthis (5), email (5), comments (5), when (5), front (5), need (5), files (5), after (5), 8080 (5), apache2 (5), pem (5), notice (5), used (5), running (5), configuration (5), block (5), first (5), should (5), associated (5), resource (5), west (5), domain (5), action (5), effect (5), event (5), dec (5), updated (5), httpie (5), calls (5), values (5), application (5), deb (5), dns (5), update (5), ssh (5), toport (5), site (4), some (4), send (4), testing (4), these (4), again (4), few (4), but (4), nginx (4), ports (4), 8888 (4), systemd (4), daemon (4), hosts (4), 172 (4), request (4), bind (4), timeout (4), ciphers (4), ecdh (4), run (4), cat (4), well (4), whitelist (4), other (4), instances (4), region (4), chose (4), above (4), statement (4), pattern (4), now (4), format (4), call (4), upstream (4), latency (4), date (4), gmt (4), length (4), accept (4), encoding (4), alb (4), points (4), requests (4), 8443 (4), true (4), value (4), description (4), jenkins (4), fromport (4), cidrip (4), ssh_group (4), web_group (4), boto3 (4), articles (3), performance (3), agile (3), 2012 (3), modifying (3), 2018 (3), what (3), plone (3), amazon (3), infrastructure (3), how (3), many (3), blocks (3), various (3), functions (3), non (3), edit (3), frontend (3), enable (3), listen (3), stop (3), reload (3), change (3), 204 (3), aesgcm (3), 3des (3), rsa (3), certbot (3), auto (3), let (3), encrypt (3), apt (3), following (3), snippet (3), html (3), any (3), addresses (3), local (3), such (3), attached (3), python (3), make (3), stream (3), same (3), streaming (3), note (3), timestamp (3), field (3), able (3), accountid (3), execution (3), ipaddress (3), its (3), setup (3), done (3), none (3), choose (3), enter (3), choice (3), object (3), contains (3), point (3), apis (3), data (3), 200 (3), host (3), headers (3), specifies (3), 60000 (3), 8001 (3), job (3), postgres (3), help (3), community (3), edition (3), dynamic (3), tagged (3), get_security_groups_for_update (3), ipprotocol (3), add_permissions (3), ssh_updated (3), web_updated (3), ingress_ports (3), lambda_handler (3), copy (3), new_permission (3), ip_range (3), under (2), creative (2), commons (2), license (2), tutorials (2), written (2), 2008 (2), write (2), search (2), selenium (2), test (2), box (2), large (2), scale (2), experiences (2), stress (2), times (2), task (2), comes (2), removing (2), updating (2), posts (2), much (2), needs (2), both (2), cached (2), achieve (2), cfg (2), udp (2), logging (2), localhost (2), lib (2), system (2), input (2), imudp (2), check (2), default_backend (2), reqadd (2), forwarded (2), proto (2), header (2), offloaded (2), crt (2), 504 (2), 503 (2), 502 (2), 500 (2), 408 (2), 403 (2), 400 (2), 50000 (2), connect (2), option (2), mode (2), global (2), aes256 (2), aes (2), servers (2), base (2), private (2), stats (2), dev (2), letsencrypt (2), live (2), webroot (2), path (2), wget (2), certificates (2), thursday (2), examples (2), wanted (2), notes (2), scenarios (2), whitelisting (2), they (2), contents (2), ones (2), useful (2), challenge (2), requireany (2), valid (2), noauth (2), authuserfile (2), htpasswd (2), authname (2), restricted (2), authtype (2), order (2), deny (2), specific (2), urls (2), includeoptional (2), through (2), network (2), 127 (2), aggregation (2), visualization (2), either (2), went (2), actions (2), instance (2), attach (2), replace (2), script (2), where (2), error (2), each (2), streams (2), hostname (2), sending (2), had (2), existing (2), subscription (2), indices (2), form (2), entries (2), visualize (2), principal (2), initial (2), specified (2), not (2), say (2), installing (2), latest (2), configuring (2), upload (2), custom (2), curl (2), putlogevents (2), createlogstream (2), createloggroup (2), allowing (2), friday (2), open (2), putting (2), microservices (2), because (2), logged (2), replacing (2), every (2), message (2), per (2), err (2), 21633901644387004745 (2), proxy (2), fri (2), gzip (2), deflate (2), uses (2), things (2), proper (2), certificate (2), balancer (2), transfer (2), chunked (2), sat (2), charset (2), utf (2), 429 (2), reply (2), pass (2), verify (2), documentation (2), plugins (2), graphql (2), pub (2), multi (2), auth (2), nodesource_setup (2), migrations (2), somepassword (2), modify (2), database (2), psql (2), postgresql (2), xenial (2), periodically (2), appropriate (2), rules (2), within (2), updatesecuritygroupwithhomeip (2), permissions (2), several (2), does (2), invocation (2), ipaddressblock (2), securitygroups (2), describe_security_groups (2), tag (2), security_group_tag (2), authorize_security_group_ingress (2), add_params (2), ipranges (2), old_prefixes (2), sshsecuritygroups (2), websecuritygroups (2), found (2), security_group_tag_for_ssh (2), security_group_tag_for_web (2), update_security_groups (2), context (2), lambdaupdate (2), urllib2 (2), hashlib (2), grep (2), lookup (2), deep (2), ip_ranges (2), security_group_id (2), travel, theme, powered, blogger, followers, tweet, visit, passion, healing, work, licensed, attribution, noncommercial, alike, presentations, over, years, report, abuse, sensu, httperf, openwebload, openvpn, monitoring, humor, cloud, buildbot, trac, labels, 2004, 2005, 2006, 2007, 2009, 2010, 2011, 2013, 2014, 2015, 2016, funct, archive, view, complete, profile, like, bitcoin, 1nqzg1dzt2jaq5prrvjsdwijzbvytfqduk, grig, gheorghiu, about, give, out, installation, customiza, part, openx, recently, completed, deployment, farms, lessons, learned, deploying, good, interview, question, tester, define, people, terms, inte, infrastructur, subscribe, atom, older, common, requirement, setting, development, staging, try, mimic, production, possible, implemented, since, handle, encrypted, terminated, before, hits, fairly, easy, way, terminate, forwarding, unencrypted, forwards, steps, sections, section, pointing, uncommenting, lines, management, flag, virtual, 514, module, provides, reception, redirect, scheme, ssl_fc, 5000, dontlognull, httplog, defaults, tune, param, 2048, options, sslv3, aes128, anull, md5, dss, hynek, hardening, information, 1ssl, enabled, listening, sockets, certs, material, locations, 30s, socket, sock, 660, level, chroot, local1, local0, fullchain, privkey, generate, combined, chain, certonly, chmod, eff, org, termination, caching, scattered, among, installations, gather, benefit, hopefully, others, depict, requirements, similar, env, setenvif, request_uri, known, acme, satisfy, authbasicprovider, office, ips, except, whitelisted, networks, monday, searching, alternative, rolling, own, elk, stack, 3rd, party, saas, solution, logentries, loggly, papertrail, expensive, splunk, sumo, logic, pointers, settings, followed, instructions, downloaded, ran, continued, types, sure, configured, indeed, sent, inside, multiple, having, those, pre, once, subsequent, reuse, important, modified, last, look, recognize, correctly, discover, screen, streamed, awslambdabasicexecutionrole, sourceip, condition, based, allowed, follow, prompts, console, writes, launching, interactive, pip, downloading, bits, cli, secret, output, destination, oct, 2000, position, end, amazonaws, downloads, describelogstreams, read, premise, don, cannot, benefits, endpoints, better, offers, experimented, bit, writing, down, future, reference, node, presentation, world, webinar, just, scratching, surface, resources, clashing, another, already, present, ended, doing, copying, transform, process, described, difference, previous, portion, entry, mac, operation, lots, means, specify, limits, units, time, details, programname, syslogseverity, text, log_level, client_errors_severity, successful_severity, server_errors_severity, 8000, plain, second, calling, standard, numbers, gets, cumbersome, cleaner, put, manager, listener, two, exceeded, 11th, quick, succession, related, passing, string, along, making, exposes, working, issuing, verifying, limited, config, john, nikolai, official, control, origin, created_at, 1513401906560, http_if_terminated, https_only, ad23a91a, b76a, 417d, 9889, 0088a13e3419, preserve_host, retries, strip_uri, upstream_connect_timeout, upstream_read_timeout, upstream_send_timeout, upstream_url, uris, total, brew, instead, far, concerned, anything, comma, separated, later, conjunction, wantedby, execstart, usr, bin, someadminuser, someadminpassword, unit, wants, npm, nodejs, bash, nodesource, setup_, ulimit, 4096, increase, pg_database, pg_password, pg_user, pg_port, 5432, pg_host, alter, password, owner, dpkg, bintray, download_file, file_path, dists, package, automated, fully, partially, simple, arises, without, static, still, automatically, their, named, achieves, goal, observations, allows, them, purpose, acts, adds, parameter, case, build, ip1, extend, iteritems, tcp, else, count, tags, identify, inbound, dig, homeip, awk, invoke, requestresponse, tail, payload, outputfile, txt, revokesecuritygroupingress, authorizesecuritygroupingress, describesecuritygroups, finally, revoke, old, authorize, copied, range, has, property, corresponding, iterate, objects, given, trivial, lookups, preferred, caller, resulting, address, sole, argument, retrieved, revoke_security_group_ingress, deepcopy, groupids, security_rule_descr, rule, xxxx, checking, did, anybody, webscale,

Text of the page (random words):
ippermissions add_params return len to_add def get_security_groups_for_update client security_group_tag filters list for key value in security_group_tag iteritems filters extend name tag key values key name tag value values value response client describe_security_groups filters filters return response securitygroups this function acts on security groups tagged web and ssh for the ones tagged web it adds new rules allowing the ip cidr block access to ports 80 and 443 for the groups tagged ssh it does the same but for port 22 the input for this function is ip1 ipaddressblock where ipaddressblock is a jenkins parameter that the user specifies when running the appropriate jenkins job in this case i used the aws lambda invocation build step in jenkins at january 11 2018 no comments email this blogthis share to x share to facebook share to pinterest friday december 29 2017 experiences with the kong api gateway kong is an open source api gateway that you can install on premise if you don t want or cannot use the aws api gateway the benefits of putting an api gateway in front of your actual api endpoints are many rate limiting authentication security better logging etc kong offers all of these and more via plugins i ve experimented with kong a bit and i am writing down my notes here for future reference installing kong on ubuntu 16 04 install kong from deb package wget https bintray com kong kong community edition deb download_file file_path dists kong community edition 0 11 2 xenial all deb dpkg i kong community edition 0 11 2 xenial all deb install postgresql apt get install postgresql create kong database and user su postgres psql psql 10 1 type help for help postgres create user kong create database kong owner kong postgres alter user kong password somepassword modify kong configuration cp etc kong kong conf default etc kong kong conf edit kong conf and set pg_host 127 0 0 1 pg_port 5432 pg_user kong pg_password somepassword pg_database kong run kong migrations job kong migrations up increase open files limit ulimit n 4096 start kong kong start install kong dashboard install updated version of node first curl sl https deb nodesource com setup_ 6 x o nodesource_setup sh bash nodesource_setup sh apt get install nodejs npm install g kong dashboard create systemd service to start kong dashboard with basic auth i specified port 8001 for kong dashboard instead of the default port 8080 cat etc systemd system multi user target wants kong dashboard service unit description kong dashboard after network target service execstart usr local bin kong dashboard start kong url http localhost 8001 basic auth someadminuser someadminpassword install wantedby multi user target systemctl daemon reload systemctl start kong dashboard adding an api to kong i used the kong admin dashboard to add a new api gateway object which points to the actual api endpoint i have note that the url name as far as kong is concerned can be anything you want in this example i chose kong1 the upstream url is your actual api endpoint the hosts value is a comma separated list of the host headers you want kong to reply to here is contains the domain name for my actual api endpoint stage mydomain com as well as a new domain name that i will use later in conjunction with a load balancer in front of kong stage api mydomain com name my kong api hosts stage mydomain com stage api mydomain com urls kong1 upstream url https stage mydomain com api graphql pub at this point you can query the kong admin endpoint for the existing apis we ll use httpie on a mac you can install httpie via brew install httpie http http stage mydomain com 8001 apis http 1 1 200 ok access control allow origin connection keep alive content type application json charset utf 8 date fri 29 dec 2017 16 23 59 gmt server kong 0 11 2 transfer encoding chunked data created_at 1513401906560 hosts stage api mydomain com stage mydomain com http_if_terminated true https_only true id ad23a91a b76a 417d 9889 0088a13e3419 name my kong api preserve_host true retries 5 strip_uri true upstream_connect_timeout 60000 upstream_read_timeout 60000 upstream_send_timeout 60000 upstream_url https stage mydomain com api graphql pub uris kong1 total 1 note that any operation that you do via the kong dashboard can also be done via api calls to the kong admin backend for lots of examples on how to do that see the official documentation as well as this blog post from john nikolai using the kong rate limiting plugin in the kong admin dashboard go to plugins add plugin name rate limiting config minute 10 this means that we are limiting the rate of calls to our api to 10 per minute you can also specify limits per other units of time see the rate limiting plugin documentation for more details verifying that our api endpoint is rate limited now we can verify that the api endpoint is working by issuing a post request to the kong url kong1 which points to our actual api endpoint http verify no v post https stage mydomain com 8443 kong1 query version post kong1 query version http 1 1 accept accept encoding gzip deflate connection keep alive content length 0 host stage mydomain com 8443 user agent httpie 0 9 9 http 1 1 200 ok connection keep alive content length 43 content type application json date sat 16 dec 2017 18 16 53 gmt server nginx 1 10 3 ubuntu via kong 0 11 2 x kong proxy latency 86 x kong upstream latency 33 x ratelimit limit minute 10 x ratelimit remaining minute 9 data version 21633901644387004745 a few things to notice in the call above we are making an ssl call by default kong exposes ssl on port 8443 we are passing a query string to the kong1 url endpoint kong will pass this along to our actual api the http reply contains 2 headers related to rate limiting x ratelimit limit minute 10 this specifies the rate we set for the rate limiting plugin x ratelimit remaining minute 9 this specifies the remaining calls we have after 9 requests in quick succession the rate limiting headers are x ratelimit limit minute 10 x ratelimit remaining minute 1 after 10 requests x ratelimit limit minute 10 x ratelimit remaining minute 0 after the 11th request we get an http 429 error http 1 1 429 connection keep alive content type application json charset utf 8 date sat 16 dec 2017 18 21 19 gmt server kong 0 11 2 transfer encoding chunked x ratelimit limit minute 10 x ratelimit remaining minute 0 message api rate limit exceeded putting a load balancer in front of kong calling kong apis on non standard port numbers gets cumbersome to make things cleaner i put an aws alb in front of kong i added a proper ssl certificate via aws certificate manager for the domain stage api mydomain com and associated it to a listener on port 443 of the alb i also created two target groups one for http traffic to port 80 on the alb and one for https traffic to port 443 on the alb the first target group points to port 8000 on the server running kong because that s the port kong uses for plain http requests the second target group points to port 8443 on the server running kong for https requests i was now able to make calls such as these http v post https stage api mydomain com kong1 query version post kong1 query version http 1 1 accept accept encoding gzip deflate connection keep alive content length 0 host stage api mydomain com user agent httpie 0 9 9 http 1 1 200 ok connection keep alive content length 42 content type application json date fri 29 dec 2017 17 42 14 gmt server nginx 1 10 3 ubuntu via kong 0 11 2 x kong proxy latency 94 x kong upstream latency 29 x ratelimit limit minute 10 x ratelimit remaining minute 9 data version 21633901644387004745 using the kong syslog plugin i added the syslog plugin via the kong admin dashboard i set the following values server_errors_severity err successful_severity notice client_errors_severity err log_level notice i then created a file called kong conf in etc rsyslog d on the server running kong cat kong conf if programname kong and syslogseverity text notice then var log kong log service rsyslog restart now every call to kong is logged in syslog format in var log kong log the message portion of the log entry is in json format sending kong logs to aws elasticsearch kibana i used the process i described in my previous blog post on aws cloudwatch logs and aws elasticsearch one difference was that for the kong logs i had to use a new index in elasticsearch this was because the json object logged by kong contains a field called response which was clashing with another field called response already present in the cwl index in kibana what i ended up doing was copying the lambda function used to send cloudwatch logs to elasticsearch and replacing cwl with cwl kong in the transform function this created a new index cwl kong in elasticsearch and at that point i was able to add that index to a kibana dashboard and visualize and query the kong logs this is just scratching the surface of what you can do with kong here are a few more resources webinar on api microservices with kong presentation on api gateway pattern and kong in a microservices world blog post on api authentication with kong at december 29 2017 no comments email this blogthis share to x share to facebook share to pinterest friday november 24 2017 using aws cloudwatch logs and aws elasticsearch for log aggregation and visualization if you run your infrastructure in aws then you can use cloudwatch logs and aws elasticsearch kibana for log aggregation searching visualization as an alternative to either rolling your own elk stack or using a 3rd party saas solution such as logentries loggly papertrail or the more expensive splunk sumo logic etc here are some pointers on how to achieve this 1 create iam policy and role allowing read write access to cloudwatch logs i created a iam policy called cloudwatch logs access with the following content version 2012 10 17 statement effect allow action logs createloggroup logs createlogstream logs putlogevents logs describelogstreams resource arn aws logs then i create an iam role called cloudwatch logs role and attached the cloudwatch logs access policy to it 2 attach iam role to ec2 instances i attached the cloudwatch logs role iam role to all ec2 instances from which i wanted to send logs to cloudwatch i went to actions instance settings attach replace iam role and attached the role 3 install and configure cloudwatch logs agent on ec2 instances i followed the instructions here for my os which is ubuntu i first downloaded a python script curl https s3 amazonaws com aws cloudwatch downloads latest awslogs agent setup py o then i ran the script in the region where my ec2 instances are python awslogs agent setup py region us west 2 launching interactive setup of cloudwatch logs agent step 1 of 5 installing pip done step 2 of 5 downloading the latest cloudwatch logs agent bits done step 3 of 5 configuring aws cli aws access key id none aws secret access key none default region name us west 2 default output format none step 4 of 5 configuring the cloudwatch logs agent path of log file to upload var log syslog destination log group name var log syslog choose log stream name 1 use ec2 instance id 2 use hostname 3 custom enter choice 1 2 choose log event timestamp format 1 b d h m s dec 31 23 59 59 2 d b y h m s 10 oct 2000 13 55 36 3 y m d h m s 2008 09 08 11 52 54 4 custom enter choice 1 3 choose initial position of upload 1 from start of file 2 from end of file enter choice 1 1 more log files to configure y i continued by adding more log files such as apache access and error logs and other types of logs you can start stop restart the cloudwatch logs agent via service awslogs start the awslogs service writes its logs in var log awslogs log and its configuration file is in var awslogs etc awslogs conf 4 create aws elasticsearch cluster not much to say here follow the prompts in the aws console for the initial access policy for the es cluster i chose an ip based policy and specified the source cidr blocks allowed to connect statement effect allow principal aws action es resource arn aws es us west 2 accountid domain my es cluster condition ipaddress aws sourceip 1 2 3 0 24 4 5 6 7 32 5 create subscription filters for streaming cloudwatch logs to elasticsearch first make sure that the log files you configured with the aws cloudwatch log agent are indeed sent to cloudwatch for each log file name you should see a cloudwatch log group with that name and inside the log group you should see multiple log streams each log stream having the same name as the hostname sending those logs to cloudwatch i chose one of the log streams went to actions stream to amazon elasticsearch service chose the elasticsearch cluster created above then created a new lambda function to do the streaming i had to create a new iam role for the lambda function i created a role i called lambda execution role and associated with it the pre existing iam policy awslambdabasicexecutionrole once this lambda function is created subsequent log subscription filters for other log groups will reuse it for streaming to the same es cluster one important note here is that you also need to allow the role lambda execution role to access the es cluster to do that i modified the es access policy and added a statement for the arn of this role effect allow principal aws arn aws iam accountid role lambda execution role action es resource arn aws es us west 2 accountid domain my es cluster 6 configure index pattern in kibana the last step is to configure kibana to use the elasticsearch index for the cloudwatch logs if you look under indices in the elasticsearch dashboard you should see indices of the form cwl 2017 11 24 in kibana add an index pattern of the form cwl it should recognize the timestamp field as the timestamp for the log entries and create the index pattern correctly now if you go to the discover screen in kibana you should be able to visualize and search your log entries streamed from cloudwatch at november 24 2017 no comments email this blogthis share to x share to facebook share to pinterest monday july 31 2017 apache 2 4 authentication and whitelisting scenarios i have these examples scattered among many apache installations so i wanted to gather my notes here for my benefit and hopefully for others as well the following scenarios depict various requirements for apache 2 4 authentication and whitelisting they are all for apache 2 4 x running on ubuntu 14 04 16 04 scenario 1 block all access to apache except to a list of whitelisted ip addresses and networks apache configuration snippet directory var www html includeoptional etc apache2 whitelist conf order allow deny allow from all directory contents of whitelist conf file local server ips require ip 127 0 0 1 require ip 172 31 2 2 office network require ip 1 2 3 0 24 other ip addresses require ip 4 5 6 7 32 require ip 5 6 7 8 32 etc scenario 2 enable basic http authentication but allow specific ip addresses through with n...