Meta tags:
keywords=data frames java,Development,InfoQ Dev Summit Boston 2025,Java,InfoQ Dev Summit,Transcripts,Best Practices,QCon Software Development Conference,InfoQ,Data,;
description=Vladimir Zakharov discusses the power of DataFrames in Java. He compares implementations like DataFrame-EC and Tablesaw against Python’s pandas, focusing on performance and memory efficiency.;

Headings (most frequently used words):

in, the, to, data, ai, with, infoq, 1brc, and, oriented, featured, engineering, your, online, of, for, an, team, agent, end, powered, java, software, dataframe, example, donut, per, methods, from, platform, related, ui, automation, how, newsletter, about, object, vs, donuts, is, ec, store, customer, topics, development, design, devops, governance, age, conversation, sarah, wells, accelerating, netflix, cross, journey, offline, road, compliance, will, internal, users, hate, chaos, gpu, clusters, on, slack, introduces, driven, testing, improve, resilience, test, by, cloudflare, identifies, race, condition, hyper, http, implementation, formal, every, engineer, future, local, first, open, source, certification, qcon, are, you, missing, frame, power, frames, architects, unlock, full, experience, summary, bio, conference, transcript, programming, what, one, billion, row, challenge, optimized, developers, toy, tablesaw, kotlin, performance, eclipse, collections, use, cases, list, popularity, order, priority, orders, tomorrow, total, spend, count, day, takeaways, resources, questions, answers, don, have, account, architecture, ml, culture, helpful, links, choose, language, practical, robustness, going, beyond, memory, safety, rust, sponsors, this, content, topic, editorial, popular, across, events, follow, us, stay, know, node, js, 26, temporal, api, enabled, default, v8, 14, round, deprecations, wordpress, ships, foundations, core, modernized, admin, new, tools, aws, expands, release, management, validate, code, before, production, build, more, resilient, applications, at, protocol, infrastructure, openai, fixes, 18, year, old, gnu, libunwind, bug, treating, crash, debugging, like, epidemiology, doordash, built, shopping, assistant, that, doesn, rely, llm, alone, enables, collaboration, creating, path, sovereign, challenges, priorities, computing, multi, approach, building, reliable, controllable, linux, foundation, launches, akrites, protect, critical, threats, github, copilot, cli, gets, tabs, no, config, file, tool, setup, redesigned, terminal, architect, security, privacy, program, san, francisco, london, 2027,

Text of the page (most frequently used words):
the (259), and (169), you (169), that (93), data (86), with (70), for (69), this (66), have (59), #dataframe (51), your (48), java (48), just (43), can (39), one (38), are (34), infoq (33), but (32), what (31), which (31), how (30), like (30), again (28), not (28), about (27), see (27), use (27), all (26), code (26), then (26), from (24), look (24), these (23), oriented (22), here (22), there (22), when (21), some (21), want (21), they (20), out (19), will (19), #tablesaw (19), challenge (19), don (18), dataframes (18), aggregation (18), example (17), donut (17), because (17), engineering (16), lot (16), pretty (16), billion (16), also (15), collections (15), time (15), memory (15), software (14), actually (14), order (14), object (14), work (13), other (13), file (13), first (13), python (13), frameworks (13), things (13), same (13), row (13), more (12), get (12), development (12), database (12), eclipse (12), could (12), need (12), much (12), let (12), kotlin (12), online (11), way (11), real (11), quantity (11), name (11), column (11), than (11), where (11), our (11), orders (11), 2026 (10), know (10), write (10), architecture (10), read (10), programming (10), table (10), very (10), donuts (10), performance (10), design (9), may (9), pure (9), them (9), standard (9), case (9), would (9), per (9), does (9), into (9), slide (9), columns (9), has (9), think (9), going (9), privacy (8), approach (8), two (8), comes (8), list (8), relational (8), customer (8), straightforward (8), similar (8), examples (8), both (8), saw (8), price (8), groupby (8), works (8), those (8), collection (8), results (8), rows (8), com (7), architect (7), senior (7), internal (7), end (7), implementation (7), infrastructure (7), business (7), different (7), maybe (7), talk (7), better (7), only (7), each (7), doing (7), logic (7), sorting (7), another (7), dsl (7), under (7), easy (7), used (7), apis (7), functions (7), register (7), qcon (6), certification (6), team (6), platform (6), methods (6), driven (6), production (6), zakharov (6), enterprise (6), their (6), really (6), now (6), figure (6), basically (6), course (6), cases (6), manipulation (6), records (6), even (6), actual (6), something (6), pandas (6), pivot (6), values (6), aggregate (6), behavior (6), specify (6), context (6), between (6), sort (6), method (6), expression (6), happening (6), world (6), well (6), was (6), efficient (6), cost (6), pattern (6), 1brc (6), immutable (6), domain (6), state (6), content (5), terms (5), architects (5), culture (5), security (5), july (5), teams (5), topics (5), early (5), every (5), over (5), developers (5), devops (5), governance (5), before (5), vladimir (5), store (5), processing (5), dataset (5), problem (5), take (5), using (5), type (5), difference (5), covers (5), priority (5), tomorrow (5), three (5), weather (5), primitive (5), top (5), looks (5), thing (5), framework (5), types (5), schema (5), station (5), articles (5), featured (5), edition (5), stay (4), new (4), august (4), events (4), developer (4), systems (4), technical (4), peers (4), protect (4), select (4), country (4), learn (4), overview (4), last (4), join (4), view (4), powered (4), agent (4), automation (4), users (4), improve (4), build (4), api (4), dev (4), necessarily (4), scenarios (4), most (4), piece (4), nice (4), practical (4), obviously (4), done (4), available (4), resources (4), benefit (4), process (4), people (4), excel (4), call (4), lists (4), missing (4), value (4), its (4), null (4), declarative (4), otherwise (4), probably (4), exactly (4), spend (4), still (4), own (4), shop (4), access (4), operations (4), takes (4), language (4), large (4), interesting (4), entities (4), customers (4), tables (4), based (4), nulls (4), features (4), objects (4), makes (4), usage (4), idea (4), bit (4), function (4), define (4), full (4), frame (4), print (4), temperature (4), implementations (4), solutions (4), model (4), maps (4), experience (4), c4media (3), notice (3), marketing (3), editors (3), feedback (3), newsletter (3), podcast (3), followers (3), 2027 (3), program (3), diversity (3), live (3), week (3), decisions (3), give (3), leaders (3), handling (3), address (3), date (3), github (3), gets (3), tool (3), open (3), source (3), chaos (3), gpu (3), clusters (3), netflix (3), local (3), collaboration (3), compliance (3), slack (3), testing (3), test (3), sarah (3), wells (3), doesn (3), rely (3), applications (3), popular (3), across (3), related (3), conference (3), best (3), transcripts (3), summit (3), topic (3), presentations (3), looking (3), life (3), pipelines (3), files (3), talking (3), whatever (3), transformation (3), change (3), links (3), couple (3), solves (3), simply (3), paradigm (3), hardware (3), count (3), day (3), implement (3), differences (3), parameters (3), multiple (3), particular (3), should (3), expressions (3), discount (3), total (3), calculate (3), externalize (3), means (3), separately (3), ide (3), definition (3), reason (3), itself (3), many (3), called (3), learning (3), answer (3), names (3), popularity (3), named (3), add (3), operate (3), since (3), pooling (3), support (3), structure (3), number (3), fairly (3), beyond (3), foreach (3), jdk (3), any (3), runtime (3), without (3), handle (3), haven (3), loading (3), execution (3), feature (3), heap (3), compare (3), level (3), pressure (3), faster (3), specifically (3), million (3), load (3), operating (3), system (3), make (3), output (3), loaded (3), mean (3), location (3), jvm (3), throw (3), readability (3), loops (3), classes (3), explains (3), shares (3), discusses (3), power (3), frames (3), rust (3), policy (2), editorial (2), general (2), london (2), san (2), francisco (2), log (2), create (2), account (2), place (2), role (2), programs (2), through (2), companies (2), engineers (2), consent (2), explained (2), enter (2), mail (2), latest (2), quick (2), adopter (2), round (2), community (2), foundation (2), critical (2), multi (2), building (2), accelerating (2), cross (2), journey (2), offline (2), challenges (2), enables (2), formal (2), engineer (2), future (2), road (2), hate (2), introduces (2), resilience (2), age (2), conversation (2), built (2), cloudflare (2), identifies (2), race (2), condition (2), hyper (2), http (2), aws (2), management (2), ships (2), default (2), boston (2), messages (2), record (2), various (2), pipeline (2), needs (2), reasons (2), enrichment (2), static (2), questions (2), major (2), points (2), tons (2), reading (2), analyze (2), correct (2), libraries (2), choices (2), fun (2), service (2), transform (2), science (2), takeaways (2), useful (2), toolkit (2), seems (2), spark (2), cluster (2), solution (2), compared (2), relatively (2), quickly (2), commodity (2), keep (2), common (2), hoc (2), shows (2), functionality (2), side (2), say (2), differently (2), try (2), multiply (2), message (2), less (2), simple (2), application (2), rather (2), next (2), external (2), requirements (2), dollar (2), bring (2), joining (2), showing (2), result (2), added (2), previous (2), govern (2), depending (2), wrong (2), tell (2), strings (2), variable (2), put (2), somewhere (2), break (2), consistent (2), slightly (2), conceptually (2), helps (2), four (2), somewhat (2), decided (2), functionally (2), sticking (2), menu (2), structures (2), bitmap (2), primitives (2), rich (2), having (2), off (2), datasets (2), grammar (2), computed (2), flexible (2), special (2), weird (2), happens (2), why (2), apache (2), commons (2), efficiency (2), speed (2), finish (2), highly (2), size (2), 100 (2), had (2), comparison (2), created (2), second (2), maximum (2), caveat (2), good (2), creates (2), says (2), part (2), infix (2), easier (2), coming (2), after (2), wondering (2), library (2), replacement (2), iterate (2), lambda (2), parsed (2), step (2), although (2), options (2), formatted (2), working (2), given (2), market (2), month (2), ran (2), made (2), sense (2), immediately (2), obvious (2), recommend (2), amazing (2), measurement (2), minimum (2), save (2), writing (2), been (2), high (2), too (2), understand (2), potentially (2), slides (2), directly (2), overhead (2), instance (2), presentation (2), class (2), blogs (2), responding (2), years (2), ecosystem (2), today (2), insights (2), scalable (2), download (2), homepage (2), bird (2), ends (2), leadership (2), aug (2), evals (2), complex (2), protocols (2), maximize (2), observability (2), cloud (2), continuous (2), safely (2), patterns (2), procedures (2), safety (2), robustness (2), updated (2), unlock (2), english (2), copyright, 2006, inc, cookie, conditions, advertising, sales, bluesky, 21k, likes, facebook, 19k, readers, rss, instagram, 26k, linkedin, 232k, youtube, follow, april, november, blog, media, kit, conferences, home, hours, weeks, reserve, becomes, changes, longer, implementing, shaping, trade, offs, direction, depend, structured, current, katharine, jarmul, luca, mezzalira, hien, luu, information, interested, published, variety, innovator, technologies, sent, tuesday, 250, 000, copilot, cli, tabs, config, setup, redesigned, terminal, linux, launches, akrites, threats, reliable, controllable, path, sovereign, priorities, computing, creating, doordash, shopping, assistant, llm, alone, openai, fixes, year, old, gnu, libunwind, bug, treating, crash, debugging, epidemiology, resilient, protocol, expands, release, validate, wordpress, foundations, core, modernized, admin, tools, node, temporal, enabled, deprecations, practices, 2025, sponsors, feb, recorded, yes, deal, detached, whether, units, within, company, financial, institutions, exchanging, final, resting, validation, recovery, point, databases, constant, inserts, mutability, assuming, concerned, participant, answers, sources, check, explore, backend, rewrite, stick, programmatically, query, notebooks, visualization, supported, addition, fit, especially, mind, playing, notebook, tend, reach, supports, dimensions, coordinates, equivalent, philosophical, discussion, feel, meaningful, error, telling, shouldn, normally, text, express, baked, externalizable, imperative, fashion, later, formula, looked, readable, everything, strongly, typed, inference, being, amount, depends, together, regular, usual, produce, harder, parse, reader, impossible, employee, submit, pull, requests, outside, accounting, department, whoever, red, squiggly, selectby, translated, executed, applied, containing, argue, able, manual, steep, curve, equally, configuration, ascending, ties, ordering, care, quantities, keepcolumns, drops, rest, keeps, specifying, visually, solving, sql, listing, grouping, descending, beginning, figuring, arbitrary, greater, equal, bob, compute, prices, represented, respective, purpose, exercise, observations, sample, 400, odd, stations, compression, mutable, boolean, marking, multimap, indexing, index, right, map, were, inspired, small, generic, tries, provide, reasonable, functional, tells, adds, ticked, advantage, applies, earlier, peel, cover, int, double, pool, dates, pooled, filters, kind, extend, touching, custom, certain, legacy, flows, interpreted, wonderful, ways, computer, perspective, mentioned, benefits, underlying, efficiencies, expose, hard, dependency, hide, worse, magnitude, sure, debugged, weren, designed, attention, paid, hurting, fastutil, stand, measured, ahead, everyone, significant, margin, great, likely, limiting, factor, run, volumes, measurements, gigs, allocated, behave, garbage, removed, skip, talked, introduce, didn, wanted, split, purple, space, physically, groups, materialized, wise, syntax, lets, dsls, situation, seeing, unusual, sortby, expected, iteration, groupingby, max, sometimes, instances, clarify, fully, compatible, existing, drop, offers, including, factory, uses, pass, physical, aggregateby, target, aggregations, start, unfortunately, controversial, builder, separator, semicolon, header, line, documentation, seen, hierarchical, flatten, hierarchy, iterator, loop, starting, read_csv, reads, csv, clear, runs, option, folks, seem, intrigued, format, except, steps, perform, console, suspect, hear, words, establish, benchmark, author, committer, picked, actively, maintained, projects, toy, flip, focused, achieving, metrics, optimized, worked, until, deadline, anything, scenario, squeeze, cpu, cycle, optimize, maintainability, finished, half, long, hundreds, lines, factored, usually, daily, coding, activities, vector, low, experimental, algorithms, commented, heard, checking, find, hope, beating, winners, classic, elaborate, task, concise, expressive, dive, gunnar, morling, came, january, 2024, fast, earth, average, optimizations, effort, alternatives, already, consider, around, reconciliation, alluded, ability, easily, filter, operates, separate, mix, tabular, nothing, question, requires, inserting, sake, filtering, fine, certainly, bunch, nested, statements, stable, refactoring, exploratory, phase, refactor, powerful, please, never, fact, evolving, sealed, matching, manipulate, streams, set, said, come, layout, costs, millions, billions, environment, abstraction, ask, calculated, stored, helper, completely, hidden, finds, selling, modern, little, concrete, distinction, throughout, introduction, main, buy, represent, modeling, hiding, details, kept, shared, mutate, bad, happen, paradigms, choice, deep, versus, mention, decision, refer, depth, understanding, approaches, treat, encapsulate, privates, ideas, polymorphism, hides, known, opposite, properties, naked, proper, brief, justice, concept, brian, goetz, his, follows, encourages, embodies, act, vlad, primarily, served, executive, committee, body, governs, evolution, member, expert, group, jsr, 335, delivered, stream, went, contrast, fitting, overall, transcript, focuses, face, gain, valuable, connect, speakers, enjoy, social, hands, twenty, sustainable, platforms, bio, serve, vital, analyzing, outperform, while, maintaining, devs, summary, 25x, horizontal, vertical, proven, tracks, apr, nov, distributed, decentralized, calls, retrieval, agents, checked, secure, sensitive, guardrails, audits, choose, helpful, bryan, oliver, frontier, scale, topologies, network, rdma, numa, misalignments, discover, seven, fault, injection, strategies, robust, containers, delivery, davide, paolis, realities, rolling, fracturing, relations, drawing, reboot, sevdesk, viable, utilize, event, alerting, automate, shift, rigid, enforcement, empathy, craftmanship, sociocracy, scrum, personal, growth, lean, kanban, agile, eng, raj, ummadisetty, ken, kurzweil, share, architectural, cloudstream, repeatable, capture, conversion, deployment, discuss, shifting, key, abstractions, stateless, stateful, move, terabytes, bulk, exploit, pathfinder, prototypes, maintain, rollout, streaming, analytics, nosql, machine, big, michael, stiefel, spoke, relationship, effectively, establishing, minimize, complexity, reduce, repetitive, tasks, targeted, checklists, help, reducing, stress, mesh, microservices, studies, scalability, andy, brinkmeyer, failure, proof, moving, ownership, enums, typestate, embed, compile, checks, eliminate, entire, bugs, manage, codebase, effortlessly, javascript, swift, net, guides, podcasts, news, logo, back, bookmark, whenever, youre, ready, anytime, minibooks, videos, training, materials, free, receive, instant, alerts, trends, matter, logging, favorite, authors, engage, exclusive, sign, search, french, japanese, chinese, facilitating, spread, knowledge, innovation, professional, toggle, navigation, close, monthly, aspiring,

Text of the page (random words):
make it a little bit more concrete let s look at this distinction between object oriented and data oriented approach in a particular domain this domain is a donut shop and this shop ships donut orders to customers we ll be using this domain throughout the presentation so this is the introduction our main entities are the donut shop itself which has customers customers have name and address then there is a menu of donuts available with their name price and discount price if you buy a lot of them then you have donut orders the way you represent it in the object oriented approach is by modeling each one of these entities as a class potentially hiding implementation details again you may not even have state on the object state is just a useful abstraction when you ask an object for a particular state like how much is this order it could be calculated value it could be directly stored it could be coming from some helper object it s all completely hidden with data oriented approach you just have three tables again could be lists of java records could be database tables then if you want to implement some behavior you just write in this case a java function operating on a list of donuts this function finds the top three best selling donuts by quantity this is basically modern java code that does data oriented programming in java data oriented programming in java in fact java support for data oriented programming has been evolving there are a lot of features added like some of them you see on the slides record sealed classes pattern matching let you manipulate these types of objects directly then of course collection streams and maps let you do set operations on collections of these entities having said that these features do come at a cost one is overhead you see this is actual overhead computed using java object layout framework for each one of these records this is just simply what it costs you to have one instance of these that s really not a lot but it will add up if you have millions or billions of these objects which happens in some real production environment the code readability is fine it s certainly better than if you had a bunch of nested loops and for loops and if statements but it s still not amazing you can t just take one look and immediately understand what s going on there also when you define java records records are still classes if your domain is not stable if you have to do a lot of refactoring maybe you re in an early exploratory phase of your development or maybe you don t even own the data schema the data structure that you work with you may have to refactor a lot and that comes at a cost of course one flexible and powerful way to model pure data in java is to use maps of maps potentially of maps and please never do it what is a dataframe now let s throw a dataframe into the mix a dataframe is just a data structure it s a tabular dataset which is made up of columns of different types again similar to a relational table here is an example of a dataframe on the slide and these are donut orders again nothing special there looks like a table of course one question could be like if this looks like a relational table why don t i just use a relational database to do operations on these datasets the answer is of course you can but that requires a database if you don t have a database in your pipeline or the database doesn t have the data that you want to operate on then inserting it just for the sake of doing some filtering aggregation transformation may be just too much in terms of the cost both development cost infrastructure cost processing time and so on as i alluded to in the previous slide dataframes give you the ability to easily transform data you can think of it conceptually like relational database operations you can aggregate you can join you can filter it does do it in a way because it operates at a high level that makes it easy for developers we ll see of course code examples it also makes it easy to separate the data manipulation logic from the data itself and even externalize that logic and take it out of your applications another thing that dataframes are good at again we ll see examples of that is that they can do a lot of optimizations under the covers they can use efficient collection frameworks which are much more efficient than your standard jdk collections they can do things like object pooling to save memory and to improve performance without any effort from the developer writing code it could be much more efficient compared to alternatives if you already have a database with all the data you need use it but if not this is something to consider these are not really specifically talking about java dataframes this framework has been around for some time and they re used in real world production scenarios for things like data transformation data enrichment reconciliation and many other use cases example the one billion row challenge 1brc let s look at some code for our first example we re going to look at the one billion row challenge problem this is a challenge that gunnar morling came up with and it ran in january of 2024 the idea was to see how fast a java program can process and aggregate a large dataset in this case one billion rows of temperature measurement values you have a file with one billion rows it has a weather station which is just some location somewhere on earth and it has temperature measurement what you need to do is calculate the minimum average and maximum temperature per weather station there is just one caveat the file has one billion rows if you haven t heard of it i highly recommend checking it out you can find the challenge and the solutions on the slide the reason we re looking at it is not because we have any hope of beating the winners of the challenge but because it s a basically classic data oriented programming problem it makes no sense to design elaborate object oriented domain for this task we ll see how well different dataframes can handle it and also how easy it is to write and to read the code how concise how expressive it is before we dive into that just a quick overview of the results of the actual challenge those results are pretty amazing the top three results all finished in about a second and a half if you look at the top results they do have a lot in common they re all long it s many hundreds of lines of well formatted well factored code they do use the apis that developers don t usually use in their daily coding activities things like vector api other low level apis some experimental jvm features because of that it s not immediately obvious how these algorithms work although some of those solutions are pretty well commented also it s not obvious what the code does functionally obviously when you look at it you know that it solves one billion row challenge because that s what it says given a solution it would take you some time to figure out what it is that it actually does all these top solutions people worked on them up until the challenge deadline the time to market for them was about a month since it ran for a month again it s a fun challenge it all made sense it doesn t mean that there is anything wrong with the code in a real life scenario we don t necessarily want to squeeze every cpu cycle out of our solutions we want to optimize for things like maintainability developer time time to market 1brc optimized for developers what if we flip our requirements and we re focused on achieving those metrics 1brc with toy data let s throw in dataframe for this part we ll look at four dataframe implementations first one is python pandas it s something that i suspect most people think about when they hear the words dataframe this is here just to establish a benchmark both in terms of performance and in terms of code readability again how easy it is to write how easy it is to read we ll look at dataframe ec and tablesaw they are pure java dataframe implementations i m the author and the committer for dataframe ec i picked tablesaw because it s another implementation which is pure java it seems to be actively maintained and it s used in various other projects the last one we ll look at is kotlin dataframe kotlin comes with a dataframe library it s not java but it runs on jvm it s an option that is available to folks working on jvm people seem to be intrigued by all things kotlin so i decided to throw it in there the file format is what you see here this is what the actual file looks like except in a real challenge there are one billion rows in there the code you re going to see will work with one billion rows we break this into three steps load data perform aggregation and sorting and then just print results on the console then at the end we ll look at performance and memory usage comparison between all these frameworks starting with python pandas the code here is pretty straightforward even if you don t know python it s easy to figure out what s happening the read_csv method reads csv file is a variable where we have file name and it has these nice named parameters you can look at it and it will be clear what s happening we get our dataframe parsed and loaded processing we want to groupby with aggregation there are two options here one does groupby and aggregation and then sort we want results to be in a consistent order we will also have a sorting step by default groupby and aggregation do sorting i read in the documentation that doing sorting separately will result in faster performance i haven t seen the difference but these are two options i m showing them on the slide it s pretty straightforward what s happening here the only thing to call out is after you do aggregation you end up with hierarchical columns which is a feature of pandas to print it out we just flatten the column hierarchy to make it easier to work with this is your standard python iterator pattern the for loop we iterate through the rows and just print the formatted view of each of the rows here we are done 1brc with dataframe ec let s look at the java examples we ll start with dataframe ec the first thing we do we get file location then we define the schema unfortunately although it s a controversial topic we don t have named parameters in java but we use this builder pattern again it s pretty straightforward i think to see what s happening here you have two columns station and temperature your separator is a semicolon and there is no header line so you specify all that then you create a dataset which links that schema to an actual physical file location and you load it as a dataframe here it is parsed and loaded aggregation and sorting again that looks pretty straightforward you think of it as doing groupby in your relational world you aggregateby and then you have a list of functions and the aggregation function takes the column you want to aggregate and the name of the target column you have these aggregations then you groupby weather station and then the next step you sort by weather station you may be wondering what this immutable business is and we ll talk about it just to clarify things dataframe ec is based on an open source library called eclipse collections and eclipse here as in eclipse foundation not eclipse ide it s a fully compatible replacement for jdk collections for the existing data types you can just do a drop in replacement it offers a lot more collection types including full support for primitive and immutable collections dataframe ec is based on eclipse collections so that s the ec in the name immutable here is just a factory method that creates an immutable list in this case of aggregation functions output just uses a standard foreach pattern we iterate over the rows of the data frame and it takes a lambda where we pass in a row object and then we just print it fairly straightforward and fairly similar to what we saw in python 1brc with tablesaw tablesaw works exactly the same way you get the file you define the schema the dataframe object in tablesaw is called table the only difference here is that if you don t have column names you can t specify them in your schema so we have to name our columns after we loaded the file aggregation groupingby and sorting work again the same way you also don t get to name the columns here so the column names mean and max are created for you but otherwise it s very similar to what you saw with dataframe ec the output works pretty much exactly the same if you look at it you may be wondering are those frameworks all the same are those apis all the same the answer is sometimes there are instances where there are major differences between the code you write based on frameworks apis and some design choices we ll see examples of those in a bit 1brc with kotlin let s look at kotlin loading bit is pretty standard processing is interesting there are a couple of things i want to call out first of all groupby actually physically groups by date and creates materialized dataframe of dataframes that has to have some cost memory wise then when you look at the aggregation you see this weird syntax where it says into that s actually a kotlin feature that lets you write internal dsls like this into is not a part of the language grammar it s an infix function which in a situation like this may make it easier to read code obviously coming from java seeing infix functions is somewhat unusual then sortby works as expected the iteration the output using foreach pattern again pretty straightforward performance we can skip that and look at the performance specifically execution time and memory usage first you see we talked about the one billion row challenge this is not the one billion row challenge this is 100 million row challenge the reason we had to introduce that was because kotlin just didn t finish the one billion row challenge but i wanted to have a comparison in the same context so we created 100 million row challenge the way to read this slide is first for each of the frameworks we have python dataframe ec tablesaw and kotlin the first column is execution time split between time to load and time to aggregate the second column the purple one is maximum heap size with the caveat that for python pandas there is no heap it s a space used by the operating system process i think we have a pretty good idea about kotlin let s look at the one billion row challenge results a couple of things stand out in terms of performance as measured by execution time python comes pretty much ahead of everyone by some relatively significant margin it s really not doing great when it comes to memory usage that s likely to be a limiting factor when you run on commodity size hardware that s just a feature of pandas if you want something that s memory efficient at large memory volumes it s not going to work for you for the java frameworks we have two measurements for each one with 16 gigs of heap allocated the other one with 32 the idea was to just compare how ...