Meta tags:
description=We introduce TurkingBench, a benchmark of tasks represented as webpages with multi-modal context to measure the generalizablity of web-based agents to a variety of naturally-constructed tasks.;

Headings (most frequently used words):

turkingbench, challenge, benchmark, for, web, agents, overview, citation,

Text of the page (most frequently used words):
the (14), and (12), web (8), for (6), #benchmark (6), tasks (6), pages (5), with (5), agents (4), turkingbench (4), text (4), models (4), this (4), tweet (3), code (3), paper (3), based (3), evaluation (3), that (3), only (3), multi (3), modal (3), various (3), form (3), university (3), more (2), kevin (2), kordi (2), yeganeh (2), sanders (2), kate (2), wang (2), yizhong (2), byerly (2), adam (2), zhang (2), jack (2), van (2), durme (2), benjamin (2), khashabi (2), daniel (2), arxiv (2), 2403 (2), 11905 (2), facilitate (2), chatbots (2), state (2), art (2), self (2), supervised (2), language (2), their (2), than (2), introduce (2), instructions (2), context (2), html (2), crowdsourcing (2), task (2), are (2), instances (2), raw (2), world (2), complex (2), template, credit, dynalang, information, check, out, summary, article, turkingbench2024xu, title, tur, ingbench, tournament, among, author, year, 2024, eprint, https, org, abs, url, archiveprefix, citation, additionally, develop, framework, connects, responses, modifications, modifying, box, checking, radio, etc, evaluate, performance, testing, range, vision, layout, combination, our, findings, reveal, these, perform, significantly, better, random, chance, yet, considerable, room, exist, improvement, hope, will, help, development, address, question, formulated, containing, textual, unlike, existing, work, which, employs, artificially, synthesized, here, use, natural, were, originally, designed, workers, annotation, purposes, each, also, instantiated, values, obtained, from, new, contains, distributed, across, 158, recent, have, demonstrated, impressive, ability, understand, communicate, however, there, example, humans, spend, long, hours, time, where, intertwined, other, modalities, accomplished, interactions, can, generalize, such, domains, overview, represented, webpages, measure, generalizablity, variety, naturally, constructed, johns, hopkins, brown, washington, challenge, learning, model,

Text of the page (random words):
learning to model the world with language turkingbench a challenge benchmark for web agents kevin xu yeganeh kordi kate sanders yizhong wang adam byerly jack zhang benjamin van durme daniel khashabi johns hopkins university brown university university of washington paper code tweet tl dr we introduce turkingbench a benchmark of tasks represented as webpages with multi modal context to measure the generalizablity of web based agents to a variety of naturally constructed tasks overview recent chatbots have demonstrated impressive ability to understand and communicate in raw text form however there is more to the world than raw text for example humans spend long hours of their time on web pages where text is intertwined with other modalities and tasks are accomplished in the form of various complex interactions can self supervised state of the art multi modal models generalize to such complex domains to address this question we introduce turkingbench a benchmark of tasks formulated as web pages containing textual instructions with multi modal context unlike existing work which employs artificially synthesized web pages here we use natural html pages that were originally designed for crowdsourcing workers for various annotation purposes the html instructions of each task are also instantiated with various values obtained from the crowdsourcing tasks to form new instances of the task this benchmark contains 32 2k instances distributed across 158 tasks additionally to facilitate the evaluation on turkingbench we develop an evaluation framework that connects the responses of chatbots to modifications on web pages modifying a text box checking a radio etc we evaluate the performance of state of the art models on this benchmark testing a range of self supervised models language only vision only layout only and their combination our findings reveal that these models perform significantly better than random chance yet considerable room exist for improvement we hope this benchmark will help facilitate the evaluation and development of web based agents citation article turkingbench2024xu title tur k ingbench a tournament among web based agents author xu kevin and kordi yeganeh and sanders kate and wang yizhong and byerly adam and zhang jack and van durme benjamin and khashabi daniel year 2024 eprint https arxiv org abs 2403 11905 url 2403 11905 archiveprefix arxiv for more information check out the paper code and tweet summary paper code tweet template credit dynalang