Meta tags:
description=A Powerful web scraper powered by LLM OpenAI, Gemini & Ollama - itsOwen/CyberScraper-2077;

Headings (most frequently used words):

scraping, uh, oh, navigation, docker, multi, page, using, code, users, saved, searches, cyberscraper, 2077, use, to, files, about, for, installation, usage, tor, contributing, license, footer, ollama, example, security, search, repositories, issues, pull, requests, provide, feedback, itsowen, menu, filter, your, results, more, quickly, folders, and, latest, commit, history, repository, features, windows, beta, network, setup, google, sheets, authentication, adjusting, playwrightscraper, settings, optional, troubleshooting, faq, contact, disclaimer, sponsor, this, project, contributors, languages, with, how, tips, effective, prerequisites, configuration, options, considerations, support, topics, resources, of, conduct, policy, stars, watchers, forks,

Text of the page (most frequently used words):
the (97), you (48), and (45), your (42), this (28), 2077 (28), cyberscraper (27), page (23), tor (23), #scraping (22), use (21), for (21), #security (19), ollama (19), with (18), docker (17), github (16), data (16), url (16), code (15), 8501 (14), https (14), com (14), can (13), all (13), scrape (13), feature (13), http (13), web (12), from (12), example (12), pages (12), using (11), want (11), install (11), through (10), support (10), run (10), license (9), website (9), host (9), reload (8), api (8), any (8), network (8), search (8), that (7), please (7), open (7), pull (7), git (7), will (7), settings (7), onion (7), view (7), there (6), was (6), while (6), contributing (6), llm (6), scraper (6), gemini (6), openai (6), software (6), are (6), add (6), captcha (6), browser (6), true (6), issues (6), client (6), localhost (6), key (6), features (6), now (6), multi (6), source (6), enterprise (6), not (5), navigation (5), error (5), loading (5), more (5), repository (5), powered (5), need (5), file (5), ensure (5), txt (5), commit (5), create (5), window (5), enter (5), bool (5), navigate (5), services (5), google (5), usage (5), additional (5), sites (5), start (5), note (5), pattern (5), content (5), app (5), itsowen (5), another (5), information (4), community (4), about (4), mit (4), readme (4), tool (4), may (4), only (4), before (4), under (4), just (4), have (4), some (4), bypass (4), bot (4), these (4), results (4), behavior (4), download (4), 127 (4), 8502 (4), name (4), setup (4), following (4), automatic (4), requests (4), linux (4), system (4), query (4), whatever (4), format (4), something (4), multiple (4), feedback (4), export (4), streamlit (4), main (4), machine (4), clone (4), actions (4), time (3), terms (3), dockerfile (3), python (3), project (3), stars (3), conduct (3), resources (3), topics (3), future (3), net (3), also (3), connection (3), always (3), into (3), see (3), end (3), amazing (3), based (3), environment (3), new (3), json (3), click (3), application (3), type (3), find (3), select (3), sheets (3), openai_api_key (3), users (3), avoid (3), automatically (3), circuit (3), detection (3), detect (3), pip (3), service (3), extract (3), number (3), csv (3), specify (3), cyberpunk (3), custom (3), ranges (3), beta (3), venv (3), activate (3), 11434 (3), llama3 (3), actual (3), address (3), below (3), container (3), set (3), windows (3), build (3), follow (3), models (3), requirements (3), files (3), insights (3), signed (3), tab (3), refresh (3), session (3), sign (3), saved (3), documentation (3), grade (3), copilot (3), platform (3), solutions (3), explore (3), personal (2), manage (2), contact (2), footer (2), sponsors (2), sponsor (2), forks (2), 348 (2), policy (2), powerful (2), built (2), chrome (2), streets (2), owen (2), someone (2), samurai (2), dark (2), out (2), digital (2), right (2), without (2), complying (2), laws (2), regulations (2), purposes (2), better (2), understand (2), email (2), got (2), don (2), robots (2), issue (2), fix (2), request (2), push (2), branch (2), changes (2), fork (2), welcome (2), complete (2), simulate_human (2), customize (2), websites (2), might (2), check (2), playwrightscraper (2), adjusting (2), optional (2), configuration (2), authorized (2), credentials (2), oauth (2), cloud (2), console (2), connections (2), than (2), local (2), creation (2), socks (2), port (2), options (2), like (2), simply (2), basic (2), packages (2), sudo (2), brew (2), macos (2), supports (2), allowing (2), access (2), safely (2), perfect (2), allows (2), requires (2), should (2), speed (2), small (2), tips (2), replace (2), different (2), structures (2), xample (2), 1279 (2), when (2), structure (2), every (2), easily (2), how (2), one (2), site (2), currently (2), extracting (2), ask (2), even (2), webpage (2), bin (2), interface (2), ollama_base_url (2), mac (2), instructions (2), google_api_key (2), steps (2), installation (2), good (2), llms (2), make (2), here (2), playwright (2), current (2), uses (2), help (2), work (2), upload (2), extracted (2), implemented (2), caching (2), netrunner (2), smart (2), stealth (2), mode (2), street (2), extraction (2), __init__ (2), code_of_conduct (2), gitignore (2), dockerignore (2), src (2), assets (2), 103 (2), commits (2), last (2), message (2), menu (2), quality (2), projects (2), appearance (2), cancel (2), available (2), searches (2), repositories (2), business (2), advanced (2), developer (2), customer (2), devops (2), perform, action, share, cookies, docs, status, privacy, 2026, inc, css, languages, contributors, learn, report, watching, watchers, activity, webscraping, love, night, city, singh, because, what, makes, criminal, getting, caught, remember, knowledge, power, but, double, edged, sword, wisely, strong, firewalls, impenetrable, stay, frosty, frontier, reserve, modify, discontinue, notice, responsible, applicable, acknowledge, doing, own, risk, guarantee, accuracy, completeness, reliability, obtained, intended, educational, research, illegal, strictly, prohibited, authors, liable, damages, losses, resulting, provided, warranty, kind, express, implied, listen, choombas, jack, risks, disclaimer, owensingh72, proton, questions, hire, gig, licensed, details, mod, sell, blame, flatlined, yes, commercial, designed, ethical, respect, their, legal, faq, ran, glitch, matrix, let, know, adding, repo, together, troubleshooting, origin, checkout, cyberpunks, netrunners, samurais, contribute, parameter, pop, back, terminal, press, its, task, adjust, target, optimal, use_stealth, false, use_custom_headers, hide_webdriver, bypass_cloudflare, fit, needs, giving, rename, client_secret, generate, redirect, uris, javascript, origins, choose, existing, delete, apis, authentication, flags, enable, offline, unreachable, sending, identifying, patient, slower, regular, vpn, addition, extra, considerations, verify, verify_connection, renew, auto_renew_circuit, timeout, circuit_timeout, default, 9050, socks_port, torconfig, tor_config, isolation, headers, verification, safety, example123abc, route, pysocks, apt, homebrew, ubuntu, debian, prerequisites, anonymously, researchers, analysts, investigators, who, gather, hidden, used, responsibly, legally, specific, give, news, ycombinator, regularly, compliance, option, natural, anti, measures, mindful, load, overloading, servers, range, test, large, effective, attempt, however, best, specifying, recommended, where, patterns, entering, suggest, detects, nearly, types, paginated, spread, across, functional, encounter, occasional, unexpected, appreciate, patience, continue, improve, products, watch, tears, faster, say, flatline, chatbot, everything, question, yet, done, fire, firewall, model, visit, internal, ipconfig, addr, show, ifconfig, image, installed, prefer, official, rest, recommend, really, sure, generation, presentation, depends, well, fine, tune, prompt, filters, yourself, required, virtualenv, virtual, higher, guide, given, won, able, maintain, version, systems, instance, which, detections, necessary, works, natively, doesn, lru, cache, dictionary, reduce, redundant, calls, scraped, straight, engram, master, parsing, lightning, fast, would, trauma, team, jealous, async, operations, huge, library, parameters, routing, html, sql, excel, fits, cyberdeck, user, friendly, gui, armed, could, sleek, utilizes, cutting, edge, parse, intelligently, whether, corpo, analyst, looking, realm, has, covered, glimpse, born, neon, lit, world, localllm, slice, defenses, unparalleled, precision, style, rip, leaving, trace, items, history, latest, date, folders, tags, branches, star, must, change, notification, notifications, public, dismiss, alert, switched, accounts, resetting, focus, qualifiers, our, filter, quickly, submit, include, contacted, read, piece, take, input, very, seriously, provide, syntax, clear, jump, pricing, premium, ons, collections, trending, archive, program, accelerator, maintainer, lab, programs, fund, developers, partners, trust, center, forum, skills, ebooks, reports, events, webinars, stories, development, topic, industries, government, manufacturing, financial, healthcare, industry, cases, devsecops, modernization, case, nonprofits, startups, medium, teams, enterprises, company, size, marketplace, changelog, blog, why, stop, leaks, they, secret, protection, secure, vulnerabilities, review, plan, track, instant, dev, environments, codespaces, automate, workflow, workflows, integrate, external, tools, mcp, registry, direct, agents, merge, write, toggle, skip,

Text of the page (random words):
ect agents from issue to merge mcp registry new integrate external tools developer workflows actions automate any workflow codespaces instant dev environments issues plan and track work code review manage code changes application security github advanced security find and fix vulnerabilities code security secure your code as you build secret protection stop leaks before they start explore why github documentation blog changelog marketplace view all features solutions by company size enterprises small and medium teams startups nonprofits by use case app modernization devsecops devops ci cd view all use cases by industry healthcare financial services manufacturing government view all industries view all solutions resources explore by topic ai software development devops security view all topics explore by type customer stories events webinars ebooks reports business insights github skills support services documentation customer support community forum trust center partners view all resources open source community github sponsors fund open source developers programs security lab maintainer community accelerator github stars archive program repositories topics trending collections enterprise enterprise solutions enterprise platform ai powered developer platform available add ons github advanced security enterprise grade security features copilot for business enterprise grade ai features premium support enterprise grade 24 7 support pricing search or jump to search code repositories users issues pull requests search clear search syntax tips provide feedback we read every piece of feedback and take your input very seriously include my email address so i can be contacted cancel submit feedback saved searches use saved searches to filter your results more quickly name query to see all available qualifiers see our documentation cancel create saved search sign in sign up appearance settings resetting focus you signed in with another tab or window reload to refresh your session you signed out in another tab or window reload to refresh your session you switched accounts on another tab or window reload to refresh your session dismiss alert message itsowen cyberscraper 2077 public uh oh there was an error while loading please reload this page notifications you must be signed in to change notification settings fork 348 star 3 1k code issues 1 pull requests 2 actions projects security and quality 0 insights additional navigation options code issues pull requests actions projects security and quality insights itsowen cyberscraper 2077 main branches tags go to file code open more actions menu folders and files name name last commit message last commit date latest commit history 103 commits 103 commits github github app app assets assets src src dockerignore dockerignore gitignore gitignore code_of_conduct md code_of_conduct md contributing md contributing md dockerfile dockerfile license license readme md readme md security md security md __init__ py __init__ py main py main py requirements txt requirements txt view all files repository files navigation readme code of conduct contributing mit license security more items cyberscraper 2077 rip data from the net leaving no trace welcome to the future of web scraping about cyberscraper 2077 is not just another web scraping tool it s a glimpse into the future of data extraction born from the neon lit streets of a cyberpunk world this ai powered scraper uses openai gemini and localllm models to slice through the web s defenses extracting the data you need with unparalleled precision and style whether you re a corpo data analyst a street smart netrunner or just someone looking to pull information from the digital realm cyberscraper 2077 has got you covered features ai powered extraction utilizes cutting edge ai models to understand and parse web content intelligently sleek streamlit interface user friendly gui that even a chrome armed street samurai could navigate multi format support export your data in json csv html sql or excel whatever fits your cyberdeck tor network support safely scrape onion sites through the tor network with automatic routing and security features stealth mode implemented stealth mode parameters that help avoid detection as a bot ollama support use a huge library of open source llms async operations lightning fast scraping that would make a trauma team jealous smart parsing structures scraped content as if it was extracted straight from the engram of a master netrunner caching implemented content based and query based caching using lru cache and a custom dictionary to reduce redundant api calls upload to google sheets now you can easily upload your extracted csv data to google sheets with one click bypass captcha bypass captcha by using the captcha at the end of the url currently only works natively doesn t work on docker current browser the current browser feature uses your local browser instance which will help you bypass 99 of bot detections only use when necessary navigate through the pages beta navigate through the webpage and scrape data from different pages for windows users please follow the docker container guide given below as i won t be able to maintain another version for windows systems installation note cyberscraper 2077 requires python 3 10 or higher clone this repository git clone https github com itsowen cyberscraper 2077 git cd cyberscraper 2077 create and activate a virtual environment virtualenv venv source venv bin activate optional install the required packages pip install r requirements txt install the playwright playwright install set openai gemini key in your environment linux mac export openai_api_key your api key here export google_api_key your api key here using ollama note i only recommend using openai and gemini api as these models are really good at following instructions if you are using open source llms make sure you have a good system as the speed of the data generation presentation depends on how well your system can run the llm you may also have to fine tune the prompt and add some additional filters yourself 1 setup ollama using pip install ollama 2 download ollama from the official website https ollama com download 3 now type ollama pull llama3 1 or whatever llm you want to use 4 now follow the rest of the steps below docker installation if you prefer to use docker follow these steps to set up and run cyberscraper 2077 ensure you have docker installed on your system clone this repository git clone https github com itsowen cyberscraper 2077 git cd cyberscraper 2077 build the docker image docker build t cyberscraper 2077 run the container docker run p 8501 8501 e openai_api_key your actual api key e google_api_key your actual api key cyberscraper 2077 using ollama with docker if you want to use ollama with the docker setup install ollama on your host machine following the instructions at https ollama com download run ollama on your host machine ollama pull llama3 1 find your host machine s ip address on linux mac ifconfig or ip addr show on windows ipconfig run the docker container with the host network and set the ollama url docker run e ollama_base_url http host docker internal 11434 p 8501 8501 cyberscraper 2077 now visit the url http localhost 8501 on linux you might need to use this below docker run e ollama_base_url http your host ip 11434 p 8501 8501 cyberscraper 2077 replace your host ip with your actual host machine ip address in the streamlit interface select the ollama model you want to use e g ollama llama3 1 note ensure that your firewall allows connections to port 11434 for ollama usage fire up the streamlit app source venv bin activate if not yet done streamlit run main py open your browser and navigate to http localhost 8501 enter the url of the site you want to scrape or ask a question about the data you need ask the chatbot to extract the data in any format select whatever data you want to export or even everything from the webpage watch as cyberscraper 2077 tears through the net extracting your data faster than you can say flatline example usage with page ranges https example com products 1 5 https example com search q cyberpunk page page 1 10 multi page scraping beta note the multi page scraping feature is currently in beta while functional you may encounter occasional issues or unexpected behavior we appreciate your feedback and patience as we continue to improve this feature cyberscraper 2077 now supports multi page scraping allowing you to extract data from multiple pages of a website in one go this feature is perfect for scraping paginated content search results or any site with data spread across multiple pages how to use multi page scraping i suggest you enter the url structure every time if you want to scrape multiple pages so it can detect the url structure easily it detects nearly all url types basic usage to scrape multiple pages use the following format when entering the url https example com page 1 5 https example com p 1 6 https example com xample something something 1279 p 1 1 3 this will scrape pages 1 through 5 of the website custom page ranges you can specify custom page ranges https example com p 1 5 7 9 12 https example com xample something something 1279 p 1 1 7 8 9 this will scrape pages 1 to 5 page 7 and pages 9 to 12 url patterns for websites with different url structures you can specify a pattern https example com search q cyberpunk page page 1 5 replace page with where the page number should be in the url automatic pattern detection if you don t specify a pattern cyberscraper 2077 will attempt to detect the url pattern automatically however for best results specifying the pattern is recommended tips for effective multi page scraping start with a small range of pages to test before scraping a large number be mindful of the website s load and your scraping speed to avoid overloading servers use the simulate_human option for more natural scraping behavior on sites with anti bot measures regularly check the website s robots txt file and terms of service to ensure compliance example url example https news ycombinator com p 1 1 3 or 1 2 3 4 if you want to scrape a specific page just enter the query please scrape page number 1 or 2 if you want to scrape all pages simply give a query like scrape all pages in csv or whatever format you want tor network scraping note the tor network scraping feature allows you to access and scrape onion sites this feature requires additional setup and should be used responsibly and legally cyberscraper 2077 now supports scraping onion sites through the tor network allowing you to access and extract data from the dark web safely and anonymously this feature is perfect for researchers security analysts and investigators who need to gather information from tor hidden services prerequisites install tor on your system ubuntu debian sudo apt install tor macos using homebrew brew install tor start the tor service sudo service tor start on linux brew services start tor on macos install additional python packages pip install pysocks requests socks using tor scraping basic usage simply enter an onion url and cyberscraper will automatically detect and route it through the tor network http example123abc onion safety features automatic onion url detection built in connection verification tor browser like request headers automatic circuit isolation configuration options you can customize the tor scraping behavior by adjusting the following settings tor_config torconfig socks_port 9050 default tor socks port circuit_timeout 10 timeout for circuit creation auto_renew_circuit true automatically renew tor circuit verify_connection true verify tor connection before scraping security considerations always ensure you re complying with local laws and regulations use a vpn in addition to tor for extra security be patient as tor connections can be slower than regular web scraping avoid sending personal or identifying information through tor some onion sites may be offline or unreachable docker support for docker users add these additional flags to enable tor support docker run p 8501 8501 network host e openai_api_key your api key cyberscraper 2077 example usage setup google sheets authentication go to the google cloud console https console cloud google com select your project navigate to apis services credentials find your existing oauth 2 0 client id and delete it click create credentials oauth client id choose web application as the application type name your client e g cyberscraper 2077 web client under authorized javascript origins add http localhost 8501 http localhost 8502 http 127 0 0 1 8501 http 127 0 0 1 8502 under authorized redirect uris add http localhost 8501 http 127 0 0 1 8501 http localhost 8502 http 127 0 0 1 8502 click create to generate the new client id download the new client configuration json file and rename it to client_secret json ️ adjusting playwrightscraper settings optional customize the playwrightscraper settings to fit your scraping needs if some websites are giving you issues you might want to check the behavior of the website use_stealth bool true simulate_human bool false use_custom_headers bool true hide_webdriver bool true bypass_cloudflare bool true adjust these settings based on your target website and environment for optimal results you can also bypass the captcha using the captcha parameter at the end of the url the browser window will pop up complete the captcha and go back to your terminal window press enter and the bot will complete its task contributing we welcome all cyberpunks netrunners and code samurais to contribute to cyberscraper 2077 fork the repository create your feature branch git checkout b feature amazing feature commit your changes git commit m add some amazing feature push to the branch git push origin feature amazing feature open a pull request troubleshooting ran into a glitch in the matrix let me know by adding the issue to this repo so that we can fix it together faq q is cyberscraper 2077 legal to use a cyberscraper 2077 is designed for ethical web scraping always ensure you have the right to scrape a website and respect their robots txt file q can i use this for commercial purposes a yes under the terms of the mit license license this project is licensed under the mit license see the license file for details use it mod it sell it just don t blame us if you end up flatlined contact got questions need support want to hire me for a gig email owensingh72 proton me website owen sh disclaimer listen up choombas before you jack into this code you better understand the risks this software is provided as is without warranty of any kind express or implied the authors are not liable for any damages or losses resulting from the use of this software this tool is intended for educational and research purposes only any illegal use is strictly prohibited we do not guarantee the accuracy completeness or reliability of any data obtain...