Meta tags:
description=Blog for my GSoC project;

Headings (most frequently used words):

week, smw, gsoc, to, for, released, pencils, down, migration, diff, of, semantic, data, major, optimization, special, pages, on, sharding, real, midterm, link, my, changes, recent, posts, archives, categories, meta,

Text of the page (most frequently used words):
the (76), this (55), and (47), for (44), smw (40), now (24), you (24), gsoc (23), 2012 (17), using (17), but (16), with (15), that (14), more (14), from (14), project (13), can (13), have (12), #special (12), was (12), very (12), caching (12), what (12), #categories (11), week (11), new (10), code (10), time (10), wiki (10), semantic (9), all (9), are (9), done (9), some (9), also (9), has (9), milestone (8), will (8), only (8), stuff (8), tables (8), pages (7), migration (7), part (7), were (7), been (7), there (7), users (7), not (7), after (7), still (7), which (7), use (7), work (7), your (7), properties (7), queries (7), site (6), com (6), july (6), wikimedia (6), org (6), where (6), about (6), version (6), script (6), got (6), property (6), result (6), query (6), diff (5), data (5), older (5), many (5), one (5), here (5), its (5), had (5), tests (5), then (5), used (5), how (5), yet (5), don (5), http (5), wordpress (4), write (4), green (4), major (4), released (4), separate (4), them (4), future (4), read (4), make (4), without (4), any (4), might (4), booleans (4), little (4), mysql (4), faster (4), actually (4), when (4), table (4), page (4), store (4), newer (4), tags (4), identified (4), few (4), out (4), simple (4), way (4), smw_ids (4), null (4), post (4), later (4), made (4), source (4), summer (4), identification (4), memcache (4), mediawiki (4), required (3), log (3), already (3), back (3), create (3), blog (3), uncategorized (3), june (3), august (3), changes (3), gerrit (3), types (3), fields (3), most (3), places (3), boolean (3), uses (3), each (3), merge (3), type (3), suggested (3), jeroen (3), such (3), proposal (3), greensmw (3), testing (3), while (3), commit (3), able (3), let (3), mostly (3), lots (3), store3 (3), sharding (3), import (3), bugs (3), want (3), need (3), highly (3), run (3), know (3), fixed (3), would (3), querying (3), method (3), really (3), large (3), smw_namespace (3), ref (3), rows (3), did (3), old (3), much (3), improvements (3), plan (3), usage (3), however (3), thus (3), efficient (3), database (3), inline (3), invalidation (3), their (3), they (3), consider (3), talk (3), mentors (3), open (3), task (3), google (3), get (2), like (2), website (2), email (2), comment (2), comments (2), content (2), sign (2), subscribed (2), subscribe (2), account (2), other (2), feed (2), entries (2), discussions (2), december (2), optimization (2), pencils (2), down (2), posts (2), branch (2), storerewrite (2), see (2), https (2), nischayn22 (2), link (2), till (2), added (2), modified (2), support (2), similar (2), unnecessary (2), two (2), enough (2), again (2), abstracted (2), could (2), performance (2), easier (2), existing (2), update (2), flies (2), releasing (2), markus (2), sqlstore3 (2), sqlstore2 (2), just (2), help (2), upgrade (2), side (2), datatype (2), finally (2), reading (2), writing (2), before (2), applied (2), unittests (2), test (2), than (2), good (2), start (2), beware (2), isn (2), lot (2), next (2), shall (2), mentor (2), midterm (2), master (2), keep (2), into (2), own (2), plus (2), local (2), refresh (2), php (2), extensively (2), should (2), marked (2), document (2), even (2), initially (2), foreign (2), keys (2), awesome (2), assign (2), yes (2), control (2), right (2), explain (2), select (2), 102 (2), order (2), _sortkey (2), limit (2), 200 (2), select_type (2), possible_keys (2), key (2), key_len (2), extra (2), ns_sk (2), 1010 (2), row (2), set (2), sec (2), runs (2), scanning (2), return (2), same (2), 250 (2), 724 (2), following (2), indexes (2), slow (2), worked (2), amazing (2), involved (2), information (2), last (2), counts (2), semanticwatchlist (2), core (2), swl (2), improve (2), behind (2), does (2), semantics (2), working (2), besides (2), left (2), switch (2), process (2), hopefully (2), developers (2), helpful (2), smwcon (2), students (2), early (2), give (2), something (2), best (2), aspect (2), participating (2), opportunity (2), software (2), stats (2), profiling (2), documentation (2), complex (2), templates (2), based (2), frequently (2), exportrdf (2), being (2), accesses (2), validation (2), writes (2), hash (2), found (2), semantic_mediawiki_1 (2), 0_released (2), world (2), started, design, name, loading, collapse, bar, manage, subscriptions, view, reader, report, privacy, join, subscribers, top, free, meta, march, april, may, archives, recent, search, owner, somehow, naming, field, columns, geo, three, layer, access, easily, enhance, separately, string, blob, elaborate, separated, layout, add, modify, ones, follows, series, schema, wow, month, discussed, though, release, keeping, pressed, sqlstore, redundant, effects, permits, suprised, popular, discovered, handling, bug, research, settled, tinyint, pgsql, checked, works, wise, space, makes, values, showed, interest, chose, making, useful, broken, rather, our, initial, attempts, running, stuck, waiting, paid, recommend, gsocers, rants, dihandler, classes, late, sum, line, below, says, 6647, true, whole, inherited, tweaking, various, looking, forward, tasks, upcoming, discuss, maybe, evaluation, come, doesn, put, expected, tiresome, job, thanks, who, along, upgradation, led, copying, files, folder, latest, mine, 14773, git, conflicts, intensive, written, massive, semanticweb, hiding, please, setup, installed, check, default, assume, far, gonna, estimate, say, likely, gibberish, mark, sql, perfectly, btw, created, rumored, fairly, uplift, less, things, load, memory, lesser, storage, requirement, compare, feature, lets, shard, admin, worry, figure, fix, heard, bragging, rights, comes, ahead, real, fast, cache, expensive, wikis, const, index, smw_sortkey, filesort, indeed, analysis, results, generating, specialproperties, joins, limited, omitted, simpler, multiple, days, specialpages, since, mind, recentsemanticchanges, currently, scope, suitable, versions, together, wasn, improving, maintaining, statistics, unusedproperties, instead, always, realizing, stupid, obtaining, shows, removed, somewhat, weeks, planning, challenges, coming, expecting, available, browse, semanticmediawiki, devel, thread, subject, small, tweaks, notice, sometime, todo, scripts, hassle, doing, refreshdata, wait, hours, complete, directly, migrate, uninterruptedly, safely, end, full, user, promise, straightforward, consuming, throughout, maintain, friendly, atmosphere, welcomes, contributors, thankful, deutschland, funding, travel, germany, suggest, continue, thing, community, previous, year, possible, intimidated, big, codes, tips, having, beneficial, challenge, changed, broke, issues, occurred, times, challenging, contributing, hundreds, people, gave, immense, exposure, development, fully, mature, produce, disrupting, activity, product, organizing, admins, takes, those, unit, covered, parts, phpunit, plans, everything, going, trivial, solution, involve, planned, different, technique, invalidate, storing, metadata, bigger, decided, approach, mwjames, nabble, dauw, td4981469, html, none, low, priority, alternative, strategy, methods, change, complicated, refactoring, easy, completed, achieved, main, deliverables, proposed, original, www, melange, review, gsoc2012, idea, completion, summary, overall, life, changing, introduced, introducing, others, october, slides, attended, wikipedia, devcamp, meet, wmf, volunteers, contributions, continues, forever, smwcon_fall_2012, improvements_in_sqlstore3, presentation, sure, happy, announce, contribution, over, home,

Text of the page (random words):
project green smw blog for my gsoc project home about project green smw blog for my gsoc project smw 1 8 released december 3 2012 1 not sure if this is my last post for gsoc 12 but i am happy to announce that my contribution to semantic mediawiki over the summer as part of my google summer of code project greensmw is now released as smw 1 8 http semantic mediawiki org wiki semantic_mediawiki_1 8 0_released overall this summer has been a life changing time for me from being introduced to a new world of open source software to introducing others to the same i also had the opportunity to give a talk on my work for smw at the smwcon in october slides can be found at http semantic mediawiki org wiki smwcon_fall_2012 improvements_in_sqlstore3 presentation later i attended the wikipedia devcamp and got to meet with lot of developers from wmf and other volunteers its an amazing world out there hopefully my contributions to open source continues like this forever http semantic mediawiki org wiki semantic_mediawiki_1 8 0_released tags gsoc smw categories uncategorized pencils down gsoc august 21 2012 0 this post is a after completion summary of my gsoc project greensmw what was the idea of this project the original proposal can be found at http www google melange com gsoc proposal review google gsoc2012 nischayn22 1 the main deliverables proposed there were validation of writes using a hash caching of special pages identification and caching of frequently made queries in special exportrdf improvements to smw s accesses to the database identification and caching of large inline queries or complex templates using memcache profiling and documentation what part of this has been achieved what was left behind validation of writes using a hash done was very easy to do got completed in very early improvements to smw s accesses to the database done this was the most complicated task as it involved lots of refactoring of old code caching of special pages an alternative strategy is being applied here the special page methods are made very efficient and now don t need any caching as such yet to commit this change identification and caching of frequently made queries in special exportrdf this was later identified as very low priority as many more places were identified to improve identification and caching of large inline queries or complex templates using memcache this task was later identified as not so trivial memcache uses time based caching which is not a good solution for query as they involve lots of invalidation we planned to work on a different technique to invalidate queries by storing their metadata this is a bigger task and we decided to do it post gsoc however users can use a memcache based approach till then as mwjames has been using http wikimedia 7 n6 nabble com re query result caching and invalidation jeroen de dauw td4981469 html none profiling and documentation mostly done but more part to be done when smw 1 8 is going to be released what was not in the plan we don t have plans for everything do we unit tests we covered some parts of smw s code using phpunit tests fixed properties side product of re organizing the db stuff wiki admins can assign separate tables for highly used properties so querying takes little time on those migration script a script to let users actually switch to smw 1 8 without disrupting their site s activity semantic diff and site stats not fully mature stuff but smw will now be able to produce a diff of the semantic data and also store stats of property usage what do you consider the best aspect of participating in gsoc the best aspect of participating was contributing to a project that hundreds of people use besides this opportunity gave me immense exposure to the process of software development in open source what do you consider the most challenging part of your summer working with existing code was a challenge i changed something here and it broke something there such issues occurred many times how were your mentors awesome having two mentors was really beneficial which tips would you give to future students talk to previous year students talk to mentors as early as possible don t be intimidated by big source codes what one thing did the wikimedia community do that you consider very helpful for your project and would suggest they continue to do developers at wikimedia have been very helpful throughout they maintain a friendly atmosphere that welcomes more contributors i am also thankful to wikimedia deutschland for funding my travel to smwcon in germany categories gsoc milestone migration to smw 1 8 week 9 august 4 2012 0 this week i made the migration scripts so users of smw can actually use all my work without any hassle of doing a refreshdata php and wait for hours for it to complete this script will let users to directly migrate all their data from the smw s older version to the newer version while still the site could run uninterruptedly with the older version thus one can safely switch to the newer version at the end and with full control on the user s part i am yet to document this migration process but i promise it is very very straightforward and hopefully not time consuming besides this week i did a few small tweaks that had been left notice sometime or were marked as todo the following weeks we are planning to start working on caching of inline queries this has many challenges coming for us and we are expecting this to only be available in a later version for more information browse semanticmediawiki devel thread with subject query caching and invalidation tags migration upgrade categories discussions gsoc milestone diff of semantic data week 8 july 25 2012 0 this week i worked on obtaining a diff of the semantic data for a wiki page this shows you what semantics got added and what semantics got removed on a page write this is somewhat similar to what semanticwatchlist does but now in core and more efficient as a result semanticwatchlist shall be using this from core smw in the newer versions and thus smw swl will be more efficient to use together however this wasn t a plan to improve swl initially this is in result of my work behind improving smw s special pages and maintaining more statistics we will now have counts for usage of properties in the database and thus special properties and special unusedproperties will just query this table instead of querying all the tables as it does now we always keep realizing how stupid we were before p i also plan to use this counts of properties usage since we already have them now to make a few more special pages for smw i have in mind right now recentsemanticchanges only however this is currently out of scope for the project and so might be only suitable as some post gsoc work tags smw specialpages categories gsoc milestone major optimization for special pages on smw july 19 2012 0 last few days i have done some major improvements to special pages for smw the old way of generating specialproperties involved unnecessary joins and limited result some result were omitted the new method is much simpler i only query the required information in multiple simple queries which should be much much faster than the old way after this i also did some analysis on using indexes in the db tables to see how mysql worked with the new method and got amazing results the following query is done with no indexes as is now it runs very slow scanning about 10 724 rows to return only 250 of them slow indeed mysql explain select from store3 smw_ids where smw_namespace 102 order by smw _sortkey limit 200 50 id select_type table type possible_keys key key_len ref ro ws extra 1 simple smw_ids all null null null null 10 724 using where using filesort 1 row in set 0 00 sec next is my new way using index smw_namespace smw_sortkey runs faster by scanning only 1010 rows to return the same result of 250 mysql explain select from store3 smw_ids where smw_namespace 102 order by smw _sortkey limit 200 50 id select_type table type possible_keys key key_len ref rows extra 1 simple smw_ids ref ns_sk ns_sk 4 const 1010 using where 1 row in set 0 00 sec still more while the new method is really fast we still need to cache stuff as this is still expensive for really large wikis categories uncategorized sharding for real week 7 july 17 2012 0 smw now has about 28 db tables yes you heard it right and that s where my bragging rights for db sharding comes from read ahead to know more smw now has about 28 db tables this has been a major uplift from what we had initially the more tables we have the less things to load into memory at a time lesser storage requirement no foreign keys and faster querying again no foreign keys to compare plus there s an awesome feature that lets you to shard the db even more the in 28 you can assign separate tables for each of your property yes you the admin can control it now this is suggested for properties that are highly used in your wiki now don t you worry how to figure this out yet this way you can have up to as many tables as you want but still beware there might be bugs to fix ok now back to how do you know what properties will be used extensively in the future this can be an estimate say a property email is more likely to be used extensively so it should be marked as a fixed property yet to document on this stuff but if you don t want all this gibberish you can mark a property as fixed property even when its in use but then you would have to run a sql script after that so your site can work perfectly ok btw this script is yet to be created but is rumored to be fairly simple to create ok now the testing part i have done some intensive testing using my own written tests for smw plus on a local wiki which has a massive import from semanticweb org and bugs are identified mostly but there might be a few of them still hiding for you if you are reading this and want to help us to test stuff please do it all you need to do is setup a local wiki with smw installed do check out from the master and then make the default store sqlstore3 then import from a highly used smw wiki which i assume you own if you have read this far p and then after the import is done run a refresh php to refresh all your smw content after that i know you are gonna comment here about the bugs tags sharding store3 categories gsoc milestone gsoc midterm july 9 2012 6 hi the midterm evaluation has come time flies doesn t it and finally i am able to put my code to master branch i had expected this to be a tiresome job with lots of git merge conflicts but thanks to my mentor markus who suggested to keep the older code along with the newer one to support easier upgradation for users this led to only copying the new files into a separate folder on the store here s the link to this latest commit of mine https gerrit wikimedia org r c 14773 but beware when you read that line on the page below which says 6647 7 this isn t true actually p as the whole lot of code isn t my work its mostly inherited code from the older store with my tweaking at various places i am looking forward to the next tasks upcoming now as i shall discuss with my mentor maybe its time for some caching now categories gsoc milestone week 5 6 july 5 2012 0 this update got a little late but let me sum it up new stuff unittests dihandler classes new table for booleans and now the rants start before i applied to gsoc i had showed interest in jeroen s proposal for writing unittests for smw now that i chose greensmw he is still making me write tests which is actually very useful to test broken stuff rather than our initial attempts of testing by running a wiki so when i get stuck for a while waiting for some commit to merge i write tests with this i am able to make very good use of my paid time i recommend this for all gsocers now back to the booleans after a little research on boolean datatype i finally settled on using tinyint 1 for mysql and boolean for pgsql i checked this stuff and it works performance wise we are using very little space for booleans now and this also makes reading and writing such values faster i was suprised that this is not a popular datatype for smw users after i discovered a bug in smw s handling of booleans also we discussed about releasing the storerewrite version and though markus suggested to release it as sqlstore3 keeping also sqlstore2 i and jeroen pressed on releasing it as just sqlstore without any redundant code from sqlstore2 there might be a migration script to help users upgrade without any side effects but only if time permits categories gsoc milestone week 4 june 20 2012 0 wow its been a month already time flies this week i abstracted read as separated the db layout for smw this will make it easier to add new di types in the future and modify the existing ones from here follows a series of update to the db schema till now as new di types were added the code was modified to support them somehow by using similar naming of fields for most di types and using some unnecessary fields in many places boolean field uses two columns but one is enough and geo uses three fields where again one is enough but now that the db is abstracted at each di layer db access could be easily modified to enhance performance for each di separately and also merge the di type string with blob will elaborate on this more in future posts categories gsoc milestone link to my changes june 19 2012 0 all the changes to smw as part of my gsoc project are done on a separate branch storerewrite you can see my changes on gerrit https gerrit wikimedia org r q owner nischayn22 n z categories gsoc older entries search for recent posts smw 1 8 released pencils down gsoc migration to smw 1 8 week 9 diff of semantic data week 8 major optimization for special pages on smw archives december 2012 august 2012 july 2012 june 2012 may 2012 april 2012 march 2012 categories discussions gsoc milestone uncategorized meta create account log in entries feed comments feed wordpress com create a free website or blog at wordpress com back to the top subscribe subscribed project green smw join 41 other subscribers sign me up already have a wordpress com account log in now privacy project green smw subscribe subscribed sign up log in report this content view site in reader manage subscriptions collapse this bar loading comments write a comment email required name required website design a site like this with wordpress com get started