Meta tags:
description=Random thoughts on code;

Headings (most frequently used words):

python, and, in, recent, codebright, blog, linear, algebra, review, numpy, tuple, trivia, globals, __main__, gotcha, edge, case, for, split, explode, reading, gzip, files, fast, standard, library, regular, expression, surprise, php, sendkeys, linux, strange, behaviour, using, ctypes, ironpython, posts, comments, archives, categories, meta,

Text of the page (most frequently used words):
the (116), print (79), #python (53), file (49), second (44), size (43), per (42), count (42), for (40), and (36), ctypes (36), dll (33), seconds (31), time (30), line (28), this (27), nadd (22), taken (21), 1000 (21), characters (21), character (21), posted (20), you (20), 50859394 (20), 194151 (20), 4661207 (20), atuple (20), alist (20), import (19), matrix (19), gzip (18), array (18), codebright (17), c_int (17), but (17), that (16), txt (15), 2011 (13), numpy (13), array3 (13), def (13), php (13), call (12), are (12), give (12), main_module (12), with (11), test (11), __main__ (11), array2 (11), files (10), from (10), first (10), code (10), group (10), produces (10), comment (9), software (9), tuple (9), strange (9), function (9), which (9), not (9), sub_module (9), blog (8), was (8), using (8), use (8), end (8), zcat (8), splitting (8), test_split (8), g_main_value (8), index (8), reading (7), sys (7), following (7), need (7), one (7), more (7), then (7), none (7), open (7), show_globals (7), multiplication (7), inverseofa (7), log (6), have (6), 2010 (6), leave (6), ironpython (6), define (6), int (6), is_file (6), sub (6), last (6), module (6), cre (6), pipe (6), res (6), sep (6), g_sub_value (6), wordpress (5), com (5), comments (5), june (5), recent (5), case (5), split (5), explode (5), output (5), integers (5), cdll (5), loadlibrary (5), restype (5), argtypes (5), try (5), call3then2_notypes (5), call2then3_notypes (5), what (5), return (5), here (5), different (5), problem (5), runs (5), day_list (5), day_index (5), mod (5), can (5), name (4), october (4), expected (4), call_voidpointer (4), call_voidpointer_addressof (4), array3type (4), array2type (4), work (4), unexpected (4), ntotal (4), start (4), linux (4), some (4), way (4), found (4), non (4), key (4), list (4), there (4), file_exists (4), when (4), example (4), written (4), mos1 (4), mos2 (4), abcd (4), regular (4), day_name (4), ttime (4), fullfn (4), elt (4), exploding (4), test_explode (4), empty (4), string (4), global (4), main_value (4), get_main_value (4), arraya (4), transpose (4), get (3), like (3), write (3), now (3), uncategorized (3), march (3), linear (3), algebra (3), review (3), posts (3), got (3), c_long_array_3 (3), pointer (3), except (3), exception (3), exc_info (3), two (3), seems (3), note (3), will (3), works (3), came (3), across (3), slightly (3), should (3), all (3), would (3), fedora (3), course (3), keystrokes (3), helpful (3), still (3), documentation (3), even (3), requires (3), results (3), waiting (3), also (3), between (3), has (3), gotchas (3), less (3), both (3), good (3), however (3), perl (3), compile (3), most (3), match (3), expression (3), collections (3), zerodict (3), speed (3), relevant (3), http (3), bugs (3), org (3), issue7471 (3), gzipfile (3), apache (3), io_method (3), subprocess (3), dirname (3), set (3), about (3), g_main_value2 (3), __name__ (3), module_name (3), get_sub_value (3), sub_test (3), sub_init (3), doesn (3), result (3), matrixa (3), site (2), required (2), content (2), sign (2), subscribed (2), subscribe (2), already (2), account (2), feed (2), create (2), may (2), february (2), april (2), july (2), tommy (2), carstensen (2), how (2), fast (2), edge (2), globals (2), gotcha (2), trivia (2), older (2), c_void_p (2), addressof (2), trying (2), wrong (2), each (2), failing (2), c_long_array_2 (2), answer (2), least (2), gcc (2), shared (2), ifdef (2), __cplusplus (2), endif (2), pndata (2), ndatalen (2), single (2), length (2), returns (2), elements (2), windows (2), find (2), out (2), behaviour (2), look (2), later (2), xautomation (2), happy (2), around (2), them (2), hard (2), above (2), ubuntu (2), sendkeys (2), equivalent (2), home (2), better (2), running (2), while (2), disappear (2), detected (2), disappearance (2), exit (2), states (2), cached (2), tests (2), than (2), same (2), libraries (2), lines (2), number (2), side (2), effects (2), cre3 (2), fix (2), changed (2), solution (2), cre2 (2), clear (2), why (2), outputs (2), adding (2), illustrates (2), traceback (2), programs (2), python25 (2), lib (2), sre_parse (2), expand_template (2), error (2), unmatched (2), recently (2), defaultdict (2), instance (2), read (2), method (2), issue (2), logfile (2), used (2), week (2), take (2), monday (2), tuesday (2), wednesday (2), thursday (2), friday (2), saturday (2), sunday (2), enumerate (2), range (2), len (2), standard (2), library (2), 400 (2), averages (2), timings (2), startswith (2), else (2), cstringio (2), 001 (2), times (2), data (2), foreach (2), after (2), behaves (2), __import__ (2), add (2), happening (2), modules (2), sub_value (2), main_test (2), main_init (2), where (2), prr (2), producing (2), instead (2), tuples (2), finally (2), final (2), element (2), produce (2), shows (2), inverse (2), multiplying (2), linalg (2), random (2), thoughts (2), started, design, website, email, loading, collapse, bar, manage, subscriptions, view, reader, report, privacy, entries, meta, categories, archives, direct, devboell, leszek, pryszcz, search, call3then2, c_long, extra, hint, follows, three, methods, learn, signature, unfortunately, getting, successful, question, correctly, prints, lets, cpp, fpic, build, msys, mingw, extern, trivial, contains, takes, sum, deals, dlls, centric, maybe, amazed, thought, document, post, examples, date, tried, had, alternative, possibility, syntax, sending, consistent, alphanumerics, sent, xdotool, hardy, heron, worked, fine, many, alpha, numeric, useful, api, enables, program, send, looking, ideally, accurate, think, minimal, creating, deleting, invariably, detection, though, biased, let, appear, true, sleep, elseif, appears, existence, expect, repeatedly, wait, long, clearstatcache, honest, hadn, noticed, writing, absence, noticeable, presence, today, ability, easily, interface, any, language, specifically, big, advantage, opinion, verbose, particular, braces, blocks, semi, colons, variable, names, only, colon, certain, new, job, involves, lot, closely, related, observations, make, comparison, entirely, occurs, place, does, capturing, notation, fixed, without, might, numbers, vary, versions, filename, linenumber, 266, filter, template, 793, raise, sre_constants, predict, outcome, anomaly, complex, idea, surprise, provides, dictionary, initialises, documented, approach, faster, rest, unchanged, whilst, processing, offers, compression, original, uncompressed, implying, trade, off, processor, disk, activity, having, succinct, built, allows, replace, gems, once, 4364, 17691, 1621, 4362, 19834, 1818, 4408, 404, 19295, 1768, 4393, 403, 19659, 1802, 4354, 399, 17123, 1569, 5237, 480, 22300, 2044, 5363, 492, 22750, 2085, 5389, 494, 22156, 2031, 5397, 495, 22516, 2064, 5293, 485, 22033, 2019, version, bytesio, stringio, listdir, sort, endswith, continue, path, join, stat, popen, stdout, communicate, assert, returncode, naverages, below, details, source, quick, summary, yes, average, consists, 950, 000, gzipped, compared, suggested, been, doing, lots, parsing, essence, interested, know, issues, mentioned, under, shift, expecting, needing, thinking, bit, returning, logical, value, languages, special, differently, print_r, during, coding, ideal, setattr, gives, clue, loaded, twice, another, wanted, usable, other, dir, getattr, values, bottom, features, lack, traps, unwary, programmer, something, happens, thankfully, these, rare, worth, knowing, consider, named, cgibin, typeerror, enough, arguments, format, operator, similar, before, lists, cases, giving, again, excludes, include, either, left, because, negative, indexes, want, matter, beyond, excluded, couple, minor, points, regarding, slice, parameters, step, needed, 256, 264, reason, 075, identity, inv, matrices, scalar, straightforward, taking, little, obvious, uses, defined, referring, individual, based, 2nd, row, 1st, column, see, resize, installing, ways, signed, stanford, subject, areas, recommended, prefer, notes, far, gnu, octave, machine, learning, skip,

Text of the page (random words):
the following ways from numpy import matrix a matrix 3 4 2 16 or a matrix 3 4 2 16 or even a matrix 3 4 2 16 a resize 2 2 you can produce the transpose of a written a t with print transpose of a n s a transpose producing output transpose of a 3 2 4 16 or if you already have a numpy array you can create it from that we ll see later why you need to use numpy matrix and not numpy array from numpy import matrix array a2 array 3 4 2 16 a matrix a2 print a 1 0 s a 1 0 this produces a 1 0 2 note that when referring to individual elements of the matrix numpy is 0 based so a 1 0 is the 2nd row 1st column adding two matrices and multiplying a matrix by scalar is straightforward taking the inverse of a matrix is a little less obvious the example uses matrix a as defined above from numpy import linalg inverseofa linalg inv a print inverseofa n s inverseofa print a inverseofa n s a inverseofa shows the inverse written a 1 and shows that multiplying a and a 1 produces the identity matrix inverseofa 0 4 0 1 0 05 0 075 a inverseofa 1 0 0 1 finally matrix multiplication is the reason to use numpy matrix and numpy array here s an example arraya array 3 4 2 16 matrixa matrix arraya print multiplication as array n s arraya arraya print multiplication as matrix n s matrixa matrixa outputs multiplication as array 9 16 4 256 multiplication as matrix 17 76 38 264 in the array multiplication result ij a ij a ij which is not what is needed for matrix multiplication posted in python 5 comments tuple trivia posted on july 5 2011 by codebright a couple of minor points regarding tuples when you take a slice of a tuple there are 3 parameters the start index the end index the step atuple 1 2 3 4 5 print atuple 1 3 print atuple 0 7 print atuple 3 1 1 which produces 2 3 1 2 3 4 5 4 3 it doesn t matter that in the second instance the end index is beyond the last index of the string also the end index is excluded if you want to go to the start of the tuple then you need to set the end index to 1 print atuple 3 1 1 but this doesn t produce the expected result this is because negative indexes are from the end of the tuple so 1 is the last element in the tuple print atuple 2 1 produces 3 4 again this excludes the final index to include either the start or the end of the tuple as the final element of the result the end index should be left empty print atuple 2 print atuple 2 1 giving 3 4 5 3 2 1 finally in most cases where you can use a tuple you can use a list instead alist list atuple print alist 1 3 print alist 0 7 print alist 3 1 1 print alist 3 1 1 print alist 2 1 print alist 2 print alist 2 1 producing similar results as before but lists instead of tuples 2 3 1 2 3 4 5 4 3 3 4 3 4 5 3 2 1 but this doesn t work for the operator print atuple 0 s atuple 1 s atuple 2 s atuple 3 s atuple 4 s atuple print alist 0 s alist 1 s alist 2 s alist 3 s alist 4 s alist atuple 0 1 atuple 1 2 atuple 2 3 atuple 3 4 atuple 4 5 traceback most recent call last file c prr cgibin data prr codebright tuple py line 29 in module print alist 0 s alist 1 s alist 2 s alist 3 s alist 4 s alist typeerror not enough arguments for format string posted in python software leave a comment globals and __main__ a python gotcha posted on june 15 2011 by codebright one of the good features is the lack of gotchas i define gotchas as traps for the unwary programmer where something unexpected happens thankfully these are rare in python but the following is one that is worth knowing about consider two python modules named main_module py and sub_module py first main_module py g_main_value none def get_main_value return g_main_value def main_test print sub_value s get_sub_value from sub_module import get_sub_value sub_test sub_init def main_init global g_main_value g_main_value 23 if __name__ __main__ main_init sub_init main_test sub_test print main_value in main_module s get_main_value then sub_module py g_sub_value none def get_sub_value return g_sub_value def sub_test print main_value in sub_module s get_main_value def sub_init global g_sub_value g_sub_value 31 from main_module import get_main_value the output is sub_value 31 main_value in sub_module none main_value in main_module 23 what s happening a global with 2 different values add the following code to the bottom of main_module def show_globals module_name mod __import__ module_name print module_name for k in dir mod if k startswith g_ print s s k getattr mod k print if __name__ __main__ print show_globals __main__ show_globals main_module show_globals sub_module the output is __main__ g_main_value 23 main_module g_main_value none sub_module g_sub_value 31 this gives us a clue as to what s happening the __main__ module may be loaded twice if used by another module if you wanted to set a global in the __main__ module that is usable by other modules then one way to do it is add the following to main_module py mod __import__ main_module setattr mod g_main_value2 34 if __name__ __main__ print show_globals __main__ show_globals main_module show_globals sub_module and the output will be __main__ g_main_value 23 g_main_value2 none main_module g_main_value none g_main_value2 34 sub_module g_sub_value 31 posted in python software leave a comment edge case for split explode posted on april 1 2011 by codebright during recent coding using php i found that the explode function behaves in a non ideal way for me when trying to split an empty string php res explode print_r res produces array 0 i was expecting needing an empty array after thinking about it a bit more it is clear that returning a single empty string is a logical return value however different languages have different returns for this special case python 2 6 4 and 3 1 2 are both like php 5 3 5 but perl 5 10 1 behaves differently for php php function test_explode txt print exploding txt to give sep res explode txt foreach res as elt print sep elt sep print n test_explode a b test_explode test_explode produces exploding a b to give a b exploding to give exploding to give for python def test_split txt print splitting s to give s txt txt split test_split a b test_split test_split produces splitting a b to give a b splitting to give splitting to give for perl sub test_split my txt shift my res split txt print splitting txt to give my sep foreach my elt res print sep elt sep print n test_split a b test_split test_split produces splitting a b to give a b splitting to give splitting to give posted in python software leave a comment reading gzip files in python fast posted on march 25 2011 by codebright i have been doing lots of parsing of apache log files recently with python speed is of the essence so i was interested to know if the speed issues mentioned in http bugs python org issue7471 are still relevant so i set out to time it under python v2 6 4 and v3 1 2 both on fedora 13 my test data consists of 950 000 lines in 5 gzipped apache log files i compared the times using the python gzip module and the zcat method suggested in http bugs python org issue7471 a quick summary of results yes the issue is still relevant the average times are for python 2 6 4 gzip open 9 5 seconds zcat and pipe 2 3 seconds for python 3 1 2 gzip open 11 6 seconds zcat and pipe 2 7 seconds below are the details first the source code to time the tests import os import sys if sys version startswith 3 import io io_method io bytesio else import cstringio io_method cstringio stringio import gzip import subprocess import time dirname test fl os listdir dirname fl sort ttime 0 0 runs 5 for i in range 2 runs st time time for fn in fl if not fn endswith gz continue cc 0 lc 0 sz 0 fullfn os path join dirname fn sz os stat fullfn 6 if i 2 0 fh gzip gzipfile fullfn r else p subprocess popen zcat fullfn stdout subprocess pipe fh io_method p communicate 0 assert p returncode 0 for line in fh lc 1 cc len line et time time dt et st ttime i 2 dt print time taken 0 2f seconds 1000 characters per second 0 0f file size per second 0 0f character count s line count s file size s dt 0 001 cc dt 0 001 sz dt cc lc sz print naverages print gzip open 0 1f seconds ttime 0 runs print zcat and pipe 0 1f seconds ttime 1 runs timings for python 2 6 4 time taken 9 71 seconds 1000 characters per second 5237 file size per second 480 character count 50859394 line count 194151 file size 4661207 time taken 2 28 seconds 1000 characters per second 22300 file size per second 2044 character count 50859394 line count 194151 file size 4661207 time taken 9 48 seconds 1000 characters per second 5363 file size per second 492 character count 50859394 line count 194151 file size 4661207 time taken 2 24 seconds 1000 characters per second 22750 file size per second 2085 character count 50859394 line count 194151 file size 4661207 time taken 9 44 seconds 1000 characters per second 5389 file size per second 494 character count 50859394 line count 194151 file size 4661207 time taken 2 30 seconds 1000 characters per second 22156 file size per second 2031 character count 50859394 line count 194151 file size 4661207 time taken 9 42 seconds 1000 characters per second 5397 file size per second 495 character count 50859394 line count 194151 file size 4661207 time taken 2 26 seconds 1000 characters per second 22516 file size per second 2064 character count 50859394 line count 194151 file size 4661207 time taken 9 61 seconds 1000 characters per second 5293 file size per second 485 character count 50859394 line count 194151 file size 4661207 time taken 2 31 seconds 1000 characters per second 22033 file size per second 2019 character count 50859394 line count 194151 file size 4661207 averages gzip open 9 5 seconds zcat and pipe 2 3 seconds timings for python 3 1 2 time taken 11 65 seconds 1000 characters per second 4364 file size per second 400 character count 50859394 line count 194151 file size 4661207 time taken 2 87 seconds 1000 characters per second 17691 file size per second 1621 character count 50859394 line count 194151 file size 4661207 time taken 11 66 seconds 1000 characters per second 4362 file size per second 400 character count 50859394 line count 194151 file size 4661207 time taken 2 56 seconds 1000 characters per second 19834 file size per second 1818 character count 50859394 line count 194151 file size 4661207 time taken 11 54 seconds 1000 characters per second 4408 file size per second 404 character count 50859394 line count 194151 file size 4661207 time taken 2 64 seconds 1000 characters per second 19295 file size per second 1768 character count 50859394 line count 194151 file size 4661207 time taken 11 58 seconds 1000 characters per second 4393 file size per second 403 character count 50859394 line count 194151 file size 4661207 time taken 2 59 seconds 1000 characters per second 19659 file size per second 1802 character count 50859394 line count 194151 file size 4661207 time taken 11 68 seconds 1000 characters per second 4354 file size per second 399 character count 50859394 line count 194151 file size 4661207 time taken 2 97 seconds 1000 characters per second 17123 file size per second 1569 character count 50859394 line count 194151 file size 4661207 averages gzip open 11 6 seconds zcat and pipe 2 7 seconds posted in python software 7 comments python standard library posted on march 13 2011 by codebright there are some gems in the python standard library which once found get regular use at least by me here are some of them 1 enumerate a built in function which allows you to replace code like this day_list monday tuesday wednesday thursday friday saturday sunday for day_index in range len day_list day_name day_list day_index print day_index day_name with the more succinct day_list monday tuesday wednesday thursday friday saturday sunday for day_index day_name in enumerate day_list print day_index day_name 2 gzip gzipfile i used this whilst processing apache log files last week it offers compression to less than 10 of the original uncompressed size but seems to take the same length of time to read write implying a good trade off between processor use and hard disk activity having changed the open code from fh open logfile r to import gzip fh gzip gzipfile logfile r the rest of the code runs unchanged note if you are running on linux then the issue of speed documented at http bugs python org issue7471 is relevant and the zcat approach is faster 3 collections defaultdict this provides a dictionary which initialises a key on first read using a method for instance import collections zerodict collections defaultdict int print zerodict key zerodict 4 produces 0 0 posted in python software leave a comment python regular expression surprise posted on february 19 2011 by codebright i came across this regular expression anomaly recently in python the regular expression was more complex but this example illustrates the idea try and predict the outcome should work with python 2 5 python 3 1 import re abcd abcd ad ad cre re compile r a bc d mos1 cre match abcd print cre sub r 1 2 3 abcd mos2 cre match ad print cre sub r 1 2 3 ad this produces a bc d traceback most recent call last file filename line linenumber in module print cre sub r 1 2 3 ad file c programs python25 lib re py line 266 in filter return sre_parse expand_template template match file c programs python25 lib sre_parse py line 793 in expand_template raise error unmatched group sre_constants error unmatched group line numbers vary with different versions of python this is python 2 5 adding the following code illustrates the problem print mos1 group 1 mos1 group 2 mos1 group 3 print mos2 group 1 mos2 group 2 mos2 group 3 for python 2 5 this outputs a bc d a none d so now it is clear why it is failing what is the solution first fix was cre2 re compile r a bc d print cre2 sub r 1 2 4 ad this fix works but it has side effects the last group number has changed so a better solution might be cre3 re compile r a bc d print cre3 sub r 1 2 3 ad using the non capturing group notation the problem is fixed without side effects i m not entirely happy that the problem occurs in the first place but it does posted in python software 1 comment php and python posted on october 20 2010 by codebright my new job involves a lot more php which seems closely related to perl there are a number of observations that i would make in the comparison between php and python 1 php is more verbose in particular php requires braces around code blocks semi colons on variable names python requires only the colon at the end of certain lines 2 both have good libraries however the ability of python to easily interface to shared libraries written in any language and not specifically written for python is a big advantage in my opinion 3 php has more unexpected gotchas i hadn t noticed this when writing in python the absence of a problem is less noticeable than the presence of the same problem one example i came across today is_file tests for the existence of a file as you would expect by the name but it is cached so if you are waiting for a file to ...