{"id":3737,"date":"2011-12-28T13:36:53","date_gmt":"2011-12-28T11:36:53","guid":{"rendered":"http:\/\/djalil.chafai.net\/blog\/?p=3737"},"modified":"2012-03-24T14:52:14","modified_gmt":"2012-03-24T13:52:14","slug":"w2m-python-module","status":"publish","type":"post","link":"https:\/\/djalil.chafai.net\/blog\/2011\/12\/28\/w2m-python-module\/","title":{"rendered":"Web to matrix Python module"},"content":{"rendered":"

The beginning of winter is often the occasion to write programs. This year I wrote just for fun w2m.py<\/a><\/strong>, answering a question raised by my co-author Charles Bordenave<\/a>. This <\/strong> Python<\/a> 2.7 module+program consists in a spider which explores part of the World Wide Web, extracts the adjacency matrix, and computes its spectrum. Below the results for Wikipedia in Kabyle language<\/a>. On the technical side, w2m.py<\/strong> has a simple object oriented<\/a> code, which makes use of threading<\/a> and numpy<\/a> + matplotlib<\/a>. I will hopefully convert it to Python 3.x some day (2to3<\/a>). For Debian\/GNU Linux, you may use apt-get install python-matplotlib python-scipy python-argparse<\/em> in order to install all the necessary packages.<\/p>\n

Related:<\/strong> The University of Florida Sparse Matrix Collection <\/a> (pointed out by Charles Bordenave<\/a>).<\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

\"Adjacency
Adjacency matrix spectrum for Wikipedia in Kabyle language (obtained in December 2011). The darkness of each point corresponds to the delocalization of the eigenvector.<\/figcaption><\/figure>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

\"Adjacency
Adjacency matrix of Wikipedia in Kabyle language (obtained in December 2011)<\/figcaption><\/figure>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n

Here is a chunk of the console output of w2m:<\/p>\n

INFO:MainThread:_time_stamp():w2m.py 2011-12-28 12:27 http:\/\/kab.wikipedia.org\/wiki\r\nINFO:MainThread:work():please be patient, this may take some time\r\nINFO:MainThread:work():urls=0(0,0) edges=0 vertices=0 threads=1\r\n...\r\nINFO:MainThread:work():urls=348(334,205) edges=3604 vertices=553 threads=20\r\n...\r\nINFO:MainThread:work():urls=743(713,0) edges=7544 vertices=743 threads=1\r\nINFO:MainThread:_time_stamp():w2m.py 2011-12-28 12:30 http:\/\/kab.wikipedia.org\/wiki\r\nINFO:MainThread:_time_stamp():w2m.py 2011-12-28 12:30 http:\/\/kab.wikipedia.org\/wiki\r\nINFO:MainThread:show():number of accepted vertices: 743\r\nINFO:MainThread:show():number of accepted edges: 7547\r\nINFO:MainThread:show():number of explored vertices: 714\r\nINFO:MainThread:show():number of http head errors: 6\r\nINFO:MainThread:show():number of http get errors: 30\r\nINFO:MainThread:show():number of analyzed urls: 744\r\nINFO:MainThread:show():vertices coverage 96%.\r\nINFO:MainThread:show():saving 'adjacency.jpg'\r\nINFO:MainThread:show():computing eigenvalues and eigenvectors\r\nINFO:MainThread:show():saving 'spectrum.jpg'\r\nINFO:MainThread:show():saving 'octave.mat', use load('octave.mat') in octave\r\nINFO:MainThread:show():saving 'vertices.lst'\r\nINFO:MainThread:show():show() finished.\r\nINFO:MainThread:_time_stamp():w2m.py 2011-12-28 12:30 http:\/\/kab.wikipedia.org\/wiki<\/pre>\n","protected":false},"excerpt":{"rendered":"
The beginning of winter is often the occasion to write programs. This year I wrote just for fun w2m.py, answering a question raised by my…<\/p>\n
Continue readingWeb to matrix Python module<\/span><\/a><\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":45},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/posts\/3737"}],"collection":[{"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/comments?post=3737"}],"version-history":[{"count":83,"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/posts\/3737\/revisions"}],"predecessor-version":[{"id":4663,"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/posts\/3737\/revisions\/4663"}],"wp:attachment":[{"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/media?parent=3737"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/categories?post=3737"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/tags?post=3737"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}