Getting file position using /proc

As I wrote in my post of 2009, "Often there is a need to check what's some process doing right now". That time I discovered this simple way of checking which file a process(e.g. grep) was using.

Another trick of this kind is to get a file position. This is a very common scenario. For example, you want to know the progress of unpacking of a really huge .gz file. You cannot just estimate this by analyzing the size of the data already unpacked.

Getting file position is as easy as getting a name of an open file. Thanks to Miklos Szeredi starting from Linux-2.6.22 this info can be obtained from /proc/PID/fdinfo/FD:

$ python
>>> f=open('/home/turist/work/big.txt')
>>> f.seek(1234)

(in another terminal, 3580 is python's PID)
$ ls -al /proc/3580/fd/
total 0
dr-x------ 2 turist turist  0 2011-09-21 14:40 .
dr-xr-xr-x 7 turist turist  0 2011-09-21 14:40 ..
lrwx------ 1 turist turist 64 2011-09-21 14:40 0 -> /dev/pts/1
lrwx------ 1 turist turist 64 2011-09-21 14:40 1 -> /dev/pts/1
lrwx------ 1 turist turist 64 2011-09-21 14:40 2 -> /dev/pts/1
lr-x------ 1 turist turist 64 2011-09-21 14:40 3 -> /home/yevgen/work/big.txt

(so 3 is our file descriptor, we will use it now)

$ cat /proc/3580/fdinfo/3
pos:	1234
flags:	0100000

Easy!

PR 1.2 - major disappointment of the year

Just updated my two N900's to PR 1.2. Well...

Pure disappointment. None of the bugs I cared about was fixed even though it's set as fixed on bugzilla:

Bug 7211(Unable to play media. Media format not supported" for recently recorded video) - not fixed.
Bug 6823(media player won't play any video files now (.avi) divx / xvid) - not fixed.

It fails even to play its own built-in "Nokia N900.avi". Since December there's been no way to play video files on this so called "Mobile Computer"!

They added Skype video calls, but it's like slideshow.

Calling with Skype? "General error" after 1 minute and no even a sign of your call on the other side. After call pulseaudio uses 20% of CPU. I found that because both of devices were too hot for no reason.

Portrait browsing? Really?! Check the screenshot. Where is the address bar? Where is the window switching button? Exit fullscreen mode button? Is this portrait browsing or portrait viewing only?

I'm not sure whether I should wait for MeeGo or just give up and buy Android next time.

Maemo DDP: a fly in the ointment

Last week I got my DDP's N900 and since I still have my Summit's device I haven't played with it much. I wish Marcin Juszkiewicz wrote his DDP device checklist earlier. I got this nasty bug 6823 "media player won't play any video files now (.avi) divx / xvid". I tried all files, even the default "Nokia N900" and "9". Since Mplayer plays all video files, this is a software fault, not hardware issue. But still it's frustrating and I'm waiting for any bug activity.

So my point is, all people who got their devices, please check for video playback in media player. And vote for the bug if necessary.

Making your hard drive faster

Some manufacturers worry too much about their customers' comfort. For example, Apple disabled multitasking on their iPhones because too many running processes could slow down the system's responsiveness and everyone would say "iPhone is slow" rather than "you have too much stuff running, close something". In this case IMHO Apple worries more about themselves, but whatever.

In my case it's Dell/WD who worried too much about my comfort. My workhorse is Dell Precision T5400 with 2 quad-core Xeons. While running anything on it in 8 processes is fun, disk is always a bottleneck.

Let me introduce AAM(wikipedia: Automatic acoustic management). This option is used for decreasing the speed of disk rotation and head positioning. And for some people it's really necessary. But at work I have other people who, combined, produce much more noise. Thus, I don't need an additional care.

For my Linux PC 'hdparm' is a trusted tool:

# hdparm
...

-M  get/set acoustic management (0-254, 128: quiet, 254: fast)
...

Checking current setting:

# hdparm -M /dev/sda

/dev/sda:
acoustic      = 128 (128=quiet ... 254=fast)

And changed to max performance:

# hdparm -M 254 /dev/sda

Useful tool for testing the average random access time - seeker.c:

Benchmark results:

AAM=128:
./seeker /dev/sda
Results: 58 seeks/second, 17.18 ms random access time


AAM=254:
./seeker /dev/sda
Results: 76 seeks/second, 13.05 ms random access time

I think I already feel this 30% improvement. Or maybe it's the placebo effect :)

First steps with Maemo 5 SDK - Stardict

It looks like there is a lot of hype about new N900. I don't think I'll be able to afford one for myself but there are still people who will need a dictionary program while they are offline.

4 hours spent to get the first working binary. And there is a huge amount of work to do:

• read Maemo 5 HIG doc
• change the layout of potentially everything as all the buttons are HUGE now and previous useful area got shrinked.
• work around the bug "Xephyr crashes on clicking any editable area". Hate to have compiled/make_installed thing in my system.
  

And, of course, a screenshot:

Look inside the process using /proc

That's what might be useful even for web-developers, not just sysadmins and geeky geeks.

Often there is a need to check what's some process doing right now. `top` is of course showing us 100% of CPU consumption but you need to have some sort of progress to decide if you can wait or you'll just kill the task and do it in a different way.

Let's take `grep` as an example of some task that may take hours on a big set of files:

user@host [/tmp/files] $ grep -e "$PATTERN" *.dat

So to use magic you don't need patch grep or to have knowledge of how to debug Linux kernel. I use `/proc` for this task:

1. Find PID of the grep process, like this "ps axuww | grep grep" :)
   
2. Exec "ls -al /proc/$PID/fd/"
3. You'll get directory list as an output. One of these file descriptors the a process is grep's current file it's processing right now.
   

lrwx------ 1 user 64 2009-07-05 04:12 0 -> /dev/pts/2
lrwx------ 1 user 64 2009-07-05 04:12 1 -> /dev/pts/2
lrwx------ 1 user 64 2009-07-05 04:12 2 -> /dev/pts/2
lr-x------ 1 user 64 2009-07-05 04:12 3 -> /tmp/files/2009-07-01_4534545435.dat

Voila!

For progress calculation you  can use "ls *.dat > list.txt" and then find which line you on right now.

Python surprises - scope of variables

Yesterday while debugging one place in some function I found one "surprise" in Python - language that I thought was designed to avoid surprises as much as possible(Principle of least astonishment).

if 1 == 1:
    print dir()
    found = True
    print dir()

print dir()
print "found:", found

Output:

['__builtins__', '__doc__', '__file__', '__name__', '__package__']
['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'found']
['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'found']
found: True

As you would have guessed coming from C++, here we had variable used out of its scope. But in Python variables live in special dictionaries - one is global where global variables and other stuff are stored and another one is local, for keeping local symbols inside the function. So all variables you create inside "for" loop or "if" scope will be stored in that local dictionary and will only be removed from there when Python's garbage collector decides.

That surprise cost me 30 minutes of wondering why some polygons on a map suddenly went nuts and lost their shapes :)

Amazon sells N810 almost for free!

Hi all,

Well, "free" in this case is actually $227, but anyway it means that those of us who can't wait until Maemo5-equipped device is released can have something right now.

I've been following the news about buy.com prices, but it was not available for non-US people. Amazon fixed this injustice :)

Nice feature of snprintf()

Hi all,

Yesterday I learned a feature which I'm sure should be described in every C book. To be honest, 'man snprintf' describes it as a part of C99, but I've never read the article completely.

Often when you construct a string using 'snprintf()' you need to allocate enough space for the buffer. But it's not possible to know how much you may need, so usually we just do something like this:

char buf1[1024];
/* or */
char *buf2 = malloc(1024);

which is not correct even if you are 100% sure that you string is just a "Hello world". Btw, you are wasting your stack space in first example.

So the trick here is to call 'snprintf()' twice:

int len = 1 + snprintf(NULL, 0, "%s, %s\n", "Hello", "world") ;
char *buf = malloc(len);
snprintf(buf, len, "%s, %s\n", "Hello", "world");

snprintf's behavior in C99 is to allow buffer to be NULL, but return number of bytes that would have been written to buffer. Note, second parameter should be zero too :)

Fring for Maemo

Hi all!

Great news - Fring is now available for maemo platform. Fring is an application that combines a number of protocols such as Skype, ICQ, GTalk into one handy program. You may find it following this link. Here is the screenshot I took from Fring's russian blog. It should work for both chinook and diablo. The announce says about N810, but I'll give it a try tomorrow on my N800.

Yandex plugin for Maemo mapper

After patching maemo-mapper application to support Yandex maps Max Lapan, their employee suggested his script that worked as CGI script and converted mmapper's links to yandex's ones. That was really awesome idea as there is no need to update application and contacting its developer all the time, we can just run script on 127.0.0.1 and pass parameters like city ID and x/y/z to it. Note, that Yandex is not that simple as other map vendors - there is no world map with all cities on it. Each city has its own ID and its own set of dimensions - Xs,Ys of upper left and bottom right corners of each rectangle. So I think we(interested people from Russia and Ukraine) will have to create list of supported cities with verified set of dimensions and IDs. So each city will have it's own repository in maemo mapper(see screenshot below).

I'm open to discussions about this implementation as GPS and other features are unlikely to work seamlessly and road traffic service is currenly unsupported. It's not possible to add it without patching maemo-mapper itself. But for me and probably for other people map is the most important feature.

How to setup a new repository:
URL for the map is "http://127.0.0.1:8000/?map=NUMBER&layer=1&x=%d&y=%d&z=%d"
where NUMBER is a special code for each city(Kiev=1600, Moscow=2000, St.Petersburg=500, and so on)
Download Zoom Steps: 1
View Zoom steps: 1
Downloadable zooms: min: 6, max: 14

So then we may download map for offline use as usual. I use 11,9,7 zoom levels + 6 for city center. It's about 50mb on a flash. It's better to move data file from "/home/user/MyDocs/.documents/Maps" to somewhere at /media/mmc* and create a symlink instead.

Python script can be downloaded following this link. To run simply put it somewhere in home dir and:

$ python yandexmap.py

Roadmap for the script:

1. Configuration file with a set of cities, dimensions, etc...
   
2. deb package in extras repository
3. install file on http://maemo.org/downloads unless I receive too many complaints about how stupid my script is.
4. Easy way of running script - probably it will be launched by maemo-mapper's .desktop shortcut and killed when mmapper exits.
   

Screenshots(clickable):

Maemo mapper + Yandex maps

Guys, I need map!

After googling for Maemo mapper format I found only this very useful link at internettablettalk with the list of supported servers. But unfortunately it's not that simple for us, Ukrainians. Google Maps does not have streets map even of our capital, Kiev. Russian yandex's map service is still the best, but its format is not that simple as google's:

Yandex Maps:
http://maps.yandex.ua/map.xml?mapID=1600&mapX=%d&mapY=%d&scale=%d
http://maps.yandex.ua/map.xml?mapID=1600&mapX=3397917&mapY=6491641&scale=4&slices=1

mapsID - city's id(from the set of cities)
mapX/Y - special coordinates in wild format:
Map is a 2^31 2-D array, coordinates are central point + shift * 2^(zoom level+8)

Google Maps:
http://mt.google.com/mt?x=%d&y=%d&zoom=%d
http://maps.google.com/?ie=UTF8&ll=50.352471,30.256348&spn=4.052146,12.44751&z=7

Very simple: long, lat and zoom :)

That's why is not that simple with Maemo mapper.

May be in a few days I'll test and polish what I got now and create special .deb package for Yandex-only mapper, because there are some changes now which break all other map repositories.

Below is a screenshot(click to see 800x480) of what's working at the moment. To be continued!

Arcanoid

3 days ago I was forced to leave civilized city and go to the place without Internet(which is 95% of Ukraine, I think). My precious N800 is a perfect toy to spent time with. What I wanted then is to have simple, easy, old, ancient, forgotten. Arcanoid game and I did not find it in . Only via ZX Spectrum emulators, etc...

It's a pity. Had to play Bomberman and NumptyPhysics again all day. So I promise to all who read this. I'll find native arcanoid/krakout and port it to maemo. Because it's classics. It's like to have Windows without solitaire :)

Any idea of what to look for first?

Labyrinth - bugfix release

It was too early to release the first version of Labyrinth without proper checking. Even considering "alpha" status of the project. The issue was found almost immediately, though I did not have a time to fix it.

Pygtk in OS2008 and in SDK 4.0 does not support method `to_string` for gtk.gdk.Color, only attributes: "pixel", "red", "green" and "blue". The method is used to store RGB color values like 127/64/0 in hex text "#007f0040000". Can someone advise why it's not working on maemo? The web site says "This method is available in PyGTK 2.12 and above.", however OS2008 contains:

>>> import gtk
>>> gtk.ver
(2, 14, 0)

This was the workaround for the issue:

def color_to_string(color):
 return "#00%02x00%02x00%02x" % (color.red, color.green, color.blue)

Now as the saving is fixed the only porting task left is maemo's Virtual Keyboard support ;)

Labyrinth - mindmap tool for maemo

Hi all!

Short preamble at first. Recently I've become a fan of mind mapping. I visited Exception #07 developers conference in March and there was a guy who did his presentation using not well-known tools like MS PowerPoint or OpenOffice Impress but he used totally different approach, Mind Map. It's the way of brainstorming or displaying your ideas just the way like human brain works - not one by one like sheets of paper but radially with multiple links and forms. It's much better than scrolling page by page during workshop, for example.

For daily use my choice was Java-based Freemind - big, but powerfull and available for Linux, Mac and less popular systems like WinXP, etc . But I wanted something small and modest to use on my Nokia N800. So I've done some search and found nothing. So I(as python/bit C/C++ dev) did some search among apps written in python and found Labyrinth - http://code.google.com/p/labyrinth/

Really don't understand people who use tons of autotool/autoconf code + python script just to install 10 .py files. But I've finally managed to release an alpha-quality port of Labyrinth for maemo - http://labyrinth.garage.maemo.org/ , available also on maemo website - http://maemo.org/downloads/product/OS2008/labyrinth/

Note! It's not ready yet for `production` as I took dev/trunk version from
the original website to have more powerful application with curves and
more export formats supported. If you are interested and want to contribute
either to application itself or to maemo port in particular - you know now where to go.

Here is what it looks like:

Bash: top keyboard shortcuts

There are people around me who've been Linux developers for years and they haven't taken a labor of reading "man bash" so far. It will not of course give you 1000% performance increase immediately but is worth reading it at least once. Just to support popular tradition, here is my list of most useful and frequently used keyboard shortcuts in bash(in order of popularity):

1. Tab - autocomplete current command, directory or file name. I think it's about 50% of the whole statistics :)
   
2. Ctrl-C - kill currently running something.
3. Ctrl-R - search command history backward.
   
4. Ctrl-L - the same as `clear` command. Wipe the screen. Very cool when you need to have blink console for the next info.
5. Ctrl-A/Ctrl-E - go to the beginning/end of the current line. Sometimes Home/End keys are broken. Just in case.
6. Ctrl-U/Ctrl-K - remove text from current position to the beginning or from current position to the end. Of course you can press <- or Delete key 200 times to get the same result..
7. Ctrl-W - Remove the word(or part of it) before current position.

And this is something I've learnt recently:

1. M-f/M-f - just forward/back one word
2. Ctrl-v Tab - insert "Tab" char. Previously I did copy from Vim and paste :(
3. M-u/M-l - change the current word to upper/lower case. Especially useful when Caps was pressed accidentally :)
4. Ctrl-x @ - auto-complete hostname
   

Stardict in maemo extras

It's been about a month since I first downloaded Maemo 4.0(Chinook) SDK and tried to compile something like stardict. But there is much more difficult thing left to do - add .deb packages to maemo repository and create .install file.

Nokia did the right thing I think - usual user only needs to click on green button and choose place where to put an link in menu. The whole work is left invisible behind - to provide easy setup developer has to:

• create SSH RSA private/public keys
• create GPG keys
• upload public key on maemo garage
  
• write changelog info
• sign changelog files
• upload packages onto "extras" repository
• create installation file

Of course, Nokia has half-draft tutorial for this. So after the long process sleepy contributor finally gets this pretty arrow:
Note: still beta!

Back to 80s - console pacman game

Some time ago I had an interview for AI developer and the task was to write a Pacman game. I was not in mood to create a nice GUI or pretty 2D/sprites graphics, the main thing to do was to show ghosts running after the player. Used a standard non-blind path-finding algorithm for this. Here is the screenshot of the game(it's ugly, I know!):

Actually that's not the first game I've ever made. The first one was "Mine Sweeper" written in BASIC when I was 14. And I can assure you python is just awesome! May be anyone can remember GOTO and GOSUB/RETURN operators. Now having all these OOP, exceptions, libraries it's like damn nightmare. I definitely will prefer python to BASIC when teaching my children ;)

The source code(in python, of course) can be found following the link

Hi mom! Meet my bots!

Just imagine - not too distant future, you use robots' help everyday as if it's as common as cleaning teeth. They help you to keep order in your house, they send you reminders, they store all your friends' birthdays(so what?! I't my weak point). What if I tell you that you can use them more from today?

Google released its translating bot farm - special service to have an easy-access dictionary/translator in your gtalk Buddy list:

Russian to English:
Eugene Antimirov: Где находится ближайшая станция метро?
ru2en@bot.talk.google.com: Where is the nearest subway station?
Eugene Antimirov: Сколько это стоит?
ru2en@bot.talk.google.com: How much does it cost?

English to Russian:
Eugene Antimirov: Hi, anybody here speaks English?!
en2ru@bot.talk.google.com: Привет, кто-то здесь говорит на английском языке!?
Eugene Antimirov: What's the problem, buddy?
en2ru@bot.talk.google.com: В чем проблема, приятель?

It's awesome idea, I think. Of course IM/Jabber bots are not something new. IRC servers have had them for years. But new web2.0 and other services may breathe new life into this idea!

Even in my daily routine I use a few. Most frequently used one I wrote to keep an eye on my servers:

Eugene Antimirov: df -h
bot.....@gmail.com: Filesystem Size Used Avail Use% Mounted on
/dev/sda2 5.1G 3.3G 1.6G 68% /
udev 493M 76K 493M 1% /dev
/dev/sda4 135G 115G 13G 90% /home
Eugene Antimirov: sudo /etc/init.d/apache restart
bot.....@gmail.com: Restarting apache... Done

Just 20 lines of python + xmpp code and voila!

One of my friends uses bot who posts latest funny quotes from bash.org.ru to amuse chat visitors on his site.
So I think there will be more and more of them. Who knows...

Stardict port for IT OS2008

Finally did something useful for Open Source. It's 1:00AM now and I've just completed compiling everything I could find for my Nokia N800. Installing maemo chinook SDK was not without confusing things but I hope they will fix everything.

So the first application was of course Stardict. Just few fixes to code from trunk and it's ready:



Thanks to Gonzo for hosting. You can find 2 deb packages following this link. I'll commit my changes during next few days. In my To-Do list I have now Freeciv, my pacman game and mplayer's new GUI.