Wed, 30 Aug 2006
Svíčky 2006
On Saturday I took part in another puzzle-solving outdoor game, Svíčky. I was a member of team Abpopa.
We had a slow start as always, in the first three stages we were at about 25th place (out of 53 teams), and we were slowly improving our position. During stages 9 or 10 (I don't remember correctly) we were at the third place.
We have however spent many hours on a not-so-hard stage, which was based on Roman numerals. We knew that the distribution of code sequences matched the distribution of letters in Czech texts, but for many hours we were not able to figure out the encoding.
We have finished at stage 15 (out of 17, I think), which we have obtained as 8th team, well after the time limit, after giving up at two other stages and getting the solution from the organizers. Overall it was a nice game, though the stage with colors was probably unsolvable.
0 replies for this story:
Reply to this story:
Fri, 25 Aug 2006
A New Router
Some time ago we have bought hardware for a new router, but until few days ago I did not have time to configure it. Yesterday we have finally moved the traffic to the new server, and should everything be OK, I will reconfigure the old one as a hot-spare backup. With a hot-spare backup I will have a testing platform for things like native IPv6 or multicast.
Apparently while the traffic is almost the same (well, Fridays are always weaker in the terms of network traffic), it seems that the CPU usage is significantly lower (the switchover happened before 23:00 yesterday). I estimate we can now route about 250-300kpps, while the old configuration peaked somewhere near 150kpps (this is with ~1700 IP tables rules, including connection tracking, and about 150 classes of HTB traffic shaping, though those do not carry high amounts of load). The new HW has faster CPUs (2.8GHz vs. 1.8GHz) and bus (800 MHz vs. 1 GHz), and the difference is visible. We can now probably match the liberouter performance with purely commodity hardware.
The next level would probably be buying a 10GbE interface. The new server has a spare PCIe x16 slot on an independent bus, conected by a separate HyperTransport channel to the CPU, so I think a 10GbE NIC would not clash with the rest of the system too much.
While preparing for the redundant configuration I have discovered that the new
HW (which also has two 1000baseT NICs, one 1000baseSX, and one 100baseT)
detects the NICs in a different order than the previous one. So for the
first time I have played with interface renaming. Fedora apparently
can locate interfaces by MAC address, and then rename them according to
its config. Now I have interfaces like uplink, trunk2,
etc.
And now a question for my lazyweb: How would you handle stateful data for
two redundant routers? I mean dhcpd.leases, Arpwatch data,
and few other files. So far I have found csync2 - a purely user-space but not near-realtime solution,
and DRBD, replication on a block device
level (which contains a kernel module, but who knows how stable it is,
especially on a bleeding-edge 64-bit SMP hardware).
0 replies for this story:
Reply to this story:
Thu, 24 Aug 2006
W3C Date and Time
Apparently W3C has defined its own date and time format, incompatible with anything else.
And what is worse, they use the "+HH:MM" format for the timezone
offset, which means W3C time format cannot be formated with
standard library's strftime(3) function.
POSIX (and ISO C99) defines
the "%Z" fromat string for the textual name of the time zone only.
GNU libc (and Perl as well) provide an extension "%z"
which leads to "+HHMM" format, suitable for formating
RFC822-style time zone offsets. But W3C had to invite yet another format,
this time with colon between hours and minutes. Why the standardization
organizations cannot reuse an existing practice, and have always to create
something different and difficult to produce with standard tools?
CPAN has DateTime::Format::W3CDTF,
so at least the Perl side is safe (altough I have not tried it yet, and from
their docs it is not clear how they handle the time zone at all). But
it uses the DateTime object as a time source, instead of
the return value of localtime() or gmtime().
5 replies for this story:
Adelton wrote:
Why not show the time in GMT/Zulu?
Yenya wrote: GMT
Good idea, altough the result isn't both machine and human readable then.
Adelton wrote: GMT
Well, Date:, Expires:, and Last-Modified: seem to be delivered in GMT time zone anyway (tested against aisa's Apache and against Google).
Ken Allan wrote:
Actually, ISO 8601 is the standard for date/time representation, and has been since 1988. The DateTime format that W3C adopted has been around for well over 25 years, and is recognized as the standard way for representing dates and time in computerized systems. Here is the exact text for the ISO 8601:1988 standard: http://hydracen.com/dx/iso8601.htm I'm not trying to be pedantic here, merely pointing out that the W3C (as usual) have done their research and done things the "proper" way, convention be damned! One may of course still argue that perhaps "convention be damned!" may not be the best way to do these things. Anyhow, enough ranting and best of wishes.
Yenya wrote: ISO 8601
OK, point taken. The problem is then conflicting standards only (ISO 8601/W3C versus ANSI/ISO/POSIX strftime()). Thanks for the explanation
Reply to this story:
Wed, 23 Aug 2006
3ware 9550SX Versus Faulty Drive
One of the drives in Odysseus
has failed. I have got a replacement today. While installing the new drive,
I have used the 3ware utility tw_cli to handle the configuration
changes. I was surprised how well designed the 3ware controller software
was.
- It is possible to upgrade the controller firmware from Linux, without needing a DOS boot disk or whatever.
- When the new unit is created, Linux immediately can see it (using the hotplug mechanism).
- S.M.A.R.T. tests can be done using standard Linux utilities (which can see the physical drives). Or the controller can be programmed to run self-tests itself on a periodical basis.
- The controller can identify the physical drive by blinking a LED over an I2C bus (provided that the chassis has the actual LEDs - mine unfortunately doesn't have any).
... and some other things as well. This controller also fully supports physical hotplug and hotremove (but again, my chassis does not have the hotplug SATA power connectors). I think with this controller, the features available to the system administrator are as good as the ones of the high-end arrays (or maybe better, see the point about SMART). Good work, 3ware!
0 replies for this story:
Reply to this story:
Tue, 22 Aug 2006
Haluz
Honza Pomikálek pointed me to an interesting site: haluz.org. It is a web-based puzzle solving game (in Slovak, sorry English-only readers!). The most unusual feature is that it is not a one-evening game, but it is designed to last for two months or so.
The game has been started three weeks ago, so most other teams have
more than two weeks head-start on us. I am in team named (guess what)
coredump,
so far with Pavlína only. The tasks are mostly interesting and inovative.
The most boring and dull one was probably
cracking the 90-digits RSA public key.
You can watch our standings
in their site as well.
0 replies for this story:
Reply to this story:
A Media Day
On Friday, there was a "media day" here. We have created (read: Mirek has created :-) a system which can discover similar documents in our base of documents (students' homeworks, theses, etc.). Almost all computer-related media and some mainstream media has published the press release, or an article based on it. On Friday, even two TV stations and a radio station Radiožurnál did a coverage of this event. I have even played a student typing into a computer in the N@va TV news :-).
In the afternoon, I did an interview (or something like that) for the Czech TV about an Internet telephony (the journalist said "about Skyping", oh horror). I hope I have managed to pass the main message "do not use proprietary protocols which are not compatible with anyone" through. Altough I suppose they can also edit this footage so that I would look as a compelte fool. Let's see how it will look like. I am glad I managed to decline an offer for another interview for the Czech TV (I have gratuitously passed it to another person; I have yet to watch the result from their web archive :-).
1 replies for this story:
Vasek Stodulka wrote:
Send some links, i want to see theese. :-)
Reply to this story:
Fri, 18 Aug 2006
Paper Work
I was fixing an UTF-8 related bug in our printing system - the bug report was from some person from the study department. The bugs was too hard to fix (it lead to these two bug reports, BTW), so after few days, they sent an urgency. After resolving the problem, I have re-read the urgency again:
They needed to print the list of students with their personal data, in order to give it to the account department people, who in turn would send payment requests to the students. Apparently they make a dead-tree version of the data, give it to the people in the next door, who then type the same data back to the computer.
When I wondered what was the reason of this unusual workflow, the answer was that it is needed for the inspection authorities. WTF?
0 replies for this story:
Reply to this story:
Thu, 17 Aug 2006
Perl: The next level
We have ran into a problem that Data::Dumper
module escapes the UTF-8 characters as \x{codepoint}.
This is probably intended, as the result is then usable regardless of
whether the "use utf8" pragma is active or not. But it is not
much readable when the data contains lots of Czech text.
I have solved it by filtering the Data::Dumper output through
the following substitution:
s/\\x{[0-9a-f]+}/"\"$&\""/geexms
This has moved me to the next level of my Perl proficiency, as probably
for the first time I have used the "/ee" in a real-world code.
4 replies for this story:
oozy wrote:
Very nice.
Adelton wrote:
Why the "xms"?
Yenya wrote: xms
Perl Best Practices - Chapter 12: Regular Expressions: "Always use /xms". I think Damian Conway is right with this one. Well, it is not suitable for the Perl Golf, but as a programming habit it is actually good.
Adelton wrote:
Benchmark: timing 10000000 iterations of normal, xms... normal: 6 wallclock secs ( 6.79 usr + 0.03 sys = 6.82 CPU) @ 1466275.66/s (n=10000000) xms: 8 wallclock secs ( 7.52 usr + 0.01 sys = 7.53 CPU) @ 1328021.25/s (n=10000000)
Reply to this story:
Wed, 16 Aug 2006
Swimming in Clothes
Last week Iva was attending swimming lessons. The last lesson was dedicated for swimming in clothes - so that children would not be so confused in case they fall into the water in clothes.
I wondered how would Iva react to the fact that she would be swimming in her pyjamas instead of the swimsuit. She was not surprised at all - she just asked why we did not bring the swimsuit, and that was all.
0 replies for this story:
Reply to this story:
Tue, 15 Aug 2006
Clean Code versus undef
I like to write a clean, readable code. Sometimes even to the point that I am slow to write or fix the code when I think the current code is too ugly, but I have no immediate idea on what would be the best way to fix it cleanly. Recently I've came across an interesting dilemma, which is probably a clash between two measures of a clean code:
- Rule 1:
- The clean code produces no warnings. The
compiler usually knows about dangerous constructs of its language, and can
warn the programmer appropriately. When the programmer really
wants to do something weird, there is often a way to switch the warning off
(such as using two nested pairs of parentheses when an assignment is to be
used as a Boolean value inside the
if (...)statement). - Rule 2:
- The clean code is readable. And by readable, I also mean short. The longer the code is, the more the reader has to keep in a short-term memory in order to grok the code. Perl programmers usually call this principle DWIM - Do What I Mean. So the symbols should be named appropriately, the meaning of the return values should be obvious, etc.
There is an interesting problem in Perl with an undef value.
The Perl interpreter usually considers undef to be something
volatile which should be handled with an extreme care. So it warns of almost
every attempt to use an undef value. Yet undef
has a very clean mental interpretation: it is nothing. So it is natural
to expect it to behave as a neutral element of the operation in question:
The concatenation undef.$string should lead to
$string, length(undef) should be zero,
lc(undef) should remain undef, ++$new_var
should return 1 for a variable with undefined value, etc.
So I am in a situation when I have to put
if (defined $var) { ... }
around various parts of the code, in order to disable warnings
about an unitialized value. But the code is perfectly readable (and thus
clean by Rule 2) even without it. Adding more code makes the readability
by Rule 2 worse.
Moreover, even Perl itself is not consistent with handling undef:
The above ++$new_var construct follows the DWIM principle,
and produces no warning. But something like
print STDERR $log_prefix, $message, "\n"; produces a run-time
warning when $log_prefix is not defined. There are definitely
usages of undef where Perl should complain loudly (such as
calling a method of undef, calling undef as
a function, etc). But I think string concatenation or other string operations
(i.e. everything that can be solved by replacing undef by an
empty string) should not fall to this category.
So what do you think? Should Rule 1 have precedence here? And what about your language of choice? I know, LISP is unreadable by definition :-), but what about other languages?
14 replies for this story:
Vasek Stodulka wrote:
I think if I have to break one of the rules, then I definitely choose rule 2. In your example - if $log_prefix is undefined, then something should be wrong - that was something to be printed, but it is not even defined yet. In this case setting $log_prefix to empty string is perfectly readable and the intepreter is also completely happy. In conclusion, I think that when the code gets shorter, then it is not automatically more readable.
Michal Schwarz wrote: undef means uninitialized
I think that variable containing undef means quite clearly that it uninitialized variable, and should behave strangely and produce warnings :) You can add something to undef, but why? Isn't "my $a=0; .... $a++" cleaner and more readable than "... $a++"?
Yenya wrote:
I don't think "my $a=0; ... $a++" more cleaner than "my $a; ... $a++". But then, the "my $a=0; ... $a++" case is trivial. Consider rather something like 'my $a=some_function(); ... print "returned $a\n"'. There is plenty of ways to get an undef to the variable, not just "my $variable;". Using the undef case as a branch different to (say) an empty string just adds to the code bloat. There are cases when returned undef value means an error, but in general, I think undef should not be treated in a special way.
Milan Zamazal wrote:
I agree with others here -- using an uninitialized variable is wrong. "my $a; ... $a++" is definitely dirty, "my $a;" clearly says "$a is uninitialized" to the kind source code reader, so using its value (as in "$a++") before initialization is very confusing and there should be a warning about it in compile time (if possible) and signalling an error in run time. As for functions, they should have well defined returned values. And in an introspection code you must be very cautios anyway, so the extra checking line is all right. Clean code doesn't mean shortest possible notation but comprehensible information without redundancy.
Yenya wrote: The shortest notation.
By clean code, I definitely do not mean "shortest possible notation". But a not-so-necessary branch which just does something like "if $var is undef, take it as an empty string" adds to the code bloat. Yes, the functions should have well-defined return values - consider the (cleanly designed) function read_file($filename), which returns the contents of the file as scalar, or undef if the file cannot be opened. And imagine you want to use this function on many files (to search for something in the contents or whatever), and you do not care whether the file exists or no. You just want to search in those files you can access. So "for my $file () { print $file, "\n" if read_file($file) =~ /string/ }" is what you want. But instead you get a warning that you are trying to match something against an undef. WTF? It is perfectly OK if undef does not match. For example in SQL, NULL values also have a clean meaning (yet different than "a neutral element"). It is possible (and perfectly OK) to use NULL in expressions.
Milan Zamazal wrote:
Cleanly designed read_file should return something like NULL value (which is a *value*, I don't know what's its equivalent in Perl) instead of undef (which I'd interpret as an undefined result, perhaps due to some unexpected error) and all should be right. If Perl doesn't have a NULL equivalent and uses undef instead, then it's a missing feature in the language.
Yenya wrote: undef _is_ NULL
The undef constant is an equivalent of NULL in SQL or (void *)0 in C - just a special value. undef in Perl does not mean "undefined" (as in "random" or "arbitrary"). Think of it as NULL (or LISP nil). Undef _is_ a value.
Milan Zamazal wrote:
I see. Then I think your assumptions are basically correct as heavy coercing is one of key Perl features. With the exception that I consider "my $a=undef;" being better notation than "my $a;" if you intend to use the initial value and that even Perl should complain when you try to use undeclared variables (I agree it should do that consistently).
Yenya wrote: New variables
The "my $a;" declaration is perfectly OK in Perl, and the variable is set to undef by definition. So "my $a=undef;" would be superfluous and thus violating the Rule 2. Warnings/errors about undeclared variables are of course the good thing, and Perl does it. But "my $a;" is the declaration (and setting to undef). META: From the discussion ti may seem that I favour the Rule 2 over the Rule 1 - it is not true, I simply do not know, and in this case (warnings when using the value undef in some string operations) they seem to clash. So when you all defended the Rule 1, I just wanted to emphasize the arguments in favour of Rule 2.
Milan Zamazal wrote: Assigning undef explicitly
"my $a=undef;" is not superfluous, it tells the code reader that undef is the *intended* value of the variable. For instance, in Lisp the forms (let ((foo nil)) ...) and (let (foo) ...) are equivalent too, but I use the first one to tell the reader that foo's value is nil and the latter one to tell him that the value is to be defined later (despite the compiler sets it to nil anyway). If I always used (let (foo) ...) then the code reader would have to distinguish the two cases by reading other parts of the code. That contradicts your Rule 2 (the reader has to extract and hold an implicit information in his memory). BTW, I think it's more important to keep the number of lines of the code low then to make the lines themselves short.
Yenya wrote: undef is always intended
I think having the newly declared variable (my $var) set to undef is always intended. Perl easily allows the variable to be declared anywhere, so the most feasible is to declare it with the first use (such as "my $var = Some::Class->new()", "my $var = some_function()", or even "open(my $fh, $filename);" and "for my $fh (@list) {...}". So actually seeing "my $var;" as a standalone statement is _the_ case where the author wanted the variable to be explicitly undef. I however agree with you in general about keeping the number of lines low (instead of the length of these lines).
Adelton wrote:
What's wrong with "no warnings 'uninitialized';" in places where you really want to print out undefined values? It's been a long time since I really used undefined values knowingly and it brought some benefits.
Yenya wrote: no warnings 'uninitialized'
I think either "no warnings 'uninitialized'" should be on by default, or $newvar++ should emit a warning as well. Why is ++ different from string concatenation?
Adelton wrote:
Actually, I agree with both your points. ;-)
Reply to this story:
Thu, 10 Aug 2006
Bluetooth File Transfer
Today I have spent some time trying to upload an image to Pavlína's phone - SonyEricsson T68i. I have succeeded, but it was not straightforward:
The positive side is, that on Fedora Core 5, the bluepin with D-Bus actually works, so it is possible to pair the devices inside the desktop environment, without writing the PIN to a config file. I have then tried gnokii to upload and download files, but without success. Gnokii supports T68i, it can identify it over Bluetooth (via a rfcomm link), but getting the list of files is not supported.
So I have tried OBEX file transfers. After some time I was able
to connect via obex_test, and receive an empty file
telecom/pb/0.vcf (from the examples found in the Web).
However, I still did not know how to send files, and under which names
(to which directories).
The next step was to use obex_push. I have misread the manpage,
and I thought the "channel" parameter is compulsory. But I did not know
the correct channel number. After a while, running obex_push
without any argument helped: it prints that it sends/receives files
on the channel #10. obex_push 10 phone:mac:addr /tmp/file.jpg
then did what I expected.
The phone did receive the image, but refused to display it. After some more googling, I have found the list of T68i supported image formats - the JPEG I sent to the phone was bigger than 640x480 pixels...
I have then moved on to explore the desktop features of Bluetooth.
gnome-bluetooth-manager can discover available devices
(one of my neighbors has a laptop and Nokia 6230i in visible mode ;-),
but did not do anything else, even when choosing "Properties" from the
menu. gnome-obex-send allows to choose the device to which
files are to be sent, but then unsuccessfully attempts to install an icon
and exits. Later I have found, that with
nautilus-sendto-bluetooth it is possible to send files with
Nautilus' right-click menu.
On the other hand, I managed to get the receiving side,
gnome-obex-server, working. It is just necessary to
set up the directory
to which files are to be uploaded.
So I think bluetooth file transfer more-or-less works, but the desktop part still needs to be improved.
4 replies for this story:
Vasek Stodulka wrote: Good backend and no frontend
I must agree. Bluetooth in Linux has a good backend (bluez), but absulutely none usable frontend. And bluez documentation is only FRTS (where "S" means source) or Google.
Vasek Stodulka wrote: s/FRTS/RTFS/
I meaned RTFS, FRTS is nonsence. :)
-- Dan wrote: btfs
btfs should be the right thing for you. http://www.mulliner.org/bluetooth/btfs.php But, I don't test it enough yet.
Yenya wrote: Re: btfs
Well, I am not a big fan of FUSE - for me, obex_push is definitely sufficient. I just wanted to explore the posibilities BT provides in modern desktops (read: for BFUs).
Reply to this story:
Wed, 09 Aug 2006
Odysseus with Ext3
OK, Odysseus is fully restored now. I have also decided to re-create the main RAID-5 volume (with the chunk size increased from the default of 64k to 512k), and running the ext3 filesystem.
So far I think ext3 is not slower or more CPU intensive than XFS was, but we will see what happens during the upcoming release of Fedora Core 6.
According to iostat(8), the average request size on the
raw disks is slightly above 64k - it is bigger than the previous
chunk size, but nowhere near the present chunk size (even while running
a purely sequential load like cat FC3-x86_64-DVD.iso > /dev/null).
Dear lazyweb, which kernel parameter should I change?
I just wonder how long the e2fsck will take, because
even mke2fs ran for 2+ hour (while the RAID-5 volume was being
rebuilt, but still it is too long, I think). Maybe I should implement
a poor man's background fsck - once in three months or so
remount the FTP volume read-only, and run fsck on it.
0 replies for this story:
Reply to this story:
Tue, 01 Aug 2006
Slow Writes
"There is one thing which is even slower than writing to a RAID-5 volume - it is writing to a RAID-5 volume, which is being rebuilt." -Yenya