Tue, 27 May 2014
MPEG Transport Stream
Today I have investigated why some files with the .MTS extension
do not have their MIME type detected. The file starts with the following
bytes:
$ od -tx1 file.mts | head -n 1 0000000 00 00 00 00 47 40 00 10 00 00 b0 11 00 00 c1 00
According to the current /usr/share/magic from Fedora 20,
it is quite similar to the following entry:
0 belong&0xFF5FFF10 0x47400010 >188 byte 0x47 MPEG transport stream data
Also, the shared-mime-info package contains something similar:
<match type="big32" value="0x47400010" mask="0xff4000df" offset="0"/>
Note that both files expect the 0x47 byte to be at the beginning of the
file, not after four NULL bytes as in my example. Yet mplayer(1)
can play these files, and ffprobe(1) can detect it as "mpegts"
with an audio and video stream. Looking into the ffmpeg source,
I have discovered it does horrible things in order to detect a file format.
For example, for mpegts, it scans the file for a 0x47 byte
at offset divisible by four, and then evaluates some other conditions.
The probe function returns score, and a file format with greatest score
is returned from the probe function. Ugly as hell, but probably needed
for handling real-world data files.
So, what should I do next? Should I submit a patch to file(1)
and shared-mime-info to accept also the magic number at offset 4?
Are we getting to the point where the already-complicated language
of the /usr/share/magic file is not powerful enough?
Tue, 13 May 2014
A Collapsed Dam
Last weekend we went to Jizerské hory, and we have visited an interesting technological sight there: the collapsed dam on the Desná river. Here are my photos together with a short description, as it apparently does not have its own English Wikipedia page, just a dead link from the Desná disambiguation page. More can be read in the Czech Wikipedia (Přehrada Desná).
The dam collapsed less than year after being built, in 1916. The forest workers spotted a tiny leak, and informed the dam keeper, who in turn informed his supervisor, and was given a permission to open the slide valves. In spite of this, the dam collapsed in about 70 minutes. The resulting flood wave killed 62 people. In the picture, there is the dam body from the side, together with the valve tower, and the river Desná flowing to the left through the gap in the dam.
The view down the river shows the huge gap where the part of the dam collapsed. Behind the new wooden bridge the entry to the water outlet tunnel can be seen.
The outlet chamber and the valve tower. There were two slide valves in the dam: one in the tower, and the second one in the outlet chamber. When the dam started to leak, the first valve was fully open, while the second one was half-open.
The second slide valve. After being informed about the leak, the dam keeper tried to fully open also the second valve, but he was forced to flee from the valve chamber in order to save his life. The valve has remained open at about 75 %.
In times of torrential rains, it was possible to divert part of the incoming water to the neighbouring dam Souš by a tunnel more than one kilometer long. Nowadays, the tunnel is closed, as it is a nature reserve of bat wintering.
Here is a modern analysis of the catastrophe and the reasons of the failure of the dam (in Czech, sorry).
An interesting fact about the political and economical system of that time: this dam and several others in Jizerské hory were built by a non-profit organization (Vodní družstvo, in English something like Water co-operative), formed by local works owners and other local persons and organizations, which were endangered by annual floods from the mountain rivers. The local people paid for the whole system of dams. I wonder why we pay enormous taxes and then build anti-flood infrastructure using the money from Brussels. I would guess the percentage of money lost in this process is huge compared to what our forefathers did.
Wed, 07 May 2014
GMail Spam Filter
Apparently, GMail spam filter got too zealous. I have my own domain, and I run my own SMTP server on it. Now it seems Google has decided to reject all mail from my server:
<my.test.gmail.account@gmail.com>: host
gmail-smtp-in.l.google.com[2a00:1450:4013:c01::1b] said: 550-5.7.1
[2a01:...my.ipv6.address...] Our system has detected that this
550-5.7.1 message is likely unsolicited mail. To reduce the amount of spam
sent 550-5.7.1 to Gmail, this message has been blocked. Please visit
550-5.7.1 http://support.google.com/mail/bin/answer.py?hl=en&answer=188131
for 550 5.7.1 more information. o49si12858332eef.38 - gsmtp (in reply to
end of DATA command)
In the mentioned page, they recommend putting "SPAM" in the subject of forwarded mail :-/ in order to trick GMail to accept it. But then, it is not forwarded mail at all, it is mail originated on the same host from which the SMTP client is trying to send it to GMail.
So, are we getting to the world where only Google and few other big players are allowed to run their own SMTP servers? And after that, they wil "suddenly" decide to stop talking to each other, as we have seen in the XMPP case with Google Talk. The morale of the story is: don't rely on services you cannot control for your private data and communication. They will drop your incoming mail as supposed spam and you will not be able to do anything about it.
Update - Wed, 21 May 2014: Workaround Available
Apparently, this is indeed IPv6-related, and the workaround is either to use IPv4 for Gmail, or better, make Postfix fall back to IPv4 after trying IPv6 first. This way, Google gets a penalty of two connections, and hopefully will have motivation to fix their problem.
The solution is described here, and more can be read in the postfix-users list archive (another source). The solution is:
Add the following to /etc/postfix/main.cf:
smtp_reply_filter = pcre:/etc/postfix/smtp_reply_filter
Create a file named /etc/postfix/smtp_reply_filter with
the following line:
/^5(\d\d )5(.*information. \S+ - gsmtp.*)/ 4${1}4$2
and reload the Postfix configuration using postfix reload command.
Fri, 02 May 2014
Rocketry
I bought a model rocket for my children (and myself, of course :-). We have assembled it, and on Thursday, we went out to give it a try.
We have found a large field near Hády, and tried the rocket with an A8-3 motor. The weather was nice with almost no wind, and the rocket has returned after a nice flight about ten meters away from the launch pad. I would guess it reached the altitude of 50 to 100 meters. Another try with the same motor type gave a similar result. Then we attempted to use the strongest motor we had, a C6-7.
In the documentation it is stated that with C6-class motors it can reach about 335 meters of altitude. It was indeed something very different than A8-3. The rocket quickly disappeared out of sight. After several seconds, we managed to get a glimpse of it, as it deployed the parachute. However, there was apparently a stronger wind above, and despite having plenty of space in the field around us, the wind has carried it above the forest.
Despite searching for the rocket for half an hour or so, we have lost it. If you visit Hády in the near future, you can look for our rocket around this place.
Anyway, model rocketry is pretty interesting. Our next project is a 3D-printed rocket. We would like to start with something like the disposable rocket (I already have my own OpenSCAD variant of this design, without slanted fins), and move towards a full-fledged rocket with a return parachute, maybe carrying an on-board camera.