Tue, 27 May 2014
MPEG Transport Stream
Today I have investigated why some files with the .MTS extension
do not have their MIME type detected. The file starts with the following
bytes:
$ od -tx1 file.mts | head -n 1 0000000 00 00 00 00 47 40 00 10 00 00 b0 11 00 00 c1 00
According to the current /usr/share/magic from Fedora 20,
it is quite similar to the following entry:
0 belong&0xFF5FFF10 0x47400010 >188 byte 0x47 MPEG transport stream data
Also, the shared-mime-info package contains something similar:
<match type="big32" value="0x47400010" mask="0xff4000df" offset="0"/>
Note that both files expect the 0x47 byte to be at the beginning of the
file, not after four NULL bytes as in my example. Yet mplayer(1)
can play these files, and ffprobe(1) can detect it as "mpegts"
with an audio and video stream. Looking into the ffmpeg source,
I have discovered it does horrible things in order to detect a file format.
For example, for mpegts, it scans the file for a 0x47 byte
at offset divisible by four, and then evaluates some other conditions.
The probe function returns score, and a file format with greatest score
is returned from the probe function. Ugly as hell, but probably needed
for handling real-world data files.
So, what should I do next? Should I submit a patch to file(1)
and shared-mime-info to accept also the magic number at offset 4?
Are we getting to the point where the already-complicated language
of the /usr/share/magic file is not powerful enough?
4 replies for this story:
petr_p wrote:
file suffers from lack of tests and the parser rules are horrible. Fedora maintainer is pondering complete rewriting becuse of some fundamental insuffuciences in the languge (e.g. early text and binary format fork). There are always funny bug reportes to file (like today's one [https://bugzilla.redhat.com/show_bug.cgi?id=1101404]). I think you can submit your patch and look forward to various regressions :)
Ondřej Caletka wrote:
As MPEG TS is consisted of almost independent 188-bytes long packet, there is no room for good header. AFAIK the only way to detect TS is to seek for 0x47 packet start mark and then check that this mark repeats every 188 bytes. However, for TS stored in file, there is no reason why 0x47 shouldn't be the first byte. Maybe your file is not a raw MPEG TS stream but some MPEG TS stream augumented with packet timestamps.
jm wrote:
This is likely a Blu-Ray video, where the 188-byte pathets are prepended with 4 byte timecodes. http://en.wikipedia.org/wiki/MPEG_transport_stream#Use_in_digital_video_cameras
jm wrote:
packets, of course :)
Reply to this story:
Tue, 13 May 2014
A Collapsed Dam
Last weekend we went to Jizerské hory, and we have visited an interesting technological sight there: the collapsed dam on the Desná river. Here are my photos together with a short description, as it apparently does not have its own English Wikipedia page, just a dead link from the Desná disambiguation page. More can be read in the Czech Wikipedia (Přehrada Desná).
The dam collapsed less than year after being built, in 1916. The forest workers spotted a tiny leak, and informed the dam keeper, who in turn informed his supervisor, and was given a permission to open the slide valves. In spite of this, the dam collapsed in about 70 minutes. The resulting flood wave killed 62 people. In the picture, there is the dam body from the side, together with the valve tower, and the river Desná flowing to the left through the gap in the dam.
The view down the river shows the huge gap where the part of the dam collapsed. Behind the new wooden bridge the entry to the water outlet tunnel can be seen.
The outlet chamber and the valve tower. There were two slide valves in the dam: one in the tower, and the second one in the outlet chamber. When the dam started to leak, the first valve was fully open, while the second one was half-open.
The second slide valve. After being informed about the leak, the dam keeper tried to fully open also the second valve, but he was forced to flee from the valve chamber in order to save his life. The valve has remained open at about 75 %.
In times of torrential rains, it was possible to divert part of the incoming water to the neighbouring dam Souš by a tunnel more than one kilometer long. Nowadays, the tunnel is closed, as it is a nature reserve of bat wintering.
Here is a modern analysis of the catastrophe and the reasons of the failure of the dam (in Czech, sorry).
An interesting fact about the political and economical system of that time: this dam and several others in Jizerské hory were built by a non-profit organization (Vodní družstvo, in English something like Water co-operative), formed by local works owners and other local persons and organizations, which were endangered by annual floods from the mountain rivers. The local people paid for the whole system of dams. I wonder why we pay enormous taxes and then build anti-flood infrastructure using the money from Brussels. I would guess the percentage of money lost in this process is huge compared to what our forefathers did.
0 replies for this story:
Reply to this story:
Wed, 07 May 2014
GMail Spam Filter
Apparently, GMail spam filter got too zealous. I have my own domain, and I run my own SMTP server on it. Now it seems Google has decided to reject all mail from my server:
<my.test.gmail.account@gmail.com>: host
gmail-smtp-in.l.google.com[2a00:1450:4013:c01::1b] said: 550-5.7.1
[2a01:...my.ipv6.address...] Our system has detected that this
550-5.7.1 message is likely unsolicited mail. To reduce the amount of spam
sent 550-5.7.1 to Gmail, this message has been blocked. Please visit
550-5.7.1 http://support.google.com/mail/bin/answer.py?hl=en&answer=188131
for 550 5.7.1 more information. o49si12858332eef.38 - gsmtp (in reply to
end of DATA command)
In the mentioned page, they recommend putting "SPAM" in the subject of forwarded mail :-/ in order to trick GMail to accept it. But then, it is not forwarded mail at all, it is mail originated on the same host from which the SMTP client is trying to send it to GMail.
So, are we getting to the world where only Google and few other big players are allowed to run their own SMTP servers? And after that, they wil "suddenly" decide to stop talking to each other, as we have seen in the XMPP case with Google Talk. The morale of the story is: don't rely on services you cannot control for your private data and communication. They will drop your incoming mail as supposed spam and you will not be able to do anything about it.
UPDATE 2014/05/21: Workaround Available
Apparently, this is indeed IPv6-related, and the workaround is either to use
IPv4 for Gmail, or better, make Postfix fall back to IPv4 after trying IPv6
first. This way, Google gets a penalty of two connections, and hopefully
will have motivation to fix their problem.
The solution is described here, and more can be read in the postfix-users list archive (another source). The solution is:
Add the following to /etc/postfix/main.cf:
smtp_reply_filter = pcre:/etc/postfix/smtp_reply_filter
Create a file named /etc/postfix/smtp_reply_filter with
the following line:
/^5(\d\d )5(.*information. \S+ - gsmtp.*)/ 4${1}4$2
and reload the Postfix configuration using postfix reload command.
4 replies for this story:
Milan Zamazal wrote:
Do you have both SPF and DKIM set up for your mail domain? If not then you should and it'll probably solve the Gmail problem. (Of course, I wouldn't recommend using Gmail to anyone, but we sometimes need to communicate with their users.)
Yenya wrote: Re: Milan Zamazal
Nope, I don't have SPF nor DKIM set up. Anyway, the outgoing mail in question was sent with envelope sender (MAIL FROM) from another domain than mine (that domain also does not have SPF/DKIM set up). However, I have also tried to send a test mail with MAIL FROM being in my own domain, and Google has rejected it as well. The strange thing is that after several test mails, it has started to pass through the Gmail filter, including the very same messages which have been previously rejected.
misch wrote: IPv6?
I had similar experience ... but ONLY on IPv6 connections. As soon as I switched to IPv4-only for sending mail via google mailservers, every mail was accepted. And when I switched back to IPv6, google started rejecting my mail again. DKIM was enabled (and working) and DNS PTR record was also set-up correctly. And, most important, every other mailserver had no problem accepting mails from me via IPv6 ... except for google. Strange.
Yenya wrote: Re: misch
Interesting. Yes, it is over IPv6.
Reply to this story:
Fri, 02 May 2014
Rocketry
I bought a model rocket for my children (and myself, of course :-). We have assembled it, and on Thursday, we went out to give it a try.
We have found a large field near Hády, and tried the rocket with an A8-3 motor. The weather was nice with almost no wind, and the rocket has returned after a nice flight about ten meters away from the launch pad. I would guess it reached the altitude of 50 to 100 meters. Another try with the same motor type gave a similar result. Then we attempted to use the strongest motor we had, a C6-7.
In the documentation it is stated that with C6-class motors it can reach about 335 meters of altitude. It was indeed something very different than A8-3. The rocket quickly disappeared out of sight. After several seconds, we managed to get a glimpse of it, as it deployed the parachute. However, there was apparently a stronger wind above, and despite having plenty of space in the field around us, the wind has carried it above the forest.
Despite searching for the rocket for half an hour or so, we have lost it. If you visit Hády in the near future, you can look for our rocket around this place.
Anyway, model rocketry is pretty interesting. Our next project is a 3D-printed rocket. We would like to start with something like the disposable rocket (I already have my own OpenSCAD variant of this design, without slanted fins), and move towards a full-fledged rocket with a return parachute, maybe carrying an on-board camera.
2 replies for this story:
Richard Lastovecki wrote: R[o|a]cket
I would never expect that you can be involved in racketeering :-)
Vasek Stodulka wrote:
Nice one. :-) I have to try it too, as a fan of Kerbal space program.