Tue, 21 Apr 2009
Statistics Problem
Is there any statistician reading this blog? Can you recommend any statistics community (web forum, mailing list, anything) where I can ask questions about one problem I am currently trying to solve? For those with login to IS MU, the description will for some time be also in the discussion forum of Faculty of Science. The problem is this:
I have a random variable with probability of exp(-a*t) for some constant a and the time t (think radioactive decay, but the real problem is something different). The problem is to calculate the constant a from the observed data.
The measurements I have are in the form of a set of pairs (ti, + or -), with the following meaning: At time 0, take a brand new "i-th atom", verify that it is not decayed, wait for the time ti, and look at it again. If the atom has not decayed yet, add a (ti, +) pair to the set of measurements. Otherwise, add (ti, -). Continue with the next new atom an the next time ti+1.
Note however, that the times t_i are given to me from the outside, I cannot choose them, and they do not have any particular distribution (e.g. being equally distributed between time of zero and some large number). Also, the number of measurements is quite small (several hundreds at most).
You can download a Perl script for generating the test data, the test data (100 rows), and the large test data (10,000 rows) generated by this script. Can you somehow compute which constants a have been used when generating these sets of data? If so, how could it be done? And how can I estimate how accurately the exp(-a*t) curve fits the real data?
4 replies for this story:
J.C. wrote:
I haven't studied your problem to such an extent that I fully understand it, so just an idea: how about applying the logarithm function to the data (y-values)? This could make things a bit easier, as it should transform the data with some-obscure-exponential-dependency to linear-like dependency.
Yenya wrote: Re: logarithms
Yes, log(y) transforms the exponential regression to a linear one. However, log(what?). I have a + or - (i.e. 1 or 0) values. So which points to apply the linear regresson to? 0 and -infinity does not sound feasible to me (remember, the data is relatively sparse and even with discretisation of the intervals you get the intervals with only - values.
Luinar wrote: Solving
Sorry that I won't bother with English but in Czech it will be much faster to explain the solution. Pro každý čas t_i si můžeš spočíst pravděpodobnost jestli dostaneš + nebo -: P(t_i,+) = exp ( -alpha t_i ) P(t_i,-) = 1 - exp ( -alpha t_i) Tj. pro každou částici ve výsledku jsi nyní schopen spočíst její pravděpodobnost jako funkci parametru alfa. No a vzhledem k tomu, že dané částice jsou nezávislé pak, pravděpodobnost toho, že dostaneš tato konkrétní data je součin pravděpodobností od jednotlivých částic. Příklad: Výsledky: 1 + 2 - 3 - 4 + 5 - Pravděpodobnost daného výsledku je: exp(-alpha) [1-exp(-2*alpha)] [1-exp(-3*alpha)] exp(-4*alpha) [1-exp(-5*alpha)] No a parametr alpha hledáš takový, aby maximalizoval tuto pravděpodobnost. Po drobných úpravách a hraní si s tím (kvůli kompaktnosti = není to jediný možný zápis) dostaneš analytický vzorec (LaTeX konvence): \sum\limits_{i=1}^N t_i \frac{ \mathrm e^{-\alpha t_i} - h_i }{ 1 - \mathrm e^{-\alpha t_i}} = 0 kde N je počet atomů, t_i jsou časy jednolivých pozorování a h_i jsou výsledky ve tvaru 0 pro -, 1 pro +. Tahle rovnice se už musí řešit numericky, jako hint je dobré si všimnout, že suma jako celek je funkcí klesající v alpha a tedy půjde to dobře řešit půlením intervalu nebo nějakou pokročilejší gradientní metodou. Pokud budou další dotazy směřujte je kdyžtak na můj mail ("Moje přezdívka" na Seznamu - k okénku na vyplnění nemám důvěru neb nevím jestli se nezobrazí veřejně a v idealním tvaru pro spamboty).
Yenya wrote: Re: Solving
Your solution is right. I am sorry, I forgot to update this blog entry, but I already had this solution - a friend of my colleague solved it about a day or two after the above blog post has been published. But thanks anyway - I hope it has been a nice mental exercise for you :-) Now the other problem is the last sentence from the blog post: when the data is a bit noisy, I would like to estimate how well the probability fits to the exponential curve. Of course, some estimate can be taken from the probability from your blog post, but the value of this highly depends on both number of measurements, and t_i values. I would like to have something which could be used for comparing even data sets with different number of measurements and/or different distributions of t_i values.
Reply to this story:
Pragocentrism
I live in a country with population of about 10 milion, with the capital Prague with about 1 milion inhabitants. Today's rant will be about narrow-minded journalists living and working in Prague.
I frequently ran into a blatant cases of pragocentrism. For example in almost every traffic news in a country-wide and state-funded radio station Radiožurnál they use formulations like this: "there is an accident in the Brno motorway in a direction to Brno". WTF? Which of the three motorways heading to Brno do they mean? The D1 from Ostrava? The D2 from Bratislava? No, of course they report from the perspective of people living in Prague, so naturally with "the Brno motorway" they mean "the motorway from Prague to Brno".
Another one was a few days ago, also on Radiožurnál. They were doing an interview with a candidate for the minister of the interior (who currently works as the head of the anti-monopoly office, the institution located in a barren countryside far away from Prague, namely in Brno :-). The first question was "Have you already get used to living in Brno instead of Prague?". Mr. Pecina replied something like: "I don't understand the question - I am from Frýdek-Místek, I have been living there for almost all of my life, except only one short stay in Prague.". The journalist had naturally expected that every important person must have been from Prague. That said, the journalist was really stupid anyway and she manifested it several other times during that interview.
Another case of Pragocentrism is more general. In the main news of the Czech TV (also state-funded), they often report about Prague-local things (such as some affairs of mayor of some part of Prague or even of a mayor of Prague, building some tunnel or some stadium in Prague) during the main part of the news, even though they have a separate part "news from the regions". Also when doing a coverage of a country-wide event such as elections, they report about the situation in Prague, and then they say something like "and now we will look into the regions". WTF? Prague is not a region? Why should the Prague-local news be forced to us by state-funded media as something important?
I know I probably sound like some women-rights or some minority-rights activist with a well-developed inferiority complex, but hey, about 90 % of citizens of this country do not live in Prague! Journalists, keep that in mind, please. My dear lazyweb, is a ${your capital}-centrism also present in your country? Is there even a Brnocentrism from people living in Brno towards people living near Brno?
10 replies for this story:
wrote:
Don't worry, it doesn't sound like "some women-rights or some minority-rights activist". Women and minority rights are important (as well as the activitsts even nowadays), your story is stupid. Sorry for such strong word, it's not about you, it's about your perception, it is stupid. I'm living in same country in Brno (anyway I'm not natural born Brno guy), I know very well such opinions, people publishing such opinions are the only people, who are creating Pragocentrism. ;) Prague is capital city and most important region in republic (anybody can say, it isn't, nevertheless, nobody will change it), D1 is most important highway in republic. News (in prime time, on premium pages) are always reporting absolutely 'useless' messages without any relation to me. I see 'useless' information about mayors and other council members of some small unimportant cities, I see information about police car hunt in centre of Brno, I see information about public traffic of several small villages hundreds km away from my place (I will never visit that region, why should I read about their current problems?) ;) however I never called my country Horni-Dolni centric. :) Nevertheless there's also something, what makes me sad, but Brno citizens, actually their accent. It's unbelievable for me, people of Moravian city have such terrible czech accent, even some Prague guys aren't not able of such sort of 'singing'. Don't you ming this? Doesn't it prove of silent dark expansion of Prague? ;-)
Yenya wrote: Dear anonymous,
you should probably reread my blog post. I am not complaining that D1 Prague-Brno motorway is more or less important than some other highway or that it is being reported in the news too often. I am complaining about calling it "the Brno highway", when for about half of the Czech republic citizens it is "the Prague highway". I know there are people with different accents speaking Czech, and while I of course find some of them being quite funny, I fully accept that my own speech can also sound funny to a person from the different part of the Czech republic. I would never rant about it in my blog. What I am complaining about is that many journalists silently expect that the listener is from Prague and report about things from the Prague perspective, even though people living in Prague are in clear minority here and even though there is a perfectly neutral way of reporting things (using the term "Prague-Brno motorway", including the Prague-local events under the "reports from regions" section instead of main news unless they are _really_ important, not expecting that any random polititian is from Prague by default, etc.). I hope I have clarified this a bit.
Hynek (Pichi) Vychodil wrote:
I fully understand you. I often hear or read in country-wide radio or news paper about some cultural event with address info containing only street but without city info included. It is Prague of course. How stupid. There are many streets with same name in tens of different cities around the country.
wrote:
Your post is absolutely understandable, no other explanation is required. Ithought my reply is understandtable as well, now I see it isn't. :( Just wantedto say people are not machines, everybody here (in country) understands the highway, Brno highway, Prague highway, Brno-Prague highway, D1 is still the same road between Brno and Prague (any other case must be explicitly stated and it correctly happens). You also claimed example of stupid guy employed by radio, it proves nothing but the stupidity of that guy. The last part (objectivity/neutrality/balance of news) I opposed by example of local related information I see in news. it is not from Prague, not from Brno, not from my native city, but it is published as well as Prague related information. That's my message, it's not about the form of publishing news, it's about your (my) perception. Please be perceiving, accept my opinion and try to think about it. ;) P.S. Neither me complain any accent some particular man. However I will always complain accent of Brno citizen. They are not using moravian accent (which is logical for moravian city), theyusing _czech_ accent, very sad for me... :(
Yenya wrote: Re: anonymous
I don't agree that the term "Brno highway" is understandable. It quite frequently makes me WTF? when I hear this. And a sole "D1" means anything between Prague and Vyškov and those new parts between Studénka and Polish borders as well. I of course do not try to say that _all_ journalists are pragocentrists. But some definitely are, and this is what the blog post is about. BTW, what is "moravian" accent? In Moravia, there are several quite different-sounding dialects: the one from around Uherské Hradiště (which BTW has been taken as a basis of the literary Czech language), the one from Haná (which is probably the closest one to the Brno dialect, altough the later has been influenced with some other dialects and languages such as German), the one from Valašsko, or even the dialect from Ostrava (remember, Ostrava is also partly in Moravia)?
avakar wrote: Re: accents
The tendency to use a lot of intonation is actually not a curiosity limited to people from Prague, but is in fact shared by a lot of popular foreign languages as well. This includes English, French, and I bet even Japanese. These languages can influence us today more than ever before. It is therefore more than understandable that one's accent does not reflect one's place of birth. So please, anonymous, don't be mad at people for the way they speak. Use your energy for something constructive, like shooting people who leave garbage on the ground.
Milan Zamazal wrote:
On a more positive note, I started to like pragocentrism last years. It's main advantage is that many idiots get exported to Prague. The fraction of people who resist values represented by pragocentrism is thus significantly higher outside Prague. And that makes life outside Prague a bit more interesting and better.
thingie wrote:
It makes a lot of sense to call D1 "Brno highway" as the whole highway system is completely pragocentric. In fact it is maybe even better than simply "D1", because it makes a distinction between the part from Praha to Brno and the second part from Brno to... um, wilderness, which are quite different, in many senses. And it doesn't really matter anyway. Why are you so upset over some silly traffic news? Their thinking and diction has to be affected by what they are doing, and in this case, it is about the highways, which simply are pragocentric. But it seems that you no longer care about that, because you assume that cars and road traffic is so important for everyone, that traffic news represents thinking of the whole nation. And while I don't see much badness caused by pragocentrism (just move there or something, blah, who cares), carcentrism is evil.
andrej wrote: same as in your east neighbour country
${your capital}-centrism is everywhere. I live in Bratislava and there is also strong Bratislavacentrism too. And not only in TV but in contrywide radio stations too.
Jan wrote:
It's a custom to call the highway after the major town on it. I never heard something like "the Prague highway", but usual terms are "brněnská", "plzeňská", "mladoboleslavská" or "královéhradecká" and everyone knows which one it is.
Reply to this story:
Fri, 03 Apr 2009
HTML <button> Tag
So I wanted to upgrade the form we use in IS MU in many places for selecting a printer, splitting the "print" and "download PDF" functionality to separate buttons. The problem is how to make it as backward-compatible as possible.
I basically wanted to have two buttons with the same name="..."
attribute, and distinguish between them by their value="..."
attribute. I have came across the cool new (for me anyway :-) HTML tag
<button>, which does exactly what I want. I am able
to use my own machine-readable value="...", and put the
button label (localized) inside the <button>...</button>.
Except that it does
not work in MSIE. That parody of a browser does not send back the
value="..." attribute contents, but rather inner text
of the <button> tag for all buttons in the
form, not just for the actually clicked one. Stupid MSIE, die already.
2 replies for this story:
Peter Kruty wrote: funnny
Stupid MSIE, die already. .... this made me laugh :).
mirka wrote: finally
Stupid MSIE, die, finally Stupid MSIE, you're dead already, go away ?