This month's most interesting messages:

All from the SETI@home team members at Berkeley. Non-official statements but interesting anyway.

Eric J. Korpela - 2000.11.09 in s.a.s./a.s.s., what he knows about the bug in the Windows 3.0 version:

It seems to be one of those pesky thread contention things again. The obvious symptom appears to be that the colored FFT graphs stop being displayed. We're not yet sure what happens to the computation thread when this happens. We've also had a few reports of systems hanging which could be a manifestation of the same problem, but we haven't been able to recreate that problem. The solution might be to go back to synchronizing the display of the FFTs with the computation if the screen isn't being blanked.

If you aren't seeing any problems with 3.0 you can continue to run it. If you haven't yet upgraded, we hope to have 3.01 out soon.

Matt Lebofsky - 2000.11.14 in s.a.s./a.s.s., the SETI@home server issue:

Hey, Folks. Despite being in the throes of SETI@home server issues at the moment, here's a brief note, strictly FYI, about an interesting client/server problem we're facing and our proposed solution. This missive exists as a piece of entertainment as well as a warning.

We are currently planning to enforce a mandatory upgrade to version 3.X for all platforms and running into a few problems in the process, the worst being the many thousands of Windows clients earlier than 1.06 which are still active and returning results on a regular basis.

We never enforced an upgrade to 1.06 (or 2.X for that matter), which is why these clients are still able to return acceptable results. Now we have version 3.X, with vastly improved security and increased science, and we would like all users to obtain and use this version.

Most of our Windows users are using clients that, when we decide to make the new version available, will obtain messages from our server about the new version. First there will be warning messages saying there is a new client on the web site. After this "grace period" (an arbitrary amount of time we feel is appropriate), a different message from our server will cause the client to complain to the user that their current version is superceded and will stop attempting to contact the server, rendering it useless.

This functionality would be enough to allow for a smooth and painless transition, but we have discovered it doesn't work for some versions earlier than 1.06. What happens with these clients, more or less, is that they get stuck in a loop trying to contact our server, forever unable to get through and submit their current result.

This wouldn't be too terrible, except there are thousands of these old clients currently running. Once we turn the version warnings on, they will all eventually try to return their result and get stuck in a mode where they are contacting the server every few seconds, thereby clobbering it with requests. In other words, they will be DOS'ing their own server.

We thought up several solutions to this problem. Our main problem is that we have one server at one IP address which all clients contact. If one client is flooding the server, then nothing can get through. So we have to move the point of contact elsewhere.

So all new clients will be contacting the same server via a different host name. The new name evaluates to the same IP as the old name. When we decide to strictly enforce the upgrade, we'll remove the old name from the DNS maps. To the old clients, the server will seem to have disappeared. None of the traffic from these clients will ever reach the server.

While this will greatly clean up the traffic hitting the server, users who have not made the upgrade will no longer be getting polite warning messages - they will be faced with ugly numerical error messages and quit. However impolite, this is still our best solution to the above problem.

Matt Lebofsky - 2000.11.14 in s.a.s./a.s.s., the SETI@home server issue (continued):

Regarding my previous post and the questions that ensued:

1) We cannot e-mail people when they send in results from an old client since we never check whether or not the user's "username" is a valid e-mail address. You'd be surprised by the number of these e-mails that bounce. It gets to the point our mail servers get too bogged down in a hefty mail queue to send anything.

In any event, I did capture about 20,000 users at one point a couple weeks ago who were sending results back, asking them to upgrade. A lot of e-mails bounced, and a lot went through. The results from that long exercise seemed insignificant.

2) As of five minutes ago, I built a version 3.01 which I will release as a semi-public beta (keep a lookout for it). This is for WINDOWS ONLY. All versions 3.01 and above will have the new server name code as noted in my previous posting. That means when we do the sneaky DNS trick version 3.0, 2.X, and 1.X will fail. That's right folks - if you already upgraded to 3.0, you'll have to upgrade again. As always, thanks for your patience and support. We do work very hard around here to avoid painful transitions.

3) And yes, we'll give *plenty* of warning before turning off the old server name. We can't expect all users to download 3.01 on all platforms in less than a couple weeks.

Matt Lebofsky - 2000.11.14 in s.a.s./a.s.s., about an auto-upgrading client:

We avoided this route since the very beginning of the project for security reasons. If somebody hacked the server such that all two million clients would automatically download a virus, that would be bad. Yeah, there are measures to protect yourself from this - maybe SETI@home II will work this way if we find time to incorporate this functionality into the clients/server.

Eric J. Korpela - 2000.11.15 in s.a.s./a.s.s., when suggested to use email warnings for client updates:

We've tried this method. Apparently the vast majority of the people still using 1.X clients either provided an invalid address, no longer read mail addressed to the account they registered with, or are ignoring our messages.

Matt Lebofsky - 2000.11.21 in s.a.s./a.s.s., the last (?) 10 results page:

Well, some of you noticed that we put the "last 10 results page" back up. Sorry to psyche y'all out, but it's back down again.

We resurrected it over the weekend (without announcing it) to see if we could handle the load, now that our science database was running smoothly. However, this clobbered the webserver more than anything. This caused regular webserving (including the pared-down user stats page) to get all funky.

To add insult to injury, we brought the science database down today for maintenance and so result lookups are impossible right now.

Now a pre-emptive answer to the question:

Q: "Why don't you just make another web server machine to handle the extra load."

A: We don't have another machine to do so. This machine needs to be (a) able to handle the load, so it needs a lot of MHz and memory, (b) a Sparc/Solaris system or else we'd need to install Informix binaries for a different platform (which we don't have) and port the CGIs as well. If we had an extra port 80 open on a hefty machine, I'd do so in a manner of seconds.

Matt Lebofsky - 2000.11.21 in s.a.s./a.s.s., a fundraising for a dedicated user-database server to fix the stats problems?

While the offer is incredibly generous, I should have made clear in my posting that we are already working on fixing the problem with resources on hand - namely we're still cleaning up the database and streamlining it, and we also have some dead machines that could be brought back to life as sister web servers.

What I *don't* have to play with is a living, breathing, Sparc/Solaris system already on the network, and already configured for our network, and with enough cpu/MHz to handle the job. So there's no *immediate* solution at hand, but there are some short-range solutions.

Of course, besides being informally in charge of web services, I'm also have to deal with the beta testing of the future 3.01 client. Ugh. And it's a holiday weekend. So don't expect anything too soon.

The dream still lives that the SETI@home web site will be a robust utility for users to check stats and various scientific results (including after the data in reduced).

Matt Lebofsky - 2000.11.22 in s.a.s./a.s.s., FYI: server outage:

Just FYI, the server outage last night was not a SETI@home outage, but a U.C. Berkeley outage. The entire campus was dropped from the internet for about 2 hours last night. And then with an outage that long, there's a huge spike of activity when we return. To add insult to injury, there's always a huge spike in the morning (california time) thanks to everybody waking up, checking their screensavers and sending results back.

Anywho.. My point being is a lot of these SETI@home server outages are not SETI@home outages at all, but random hiccups in the general U.C. Berkeley network. Since we aren't a commercial operation with gobs of cash to throw at an ISP who will guarantee an unbreakable 100-baseT connection, we have to make do with what we have.

As always, there's much hope on the horizon in many forms, including a streamlined database, more hardware, and better client/server services. Stay tuned.

Eric J. Korpela - 2000.11.23 in s.a.s./a.s.s., a fundraising for a dedicated user-database server to fix the stats problems? (part 2)

One thing were looking at in solving this problem is spreading out the science work to multiple machines. One of the ideas in circulation is that we either parallelize the database across multiple machines, or that we make clones of the database to do post processing on. In either case it's likely that the machines we would be using would be fairly low cost intel boxes running solaris or linux. (2 to 4 processors, 0.3 to 1 Tb of disk space, ~$10k per). It's likely that we'll be trying out a test system in the next month or so when our new science programmer comes on staff. Once we're sure what we need, we'll let everyone who wants to donate equipment or $ know what we're looking for.

The bind in the user database performance is harder to fix. The user database is currently running at 100% CPU and 80% I/O capactity. That's the main reason we're having trouble keeping the stats updated and the reason user stats lookups take so long. We have the means to fix the I/O bottleneck, but we'd need to add a couple of CPUs to the machine (a currently dual processor Sun Enterprise 450) to get the CPU performance issues under control. I certainly wouldn't object if someone wanted to donate them, or the funds needed to buy them.

The problems with the web server machine are a little less severe because we could probably set up a parallel server once we get one of our broken machines repaired.

Donating to SETI@home for a specific purpose [ >8 ] is a good idea. I'll run it by the rest of the crew.

Eric J. Korpela - 2000.11.23 in a.s.s., team stats and top 200's not acurate:

I only recently became aware of the problem. At some point, in an attempt to speed up the stats processing we started caching group database entries in memory for large periods of time between database updates. It worked, we are able to keep up with the stats fairly well. Unfortunately this introduced a bug when people joined groups. The database entry would be updated, but this update would be lost the next time a cached entry was written to the database.

I've fixed the problem, additions to the group will get their preexisting workunit totals updated in the database once a week.

Eric J. Korpela - 2000.11.28 in s.a.s., time base of last_result_time and other times in user_info.sah:

They are in exact Julian Date (Number of Days Since noon GMT 1 Jan 4713 B.C. in the Julian calendar) As an example the Unix time reference ( 0:00 GMT 1 Jan 1970 ) has an exact Julian Date of 2440587.5

You can get more info at http://serendipity.magnet.ch/hermetic/cal_stud/jdn.htm

Matt Lebofsky - 2000.11.28 in a.s.s., Question! Amiga client?

I'd figured I'd chime in since I am an Amiga-head as well. I actually got my Amiga 500 dirt cheap, with monitor and 1 MB memory card - I think like $650 for the whole shebang new! How? Well I just happened to be in the right place at the right time - The model I bought was the machine they used to photograph some of the first Amiga 500 magazine ads. Unfortunately, it doesn't work so well anymore.

But at the time it kicked my Apple II+'s butt.

In any event, I don't think an Amiga client is going to happen any time soon. Not with our resources.