Admin: Another Database Failure

Should we all pitch in and have Einsteins deliver coffee and bagels?

There is nothing I can do to help because I am not a programmer. But, rest assurred I will remain loyal to PN and will hang in there until the problem is resolved. You guys have my heart felt support. Thanks for working so hard to fix the problem.

Where's all of photo.nets new wealth going?

You guys are having keg parties with it aren't you? And spilling beer on the servers! I knew it!

In all seriousness I hope it isn't a serious problem and that it can be fixed soon without too much effort.

I'm just glad that you're back up and on line again. Thanks for all your efforts and hard work!

Good luck in solving the problem. Photo.net being down meant I actually got some work done today :)

So, I sent a private e-mail weeks ago to someone who posted a status on some disruption a month or two ago that I never saw a reply to.

There are storage systems that have online backup capabilies that don't require full mirroring (and doubling the cost of the storage) intended to reduce recovery time.

Frankly, I'm flabbergasted. Is anyone interested in the management of photo.net in a conversation on reliable storage architectures for web and database applications? Please contact me. I'm serious. My impression at this point is you are doing your restores from tape? That was mildly amusing about 6 years ago.

There are several vendors and solutions out there that can at least reduce the downtime and data loss associated with these disruptions.

beepy

There are frequent snide comments round here about dpreview, but their servers are up whenever you want to go there. Maybe ask them how to keep your servers running. This down time does seem to be a regular thing lately.

I'd like to echo Howard Dion's comments. I can only imagine how exhausting (both physically and mentally) this recent string of failures has been. Thanks for all of your hard work, it is appreciated.

Carl, the problems are due to the fact that we spent our "new wealth" on new hardware that is failing. We were better off when the site was too poor to upgrade its hardware. Things got slower and slower and we had 20 minutes of downtime occasionally while we restarted the Oracle database, but basically everything stayed up and no more than a few minutes of data was lost for years due to hardware problems. The new hardware is failing, in part, because there is too much hardware in the cabinet now. We know that one of the failures was due to the power short-circuiting. And there were thermal issues definitely indicated in one of the other failures. I think most of the failures are actually due to heat. We are trying to get another cabinet as quickly as possible, and yesterday we decided to leave the cabinet doors open to help dissipate heat. This is not very secure, since anybody else in the data center could pull the plug maliciously when I am not there watching. Fortunately, the data center is usually deserted except for the security guards, and I think this is low risk, if not ideal.

Brian P, our approach to backups is as follows:

I. For everything except for the Oracle database, that is, all the static HTML files, scripts, photos, etc, we have everything on mirrored drives. We do a twice-daily rsync of these onto a Permeon disk array, which looks like an NFS-mounted disk. The Permeon software maintains differences. In theory, we can restore the state of our site to any point in the past. That does mean that we could lose a maximum of 12 hours of photo uploads and other changes. We did lose several hours of photos twice during the last six weeks, once because a power failure corrupted the RAID array they were on, and once because of a human error by a Permeon support engineer, who was in fixing a problem with the Permeon array. (The classic "rm -rf *" on the entire photo database directory. Oops.)

We also do a daily rsync of all these files onto an external Firewire drive, which we swap out once per week. We have several of these stored off-site, and they are rotated. In the case of a total disaster such as fire in the data center, where we lost the mirrored drives and the Permeon array, we would lose a maximum of one week of data (assuming the off-site backups were sound). This wouldn't be good, but to reduce that would require more frequent trips to swap out the external drives. These trips chew up about 2.5 hours of time for the round trip.

II. For the Oracle database, which is where we have been having trouble in the last few days, we have been doing logical backups (ie. exports) twice per day to Network Attached Storage boxes. These exports are also put on the external Firewire drives and stored off-site the same as above. The database files themselves are on hardware mirrored drives, until yesterday a Dell PowerVault RAID array. We do a database export twice per day, at 4:30 AM and 5:30 PM Eastern Time. We have not been running the database in ARCHIVELOG mode, which means that in the case of a failure we would lose data to the last good export, because there would be no logs with which to redo transactions forward from the last logical backups. We should have been running in ARCHIVELOG mode, but on the E450 we did not have the space for both archive logs and logical exports, and we had not yet converted to archive logging on the PE6650.

The problems we have been having are due to the fact that the PowerVault Storage array not only had failing disks since last Friday, which could be rebuilt, and indeed have been, but there is a further hardware problem that results in database control files and tablespace files being corrupted (with the corruptions propagated to all members of the RAID arrays). Because we are not running in ARCHIVELOG mode, that means we have to drop back to the exports. That is what we did on Friday, when we lost a few hours of data. We had the same problem again yesterday, but the last two exports, on Sunday afternoon and Monday morning, failed. So we had to backup to Sunday morning's export. We have stopped using the PowerVault for now, and are running off the internal mirrored drives in the server, which are big enough.

To be able to guarantee no data loss in the worst case requires significant resources, both hardware and people, which we don't have. We haven't experienced the worst case, since the data center hasn't burned down, but the last month, and especially the last few days, have been a pretty severe test of backup policies. We have been on the verge of losing the entire site several times during the past six weeks, and were saved by backups. But we can certainly do better.

Brian,

Thank you for the detailed technical update. I hope this alleviates some of the less-than-informed suggestions from technically superior technophytes. As a System Admin, I always appreciate the ever so helpful suggestions (rantings?) of those outside of the situation.

I would like to say that I, and many others, appreciate your efforts to provide a place where collaboration and information can be freely shared, even that which is less than appropriate.

Thank you.

I can't imagine maintaining a site such as this. I have learned so much here and when the site is down, yeah I'm bummed, but it always comes back online soon. Keep up the good work and look for my $25.00 this month. No longer will I continue to take a free ride.

Thanks for all the hardwork keeping this place running. Just out of curiousity, how many people are involved in running photo.net? It'd be nice to be able to thank everyone by name.

Brian, Bob and all others,

You guys do a great job with this site and I know you all work your butts off to keep it up and running. Your work is very much appreciated by a lot of people. Many, many thanks for all you do. I have faith in you all that these glitches will be overcome. And as soon as Paypal decide they want to work in Taiwan (or some other way of paying for non-US users becomes available), I'll be sending you some money.

Many many thanks, Craig

Brian M

Thanks for keeping us in the picture. As a one time techie I know how difficult it can be to sort these kinds of issues out. I too will be coming off the freeload route and subsscribing - it's sobering to realise how big a gap these outages have left in my browsing life!

I would like to echo Brian P though: without wanting to be facetious (and in my currrent role of 'sourcing advisor'!), have you considered handing over the hosting of your servers to a specialist organisation (Rackspace etc.?). That kind of a set up can be significantly more resilient and cost effective, because you then become one of many servers managed by a small team in a dedicated, failsafe environment. I'd be happy to advise / negotiate for you foc if you were interested in exploring that route...

Matt, there is one full-time employee, who does everything from Sys Admin to programming/site design, to selling advertising. That is me. Rajeev Surati, who worked full-time on the site from 1999 to 2001, is also involved part-time, and since he is a technical person (in fact, an MIT PhD), he can and does help out in a pinch with technical problems. He is "the boss", but he more or less lets me run things as the "Editor in Chief/Publisher" of the site. The only other paid person is Bob Atkins, who works part-time as the editor of the "Learn" and "Equipment" sections. There are many volunteer moderators, elves, and other people without whom the site could not operate. A few of these, notably Patrick Hudepohl and Bob, have shell access to the servers and help in a crisis. It was Bob who kept the "Come Back" message updated the other day. For a full list of people involved in the site, check out the "About Us" link. It may not be quite up to date, but almost.

Mike, I haven't investigated what it would cost to outsource the entire operation to RackSpace. I did get a quote recently from them to run a 1U server to which we could transmit incremental backups, off-site, over the net. It was about $300 per month. That doesn't seem like much, but it was for a small single-proce`ssor system with about 40GB of disk, and all the bandwidth was essentially in-bound, which is practically free in their bandwidth pricing scheme.

So I hate to think what a full rack, multiple servers, RAID arrays, firewalls, load balancers, etc, and 15Mbps outbound bandwidth would cost.

but there is a further hardware problem that results in database control files and tablespace files being corrupted (with the corruptions propagated to all members of the RAID arrays).

I'm assuming this is in reference once again to the PowerVault NAS.

Just a bit 'O' passed on experience here, but I've found NAS devices to be lousy in terms of keeping the integrity of heavy traffic transaction based servers. I can't tell you how many times I've had this fight with DBA's, but network attached devices are not the same as a logical internal drive and can't be relied on for integrity unless your database engine is so redundant and buffered it can maintain itself across a WAN. Regardless of the marketing, these things do not have the same protective layer as a dedicated server, and in the case of the later, you need a database engine that is fully aware of the software environment it's working in anyways to be fully reliable across a simple network.

I'm preaching to the choir of course, and you still might find some performance switch on the PowerVault that fixes the problem, but why not dedicate it as a dumb storage device, or at the most handle one-way non critical transactions like photo-uploads? Snap servers, HP storage arrays, etc....never had much luck with them other than simple data dumps or read only libraries.

In support of the "crew" there, I want to say you are doing an especially good job given your constraints. I do database consulting for a living ( for > 20 yrs now) , and running a 12x7 system with (1) person who is split over sys admin/ DBA/backup/hardware/troubleshooting/networking/etc duties is no minor task. These days, each is a "specialty" unto itself, and in the commercial world, one would easily find a single individual (or more) dedicated to each function!

Several posters have pointed out areas that I am sure they feel should/could be addressed, but they must keep in mind the constraints of your situation there. I am glad that you have already taken to "mining" for volunteers among the ranks, which may help fill some of the knowledge gaps if you can locate those with the proper professional skills.

Hardware failues are a fact of life. They also commonly occur 30-90 days after installation of new equipment...either the new equipment will fail, or the old equipment does not like the wire pulling/replugging/jostling. In hindsight perhaps a little more "trial time" might have been allocated before the recent upgrades went "live", but even that depends on having the support staff/ available space/money $$ to run simultaneous systems during the "burn in " phase...which you obviously do not.

So why doesn't everyone give the folk a little breathing room. I suspect sleep has been hard to come by lately for them, and they are doing their best to keep things running smoothly.

Dr. Robert Young, ALI Database Consultants

PS: As a purely technical interest of mine, which OS and version of Oracle are you running?

Hi Scott,

We do have 2 PowerVault NAS boxes. One of them is used for the photo database, and the other one is currently spare, which we use for staging files to the external firewire drive.

But the PowerVault Storage Array that I am talking about is a 14-disk split-blackplane, dual-channel SCSI Storage Enclosure which is directly connected to a "PERC4" SCSI/RAID controller card in the PowerEdge 6650 server. We have it set up in the split-blackplane configuration (each half on a separate channel) in a single 8 disk RAID 0+1 array (2 4-Disk RAID 0 stripes, one on each channel, mirrored with RAID 1). (There are 6 empty slots.)

On Friday, one disk out of the 8 had failed, according to the RAID controller BIOS, but there were no media errors, and the one disk could be rebuilt. There was a temperature failure light on the front panel of the PowerVault box, and numerous SCSI errors in the kernel log. The bad part was that the files had been corrupted, requiring media recovery in Oracle.

Yesterday morning, I actually went in to data center because I thought the firewall was down or there had been another power fail, because I couldn't get into any of the systems. When I got there, the firewall had indeed failed, and there was a kernel panic on the console due to an unexpected interrupt. The firewall rebooted with no problem.

But the bad news was that the warning lights were also lit on one whole side of the PowerVault Storage array, and when I rebooted the server the BIOS reported all the disks on one channel as having failed. Still no media errors, and they were rebuildable. But, again, the files on the "good" disks were all corrupted, and the database could not be mounted. There had been weird errors with the database during the weekend, which I initially thought were problems caused by my not reimporting properly on Friday, but which I now think were most likely due to another corruption, which probably first happened Saturday.

All of these failures, such as in the firewall (which has been pretty reliable), suggest thermal problems. There have been quite a few other failures during the last six weeks, many more than I think site visitors are even aware of, not to mention all the ones that they have been aware of.

Right now, we are running the database off the internal disks in the server, not using the hardware RAID controller or the external storage array.

Thanks for the explanation Brian, that's what I was interested in.

Good luck with the new hardware. I've dealt with this stuff on a smaller scale, building and upgrading personal computers. I can't imagine what it's like running a site of this magnitude.

Robert, thanks for the moral support. On the database box, we are running Red Hat Enterprise Linux Advanced Server 2.1 with Oracle 9IR2. On all the other systems we run RH 9.0, with the one exception of the former database server box, which is a Sun E450 running Solaris 7 and Oracle 8.5.6. This machine is now mainly used as an NFS server, although the Oracle instance is still running with the philip.greenspun.com database. We might need to go back to this, although I am hoping to avoid it.

This would be a great case study for a Sun salesman ;)

Good luck with everything.

Quote from Brian Pawlowski: >There are several vendors and solutions out there that can at least >reduce the downtime and data loss associated with these >disruptions.

Maybe your not in the server admin business because apparently you don't know the difference between vendor hype and theory and actual reality. If you've ever been through a bad server crash you will know those "vendor solutions" often turn into disasters in the worst of moments. Don't criticize unless you know the full story behind a situation.

>>All of these failures, such as in the firewall (which has been pretty reliable), suggest thermal problems.<<

DUH! It's because your RAID is too close to the FIREwall, or you may be running the firewall too hot! At least shield the RAID with some tinfoil to reflect the radiant heat!

I can see a lot of hard work to keep photo.net running... thanks for your effort. I would like to know how many pictures are in photo.net (aprox.)

Juan, there are about 750,000 photos, or 2.2 million if you count each of three views (Small, Medium, and Large) separately.

Should we refrain from uploading for critique until the new cabinet gets their (or whatever your solution might be)? Or Just keep packing it in? ....;)...J

Right now, it looks OK. The PowerVault Storage unit is switched off, and is not contributing any heat. The cabinet doors are open, which is keeping things cool. The Remote Power switch has a thermometer that I can monitor remotely and at least the upper part of the cabinet where it is located seems OK, around 80 degrees F. I might submit critique requests, but I think I would hold off submitting 20 new photos all in one go, at least for a couple more days.

I used to like working through server crashes as it demonstrated to management that SAs/DBAs serve some purpose. Hopefully this downtime will convince all PN members of the sterling work carried out by Brian and his skeleton crew.

Stuart

> I used to like working through server crashes as it demonstrated to management that SAs/DBAs serve some purpose

Where I work (a university non-academic department), we've had what I humbly consider a pretty darn good "uptime" record--in the past six years, we've provided pretty much 24/7 service except for two times (once with a malfunctioning RAID controller in the file server, and the other time with outdated firmware on our switches) when the service outage occured for longer than 12 hours. But to our customers, it's the outages and interruptions that matter, not the 99.9% uptime we provide. Until something goes wrong, what we do is taken for granted, and when something does go wrong, it's almost amusing (if it weren't so personal!) how outraged and concerned the customers get.

> Hopefully this downtime will convince all PN members of the sterling work carried out by Brian and his skeleton crew

Yes... then there'll be those (usually non-subscribers) whose sense of entitlement knows no bounds and who will be mightily ticked off that their irreplaceable, superlative contribution to the photographic world and posterity they made yesterday got deleted.

Brian, congrats with your performance under adversity, very easy to have tons of $$$ to make things work, much harder to work with severe limits and still produce far more effectively than others with all of the toys. I accept the outages as part of the photo.net "thing". In my former eng. shop, we were also limited with funds, but fortunate enough to have an original 1960's computer room with double floors and a MASSIVE air conditioning unit the size of a restaurant ice making machine...so cooling was never a problem..

I am curious -- how many servers are there, what are their specs, and how much bandwidth are they using?

Hats off to you guys...all three of ya...

Question: Looks like some of the custom data fields got reaaranged... No big deal, just tryin' to figure out why it rearranged only a couple of the fields... !@#$ databases....sure glad I design the infratructure (conduit systems)...not enough $$$ in the world to put up with networking/database headaches....

Laura, there are

• 4 web server boxes. 2 of these are relatively new dual processor 2.6Ghz Pentium IV systems with 2GB of memory, and 2 are older 900Mhz Pentium 3 dual processor systems, also with substantial memory). They run 14 separate AOLServer processes.
• an 800Mhz Pentium 3 firewall, with 512MB of memory.
• an Alteon AD3 Load Balancer
• 2 Network Attached Storage boxes, one with 800GB of raw disk space, the other with 1TB of raw disk space. The smaller of these is used for the photo database, and the big one is spare
• 5 boxes for the Permeon backup system. This has over 1TB of storage which is used for generational disk backups of everything else.
• a Sun E450. This was the main database server until last month, and it is still used as the database server for philip.greenspun.com. It is a quad processor 400Mhz system with 4GB of memory and about 400GB of raw disk space, which used to be the storage for the entire site. Now it is just used for the web files, scripts, and Philip Greenspuns photos.
• a Dell PowerEdge 6650. This is the new database server. It is a dual processor 2GHZ system with 16GB of memory.
• a Dell PowerVault SCSI Storage Enclosure with slots for 14 disks, 8 of which are currently used.
• miscelaneous Ethernet hubs and switches.

The bandwidth averages around 15Megabits per second. We burst to 20Mbps, occasionally, judging from watching the indicators on the network switch that connects us to the outside world, although we've never been billed for that much.

To Brian Pawlowski. the synthetic full backups, if that's what you are alluding to, is a recepie for disaster from my experience. For one thing, "flash" backup implementations require a LOT more expertise (read consulting $$$) and hardware resources than a classic ATL based solution. At my company the backups of several TB of data happen in just few hours using your average SAN and ATLs and hardly ever fail.

This is the same technology powering the new breed of digital cameras, is there a nikon f2 version of your database?

Brian,

I've had good luck with Dell PERC3/4 RAID controllers, and even though I'm not a fan of RAID striping in any form, I've had the least amount of problems with complex striped arrays with PERC3/4 than IBM and Compaq. I helped set up a SANs last summer with the same Dell unit, even though for the life of me I can't remember what striping we used. For a second there I thought you were using the NAS units for your main transactions, and that made me cringe, but it's not an issue here.

Classic case of an unknown point of failure because with all those redundant drives. They just happily replicated the database errors to the opposite 4 disk array before giving you the courtesy of a warning light. I just love that with RAID :-) This was such a problem with IBM's nasty 3L and 3H controllers I asked their tech support if the units could play the .WAV "It's the end of the world as we know it" when they faulted so much. Never had a problem with PERC3/4, but I'm not familiar with the controllers inside the drive unit, or how sensitive they are.

As for heat, it might be a problem in a confined case with SANs as big as that one, and those things generate *massive* heat. You probably got 8,000 fans in there, but they are likely just moving their immediate surrounding air around and recirculating it, which means air inside the individual boxes can get very, very warm because it doesn't go anywhere.

Easy enough to fix. The owner at the current company I'm at was pi$$ed when on my second day I pulled all the doors off his fancy server racks and used them for work benches. When he confronted me, I put one of the doors back one, shut off the room environmental control, and waited for 4 minutes until alarms went off inside the main server rack. I then took the doors off and turned on a house fan behind the rack, which shut the alarms off in 30 seconds. He walked away and said "oh" and hasn't bugged me since. From an engineering standpoint server racks with glass doors are better at keeping things warm than cold.

Scott, I got them to pull the doors off the rack on Monday. But this is a colo and I don't think we can run that way for too long. When we get the second cabinet, we're planning to put the new server and the storage array in it by themselves. I hope that this will keep the heat under control.

While the room has the usual huge cooling units and the rows of cabinets are on a raised floor, the interiors of the cabinets actually only have one small fan at the top. Our rack is within 3 or 4U of being full, and the clearance between the fan and the top of the equipment is only a couple of inches. I don't think we can be getting very good air circulation, especially at the bottom where the new Dell equipment is. When I open the back door in the rack, I always get a blast of hot air in my face.

Sad to see the heart of photo.net is no longer running on sun hardware. They're expensive but generally worth it. The e450 might have been getting slow with the new features and increased traffic but it held up quite well not usually missing a beat. Considering the server was probably 7-8 years old, that's not bad.

A new v480 (the current equivalent of the e450) probably would have run similar to what the 6650 cost in a similar configuration. You can even get some incredible deals on refurbished sun hardware probably making a 4way v480 either the same price or cheaper than the 6650. I've even seen factory refurbished 24way e10k's going for under $20k. With the way the 10k scales with additional processors, it would have been the equivalent of 6x the performance of the e450 (it had 24 400Mhz UltraSparc II chips like the e450). Plus there's something cool about having a server that's almost as tall as you :)

Sun's storage arrays also have a very good reputation under demanding circumstances though are quite pricey. You can find really good deals on both sun and emc storage in the refurbished market. Most places usually include some sort of warranty. On servers I see full warranties for up to one year.

From what I remember reading, the e450 was donated by sun. Now with photo.net going the for profit route, I doubt that it could have gotten the same sweet deal though I'm sure something could have been worked out with sun for better terms than retail. Hell, if you contacted sun now and told them about the problems you were having I wouldn't be surprised if you get some help so that they will be able to write a press release stating how they came in to save the day after the dell hardware kept failing. Wouldn't be surprised if you hear some talk about converting photo.net to j2ee. CCM (java version of ACS) seems to have died but if there's every a push to rewrite photo.net on a j2ee base count me in for a few hours a week. I remember a while back trying to do something and just couldn't get aolserver/acs working correctly after over a week of trying.

Don't know what's happening to the e450 but if I could throw in a suggestion it would be nice to see it be part of a dedicated staging environment. Too often recently, new features have been put into production that haven't quite worked right and once in usually don't get fixed for a while. Having an environment that closely simulates the production site, where some users can go in and beta test would be a big plus.

Nice to see things happening. When the site is up the performance difference is quite drammatic. So hopefully these issues get resolved soon as the time and money is making a difference. Excuse the rant above. I just hate it when reliable server components get swapped out with commodity based servers. Even when I've seen fairly significant performance increases going the other way, it usually doesn't translate into the same reliability.

P.S. I've also seen somethings about the dell raid controllers having problems and people resorting to using either other raid controllers or sw based raid instead.

Hopefully you get everything working soon - I'm pretty sure the current situation is far more irritating for you than it is for us. If you've got the HW, then using archivelog would be cool for recovery. Anyway, keep up the good work.

We looked into a Sun replacement, but they don't really compare on price. Refurbished Dell units are quite a bit cheaper than list price, and we did purchase a refurbished unit. You can't really compare the list price Dell with a refurbished Sun on price. But the deciding factor was really the cost of the support contracts. I don't know how Dell makes money on support at the prices they charge. It should be the other way around, because as you very correctly point out the Sun machines are super reliable. The E450 certainly has been, staying up for nearly a year at a time. The Dell has had more crashes in the first month than the E450 has in the last two years.

The E450 wasn't donated by the way. It was purchased by photo.net from Ars Digita for about $20K several years ago. (If aD had it donated, then they pulled a fast one.) I think you are probably thinking of the HP Pavilion that was the predecessor of the E450 in the mid nineties when photo.net was running from an MIT lab. That was contributed by HP, I believe.

We don't plan on getting rid of the E450 for the time being. It is doing service right now as an NFS server and it is the fallback we have if the Dell falls over. Using it for a staging database is a good idea.

Thanks for the offer of help. After your experience with Aolserver last year, I don't generally take people up on offers of help anymore. You aren't the only person who has had trouble. Getting a development environment installed and working does seem to stop almost everybody in their tracks. Actually, Aolserver isn't such a pain. It's Oracle on Linux. If you don't have the exact right version of Oracle with the exact right version of Linux, you have to apply patches even to get Oracle to install. Those patches might be available to people with Oracle support contracts, but people with just a development license and no support contract are often just stuck. By the way, one combination that does work without a lot of trouble is Oracle 9IR2 and Red Hat Enterprise Linux Advanced Server 2.1.

I'd imagine that you already have some, but if you're encountering some power problems, what about some UPS?

Hi Brian - I know you are rarely accepting offers of help nowadays, but I have been working with Oracle on Unix for the last 14 years. RH AS 2.1 running O9iR2? I have a 600G GIS database running on the same combination as we speak. If you do need Oracle help, please let me know. I'll take a look free of charge.

- John Wall

Brian,

You might want to consider looking at updating scsi drivers for any storage boxes. Dell has been known to require driver updates. Dell boxes (some) have alarm lights that might indicate a temperature thing or a log somewhere.

Furthermore, I am not sure of the whole database tie-in to a website, but it may be time to consider how to mirror sites. I guess it is all complicated by the database replication issue, but a metropolitan fiber link may be in order, where the other end is a replica (perhaps read-only) of the database, etc.

Also, what about mirrored sites located throughout the world. There may be volunteers out there to host such a thing, if your software is set up to allow it.

Just food for thought.

Steve

What ever you make from this time consuming job it ain't enough! Kind of like taming lions with a shoe string. You are very much appreciated by thousands for the enormous work load you have assumed. I know with your talent you are basically being grossly under paid. Sort of why your two predecessors took a left turn. (Burn out, baby, burn out). On the other hand it is one hell of a learning opportunity - which you probably don't need. Hopefully it will at least look good on your resume. :^) Fortunatly you can get help from two (at least) incredibly brilliant people. (Phds from MIT are hard to come by!)

What ever you make from this time consuming job it ain't enough! Kind of like taming lions with a shoe string. You are very much appreciated by thousands for the enormous work load you have assumed. I know with your talent you are basically being grossly under paid. Sort of why your two predecessors took a left turn. (Burn out, baby, burn out). On the other hand it is one hell of a learning opportunity - which you probably don't need. Hopefully it will at least look good on your resume. :^) Fortunatly you can get help from two (at least) incredibly brilliant people. (Phds from MIT are hard to come by!)

Not a problem where I'm sitting. Personally, I support the first idea from: E. C. Rice , mar 15, 2004; 09:10 p.m.

"Should we all pitch in and have Einsteins deliver coffee and bagels?"

Let us know what you need from your loyal patrons and we shall do what we can. You all do a smashing job managing this site. You're very appreciated. Don't stress too much over this tiny setback. You'll get it back to the format and function you desired for PN Members and all will be as it was.

I come from Starbuck's Country. Let me know if you could use a small shipment of some good coffee.

Best regards, T.C. Reed

Heat Heat Heat! I'll probably get kicked around for this one ... but have you tried the "high tech" method of Water Cooling - I've heard that it has great heat dispersion qualities.

Brian,

I just reduced my PayPal account by $25.00.

Keep up the good work. I am sure you will work these issues out. In the grand scheme of things.......

BTW - I am a admin for a large company - been there, done that, It's a bad tape, I swear..... :-)

Perhaps you could find some photonetters who might be willing to donate expertise or material support in the form of better hardware and software interface to ease the burden of time and expenses? You could always provide them with an incentive or reward, like a lifetime subscription or something? :-)

I can't think of anyone offhand who might know how to help fix this...so sorry. :-(

Actually, considering the amount of administrators that seem to have a very educated background from prestigious colleges, perhaps you could offer some interested students in IT/EE backgrounds some service/internship credit to help maintain the website?

Just a couple ideas to help...

Hi! I am a new new newbie, as I am both an amateur photographer and new to this site. But in my short and all to few visits I have come to enjoy this site very much! I am sad to say, that I can´t contribute with any knowledge as I operate on a far below niveau regardig servers, but I thought you should now that I and others are aware of your heroic battle with this server and the pressure to keep it up. I can give you no more than moral support as of now, since my economic situation is worse than bad at the moment, but rest assured, that you have raised yourself on my prioritylist with this thread.

You are doing a fantastic work, and I appreciate it very much! For what it is worth, I think you all are being very altruistic in this matter and you are setting the standard for many a website on the net...

You "may" see more hardware problems with the Dell & RH when mixing in 3rd party equipment than you did with Sun. There is something to be said for largely "proprietary" systems at times..Sun, HP, & DEC/Compaq/HP Alphaservers come to mind...hardware is designed by the people who designed the OS who designed the drivers.....since most of the hardware/software offerings are (mostly) from the primary vendor, upgrades etc tend to be "friendlier" to each other. Of course those addons cost 3X as much.....

BTW: Be aware that when you go to ARCHIVELOG mode, the performance of the database will take a hit. I know several places that have tried it out, only to turn it off at a later date. It depends on the amount of traffic you are getting.

I'll take a performance hit anytime over the loss of a whole days data.

As I'm earning a living with business critical systems, I know one thing for sure. I won't offer advice as the poor guys running this site probably know their trade and have to make do with the hardware they have.

I signed up for a subscription instead, hope this helps and at least it shows my apreciation for the good work!

Regards

Volker

Just thought they should be made aware of it. On paper it sounds great. Often times it is great. However, some have had to drop it due to performance issues. There are several ways to "cover oneself" with or without it being enabled. I would certainly give it a try, but one should realize that turning it off is an option it users try to storm your "castle" with pitch forks and flaming torches :-)

"Of course those addons cost 3X as much....."

3x as much is way off. I'm not sure where you're getting your numbers from because it doesn't seem to be from current data unless your comparing a desktop system that someone decided to put in a rackmount chasis. The prices are no where near that different and the commodity hardware isn't always cheaper. Sometimes significantly cheaper. If you need a 64bit platform it seems dell only offers itaniums in their 3250 that I could find. A 2way 1.0Ghz IA-64 system with 36Gig SCSI drive and 2Gig's of mem runs about 7.5k with no OS. A Sun V240 with 2 1.28GHz UltraSparc IIIi Cu processors, 2Gig ram, 2 36Gig SCSI drives and solaris (and everything they throw in with it) goes for 6,695. You can get the v210 for less than 5k. Or even a similar 2cpu Opteron processor for less than 4k. You can even get single proc ultrasparc servers for under 1k.

When comparing the 6650 that photo.net acquired, new it would go for about 20k. The v440's and 480's are in the same league. The difference between the v440 and v480 is that the chips are different versions (1MB vs 8MB on board cache) and SCSI vs Fiber Channel hard drives respectively. 4way systems in these series range in price from 16k to about 42k for a 4way 1.2Ghz system with 16gigz of ram, 2 72Gig 10K FC-AL drives, redundant power and solaris.

It's like saying import cars are 3 times more expensive than domestic because you're comparing a Ford Escort with a Mercedez.

If you're looking at higher end systems Sun and IBM system are sometimes a bit more expensive but I haven't seen anything 3x the cost on comparable systems. When sites are down, they lose revenue. Not just on the days they were down but because of the perception that they may not be stable. Would you want to pay 10k on an advertising campaign on a site that you're not sure will be online the most important day? For photo.net maybe this doesn't make as big an impact as sites that generate greater revenues. Also keep in mind. The problem isn't "fixed" it's been patched and the system will have to come down to actually reattach the storage device and sync it up with the current data, most likely losing some in the process.

Both have similarly priced support contracts with Dell having an edge in that their base service provides nedt day onsite support and I'm not sure if Sun's inclueded warranty does. If you want same day support from either you pay a decent amount for it.

I don't really know much about dell refurbished systems and pricing but I'm sure they got a good deal. How good it seems depends on what happens in the future. The problem is being attributed to heat, and it may well be, but 80 degF is well within the 95 degF ceiling for normal operating conditions. A second rack and spreading out the servers may help but now there's the monthly expense of the other rack before photo.net had a chance to capitalize on it's new equipment purchases. Or maybe just the one time expense of a more effective cooling system plus an additional circuit to the rack since it seems to be having power problems as well.

Maybe it all works out to still make sense financially in the end, I have no idea. My point is just that 3x the price is a popular myth that keeps people from going with manufacturers that have been developing servers for the datacenter for a long time.

Also as far as ARCHIVELOG. It shouldn't be that much of a performance hit. If it was it might be due to a bad configuration. You want to make sure you have your redo log file size set to a large enough number. Also having your online and archived redo logs on seperated physical drives (and data on the attached storage device) will give you the least performance hit. If you have the archive redo logs and the online redo logs on the same drive you are going to see more of a performance hit.

@Tom.
Thats my experience. I've got a v480 as the main OPI Server in a print shop. We tested homegrown Intel Xeon PC's with RedHat as an alternative, but under load the Sun performs better and handels both Oracle and Helios without a glitch. We lost one HD in an external RAID due to no cooling last summer, Sun changed the faulty drive within four hours while the system was running print jobs on two Xerox digital offset presses simultaniously.
At a small advertising agency I have two Dell 1750 with Debian stable running loadbalanced Apache Webserver and replicated MySQL Databases. Webpages are on a Dell Powervault and mounted via NFS.
I'd prefer the IBM e- and p- servers with EMC² Storage I have running Informix and Oracle Databases for Warehouse management and DMS, but multinational Car producers can't be compared to a relative small comunity website like this one :-)

So signing up was the best I could do to show my apreciation and ongoing support in bad times.

Regards
Volker
P.S. for a change, IMHO one of the best books about Database Backup Unix Backup and Recovery

What does the Permeon system provide for you? I took a look at their site and I did see a mention of providing snapshot backups of systems. Are the Permeon systems raid as well, or are they just arrays of big single non raid hard drives?

Thanks for the informative descriptions of your problems, interesting reading.

The Permeon system is essentially a backup "appliance". It is mounted via NFS and files can be copied to the Permeon filesystem using the usual Unix utilities. We use rsync, but obviously cp, tar, and so forth, would also work. On a regular schedule, snapshots of the filesystem are taken, and the snapshots are maintained indefinitely. File updates are stored as differences, so the snapshots do not take up as much space as multiple versions of the same file would. The binary differencing algorithm, and the storage format for the differences, are among the core Permeon technologies. You can mount the filesystem either in its current state or at any snapshot point in the past. Our snapshots are taken daily.

The system is a black box; in our case, it is five 1U Intel-based systems, which I happen to know run Linux, but you don't need to, and indeed cannot, administer them yourself. Permeon takes care of that. They don't use RAID, but they use a RAID-like technology for striping and mirroring information across the multiple systems. One time, their CTO referred to their technology as "RAIS" (Redundant Array of Independent Systems), but I don't know if that is really a term that they commonly use.

photo.net has been a beta test site for Permeon (previously Permabit) for the last couple of years, and we received the hardware and software from them in return for testing. We had been using it for online backups of the photo database, but now we use it for online backups of all our files. (We also do external backups, of course.) Permeon released "V1.0" of their system in December, and we recently deleted our previous snapshots and started over from scratch.

Brian, I used to work as a Systems Engineer for Dell and then finally at EMC. I do know the escalation paths within both companies if you need the assistance. The Clariion array has the ability to have a call back feature...meaning the array will dial EMC if there's a failure. Not sure that Dell uses that feature but it's a big help if you can get Dell's server support folks to escalate this to Engineering quickly and get both companies working on it simultaneously. Also not sure if you're using the IP4700 (EMC version of their NAS device) but it also has software capabilities to do a point in time snapshot backup and recovery which may help in faster recovery times versus tape backup and restore. Let me know if I can help point you in a direction. My email is mike@picturetex.com . Send me an email and I can give you some contacts there that may be able to help. Mike

Thank you all for the hard work.....P net is really a great resource for many of us...

Good luck!