Storage on Nanikore

Home Server and Storage

Sun, 19 May 2013 08:52:00 +0000

I need to address my home server storage capacity problem. For a while I’ve been using a single disk 2TB Buffalo.)

That NAS has filled up with family video and photo backups, my over-enthusiastic GoPro footage, sound recordings, rips of my DVD and music collection (as FLAC), and so space has run out.

I decided I wanted to spec something that would last for a few more years and decided that:

It had to be reliable and quiet, even if it was going in a cupboard.
If the system failed, I needed a decent chance of getting my data back.
I needed 4TB of space, with a local 1:1 copy of that - that’s 8TB of space total.
I needed to be able to Crashplan/other internet backup it offsite.
It was going to be on a budget!

There are lots of ways to approach this issue. For example - a simple way is to buy a big drive or drive array and attach via USB to a current machine. I like the idea, but it means I need that machine to be on, or in sleep mode, all the time, which is something I wanted to avoid.

Another solution is a home NAS like my existing one. These go from single drive units up to 5 drive and above RAID systems. They are relatively cost effective, but when they die, you’re often stuck trying to find the same model to use to get data back, and since many are embedded Linux, if it dies, your only real chance is to hook up what’s left to another unit and hope.

Another option: an old style home server. Essentially it’s a PC, with a lot of drives and that’s it. After spending time reading reviews, especially at Silent PC Review for quieter parts and considerations, and kakaku.com for pricing (I live near Tokyo so I’m spoiled for PC parts at retail) I came up with the following:

Case: I looked through SPCR’s recommended cases, and saw the Fractal Design Refine R3 - the R4 had recently been released. I looked at one, and some of the competition, and decided to give it a try. One minor note, I got the ‘Arctic’ white one so it wouldn’t stand out as much where I was going to place it, and since this thing is heavy, I bought it from Amazon Japan for pretty much what I would have paid in Akiba, but without having to carry it home on the train. Mounting components is simple, and it comes with plenty of brackets and rubber washers to help isolate vibration from hard drives.

Motherboard and CPU: In my price range AMD have some good chips, but so do Intel. I was looking at the AMD A6-5400K APU, and the Pentium G2020 - yes, a Pentium, but based on the new Ivy Bridge core - this part is lobotomised though, and only has Intel HD graphics, but for what I wanted, it would do. Then I looked at the mobos - either Intel’s 1155 socket based, or AMD’s FM2. In nearly all cases, the AMD chipsets had 5-6 SATA3 connectors, whilst the Intel ones had only 2. That was the deal breaker for me - I wanted to make sure the machine would have as much SATA 6Gb capacity as possible for the future, so I went with the AMD combo. I chose an ATX board from [ASRock](http://www.asrock.com/mb/AMD/FM2A75 Pro4-M/), which have always treated me well, and between the two I’d have everything built in.

Drives: For the OS I picked up a Samsung 840 SSD in 120GB - I wanted a fast OS disc to get the machine up from whatever sleep / reboots it had to do. Also, it would mean a cooler, quieter machine. For storage I went for 4 * 2TB discs - Western Digital Green - these are the only thing that I wonder about, as there are some stories about the Green drives not withstanding this kind of role. That said, I’ve been using them aggressively for years and not had an issue. We’ll see.

Memory: Corsair XMS3 in 2 * 4GB - nothing special, but a decent brand.

Power Supply: I like the Antec units, and thanks to a sale, I got an Antec EarthWatts platinum 450W power supply for a good price.

So that’s the hardware

For Operating System, I was actually planning to go with Linux, but since I have an MS Technet account, I decided to give Microsoft Server 2012 a run and it’d give me an excuse to spend some time with it. One thing I liked about a Windows solution was that I could use Storage Spaces, which allows Windows to group physical discs as virtual discs, meaning I could buy cheaper smaller disks and let the system see them as a single larger disc. This doesn’t buy safety against drive failure, so the other discs made an identical space, and used Windows Backup Service to do a nightly copy. The benefit here is that if the OS dies, or the machine is unusable, as all meta data is in the Space, you can hook the same two discs (in my case) up to a Windows 8 box, or a rebuilt Server 2012 box, and still use the discs. I decided to to try this, and built the system up on Server 2012 Essentials (the old home server plus small business server) but then rebuilt it on Server 2012 base, and it saw the Storage Spaces no problem, so I’m relatively happy that’ll work in a disaster scenario.

Finally, as it’s Windows, I can run Crashplan on it as normal and have that extra offsite backup mode.

[Update: August 2013 - Just a couple of months later, MS killed Technet, so I’ve happily rebuilt this with Mint Linux 15 (into text runlevel) using LVM over StorageSpaces. Crashplan still runs superbly.]

For scale, 2 * 4TB Buffalo NAS cost around 66,000yen (~420GBP/~640USD). This machine came in a bit more expensive, around 74,000yen (with 32,000en of that being the 4 WD drives!).

In practice, after a month of usage, it’s been a great success - it’s fast and reliable, sleeps well and even when on generates very little noise, so I think I managed to hit all of my goals.

Photograph Digitising and Preservation

Mon, 26 Sep 2011 23:38:07 +0000

Perhaps as you get older, and certainly when you have children, you start thinking a bit more about preserving family photos. All photos of my kids have always been digital, so for me preservation and archiving that has been a digital storage and backup dilemma, so have a look down the bottom of this post for my solutions on that.

However, almost all of my own childhood pictures are on single pieces of paper in boxes, some in albums, and some of which we still have the single negatives of. For me, that’s an issue as I live on the other side of the planet from the rest of my family and those photos, and there’s only that single copy, which we’d like to duplicate to have here in Japan as we start our own family, and not just to reduce that single point of disaster. I want to give my kids a giggle when they look at what I did when I was a kid.

Most of my old family photos were shot on cheap 110 film cameras with those stick flashes which melted after one use (my children will laugh at this post when they’re old enough to read it!)

110 film was widely available in the seventies and eighties, and roundly regarded as not very good nowadays. This means many of my childhood photos aren’t great technically, but from going through this process with a hundred or so photos so far, it isn’t the image quality that matters, it’s the memory of the moment, so don’t worry if that awesome memory you have in your head seems to have a coffee ring on the corner of the print and is slightly out of focus on the paper - it doesn’t detract at all.

Scanning

I have a recent, but not new, Epson GT-X770 scanner - it’s a mid range home scanner, and supports slide and negative scanning via a top lid light, and plastic negative holders. I did though, have to make my own holder for 110 film negative, as virtually no scanners support it natively. It also has some solid scan drivers and hardware elements for colour restoration, scratch removal etc.. I tend to scan two versions of each print; one a straight un-modded one, and one with colour restored because I actually find the scanner seems to do a better job of this that Photoshop for some reason.

The resolution I scan at depends on the source and the photo. There are plenty of guides out there, though I find some to be a bit off to me - a good website for reviewing some of the more technical aspects is ScanTips

Generally I scan colour paper print photos at 300dpi, and a few at 600dpi [dots per inch].

300dpi is easily good enough for most prints, and recently, what most photo print shops printed at anyway. From what I can tell from testing on the 110 prints, 300dpi is already far above what was captured on the paper anyway. I use 600dpi for some black and white prints from good sources, though again, I’m probably going too high and am pretty much capturing high resolution grain some of the time.

For negatives, you’ll use a much higher dpi setting (as the negatives are so small) and you start to see why scanners often have fantastically high scan numbers.

I tend to go for ~2400-3200dpi, but again, you run the risk of purely scanning grain in the negative, rather than capturing any real value, so no need to put the scanner on max, unless you have pristine, well shot slides or negatives. Again, for me, my sources are relatively poor, so I don’t need to go too far - to around 3200dpi for some 35mm negatives I had from early backpacking trips with my old Olympus.

Other Scanner Settings
I tend to just try to capture everything in the scanner, and then ‘fix’ in software on a copy. The only exception is really colour restore as I said, and some hardware features. Unsharp mask etc. I just leave for software adjustments later on.

File Format

When saving anything you’ve digitized from an analogue source, you want it to be in a loss-less format - this means that it’s exactly what you captured. The other system is ’lossy’ and generally you don’t see the difference, except when you do multiple generations of saves.

I save the files as loss-less TIFF files. This seems to be the most widely supported format, and holds comments and other things fairly well, it also has some lossless compression options, and handles high colour depth for those with higher end scanners. I’ve saved some of my own scans as PNG format also, as there’s nothing wrong with it, and has some advantages over TIFF in loss-less compression, but lacks support in some applications.

Never, ever, save as JPG - just don’t - always try to save a loss-less version as your ‘gold master’, and take JPGs from that. For any file conversion, I recommend IrfanView for Windows, and GraphicConverter for OS X.

Outsourcing

If the thought of getting a scanner and taking the time and effort to do all those photos seems scary or just plain painful, you can pay someone to scan your prints and negatives, and return them to you. This might also be an option if you’re looking at a huge archive.

There was some controversy a few years ago that many of the companies ship your photos to India for scanning and clean up, though there never seemed to be anyone who’d lost photos or had any other issues with any of these services, and as long as the end result is good, I fail to see how it’s an issue.

That Digital Storage
Backups for my computers used to really be about things I’d written and so on, but basically, like this blog - I always have a copy online these days, but I will confess I still backup a WordPress export locally now and then, but I think that’s relatively safe.

Prints are on paper, and you have a negative. There’s no worry about obsolete file formats, or applications, you just look at them in an album - risk of deletion is fairly low.

For any format though, there’s the risk of natural disaster, fires, theft and all manner of things - I tragically saw a lot of this when I went up to Tohoku to help clean up the tsunami damage - we all kept a keen eye out for any photos, CDs, hard drives or negatives which may help someone put their family memories back together.

I see a lot of people saying to keep a USB drive as a backup, but I’m not sure that’s safe enough. I believe in 3 copies - two different media locally, and one off-site copy. For me, most of my photos are on a Mac, that’s backed up sort-of live to a USB drive (not a portable one) on my desk via Time Machine, and then I use CrashPlan to upload another copy to their servers over the internet.

I used to back up to DVDs, but the libraries became just too large to burn, and I can’t trust small hard drives, though for a while i would keep one in my office drawer as a backup.

So that’s my setup for archiving the analogue photographic past. There are cheaper, simpler, or more expensive ways of doing it, but this is working for me. I don’t think you can go too wrong as long as you get a basic loss-less image file at a decent resolution, and back it up - and keep that original print or negative.

Original Comments

Comments from the original WordPress blog post.

sadie — 2011-10-03

Greetings! I have been trying to get in touch with people that might know something about the salvaging of Tohoku family photos, so perhaps you could help. I am an American artist living in Tokyo and have been trying to get this next body of work underway for a last minute show at the end of this month concerning our neighbors up north. Many artists here in Japan have focused on the Daichi plant and the problems with nuclear power and radiation (very important and frightening for all of us of course) but I feel like in the city many have forgotten the individuals that were effected. I will be in an art fair here in October and really want to get copies of the family photos left behind in the debris to paint, whether the subjects are known or unknown, as a memorial. What do you think? If you can offer any help or advice, I certainly appreciate it! Also, if you are still taking volunteers I am very interested of course.

Many many thanks, Sadie

gurahamu — 2011-10-03

Hello, and thanks for the comment.

I’m not really sure where would be best to start, You could try the organisation we worked through on the clean-up trips - http://tonomagokoro.net/ and perhaps they can walk you through the scale of the media recovered and what happens to them. We basically took all photos and negatives, and any CD-Rs or camcorder tapes and separated them into their own containers, and then they were taken somewhere else. I know one team spent several hours trying to get some boxes of photos out from the sludge in a basement, which again were taken away. Sorry I can’t help much more with ‘what happened next’, but I’d still try with the volunteer coordinators first.

Let me know how you get on.

[Not] Unlimited Mozy

Tue, 08 Feb 2011 13:40:22 +0000

Something interesting happened on February 1st. in the incredibly exciting realm of online backups we found that it’s not unlimited at Mozy. Let’s just address one thing first: backups. Please make backups and make them regularly. Even better, have an onsite and an offsite backup. For me, offsite used to mean burning to DVD-R and shipping to the parents, nine time zones away, and thus preserving all my family’s pictures, video and a few other bits in case something Bad (™) happened.

Initially I went with JungleDisk/Amazon S3 for this, but in November 2009 shifted to Mozy - unlimited backup for 5USD a month which made more sense for me and meant I could also backup all the .dv videos and music, then about 140GB in total, now about 175GB.

As I say then, on Feb. 1st. Mozy dropped their unlimited plan, and implemented a tier based system. Instead of unlimited for 5 USD/month, it would be 50GB for 6USD/month and additional fees per 20GB above that. For me then, that would be about 21USD / month - a decent price increase.

As I posted on their forums, on the now 847 post complaint thread:

“I appreciate storage (even with EMC behind you) costs money. Bandwidth costs money. I can understand your move to a tiered, non unlimited plan, I really can. That said, here’s my two thoughts:

- you set the baseline too low - I see a lot of people on Twitter, and myself between 150-200GB, and for us, there’s just no economic reason to stay once our plans expire. For me that’s 10months, though I need to be careful to avoid the auto-renewal if I decide to leave (and unfortunately take my parents with me). I don’t think we’re abusing your generous unlimited offer with that amount per month. My fee likely covers my cost to you.

- but honestly, what irks me most is that I learn about this on Twitter, and on Lifehacker. I don’t have an e-mail from you, I don’t see anything on the blog from you. That just seems to imply you’re not interested in customer relations. As I implied above, that I only get mail from Mozy after they’ve charged my credit card, but not for a product offering shift like this means that I think you’re going to have a painful reaction from users who don’t watch every product announcement not in their mailbox.

Again, it’s a bold move by Mozy and I’m sure others will follow, and users will chose to stay or go, but really, whoever handled the rollout of this announcement needs to be given a little talking to.”

The first company blog entry for the change came on February 3rd. A touch late for many. That support forum thread is 99% complaints, accusations, as you can imagine.

I didn’t get the email notification Mozy say they sent. I use Google Apps for mail and had a look but found nothing. I asked about this and Community Manager Mike responded that so many e-mails going out at once probably set many ISPs spam alarm bells ringing. I find this fairly plausible, but given I’d already seen it on Twitter it was essentially moot within 24hours anyway.

“Mike, thanks for posting, and I wish you all the best too.

Business needs change, strategies change, sheer costs change, and that’s part of the world. Per my previous post in this thread, one thing I hope Mozy is hearing, aside from people’s unhappiness with the costing, is the communication. I found out via Lifehacker - no email for me. That’s not impressive. When I went to the blog at http://mozy.com/blog/ there’s nothing there either.

Given this thread, and the pounding #mozy is taking on Twitter and elsewhere, which I’m sure Mozy anticipated, it’s obvious a lot of people liked the service as it was, and many will leave due to the change. I hope those who work in Mozy stay employed in whatever direction Mozy is going. A company is always made up of it’s staff and it’s customers.

As for me, I have a few months for myself and my parents until renewal, so I’ll likely wait and see what happens.”

To which Mike responded:

“Thanks. The emails went out last night, however because of the volume of emails, it took a while for all of them to be sent. Also, because of the volume, I wouldn’t be surprised if it hit a lot of spam filters. The Mozy blog was supposed to contain this content here, but they moved it because of the high traffic (they didn’t want to bring mozy.com to a crawl).”

I’m actually finding this quite interesting, both in how to manage this from Mozy’s point of view, and how to seize some advantage. You only have to check out some Twitter streams for Feb. 1st. to 3rd. to see how this works: mentions of Mozy, #Mozy

Note that the hashtag seems to have a lot of spam like entries still pimping the unlimited and free plans - it’s fair to say Mozy has zero control over this.

Basically, it’s a pounding at best. Mozy in their Twitter stream is obviously on the defensive and they don’t really have any carrot to dangle. Also, note that up until the end of January, Mozy would go for days without posting - now it’s more than hourly.

Now see how competitor Crashplan is working this with their ‘mozyonover’ 15% discount link thrown out all over. @crashplan.

There’s a key couple of key phrases repeated in there such as “We appreciate your business.” and “Welcome” as well as careful retweets, and the re-iteration of their migration discount URL.

What comes out of things like this is that it makes people go and take a look at competitors - I looked at quite a few but I should say that I’ve gone with Crashplan. Crashplan have an interesting client which allows me to use their app to backup to another local machine as well as their online service (or even a friend’s PC).

Sustainable?

One thing that did occur to me is why. Obviously Mozy wants out of the large storage consumer market - chances are it just doesn’t have the profit margin they need and is more trouble than they feel it’s worth. I suspect the sweet spots are indeed these sub 50GB people, and the small business users. I wonder how long till others decide to drop ‘unlimited’; I suspect that may depend on how Mozy fares.

I think there are questions over anything called ‘unlimited’ though I honestly believe 170GB isn’t a huge amount of data, people backing up terabytes is almost impressive. Perhaps its because to me Mozy was a last resort - my main machine, a Mac - has a Time Machine drive as live backup, and I take weekly snapshots to a couple of attached hard drives.

Mozy, with it’s download speed even on my nice fibre connection, isn’t going to rival an eSATA2 HDD for restore. However, if someone walks off with that drive and the diminutive Mac Mini, then Mozy just paid for itself.

As you can see from my posts, and some of the others, I just can’t get angry about this. Yes, it’s a pain uploading all that data to another site, but my Mac is happy to do that whilst I sleep, and it’s still on Mozy whilst I move. If anything, I hope it makes people think more about offsite storage.