Home
Posted By: kevs Opinon on time machine - 02/14/11 03:33 AM
I've avoided time machine since it was invented.
I use Super Duper to clone hardrives overnight.

One of my hardrives died in the middle of the day recently.

I did not lose data, but could have.

It seems the drain on the system and storage would be enormous with time machine. any opinions? thanks.
Posted By: ganbustein Re: Opinon on time machine - 02/14/11 03:56 AM
It's hard to find anything to complain about with Time Machine. You really should be using it.

The only downside of Time Machine that I know of is that its backups are not bootable. By all means, continue using SuperDuper alongside Time Machine. Having two different kinds of backup gives you not only added protection against media failure, but adds a layer of protection against systemic errors. (By that, I mean that if someday it's discovered that one or the other method has been doing something horribly wrong all this time and no one ever noticed, or started doing something horribly wrong with a new release, you could still recover using the other.)

Do use a different physical drive for Time Machine than the one you're using with SuperDuper. Two backups to the same drive amount to only one backup.

The "drain on the system" for Time Machine is miniscule. It's extremely parsimonious of disk space, since it doesn't copy a file again unless it changes. The time requirement is also miniscule. On my machine, Time Machine needs only about 30 seconds out of every hour to do its thing. Your Mileage May Vary, of course. The initial backup will take considerably longer, because it has to copy everything. The first backup after a system upgrade is also longer than normal, because it does a deep scan to double-check that it hasn't missed anything. (It still only copies what's changed, but the deep scan adds some time.)

Time Machine never gets in your way. It does low-priority disk I/O so it won't slow you down, and it will cheerfully postpone whatever it's doing if you want to unmount the drive it's using.
Posted By: MicroMatTech3 Re: Opinon on time machine - 02/14/11 04:03 AM
Both Time Machine and Super Duper are highly reliable, but in the last few months, I had Time Machine backups that appeared to be lacking recently-modified files, and a friend of mine found that his mother's Super Duper backup was being reported as larger than the size of the entire drive on which it resided. Having both of these types of backups, as suggested by gangbustein, and keeping both of them up-to-date, is the right approach.
Posted By: jchuzi Re: Opinon on time machine - 02/14/11 10:27 AM
My experience has been the same as Ganbustein's. I too have a Super Duper backup as well as a Time Machine backup, and on separate hard drives.
Posted By: joemikeb Re: Opinon on time machine - 02/14/11 04:06 PM
I have said this before, but there is a major flaw in using Super Duper, or Carbon Copy Cloner as the only backup. Assuming you are using SD or CCC as your only backup consider this sceneario.

  • Day 1: You edit an important file and at the end of the day you backup using SD or CCC
  • Day 2: you edit the file but unknown to you there is a glitch of some sort and the file is irrepairably damaged. At the end of the day you backup using SD or CCC
  • Day 3: You attempt to open the file you have been working on for the past few days and discover it is damaged beyond repair. No problem, you simply copy the file from your SD or CCC backup. Unfortunatly that backup contains the damaged copy of the file from the Day 2 backup. You have lost all the work you have done on the file.

With Time Machine you just go back in time one more hour or one more day and recover an undamaged copy of the file having lost only an hour or so of work.

FWIW, after a logic board crash I was able to completely restore my entire system using Migration Assistant and a Time Machine backup. I will also admit I have had a TM backup set go sour for reasons I was never able to determine and I was never able to recover the backup set which forced me to start the TM backup over again from scratch.
Posted By: ryck Re: Opinon on time machine - 02/14/11 05:04 PM
Originally Posted By: jchuzi
I too have a Super Duper backup as well as a Time Machine backup, and on separate hard drives.

Ditto. And, although I've been fortunate never to need the SD backup, I have occasionally used Time Machine for reverting to an earlier version of a file.

ryck
Posted By: Virtual1 Re: Opinon on time machine - 02/14/11 06:45 PM
I use a custom written rsync script, which backs up to another computer at my house. It's geared to be able to go over the internet if the computer is not available on the LAN, so my backups go uninterrupted as long as I have internet connectivity. I can also access my backups remotely, which I've needed to do on occasion.

The backup it makes is bootable, though it's not at the root folder of the hard drive it resides on. The hard drive has backups of multiple computers, so one large hard drive is backing up 4 machines. (something you can only get with a time capsule when using time machine) Though moving any one of the four to the root folder would turn any of them into a bootable backup.

The only downfall is the lack of versions. While I recognize the shortcoming, I have yet to run into a need for it. Every time I've needed to snag a backup it's had what I needed. But that's the one thing I admire in time machine over my solution.
Posted By: kevs Re: Opinon on time machine - 02/15/11 01:53 AM
Thanks all, great comments.

I have all my data on one external which copies/ clones to another external the same size via SD at 4am. I also have a clone of this at the Bank- security deposit.

Been doing that for years - no time machine.

But as my hard drive fried on the spot I realized as Joe mentioned what if you were doing critical work that whole day.

Ok. questions for a time machine newbie:
How do I get started? What documentation should I read.
My new backup hard drive does have a ton of space -- over 1TB right now. My Mac HD also has one TB -- its empty except for the OS. I also have many 500GB externals I could use.

Do you make folder and back everything into it? thanks.
Posted By: tacit Re: Opinon on time machine - 02/15/11 04:32 AM
There are two different kinds of backups: backups for version control, and backups that protect against hardware failure.

Time Machine is version control backup. It protects you from accidentally editing, overwriting, or deleting an important file; with Time machine, you can go back in time, so to speak, and see earlier versions of the file. Versioning backup requires a lot of space, as it spans multiple versions of a file's history.

Time machine does not protect against hard drive failure, in the sense that if you lose a hard drive you can't simply switch over to your Time Machine drive and keep going.

Super Duper, Carbon Copy Cloner, and similar programs are hardware failure backups. They create mirrors of your current hard drive, in a bootable form. If oyur hard drive suddenly ends up deep six on you, you simply boot from the copy and keep right on going, losing only any files that have changed since the last clone.

They do not, however, do version control backup. You don't get multiple snapshots of a file's revision history; you merely get an exact clone of your drive as it was at the moment you made the backup. Using CCC or SuperDuper, you can not revisit a file in previous versions to recover accidentally modified or overwritten versions.

A good backup strategy requires both. This is something I tell my clients; you need both versioning backup and hardware failure backup.
Posted By: MarkG Re: Opinon on time machine - 02/15/11 05:19 AM
This may be useful. Mac 101: Time Machine
Learn how to set up Time Machine to perform backups, how to restore items (or your entire system) from a backup, how to use existing backups on a new Mac, and more.
http://support.apple.com/kb/HT1427

This should have been a reply to kev, not to tacit
Posted By: Virtual1 Re: Opinon on time machine - 02/15/11 04:03 PM
Originally Posted By: tacit
A good backup strategy requires both. This is something I tell my clients; you need both versioning backup and hardware failure backup.


A boot drive that is a mirror, in combination with a time machine backup, is the easiest way to accomplish both. SuperDuper/CCC are effective, but not nearly as convenient as a good ol mirror, which is automatically and continuously updated, unlike CCC etc that require you to explicitly run them.
Posted By: joemikeb Re: Opinon on time machine - 02/15/11 04:42 PM
Quote:
A boot drive that is a mirror, in combination with a time machine backup

As long as I am dreaming and money is no object, my preference would be a boot drive that is a RAID 5 array along with a Time Machine backup on another RAID 5 array. grin
Posted By: Ira L Re: Opinon on time machine - 02/15/11 08:50 PM
I too live with Time Machine and a cloned copy to an external drive. However, like MicroMat observed above, Time Machine seemed to be missing a recent (2-3 days old) copy of a modified file. Never really noticed it ever before, and having another type of back up can potentially avoid this problem.
Posted By: kevs Re: Opinon on time machine - 02/16/11 11:06 PM
thanks guys, great tips.
I will get going.

I would like to perhaps either target time machine backups to an older external hardrive or -- my Mad HD has about almost the entire Terra-byte free, but is it too late to parition it?

What do you all recommend on that small issue -- where to target the backups to.
Posted By: joemikeb Re: Opinon on time machine - 02/17/11 03:15 PM
Apple's recommendation is a drive or partition that is two to three times the size of the drive being backed up. However, with TB capacity drives that does seem to get a bit out of hand. My recommendation would be to
  1. estimate the maximum data you will ever have on your hard drive
  2. multiply that by two because it is almost a dead certainty you will underestimate by at least that much
  3. multiply that by three to get the size of the volume you should have for the Time Machine backup.

The most likely failure is not a failure in one volume or the other, it is the failure of the drive itself which immediately renders the backup a pointless exercise. Time Machine should be on a separate, preferably external, drive.

I don't partition any drive. Partitions establish fixed limits on volume capacity and you already have enough of a limit posed by the capacity of the drive. I quit partitioning drives when I upgraded to OS X 10.1. My experience with partitions had been 100% consistent: I invariably use up the space in one partition while there is still lots of empty but unavailable space available in the other partition.

Admittedly I tend to be very conservative about these things. I have a Time Machine/Time Capsule backup, a SuperDuper Clone on a industrial quality drive with a five year warranty; and I back up preferences, databases, and financial records to a MobileMe iDisk. At various times, I have used each of these to recover from a drive or computer failure.
Posted By: kevs Re: Opinon on time machine - 02/17/11 10:48 PM
Joe,
I'm just doing a 150GB partition for the OS backup. The other 2.85 TB is data, and currently 1.4 TB is free.

I also back up certain things to CDR's but I think having one back up in the house and then another monthly backup hardrive in security bank deposit is pretty good backup.

Last question: Time Machine. I'm mainly doing it to be honest, just for that very rare occurrence when and if, the primary external fails, I'll have the work from that day somewhere. I dont' really need tons and tons of back ups (although that's a nice luxury) because if the overnight backup via Super Duper works -- I'm covered. I just want time machine to instantly back up what is new and not backed up by SD yet (which I schedule 1x day in middle of the night)

I'm taking about needed a backup for just a 8-10 hour period before Super Duper does its thing once a day.
Posted By: Pendragon Re: Opinon on time machine - 02/18/11 10:44 AM
Originally Posted By: kevs
I'm taking about needed a backup for just a 8-10 hour period before Super Duper does its thing once a day.


kevs,

Just curious. If you all you need is a backup for the 8-10 hour period before SD does its thing, why not merely create an additional SD schedule to that effect?
Posted By: artie505 Re: Opinon on time machine - 02/18/11 02:09 PM
> I'm taking about needed a backup for just a 8-10 hour period before Super Duper does its thing once a day.

> I also back up certain things to CDR's but I think having one back up in the house and then another monthly backup hardrive in security bank deposit is pretty good backup.

I dunno... On the one hand, you're concerned about an 8-10 hour period, and on the other, you're perfectly happy with having nothing more recent than a month old backup if you experience an in-house catastrophe on the last day of the month.

Sounds like in addition to whatever scheme you wind up with you also need an electronic, off-premises, daily backup..."cloud" based or otherwise.
Posted By: artie505 Re: Opinon on time machine - 02/18/11 02:10 PM
Originally Posted By: Pendragon
Originally Posted By: kevs
I'm taking about needed a backup for just a 8-10 hour period before Super Duper does its thing once a day.

kevs,

Just curious. If you all you need is a backup for the 8-10 hour period before SD does its thing, why not merely create an additional SD schedule to that effect?

Or would a mirror setup be more appropriate?
Posted By: joemikeb Re: Opinon on time machine - 02/18/11 03:25 PM
Originally Posted By: keys
I just want time machine to instantly back up what is new and not backed up by SD yet (which I schedule 1x day in middle of the night)

Time Machine does not provide "instant" backups. It only backs up at hourly intervals. I am not aware of any application that provides instant backups of anything on your computer with the sole exception of synchronizing selected system data with MobileMe and that is dependent on the settings in System Preferences > MobileMe > Synch.

I'm with Artie, If all you are concerned about is that very narrow period of time and your requirement is for "instant" backups a RAID array of disks is the only option. The cheapest to implement would, of course, be RAID 1 (mirrored) array. RAID 1 can be successfully implemented in either hardware or software. If you have a fat wallet then my choice would be RAID 5 which AFIK has to be implemented in hardware but is by far the most reliable and safest. It appears you have no concern about "versioning" and if that is true, a RAID 1 or 5 array would eliminate the need for your clone backup as well.
Posted By: Virtual1 Re: Opinon on time machine - 02/18/11 04:04 PM
Originally Posted By: joemikeb
Time Machine does not provide "instant" backups. It only backs up at hourly intervals. I am not aware of any application that provides instant backups of anything on your computer with the sole exception of synchronizing selected system data with MobileMe and that is dependent on the settings in System Preferences > MobileMe > Synch.


Would a folder action work for that?
Posted By: kevs Re: Opinon on time machine - 02/18/11 08:54 PM
Joe, others, Yeah, the hourly thing sounds good.
Posted By: ganbustein Re: Opinon on time machine - 02/19/11 12:43 AM
Originally Posted By: kevs
I think having one back up in the house and then another monthly backup hardrive in security bank deposit is pretty good backup.

Last question: Time Machine. I'm mainly doing it to be honest, just for that very rare occurrence when and if, the primary external fails, I'll have the work from that day somewhere. I dont' really need tons and tons of back ups (although that's a nice luxury) because if the overnight backup via Super Duper works -- I'm covered.

The SuperDuper backup is important, but it won't protect you against a file being corrupted. SuperDuper will simply back up the corrupted file. Same thing for a file that gets mysteriously deleted. Once it's gone from the primary volume, the next SD backup will delete it from the secondary volume as well.

Mirroring is useful only as a defense against physical drive failure. It offers absolutely no protection against corrupted files, mysteriously deleted files, or worse, a corrupted disk catalog. At least with SuperDuper you have a few hours to notice the problem and fix it; with mirroring, damage propagates instantly to both drives.

Time Machine protects against all those things. Its only drawback is that its backups are not bootable. (And, it's only one kind of backup. You really should have two kinds of backup.)

For offsite backup, I rotate between two Time Machine backups. I actually have three TM backups: one stays at home and does hourly backups. The other two rotate, one in-house and the other off-site at any one time. For the sake of clarity, call them A, B1, and B2.

Once a week, I tell TM to back up to B1 (assuming that's the drive that happens to be onsite) and let it do one backup. Then I tell it to go back to using A for backups. I carry B1 offsite and swap it for B2. As soon as I get B2 home, I again tell TM to do one backup to B2, then switch it back to A. Next week, I reverse the roles of B1 and B2.

In effect, both B1 and B2 contain only weekly backups, each containing a backup that is at worst a little over a week old. Meanwhile, A continues to give me rapid access to recent versions, with the normal mix of hourly, daily, and weekly backups. A SuperDuper backup completes the coverage.

That sounds like a lot of backup, but I've had TM backup drives fail on me, and it's disconcerting to have all that history disappear. I haven't ever lost any data, but just like in a car, any time you use your spare you no longer have a spare. Besides, that history is useful for other things besides recovery. Sometimes you just want to know what changed and when.

(Case in point: it once was important to me to know when a particular preference got set. The modify date on the preference file wouldn't tell me, because that file got updated several times a day. But TM still had many versions of that file, and it was easy to find the first one that had the particular preference set. Losing your only TM backup makes such questions suddenly unanswerable.)

If recovery requires that I actually use my offsite backup, I've probably also lost the computer itself to fire or burglary, and the extra hour or so it takes to recover from a non-bootable backup pales beside the time it takes to replace the hardware.
Posted By: roger Re: Opinon on time machine - 02/19/11 03:05 AM
since we're talking about what TM does, I hope that this is an acceptable place for this query.

if I run TM, then delete files from the computer, then run TM again, the deleted files still exist on the TM drive, correct? TM just creates a second, updated backup, leaving the first intact?
Posted By: kevs Re: Opinon on time machine - 02/19/11 03:52 AM
nice post Gan,
I just read the Apple bit on time machine.

I assumed:

1) I could use an old 250 GB external, and then time machine would back up all files that are changed starting now.

no go? the documentation implied I need to first backup everything I own, meaning I have to buy another new 3TB external is this correct? bummer if so.

2) I would also assume I can include/exclude which folders get backed up, but I did not see this in the documentation. Not possible?
Posted By: tacit Re: Opinon on time machine - 02/19/11 06:50 AM
Originally Posted By: roger
since we're talking about what TM does, I hope that this is an acceptable place for this query.

if I run TM, then delete files from the computer, then run TM again, the deleted files still exist on the TM drive, correct? TM just creates a second, updated backup, leaving the first intact?


Yep!
Posted By: MicroMatTech3 Re: Opinon on time machine - 02/19/11 04:07 PM
I am not able at the moment to find a definitive explanation of how Time Machine works, but it was my understanding that it is a backup utility, not an archiving utility. At some later time, the only copy in the Time Machine backups of the file that was deleted from the source volume after being backed up by Time Machine will exist only in what will have become a very old Time Machine backup. If that backup is deleted to create more free space, the file will be nowhere.

If a file still exists on the source volume, then it should be present in at least one of the Time Machine backups, but if it has been deleted from the source volume, it may not be in the Time Machine, depending on elapsed time and disk space use.

Please correct me if I have misunderstood how Time Machine works. Also, I do not understand why this text is frequently quoted but seems to have no original source on the Web:

(from http://en.wikipedia.org/wiki/Time_Machine_%28Mac_OS%29)

"Time Machine is a backup utility, not an archival utility, it is not intended as offline storage. Time Machine captures the most recent state of your data on your disk. As snapshots age, they are prioritized progressively lower compared to your more recent ones."[citation needed]
Posted By: roger Re: Opinon on time machine - 02/19/11 04:44 PM
thanks, tacit and MMT3! I think what I might do is copy the files I want to keep, but don't need on the laptop, into a separate folder on the TM drive (there's plenty of room, it's 2Tb). then I'm assured that I have them, and wouldn't need to worry about the TM backup disappearing.
Posted By: kevs Re: Opinon on time machine - 02/19/11 06:08 PM
Micromat, thanks -- were you answering my final questions? I'm not sure clear?

Here are my finals again, (sorry for being redundant)


I assumed:

1) I could use an old 250 GB external, and then time machine would back up all files that are changed starting now.

no go? the documentation implied I need to first backup everything I own, meaning I have to buy another new 3TB external is this correct? bummer if so.

2) I would also assume I can include/exclude which folders get backed up, but I did not see this in the documentation. Not possible?
Posted By: Virtual1 Re: Opinon on time machine - 02/19/11 07:36 PM
Originally Posted By: roger
if I run TM, then delete files from the computer, then run TM again, the deleted files still exist on the TM drive, correct? TM just creates a second, updated backup, leaving the first intact?


It doesn't delete them immediately tho I'm not sure at what point it decides to delete them. There is an option to delete a file along with all time machine backups of the time, I don't recall offhand how you invoke it. (secure erase of trash?)
Posted By: jchuzi Re: Opinon on time machine - 02/19/11 07:50 PM
Quote:
I would also assume I can include/exclude which folders get backed up, but I did not see this in the documentation. Not possible?
Very possible. Go to System Preferences > Time Machine and click the Options button. You can exclude items from there.
Posted By: joemikeb Re: Opinon on time machine - 02/19/11 09:43 PM
Originally Posted By: roger
if I run TM, then delete files from the computer, then run TM again, the deleted files still exist on the TM drive, correct? TM just creates a second, updated backup, leaving the first intact?

Correct and that file will remain in the TM backup until you run out of room on the TM drive and the backup it is contained in "fall off the backside" of the entire backup set.
Posted By: kevs Re: Opinon on time machine - 02/20/11 12:42 AM
JON,
thanks, it looks like you can exclude only hardrives, (which is good), but not folders. correct? And will it just start filling up with whatever I now update or do I have to have a external hardrive with the space to match the 2TB of data I have on my main drive?
Posted By: Hal Itosis Re: Opinon on time machine - 02/20/11 03:15 AM
Originally Posted By: MicroMatTech3
I am not able at the moment to find a definitive explanation of how Time Machine works, but it was my understanding that it is a backup utility, not an archiving utility.

As one ComputerWorld.com article said, Apple keeps most of those details close to the vest. [i'm pretty sure there's no developer or knowledge-base doc which describes TM's decision-making process to any deep extent, but it would be a pleasure to be proven wrong about that.]

Our fellow member ganbustein has written several posts on the matter, including:
Posted By: Hal Itosis Re: Opinon on time machine - 02/20/11 03:25 AM
Originally Posted By: kevs
it looks like you can exclude only hardrives, (which is good), but not folders. correct?

Folders can be individually excluded... but, they must reside on a disk which is already included.
Posted By: MicroMatTech3 Re: Opinon on time machine - 02/20/11 04:07 AM
Hal,

Thanks for the details, especially the fact that the excluded folder must be on an included volume. MicroMat suggests that users of TechTool Pro exclude the backups of the disk directory made by the TechTool Protection feature, which are at /Library/Application Support/TechTool Protection/Volume_Name.

If I recall correctly, you had a list of the files that Apple automatically excludes from Time Machine. Ah, Yojimbo finds it, provided I can find one posting somewhere that mentions one distinctive excluded file:

(from the ancient forums)

Leopard tips? What's good?
10/28/07 08:50 AM

Hal Itosis
MacWizard

Reged: 08/23/99
Posts: 5114
Loc: 10.5.0 (build 9A581)


Files auto-excluded from Time Machine backups [re: Hal Itosis]
11/09/07 09:43 PM
Reply

Files auto-excluded from Time Machine backups...
code:

$ defaults read /System/Library/CoreServices/backupd.bundle/Contents/Resources/StdExclusions
{
ContentsExcluded = (
"/Volumes",
"/Network",
"/automount",
"/.vol",
"/tmp",
"/cores",
"/private/tmp",
"/private/Network",
"/private/tftpboot",
"/private/var/automount",
"/private/var/log",
"/private/var/folders",
"/private/var/log/apache2",
"/private/var/log/cups",
"/private/var/log/fax",
"/private/var/log/ppp",
"/private/var/log/sa",
"/private/var/log/samba",
"/private/var/log/uucp",
"/private/var/run",
"/private/var/spool",
"/private/var/tmp",
"/private/var/vm",
"/private/var/db/dhcpclient",
"/private/var/db/fseventsd",
"/Library/Caches",
"/Library/Logs",
"/System/Library/Caches",
"/System/Library/Extensions/Caches"
);
PathsExcluded = (
"/.Spotlight-V100",
"/.Trashes",
"/.fseventsd",
"/.hotfiles.btree",
"/Backups.backupdb",
"/Desktop DB",
"/Desktop DF",
"/Network/Servers",
"/Previous Systems",
"/Users/Shared/SC Info",
"/Users/Guest",
"/dev",
"/home",
"/net",
"/private/var/db/Spotlight",
"/private/var/db/Spotlight-V100"
);
UserPathsExcluded = (
"Library/Application Support/MobileSync",
"Library/Application Support/SyncServices",
"Library/Caches",
"Library/Logs",
"Library/Mail/Envelope Index",
"Library/Mail/AvailableFeeds",
"Library/Mirrors",
"Library/PubSub/Database",
"Library/PubSub/Downloads",
"Library/PubSub/Feeds",
"Library/Safari/Icons.db",
"Library/Safari/HistoryIndex.sk"
);
}


____

verified perms? • done fsck or safe boot? • cleaned caches & logs?
Posted By: ganbustein Re: Opinon on time machine - 02/20/11 05:13 AM
Originally Posted By: Hal Itosis
[quote=MicroMatTech3]As one ComputerWorld.com article said, Apple keeps most of those details close to the vest.

Our fellow member ganbustein has written several posts on the matter, including:

First off, thank you very much for providing those links to my previous posts. You've spared me a lot of work.

But I'm not sure what details the ComputerWorld.com article thinks Apple is keeping close to the vest. The Time Machine preference pane (in System Preferences) provides, in 32 words, a very nice summary of how it works that is detailed enough for the vast majority of users. The "magic" that makes it all possible boils down to two features new to Leopard: the FSEvent mechanism that lets TM (or any application) rapidly discover where files have changed since an arbitrary time in the past, and the ability to make hard links to folders on an HFS+ volume, which TM uses to save backup space when nothing in a folder has changed. TM also makes use of extended attributes and ACLs, but that's only for fine-tuning. (It uses extended attributes to record when a file was first backed up and when it was replaced, so it can quickly jump to the prior or next version without having to scan intermediate backups looking for the inode number to change. It uses ACLs to make everything in the backup non-deleteable, so users won't be tempted to "tune" the backup behind TM's back by deleting or renaming individual files.)

All four of those features are independently documented in the developer documentation. How exactly TM puts them to use is of interest only to people like me who like to dig into how things work. The details are not complex, and are in any event easily discoverable. It's much less complex than even one PhotoShop image filter, but no one complains that Adobe "keeps the inner workings of 'Sharpen Image' close to the vest." What it (either Time Machine or Sharpen Image) does is apparent; how it does it is not important to anyone but the programmers.
Posted By: kevs Re: Opinon on time machine - 02/20/11 04:01 PM
So guys, for this to work, my 500GB drive wont work?

I have to buy a new expensive 3TB drive to match the 3TB of data I have?

It just can't start on the cuff backing up data from my new 3TB external hard drive over to my backup 500GB?

I'm not sure I'm willing to buy a new 3tB drive just for time machine when I have an excellent 500GB drive I'm not using.
Posted By: Virtual1 Re: Opinon on time machine - 02/20/11 05:03 PM
initially it need to make one full copy of anything not being excluded. So if you have a 3tb drive, with say 1tb of data on it, it can't even make the first initial backup on a 500gb drive.

If you truly want time machine to be an incremental backup the general rule of thumb I've heard is to have the backup drive be twice the capacity of the drive it backs up.

For a 3tb drive, I suppose you'd be looking at getting a drobo and stuffing it with 2tb drives?
Posted By: kevs Re: Opinon on time machine - 02/20/11 05:44 PM
It looks like I wont be using time machine after all.
Pity it just cannot start off the cuff backing up stuff to any size drive.

Also curious how does it work? It need all this data on the TM back up disk. But aren't we backing up from the primary drives?
It "sees" what I did on a primary drive and then find the corresponding file and backs that up on TM?
Posted By: Hal Itosis Re: Opinon on time machine - 02/20/11 11:05 PM
Originally Posted By: ganbustein
But I'm not sure what details the ComputerWorld.com article thinks Apple is keeping close to the vest. The Time Machine preference pane (in System Preferences) provides, in 32 words, a very nice summary of how it works that is detailed enough for the vast majority of users.

I'll just <link to the article> then, and quote the specific passage:

Originally Posted By: Ryan Faas

Inside Leopard's Time Machine: Backups for the rest of us (excerpt from page 5):

Finally, it is worth mentioning that Time Machine is very smart in how it determines what to delete when a disk begins to get full. It doesn't simply delete the oldest backups and all their files. Instead, when Time Machine deletes an older backup, it deletes only the files that were unique to that backup (i.e., files that no longer exist anywhere in the file system).

Time Machine also doesn't delete just the oldest backups. While it does keep many recent backups at frequent intervals, it also keeps a range of older backups from a wide range of dates, letting you browse as far back in time as feasible. Exactly what computation goes into determining this approach isn't quite certain (and Apple may have a competitive advantage if it chooses to keep some of Time Machine's secrets close to the vest), but in early testing, it does seem an effective solution.



edit: note that that comes from a very early review, dated October 29, 2007.
Posted By: roger Re: Opinon on time machine - 02/21/11 02:05 AM
Originally Posted By: Hal Itosis

Originally Posted By: Ryan Faas
it deletes only the files that were unique to that backup (i.e., files that no longer exist anywhere in the file system).


does anyone else thing that's a bit backward, or am I reading it wrongly? wouldn't you want it to keep the unique files as long as possible, and delete the ones that were available on other backups?
Posted By: Hal Itosis Re: Opinon on time machine - 02/21/11 03:51 AM
Originally Posted By: roger
does anyone else thing that's a bit backward, or am I reading it wrongly? wouldn't you want it to keep the unique files as long as possible, and delete the ones that were available on other backups?

Hmm, hard to say (or rather... i'd need to think more before replying wink ).

If a file is unique, then that implies it isn't multi-linked to in any other snapshot. That condition further implies that the item must have been deleted (from the HD) a good while ago [edit: or, alternatively, it didn't exist on the HD for any significant length of time to begin with]... and is therefore more dispensable than other items. [but i know what you're saying... and i'm just guessing myself.]

The other factor is this: it's only by removing unique items that we can free up space. When there are multi-linked items mingling in Time Machine's maze, "removing" one of them merely decrements the link count... thus no actual space gets freed by that process.

Note that Ryan was talking there about when a disk begins to get full.

Looking back on some of the exchanges you've had in this thread, i will say that this unique file business ties in nicely with what MicroMatTech3 said: Time Machine is not an archiver... so make good use of that "extra" folder on the backup disk to store items deleted from the HD. Yes, they do exist (perhaps) in some weekly snapshot... but if they're no longer on the HD, then their heads are on the chopping block when TM needs more room.

--

[i'll probably edit this and say more later. meanwhile, i'm just grateful that ganbustein has a good grip on all this.]

EDIT: i just want to add some thoughts here, because the whole magic behind Time Machine (rsync snapshots) is that it already saves a ton of space by its very nature (the --link-dest option to rsync). Realize that —within the Backups.backupd directory —we eventually accumulate a great number of "snapshots". 23 (???) hourly snapshots, 30 (???) daily snapshots, and an unknown number of weekly snapshots. So we typically have something like over 50 discrete 'versions' of our entire HD sitting there in that folder.

When i used the word 'maze' a few paragraphs back, i meant it. That Backups.backupd directory is no ordinary folder. It contains an intricate matrix of many possible HDs. If you ever try to measure that folder's "size" ...you're in for some fun (depending on which tool you use).
Posted By: ganbustein Re: Opinon on time machine - 02/21/11 05:49 AM
I'd say the clumsy wording comes from ComputerWorld's not having a good understanding about how hard links work.

Suppose you had three files (A, B, and C) on your main volume when TM took its first snapshot. (This is only for the sake of exposition. You'd typically have millions of files.) Those three files would be copied to backup.

Before your second backup, suppose you delete A and add D. For TM's second snapshot, it would copy over the new file, D, and round out the new snapshot by making hard links to B and C. The backup now contains:
Snapshot 1: A(1), B(2), C(2)
Snapshot 2: B(2), C(2), D(1)
where the numbers in parentheses denote the link count, that is, the number of places in the filesystem where the exact same file occurs.

Between then and the third backup, you delete B and add E. The third backup copies the new file E and increments the link counts on the still-present C and D to produce:
Snapshot 1: A(1), B(2), C(3)
Snapshot 2: B(2), C(3), D(2)
Snapshot 3: C(3), D(2), E(1)

There are five physical files (aka inodes) in the backup: A, B, C, D and E. C is listed as C(3) because that one file appears in three snapshots. It's linked to from three directories, so it has three different pathnames, but all three pathnames lead to the same inode. It's a file's inode that contains its data, so the data appears on disk only once.

Suppose we now delete Snapshot 1. In doing that, the reference count of each file in the snapshot is decremented, and any files whose counts go to zero get deleted. This is not unique to Time Machine. It's standard behavior for hard links. In fact, the Unix command rm is really just an alias for unlink, which removes one link from a file and, if and only if that reduces the link count to 0 deletes the underlying inode.

In this case, the only file whose link count goes to zero is A, so that's the only inode that gets deleted. Even though B and C are "removed" (really, "unlinked"), their inodes remain because they're still linked to from snapshots 2 and 3.

This is the behavior that ComputerWorld is talking about when they say
Quote:
Time Machine is very smart in how it determines what to delete when a disk begins to get full. It doesn't simply delete the oldest backups and all their files. Instead, when Time Machine deletes an older backup, it deletes only the files that were unique to that backup (i.e., files that no longer exist anywhere in the file system).
but I repeat that that's nothing smart or clever on TM's part. TM really does just delete the oldest backup; the only cleverness comes from hard links doing what hard links normally do. CW seems to think that deleting Snapshot 1 should physically remove all the inodes it references, and thinks TM is being smart to somehow shield B and C from deletion.

When they refer to "the files that were unique to that backup", they're referring to files like A that are referenced from that backup and no other. The way a file becomes referenced only by the oldest backup is by being a file that was deleted from the main volume longest ago, even before the second-oldest snapshot was taken.

When you later delete Snapshot 2, it'll take with it the inode underlying B, the second file to have been deleted from the main volume. C, D, and E will still remain. Even though C was first backed up with the very first snapshot, and even though it was part of two snapshots that have both been deleted since then, it's still there on the backup (as part of Snapshot 3) because it's still there on the main volume. D was the only file copied to make Snapshot 2, but that's not what they mean by a "file unique to that backup", because D was also made part of Snapshot 3 via hard linking.

The net effect: Time Machine forgets files in the same order you delete them, not the order TM learns about them. (More or less. Any files created and deleted the same day will be forgotten by TM roughly 24 hours after you delete them, when the last hourly snapshot that captured them goes away. Two such files deleted at different times during that day will be forgotten the next day in the same order they were deleted. LIkewise for files created and deleted the same week; TM will forget them a month later, in the same order they were deleted. Files created one week but not deleted until a later week will be remembered as long as possible, but when they are forgotten they'll be forgotten in the order they were deleted.
Posted By: artie505 Re: Opinon on time machine - 02/21/11 06:09 AM
Nice, clear explanation; thanks.

This thread has become a great Time Machine resource.
Posted By: artie505 Re: Opinon on time machine - 02/21/11 07:34 AM
Originally Posted By: kevs
So guys, for this to work, my 500GB drive wont work?

I have to buy a new expensive 3TB drive to match the 3TB of data I have?

It just can't start on the cuff backing up data from my new 3TB external hard drive over to my backup 500GB?

I'm not sure I'm willing to buy a new 3tB drive just for time machine when I have an excellent 500GB drive I'm not using.

Is all the data on your 3Tb HD current, or can you archive some/a lot of it elsewhere and start Time Machine with a smaller database?
Posted By: ganbustein Re: Opinon on time machine - 02/21/11 08:17 AM
Originally Posted By: roger
does anyone else thing that's a bit backward, or am I reading it wrongly? wouldn't you want it to keep the unique files as long as possible, and delete the ones that were available on other backups?

First, let's agree on terminology.

Suppose you want to create a hierarchical filesystem on top of another filesystem. Call files in the underlying filesystem "inodes" (internal nodes), and their names, as used by the underlying filesystem, "inode names". The only way to access a real file is through its inode name, but you can create hierarchical pathnames of your own choosing.

Start by creating special files called "directories", each of which contains nothing but a map from filename to inode name. Let there be one such directory file, the root directory, whose inode name is well-known. To interpret a pathname like "/a/b/c", look up filename "a" in the root directory to get the inode name for "/a". With that inode name in hand, you can look in that directory file for "b" to get the inode name for "/a/b", and look therein for "c" to get the inode name corresponding to the pathname "/a/b/c".

In the original Unix Filesystem (UFS), the underlying filesystem was as simple as could be. An "inode name" was just a number, indexing into a table of inode descriptors, so in Unix we call it an "inode number" instead of an inode name.

It's quite possible for the same inode number to appear more than once among all the directory files in a system. That's how hard links are created.

In UFS, the directory entry for a filename contains nothing but the corresponding inode number. Everything else about the file is attached to the inode itself: owner, group, permissions, create/modify/access date, flags, size, location on disk, etc. The only thing the inode does not have is a filename. It does, however, contain a link count, a count of the number of times that inode is referenced from any directory anywhere in the system.

All of that is conceptual, of course. Some filesystems, HFS for example, were designed to be hierarchical from the get-go. In HFS, files may but usually do not have inode numbers, directories always have inode numbers even though they have no corresponding inode, being merely records within the catalog and not standalone files. HFS+, among many other changes, requires all file to have inode numbers, and makes hard-linking possible through an elaborate and contrived process.

(Aside: the git version control system also implements hierarchical names the same way, except that an inode's name is a hash of its contents, which is always a valid name in whatever underlying filesystem you're using. This makes all inodes immutable, since the slightest change in content changes the inode name.)

But never mind the mechanics. It's conceptually easier to imagine that even HFS+ works like UFS. When we talk about a "real file" or a "physical file", we mean "inode". When we just say "file", it's ambiguous whether we mean "pathname" or "inode". Thus, if /a/b/c is hard-linked to /d/e, is that one file or two? It's two pathnames, but only one inode, so the answer is different depending on what we mean by "file".

That ambiguity is at the heart of your question.

The original
Quote:
it deletes only the files that were unique to that backup (i.e., files that no longer exist anywhere in the file system).
should have been phrased
Quote:
it deletes only the inodes which had no pathnames outside the backup (i.e., not anywhere else in the file system).


You're thinking it's OK to delete a file that you have other copies of. But if by "file" you mean "inode", you never have other copies. inodes are unique.

If by "file" you mean "pathname", then it's OK to delete a pathname to an inode that's still reachable by other pathnames. As long as any pathname reaches it, the inode remains, the other pathnames still reach it, and no data is lost.

But if the backup you're deleting contains the only remaining pathname to a given inode, then yeah, it's time to delete the inode, because an inode that can no longer be reached is just wasting disk space.


BTW: this tells you how to protect any particular version of file from being purged from backup. Just make a hard link to it, putting the hard link anywhere outside the backup folder. Time Machine let's the normal hard link counting mechanism determine when to delete the inode. Your hard link bumps the link count, and TM can never get it back down to zero.

One drawback is that the inode carries all security with it, including the "group:everyone deny delete" ACL that TM slaps on everything it backs up. You won't be able to easily unlink your new hard link.

Another is that it isn't easy to protect an entire folder this way. HFS+ allows hard links to folders, but I don't know of any unix command line tool that does. You'd probably have to write your own tool in C. (Or some kind soul will chime in and tell us which command to use.)

Soft links won't work. A soft link doesn't bump the link count of the target, so it breaks when the target pathname goes away (even if the target's inode doesn't).
Posted By: roger Re: Opinon on time machine - 02/21/11 03:12 PM
thank you all. it's making sense to me now.

the references in TM to a "file" (the actual data, say an mp3) are more like aliases, and so deleting them when space is needed is pretty worthless, so of course TM would need to delete the actual file to gain any space. I might have lots of aliases to a particular file within TM, but only one that contains the actual data.

next question: how does one create a "hard link"? I take it that is more than an alias.

thanks again! laugh
Posted By: kevs Re: Opinon on time machine - 02/21/11 03:13 PM
Artie,
All my data is now on one 3TB drive called Main Hard Drive. ON that drive there is about 2TB of data.

I don't need snapshots of all that 2TB, I could be happy if time machine just backed up some important folders which are well unter 500GB.

But how?

TM options shows all hardrives and partitions on the computer. I tick to exclude the partitions I don't need, but it does not offer the ability to exclude specific folders on Main Hard DRive.

It seems you have to offer up to the TM Gods one hard drive in full, am I wrong? I would love to just offer up a few specific folders from Main Hard Drive and test it out.
Posted By: artie505 Re: Opinon on time machine - 02/21/11 03:32 PM
Not being a Time Machine aficionado (or even user), but having tried to digest this thread, my best guess is that the only way you can do what you want to do is by partitioning your 3Tb HD, archiving everything that isn't currently important on a really large partition, and keeping your current stuff, the stuff that needs to be TM'ed, on a partition that is small enough for you to work with.

But remember that you'll still need your backup HD to be no less than a Tb, and also remember that, in addition, you'll probably need to reformat your HD.
Posted By: kevs Re: Opinon on time machine - 02/21/11 04:32 PM
too late to partition, already done that, way too late.

But also curious:
how does TM work?

you have to copy everything on a TM drive and it sees what was changed on a completely different drive and changes the file that matches on the TM drive?

and what about new files you make on the non TM drive?
Posted By: ganbustein Re: Opinon on time machine - 02/22/11 12:31 AM
Originally Posted By: roger
the references in TM to a "file" (the actual data, say an mp3) are more like aliases, and so deleting them when space is needed is pretty worthless, so of course TM would need to delete the actual file to gain any space. I might have lots of aliases to a particular file within TM, but only one that contains the actual data.

next question: how does one create a "hard link"? I take it that is more than an alias.

The terms alias and symbolic link refer to more or less the same thing: a file that provides a layer of indirection by pointing to another file somewhere else. You create an alias file in Finder by using its File->Make Alias menu command, or by command-option-dragging an icon into a different folder. You create a symbolic (also sometimes called soft) link at the Unix command line using the ln tool with the -s option.

Although they serve similar purposes, they have different file layouts. The content of their inodes is different, and they use different metadata flag bits to advise the OS of their special nature.

One thing they share is that they can both be broken. Since they only point to a file somewhere else, if that file goes away they're left dangling, no longer pointing at anything.

A symbolic link contains a pathname as its content. If you're following a pathname, say /a/b/c/d, you would expect to find only directories along the way. That is, /a, /a/b, and /a/b/c would all be directories, in which you'd look up the next component of the pathname. But you could discover that /a/b is a symbolic link instead, containing perhaps the pathname /x/y/z. In that case, expand /x/y/z in the usual way, to reach the directory that "c" should be looked up in. That is, the actual path to look up is /x/y/z/c/d. Of course, /x/y could itself be a symbolic link, incurring additional levels of indirection.

Since a symbolic link contains only a pathname, it breaks if its target moves. (A rename is a special case of a move.) Moving a different file (inode) to the same place in the hierarchy re-connects the link.

To make links less fragile, an alias file contains much more data. In addition to the pathname that a symbolic link contains, an alias file contains also the inode number of every directory along the path, along with information about how to find the disk volume in case it gets renamed or unmounted. When Finder is following an alias, it will even go so far as to log into a remote server to mount a remote volume, or open a disk image file if the target is inside. Any passwords needed along the way can be automatically retrieved from the keychain.

If Finder finds a symbolic link, it'll follow it the same way Unix does. In fact, in Finder you can't tell the difference between an alias and a symbolic link (except that the latter is more fragile). Unix doesn't reciprocate, treating an alias file as some unknown file type.


The purpose of all that discussion was to dispense with the subject of aliases/soft links, getting them out of the way. Time Machine makes no use of either.

Time Machine relies instead on hard links. A hard link is what a directory creates when it resolves a pathname to an inode. We usually don't call it a hard link until there are at least two different pathnames leading to the same inode, but that's not fundamentally different from the normal case where an inode has only one pathname leading to it.

You can't refer to an inode without using at least one of its pathnames. (Once a program opens a file, the I/O subsystem refers to it solely by inode number. Changing a file's name or moving it elsewhere does not disturb programs that already have the file open.) As I've said before, an inode is aware of how many pathnames lead to it. Removing a file means removing one pathname to it, thereby decrementing the link count in the inode. The inode itself is deleted when the link count goes to zero, because at that point there's no longer any way to reach the inode. (On some filesystems, opening a file also increments the link count, so the inode still doesn't go away until the last program using it closes the file.)

At the Unix command line, you create a hard link (that is, add a new pathname to the same inode) using the ln command without the -s flag, or by using the link command.

You can see the link count on (the inode of) a file as part of the information produced by the ls command with the -l option. Adding the -i flag to that shows also the inode number.

Here's an example:

# create an empty folder and add two files to it
ronk@Zebra:Desktop(0)$ mkdir tmp
ronk@Zebra:Desktop(0)$ cd tmp
ronk@Zebra:tmp(0)$ date > f
ronk@Zebra:tmp(0)$ date > g

# peek at the contents of these two files
ronk@Zebra:tmp(0)$ cat f
Mon Feb 21 16:19:04 PST 2011
ronk@Zebra:tmp(0)$ cat g
Mon Feb 21 16:19:08 PST 2011

# look at the link counts
ronk@Zebra:tmp(0)$ ls -l
total 16
-rw-r--r-- 1 ronk _ronk 29 Feb 21 16:19 f
-rw-r--r-- 1 ronk _ronk 29 Feb 21 16:19 g

# see how much disk space we're using
# (note that 4K is the minimum file allocation, even for these 29-byte files)

ronk@Zebra:tmp(0)$ du -h
8.0K .

# create a (new) hard link. The new path h leads where the old path f did
ronk@Zebra:tmp(0)$ ln f h

# look again at the counts, this time throwing in inode numbers as well
ronk@Zebra:tmp(0)$ ls -li
total 24
56733482 -rw-r--r-- 2 ronk _ronk 29 Feb 21 16:19 f
56733483 -rw-r--r-- 1 ronk _ronk 29 Feb 21 16:19 g
56733482 -rw-r--r-- 2 ronk _ronk 29 Feb 21 16:19 h

# are we using any more disk space now?
ronk@Zebra:tmp(0)$ du -h
8.0K .

# delete the "original" of the hard link, and verify that the inode remains
ronk@Zebra:tmp(0)$ rm f
ronk@Zebra:tmp(0)$ ls -li
total 16
56733483 -rw-r--r-- 1 ronk _ronk 29 Feb 21 16:19 g
56733482 -rw-r--r-- 1 ronk _ronk 29 Feb 21 16:19 h
ronk@Zebra:tmp(0)$ cat h
Mon Feb 21 16:19:04 PST 2011


Posted By: ganbustein Re: Opinon on time machine - 02/22/11 12:43 AM
Originally Posted By: kevs
you have to copy everything on a TM drive and it sees what was changed on a completely different drive and changes the file that matches on the TM drive?

Each time TM takes a snapshot, it copies over everything that's new (and isn't excluded).

The operative word here is new. New since when?

The answer is, new since the last good snapshot. In the beginning, when it hasn't yet made any snapshots, everything is new.

Normally, it'll try to make a new snapshot every hour, but what if it can't? For example, you might have disconnected the backup drive. If it can't take a snapshot, it doesn't, but the next time it can it looks on the backup volume to see when the last good snapshot was, and backs up everything new since then.
Posted By: kevs Re: Opinon on time machine - 02/22/11 01:00 AM
Ok, Gan, so it sees the primary drive and then see it has a clone on the TM drive, and when I change something on the primary drive, it snapshots and updates to the TM drive?

Pity you have to copy everything to begin with to the TM drive. That is just cannot work out of the gate with a blank drive.

Also, what about new files I create from the primary drive??
Posted By: Virtual1 Re: Opinon on time machine - 02/22/11 01:16 AM
There's a much easier solution to this than buying another 3T hdd. I use it with my backup. I have a folder called DATA_XFER in my home. ALL my backup scripts are keyed to 100% ignore folders by that name on the source and target.

So you could just make a folder in your home that has information that does not need to be backed up, and privatize that to time machine. And you could do what you needed to separately back that up if needed. Burning to discs, copying to 2nd hard drive etc. Time Machine's ability to exclude folders is perfect for this.
Posted By: roger Re: Opinon on time machine - 02/22/11 02:05 AM
thank you so much for all of that. I know enough now to know that I know not much!!

seriously, I truly appreciate you taking the time to spell that all out for me.
Posted By: Hal Itosis Re: Opinon on time machine - 02/22/11 02:23 AM
Originally Posted By: kevs
Pity you have to copy everything to begin with to the TM drive. That is just cannot work out of the gate with a blank drive.

Wut? no... it can (in fact, it must).

Originally Posted By: kevs
Also, what about new files I create from the primary drive??

Create from? Wut?
Just try it... TM is pretty simple to operate. Then you can go look and *see* what happens.
Posted By: MG2009 Re: Opinon on time machine - 03/01/11 08:08 PM
For an "instant" backup - if one does not wish to wait for the next scheduled hourly b/u - one could select BACK UP NOW from the Time Machine drop down menu.

P.S. Perhaps not a perfect solution as it is a conscious manual operation rather than an automatic scheduled function. (Simple, but effective.)
© FineTunedMac