KISS 🇺🇦

Stop the war!

Stop the war in Ukraine! Fuck putin!

More information is at: https://war.ukraine.ua/.

There is a fund to support the Ukrainian Army: https://savelife.in.ua/en/donate/, and there is a special bank account that accepts funds in multiple currencies: https://bank.gov.ua/en/about/support-the-armed-forces. I donated to them. Please donate if you can!

Killer putin

Killer putin. Source: politico.eu.

Arrested putin

"It hasn't happened yet, but it will happen sooner or later. Beautiful photo, isn't it?" Source: twitter.

How I manually "reassociated" not quite deleted Time Machine backups

| comments

The problem statement

It all started with the need to reinstall my OSX. I have Time Machine backups, so I planned on making the final one before update, but there is no way to actually lock it. So a good idea is to make an image of your drive with another tool. I needed to clean the old TM backups first, so I ran:

1
2
3
4
5
6
7
# get the earliest ones
$ tmutil listbackups | head -10
2016-10-01-000000

# remove them
$ tmutil listbackups | xargs sudo tmutil delete

…and I missed the | head part there, which meant it’d remove all the backups. Since it’s very slow at removing them (not just rm -rf), I stopped the process at the first backup, so the rest was still there, but tmutil listbackups displayed an empty list from then on. I tried remounting the drive, running tmutil inheritbackup, tmutil associatedisk, disabling and reenabling the backup destination, selecting the backup directory by Option-clicking the TM icon — nothing worked to repopulate the list, only the new backups were visible. The official Time Machine UI displayed the same situation.

I tried searching online, but stackoverflow didn’t bring any answers, and the apple’s forums are completely unhelpful. And of course apple doesn’t provide any important details whatsoever (I hate this closeness about the OS).

Nota Bene! What’s described below is somewhat risky because I’m working on my real TM backup tree. If you need to do the same stuff, be very careful about the commands!

Extended attributes, part I

I thought that some of the information is kept in the extended attributes of the backup directories. Let’s see:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
$ cd /Volumes/backups/Backups.backupdb/mac/
$ xattr -l 2018-02-26-140000/Macintosh\ HD 2018-02-26-180000/Macintosh\ HD
2018-02-26-140000/Macintosh HD: com.apple.backupd.SnapshotVolumeFSEventStoreUUID:
00000000  45 45 45 45 45 45 45 45 2D 36 34 30 41 2D 34 41  |EEEEEEEE-640A-4A|
00000010  41 46 2D 39 31 30 38 2D 37 36 32 36 39 41 34 46  |AF-9108-76269A4F|
00000020  32 35 41 33 00                                   |25A3.|
00000025
2018-02-26-140000/Macintosh HD: com.apple.backupd.SnapshotVolumeLastFSEventID:
00000000  31 34 36 36 39 36 34 35 30 31 00                 |1466964501.|
0000000b
2018-02-26-140000/Macintosh HD: com.apple.backupd.SnapshotVolumeUUID:
00000000  31 31 31 31 31 31 31 31 2D 43 43 30 35 2D 33 36  |11111111-CC05-36|
00000010  31 31 2D 42 30 30 30 2D 45 32 32 37 46 37 35 30  |11-B000-E227F750|
00000020  31 31 45 36 00                                   |11E6.|
00000025
2018-02-26-140000/Macintosh HD: com.apple.backupd.VolumeBytesUsed: 226109964288
2018-02-26-140000/Macintosh HD: com.apple.backupd.VolumeIsCaseSensitive: 0
2018-02-26-140000/Macintosh HD: com.apple.metadata:_kTimeMachineNewestSnapshot:
00000000  62 70 6C 69 73 74 30 30 33 41 C0 22 64 1E 00 00  |bplist003A."d...|
00000010  00 08 00 00 00 00 00 00 01 01 00 00 00 00 00 00  |................|
00000020  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 11                                            |..|
00000032
2018-02-26-140000/Macintosh HD: com.apple.metadata:_kTimeMachineOldestSnapshot:
00000000  62 70 6C 69 73 74 30 30 33 41 C0 22 63 F6 00 00  |bplist003A."c...|
00000010  00 08 00 00 00 00 00 00 01 01 00 00 00 00 00 00  |................|
00000020  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 11                                            |..|
00000032
2018-02-26-180000/Macintosh HD: com.apple.backupd.PreviousSnapshotVolumeName:
00000000  4D 61 63 69 6E 74 6F 73 68 20 48 44 00           |Macintosh HD.|
0000000d
2018-02-26-180000/Macintosh HD: com.apple.backupd.PreviousSnapshotVolumeUUID:
00000000  31 31 31 31 31 31 31 31 2D 43 43 30 35 2D 33 36  |11111111-CC05-36|
00000010  31 31 2D 42 30 30 30 2D 45 32 32 37 46 37 35 30  |11-B000-E227F750|
00000020  31 31 45 36 00                                   |11E6.|
00000025
2018-02-26-180000/Macintosh HD: com.apple.backupd.SnapshotVolumeFSEventStoreUUID:
00000000  45 45 45 45 45 45 45 45 2D 36 34 30 41 2D 34 41  |EEEEEEEE-640A-4A|
00000010  41 46 2D 39 31 30 38 2D 37 36 32 36 39 41 34 46  |AF-9108-76269A4F|
00000020  32 35 41 33 00                                   |25A3.|
00000025
2018-02-26-180000/Macintosh HD: com.apple.backupd.SnapshotVolumeLastFSEventID:
00000000  31 34 36 37 30 37 39 32 38 34 00                 |1467079284.|
0000000b
2018-02-26-180000/Macintosh HD: com.apple.backupd.SnapshotVolumeUUID:
00000000  31 31 31 31 31 31 31 31 2D 43 43 30 35 2D 33 36  |11111111-CC05-36|
00000010  31 31 2D 42 30 30 30 2D 45 32 32 37 46 37 35 30  |11-B000-E227F750|
00000020  31 31 45 36 00                                   |11E6.|
00000025
2018-02-26-180000/Macintosh HD: com.apple.backupd.VolumeBytesUsed: 222954557440
2018-02-26-180000/Macintosh HD: com.apple.backupd.VolumeIsCaseSensitive: 0
2018-02-26-180000/Macintosh HD: com.apple.metadata:_kTimeMachineNewestSnapshot:
00000000  62 70 6C 69 73 74 30 30 33 41 C0 22 80 2E 00 00  |bplist003A."....|
00000010  00 08 00 00 00 00 00 00 01 01 00 00 00 00 00 00  |................|
00000020  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 11                                            |..|
00000032
2018-02-26-180000/Macintosh HD: com.apple.metadata:_kTimeMachineOldestSnapshot:
00000000  62 70 6C 69 73 74 30 30 33 41 C0 22 7F 97 80 00  |bplist003A."....|
00000010  00 08 00 00 00 00 00 00 01 01 00 00 00 00 00 00  |................|
00000020  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 11                                            |..|
00000032

The first path is the last forgotten backup, and the second is the first known to TM backup. There is definitely some stuff there, but not very clear if there is any difference.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
$ diff -U2 <(xattr -l 2018-02-26-140000/Macintosh\ HD) <(xattr -l 2018-02-26-180000/Macintosh\ HD)
--- /dev/fd/11  2018-03-02 19:30:11.000000000 -0800
+++ /dev/fd/12  2018-03-02 19:30:11.000000000 -0800
@@ -1,2 +1,10 @@
+com.apple.backupd.PreviousSnapshotVolumeName:
+00000000  4D 61 63 69 6E 74 6F 73 68 20 48 44 00           |Macintosh HD.|
+0000000d
+com.apple.backupd.PreviousSnapshotVolumeUUID:
+00000000  31 31 31 31 31 31 31 31 2D 43 43 30 35 2D 33 36  |11111111-CC05-36|
+00000010  31 31 2D 42 30 30 30 2D 45 32 32 37 46 37 35 30  |11-B000-E227F750|
+00000020  31 31 45 36 00                                   |11E6.|
+00000025
 com.apple.backupd.SnapshotVolumeFSEventStoreUUID:
 00000000  45 45 45 45 45 45 45 45 2D 36 34 30 41 2D 34 41  |EEEEEEEE-640A-4A|
@@ -5,5 +13,5 @@
 00000025
 com.apple.backupd.SnapshotVolumeLastFSEventID:
-00000000  31 34 36 36 39 36 34 35 30 31 00                 |1466964501.|
+00000000  31 34 36 37 30 37 39 32 38 34 00                 |1467079284.|
 0000000b
 com.apple.backupd.SnapshotVolumeUUID:
@@ -12,8 +20,8 @@
 00000020  31 31 45 36 00                                   |11E6.|
 00000025
-com.apple.backupd.VolumeBytesUsed: 226109964288
+com.apple.backupd.VolumeBytesUsed: 222954557440
 com.apple.backupd.VolumeIsCaseSensitive: 0
 com.apple.metadata:_kTimeMachineNewestSnapshot:
-00000000  62 70 6C 69 73 74 30 30 33 41 C0 22 64 1E 00 00  |bplist003A."d...|
+00000000  62 70 6C 69 73 74 30 30 33 41 C0 22 80 2E 00 00  |bplist003A."....|
 00000010  00 08 00 00 00 00 00 00 01 01 00 00 00 00 00 00  |................|
 00000020  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
@@ -21,5 +29,5 @@
 00000032
 com.apple.metadata:_kTimeMachineOldestSnapshot:
-00000000  62 70 6C 69 73 74 30 30 33 41 C0 22 63 F6 00 00  |bplist003A."c...|
+00000000  62 70 6C 69 73 74 30 30 33 41 C0 22 7F 97 80 00  |bplist003A."....|
 00000010  00 08 00 00 00 00 00 00 01 01 00 00 00 00 00 00  |................|
 00000020  00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|

Much better, we now see some minor changes there, but also two properties are missing from the old backup: com.apple.backupd.PreviousSnapshotVolume{Name,UUID} — those seem important. Let’s replicate them and see if that helps. First, disconnect the disk from Time Machine, just in case. Then we need to remove the very restrictive ACL permissions (sudo alone won’t help), set the attributes, and reapply the ACL:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ sudo tmutil removedestination "$( tmutil destinationinfo | sed -nE 's/ID +: +(.*)/\1/p' )"

$ cd /Volumes/backups/Backups.backupdb/mac

$ ls -led 2018-02-26-140000/Macintosh\ HD
drwxr-xr-x@ 28 root  wheel  952 26 Feb 07:42 2018-02-26-140000/Macintosh HD/
 0: group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown

$ sudo chmod -a\# 0 2018-02-26-140000/Macintosh\ HD

$ xattr -p com.apple.backupd.PreviousSnapshotVolumeUUID 2018-02-26-180000/Macintosh\ HD > ../../voluuid
$ xattr -p com.apple.backupd.PreviousSnapshotVolumeName 2018-02-26-180000/Macintosh\ HD > ../../volname

$ sudo xattr -wx com.apple.backupd.PreviousSnapshotVolumeName "$( cat ../../volname )" 2018-02-26-140000/Macintosh\ HD
$ sudo xattr -wx com.apple.backupd.PreviousSnapshotVolumeUUID "$( cat ../../voluuid )" 2018-02-26-140000/Macintosh\ HD

$ sudo chmod +a "group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown" 2018-02-26-140000/Macintosh\ HD

$ sudo tmutil setdestination /Volumes/backups
$ tmutil listbackups

Nope, the backup wasn’t in the list.

The commands to copy the extended attributes are from https://stackoverflow.com/questions/21591485/using-xattr-to-set-the-mac-osx-quarantine-property/21592337#21592337.

Note that I’m escaping # in chmod -a\# 0 in zsh. I don’t know if that is necessary in bash.

Extended attributes, part II

Later I had an idea to check the timestamped directory as well:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
$ diff -U2 <(xattr -l 2018-02-26-140000) <(xattr -l 2018-02-26-180000)
--- /dev/fd/11  2018-03-04 09:05:24.000000000 -0800
+++ /dev/fd/12  2018-03-04 09:05:24.000000000 -0800
@@ -1,12 +1,16 @@
-com.apple.backup.SnapshotNumber: 218501
+com.apple.backup.SnapshotNumber: 218601
 com.apple.backup.SnapshotVersion: 2
+com.apple.backupd.SnapshotCompletionDate:
+00000000  31 35 31 39 37 30 30 31 38 38 33 33 38 32 38 38  |1519700188338288|
+00000010  00                                               |.|
+00000011
 com.apple.backupd.SnapshotStartDate:
-00000000  31 35 31 39 36 38 35 37 34 30 34 37 39 36 34 38  |1519685740479648|
+00000000  31 35 31 39 36 39 39 38 38 37 35 39 31 37 33 33  |1519699887591733|
 00000010  00                                               |.|
 00000011
 com.apple.backupd.SnapshotState:
-00000000  31 36 00                                         |16.|
-00000003
-com.apple.backupd.SnapshotTotalBytesCopied: 85788099
+00000000  34 00                                            |4.|
+00000002
+com.apple.backupd.SnapshotTotalBytesCopied: 273602491
 com.apple.backupd.SnapshotType:
 00000000  31 00                                            |1.|

Two interesting things here: first, the visible backup has the com.apple.backup.SnapshotCompletionDate; second, its .SnapshotState is "4" whereas the invisible one’s is "16". I couldn’t find any explanation online for the .SnapshotState attribute, there is only one post about .SnapshotType however: https://d4rkw1ll0w4n6.wordpress.com/2015/02/12/timemachine-4n6/:

xattr paired with the com.apple.backupd.SnapshotType will show what type of backup it it, Monthly (1), Weekly (2), or Daily (3).

Nope, we don’t need to change it. So let’s try to replicate those two attributes, with the same basic steps:

1
2
3
4
5
6
7
8
9
10
11
$ sudo chmod -a\# 0 2018-02-26-140000
$ sudo xattr -wx com.apple.backupd.SnapshotState "34 00" 2018-02-26-140000

# for simplicity I just add 1000 to the .SnapshotStartDate
$ sudo xattr -wx com.apple.backupd.SnapshotCompletionDate "31 35 31 39 36 38 35 37 34 30 34 38 30 36 34 38 00" 2018-02-26-140000
$ sudo chmod +a "group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown" 2018-02-26-140000

$ sudo tmutil setdestination /Volumes/backups
$ tmutil listbackups
/Volumes/backups/Backups.backupdb/mac/2018-02-26-140000

Wow! It works, the backup was recognized and appeared in the list!

Small error

Fun fact (not so fun actually). First I set the date to … 37 40 36 34 38 00, enabled TM, went to the TM UI, and saw that the earliest backup was created on January 01, 1970! That’s weird. Let’s check:

1
2
3
4
5
6
7
$ xattr -l 2018-02-26-140000
com.apple.backup.SnapshotNumber: 218501
com.apple.backup.SnapshotVersion: 2
com.apple.backupd.SnapshotCompletionDate:
00000000  31 35 31 39 36 38 35 37 34 30 34 37 40 36 34 38  |151968574047@648|
00000010  00                                               |.|
00000011

Oh, the @ symbol doesn’t look right. How do you like that?! 37 39 + 01 is not 37 40, it is 38 30! With the fixed string the backup date is correct now!

Important points

So here you go, the necessary step to reassociate the mistakenly scheduled for deletion backup is setting the com.apple.backupd.Snapshot{State,CompletionDate} attributes on the timestamped directory. The observed result seems correct, however I don’t know if I reverted all the changes made by tmutil delete.

Setting the extended attributes com.apple.backupd.PreviousSnapshotVolume{Name,UUID} to the disk directory inside the timestamped directory is unnecessary and in fact the newest, proper TM backups don’t have them — so those must be the effects of running tmutil associatedisk or tmutil inheritbackup.

Here is a small script that helped me automate the process:

(tmrestore.sh) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#!/usr/bin/env bash
# https://egeek.me/

set -eou pipefail

DEFAULT_ACL="group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown"
EXPECTED_ACL=" 0: ${DEFAULT_ACL}"

function rstbkp() {
  TSDIR="${1:?need backup name}"

  TSACL="$( find . -maxdepth 1 -type d -name "$TSDIR" -print0 | xargs -0 ls -led | tail -1 )"
  [[ "$TSACL" != "$EXPECTED_ACL" ]] && { echo "$TSDIR has unexpected ACL: '$TSACL'"; exit 1; }

  TSSTATE="$( xattr -px com.apple.backupd.SnapshotState "$TSDIR" )"
  [[ "$TSSTATE" != "31 36 00" ]] && { echo "$TSDIR has unexpected .SnapshotState: '$TSSTATE'"; exit 2; }

  chmod -a\# 0 "$TSDIR"

  START_DATE="$( xattr -px com.apple.backupd.SnapshotStartDate "$TSDIR" | xxd -r -p | tr -d '\0' )"
  END_DATE_BIN="$( echo $(( START_DATE + 1000 )) | tr '\n' '\0' | xxd -p )"
  xattr -wx com.apple.backupd.SnapshotCompletionDate "$END_DATE_BIN" "$TSDIR"

  xattr -wx com.apple.backupd.SnapshotState "34 00" "$TSDIR"

  chmod +a "$DEFAULT_ACL" "$TSDIR"
}

rstbkp "$1"

Do not just run it! It worked for me, but your case may be different. I cd /Volumes/backups/Backups.backupdb/mac/, ran sudo ~/bin/tmrestore.sh 2018-02-25-200000 and then on all the previous backups using a for loop.

Conclusion

The TM behavior observed here makes sense because if you request the removal of a set of backups, the system first marks them as unusable and then deletes them slowly. The problem of course is that there is no way to say that those later backups are in fact still complete so we can still use them.

This all happened on OSX 10.11.6.

ps. Before I manually adopted all the old backups, a very strange behavior happened: after each regular backup, the “post-backup thinning stage” removed only one, the oldest, not adopted backup, even though there was enough space on the disk, and it didn’t remove any of the hourly more recent backups.

Comments