An open community 
of Macintosh users,
for Macintosh users.

FineTunedMac Dashboard widget now available! Download Here

Previous Thread
Next Thread
Print Thread
Page 2 of 2 1 2
Re: Surface scan doesn't run from Terminal?
artie505 #7681 01/18/10 11:06 AM
Joined: Aug 2009
Likes: 7
Online

Joined: Aug 2009
Likes: 7
Stuffit Expander worked when I tried V1's file.


Jon

macOS 11.7.10, iMac Retina 5K 27-inch, late 2014, 3.5 GHz Intel Core i5, 1 TB fusion drive, 16 GB RAM, Epson SureColor P600, Photoshop CC, Lightroom CC, MS Office 365
Re: Surface scan doesn't run from Terminal?
jchuzi #7682 01/18/10 11:39 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
Thanks, Jon.


The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Re: Surface scan doesn't run from Terminal?
artie505 #7690 01/18/10 04:53 PM
Joined: Sep 2009
Offline

Joined: Sep 2009
Originally Posted By: artie505
I don't mind its not stopping when it finds an error...morbid curiosity, but have I misunderstood the command?

Will it report an I/O error and keep going, or will it keep going without reporting what it finds?

I tend to read and believe man pages, until i can prove they're wrong. If you have a known bad disk which can be used to test dd with various options, and can prove that the section i quoted in green is wrong (my 2nd post on page 1), then i'd be extremely interested (and quite surprised).

[i.e., the answer (until proven otherwise) is: yes, dd reports the error instantly AND it continues to chug along... at least that's my interpretation of the words written on the man page.]

Last edited by Hal Itosis; 01/18/10 05:11 PM.
Re: Surface scan doesn't run from Terminal?
Hal Itosis #7694 01/18/10 06:42 PM
Joined: Aug 2009
Offline

Joined: Aug 2009
Originally Posted By: Hal Itosis
Originally Posted By: Virtual1
You NEED to remove the "conv=noerr", otherwise dd will not stop on io error.

Hence ctrl-C (if need be).


I think dd behaves differently than you are expecting. If it hits an error that stumps it (hangs dd, either temporarily or permanently) then ctrl-c won't save you. It'll be blocking in the kernel until it finishes the block it was reading. THEN ctrl-c may stop it. But it's been my experience that unless you're using a very small block size (512 etc) then you're in for an awful long wait, as each disc block (512) may take 1-2 minutes to pass. If your dd block size is 1mb, it could be 20 minutes or more before it releases with ctrl-c. Some block errors never complete and block until killed.

The reason to not use conv=noerror is that if it gets an error (that passes quickly, say one bad disk block in the entire dd block) then it will display the io error to console, but with conv=noerror, it WON'T STOP, and will NOT return an error code when dd finally finishes. This isn't usually how you want it to behave when looking for errors, unless you are trying to count how many you have. (I usually consider one to be more than enough!) If you omit that conv, dd will stop at its first bad block encountered, and will return an error code.

(fyi, dd pads unreadable blocks with zeros if you use conv=noerror)


I work for the Department of Redundancy Department
Re: Surface scan doesn't run from Terminal?
artie505 #7701 01/18/10 10:02 PM
Joined: Aug 2009
Likes: 1
Moderator
Offline
Moderator

Joined: Aug 2009
Likes: 1
Originally Posted By: artie505
I haven't got Stuffit on my deuced Mac(hina); [...] which Stuffit app do I need to d/l?

StuffIt Expander 2010 - 14.0.1. Alternatively The Unarchiver will work too, and weighs in at about 1/10th the size of SE.


alternaut moderator
Re: Surface scan doesn't run from Terminal?
alternaut #7703 01/19/10 12:56 AM
Joined: Aug 2009
Offline

Joined: Aug 2009
ok try this disk image instead. Sorry for me stuffing something is as easy as cmd-S. Making a disk image is entirely more complicated than should be necessary. (I have deleted the previous .SIT file)

Finder needs a nice contextual menu item for "create disk image" that would DMG a folder, or if it was a file, made a folder by the same name, put it in the folder, and DMG'd that. (or is there something like that available you know of?)


I work for the Department of Redundancy Department
Re: Surface scan doesn't run from Terminal?
Hal Itosis #7704 01/19/10 01:04 AM
Joined: Aug 2009
Offline

Joined: Aug 2009
Originally Posted By: Hal Itosis
I tend to read and believe man pages, until i can prove they're wrong.

It's been my experience that the man pages are usually very accurate, but are often incomplete. Rsync for example is a very large, detailed, accurate page, but is outright missing several important flags.


I work for the Department of Redundancy Department
Re: Surface scan doesn't run from Terminal?
Virtual1 #7709 01/19/10 05:32 AM
Joined: Sep 2009
Offline

Joined: Sep 2009
Originally Posted By: Virtual1
I think dd behaves differently than you are expecting. If it hits an error that stumps it (hangs dd, either temporarily or permanently) then ctrl-c won't save you. It'll be blocking in the kernel until it finishes the block it was reading. THEN ctrl-c may stop it. But it's been my experience that unless you're using a very small block size (512 etc) then you're in for an awful long wait, as each disc block (512) may take 1-2 minutes to pass. If your dd block size is 1mb, it could be 20 minutes or more before it releases with ctrl-c. Some block errors never complete and block until killed.

Yes, i would expect that. I mean, bad blocks can make just logging in or opening a Finder window take 20 minutes. Such pain is not an exclusive feature of Terminal or dd under those circumstances.

Originally Posted By: Virtual1
The reason to not use conv=noerror is that if it gets an error (that passes quickly, say one bad disk block in the entire dd block) then it will display the io error to console, but with conv=noerror, it WON'T STOP,

All good (for those who wanted that to happen).

Originally Posted By: Virtual1
and will NOT return an error code when dd finally finishes. This isn't usually how you want it to behave when looking for errors, unless you are trying to count how many you have. (I usually consider one to be more than enough!)

I think there's where our preferences differ. In your context (at a work place), you're mainly interested in strict PASS/FAIL type of analysis... so you can swiftly move on to what else needs doing. (i.e., probably tackle another broken machine that's been waiting for your attention).

We folks at home are focused on our sole problem machine (or device), and are more inclined to want to find more details about how badly the disk's condition was degraded... even if it takes a little time. And if it turns out to be a total of 1 or 2 bad blocks... then maybe a quick full erase [to remap] could make the disk last another day or 3, while we wait for FedEx to deliver the replacement. I remember my first one was a doozy... the main boot partition was hosed with hundreds of bad blocks. I was able to determine the count thanks to ditto -V, which i used to salvage what i could (by copying it off to an external). The copying ditto did took around 28 hours (and that was just the /Users folder, IIRC). Bad blocks are a bitch.



Originally Posted By: Virtual1
(fyi, dd pads unreadable blocks with zeros if you use conv=noerror)

True, but /dev/null won't mind in the least "bit". wink


Originally Posted By: Virtual1
Finder needs a nice contextual menu item for "create disk image" that would DMG a folder, or if it was a file, made a folder by the same name, put it in the folder, and DMG'd that. (or is there something like that available you know of?)

Yes, there is Compress in Finder's contextual menu. [it makes a zip archive.] We can actually tweak a few of its "features" which get saved into a com.apple.archiveutility.plist in our home library (so we're not monkeying with the system really).

One simple way to access those prefs is to create a symbolic link to it:
Code:
ln -s /System/Library/CoreServices/Archive\ Utility.app/Contents/Resources/Archives.prefPane ~/Library/PreferencePanes/

That command puts a symlink in our ~/Library/PreferencePanes/ folder, so when we open System Prefs, a new prefpane will appear there. See this article for more info (and pictures): New hidden PrefPanes in Leopard

Last edited by Hal Itosis; 01/19/10 05:56 AM.
Re: Surface scan doesn't run from Terminal?
alternaut #7710 01/19/10 06:38 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
V1's dmg makes it moot, but thanks for the alternative to Stuffit.


The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Re: Surface scan doesn't run from Terminal?
Virtual1 #7711 01/19/10 06:40 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
> ok try this disk image instead.

Got it!

Thanks. (As I said, I'll get to it and let you know how it works for me when I get a chance.)


The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Re: Surface scan doesn't run from Terminal?
Virtual1 #7713 01/19/10 07:20 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
This thread has turned pretty educational; now let's see if I actually understand what you and Hal have been discussing...

> and will NOT return an error code when dd finally finishes.

> (fyi, dd pads unreadable blocks with zeros if you use conv=noerror)

> it will display the io error to console

I think that-all means that:

1.
Code:
Artie-s-Computer-4:~ artie$ sudo dd if=/dev/rdisk0s2 of=/dev/null conv=noerror bs=10240000
Password:
2610+1 records in
2610+1 records out
26730483712 bytes transferred in 481.737618 secs (55487640 bytes/sec)

does not necessarily mean that my partition has no bad blocks?

2. "dd's" padding of unreadable blocks with zeros will result in "records in" = "records out" even if it finds bad blocks?

3. after "dd" finishes its run I have to go to Console to find out if it actually found any bad blocks? (If that's the case, then still none. smile )

Thanks to both you and Hal for sharing your expertise.


The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Re: Surface scan doesn't run from Terminal?
Hal Itosis #7722 01/19/10 02:47 PM
Joined: Aug 2009
Offline

Joined: Aug 2009
Oh I thought that compress CMM was added by stuffit, and I didn't realize it would do folders too. Does the zip it makes have good backward and sideways compatibility, or will I have problems with someone trying to unzip it on say, windows xp?

But I'd say anyone that wants to know "how bad" the errors on the hard drive are, isn't too interested in the safety of their information wink



I work for the Department of Redundancy Department
Re: Surface scan doesn't run from Terminal?
artie505 #7723 01/19/10 02:58 PM
Joined: Aug 2009
Offline

Joined: Aug 2009
DD should still send io error notices to stderr which should display for you in the same way the number of blocks copied displays. you should also get an io error in the System log in console.

without the conv, you may see something like this:

dd: /dev/disk0: Input/output error
5+0 records in
5+0 records out
5242880 bytes transferred in 41.363104 secs (126753 bytes/sec)

iirc, if you get say one io error, the records in will not count the bad block, so records in will be smaller than records out. (since it's outputting zeros) BTW, 'records' refers to dd blocks, so 1mb blocksize means each record is 1mb.

That also shows the very long delay that you get from encountering a single io error. If you conv=noerror and it runs into a pack of say, 500 bad blocks, you're in for a long wait. Not all errors take a long time to resolve, but most do.

The error in console will be the same as dd: "/dev/disk0: Input/output error"

does not necessarily mean that my partition has no bad blocks?

Yes and no. Since you didn't see any io error notices, there were no bad blocks in the area you scanned. But if you scanned a diskXsY instead of a diskX, then you skipped other partitions, drivers, partition map, and boot block, so I suppose there could be an error in the skipped space. (tho statistically unlikely)

I also consider "slow blocks" to be errors, as they almost always foretell imminent failure. If you just use the one-liner to read the entire HD, it may encounter many slow blocks on the way but will not inform you and will complete without complaint. The surface scan script intentionally scans in small chunks, timing them, and displays notice if it finds a range of blocks that took too long, making it a much better test.


I work for the Department of Redundancy Department
Re: Surface scan doesn't run from Terminal?
artie505 #7731 01/19/10 04:35 PM
Joined: Sep 2009
Offline

Joined: Sep 2009
Originally Posted By: artie505
I think that-all means that:
1.
Code:
Artie-s-Computer-4:~ artie$ sudo dd if=/dev/rdisk0s2 of=/dev/null conv=noerror bs=10240000
Password:
2610+1 records in
2610+1 records out
26730483712 bytes transferred in 481.737618 secs (55487640 bytes/sec)

does not necessarily mean that my partition has no bad blocks?

I don't see any error message, so that means dd did not encounter one. [then again, dd wasn't designed to be a "bad block detector" -- so i don't know what its threshold for defining an error is. maybe a disk could still be wonky and return good data very slowly or something. idunno.]


Originally Posted By: artie505
2. "dd's" padding of unreadable blocks with zeros will result in "records in" = "records out" even if it finds bad blocks?

Right. (AFAIK). <-- looks like we're both wrong there. See Virtual1's answer.


Originally Posted By: artie505
3. after "dd" finishes its run I have to go to Console to find out if it actually found any bad blocks? (If that's the case, then still none. smile )

Not right. Here again...
Originally Posted By: man dd
When an input error occurs, a diagnostic message followed by the current input and output block counts will be written to the standard error output in the same format as the standard completion message.

When running a command using a *terminal* device, the phrase "standard error output" means the same screen into which you typed the command. (Though redirection to other file descriptors is certainly possible, standard error is always echoed to the screen [when there is one]).

What's really needed here is someone with a known bad disk to do a few runs and then *show* us the results.

Last edited by Hal Itosis; 01/19/10 05:23 PM. Reason: strikeout answer #2
Re: Surface scan doesn't run from Terminal?
Virtual1 #7732 01/19/10 05:02 PM
Joined: Sep 2009
Offline

Joined: Sep 2009
Originally Posted By: Virtual1
Oh I thought that compress CMM was added by stuffit, and I didn't realize it would do folders too. Does the zip it makes have good backward and sideways compatibility, or will I have problems with someone trying to unzip it on say, windows xp?

I haven't run xp since maybe 2003, and i never did much with it (other than downloading service packs). The only "unusual" thing about OSX zip archives is that they *may* include resource fork and other Mac extended attributes, if the source files had any such content/metadata. But a plain text file (such as your surface_scan.command) should be cleanly extracted anywhere.



Originally Posted By: Virtual1
But I'd say anyone that wants to know "how bad" the errors on the hard drive are, isn't too interested in the safety of their information wink

Oh, I totally understand where you're coming from. I definitely wouldn't want a single bad block on some hard drive in the next jet i fly on, for example.

But presumably our personal data is backed up... so measuring the degree of damage is just to satisfy our curiosity and desire for more complete knowledge about the extent of a problem. If Quantum disks average 13 bad blocks after 5 years of usage... and Western Digital drives average 47 bad blocks in the same time period... guess which one i'm gonna buy next time i need a new disk? (if all we ever measure is *one* error, then maybe we miss out on a useful part of the picture).


Last edited by Hal Itosis; 01/19/10 05:26 PM.
Re: Surface scan doesn't run from Terminal?
Hal Itosis #7778 01/21/10 08:27 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
Thanks to both you and V1 for explaining some confusing stuff.

> What's really needed here is someone with a known bad disk to do a few runs and then *show* us the results.

Not me, thank goodness. smile


The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Re: Surface scan doesn't run from Terminal?
Virtual1 #7919 01/25/10 09:53 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
Does your "Surface Scanner" generate a time record, as does Terminal, when it's finished its work?

If I'm going to compare it with TT Deluxe>Surface Scan I'll need to know whether I've got to sit here with my eye on my deuced Mac(hina) during both operations in order to get accurate time readings.


The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Re: Surface scan doesn't run from Terminal?
artie505 #7936 01/26/10 03:43 AM
Joined: Aug 2009
Offline

Joined: Aug 2009
Does your "Surface Scanner" generate a time record, as does Terminal, when it's finished its work?

I've considered having it keep samples and generate a basic graph when it's done to see if there are any (downward) spikes encountered during the scan. I'm also just basically curious to see how scan speeds vary across the surface of a disk - I assume there is a change as scanning farther from the center should yield faster data transfer rate.

For now it just has a hardcoded time limit that if the particular pass doesn't complete in time it stops and errors out.

I have a much more sophisticated script that runs here on several machines that have multiple attached drives, that is set to actually monitor speeds and alert me when they drop to 40% of expected or less. I've been getting emails lately though and I'm not sure why...

2010-01-25 02:20:31 WD2: disk1 "(RAID_slice)" Scan speed 19 below acceptable (norm 67)
2010-01-25 02:20:31 WD2: FAIL: dd if=/dev/rdisk1 of=/dev/null count=142 iseek=371472 bs=1048576 > /dev/null

It's always that same drive too. And the two of them are scanning over a dozen total drives. Makes me suspicious of that one. I think what's happening though is it's hitting at the same time as a cronned rsync from the boot drive, and I think that's the "preferred slice" for the mirror to read from, and rsync building its send list is what's causing things to slow down so much.


I work for the Department of Redundancy Department
Re: Surface scan doesn't run from Terminal?
Virtual1 #7940 01/26/10 06:26 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
OK... As I said, I'll run it against TT Deluxe>Surface Scan for comparative purposes when I've got a coupl'a free hours.


The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Re: Surface scan doesn't run from Terminal?
Virtual1 #8857 03/15/10 09:38 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
Your dmg has been staring back at me from my desktop for the last 7 weeks, but it took the current discussion about cluttered desktops (not that mine is all that bad) to move me to decide to get it out of my sight.

After TechTool Deluxe completed a surface scan of my 25Gb volume in 06:55 I launched surface_scan.command.2010.01.13.A.dmg to run a comparison test...

[Bug False Alarm Report]

After getting to this point with your script

Code:
Available Block Devices:
-------------------------------------------------------------
1) (DEVICE)               *232.9 Gi   disk0
2) HD                      24.9 Gi    disk0s2
3) HD2                     207.5 Gi   disk0s3
4) (DEVICE)               *9.1 Mi     disk1
5) surface_scan.command.2010.01.13.A9.1 Mi     disk1s2

Select device by number: 


I discovered that both options 2 and 3 defaulted to scanning disk0.

Oops!

[/Bug False Alarm Report]

With no other options available I just timed how long it took your script to scan 25Gb (06:47...8 sec faster than TTD!), but I've absolutely no idea if the two results are comparable.

Thanks, again, for the script, and sorry to have taken so long to get back to you.

Last edited by artie505; 03/16/10 05:50 AM. Reason: Bug-->False Alarm

The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Re: Surface scan doesn't run from Terminal?
artie505 #8866 03/15/10 06:19 PM
Joined: Sep 2009
Offline

Joined: Sep 2009
Originally Posted By: artie505
After getting to this point with your script

Code:
Available Block Devices:
-------------------------------------------------------------
1) (DEVICE)               *232.9 Gi   disk0
2) HD                      24.9 Gi    disk0s2
3) HD2                     207.5 Gi   disk0s3
4) (DEVICE)               *9.1 Mi     disk1
5) surface_scan.command.2010.01.13.A9.1 Mi     disk1s2

Select device by number: 

I discovered that both options 2 and 3 defaulted to scanning disk0.

According to the table, HD and HD2 are both slices on the same device (disk0). Therefore they share the same (master) partition map... which is the first (some number of) blocks on "disk0". I guess lumping the two "HDs" makes sense from the viewpoint that: if one is bad, then you'll still need to replace both (when a new HD is installed).

But, i should let V1 speak for his own product.

--

EDIT: this forum software doesn't let us put styles (colors, bold, italics, etc.) inside code tags. What a drag.

EDIT#2: and that first edit took almost 5 minutes to reload [in contrast to places like macosxhints or macrumors, where the pages fly open... even with hundreds of members logged in (and perhaps thousands of guests lurking), while dozens of users are posting almost simultaneously.]

Last edited by Hal Itosis; 03/15/10 06:32 PM.
Re: Surface scan doesn't run from Terminal?
Hal Itosis #8867 03/15/10 06:34 PM
Joined: Aug 2009
Offline

Joined: Aug 2009
It's listing all the available character devices for scanning. Each physical disk normally shows up at least twice, once for the entire device, ("disk0") and once for each formatted HFS+ partition on it. ("disk0s2", "disk0s5", etc)

This allows you to surface scan an unformatted disk, as well as to scan a specific disk by volume name if you don't know the device number.

In any event, no matter which "disk0" you select, it will scan the entire disk0 device.

In the past it was set up to only scan the specific partition if you selected one, but I decided that was silly since you really need to know if the entire drive the volume is on is bad, not just to see if there's an error on the specific partition. (since an error ANYWHERE on the drive can cause it to beachball when accessing ANY partition on the drive) My philosophy is that a drive with even one slow block or one bad block needs to be replaced immediately.


I work for the Department of Redundancy Department
Re: Surface scan doesn't run from Terminal?
Hal Itosis #8868 03/15/10 06:40 PM
Joined: Aug 2009
Offline

Joined: Aug 2009
I also discovered a pleasant surprise the other day -- apple's hardware test disks, when you select to do a full disk scan, will check for slow blocks in addition to bad blocks. So for my purposes, that's probably as acceptable of a check as my script.


I work for the Department of Redundancy Department
Re: Surface scan doesn't run from Terminal?
Hal Itosis #8871 03/15/10 06:57 PM
Joined: Aug 2009
cyn Online
Administrator
Online
Administrator

Joined: Aug 2009
Regarding your edits, especially the 2nd, please post such comments in FineTunedMac Feedback. That way threads like this won't get derailed with off-topic discussion and you increase the likelihood tacit will see your observations/complaints.


FineTunedMac Forums Admin
Re: Surface scan doesn't run from Terminal?
Virtual1 #8885 03/16/10 07:05 AM
Joined: Aug 2009
Likes: 15
OP Online

Joined: Aug 2009
Likes: 15
The instant I saw Hal's post I realized that I was so hell-bent on comparing scan times for my HD partition that I breezed right past the "DEVICE" (not "device," mind you, but "DEVICE") thing. (See my sig for more.)

My bad. Sorry.

> In the past it was set up to only scan the specific partition if you selected one, but I decided that was silly since you really need to know if the entire drive the volume is on is bad, not just to see if there's an error on the specific partition. (since an error ANYWHERE on the drive can cause it to beachball when accessing ANY partition on the drive)

...which makes you wonder why AHT allows you to scan partitions one-at-a-time?


The new Great Equalizer is the SEND button.

In Memory of Harv: Those who can make you believe absurdities can make you commit atrocities. ~Voltaire
Page 2 of 2 1 2

Moderated by  alternaut, dkmarsh, joemikeb 

Link Copied to Clipboard
Powered by UBB.threads™ PHP Forum Software 7.7.4
(Release build 20200307)
Responsive Width:

PHP: 7.4.33 Page Time: 0.053s Queries: 65 (0.044s) Memory: 0.7245 MB (Peak: 0.9205 MB) Data Comp: Zlib Server Time: 2024-03-28 21:50:22 UTC
Valid HTML 5 and Valid CSS