evan_tech -- flipped bits

03:11 pm, 22 Oct 06

flipped bits

% bdiff /mnt/disk1/large_file /mnt/disk2/same_large_file
diffing: [........................................]
      offset  diff
   346360387: 01011111
            : 01001111
   346360453: 10111010
            : 10111110
   346360515: 11010001
            : 11000001
   346360517: 01001100
            : 01001000

3smallishmagi

how did that happen? That's a byte offset? (yes?) in the same sector? (yes?) and some sort of CRC check is written with the sector. The sector is mostly good ( otherwise the two reads would have lots of different bytes) but some bits return a random value each read. When you just flip a few bits CRC values I guess are quite close. When a CRC error is encountered the probable response is to try again and again with a different set of values returned each time which subverts the value of a CRC check. I wonder what sort of checksum is used, and if either of the two sets of values is correct.

reply to this

evan

Yeah, those are raw byte offsets and it' sa single bit flipped in each case. I woudl think CRCs ought to be very different for single-bit flips as that's probably the most common case for data errors. I don't know much about how/whether disks use CRCs, though.

I'm trying to figure out which of the files is corrent, but unfortunately much of this data is large media which means that a flipped bit may just cause a single-pixel glitch. (On the other hand, if it's hard for me to determine when looking at the files carefully, then they're effectively both "correct".) However, in one jpeg I found it was enough to ruin the bottom of it.

reply to this

3smallishmagi	I used something ( think it might have been http://www.cgsecurity.org/photorec.html ) to recover jpegs off a CF card for someone once. If there's an offset option you could point it to near the bad area in each image and see what it finds.
reply to this

edm

Disk CRCs

There are per-sector CRCs on hard drives, which allows it to tell that the sector didn't read cleanly (which should, eg, cause the OS to retry). However, that doesn't help you if the bits were flipped somewhere other than reading off the disk media, such as in the IO bus, or chipset. (Eg, about 5 years back there were a bunch of IDE chipsets which were discovered to be very... buggy, when faced with I/O on both of the IDE channels they supported.) Or even in bad RAM, which then got flushed out to disk.

And yes, a single bit flip is the main thing that a CRC is good at catching, particularly the relatively small CRCs used on disk sectors. (I think they're CRC16.)

Ewen

reply to this

eqe	Re: Disk CRCs Nah, disk CRCs are much larger than that nowadays (although I can't find a comprehensive reference, this article has some relevance). Note that manufacturers are speccing bit error rates of 2^-50 and better; while I'm not an expert, it would seem that that implies a better ECC.
reply to this

xaosenkosmos	A random aside The JPEG corruption is a matter of getting pretty damned lucky: you've got to hit a DC coefficient or one of the EOB markers. Wikipedia article on JPEG is surprisingly helpful, and i'm wishing it had been there a year or so back, when i implemented a bunch of JPEG encoding for fun...
reply to this

eqe	The lack of a pattern is at least evidence that it's not a single intermittent bitpath in your CPU or memory. (If they were all in bit 17, for example, that would be the expectation.) Is the lossage reproducible?
reply to this

evan_tech

flipped bits

Disk CRCs

Re: Disk CRCs

A random aside