While I was testing the cloning and resizing of the user data file system, I came across one drive which FreeBSD was inexplicably unable to access again after the PS3 had mounted it. I knew the file system was good, because the PS3 was still able to use it and I could mount it read-only under Linux, but any attempts to access it via FreeBSD resulted in a UFS2 superblock error:
UFS2 superblock failed: fs->fs_fsbtodb (0) != ILOG2(fs->fs_fsize / sectorsize) (3)
Testing the drive further, I confirmed the same thing happened when the drive was natively formatted by the PS3; FreeBSD was still unable to access it. So this wasn't an issue being introduced by cloning or resizing the file system.
What was going wrong? What was different about this drive?
Meet Advanced Format 512-byte Emulation Disk Drives
It turns out that I had unknowingly been using a so called 512e disk drive; an Advanced Format drive that presents as a traditional 512 bytes per sector through emulation. Such a drive appears to the device and Operating System with 512 bytes logical sectors and 4096 bytes physical sectors.
The following acronyms are used throughout this article to refer to the different sector sizes
| Name | Description | Logical Sector Size | Physical Sector Size |
|---|---|---|---|
| 512n | Native 512 byte sectors Almost every "modern-era" disk drive prior to 2010 | 512 | 512 |
| 512e | Emulated 512 byte sectors | 512 | 4096 |
| 4Kn | Native 4096 byte sectors | 4096 | 4096 |
One of the purposes of 512e is backwards compatibility, but despite this it still appears that there needs to be a level of understanding of such drives at the Operating System level - given that they present as 4096 bytes physical sectors - otherwise unexpected things may happen. There is a quote from Microsoft on support for 512e drives in OS versions prior to Windows Vista which would support this understanding:
Windows XP, Windows Server 2003, and Windows Server 2003 R2 do not support 512e or 4Kn media. While the system may boot up and be able to operate minimally, there may be unknown scenarios of functionality issues, data loss, or sub-optimal performance.
I suspect that is what is happening in this case; the PS3 OS lacks proper support for 512e disk drives and is interpreting the 4096 bytes physical sector value that the drive is presenting in a way which affects how the UFS2 file system is created and accessed.
Investigating what the PS3 is doing by using NetBSD
In trying to understand what was going wrong, I was tipped off to using NetBSD rather than FreeBSD.
The PS3 UFS2 file system is big-endian, and FreeBSD's UFS2 implementation is endian dependant (ie. it can only interact with UFS2 file systems that match the endianness of the host CPU). Up to this point my testing had been done by emulating a PowerPC 64bit computer, as this CPU is big-endian and relatively easily emulated with solutions like QEMU alongside current versions of FreeBSD (although note emulating legacy versions of FreeBSD is much more difficult).
NetBSD's UFS2 implementation is endian independent, and this means that you can access either big-endian or little-endian file systems regardless of your CPU. So with NetBSD you can virtualise - rather than emulate - an AMD64 version of the OS, and this also means you can more easily access legacy versions of NetBSD that would be era appropriate to when the PS3 OS was created.
This may make NetBSD appear to be the perfect candidate for general access to PS3 disk drives, however attempts to mount the drive should be undertaken with caution.
While BSD UFS (FFS) implementations may have the same origin, their development over the years has diverged significantly and there is not a guarantee of compatibility between file systems created under different BSDs. Expect a separate article on the subject of file system write access from BSD.
dumpfs superblock comparison
Using NetBSD 5.2.3, it was possible to examine the file system superblock with dumpfs -s. This made it easy to compare the superblock of the 512e drive (which FreeBSD reports as having a superblock error) with the superblock of a 512n drive (which FreeBSD is able to use).
This comparison showed that the fsbtodb property on the 512e drive was 0 with the same property on the 512n drive being 3. The original FreeBSD error had made reference to fs_fsbtodb, so it makes sense that this would come up as a difference.
fsbtodb seems to be a file system level indicator of the logical sector size of the drive that the file system was created on. When this is set to 0 it indicates the sector size is 4096 bytes (with the left-hand side of the image indicating that the superblock is in sector 16). With it set to 3 it indicates a sector size of 512 bytes (right-hand side showing the superblock being in sector 128).
You can demonstrate that the superblock is in the same physical location of the 512e drive regardless of whether it is addressed in 512 byte sectors or 4096 byte sectors by using dd (it starts at byte 65536 in both cases).
dd if=/dev/ld1d bs=4096 skip=16 count=1 | hexdump -C
00000000 00 00 00 00 00 00 00 00 00 00 00 14 00 00 00 18 |................|
00000010 00 00 00 1c 00 00 09 24 00 00 00 00 00 00 00 00 |.......$........|
00000020 00 00 00 00 00 00 00 00 00 00 78 10 00 00 00 06 |..........x.....|
00000030 00 00 40 00 00 00 10 00 00 00 00 04 00 00 00 08 |..@.............|
00000040 00 00 00 00 00 00 00 00 ff ff c0 00 ff ff f0 00 |................|
00000050 00 00 00 0e 00 00 00 0c 00 00 00 08 00 00 08 00 |................|
00000060 00 00 00 02 00 00 00 00 00 00 10 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 08 00 00 00 00 40 00 00 00 00 |...........@....|
00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000090 00 00 00 00 f5 35 bd 07 00 00 00 00 00 00 10 00 |.....5..........|
000000a0 00 00 40 00 00 00 00 00 00 00 00 00 00 00 00 00 |..@.............|
000000b0 00 00 00 00 00 00 00 00 00 00 90 80 00 01 20 ac |.............. .|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 00 01 00 80 2f 63 65 6c 6c 5f 6d 77 5f 63 66 73 |..../cell_mw_cfs|
000000e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
dd if=/dev/ld1d bs=512 skip=128 count=1 | hexdump -C
00000000 00 00 00 00 00 00 00 00 00 00 00 14 00 00 00 18 |................|
00000010 00 00 00 1c 00 00 09 24 00 00 00 00 00 00 00 00 |.......$........|
00000020 00 00 00 00 00 00 00 00 00 00 78 10 00 00 00 06 |..........x.....|
00000030 00 00 40 00 00 00 10 00 00 00 00 04 00 00 00 08 |..@.............|
00000040 00 00 00 00 00 00 00 00 ff ff c0 00 ff ff f0 00 |................|
00000050 00 00 00 0e 00 00 00 0c 00 00 00 08 00 00 08 00 |................|
00000060 00 00 00 02 00 00 00 00 00 00 10 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 08 00 00 00 00 40 00 00 00 00 |...........@....|
00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000090 00 00 00 00 f5 35 bd 07 00 00 00 00 00 00 10 00 |.....5..........|
000000a0 00 00 40 00 00 00 00 00 00 00 00 00 00 00 00 00 |..@.............|
000000b0 00 00 00 00 00 00 00 00 00 00 90 80 00 01 20 ac |.............. .|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 00 01 00 80 2f 63 65 6c 6c 5f 6d 77 5f 63 66 73 |..../cell_mw_cfs|
000000e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
A closer look at the file system
While dumpfs and fsck_ffs under NetBSD can read information about the file system, it still isn't possible to mount it. While it does technically mount, trying to browse the directory results in an error.
# mount -o rdonly -v /dev/ld1d /cell_mw_cfs/
exec: mount_ffs -o rdonly /dev/ld1d /cell_mw_cfs
/dev/ld1d on /cell_mw_cfs type ffs (read-only, local, fsid: 0x1313/0x78b, reads: sync 1 async 0, writes: sync 0 async 0)
# ls /cell_mw_cfs/
ls: /cell_mw_cfs/: Not a directory
# df /cell_mw_cfs
df: /cell_mw_cfs: Bad file descriptor
It is, however, possible to browse the file systems inode data using fsdb. This gives some insights into differences between the file system created on the 512e drive compared with the 512n one. To demonstrate this I can compare a standard file which exists on the PS3 drive: /home/00000001/community/PTL.TMP
fsdb -n -f /dev/ld1d
A quick fsdb guide
It is relatively easy to navigate through inodes using fsdb. When starting the application it should automatically select inode for the root of the file system.
Similar to browsing a file system, you can use ls to display inodes contained within the current one.
fsdb (inum: 2)> ls
slot 0 ino 2 reclen 12: directory, `.'
slot 1 ino 2 reclen 12: directory, `..'
slot 2 ino 73984 reclen 24: directory, `crash_report'
slot 3 ino 110976 reclen 12: directory, `vm'
slot 4 ino 36992 reclen 12: directory, `mms'
slot 5 ino 147968 reclen 12: directory, `vsh'
slot 6 ino 36994 reclen 12: directory, `drm'
slot 7 ino 36995 reclen 12: directory, `tmp'
slot 8 ino 147975 reclen 16: directory, `data'
slot 9 ino 73988 reclen 388: directory, `home'
You can then use inode [number] to move to that inode. To move to inode 73988 for /home:
fsdb (inum: 2)> inode 73988
current inode: directory
I=73988 MODE=40700 SIZE=512
MTIME=Jan 1 00:00:15 2024 [0 nsec]
CTIME=Jan 1 00:00:15 2024 [0 nsec]
ATIME=Jan 1 00:00:20 2024 [0 nsec]
OWNER=root GRP=wheel LINKCNT=3 FLAGS=0x0 BLKCNT=0x1 GEN=0x630c4bba
You can then use blks to display the file system blocks used by this inode:
fsdb (inum: 73988)> blks
I=73988 1 blocks
Direct blocks:
0: 166516
slot 3 ino 111013 reclen 472: regular, `PTL.TMP'
fsdb (inum: 110997)> inode 111013
current inode: regular file
I=111013 MODE=100600 SIZE=24048
MTIME=Jan 1 00:01:06 2024 [0 nsec]
CTIME=Jan 1 00:01:06 2024 [0 nsec]
ATIME=Jan 1 00:01:06 2024 [0 nsec]
OWNER=root GRP=wheel LINKCNT=1 FLAGS=0x0 BLKCNT=0x6 GEN=0xb905c1ca
fsdb (inum: 111013)> blks
I=111013 6 blocks
Direct blocks:
0: 248956 249022
slot 3 ino 111013 reclen 472: regular, `PTL.TMP'
fsdb (inum: 110997)> inode 111013
current inode: regular file
I=111013 MODE=100600 SIZE=24048
MTIME=Jan 1 00:01:27 2024 [0 nsec]
CTIME=Jan 1 00:01:27 2024 [0 nsec]
ATIME=Jan 1 00:01:27 2024 [0 nsec]
OWNER=root GRP=wheel LINKCNT=1 FLAGS=0x0 BLKCNT=0x30 GEN=0xb905c1ca
fsdb (inum: 111013)> blks
I=111013 48 blocks
Direct blocks:
0: 251544 248986
To try to understand and give some context to this:
This is somewhat complicated, and difficult to find detailed explanations on, so if any information here has been misunderstood please do let me know via GitHub.
- The file is the same size on both file systems (24048 bytes)
- On the 512n drive, the file occupies 48 blocks in
BLKCNT(512 bytes x 48 = 24576 bytes) - On the 512e drive, the file occupies 6 blocks in
BLKCNT(4096 bytes x 6 = 24576 bytes) - The fragment (or frag) size for the UFS2 file system is 4096 bytes on both drives
This is the smallest amount of space a file will occupy on the file system - The block size for the UFS2 file system is 16384 bytes on both drives
The file occupies 2 file system blocks on both drives (displayed byblks)
Further broken down, the file occupies 6 fragments with the file system fiting 4 fragments per block
This does make for confusing reading as it seems that the term blocks is used to refer to two different things (ie. on the 512n drive this is saying that the file occupies 48 blocks, and yet it only uses 2 blocks when running blks).
My interpretation of this (as shown in the workings above) is that the inode BLKCNT is the number of logical sectors used on the physical drive, with the blocks referred to by the blks command being the the file system block size (ie. the file system block size is 16384 bytes, and so a file that is 24048 bytes would occupy two of these blocks).
If you were the run fsck_ffs on the 512e file system, it would highlight these exact differences as inconsistencies.
INCORRECT BLOCK COUNT I=111013 (6 should be 48)
CORRECT? no
In summary, the inode block (sector) count for every file and directory is different when compared with the 512n drive (and reported as an inconsistency by fsck_ffs in NetBSD); but other aspects of the file system appear to be the same.
Example of how this might cause problems
You can use the du application to demonstrate how an incorrect block count in the inode might have an effect on application, as in this case it causes it to miscalculate file sizes.
These examples are shown from Linux Mint rather than a BSD, as the Linux UFS module can read the 512e filesystem without having to modify the superblock. However the same output would be seen from running du on either FreeBSD or NetBSD.
root@mint:/home/andrew/ps3/storage/hdd/dev_hdd0# du -h
512 ./crash_report/game
512 ./crash_report/vsh
512 ./crash_report/bdp
2.0K ./crash_report
61M ./vm
512 ./home/00000001/savedata
512 ./home/00000001/http
5.5K ./home/00000001/community
1.0K ./home/00000001/etc
9.0K ./home/00000001
9.5K ./home
63M .
Two issues are immediately apparently. Firstly, the smallest file is shown as 512. We know this is wrong, because the fragment size of the file system is 4096 bytes (ie. this is smallest space a file can occupy). Secondly, it shows total space used as 63M which we also know is incorrect. It is undercounting file sizes by a factor of 8 (ie. 512 x 8 would become 4096, 63M x 8 would become 504M).
The df application does not have the same issue, and running that against the file system shows the correct sizes:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ps3hdd2 1.5G 503M 848M 38% /home/andrew/ps3/storage/hdd/dev_hdd0
To compare du output with a 512n drive, you can see that the values are what would be expected with the smallest file reported as 4.0K and the total file size 503M.
root@mint:/home/andrew/ps3/storage/hdd/dev_hdd0# du -h
4.0K ./crash_report/game
4.0K ./crash_report/vsh
4.0K ./crash_report/bdp
16K ./crash_report
481M ./vm
4.0K ./home/00000001/savedata
4.0K ./home/00000001/http
44K ./home/00000001/community
8.0K ./home/00000001/etc
72K ./home/00000001
76K ./home
503M .
Editing the superblock to enable access
Aside from the differing inode block counts, the actual structure of the file systems appear to be identical between the 512e drive and the 512n drive.
It would seem that the only factor preventing access to the drive via FreeBSD or NetBSD is that the fsbtodb property in the UFS2 superblock is set to 0, it's therefore reasonable to think that if this value were to be set to 3 (which both FreeBSD and NetBSD are expecting for a 512e drive) then they would be able to access the file system.
It is possible to change this value by directly altering the first superblock in the file system.
printf '\x03' | dd of=/dev/vtbd1 bs=1 seek=65639 count=1 conv=notrunc
printf '\x03' | dd of=/dev/ld1d bs=1 seek=65639 count=1 conv=notrunc
With this changed both FreeBSD and NetBSD can now access the file system without error.
If you do decide to mount the file system with read/write access then you should use FreeBSD. Doing this from NetBSD will remove the soft-updates flag from the superblock (even if you mount using -o softdep), and with this flag removed the PS3 will not mount the drive. NetBSD seems to lack facility to restore this flag if it is removed, so you must use FreeBSD's tunefs to restore it.
tunefs -n enable
If you are mounting read only (-o rdonly), then you can use either NetBSD or FreeBSD.
You should consider that any files created while the file system is mounted will have the opposite problem of files created by the PS3. The inode block count for these files will be based on 512 byte sector sizes, and it is unclear how the PS3 may react to block counts that are larger than it is expecting. For example, if I were to create the file demonstrated above from FreeBSD then the inode block count would be 48 (512 bytes x 48 = 24,576 bytes). The PS3 would then interpret this based on 4096 bytes (so 4096 bytes x 48) which would be 196,608 bytes - some eight times larger. In theory it should know that is wrong, as the inode file size would still be recorded as 24,048 bytes, but it does open the system to the potential of unexpected behaviour, especially if the PS3 equivalent of fsck cannot correct the counts.
When the drive is next mounted by the PS3 then the fsbtodb value is reset to 0.
Is this an actual a problem, and should 512e drives be avoided
The PS3 does appear to be misbehaving in how it is creating the UFS2 file system when a 512e drive is being used.
The only way I can replicate such a file system (ie. one with fsbtodb set to 0) is when passing the -S 4096 parameter to the newfs command. The man page has the following to say about this setting:
The following options override the standard sizes for the disk geome-
try. Their default values are taken from the disk label. Changing
these defaults is useful only when using newfs to build a file system
whose raw image will eventually be used on a different type of disk
than the one on which it is initially created (for example on a write-
once disk). Note that changing any of these values from their defaults
will make it impossible for fsck(8) to find the alternate superblocks
if the standard superblock is lost.
-S sector-size
The size of a sector in bytes (almost never anything but 512).
The file system should be created with the default sector size of 512, because that is what the 512e drive has. The way that the PS3 is creating and accessing the file system is as if it thinks it is a 4Kn drive, which it is not and which I do not believe that the PS3 has any support for.
So on the basis of this apparent misdemeanor, my advice would be that you should favour using a 512n drive with your PS3. This is especially true if you want an easier chance of data access or recovery via a BSD, as a 512e drive will require manual manipulation of the file system superblock before you can properly access the drive.
While the investigation detailed in this article does appear to show that the PS3 is still producing a working file system on the 512e drive, the inode block counts being based on 4096 byte sector sizes when the drive is actually 512 byte sectors could result in unpredictable behaviour. Without low-level access to the PS3 OS to understand exactly why it is behaving this way, the file system may appear to be operating normally but it should be considered unreliable. It could only be a matter of time until something happens which breaks it, requiring you to re-format the drive (eg. something internally sees the drive as the 512 byte sectors that it really is, and gets confused by the inode block size counts being based on 4096 byte sector sizes).
It is difficult to determine exactly when support for Advanced Format disk drives was added to BSD. With the PS3 OS being a BSD variant that was likely forked ~2005 or earlier, it is unlikely to have had support at that time and it is unclear how up-to-date Sony would have kept their OS over the years that the PS3 was in active support, at least relative to other BSD versions.
How do I know if my drive is 512e
You can confirm the format of a drive you have by connecting it to a computer and running the following commands.
The drive must be connected directly to a SATA connection (ie. on the motherboard or eSATA). You cannot read this information from a drive connected by a USB-to-SATA adapter, as these adapters may mask the real sector sizes of the connected device.
A drive with a physical sector size of 4096 and a logical sector size of 512 is a 512e drive; a drive with both sector sizes reported as 512 is a 512n drive.
Get-PhysicalDisk | where BusType -eq "SATA" | Select FriendlyName, Size, PhysicalSectorSize, LogicalSectorSize, MediaType | Format-Table
FriendlyName Size PhysicalSectorSize LogicalSectorSize MediaType
------------ ---- ------------------ ----------------- ---------
CT1000MX500SSD1 1000204886016 4096 512 SSD
ST3000DM001-1ER166 3000592982016 4096 512 HDD
Samsung SSD 850 EVO 500GB 500107862016 512 512 SSD
Samsung SSD 850 EVO 500GB 500107862016 512 512 SSD
SanDisk SDSSDH32000G 2000398934016 512 512 SSD
lsblk --output NAME,PATH,MODEL,SIZE,PHY-SEC,LOG-SEC,TYPE -S
NAME PATH MODEL SIZE PHY-SEC LOG-SEC TYPE
sda /dev/sda ST3000DM001-1ER166 2.7T 4096 512 disk
sdb /dev/sdb Samsung SSD 850 EVO 500GB 465.8G 512 512 disk
sdc /dev/sdc Samsung SSD 850 EVO 500GB 465.8G 512 512 disk
sdd /dev/sdd SanDisk SDSSDH32000G 1.8T 512 512 disk
sdf /dev/sdf CT1000MX500SSD1 931.5G 4096 512 disk
Which drives are 512n
It is surprisingly difficult to find information on the sector format of consumer tier disk drives. If your drive was manufactured prior to 2010 it will almost certainly be a 512n drive. For anything manufactured since then we apparently have to rely on crowd sourcing the information from people who own the drives.
I was specifically interested in finding a 512n SSD, as this is what I wanted to used with my PS3, so this is the information I have primarily gathered here.
List of known 512n drives
This table is based on a small sample of drives from user submitted data. It is possible that there is variation across models, for example see the Crucial MX500 which appears to be manufactured in both 512e and 512n formats.
While I have listed drives of all sizes here for information, you should prefer to use no larger than a 1TB drive to retain all PS3 system file system functionality.
| Drive | Size | Firmware Version |
|---|---|---|
| Crucial BX100 256GB | 256 GB | MU02 |
| Crucial M4 256GB | 256 GB | 070H |
| CT1000BX500SSD1 | 1 TB | M6CR056 |
| Plextor PX-256M6Pro | 256 GB | |
| Samsung SSD 840 EVO 1TB | 1 TB | |
| Samsung SSD 850 EVO 250GB | 250 GB | EMT02B6Q |
| Samsung SSD 850 EVO 500GB | 500 GB | |
| Samsung SSD 870 QVO 8TB | 8 TB | SVQ02B6Q |
| SanDisk SDSSDH32000G | 2 TB | |
| WD Blue SA510 2.5 1000GB | 1 TB | 52048100 |
| WD WDS120G2G0B-00EPW0 M.2 | 120 GB | |
| WDC WDS100T2B0A-00SM50 | 1 TB | 415020WD |
Reference 1: Overclockers UK Forum - SATA SSDs which operate in 512n Sector Size
Beware the Crucial MX500
The Crucial MX500 is a very popular and well regarded SSD that has been produced over several years, however it appears there are variants in both 512n and 512e format. There does not seem to be any obvious indicators to look for to guarantee that the model you may be purchasing is a 512n format version. I personally have one manufactured in 2018 and one in 2023, and both of them are 512e variants. It would seem that this variance is not restricted to drives manufactured during a certain time, or with certain firmware versions, and to date they continue to manufacture these drives in both variants. The table below shows information about a variety of these drives, including their serial numbers and firmware versions where known along with whether they are 512e or 512n.
The start of the serial number appears to indicate manufacture date, for example:
1826E... would be 2018 week 26
2336E... would be 2023 week 36
| Drive | Firmware Version | Serial | Size | Physical Sector Size | Logical Sector Size |
|---|---|---|---|---|---|
| CT1000MX500SSD1 | M3CR023 | 1826E145F98B | 1 TB | 4096 | 512 |
| CT1000MX500SSD1 | 1845E1D5EC2D | 1 TB | 4096 | 512 | |
| CT1000MX500SSD1 | 1845E1D63FE1 | 1 TB | 4096 | 512 | |
| CT1000MX500SSD1 | 1844E1D4FA79 | 1 TB | 512 | 512 | |
| CT1000MX500SSD1 | 1845E1D610B1 | 1 TB | 512 | 512 | |
| CT1000MX500SSD1 | 1920E203CC08 | 1 TB | 4096 | 512 | |
| CT1000MX500SSD1 | M3CR023 | 1 TB | 512 | 512 | |
| CT1000MX500SSD1 | M3CR046 | 1 TB | 512 | 512 | |
| CT1000MX500SSD1 | M3CR046 | 2336E... | 1 TB | 4096 | 512 |
| CT2000MX500SSD1 | M3CR033 | 2 TB | 512 | 512 | |
| CT2000MX500SSD1 | M3CR046 | 2 TB | 512 | 512 |
Reference 1: Overclockers UK Forum - SATA SSDs which operate in 512n Sector Size
Reference 2: A comment on Chris Siebenmann's blog
Additional notes
This testing was conducted using a PS3 Slim model CECH-2503A, running firmware 4.90 (the latest official firmware at the time of writing).