ref: dde3628d54073e0569bf565e6ec861e42cfa35f6
dir: /sys/src/cmd/atazz/atazz.ms/
.FP lucidasans
.TL
ATA au Naturel
.AU
Erik Quanstrom
quanstro@quanstro.net
.AB
The Plan 9
.I sd (3)
interface allows raw commands to be sent. Traditionally,
only SCSI CDBs could be sent in this manner. For devices
that respond to ATA/ATAPI commands, a small set of SCSI CDBs
have been translated into an ATA equivalent. This approach
works very well. However, there are ATA commands such as
SMART which do not have direct translations. I describe how
ATA/ATAPI commands were supported without disturbing
existing functionality.
.AE
.SH
Introduction
.PP
In writing new
.I sd (3)
drivers for plan 9, it has been necessary to copy laundry
list of special commands that were needed with previous
drivers. The set of commands supported by each device
driver varies, and they are typically executed by writing a
magic string into the driver's
.CW ctl
file. This requires code duplicated for each driver, and
covers few commands. Coverage depends on the driver. It is
not possible for the control interface to return output,
making some commands impossible to implement. While a
work around has been to change the contents of the control
file, this solution is extremely unwieldy even for simple
commands such as
.CW "IDENTIFY DEVICE" .
.SH
Considerations
.PP
Currently, all
.I sd
devices respond to a small subset of SCSI commands
through the raw interface and the normal read/write interface uses
SCSI command blocks. SCSI devices, of course, respond natively
while ATA devices emulate these commands with the help
.I sd .
This means that
.I scuzz (8)
can get surprisingly far with ATA devices, and ATAPI
.I \fP(\fPsic. )
devices
work quite well. Although a new implementation might not
use this approach, replacing the interface did not appear
cost effective and would lead to maximum incompatibilities,
while this interface is experimental. This means that the raw interface will need
a method of signaling an ATA command rather than a SCSI CDB.
.PP
An unattractive wart of the ATA command set is there are seven
protocols and two command sizes. While each command has a
specific size (either 28-bit LBA or 48-bit LLBA) and is
associated with a particular protocol (PIO, DMA, PACKET,
etc.), this information is available only by table lookup.
While this information may not always be necessary for simple
SATA-based controllers, for the IDE controllers, it is required.
PIO commands are required and use a different set of registers
than DMA commands. Queued DMA commands and ATAPI
commands are submitted differently still. Finally,
the data direction is implied by the command. Having these
three extra pieces of information in addition to the command
seems necessary.
.PP
A final bit of extra-command information that may be useful
is a timeout. While
.I alarm (2)
timeouts work with many drivers, it would be an added
convenience to be able to specify a timeout along with the
command. This seems a good idea in principle, since some
ATA commands should return within milli- or microseconds,
others may take hours to complete. On the other hand, the
existing SCSI interface does not support it and changing its
kernel-to-user space format would be quite invasive. Timeouts
were left for a later date.
.SH
Protocol and Data Format
.PP
The existing protocol for SCSI commands suits ATA as well.
We simply write the command block to the raw device. Then
we either write or read the data. Finally the status block
is read. What remains is choosing a data format for ATA
commands.
.PP
The T10 Committee has defined a SCSI-to-ATA translation
scheme called SAT[4]. This provides a standard set of
translations between common SCSI commands and ATA commands.
It specifies the ATA protocol and some other sideband
information. It is particularly useful for common commands
such as
.CW "READ\ (12)"
or
.CW "READ CAPACITY\ (12)" .
Unfortunately, our purpose is to address the uncommon commands.
For those, special commands
.CW "ATA PASSTHROUGH\ (12)"
and
.CW "(16)"
exist. Unfortunately several commands we are interested in,
such as those that set transfer modes are not allowed by the
standard. This is not a major obstacle. We could simply
ignore the standard. But this goes against the general
reasons for using an established standard: interoperability.
Finally, it should be mentioned that SAT format adds yet
another intermediate format of variable size which would
require translation to a usable format for all the existing
Plan 9 drivers. If we're not hewing to a standard, we should
build or choose for convenience.
.PP
ATA-8 and ACS-2 also specify an abstract register layout.
The size of the command block varies based on the “size”
(either 28- or 48-bits) of the command and only context
differentiates a command from a response. The SATA
specification defines host-to-drive communications. The
formats of transactions are called Frame Information
Structures (FISes). Typically drivers fill out the command
FISes directly and have direct access to the Device-to-Host
Register (D2H) FISes that return the resulting ATA register
settings. The command FISes are also called Host-to-Device
(H2D) Register FISes. Using this structure has several advantages. It
is directly usable by many of the existing SATA drivers.
All SATA commands are the same size and are tagged as
commands. Normal responses are also all of the same size
and are tagged as responses. Unfortunately, the ATA
protocol is not specified. Nevertheless, SATA FISes seem to
handle most of our needs and are quite convenient; they can
be used directly by two of the three current SATA drivers.
.SH
Implementation
.PP
Raw ATA commands are formatted as a ATA escape byte, an
encoded ATA protocol
.CW proto
and the FIS. Typically this would be a
H2D FIS, but this is not a requirement. The escape byte
.R 0xff ,
which is not and, according to the current specification,
will never be a valid SCSI command, was chosen. The
protocol encoding
.CW proto
and other FIS construction details are specified in
.CW "/sys/include/fis.h" .
The
.CW proto
encodes the ATA protocol, the command “size” and data
direction. The “atazz” command format is pictured in \*(Fn.
.F1
.PS
scale=10
w=8
h=2
define hdr |
[
box $1 ht h wid w
] |
define fis |
[
box $1 ht h wid w
] |
F: [
A: hdr("0xff")
hdr("proto")
hdr("0x27")
hdr("flags")
B: hdr("cmd") at A+(0, -h)
hdr("feat")
hdr("lba0")
hdr("lba8")
C: hdr("lba16") at B+(0, -h)
hdr("dev")
hdr("lba24")
hdr("lba32")
D: hdr("lba40") at C+(0, -h)
hdr("feat8")
hdr("cnt")
hdr("cnt8")
E: hdr("rsvd") at D+(0, -h)
hdr("ctl")
]
G: [
fis("sdXX/raw")
] at F.se +(w*2, -h)
arrow from F.e to G.w
H: [
fis("data")
] at G.sw +(-w*2, -h)
HH: [
fis("sdXX/raw")
] at H.se +(w*2, -h)
arrow from H.e to HH.w
Q: [
fis("sdXX/raw")
] at HH +(0, -2*h)
I: [
K: hdr("0xff")
hdr("proto")
hdr("0x34")
hdr("port");
L: hdr("stat") at K+(0, -h)
hdr("err")
hdr("lba0")
hdr("lba8")
M: hdr("lba16") at L+(0, -h)
hdr("dev")
hdr("lba24")
hdr("lba32")
O: hdr("lba40") at M+(0, -h)
hdr("feat8")
hdr("cnt")
hdr("cnt8")
P: hdr("rsvd") at O+(0, -h)
hdr("ctl")
] at Q.sw +(-w*3.5, -3*h)
arrow from Q.w to I.e
.PE
.F2
.F3
.PP
Raw ATA replies are formatted as a one-byte
.R sd
status code followed by the reply FIS.
The usual read/write register substitutions are
applied; ioport replaces flags, status replaces cmd, error
replaces feature.
.PP
Important commands such as
.CW "SMART RETURN STATUS"
return no data. In this case, the protocol is run as usual.
The client performs a 0-byte read to fulfill data transfer
step. The status is in the D2H FIS returned as the status.
The vendor ATA command
.R 0xf0
is used to return the device signature FIS as there is no
universal in-band way to do this without side effects.
When talking only to ATA drives, it is possible to first
issue a
.CW "IDENTIFY PACKET DEVICE"
and then a
.CW "IDENTIFY DEVICE"
command, inferring the device type from the successful
command. However, it would not be possible to enumerate the
devices behind a port multiplier using this technique.
.SH
Kernel changes and Libfis
.PP
Very few changes were made to devsd to accommodate ATA
commands. the
.CW SDreq
structure adds
.CW proto
and
.CW ataproto
fields. To avoid disturbing existing SCSI functionality and
to allow drivers which support SCSI and ATA commands in
parallel, an additional
.CW ataio
callback was added to
.CW SDifc
with the same signature as
the existing
.CW rio
callback. About twenty lines of code were
added to
.CW port/devsd.c
to recognize raw ATA commands and call the
driver's
.CW ataio
function.
.PP
To assist in generating the FISes to communicate with devices,
.CW libfis
was written. It contains functions to identify and
enumerate the important features of a drive, to format
H2D FISes And finally, functions for
.I sd
and
.I sd
-devices to build D2H FISes to
capture the device signature.
.PP
All ATA device drivers for the 386 architecture have been
modified to accept raw ATA commands. Due to consolidation
of FIS handling, the AHCI driver lost
175 lines of code, additional non-atazz-related functionality
notwithstanding. The IDE driver remained exactly the same
size. Quite a bit more code could be removed if the driver
were reorganized. The mv50xx driver gained 153 lines of
code. Development versions of the Marvell Orion driver
lost over 500 lines while
.CW libfis
is only about the same line count.
.PP
Since FIS formats were used to convey
commands from user space,
.CW libfis
has been equally useful for user space applications. This is
because the
.I atazz
interface can be thought of as an idealized HBA. Conversely,
the hardware driver does not need to know anything about
the command it is issuing beyond the ATA protocol.
.SH
Atazz
.PP
As an example and debugging tool, the
.I atazz (8)
command was written.
.I Atazz
is an analog to
.I scuzz (8);
they can be thought of as a driver for a virtual interface provided
by
.I sd
combined with a disk console.
ATA commands are spelled out verbosely as in ACS-2. Arbitrary ATA
commands may be submitted, but the controller or driver may
not support all of them. Here is a sample transcript:
.P1
az> probe
/dev/sda0 976773168; 512 50000f001b206489
/dev/sdC1 0; 0 0
/dev/sdD0 1023120; 512 0
/dev/sdE0 976773168; 512 50014ee2014f5b5a
/dev/sdF7 976773168; 512 5000cca214c3a6d3
az> open /dev/sdF0
az> smart enable operations
az> smart return status
normal
az> rfis
00
34405000004fc2a00000000000000000
.P2
.PP
In the example, the
.CW probe
command is a special command that uses
.CW #S/sdctl
to enumerate the controllers in the system.
For each controller, the
.CW sd
vendor command
.CW 0xf0
.CW \fP(\fPGET
.CW SIGNATURE )
is issued. If this command is successful, the
number of sectors, sector size and WWN are gathered
and and listed. The
.CW /dev/sdC1
device reports 0 sectors and 0 sector size because it is
a DVD-RW with no media. The
.CW open
command is another special command that issues the
same commands a SATA driver would issue to gather
the information about the drive. The final two commands
enable SMART
and return the SMART status. The smart status is
returned in a D2H FIS. This result is parsed the result
is printed as either “normal,” or “threshold exceeded”
(the drive predicts imminent failure).
.PP
As a further real-world example, a drive from my file server
failed after a power outage. The simple diagnostic
.CW "SMART RETURN STATUS"
returned an uninformative “threshold exceeded.”
We can run some more in-depth tests. In this case we
will need to make up for the fact that
.I atazz
does not know every option to every command. We
will set the
.CW lba0
register by hand:
.P1
az> smart lba0 1 execute off-line immediate # short data collection
az> smart read data
col status: 00 never started
exe status: 89 failed: shipping damage, 90% left
time left: 10507s
shrt poll: 176m
ext poll: 19m
az>
.P2
.PP
Here we see that the drive claims that it was damaged in
shipping and the damage occurred in the first 10% of the
drive. Since we know the drive had been working before
the power outage, and the original symptom was excessive
UREs (Unrecoverable Read Errors) followed by write
failures, and finally a threshold exceeded condition, it is
reasonable to assume that the head may have crashed.
.SH
Stand Alone Applications
.PP
There are several obvious stand-alone applications for
this functionality: a drive firmware upgrade utility,
a drive scrubber that bypasses the drive cache and a
SMART monitor.
.PP
Since SCSI also supports a basic SMART-like
interface through the
.CW "SEND DIAGNOSTIC"
and
.CW "RECEIVE DIAGNOSTIC RESULTS"
commands,
.I disk/smart (8)
gives a chance to test both raw ATA and SCSI
commands in the same application.
.PP
.I Disk/smart
uses the usual techniques for gathering a list of
devices or uses the devices given. Then it issues a raw ATA request for
the device signature. If that fails, it is assumed
that the drive is SCSI, and a raw SCSI request is issued.
In both cases,
.I disk/smart
is able to reliably determine if SMART is supported
and can be enabled.
.PP
If successful, each device is probed every 5 minutes
and failures are logged. A one shot mode is also
available:
.P1
chula# disk/smart -atv
sda0: normal
sda1: normal
sda2: normal
sda3: threshold exceeded
sdE1: normal
sdF7: normal
.P2
.PP
Drives
.CW sda0 ,
.CW sda1
are SCSI
and the remainder are ATA. Note that other drives
on the same controller are ATA.
Recalling that
.CW sdC0
was previously listed, we can check to see why no
results were reported by
.CW sdC0 :
.P1
chula# for(i in a3 C0)
echo identify device |
atazz /dev/sd$i >[2]/dev/null |
grep '^flags'
flags lba llba smart power nop sct
flags lba
.P2
So we see that
.CW sdC0
simply does not support the SMART feature set.
.SH
Further Work
.PP
While the raw ATA interface has been used extensively
from user space and has allowed the removal of quirky
functionality, device setup has not yet been addressed.
For example, both the Orion and AHCI drivers have
an initialization routine similar to the following
.P1
newdrive(Drive *d)
{
setfissig(d, getsig(d));
if(identify(d) != 0)
return SDeio;
setpowermode(d);
if(settxmode(d, d->udma) != 0)
return SDeio;
return SDok;
}
.P2
However in preparing this document, it was discovered
that one sets the power mode before setting the
transfer mode and the other does the opposite. It is
not clear that this particular difference is a problem,
but over time, such differences will be the source of bugs.
Neither the IDE nor the Marvell 50xx drivers sets the
power mode at all. Worse,
none is capable of properly addressing drives with
features such as PUIS (Power Up In Standby) enabled.
To addresses this problem all four of the ATA drivers would
need to be changed.
.PP
Rather than maintaining a number of mutually out-of-date
drivers, it would be advantageous to build an ATA analog
of
.CW pc/sdscsi.c
using the raw ATA interface to submit ATA commands.
There are some difficulties that make such a change a bit
more than trivial. Since current model for hot-pluggable
devices is not compatible with the top-down
approach currently taken by
.I sd
this would need to be addressed. It does not seem that
this would be difficult. Interface resets after failed commands
should also be addressed.
.SH
Source
.PP
The current source including all the pc drivers and applications
are available
in the following
.I contrib (1)
packages on
.I sources :
.br
.CW "quanstro/fis" ,
.br
.CW "quanstro/sd" ,
.br
.CW "quanstro/atazz" ,
and
.br
.CW "quanstro/smart" .
.PP
The following manual pages are included:
.br
.I fis (2),
.I sd (3),
.I sdahci (3),
.I sdaoe (3),
.I sdloop (3),
.I sdorion (3),
.I atazz (8),
and
.I smart (8).
.SH
Abbreviated References
.nr PI 5n
.IP [1]
.I sd (1),
published online at
.br
.CW "http://plan9.bell-labs.com/magic/man2html/3/sd" .
.IP [2]
.I scuzz (8),
published online at
.br
.CW "http://plan9.bell-labs.com/magic/man2html/8/scuzz" .
.IP [3]
T13
.I "ATA/ATAPI Command Set\ \-\ 2" ,
revision 1, January 21, 2009,
formerly published online at
.CW "http://www.t13.org" .
.IP [4]
T10
.I "SCSI/ATA Translation\ \-\ 2 (SAT\-2)" ,
revision 7, February 18, 2007,
formerly published online at
.CW "http://www.t10.org" .