Accessing a PCI Device's Configuration Space Registers

It isn't necessary to do anything extra to access your device's configuration space registers. They are always accessible.

Base Address Register Setting

The base address registers in the configuration space are set by the CPU PROM.

The CPU PROM (not the PCI card's FCode PROM) allocates the base address for memory and/or I/O space on your PCI device and for the FCode PROM.

System Cache Line Size

To determine the system's cache line size from the FCode to write into the cache-line-size configuration space register of your PCI device, look in the cache-line-size register in the configuration space; it refers to the cache line size supported by the PCI device.

Sun Ultra-30 UPA/PCI-Related Nodes

The PCI-related nodes on the Sun Ultra-30 UPA/PCI system are /pci@1f,4000 and /pci@1f,2000. pcia and pcib as needed for the NVRAM variables pcia-probe-list and pcib-probe-list are determined in the following manner.

Each PCI bus has a property named "slot-names" which gives information about slots on that PCI bus. It could sometimes indicate which NVRAM variable corresponds to it.

To get a human-readable value for that property, do the following:

ok " </pci-bus-node>" select-dev ok " slot-names" get-my-property drop decode-int .h cr type

For example, for a PCI bus at /pci@1f,2000:

ok " /pci@1f,2000" select-dev ok " slot-names" get-my-property drop decode-int .h cr type

will display something like:

6 pcia slot 1pcia slot 2

This is an indication that devices under /pci@1f,2000 relate to pcia.

In a Sun Ultra-30 UPA/PCI with 4 plug-in PCI slots, only slot 1 is physically present for pci@1f,2000. It can also support 66 Mhz., 64 bit PCI devices.

ok " /pci@1f,4000" select-dev ok " slot-names" get-my-property drop decode-int .h cr type 34 pcib slot 2pcib slot 4pcib slot 5

This is an indication that devices under /pci@1f,4000 relate to

pcib,

In a Sun Ultra-30 UPA/PCI with 4 plug-in PCI slots:

• Slots 2, 4 and 5 under /pci@1f,4000 support 33 Mhz., 32 bit PCI devices.
• Slot 3 under /pci@1f,4000 is for an on-board SCSI device.

Note that the value of the "slot-names" property differs for different systems. Some systems may not indicate which PCI bus is which by the value of the "slot-names" property.

Also in different releases of the PROM for the same system, the value of the "slot-names" property may change. You may need to refer to the system documentation for details about PCI buses on the system.

Alternatively, you can find which NVRAM variable refers to which PCI bus by setting the NVRAM variables to different values and/or by plugging PCI card(s) in different slot(s).

Finding and Using Physical Addresses

To find and use physical addresses to access, for instance, configuration space registers on a Sun Ultra-30 UPA/PCI system, do MMU bypassing with the choice of the correct ASI space. The arguments for the space {c,d,w,l,x} command includes an address and ASI code (and data for a write operation.)

On Sun Ultra-30 UPA/PCI systems, a PCI device's configuration registers are viewed using the following address:

1fe.0100.0000 + X

where the 32-bit value of X is represented in bit format as:

0000.0000.bbbb.bbbb.dddd.dfff.rrrr.rrrr

where

where bbbb.bbbb is an eight bit bus number, dddd.d is a five bit device number fff is a three-bit function number

rrrr.rrrr is an eight-bit register number

So, if the bus number is 81, device number is 0 and function number is 1, then X will be 81.0100, giving you a configuration register base.

So you'll access the 0th configuration register at 1fe.0181.0100 (physical address). On Sun Ultra-30 UPA/PCI systems, you can use ASI 0x15 for a non-cacheable address being accessed by MMU bypass. If you are accessing a little-endian device, use ASI 0x1d.

You can get bus number, device number, and function number from my-space after selecting that device or from the "reg" property value for that device. Look in IEEE 1275/PCI binding for the "reg" property format.

In general, for any system, to get physical addresses for registers in any space (configuration space, 32- bit memory space, and others.) use the map-in command. map-in requires the phys.lo, phys.mid, phys.hi, and length arguments. Phys.lo, phys.mid, and phys.hi numbers can be taken from the corresponding "reg" property.

In the case of configuration space, getting the physical address is easy since phys.lo and phys.mid are always zero. phys.hi is just the configuration space address.

Here is an example of getting the physical address of the configuration space registers, using onboard ethernet on a Sun Ultra-30 UPA/PCI system:

ok " /pci@1f,4000/network@1,1" begin-select-dev ok pwd /pci@1f,4000/network@1,1 ok .properties . . reg (Config Space ---->)00000900 00000000 00000000 00000000 00000000 (32bit memory space ---->) 02000910 00000000 00000000 00000000 00007020 . . . ok 0 0 900 100 " map-in" $call-parent constant my-cfg-vaddr ok my-cfg-vaddr . fff80900 ok my-cfg-vaddr map? VA:fff80900 G:0 W:1 P:1 E:1 CV:0 CP:0 L:0 Soft1:1 PA[40:13]:ff00800 PA:1fe01000000 Diag:0 Soft2:0 IE:0 NFO:0 Size:0 V:1 PA:1fe01000900

Hence the physical address for the base of configuration registers is 1fe.0100.0900 for this device. For plug-in PCI devices, the registers' physical address may vary if the device is plugged into a different slot, or if other devices are present. Similarly, using the "reg" entry for memory or I/O space, you can find a physical address for those spaces.

Controlling PCI Slot Probing on an Ultra-30 UPA/PCI System

You can control probing of PCI slots on your Sun Ultra-30 UPA/PCI system as follows. On Sun Ultra-30 UPA/PCI system, during normal system initialization, there are NVRAM variables which indicate to the CPU PROM what slots to probe and in what order. On the Sun Ultra-30 UPA/PCI system they are: pcia-probe-list and pcib-probe-list. The default value for pcia-probe-list is 1,2; for pcib-probe-list, it is 3,2,4,5. To disable slot 4 probing on pcib, during normal initialization after a reset, change pcib-probe-list to:

ok setenv pcib-probe-list 3,2,5

After a reset, to probe slot 4 on pcib manually do:

ok 4 probe-pci-slot /pci@1f,4000

Note that not all CPU PROMs have the probe-pci-slot command. Also, in future PROMs, behavior of this command may change, including the possibility of its deletion.

Using 3.x Tokenizer and 3.x CPU PROMs

Here are some points to consider while using the 3.x tokenizer: While you are testing FCode under CPU OpenBoot PROM 3.x versions, make sure that you have OpenBoot PROMs version 3.1 or later. Pre-3.1 PROMs will need the following NVRAM patch:

ok nvedit 0:: nl-move( src dst len -- ) rot n->l rot n->l rot n->l (move); 1: ['] nl-move is move 2: ['] l>>a 2 la+ dup l@ h# 1000 invert and swap l! 3: ['] lrshift 2 la+ dup l@ h# 1000 invert and swap l! 4: ^C ok nvstore ok setenv use-nvramrc? true ok reset-all

Also, note that, while using the 2.x or 3.x tokenizer, literals or numbers that have bit 31 set to 1 will extend this bit (1) to bit 63 on 3.x CPU PROMs. For example:

8000.0000 constant xxx

will in reality be giving a value as: ffff.ffff.8000.0000. When such words or constants are used in address manipulation or otherwise, your code should clip them to a 32 bit value:

Get a real 8000.0000 by:

ff ff ff ff bljoin constant x-num
: clip-num ( n -- l )  x-num and ;
8000.0000 clip-num constant  xxx

or

use xxx clip-num wherever "xxx" is being used.

PCI Device Configuration Register Access

To find the address to use for configuration register access on your PCI device, look in the format for the physical address of the "reg" property. Use the phys.hi cell of the first entry in the "reg" property as the base address for the configuration space. The first entry in the "reg" property must be the configuration space entry (bbbb.bbbb.dddd.dfff.0000.0000 binary). Using this or any other method, obtain the values of bbbb.bbbb, ddddd and fff for your device. Then use:

ok "< parent-pci-bus-node>" select-dev ok <bbbb.bbbb.dddd.dfff>XX config-l@

(XX is the offset for that register configuration.) For example, if the bus number is 1000.0001 (0x81), the device number is 0.0000, and the function number is 001 (0x01), then use

ok " /pci@1f,2000" select-dev ok 81.0100 config-l@ ( to read device id and vendor id) ok 81.0104 config-w@ ( to read command register ) ok 81.0130 config-l@ ( to read the expansion PROM base address register)

Boot Software Roles

Three types of software involved during a boot: the kernel, FCode, and the OS driver. This section describes the normal Solaris operating environment boot scenario, including the functions of each and the order in which they begin.

At power-on, the CPU PROM begins execution. It probes all on-board devices and plug-in cards, thus interpreting the FCodes on all FCode PROMs. In the FCode probing process, FCode PROMs generate properties for devices. Some FCode PROMs execute commands to reset the device and perform other initialization.

Then, the CPU PROM boots over the specified boot device (using its FCode boot driver), loads bootblk (or inetboot for network booting), and passes control to the bootblk code. Then, the bootblk code loads the kernel and modules, and passes control to the kernel. The kernel at some point starts to use the OS-device driver.

So, the order is:

• CPU-PROM
• FCode-PROM
• bootblk
• Kernel
• OS driver.

Enabling Access to a PCI Device's Memory Space Locations

If a developer is loading FCode and can't access memory space locations, how should they go about enabling access to memory space locations for their PCI device?

Look in the format for the physical address of "reg" property. Using that (or any other method), obtain the values of bbbb.bbbb, ddddd and fff for your device. Then use

ok " <parent-pci-bus-node>" select-dev ok 3 <bbbb.bbbb.dddd.dfff>04 config-w!

This will write to the configuration space command register and thus enable access to memory and I/O space. This sets bit[0] and bit[1] of the command register. In the same way, you may set other bits in the command register, if needed by your application. If the bus number is 1000.0001 (0x81), the device number is 0.0000, and the function no. is 001 (0x01), you will then use

ok " /pci@1f,2000" select-dev ok 81.0104 config-w@ 3 or 81.0104 config-w!

Normally, your FCode driver's open routine should enable such access. FCode can use the value returned by my-space and add an offset of 4 to get the address of the command register. Then it can set various bits in the command register to enable the desired access. The close routine should disable that access.

Expansion FCode PROM

If a developer is unable to access his expansion FCode PROM, how can access to it be enabled?

To enable access, look in the format for the physical address of the "reg" property. Using that (or any other method), obtain the values of bbbb.bbbb, ddddd and fff for your device. Then use

ok "< parent-pci-bus-node>" select-dev ok <bbbb.bbbb.dddd.dfff>04 config-w@ 3 or <bbbb.bbbb.dddd.dfff>04 config-w! ok <bbbb.bbbb.dddd.dfff>30 config-l@ 1 or <bbbb.bbbb.dddd.dfff>30 config-l!

This will first enable memory and I/O space access. Then, it will read the value from the expansion PROM configuration space base address register (at offset 0x30) or 1 to it, and write the value in the expansion PROM base address register to enable access to your FCode PROM. This sets bit[0] of the expansion PROM base address register. In other words, if the bus number is 1000.0001 (0x81), the device number is 00000, and the function number is 001 (0x01), then use:

ok " /pci@1f,2000" select-dev ok 81.0104 config-w@ 3 or 81.0104 config-w! ok 81.0130 config-l@ 1 or 81.0130 config-l!

If for any reason, (for example, to access Vital Product Data stored in the PROM) the FCode needs to access PROM data, then the FCode should enable PROM access by using the value returned by my-space and adding an offset of 0x30 as the register address. The FCode should read the value from the address, or the value with 1, and write the result back to that address.

Also, since the FCode would have been copied in memory, devices' memory and I/O spaces may not be enabled. So, the FCode must enable them, using FCode:

ok my-space h# 30 + dup config-l@ 1 or swap config-l! ( enable PROM access) ok my-space h# 4 + dup config-w@ 3 or swap config-w! ( enable I/O, memory access) Using the example above, you can disable expansion PROM access as: ok my-space h# 30 + dup config-l@ 1 invert and swap config-l! ( disable PROM access)

Packaging Error with Ethernet FCode

In trying to load the FCode from Ethernet, the code seems to load without errors; however, when the developer tries to build the package, she gets an error:

ok 4000 dload /stand/cheerio.o Boot device: /pci@1f,4000/network@1,1:,|stand|cheerio.o File and args: ok 0 0 " 0,1" " /pci@1f,2000" begin-package ok 4000 1 byte-load Unimplemented FCode token before address 4004 Warning: FCode sequence resulted in a net stack depth change of 1 ok

The error may be due to either of these causes:

1. The PCI header is attached to the PROM image, or

2. my-address is 2 32-bit numbers for PCI and only 1 32-bit number for SBus.

If the cause is 1, dump the download image beginning at 4000, for example

4000 60 dump

and see where f1 or fd starts. It is the beginning of the FCode data for the byte-load. For instance, if the FCode data starts at x, use the address x in

ok X 1 byte-load

f1 or fd is the beginning of the FCode header, 8 bytes long, as:

f1, <reserved byte>,<2 reserved bytes>,<4 bytes of FCode length>

---

Note - If you begin your FCode source with fcode-version1, the first FCode data is fd, but if you use fcode-version2 or fcode-version3, the first FCode data is f1.

---

If the cause is 2, change your FCode to handle two numbers returned from my-address.

One way to do this is:

my-address constant my-bus-addr-mid constant my-bus-addr-low : my-bus-addr ( -- paddr.low paddr.mid ) my-bus-addr-low my-bus-addr-mid ;

Then use my-bus-addr in the creation of the "reg" property.