Configuring the crate for PCIe x8
- Benoît Roche
- Matt Taylor (Unlicensed)
Factory state
By default AMC720 and UTC002 are configured for PCIe x4 operation. The MBF can be operated in this mode, but it will not be optimal in term of data transfer speed.
If you want to use the MBF with a PCIe x4 link (hence avoiding the modifications below), you need to change the e-key configuration of both AMC720 and AMC525 as compared to what is describe in Bringing up MBF: you will use PCIe ports 4-7 instead of ports 4-11.
Change DIP switch configuration on AMC720
The DIP switch SW4-4 has to be set to OFF (while it is ON by default) to enable PCIe x8 operation.
The AMC720 description for this switch is:
| Switch | Description | OFF | ON |
|---|---|---|---|
| SW4-4 | PCIe Port Bifurcation Strap | x8 | Dual x4 (Default) |
When this switch is set to OFF, it looks like this:
E-key configuration
If it wasn't done before, e-key has to be configured to enable PCIe link in x8 mode (see Bringing up MBF).
Configure DA135 for 8 lanes
PCIe bus is managed by the PCIe Fabric DA135 installed on the UTC002. By default the DA135 is configured to handle x4 lanes PCIe link. You have to upgrade its configuration to be able to use 8 lanes PCIe.
To check current status of the DA135, log on the UTC002 (default password is 'root') and use pcie command:
l-utca:~ # ssh root@192.168.40.250
root@192.168.40.250's password:
[root@vtipmi startup]# pcie
Virtual Link Max Link Link
Device Physical Switch Status Width Width Speed Type
--------------------------------------------------------------------------
1 8 0 Disabled No Link x4 2.5 GT/s Downstream
2 2 0 Disabled No Link x4 2.5 GT/s Downstream
3 0 0 Disabled No Link x4 2.5 GT/s Upstream
4 6 0 Disabled No Link x4 2.5 GT/s Downstream
5 4 0 Disabled No Link x4 2.5 GT/s Downstream
6 14 0 Disabled No Link x4 2.5 GT/s Downstream
7 9 0 Disabled No Link x4 2.5 GT/s Downstream
8 3 0 Disabled No Link x4 2.5 GT/s Downstream
9 1 0 Disabled No Link x4 2.5 GT/s Downstream
10 7 0 Disabled No Link x4 2.5 GT/s Downstream
11 5 0 Disabled No Link x4 2.5 GT/s Downstream
12 15 0 Disabled No Link x4 2.5 GT/s Downstream
13 12 0 Disabled No Link x4 2.5 GT/s Downstream
The maximum PCIe width is x4.
Then you need to get the file DA135-PCIE-x8-copper-1.4.0.tgz from Vadatech Products Download Center Portal under UTC002 / PCI Fabric / DA135 PCIe Gen3 Virtual Switch FRU Information.
Copy the file in /upgrade (from main CPU using scp, its IP address is 192.168.40.1 here) and upgrade the system (you have to use the /upgrade folder as shown below):
[root@vtipmi ~]# cd /upgrade [root@vtipmi upgrade]# scp 192.168.40.1:/your_folder/DA135-PCIE-x8-copper-1.4.0.tgz ./ [root@vtipmi upgrade]# tar xvzf DA135-PCIE-x8-copper-1.4.0.tgz DA135-PCIE-x8-copper-1.4.0 DA135-PCIE-x8-copper-1.4.0/eeprom.pex DA135-PCIE-x8-copper-1.4.0/upgrade DA135-PCIE-x8-copper-1.4.0/fruDA.xml DA135-PCIE-x8-copper-1.4.0/plx.eeprom DA135-PCIE-x8-copper-1.4.0/system.ini DA135-PCIE-x8-copper-1.4.0/version DA135-PCIE-x8-copper-1.4.0/fruGenMCMC DA135-PCIE-x8-copper-1.4.0/redriver.bin DA135-PCIE-x8-copper-1.4.0/vtpcie_ekey.conf DA135-PCIE-x8-copper-1.4.0/da.conf [root@vtipmi upgrade]# cd DA135-PCIE-x8-copper-1.4.0 [root@vtipmi DA135-PCIE-x8-copper-1.4.0]# vtipmi stop Stopping MCMC Stopping Carrier Manager Stopping Shelf Manager Stopping failover agent Stopping file monitor Stopping SNMP server Stopping SNMP trap handler Stopping ethernet monitor Stopping execHelper Stopping Memory monitor Stopping vtipmi custom [root@vtipmi DA135-PCIE-x8-copper-1.4.0]# ./upgrade Backing up /etc/da.conf to /etc/da.conf.bak Backing up /etc/vtpcie_ekey.conf to /etc/vtpcie_ekey.conf.bak Creating Daughter Card FRU inventory warning, no module power information found. warning, no zone3 compatibility information found. Updating EEPROM at address 0x50. Please wait ... (try 1) Verified data in EEPROM 0x50 Erasing the EEPROM... Resetting the swtich...Done Writing to EEPROM: Done Programming PCIe Redriver Configuration EEPROM... Successfully configured daughter card. Please power cycle for the changes to take affect. [root@vtipmi DA135-PCIE-x8-copper-1.4.0]# vtipmi start Starting execHelper Starting heartbeat monitor Starting watchdog timer Starting memory monitor Starting MCMC Starting Managers Starting trap handler Starting vtipmi custom
Check Configuration
After DA135 upgrade you can check that pcie command's output changed:
[root@vtipmi ~]# pcie
Virtual Link Max Link Link
Device Physical Switch Status Width Width Speed Type
--------------------------------------------------------------------------
1 8 0 Enabled No Link x16 2.5 GT/s Endpoint
2 2 0 Enabled No Link x16 2.5 GT/s Endpoint
3 0 0 Enabled No Link x16 2.5 GT/s Endpoint
4 6 0 Enabled No Link x16 2.5 GT/s Endpoint
5 4 0 Enabled No Link x16 2.5 GT/s Endpoint
6 14 0 Enabled No Link x16 2.5 GT/s Endpoint
7 9 0 Enabled No Link x16 2.5 GT/s Endpoint
8 3 0 Enabled No Link x16 2.5 GT/s Endpoint
9 1 0 Enabled No Link x16 2.5 GT/s Endpoint
10 7 0 Enabled No Link x16 2.5 GT/s Endpoint
11 5 0 Enabled No Link x16 2.5 GT/s Endpoint
12 15 0 Enabled No Link x16 2.5 GT/s Endpoint
13 12 0 Enabled No Link x16 2.5 GT/s Endpoin
However you have to reboot UTC002 (just type 'reboot'). Log back on the UTC002 and pcie command will give the correct output:
[root@vtipmi ~]# pcie
Virtual Link Max Link Link
Device Physical Switch Status Width Width Speed Type
--------------------------------------------------------------------------
1 8 0 Disabled No Link x8 2.5 GT/s Downstream
2 1 0 Disabled No Link x8 2.5 GT/s Downstream
3 0 0 Disabled No Link x8 2.5 GT/s Upstream
4 5 0 Disabled No Link x8 2.5 GT/s Downstream
5 4 0 Disabled No Link x8 2.5 GT/s Downstream
6 13 0 Disabled No Link x8 2.5 GT/s Downstream
13 12 0 Enabled No Link x8 2.5 GT/s Downstrea
PCIe link can be x8 at maximum. It FPGA image is loaded the output of pcie is slightly different. The important point here is to have a max width of x8.
Back on the AMC720, first reboot it to take the changes into account. Then you can check that PCIe link is operational (only the last 8 lines are relevant):
l-utca:~ # lspci 00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 00:1d.0 USB controller: Intel Corporation DH89xxCC USB2 Enhanced Host Controller #1 (rev 08) 00:1f.0 ISA bridge: Intel Corporation DH89xxCC LPC Controller (rev 08) 00:1f.2 SATA controller: Intel Corporation DH89xxCC 4 Port SATA AHCI Controller (rev 08) 00:1f.3 SMBus: Intel Corporation DH89xxCC SMBus Controller (rev 08) 00:1f.6 Signal processing controller: Intel Corporation DH89xxCC Thermal Subsystem (rev 08) 00:1f.7 System peripheral: Intel Corporation DH89xxCC Watchdog Timer (rev 08) 01:00.0 Co-processor: Intel Corporation DH89XXCC Series QAT (rev 21) 01:00.1 Ethernet controller: Intel Corporation DH8900CC Series Gigabit Backplane Network Connection (rev 21) 01:00.2 Ethernet controller: Intel Corporation DH8900CC Series Gigabit Backplane Network Connection (rev 21) 01:00.3 Ethernet controller: Intel Corporation DH8900CC Series Gigabit Network Connection (rev 21) 01:00.4 Ethernet controller: Intel Corporation DH8900CC Series Gigabit Network Connection (rev 21) 02:00.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:00.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:01.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:04.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:05.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:08.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:09.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:0c.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab)