...
By default AMC720 and UTC002 are configured for PCIe x4 operation. The MBF can be operated in this mode, but it will not be optimal in term of data transfer speed.
...
The AMC720 description for this switch is:
| Switch | Description | OFF | ONOFF |
|---|---|---|---|
| SW4-4 | PCIe Port Bifurcation Strap | x8 | Dual x4 (Default) |
...
PCIe bus is managed by the PCIe Fabric DA135 installed on the UTC002. By default the DA135 is configured to handle x4 lanes PCIe link. You have to upgrade its configuration to be able to use 8 lanes PCIe.
...
Then you need to get the file DA135-PCIE-x8-copper-1.24.0.tgz from Vadatech Products Download Center Portal under Switches / UTC002 / PCI Fabric / DA135 PCIe Gen3 Virtual Switch FRU Information.
Copy the file in /upgrade (from main CPU using scp, its IP address is 192.168.40.1 here) and upgrade the system (you have to use the /upgrade folder as shown below):
| Code Block | ||
|---|---|---|
| ||
[root@vtipmi ~]# cd /upgrade [root@vtipmi upgrade]# scp 192.168.40.1:/your_folder/DA135-PCIE-x8-copper-1.24.0.tgz ./ [root@vtipmi upgrade]# tar xvzf DA135-PCIE-x8-copper-1.24.0.tgz DA135-PCIE-x8-copper-1.24.0 DA135-PCIE-x8-copper-1.24.0/vtpcie_ekeyeeprom.confpex DA135-PCIE-x8-copper-1.24.0/da.confupgrade DA135-PCIE-x8-copper-1.24.0/upgradefruDA.xml DA135-PCIE-x8-copper-1.24.0/systemplx.inieeprom DA135-PCIE-x8-copper-1.2.4.0/system.ini DA135-PCIE-x8-copper-1.4.0/version DA135-PCIE-x8-copper-1.24.0/plx.eepromfruGenMCMC DA135-PCIE-x8-copper-1.24.0/fruDAredriver.xmlbin DA135-PCIE-x8-copper-1.24.0/fruGenMCMCvtpcie_ekey.conf DA135-PCIE-x8-copper-1.24.0/eepromda.pexconf [root@vtipmi upgrade]# cd DA135-PCIE-x8-copper-1.24.0 [root@vtipmi DA135-PCIE-x8-copper-1.24.0]# vtipmi stop [root@vtipmi DA135-PCIE-x8-1.2.0]# ./upgrade Backing up /etc/da.conf to /etc/da.conf.bak Backing up /etc/vtpcie_ekey.conf to /etc/vtpcie_ekey.conf.bak Creating Daughter Card FRU inventory warning, no module power information found. warning, no zone3 compatibility information found. Updating EEPROM at address 0x50. Please wait ... (try 1) Verified data in EEPROM 0x50 Erasing the EEPROM... Resetting the swtich...Done Writing to EEPROM: Done Successfully configured daughter card. Please power cycle for the changes to take affect. [root@vtipmi DA135-PCIE-x8-1.2.0]# vtipmi start |
Check Configuration
You have to power cycle the crate for the changes to be taken into account.
After power cycling the PCIe link can be x16 at maximum. If you log on the UTC002, you will see:
| Code Block | ||
|---|---|---|
| ||
[root@vtipmi DA135-PCIE-x8-1.2.0]# pcie Virtual Link Max Link Link Device Physical Switch Status Width Width Speed Type ---------Stopping MCMC Stopping Carrier Manager Stopping Shelf Manager Stopping failover agent Stopping file monitor Stopping SNMP server Stopping SNMP trap handler Stopping ethernet monitor Stopping execHelper Stopping Memory monitor Stopping vtipmi custom [root@vtipmi DA135-PCIE-x8-copper-1.4.0]# ./upgrade Backing up /etc/da.conf to /etc/da.conf.bak Backing up /etc/vtpcie_ekey.conf to /etc/vtpcie_ekey.conf.bak Creating Daughter Card FRU inventory warning, no module power information found. warning, no zone3 compatibility information found. Updating EEPROM at address 0x50. Please wait ... (try 1) Verified data in EEPROM 0x50 Erasing the EEPROM... Resetting the swtich...Done Writing to EEPROM: Done Programming PCIe Redriver Configuration EEPROM... Successfully configured daughter card. Please power cycle for the changes to take affect. [root@vtipmi DA135-PCIE-x8-copper-1.4.0]# vtipmi start Starting execHelper Starting heartbeat monitor Starting watchdog timer Starting memory monitor Starting MCMC Starting Managers Starting trap handler Starting vtipmi custom |
Check Configuration
After DA135 upgrade you can check that pcie command's output changed:
| Code Block | ||
|---|---|---|
| ||
[root@vtipmi ~]# pcie Virtual Link Max Link Link Device Physical Switch Status Width Width Speed Type -------------------------------------------------------------------------- 1 8 0 Enabled No Link x16 2.5 GT/s Endpoint 2 2 0 Enabled No Link x16 2.5 GT/s Endpoint 3 0 0 Enabled No Link x16 2.5 GT/s Endpoint 4 6 0 Enabled No Link x16 2.5 GT/s Endpoint 5 4 0 Enabled No Link x16 2.5 GT/s Endpoint 6 14 0 Enabled No Link x16 2.5 GT/s Endpoint 77 9 0 Enabled No Link x16 2.5 GT/s Endpoint 8 3 0 Enabled No Link x16 2.5 GT/s Endpoint 9 1 0 Enabled No Link x16 2.5 GT/s Endpoint 10 7 0 Enabled No Link x16 2.5 GT/s Endpoint 11 5 0 Enabled No Link x16 2.5 GT/s Endpoint 12 15 0 Enabled No Link x16 2.5 GT/s Endpoint 13 12 0 Enabled No Link x16 2.5 GT/s Endpoin |
However you have to reboot UTC002 (just type 'reboot'). Log back on the UTC002 and pcie command will give the correct output:
| Code Block | ||
|---|---|---|
| ||
[root@vtipmi ~]# pcie Virtual Link Max Link Link Device Physical Switch Status Width Width Speed Type -------------------------------------------------------------------------- 1 98 0 Enabled Disabled No Link x8 x16 2.5 GT/s EndpointDownstream 82 31 0 Enabled Disabled No Link x8 x16 2.5 GT/s EndpointDownstream 93 10 0 EnabledDisabled No Link x16x8 2.5 GT/s Upstream Endpoint4 10 75 0 EnabledDisabled No Link x16x8 2.5 GT/s Downstream Endpoint5 11 54 0 Enabled Disabled No Link x8 x16 2.5 GT/s Downstream Endpoint6 12 1513 0 Enabled Disabled No Link x8 x16 2.5 GT/s EndpointDownstream 13 12 0 Enabled No Link x16 2.5 GT/s Endpoint |
...
No Link x8 2.5 GT/s Downstrea |
PCIe link can be x8 at maximum. It FPGA image is loaded the output of pcie is slightly different. The important point here is to have a max width of x8.
Back on the AMC720, first reboot it to take the changes into account. Then you can check that PCIe link is taking into account by the systemoperational (only the last 8 lines are relevant):
| Code Block | ||
|---|---|---|
| ||
l-utca:~ # lspci 00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) 00:1d.0 USB controller: Intel Corporation DH89xxCC USB2 Enhanced Host Controller #1 (rev 08) 00:1f.0 ISA bridge: Intel Corporation DH89xxCC LPC Controller (rev 08) 00:1f.2 SATA controller: Intel Corporation DH89xxCC 4 Port SATA AHCI Controller (rev 08) 00:1f.3 SMBus: Intel Corporation DH89xxCC SMBus Controller (rev 08) 00:1f.6 Signal processing controller: Intel Corporation DH89xxCC Thermal Subsystem (rev 08) 00:1f.7 System peripheral: Intel Corporation DH89xxCC Watchdog Timer (rev 08) 01:00.0 Co-processor: Intel Corporation DH89XXCC Series QAT (rev 21) 01:00.1 Ethernet controller: Intel Corporation DH8900CC Series Gigabit Backplane Network Connection (rev 21) 01:00.2 Ethernet controller: Intel Corporation DH8900CC Series Gigabit Backplane Network Connection (rev 21) 01:00.3 Ethernet controller: Intel Corporation DH8900CC Series Gigabit Network Connection (rev 21) 01:00.4 Ethernet controller: Intel Corporation DH8900CC Series Gigabit Network Connection (rev 21) 02:00.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:00.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:01.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:04.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:05.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:08.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:09.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) 03:0c.0 PCI bridge: PLX Technology, Inc. Device 8764 (rev ab) |
...