...
The MBF control system at Diamond consists of the following hardware:
| Hardware | Vendor | Description |
|---|---|---|
| VT814 | Vadatech | 2U μTCA.4 chassis with 6 AMC slots |
| UTC002 | μTCA MCH (MicroTCA Carrier Hub) with support for 8 lanes of PCIe gen 3. | |
| AMC720 | Intel Xeon general purpose AMC processor card with 16GB RAM, 30GB SSD. | |
| AMC525 | AMC FPGA carrier for dual HPC FMC with Virtex-7 690. | |
| FMC-500M | Innovative Integration | FMC module with dual 500 Ms/s 14-bit ADC, dual 1200 Ms/s 16-bit DAC. |
| CTI-FMC-DIO | Creotech | FMC module with five channels of TTL I/O. |
Note that because of an address conflict on the shared IPMI I2C bus between the programmable input threshold DAC on the CTI-FMC-DIO and a temperature sensor on the AMC525, it is necessary to make a modification to the DIO card. Fortunately the default DAC output does not need changing, so it is enough to cut the control line to the DAC.
...
Once the hardware has been configured as above the crate can be powered on and the e-keying for the CPU and AMC525 must be set up. This will be particularly important to ensure that the PCIe link works properly.
CPU e-key
| Code Block | ||
|---|---|---|
| ||
Payload SOL Port : Serial1 Gbe Management Port: back0 Shutdown delay : 30s Commands: ekey - Configure Electronic Keying lan - Configure LAN Parameters plser - Configure Payload Serial Port mgtport - Configure Gbe Management Port wdcfg - Configure Watchdog Parameters shutdowndelay - Configure shut down delay > ekey E-Keying configuration * 1) Gigabit Ethernet - Port 0 * 2) Gigabit Ethernet - Port 1 * 3) PCIe Root - Ports 4-11 4) PCIe Root - Ports 4-7 5) PCIe Root - Ports 8-11 6) SATA Client - Port 2 7) SATA Client - Port 3 Commands: 1-7 to toggle option. save, cancel |
...
Connect to the MGT-RS232 port on the AMC525 front panel, and type ekey at the prompt. Adjust for the following settings:
| Code Block | ||
|---|---|---|
| ||
AMC525 > ekey E-Keying configuration * 1) Gigabit Ethernet - Port 0 * 2) Gigabit Ethernet - Port 1 3) SATA Server - Port 2 4) SATA Client - Port 2 5) SATA Server - Port 3 6) SATA Client - Port 3 7) SAS - Port 2 8) SAS - Port 3 9) PCIe Root - Ports 4-11 10) PCIe Root - Ports 4-7 11) PCIe Root - Ports 8-11 * 12) PCIe Node - Ports 4-11 13) PCIe Node - Ports 4-7 14) PCIe Node - Ports 8-11 15) SRIO 3.125 Gbaud - Ports 4-7 16) SRIO 2.5 Gbaud - Ports 4-7 17) SRIO 1.25 Gbaud - Ports 4-7 18) SRIO 3.125 Gbaud - Ports 8-11 19) SRIO 2.5 Gbaud - Ports 8-11 20) SRIO 1.25 Gbaud - Ports 8-11 21) XAUI - Port 4-7 22) XAUI - Port 8-11 Commands: 1-22 to toggle option. save, cancel |
...
Also the vadj setting must be configured as otherwise the FMC cards will not be powered on.
| Code Block | ||
|---|---|---|
| ||
FMC0 EEPROM is invalid / missing FMC1 VADJ range is 0.00V - 0.00V VADJ setting is 1.80V AMC525 > vadj 0: 1.80V c: Cancel Select Voltage (0-0, c): |
...
| Address | Device | |||||
|---|---|---|---|---|---|---|
| 192.168.40.250 | MCH. Connect to this as root with password root, or use
| |||||
| 192.168.40.200 | AMC525. If more than one AMC525 is installed the network address will need to be changed by logging on through the serial console (user root, no password) and editing This interface will be used for programming the FPGA. |
...
First of all the FPGA must be loaded onto the AMC card. The following commands run on the target system will do the necessary work:
| Code Block | ||
|---|---|---|
| ||
BIT_FILE=amc525_lmbf.bit ip=192.168.40.200 scp "$BIT_FILE" root@$ip:/tmp/amc525_lmbf.bit ssh -x root@$ip amc525_lbtool fpga_load /tmp/amc525_lmbf.bit |
At this point run lspci -v, and you should see something like the following:
| Code Block | ||
|---|---|---|
| ||
04:00.0 Signal processing controller: Xilinx Corporation FPGA Card XC7VX690T
Subsystem: Xilinx Corporation Device 0007
Physical Slot: 0
Flags: bus master, fast devsel, latency 0, IRQ 70
Memory at fe300000 (64-bit, non-prefetchable) [size=1M]
Memory at fe400000 (64-bit, non-prefetchable) [size=64K]
Capabilities: <access denied>
Kernel driver in use: amc525_lmbf |
...
If we see this then the FPGA has been successfully loaded and the PCIe link is working correctly. Alas, the PCIe IO memory cannot be mapped at this stage because it is too late for the BIOS to identify it. Now reboot the processor card and re-run lspci -v and we should now see:
| Code Block | ||
|---|---|---|
| ||
04:00.0 Signal processing controller: Xilinx Corporation FPGA Card XC7VX690T
Subsystem: Xilinx Corporation Device 0007
Physical Slot: 0
Flags: bus master, fast devsel, latency 0, IRQ 70
Memory at fe300000 (64-bit, non-prefetchable) [size=1M]
Memory at fe400000 (64-bit, non-prefetchable) [size=64K]
Capabilities: <access denied>
Kernel driver in use: amc525_lmbf |
...
Each time a new AMC card is commissioned it may be necessary to measure the timing skew from the ADC to the FPGA, and should also be checked when the FPGA image is changed. This test can be run without any clocks connected (depending on the selected clocking mode) and needs no data input.
| Code Block | ||
|---|---|---|
| ||
$ tools/scan_idelay -m 500MHz Warning: clock not locked 00 5554 aaa8 aaa8 5554 True False 01 5554 aaa8 aaa8 5554 True False 02 5554 aaa8 aaa8 5554 True False 03 5554 aaa8 aaa8 5554 True False 04 5554 aaa8 aaa8 5554 True False 05 5553 aaa8 aaa8 5554 False False 06 155c aaa8 e659 5554 False False 07 1554 aaa8 eaa8 5554 False False 08 945e aaa8 7fa7 5554 False False 09 a85a aaa8 6e36 5554 False False 0a aaa8 aaa8 5554 5554 True True 0b aaa8 aaa8 5554 5554 True True 0c aaa8 aaa8 5554 5554 True True 0d aaa8 aaa8 5554 5554 True True 0e aaa8 aaa8 5554 5554 True True 0f aaa8 aaa8 5554 5554 True True 10 aaa8 aaa8 5554 5554 True True 11 aaa8 aaa8 5554 5553 False False 12 aaa8 cc53 5554 15d9 False False 13 aaa8 eaa8 5554 1554 False False 14 aaa8 b03f 5554 94cc False False 15 aaa8 6769 5554 aa97 False False 16 aaa8 5554 5554 aaa8 True False 17 aaa8 5554 5554 aaa8 True False 18 aaa8 5554 5554 aaa8 True False 19 aaa8 5554 5554 aaa8 True False |
...
The -m parameter selects the timing mode, and for frequencies other than the default 500MHz a second -f parameter can be used to set the expected range of delays. For example (output abbreviated):
| Code Block | ||
|---|---|---|
| ||
$ tools/scan_idelay -m 352MHz -f 352 Only scanning 32 of 37 steps 00 5554 aaa8 aaa8 5554 True False ... 0a aaa8 aaa8 5554 5554 False False 0b aaa8 aaa8 5554 5554 True True ... 11 aaa8 aaa8 5554 5554 True True 12 aaa8 aaab 5554 4cee False False 13 aaa8 eaa8 5554 1554 True False ... 1f aaa8 5554 5554 aaa8 True False |
...