...

Preparing Hardware

Populating the crate

•  Michael Abbott The hardware list is where...?
•  Michael Abbott The DIO mod is documented where?

• The two FMC cards must be placed in the AMC 525 carrier in the correct slots, with the Digital I/O card in slot 0 and the FMC 500 in slot 1.  See the image below to verify the correct configuration:
  Image Removed
• Note that in order to avoid an I₂C collision with an AMC temperature sensor (and thus a MTCA IPMI alert which won't go away) it is necessary to cut a line on the Digital I/O card (is this documented anywhere?
• When placing the AMC cards in the crate it seems that it is necessary to be careful about which cards are (logically) adjacent to the processor card.  To avoid problems, at DLS we have install the processor card in slot 6 and the carrier cards in slots 2 and 3 (note that these slot numbers don't correspond to the PCIe addresses which we'll encounter later).  Specifically, it seems that with the processor card in slot 6 we need to avoid placing a carrier card in slot 5, as otherwise the processor card tries to boot from non-existent mass storage in this slot!
• At this point we recommend that all serial ports are configured with the same data rate.  The default line speed is 115200 8N1 for all ports except for the CPU console which defaults to 9600 8N1.
• Now e-keying must be configured.

Table of relevant serial ports

...

AMC720

(CPU)

...

E-key configuration

Once the hardware has been configured as above the crate can be powered on and the e-keying for the CPU and AMC525 must be set up.  This will be particularly important to ensure that the PCIe link works properly.

CPU e-key

Payload SOL Port   : Serial1
Gbe Management Port: back0
Shutdown delay     : 30s

Commands:
  ekey          - Configure Electronic Keying
  lan           - Configure LAN Parameters
  plser         - Configure Payload Serial Port
  mgtport       - Configure Gbe Management Port
  wdcfg         - Configure Watchdog Parameters
  shutdowndelay - Configure shut down delay
> ekey

E-Keying configuration
*  1) Gigabit Ethernet - Port 0        *  2) Gigabit Ethernet - Port 1        
*  3) PCIe Root -  Ports 4-11             4) PCIe Root -  Ports 4-7           
   5) PCIe Root -  Ports 8-11             6) SATA Client - Port 2             
   7) SATA Client - Port 3             

Commands:
  1-7 to toggle option.
  save, cancel

Ensure that the following are configured:

...

Note: Port 1 is enabled but not used.

AMC e-key

Connect to the MGT-RS232 port on the AMC525 front panel, and type ekey at the prompt.  Adjust for the following settings:

...

Hardware

Here we focus entirely on the control system.  A complete multi-bunch feedback system consists of a Beam Position Monitor button block, typically an RF hybrid assembly to convert button signals to scaled position and intensity, an RF front end to down-convert button signals at an intermediate frequency (1.5GHz at Diamond) before the signal is sampled and processed by the control system.  The output from the control system is then up-converted if necessary, amplified, and driven onto the beam, either through strip-lines (for transverse operation) or through a dedicated RF cavity for longitudinal operation.

The MBF control system at Diamond consists of the following hardware:

Note that because of an address conflict on the shared IPMI I²C bus between the programmable input threshold DAC on the CTI-FMC-DIO and a temperature sensor on the AMC525, it is necessary to make a modification to the DIO card.  Fortunately the default DAC output does not need changing, so it is enough to cut the control line to the DAC.

•  Michael Abbott The DIO mod is documented where?
  

Populating the crate

• The two FMC cards must be placed in the AMC 525 carrier in the correct slots, with the Digital I/O card in slot 0 and the FMC 500 in slot 1.  See the image below to verify the correct configuration:
  Image Added
• Note that in order to avoid an I₂C collision with an AMC temperature sensor (and thus a MTCA IPMI alert which won't go away) it is necessary to cut a line on the Digital I/O card (is this documented anywhere?
• When placing the AMC cards in the crate it seems that it is necessary to be careful about which cards are (logically) adjacent to the processor card.  To avoid problems, at DLS we have install the processor card in slot 6 and the carrier cards in slots 2 and 3 (note that these slot numbers don't correspond to the PCIe addresses which we'll encounter later).  Specifically, it seems that with the processor card in slot 6 we need to avoid placing a carrier card in slot 5, as otherwise the processor card tries to boot from non-existent mass storage in this slot!
• At this point we recommend that all serial ports are configured with the same data rate.  The default line speed is 115200 8N1 for all ports except for the CPU console which defaults to 9600 8N1.
• Now e-keying must be configured.

Table of relevant serial ports

E-key configuration

Once the hardware has been configured as above the crate can be powered on and the e-keying for the CPU and AMC525 must be set up.  This will be particularly important to ensure that the PCIe link works properly.

CPU e-key

Payload SOL Port   : Serial1
Gbe Management Port: back0
Shutdown delay     : 30s

Commands:
  ekey          - Configure Electronic Keying
  lan           - Configure LAN Parameters
  plser         - Configure Payload Serial Port
  mgtport       - Configure Gbe Management Port
  wdcfg         - Configure Watchdog Parameters
  shutdowndelay - Configure shut down delay
> ekey

E-Keying configuration
*  1) Gigabit Ethernet - Port 0        *  2) Gigabit Ethernet - Port 1        
*  3) PCIe Root -  Ports 4-11             4) PCIe Root -  Ports 4-7           
   5) PCIe Root -  Ports 8-11                 46) SATA Client - Port 2             
   57) SATA ServerClient - Port 3             

Commands:
6) SATA Client 1-7 Portto 3toggle option.
  save,         
   7) SAS - Port 2                        8) SAS - Port 3                     
   9) PCIe Root -  Ports 4-11            10) PCIe Root -  Ports 4-7cancel

Ensure that the following are configured:

Note: Port 1 is enabled but not used.

AMC e-key

Connect to the MGT-RS232 port on the AMC525 front panel, and type ekey at the prompt.  Adjust for the following settings:

AMC525 > ekey

E-Keying configuration
*  1) Gigabit Ethernet - Port 0        *   
  112) PCIeGigabit RootEthernet - Port Ports1 8-11       
  * 123) PCIeSATA NodeServer - Port 2  Ports 4-11             134) PCIeSATA NodeClient - Port 2   Ports 4-7         
   145) PCIeSATA NodeServer - Port Ports 8-113                156) SATA SRIOClient 3.125- GbaudPort -3 Ports 4-7       16) SRIO 2.5  
Gbaud - Ports 4-7) SAS - Port 2    17) SRIO 1.25  Gbaud - Ports 4-7       18) SRIO 3.125 Gbaud - Ports 8-11) SAS - Port 3         19) SRIO 2.5   Gbaud - Ports 8-11    
 20) SRIO 1.259) PCIe GbaudRoot -  Ports 84-11        21) XAUI - Port 4-710) PCIe Root -  Ports 4-7           
  2211) PCIe XAUIRoot -  PortPorts 8-11          * 12) PCIe Node -  Ports 4-11    Commands:   1-22 to toggle option.
  save, cancel

Ensure that the following are configured:

...

Also the vadj setting must be configured as otherwise the FMC cards will not be powered on.

  FMC0 EEPROM is invalid / missing
  FMC1 VADJ range is 0.00V - 0.00V
  VADJ setting is 1.80V
AMC525 > vadj
0: 1.80V
c: Cancel
Select Voltage (0-0, c): 

If VADJ is not already set to 1.80V select 0 above to configure.

Cabling

Connections to Digital I/O card.

These are Limo ERA.00.250 sockets taking FFS.00.250 connectors.  All inputs are configured as 3.3V TTL high impedance (though 50Ω termination can be configured in software if desired), all outputs are driven as 3.3V TTL.  All inputs are acted on once per machine revolution in response to a rising edge, but are continuously sampled (at machine RF frequency).

...

Connections to FMC500 card

These are all SSMC sockets.  The connectors should not be tighter than finger tight, our experience is that over-tightening can use the inner conductor to lose contact.  The official documentation (see page 26) recommends 0.2 Nm coupling torque.

...

DAC 1 OUT-

DAC 1 OUT+

...

DAC 0 OUT-

DAC 0 OUT+

...

System Software Setup

At this point the MTCA crate and all cards should be in a position to power up.  Now tangle with the BIOS and install your choice of Linux (we use RHEL 7 for this at DLS).  No special drivers (except for the MBF driver, we'll get onto that) need to be installed, but you'll probably want IPMI administration configured.

Networking

Two out of the four available network ports need to be configured:

...

The administration network connects to the MCH and the P2040 CPU on the AMC card.  By default the following addresses are assigned:

...

ipmitool -I lan -H 192.168.40.250 -U '' -P '' shell

...

AMC525.  If more than one AMC525 is installed the network address will need to be changed by logging on through the serial console (user root, no password) and editing /etc/network/interfaces.  Note that this CPU sees 4 network ports, but only eth0 is used (eth1 connects to MTCA port 1 which we're not using, and eth2 and eth3 connect to the FPGA where they will be ignored).

This interface will be used for programming the FPGA.

We can now validate the system by logging on to the MCH and AMC525 cards through the internal network.

MBF Kernel Driver

At some point before running the software the MBF kernel driver will need to be installed on the AMC720 processor. The following top level makefile targets may be helpful:

...

The details of how to manage driver installation are somewhat distribution specific.  Once the module has been installed it should be automatically picked up when the FPGA is loaded.

Bringing up the FPGA

•  Michael Abbott The automation tools here need to be modified to be generic and the instruction here fixed

First of all the FPGA must be loaded onto the AMC card.  The following commands run on the target system will do the necessary work:

BIT_FILE=amc525_lmbf.bit
ip=192.168.40.200
scp "$BIT_FILE" root@$ip:/tmp/amc525_lmbf.bit
ssh -x root@$ip amc525_lbtool fpga_load /tmp/amc525_lmbf.bit

At this point run lspci -v, and you should see something like the following:

04:00.0 Signal processing controller: Xilinx Corporation FPGA Card XC7VX690T
    Subsystem: Xilinx Corporation Device 0007
    Physical Slot: 0
    Flags: bus master, fast devsel, latency 0, IRQ 70
    Memory at fe300000 (64-bit, non-prefetchable) [size=1M]
    Memory at fe400000 (64-bit, non-prefetchable) [size=64K]
    Capabilities: <access denied>
    Kernel driver in use: amc525_lmbf

•  Michael Abbott Need to capture the above on a freshly booted system

If we see this then the FPGA has been successfully loaded and the PCIe link is working correctly.  Alas, the PCIe IO memory cannot be mapped at this stage because it is too late for the BIOS to identify it.  Now reboot the processor card and re-run lspci -v and we should now see:

04:00.0 Signal processing controller: Xilinx Corporation FPGA Card XC7VX690T
    Subsystem: Xilinx Corporation Device 0007
    Physical Slot: 0
    Flags: bus master, fast devsel, latency 0, IRQ 70
    Memory at fe300000 (64-bit, non-prefetchable) [size=1M]
    Memory at fe400000 (64-bit, non-prefetchable) [size=64K]
    Capabilities: <access denied>
    Kernel driver in use: amc525_lmbf

If the kernel driver has not yet been installed, now install it and run ls /dev.  The following files should be present:

...

The system is now ready to run the software.  Note that once the processor has been booted with a loaded FPGA present it won't need to be rebooted again if a fresh FPGA image is loaded until the whole system is power-cycled.

Note also that reloading the FPGA while the software is running is likely to trigger a kernel panic on the main processor (interrupted PCIe transactions don't seem to be handled well!), and hot-swapping the AMC is equally likely to do this.

...

13) PCIe Node -  Ports 4-7             14) PCIe Node -  Ports 8-11          
  15) SRIO 3.125 Gbaud - Ports 4-7       16) SRIO 2.5   Gbaud - Ports 4-7     
  17) SRIO 1.25  Gbaud - Ports 4-7       18) SRIO 3.125 Gbaud - Ports 8-11    
  19) SRIO 2.5   Gbaud - Ports 8-11      20) SRIO 1.25  Gbaud - Ports 8-11    
  21) XAUI - Port 4-7                    22) XAUI - Port 8-11                 

Commands:
  1-22 to toggle option.
  save, cancel

Ensure that the following are configured:

Also the vadj setting must be configured as otherwise the FMC cards will not be powered on.

  FMC0 EEPROM is invalid / missing
  FMC1 VADJ range is 0.00V - 0.00V
  VADJ setting is 1.80V
AMC525 > vadj
0: 1.80V
c: Cancel
Select Voltage (0-0, c): 

If VADJ is not already set to 1.80V select 0 above to configure.

Cabling

Connections to Digital I/O card.

These are Limo ERA.00.250 sockets taking FFS.00.250 connectors.  All inputs are configured as 3.3V TTL high impedance (though 50Ω termination can be configured in software if desired), all outputs are driven as 3.3V TTL.  All inputs are acted on once per machine revolution in response to a rising edge, but are continuously sampled (at machine RF frequency).

Connections to FMC500 card

These are all SSMC sockets.  The connectors should not be tighter than finger tight, our experience is that over-tightening can use the inner conductor to lose contact.  The official documentation (see page 26) recommends 0.2 Nm coupling torque.

System Software Setup

At this point the MTCA crate and all cards should be in a position to power up.  Now tangle with the BIOS and install your choice of Linux (we use RHEL 7 for this at DLS).  No special drivers (except for the MBF driver, we'll get onto that) need to be installed, but you'll probably want IPMI administration configured.

Networking

Two out of the four available network ports need to be configured:

The administration network connects to the MCH and the P2040 CPU on the AMC card.  By default the following addresses are assigned:

ipmitool -I lan -H 192.168.40.250 -U '' -P '' shell

We can now validate the system by logging on to the MCH and AMC525 cards through the internal network.

MBF Kernel Driver

At some point before running the software the MBF kernel driver will need to be installed on the AMC720 processor. The following top level makefile targets may be helpful:

The details of how to manage driver installation are somewhat distribution specific.  Once the module has been installed it should be automatically picked up when the FPGA is loaded.

Bringing up the FPGA

•  Michael Abbott The automation tools here need to be modified to be generic and the instruction here fixed

First of all the FPGA must be loaded onto the AMC card.  The following commands run on the target system will do the necessary work:

BIT_FILE=amc525_lmbf.bit
ip=192.168.40.200
scp "$BIT_FILE" root@$ip:/tmp/amc525_lmbf.bit
ssh -x root@$ip amc525_lbtool fpga_load /tmp/amc525_lmbf.bit

At this point run lspci -v, and you should see something like the following:

04:00.0 Signal processing controller: Xilinx Corporation FPGA Card XC7VX690T
    Subsystem: Xilinx Corporation Device 0007
    Physical Slot: 0
    Flags: bus master, fast devsel, latency 0, IRQ 70
    Memory at fe300000 (64-bit, non-prefetchable) [size=1M]
    Memory at fe400000 (64-bit, non-prefetchable) [size=64K]
    Capabilities: <access denied>
    Kernel driver in use: amc525_lmbf

•  Michael Abbott Need to capture the above on a freshly booted system

If we see this then the FPGA has been successfully loaded and the PCIe link is working correctly.  Alas, the PCIe IO memory cannot be mapped at this stage because it is too late for the BIOS to identify it.  Now reboot the processor card and re-run lspci -v and we should now see:

04:00.0 Signal processing controller: Xilinx Corporation FPGA Card XC7VX690T
    Subsystem: Xilinx Corporation Device 0007
    Physical Slot: 0
    Flags: bus master, fast devsel, latency 0, IRQ 70
    Memory at fe300000 (64-bit, non-prefetchable) [size=1M]
    Memory at fe400000 (64-bit, non-prefetchable) [size=64K]
    Capabilities: <access denied>
    Kernel driver in use: amc525_lmbf

If the kernel driver has not yet been installed, now install it and run ls /dev.  The following files should be present:

The system is now ready to run the software.  Note that once the processor has been booted with a loaded FPGA present it won't need to be rebooted again if a fresh FPGA image is loaded until the whole system is power-cycled.

Note also that reloading the FPGA while the software is running is likely to trigger a kernel panic on the main processor (interrupted PCIe transactions don't seem to be handled well!), and hot-swapping the AMC is equally likely to do this.

Bringing up Software

Measuring timing skew

Each time a new AMC card is commissioned it may be necessary to measure the timing skew from the ADC to the FPGA, and should also be checked when the FPGA image is changed.  This test can be run without any clocks connected (depending on the selected clocking mode) and needs no data input.

$ tools/scan_idelay -m 500MHz
Warning: clock not locked
00 5554 aaa8 aaa8 5554 True False
01 5554 aaa8 aaa8 5554 True False
02 5554 aaa8 aaa8 5554 True False
03 5554 aaa8 aaa8 5554 True False
04 5554 aaa8 aaa8 5554 True False
05 5553 aaa8 aaa8 5554 False False
06 155c aaa8 e659 5554 False False
07 1554 aaa8 eaa8 5554 False False
08 945e aaa8 7fa7 5554 False False
09 a85a aaa8 6e36 5554 False False
0a aaa8 aaa8 5554 5554 True True
0b aaa8 aaa8 5554 5554 True True
0c aaa8 aaa8 5554 5554 True True
0d aaa8 aaa8 5554 5554 True True
0e aaa8 aaa8 5554 5554 True True
0f aaa8 aaa8 5554 5554 True True
10 aaa8 aaa8 5554 5554 True True
11 aaa8 aaa8 5554 5553 False False
12 aaa8 cc53 5554 15d9 False False
13 aaa8 eaa8 5554 1554 False False
14 aaa8 b03f 5554 94cc False False
15 aaa8 6769 5554 aa97 False False
16 aaa8 5554 5554 aaa8 True False
17 aaa8 5554 5554 aaa8 True False
18 aaa8 5554 5554 aaa8 True False
19 aaa8 5554 5554 aaa8 True False

If no clock is connected, or the input clock frequency does not match the programmed mode, or if Passthrough mode is selected then the Warning: clock not locked message will be generated.  Each line of output represents a delay (from data to clock, in steps of 78ps), and the only lines of interest are those ending True True.  From the example above any timing selection in the range 10 to 16 will be valid; our default is 12.

The -m parameter selects the timing mode, and for frequencies other than the default 500MHz a second -f parameter can be used to set the expected range of delays.  For example (output abbreviated):

$ tools/scan_idelay -m 352MHz -f 352
Only scanning 32 of 37 steps
00 5554 aaa8 aaa8 5554 True False
...
0a aaa8 aaa8 5554 5554 False False
0b aaa8 aaa8 5554 5554 True True
...
11 aaa8 aaa8 5554 5554 True True
12 aaa8 aaab 5554 4cee False False
13 aaa8 eaa8 5554 1554 True False
...
1f aaa8 5554 5554 aaa8 True False

Note that we only have 32 steps of delay available for a maximum delay of 2.5ns.