The procedure for bringing up the Multi-Bunch Feedback processor system for the first time in a new environment is a little involved and is documented below.

Loose ends:

Task report

pages	79857402
columns	description,assignee

Preparing Hardware

Hardware

...

The MBF control system at Diamond consists of the following hardware:

Hardware	Vendor	Part Number	Description
VT814	Vadatech	VT814-200-000-000	2U μTCA.4 chassis with 6 AMC slots
UTC002		UTC002-210-801-100	μTCA MCH (MicroTCA Carrier Hub) with support for 8 lanes of PCIe gen 3.
AMC720		AMC720-122-010-000	Intel Xeon general purpose AMC processor card with 16GB RAM, 30GB SSD.
AMC525		AMC525-012-023-000	AMC FPGA carrier for dual HPC FMC with Virtex-7 690.
FMC-500M	Innovative Integration	80281-2-L0 FMC-500	FMC module with dual 500 Ms/s 14-bit ADC, dual 1200 Ms/s 16-bit DAC.
CTI-FMC-DIO-5chttl	Creotech	Open Hardware (Creotech, Sevensols)		FMC module with five channels of TTL I/O.

Populating the crate

The two FMC cards must be placed in the AMC 525 carrier in the correct slots, with the Digital I/O card in slot 0 and the FMC 500 in slot 1 (more details and pictures in this section). See the image below to verify the correct configuration:
Image Added
Note that because of an address conflict on the shared IPMI I²C bus between the programmable input threshold DAC on the CTI-FMC-DIO and a temperature sensor on the AMC525

...

(and thus a MTCA IPMI alert which won't go away), it is necessary to make a modification to the DIO card to reassign the address to an unused address

...

. This done by cutting the track to ADDR1:

Image Modified

Populating the crate

The two FMC cards must be placed in the AMC 525 carrier in the correct slots, with the Digital I/O card in slot 0 and the FMC 500 in slot 1. See the image below to verify the correct configuration:
Image Removed
Note that in order to avoid an I²C collision with an AMC temperature sensor (and thus a MTCA IPMI alert which won't go away) it is necessary to cut a line on the Digital I/O card (is this documented anywhere?
When placing the AMC cards in the crate it seems that it is necessary to be careful about which cards are (logically) adjacent to the processor card. To avoid problems, at DLS we have install the processor card in slot 6 and the carrier cards in slots 2 and 3 (note that these slot numbers don't correspond to the PCIe addresses which we'll encounter later). Specifically, it seems that with the processor card in slot 6 we need to avoid placing a carrier card in slot 5, as otherwise the processor card When placing the AMC cards in the crate it seems that it is necessary to be careful about which cards are (logically) adjacent to the processor card. To avoid problems, at DLS we installed the processor card in slot 6 and the carrier cards in slots 2 and 3 (note that these slot numbers don't correspond to the PCIe addresses which we'll encounter later). Specifically, it seems that with the processor card in slot 6 we need to avoid placing a carrier card in slot 5, as otherwise the processor card tries to boot from non-existent mass storage in this slot!
At this point we recommend its a good idea to ensure that all serial ports are configured with the same data rate. The default line speed is 115200 8N1 for all ports except for the CPU console which defaults to 9600 8N1.
Now e-keying must be configured.

...

These are all SSMC sockets. The connectors should not be tighter than finger tight, our experience is that over-tightening can use the inner conductor to lose contact. The official documentation (see page 26) recommends 0.2 Nm coupling torque.

Port	Dir	Signal	Description
DAC 1 OUT- DAC 1 OUT+	Out	±1V into 50Ω, DC to 500 MHz	Channel 1 differential output. Outputs are driven directly by a LMH6554 amplifier. Unused outputs should probably be terminated into 50Ω.
DAC 0 OUT- DAC 0 OUT+	Out	±1V into 50Ω, DC to 500 MHz	Channel 0 differential output.
CLK IN	In	±0.3 to 3.3V into 50Ω, 250 to 500 MHz.	Machine RF clock.
CLK OUT	Unused (except for development and debugging)
EXT TRG	In	0 to 3.3V into 50Ω, threshold at 1.2V.	To ensure synchronisation with the machine revolution a fast rising edge is expected at machine revolution frequency. This signal is synchronised to once during startup, and is sampled continuously thereafter.
ADC 0	In	±1V into 50Ω, DC to 250 MHz	Channel 0 single ended input.
ADC 1	In	±1V into 50Ω, DC to 250 MHz	Channel 1 single ended input.

System Software Setup

At this point the MTCA crate and all cards should be in a position to power up. Now tangle with the BIOS and install your choice of Linux (we use RHEL 7 for this at DLSDLS, Debian 9 at the ESRF (see installation notes here)). No special drivers (except for the MBF driver, we'll get onto that) need to be installed, but you'll probably want IPMI administration configured.

...

Target	Description
`driver`	Builds kernel module `amc525_lmbfmbf.ko` in `build/kbuild-$(uname -r)`. Needs to be run on target system for module to be usable.
`insmod`	Ensures kernel module is built and runs `insmod` to add to the current kernel. Needs to be run on target system.
`driver-rpm`	Builds DKMS based RPM for driver. Can be run on any system, resulting RPM can be permanently installed on target system.

The details of how to manage driver installation are somewhat distribution specific. Once the module has been installed it should be automatically picked up when the FPGA is loaded.

Bringing up the FPGA

Michael Abbott The automation tools here need to be modified to be generic and the instruction here fixed

First of all the FPGA must be loaded onto the AMC card. The following commands ./load_fpga script does the work: this must be run on the target system will do the necessary workAMC720 processor card:

Code Block

language	bash

BIT_FILE=amc525_lmbf.bit
ip=192.168.40.200
scp "$BIT_FILE" root@$ip:/tmp/amc525_lmbf.bit
ssh -x root@$ip amc525_lbtool fpga_load /tmp/amc525_lmbf.bit

...

./load_fpga -f path/to/amc525_mbf.bit

(The -f bit-file argument can be omitted if the bit file is in the working directory.)

At this point run lspci -v, and you should see something like the following:

Code Block

language	text

04:00.0 Signal processing controller: Xilinx Corporation FPGA Card XC7VX690T
    Subsystem: Xilinx Corporation Device 0007
    Physical Slot: 0
    Flags: bus master, fast devsel, latency 0,
IRQ 70     Memory at fe300000<unassigned> (64-bit, non-prefetchable) [size=1Mdisabled]
    Memory at fe400000<unassigned> (64-bit, non-prefetchable) [size=64Kdisabled]
    Capabilities: <access denied>
    Kernel driver in use: amc525_lmbf

Michael Abbott Need to capture the above on a freshly booted system

...

If we see this then the FPGA has been successfully loaded and the PCIe link is working correctly. Alas, Note however that the PCIe IO memory cannot be mapped at this stage because it is too late for the BIOS to identify it: hence the Memory at <unassigned> lines (we see Capabilities: <access denied> merely because we're not running lspci as root). Now reboot the processor card and re-run lspci -v and we should now see:

Code Block

language	text

04:00.0 Signal processing controller: Xilinx Corporation FPGA Card XC7VX690T
    Subsystem: Xilinx Corporation Device 0007
    Physical Slot: 0
    Flags: bus master, fast devsel, latency 0, IRQ 70
    Memory at fe300000 (64-bit, non-prefetchable) [size=1M]
    Memory at fe400000 (64-bit, non-prefetchable) [size=64K]
    Capabilities: <access denied>
    Kernel driver in use: amc525_lmbfmbf

If the kernel driver has not yet been installed, now install it and run ls /dev. The following files should be present:

Device Node	Description
`amc525_lmbfmbf.0.reg`	Register interface device node. Used for control of MBF system.
`amc525_lmbfmbf.0.ddr0`	Access to fast memory buffer for bunch by bunch capture readout.
`amc525_lmbfmbf.0.ddr1`	Access to slow memory buffer for detector readout.
`amc525_lmbfmbf/`	Directory containing physical address links to device nodes.

...

Note also that reloading the FPGA while the software is running is likely to trigger a kernel panic on the main processor (interrupted PCIe transactions don't seem to be handled well!), and hot-swapping the AMC is equally likely to do this.

Bringing up Software

Measuring timing skew

...

Preparing IOC Configuration

To prepare to run a new IOC a new configuration and startup file in the iocs/ directory:

Code Block

language	text

$ tools/scan_idelay -m 500MHz
Warning: clock not locked
00 5554 aaa8 aaa8 5554 True False
01 5554 aaa8 aaa8 5554 True False
02 5554 aaa8 aaa8 5554 True False
03 5554 aaa8 aaa8 5554 True False
04 5554 aaa8 aaa8 5554 True False
05 5553 aaa8 aaa8 5554 False False
06 155c aaa8 e659 5554 False False
07 1554 aaa8 eaa8 5554 False False
08 945e aaa8 7fa7 5554 False False
09 a85a aaa8 6e36 5554 False False
0a aaa8 aaa8 5554 5554 True True
0b aaa8 aaa8 5554 5554 True True
0c aaa8 aaa8 5554 5554 True True
0d aaa8 aaa8 5554 5554 True True
0e aaa8 aaa8 5554 5554 True True
0f aaa8 aaa8 5554 5554 True True
10 aaa8 aaa8 5554 5554 True True
11 aaa8 aaa8 5554 5553 False False
12 aaa8 cc53 5554 15d9 False False
13 aaa8 eaa8 5554 1554 False False
14 aaa8 b03f 5554 94cc False False
15 aaa8 6769 5554 aa97 False False
16 aaa8 5554 5554 aaa8 True False
17 aaa8 5554 5554 aaa8 True False
18 aaa8 5554 5554 aaa8 True False
19 aaa8 5554 5554 aaa8 True False

If no clock is connected, or the input clock frequency does not match the programmed mode, or if Passthrough mode is selected then the Warning: clock not locked message will be generated. Each line of output represents a delay (from data to clock, in steps of 78ps), and the only lines of interest are those ending True True. From the example above any timing selection in the range 10 to 16 will be valid; our default is 12.

The -m parameter selects the timing mode, and for frequencies other than the default 500MHz a second -f parameter can be used to set the expected range of delays. For example (output abbreviated):

Code Block

language	text

$ tools/scan_idelay -m 352MHz -f 352
Only scanning 32 of 37 steps
00 5554 aaa8 aaa8 5554 True False
...
0a aaa8 aaa8 5554 5554 False False
0b aaa8 aaa8 5554 5554 True True
...
11 aaa8 aaa8 5554 5554 True True
12 aaa8 aaab 5554 4cee False False
13 aaa8 eaa8 5554 1554 True False
...
1f aaa8 5554 5554 aaa8 True False

Note that we only have 32 steps of delay available for a maximum delay of 2.5ns.

Preparing IOC Configuration

To prepare to run a new IOC a new configuration and startup file in the iocs/ directory:

Code Block

language	bash

ioc_name=new-ioc-name
cd iocs
ln -s start-ioc $ioc_name
cp TS-DI-TMBF-02.config $ioc_name

Now edit the .config file. The options are as follows:

...

procserv_port

...

device_address

...

adc_idelay

...

dio_termination

...

clock_mode

...

This determines how the PLL is initialised. The following timing modes are supported:

Mode Name	Description
500MHz	PLL is locked to 500MHz RF input.
352MHz	PLL is locked to 352MHz RF input.
Passthrough	RF clock is passed through without locking and regeneration.

...

epics_name

...

axis0_name

...

axis1_name

...

lmbf_mode

...

bunches_per_turn

...

revolution_frequency

...

lmbf_fir_offset

...

mms_poll_interval

...

persistence_file

...

persistence_interval

...

pv_log_array_length

...

memory_readout_length

...

detector_length

...

data_port

...

bash

ioc_name=new-ioc-name
cd iocs
ln -s start-ioc $ioc_name
cp TS-DI-TMBF-02.config $ioc_name

Now edit the .config file. The options are as follows:

Key Description

procserv_port

If the IOC is started using the ./start_ioc script this defines the telnet port that procServ will use. Otherwise this key is ignored.

device_address

This is of the form pci-0000:nn:00.0 where nn identifies the slot where the appropriate AMC525 card is inserted (though the slot numbers and PCIe numbers don't match). Inspect /dev/amc525_mbf/ for currently recognised cards.

dio_termination

If 50Ω termination is wanted for the three digital I/O trigger inputs then set this to 7, otherwise set to 0 for high impedance termination.

clock_mode

This determines how the PLL is initialised. The following timing modes are supported:

Mode Name	Description
499_682	PLL is locked to 499.682 MHz RF input.
352_202	PLL is locked to 352.202 MHz RF input.
352_372	PLL is locked to 352.372 MHz RF input.
Passthrough	RF clock is passed through without locking and regeneration.

Alternatively this key can be set to sequence of six numbers defining the PLL configuration, for example the following is equivalent to 499_682:

Code Block

language	text

13 0 5 2 381 61

See the documentation for PLL_ratios in python/mbf/driver/setup_pll.py for more detail.

epics_name

This determines the top level device name and generally should match $ioc_name determined above.

axis0_name

Name of channel 0. Typically X for transverse mode, I for longitudinal mode.

axis1_name

Name of channel 1. Typically Y for transverse mode, Q for longitudinal mode.

lmbf_mode

Set to 0 if operating in transverse mode, set to 1 if longitudinal mode. This will determine how the FPGA is configured and some details of the behaviour of the IOC.

bunches_per_turn

Set to the number of RF buckets per machine revolution. Must be no more than 1024 (with the current FPGA build), and there may be problems with particularly small values.

revolution_frequency

Set to machine revolution frequency in Hz. Only used for time estimates in display.

lmbf_fir_offset

Adjustment of FIR coupling between I and Q axes in LMBF mode.

mms_poll_interval

Used to control polling frequency for MMS readout. If MMS overrun is reported this number needs to be reduced, but check CPU usage with top. Reading MMS data is time consuming.

persistence_file

This should be an absolute path to a writeable location where the IOC can save the persistent state of all of its PVs. It is wise to keep this file backed up and archived as the configuration of MBF can be quite complex and potentially difficult to recreate.

persistence_interval

This determines the interval (in seconds) between checks for writing the persistent state. Too small a value will generate a lot of writes, too large a value can result in lost configuration state if the IOC is forcibly restarted.

pv_log_array_length

Determines how many points of changed waveforms are logged.

memory_readout_length

Determines the length of the memory readout waveform PV.

detector_length

Determines the length of the detector readout waveform PV.

data_port

Determines socket number for fast data readout.

Now the IOC can be started, either by running iocs/$ioc_name or preferably ./start_ioc $ioc_name to run the IOC under procServ.

Note: If the environmental variable EPICS_CA_MAX_ARRAY_BYTES is not defined in your environment, it is advised to set it to a higher value than the default one (16384 bytes) for both the client and the server. At DLS, it is set to 6000000 in all profiles.

Initial State

When MBF is started for the first time, or when the state file is deleted, the initial state of nearly all PVs is set to zero. Before trying to use the system the input and output compensation filters should be reset to passthrough and the control gains should be set to 1. The tools/initial-state script will perform this operation:

Code Block

language	bash

tools/initial-state $ioc_name

Also note that every time MBF is restarted, the DAC output is disabled. This is deliberate to avoid accidentally driving onto the beam after a restart, but means that after every restart the :DAC:ENABLE_S pvs must be manually set.

Helper Tools

A number of Helper Tools are documented on the linked page.

Restarting the IOC

The IOC can be restarted at any time. If however the crate is power-cycled then the procedure is slightly more involved:

Power on crate, wait for PC to boot
Reload FPGA with ./load_fpga script run on PC
Reboot PC (without power cycling) to pick up full PCIe functionality
Start IOC
Renable DAC outputs, rearm triggers

Version	Old Version 16	New Version Current
Changes made by	Michael Abbott	Guenther Rehm
Saved on	Mar 08, 2018	Nov 04, 2021

Content Comparison

Versions Compared

Key

Preparing Hardware

Hardware

Populating the crate

Populating the crate

System Software Setup

Bringing up the FPGA

Bringing up Software

Measuring timing skew

Preparing IOC Configuration

Preparing IOC Configuration

Initial State

Helper Tools

Restarting the IOC