...
The procedure for bringing up the Multi-Bunch Feedback processor system for the first time in a new environment is a little involved and is documented below.
Loose ends:
| Task report | ||||
|---|---|---|---|---|
|
Preparing Hardware
Hardware
...
The MBF control system at Diamond consists of the following hardware:
| Hardware | Vendor | Part Number | Description | |
|---|---|---|---|---|
| VT814 | Vadatech | VT814-200-000-000 | 2U μTCA.4 chassis with 6 AMC slots | |
| UTC002 | UTC002-210-801-100 | μTCA MCH (MicroTCA Carrier Hub) with support for 8 lanes of PCIe gen 3. | ||
| AMC720 | AMC720-122-010-000 | Intel Xeon general purpose AMC processor card with 16GB RAM, 30GB SSD. | ||
| AMC525 | AMC525-012-023-000 | AMC FPGA carrier for dual HPC FMC with Virtex-7 690. | ||
| FMC-500M | Innovative Integration | 80281-2-L0 FMC-500 | FMC module with dual 500 Ms/s 14-bit ADC, dual 1200 Ms/s 16-bit DAC. | |
| CTI-FMC-DIO-5chttl | Creotech | Open Hardware (Creotech, Sevensols) | FMC module with five channels of TTL I/O. |
...
- The two FMC cards must be placed in the AMC 525 carrier in the correct slots, with the Digital I/O card in slot 0 and the FMC 500 in slot 1 (more details and pictures in this section). See the image below to verify the correct configuration:
Note that because of an address conflict on the shared IPMI I2C bus between the programmable input threshold DAC on the CTI-FMC-DIO and a temperature sensor on the AMC525 (and thus a MTCA IPMI alert which won't go away), it is necessary to make a modification to the DIO card to reassign the address to an unused address. This done by cutting the track to ADDR1:
- When placing the AMC cards in the crate it seems that it is necessary to be careful about which cards are (logically) adjacent to the processor card. To avoid problems, at DLS we have install installed the processor card in slot 6 and the carrier cards in slots 2 and 3 (note that these slot numbers don't correspond to the PCIe addresses which we'll encounter later). Specifically, it seems that with the processor card in slot 6 we need to avoid placing a carrier card in slot 5, as otherwise the processor card tries to boot from non-existent mass storage in this slot!
- At this point its a good idea to ensure that all serial ports are configured with the same data rate. The default line speed is 115200 8N1 for all ports except for the CPU console which defaults to 9600 8N1.
- Now e-keying must be configured.
...
At this point the MTCA crate and all cards should be in a position to power up. Now tangle with the BIOS and install your choice of Linux (we use RHEL 7 for this at DLSDLS, Debian 9 at the ESRF (see installation notes here)). No special drivers (except for the MBF driver, we'll get onto that) need to be installed, but you'll probably want IPMI administration configured.
...
| Target | Description |
|---|---|
driver | Builds kernel module amc525_lmbfmbf.ko in build/kbuild-$(uname -r). Needs to be run on target system for module to be usable. |
insmod | Ensures kernel module is built and runs insmod to add to the current kernel. Needs to be run on target system. |
driver-rpm | Builds DKMS based RPM for driver. Can be run on any system, resulting RPM can be permanently installed on target system. |
...
| Code Block | ||
|---|---|---|
| ||
./load_fpga -f path/to/amc525_lmbfmbf.bit |
(The -f bit-file argument can be omitted if the bit file is in the working directory.)
...
| Code Block | ||
|---|---|---|
| ||
04:00.0 Signal processing controller: Xilinx Corporation FPGA Card XC7VX690T
Subsystem: Xilinx Corporation Device 0007
Physical Slot: 0
Flags: bus master, fast devsel, latency 0, IRQ 70
Memory at fe300000<unassigned> (64-bit, non-prefetchable) [size=1Mdisabled]
Memory at fe400000<unassigned> (64-bit, non-prefetchable) [size=64Kdisabled]
Capabilities: <access denied>
Kernel driver in use: amc525_lmbf |
- Michael Abbott Need to capture the above on a freshly booted system to correctly show unreachable registers
...
If we see this then the FPGA has been successfully loaded and the PCIe link is working correctly. Alas, Note however that the PCIe IO memory cannot be mapped at this stage because it is too late for the BIOS to identify it: hence the Memory at <unassigned> lines (we see Capabilities: <access denied> merely because we're not running lspci as root). Now reboot the processor card and re-run lspci -v and we should now see:
| Code Block | ||
|---|---|---|
| ||
04:00.0 Signal processing controller: Xilinx Corporation FPGA Card XC7VX690T
Subsystem: Xilinx Corporation Device 0007
Physical Slot: 0
Flags: bus master, fast devsel, latency 0, IRQ 70
Memory at fe300000 (64-bit, non-prefetchable) [size=1M]
Memory at fe400000 (64-bit, non-prefetchable) [size=64K]
Capabilities: <access denied>
Kernel driver in use: amc525_lmbfmbf |
If the kernel driver has not yet been installed, now install it and run ls /dev. The following files should be present:
| Device Node | Description |
|---|---|
amc525_lmbfmbf.0.reg | Register interface device node. Used for control of MBF system. |
amc525_lmbfmbf.0.ddr0 | Access to fast memory buffer for bunch by bunch capture readout. |
amc525_lmbfmbf.0.ddr1 | Access to slow memory buffer for detector readout. |
amc525_lmbfmbf/ | Directory containing physical address links to device nodes. |
...
Note also that reloading the FPGA while the software is running is likely to trigger a kernel panic on the main processor (interrupted PCIe transactions don't seem to be handled well!), and hot-swapping the AMC is equally likely to do this.
Bringing up Software
Measuring timing skew
Each time a new AMC card is commissioned it may be necessary to measure the timing skew from the ADC to the FPGA, and should also be checked when the FPGA image is changed. This test can be run without any clocks connected (depending on the selected clocking mode) and needs no data input.
| Code Block | ||
|---|---|---|
| ||
$ tools/scan_idelay -m 499_682
Warning: clock not locked
00 5554 aaa8 aaa8 5554 True False
01 5554 aaa8 aaa8 5554 True False
02 5554 aaa8 aaa8 5554 True False
03 5554 aaa8 aaa8 5554 True False
04 5554 aaa8 aaa8 5554 True False
05 5553 aaa8 aaa8 5554 False False
06 155c aaa8 e659 5554 False False
07 1554 aaa8 eaa8 5554 False False
08 945e aaa8 7fa7 5554 False False
09 a85a aaa8 6e36 5554 False False
0a aaa8 aaa8 5554 5554 True True
0b aaa8 aaa8 5554 5554 True True
0c aaa8 aaa8 5554 5554 True True
0d aaa8 aaa8 5554 5554 True True
0e aaa8 aaa8 5554 5554 True True
0f aaa8 aaa8 5554 5554 True True
10 aaa8 aaa8 5554 5554 True True
11 aaa8 aaa8 5554 5553 False False
12 aaa8 cc53 5554 15d9 False False
13 aaa8 eaa8 5554 1554 False False
14 aaa8 b03f 5554 94cc False False
15 aaa8 6769 5554 aa97 False False
16 aaa8 5554 5554 aaa8 True False
17 aaa8 5554 5554 aaa8 True False
18 aaa8 5554 5554 aaa8 True False
19 aaa8 5554 5554 aaa8 True False |
If no clock is connected, or the input clock frequency does not match the programmed mode, or if Passthrough mode is selected then the Warning: clock not locked message will be generated. Each line of output represents a delay (from data to clock, in steps of 78ps), and the only lines of interest are those ending True True. From the example above any timing selection in the range 10 to 16 will be valid; our default is 12.
The -m parameter selects the timing mode, and for frequencies other than the default 499.682 MHz a second -f parameter can be used to set the expected range of delays. For example (output abbreviated):
| Code Block | ||
|---|---|---|
| ||
$ tools/scan_idelay -m 352_202 -f 352
Only scanning 32 of 37 steps
00 5554 aaa8 aaa8 5554 True False
...
0a aaa8 aaa8 5554 5554 False False
0b aaa8 aaa8 5554 5554 True True
...
11 aaa8 aaa8 5554 5554 True True
12 aaa8 aaab 5554 4cee False False
13 aaa8 eaa8 5554 1554 True False
...
1f aaa8 5554 5554 aaa8 True False |
Note that we only have 32 steps of delay available for a maximum delay of 2.5ns.
Preparing IOC Configuration
To prepare to run a new IOC a new configuration and startup file in the iocs/ directory:
...
| language | bash |
|---|
...
Preparing IOC Configuration
To prepare to run a new IOC a new configuration and startup file in the iocs/ directory:
| Code Block | ||
|---|---|---|
| ||
ioc_name=new-ioc-name
cd iocs
ln -s start-ioc $ioc_name
cp TS-DI-TMBF-02.config $ioc_name |
...
| Key | Description | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
procserv_port | If the IOC is started using the ./start_ioc script this defines the telnet port that procServ will use. Otherwise this key is ignored. | |||||||||||||||||
device_address | This is of the form pci-0000:nn:00.0 where nn identifies the slot where the appropriate AMC525 card is inserted (though the slot numbers and PCIe numbers don't match). Inspect /dev/amc525_lmbfmbf/ for currently recognised cards. | adc_idelay | This is the timing skew described in the previous section. | |||||||||||||||
dio_termination | If 50Ω termination is wanted for the three digital I/O trigger inputs then set this to 7to 7, otherwise set to 0 for high impedance termination. | |||||||||||||||||
clock_mode | This determines how the PLL is initialised. The following timing modes are supported:
Alternatively this key can be set to sequence of six numbers defining the PLL configuration, for example the following is equivalent to
| PLL is locked to 352.372 MHz RF input. | Passthrough | RF clock is passed through without locking and regeneration|||||||||||||||
epics_name | This determines the top level device name and generally should match $ioc_name determined above. | |||||||||||||||||
axis0_name | Name of channel 0. Typically X for transverse mode, I for longitudinal mode. | |||||||||||||||||
axis1_name | Name of channel 1. Typically Y for transverse mode, Q for longitudinal mode. | |||||||||||||||||
lmbf_mode | Set to 0 if operating in transverse mode, set to 1 if longitudinal mode. This will determine how the FPGA is configured and some details of the behaviour of the IOC. | |||||||||||||||||
bunches_per_turn | Set to the number of RF buckets per machine revolution. Must be no more than 1024 (with the current FPGA build), and there may be problems with particularly small values. | |||||||||||||||||
revolution_frequency | Set to machine revolution frequency in Hz. Only used for time estimates in display. | |||||||||||||||||
lmbf_fir_offset | Adjustment of FIR coupling between I and Q axes in LMBF mode. | |||||||||||||||||
mms_poll_interval | Used to control polling frequency for MMS readout. If MMS overrun is reported this number needs to be reduced, but check CPU usage with top. Reading MMS data is time consuming. | |||||||||||||||||
persistence_file | This should be an absolute path to a writeable location where the IOC can save the persistent state of all of its PVs. It is wise to keep this file backed up and archived as the configuration of MBF can be quite complex and potentially difficult to recreate. | |||||||||||||||||
persistence_interval | This determines the interval (in seconds) between checks for writing the persistent state. Too small a value will generate a lot of writes, too large a value can result in lost configuration state if the IOC is forcibly restarted. | |||||||||||||||||
pv_log_array_length | Determines how many points of changed waveforms are logged. | |||||||||||||||||
memory_readout_length | Determines the length of the memory readout waveform PV. | |||||||||||||||||
detector_length | Determines the length of the detector readout waveform PV. | |||||||||||||||||
data_port | Determines socket number for fast data readout. |
...
When MBF is started for the first time, or when the state file is deleted, the initial state of nearly all PVs is set to zero. Before trying to use the system the input and output compensation filters should be reset to passthrough and the control gains should be set to 1. The epicstools/initial-state script will perform this operation:
| Code Block | ||
|---|---|---|
| ||
epicstools/initial-state $ioc_name |
Also note that every time MBF is restarted, the DAC output is disabled. This is deliberate to avoid accidentally driving onto the beam after a restart, but means that after every restart the :DAC:ENABLE_S pvs must be manually set.
...