...
The procedure for bringing up the Multi-Bunch Feedback processor system for the first time in a new environment is a little involved and is documented below.
Loose ends:
| Task report | ||||
|---|---|---|---|---|
|
Preparing Hardware
Hardware
...
The MBF control system at Diamond consists of the following hardware:
| Hardware | Vendor | Part Number | Description |
|---|---|---|---|
| VT814 | Vadatech | VT814-200-000-000 | 2U μTCA.4 chassis with 6 AMC slots |
| UTC002 | UTC002-210-801-100 | μTCA MCH (MicroTCA Carrier Hub) with support for 8 lanes of PCIe gen 3. | |
| AMC720 | AMC720-122-010-000 | Intel Xeon general purpose AMC processor card with 16GB RAM, 30GB SSD. | |
| AMC525 | AMC525-012-023-000 | AMC FPGA carrier for dual HPC FMC with Virtex-7 690. | |
| FMC-500M | Innovative Integration | 80281-2-L0 FMC-500 | FMC module with dual 500 Ms/s 14-bit ADC, dual 1200 Ms/s 16-bit DAC. |
| CTI-FMC-DIO-5chttl | Open Hardware (Creotech, Sevensols) | FMC module with five channels of TTL I/O. |
...
- The two FMC cards must be placed in the AMC 525 carrier in the correct slots, with the Digital I/O card in slot 0 and the FMC 500 in slot 1 (more details and pictures in this section). See the image below to verify the correct configuration:
Note that because of an address conflict on the shared IPMI I2C bus between the programmable input threshold DAC on the CTI-FMC-DIO and a temperature sensor on the AMC525 (and thus a MTCA IPMI alert which won't go away), it is necessary to make a modification to the DIO card to reassign the address to an unused address. This done by cutting the track to ADDR1:
- When placing the AMC cards in the crate it seems that it is necessary to be careful about which cards are (logically) adjacent to the processor card. To avoid problems, at DLS we have install installed the processor card in slot 6 and the carrier cards in slots 2 and 3 (note that these slot numbers don't correspond to the PCIe addresses which we'll encounter later). Specifically, it seems that with the processor card in slot 6 we need to avoid placing a carrier card in slot 5, as otherwise the processor card tries to boot from non-existent mass storage in this slot!
- At this point its a good idea to ensure that all serial ports are configured with the same data rate. The default line speed is 115200 8N1 for all ports except for the CPU console which defaults to 9600 8N1.
- Now e-keying must be configured.
...
At this point the MTCA crate and all cards should be in a position to power up. Now tangle with the BIOS and install your choice of Linux (we use RHEL 7 for this at DLSDLS, Debian 9 at the ESRF (see installation notes here)). No special drivers (except for the MBF driver, we'll get onto that) need to be installed, but you'll probably want IPMI administration configured.
...
| Target | Description |
|---|---|
driver | Builds kernel module amc525_lmbfmbf.ko in build/kbuild-$(uname -r). Needs to be run on target system for module to be usable. |
insmod | Ensures kernel module is built and runs insmod to add to the current kernel. Needs to be run on target system. |
driver-rpm | Builds DKMS based RPM for driver. Can be run on any system, resulting RPM can be permanently installed on target system. |
...
| Code Block | ||
|---|---|---|
| ||
./load_fpga -f path/to/amc525_lmbfmbf.bit |
(The -f bit-file argument can be omitted if the bit file is in the working directory.)
...
| Code Block | ||
|---|---|---|
| ||
04:00.0 Signal processing controller: Xilinx Corporation FPGA Card XC7VX690T
Subsystem: Xilinx Corporation Device 0007
Physical Slot: 0
Flags: bus master, fast devsel, latency 0, IRQ 70
Memory at fe300000 (64-bit, non-prefetchable) [size=1M]
Memory at fe400000 (64-bit, non-prefetchable) [size=64K]
Capabilities: <access denied>
Kernel driver in use: amc525_lmbfmbf |
If the kernel driver has not yet been installed, now install it and run ls /dev. The following files should be present:
| Device Node | Description |
|---|---|
amc525_lmbfmbf.0.reg | Register interface device node. Used for control of MBF system. |
amc525_lmbfmbf.0.ddr0 | Access to fast memory buffer for bunch by bunch capture readout. |
amc525_lmbfmbf.0.ddr1 | Access to slow memory buffer for detector readout. |
amc525_lmbfmbf/ | Directory containing physical address links to device nodes. |
...
| Key | Description | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
procserv_port | If the IOC is started using the ./start_ioc script this defines the telnet port that procServ will use. Otherwise this key is ignored. | |||||||||||||||
device_address | This is of the form pci-0000:nn:00.0 where nn identifies the slot where the appropriate AMC525 card is inserted (though the slot numbers and PCIe numbers don't match). Inspect /dev/amc525_lmbfmbf/ for currently recognised cards. | |||||||||||||||
dio_termination | If 50Ω termination is wanted for the three digital I/O trigger inputs then set this to 7, otherwise set to 0 for high impedance termination. | |||||||||||||||
clock_mode | This determines how the PLL is initialised. The following timing modes are supported:
Alternatively this key can be set to sequence of six numbers defining the PLL configuration, for example the following is equivalent to
See the documentation for | |||||||||||||||
epics_name | This determines the top level device name and generally should match $ioc_name determined above. | |||||||||||||||
axis0_name | Name of channel 0. Typically X for transverse mode, I for longitudinal mode. | |||||||||||||||
axis1_name | Name of channel 1. Typically Y for transverse mode, Q for longitudinal mode. | |||||||||||||||
lmbf_mode | Set to 0 if operating in transverse mode, set to 1 if longitudinal mode. This will determine how the FPGA is configured and some details of the behaviour of the IOC. | |||||||||||||||
bunches_per_turn | Set to the number of RF buckets per machine revolution. Must be no more than 1024 (with the current FPGA build), and there may be problems with particularly small values. | |||||||||||||||
revolution_frequency | Set to machine revolution frequency in Hz. Only used for time estimates in display. | |||||||||||||||
lmbf_fir_offset | Adjustment of FIR coupling between I and Q axes in LMBF mode. | |||||||||||||||
mms_poll_interval | Used to control polling frequency for MMS readout. If MMS overrun is reported this number needs to be reduced, but check CPU usage with top. Reading MMS data is time consuming. | |||||||||||||||
persistence_file | This should be an absolute path to a writeable location where the IOC can save the persistent state of all of its PVs. It is wise to keep this file backed up and archived as the configuration of MBF can be quite complex and potentially difficult to recreate. | |||||||||||||||
persistence_interval | This determines the interval (in seconds) between checks for writing the persistent state. Too small a value will generate a lot of writes, too large a value can result in lost configuration state if the IOC is forcibly restarted. | |||||||||||||||
pv_log_array_length | Determines how many points of changed waveforms are logged. | |||||||||||||||
memory_readout_length | Determines the length of the memory readout waveform PV. | |||||||||||||||
detector_length | Determines the length of the detector readout waveform PV. | |||||||||||||||
data_port | Determines socket number for fast data readout. |
...
When MBF is started for the first time, or when the state file is deleted, the initial state of nearly all PVs is set to zero. Before trying to use the system the input and output compensation filters should be reset to passthrough and the control gains should be set to 1. The epicstools/initial-state script will perform this operation:
| Code Block | ||
|---|---|---|
| ||
epicstools/initial-state $ioc_name |
Also note that every time MBF is restarted, the DAC output is disabled. This is deliberate to avoid accidentally driving onto the beam after a restart, but means that after every restart the :DAC:ENABLE_S pvs must be manually set.
...