Management of non-intelligent RTM

So nice of you, the install went fine !

But sadly I still get the same issue:

$ devkit_ctrl -a get_power
Traceback (most recent call last):
  File "/home/pi/SylvainL/ipmc-tester/ipmc-tester/DevKITControl.py", line 581, in <module>
    IPMCDevObject = IPMCDevCom.IPMCDevCom(umgt_port = umgt_port)
TypeError: __init__() got an unexpected keyword argument 'umgt_port'

Hello,

I just tried it myself from scratch and it works for me. This probably means you are still pointing to the old version of the underlying communication library from ipmc-devkit. Can you please do the following:

cat which devkit_ctrl

And list the contents of the directory that PYTHONPATH points to? In particular the first lines of the following file are of interest:

IPMCDevLib/IPMCDevCom.py

cheers,

Stefan

Hi,

I of course checked all this twice prior to bothering you once more:

$ cat `which devkit_ctrl`
### Original: ###
#python3 /ipmc/ipmc-tester/ipmc-tester/DevKITControl.py "$@"

### As of 2025.04.09 following https://cern-ipmc-forum.web.cern.ch/t/management-of-non-intelligent-rtm/319/15
export PYTHONPATH=/home/pi/SylvainL/ipmc-tester/ipmc-devkit
python3 /home/pi/SylvainL/ipmc-tester/ipmc-tester/DevKITControl.py "$@"

$ devkit_ctrl -a get_power
Traceback (most recent call last):
  File "/home/pi/SylvainL/ipmc-tester/ipmc-tester/DevKITControl.py", line 581, in <module>
    IPMCDevObject = IPMCDevCom.IPMCDevCom(umgt_port = umgt_port)
TypeError: __init__() got an unexpected keyword argument 'umgt_port'

As suggested, I proceed to clean and reinstall everything from scratch, and now I get this:

$ devkit_ctrl -a get_power
[Info] using serial port /dev/ipmc-umgt to communicate with uMGT
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/serial/serialposix.py", line 322, in open
    self.fd = os.open(self.portstr, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
FileNotFoundError: [Errno 2] No such file or directory: '/dev/ipmc-umgt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pi/SylvainL/ipmc-tester/ipmc-tester/DevKITControl.py", line 581, in <module>
    IPMCDevObject = IPMCDevCom.IPMCDevCom(umgt_port = umgt_port)
  File "/home/pi/SylvainL/ipmc-tester/ipmc-devkit/IPMCDevLib/IPMCDevCom.py", line 28, in __init__
    with serial.Serial(port, 9600, timeout=1, write_timeout=1) as ser:
  File "/usr/lib/python3/dist-packages/serial/serialutil.py", line 244, in __init__
    self.open()
  File "/usr/lib/python3/dist-packages/serial/serialposix.py", line 325, in open
    raise SerialException(msg.errno, "could not open port {}: {}".format(self._port, msg))
serial.serialutil.SerialException: [Errno 2] could not open port /dev/ipmc-umgt: [Errno 2] No such file or directory: '/dev/ipmc-umgt'
$ file /dev/ipmc-umgt
/dev/ipmc-umgt: cannot open `/dev/ipmc-umgt' (No such file or directory)

Is there any installation script or udev rules than I should be aware of ?

Best regards,
Sylvain

Hello Sylvain,

this is actually much better. I assume there is a udev rule missing on your RPi. This will have to wait until Monday when Ralf is back. In the meantime you could try making a symbolic link from /dev/ttyUSB0 to /dev/ipmc-umgt. Also make sure that you have read and write access rights as a normal user. This is what I have on the RPi I’m using:

haass@ipmc-tester14:~/work$ ls -l /dev/ipmc-umgt
lrwxrwxrwx 1 root root 7 Mar 5 16:17 /dev/ipmc-umgt → ttyUSB0

I hope it will work then.

cheers,

Stefan

Yes that made it thank you.

Hi Stefan,

Our updated CERN-IPMC Raspberry Pi is now operational, thanks to your help and patience.

After spending several hours debugging and fixing the code you provided us with line by line to decipher and fix each compilation error I encountered (still strange to me BTW, as you stated that it compiled fine on your side… maybe we use different compilers ?), I am stuck on the error hereafter, that must be linked only to the RTM code (as everything else compile and run fine when removing all the RTM-related code in user_defs.h):

→ Configuration process successfully finished

→ Compile IPMC source code

make: Entering directory ‘/tmp/comp-0UV0gRHQAl’

/usr/local/arm-toolchain/bin/arm-none-eabi-ar: creating ipmc-core/app/libcompat/libcompat.a

/usr/local/arm-toolchain/bin/arm-none-eabi-ar: creating ipmc-core/net/libnet.a

/usr/local/arm-toolchain/bin/arm-none-eabi-ar: creating ipmc-core/hal/libhal.a

/usr/local/arm-toolchain/bin/arm-none-eabi-ar: creating ipmc-core/lib/lib.a

/usr/local/arm-toolchain/bin/arm-none-eabi-ar: creating ipmc-core/hal/actel/libactel.a

crc32 for the cold data is 0xa7e8a6b3 (12 vars total)

compat_record checksums: 0xfc 0xde

File name: ipmc-core/hal/firmware.bin size: 148408

Checksum section offset: 000004

Verify post_sram.c code

File name: ipmc-core/boot/boot.bin size: 18116

Checksum section offset: 0002A0

make: *** No rule to make target ‘ipmc-config/rtm-fru-info.bin’

make: Leaving directory ‘/tmp/comp-0UV0gRHQAl’

Error: IPMC configuration failed. This issue has been automatically reported

Does this message speak to you in any way ?

I attached a zip of the 2 RTM-related files we compile with, if of any help for your investigation.

Maybe can we also appoint a Zoom call at your convenience, to look at this interactively if you think this to be easier/faster ?

We also considered coming at CERN to meet you, but for a productive session we would have to first dismantle, then travel back and forth with, and then reinstall our setup back here, which is obviously not the simplest, safest and preferred solution for us.

Anyway, please tell us what you think ?

Amicalement,
Sylvain L.

PS: I tried to send you this message by mail first, but got rejected for whatever security policy reason…

Hello Sylvain,

unfortunately, Discourse (the tool we use for the forum) also ate your attachments.

I am using exactly the same remote compilation tool as you. However you need to add the following key to the XML:

I realized that I forgot to mention this in my previous messages. I can also send you an archive with the files I use to compile to your e-mail, hoping it will get through. If none of this fixes it, we can arrange a zoom call tomorrow.

cheers,

Stefan

Hello Stefan,

I replied to you by mail with log files attached to go further.

Best regards,
Sylvain L.

Hello,

Do you please have any update on this issue on your side ?

Best regards,
Sylvain

Hello Sylvain,

I’m just back at CERN, I managed to reproduce the issue with the firmware upgrade. I’m currently looking into what could have gone wrong.

cheers,

Stefan

Hello Sylvain,

I managed to get your firmware loaded on one of our IPMCs. Can you please try to upgrade first only the firmware, then the bootloader and then activate as follows?

ipmitool -I lan -U ‘’ -P ‘’ -H hpm upgrade hpm1fw.img force
ipmitool -I lan -U ‘’ -P ‘’ -H hpm upgrade hpm1boot.img force
ipmitool -I lan -U ‘’ -P ‘’ -H hpm activate norollback

After that I was able to upgrade using the full HPM.1 file (hpm1all.img) in the normal way. Let me know if this doesn’t work for you.

BTW, I also enabled debug messages from the RTM part of the software.

cheers,

Stefan

Hello Stefan,

Yes I confirm I can now load my firmware correctly.
What king of log should I expect now that you enabled it please ?

Best regards,
Sylvain

Hello Sylvain,

good to hear that the firmware upgrade works for you as well now.

The messages concerning the RTM are debug messages, for example I see:

RTM: ps=0 mp_en=0 mp_good=0 state=0 dead=0

I’m afraid since we do not have an RTM, I cannot tell you what to expect in case it is working.

cheers,

Stefan

Oops I do not get this one yet :frowning:
Anything special to enable on my side maybe ?

Hello Sylvain,

you need to rerun the remote compilation to get this latest change.

cheers,

Stefan

I of course did to no avail for me…

I assume you re-enabled the RTM support?

Hello Sylvain,

I also changed the following defines in the user_defs.h file:

// RTM FRU EEPROM on the front blade
#define CFG_RTM_FRU_INFO_EEPROM_SIZE 512
#define CFG_RTM_FRU_INFO_EEPROM_ADDR 0xa2
#define CFG_RTM_FRU_INFO_EEPROM_OFFSET (6 * 512)

Maybe you could do the same and see if this helps?

cheers,

Stefan

Hello Stefan,

RTM support is enabled, and the specified defines were added.
Lots of new warning during compilation:


./ipmc-user/user_defs.h:180: warning: “CFG_RTM_FRU_INFO_EEPROM_SIZE” redefined

And no RTM traces on the serial output:

[16:49:50.834] <>: IPMI session established (session ID = 00000002)
[16:49:54.514] <
>: IPMI session closed (session ID = 00000002)
[16:49:54.546] <_>: IPMI session established (session ID = 00000004)

[16:49:54.914] BMR-A2F Boot Loader v1.3.0.20250513 (c) Pentair Technical Products, Inc.
[16:49:54.966] boot_first_time: 00
[16:49:54.993] boot_type: 12
[16:49:54.994] wdt_reset_type: A3
[16:49:55.010] fw_version: 02
[16:49:55.026] wdt_reset_occurred implemented: false
[16:49:55.058] Backing up active image to external Flash…
[16:49:55.091] Copying firmware from 00006000 to 00300000 size 00023CD4

[16:49:56.097] bl_hdr:00000000
[16:49:56.113] fw_version: 02
[16:49:56.114] boot_flag: FA
[16:49:56.130] Active partition is #00
[16:49:56.146] active status:07
[16:49:56.162] wdt_reset_occurred:01
[16:49:56.178] reset_type:A3
[16:49:56.194] Firmware address:00006000
[16:49:56.225] Firmware size: 00023CD4
[16:49:56.241] Firmware image (END-START = 00040000) - checksum: 00
[16:49:56.353] Active partition succesfully cheked: 07
[16:49:56.386] boot_flag: FA
[16:49:56.401] active status:07
[16:49:56.402] Booting at address #00006000

[16:49:56.447] <>: BMR-A2F Firmware (v1.3.0)
[16:49:56.447] <
>: Pentair Technical Products, Inc. (c) Copyright 2004-2016.
[16:49:56.670] <>: Reset type: cold, reset cause: software
[16:49:56.692] MAC Address loaded from EEPROM [80:D3:36:00:40:F7]
[16:49:56.693] <
>: Perform Power-on-Self Testing.
[16:49:57.000] <>: Device type: A2F200M3F-CS288
[16:49:57.000] <
>: MSS clock frequency: 80 MHz
[16:49:57.001] <>: Fabric clock frequency: 20 MHz
[16:49:57.002] <
>: Fast delay calibration: 2665 cycles per 100uS
[16:49:57.002] <>: FPGA design: BMR-A2F-ATCA, version 4.0.0
[16:49:57.003] <
>: Flash: 256 KB (00000000 - 00040000)
[16:49:57.004] <>: SRAM: 64 KB (20000000 - 20010000)
[16:49:57.005] <
>: Operating mode: normal
[16:49:57.005] <>: Hardware address: 0x43
[16:49:57.006] <
>: Setting RTC with the default date
[16:49:57.006] <>: FRU 0 state: M0->M1, cause = 0
[16:49:57.007] <
>: LAN iface 0: MAC = 80:D3:36:00:40:F7, IP = 192.168.1.74
[16:49:57.008] <>: SOL mode: Direct Ethernet (with Core10/100)
[16:49:57.023] <
>: HPM3: Iface 0: reset IPv4 address to 0.0.0.0
[16:49:57.245] <>: FRU 0 state: M1->M2, cause = 2
[16:49:57.341] <
>: FRU 0 state: M2->M3, cause = 1
[16:49:57.725] : [FM3 FM2 FM1 FC DCDC] Power UP sequence completed SUCCESSFULLY.
[16:49:57.749] <>: FRU 0 state: M3->M4, cause = 0
[16:49:57.906] : Sensor alert
[16:50:00.002] <
>: CORE_10100: Link is UP, 100Mb FD
[16:50:02.897]
: Sensor alert
[16:50:07.872]
: Sensor alert
[16:50:12.847]
: Sensor alert
[16:50:17.839]
: Sensor alert
[16:50:22.814]
: Sensor alert
[16:50:26.349] <_>: LAN iface 0: IP = 192.168.1.74
[16:50:27.805]
: Sensor alert
[16:50:32.780]
: Sensor alert
[16:50:37.755]
: Sensor alert
[16:50:42.746]
: Sensor alert
[16:50:47.721]
: Sensor alert
[16:50:52.697]
: Sensor alert
[16:50:56.632] iface_ipmb_send_message_ipmb: i2c NACK
[16:50:57.688]
: Sensor alert
[16:51:02.663]
: Sensor alert

Best regards,
Sylvain

Hello Sylvain,

can you please upload your project files to cernbox so that I can try it?

Also, I see that in your case the power-up sequence finishes successfully. Are you testing this on an IPMC tester or with a LATOURNETT board?

cheers,

Stefan