Hi Petr,
Any news? Are the IPMCs still crashing?
Cheers,
Ralf
Hi Ralf,
Thanks for coming back to this. Something changed: I got an email notification about your message!
I will try to answer all the questions from your previous message:
What is your network setup?
The IPMC on the EP-ESE development board is connected directly to the GPN network in B186, socket 0717/01.
And what are the first messages that you see as soon as you connect the IPMC to the network?
Only the exception message. Nothing else is printed when it is connected to the network.
From the logs I can see that the IPMC uses IP address 0.0.0.0, which in my understanding means that it will listen to ANY message passing on the network! I can understand that this will completely overwhelm it.
I’m assuming that the IPMC reports 0.0.0.0 because we do not manually assign a static IP address. We rely on DHCP.
According to RFC 2131, during DHCP (Discovery, etc.), the client uses 0.0.0.0 as its source IP address before the DHCP process has completed and an IP address has been assigned. In this state, the TCP/IP stack should only listen for broadcast traffic and UDP packets on the DHCP ports 67 and 68.
I don’t know which TCP/IP stack the IPMC is using. Maybe lwIP? If so, would it be possible to find out which release/version is being used?
The rate of packets reaching the IPMC is very small, on the order of 10 packets/sec.
I am further wondering about the “IPMB-A error: IPMC 20: not ready” (same for IPMB-B) messages…
These are artefacts caused by the fact that the IPMC does not have a connection to the Shelf Manager when it is running on the development board. One has to manually set the address via the management interface, e.g. with SETHA 0x8C. After this command, the IPMC continues operating.
The development board is from EP-ESE, so I believe that you must have encountered this already. This does not happen when the IPMC is on the DTH in the ATCA crate and has a connection to the Shelf Manager. Therefore, this can be ignored.
Unfortunately, the logs don’t contain any timestamps, so it is a bit difficult to follow what happens when and what the delays between events are.
The IPMC software/firmware is developed for an embedded system and works as a tight loop acting on messages (of all network and internal events). Its resources are limited. So, a correct IP network setup is essential.
The packet rate reaching the IPMC is very small, on the order of 10 packets/sec. Therefore, this does not seem consistent with a resource limitation problem.
When using, for example, remote JTAG, the packet rate is significantly higher, and the IPMC does not seem to have any problems processing it.
Best regards,
Petr
Hello Petr,
I have several comment/questions, see below:
The IPMC on the EP-ESE development board is connected directly to the GPN network in B186, socket 0717/01.
I could not find any device with the MAC address 80:D3:36:00:42:74 from the log below in LanDB:
https://gitlab.cern.ch/hardware/phase2/ipmc_tests/-/blob/master/logs/IPMC_barebone.txt#L62
Did you not register the device on the network?
I don’t know which TCP/IP stack the IPMC is using. Maybe lwIP? If so, would it be possible to find out which release/version is being used?
No external TCP/IP stack is used, the light-weight networking layer implemented on the IPMC was developed by PigeonPoint.
These are artefacts caused by the fact that the IPMC does not have a connection to the Shelf Manager when it is running on the development board. One has to manually set the address via the management interface, e.g. with
SETHA 0x8C. After this command, the IPMC continues operating.
The development board is from EP-ESE, so I believe that you must have encountered this already. This does not happen when the IPMC is on the DTH in the ATCA crate and has a connection to the Shelf Manager. Therefore, this can be ignored.
Operating the IPMC in normal mode without a connection to a shelf manager may cause issues. You should really reboot the IPMC in stand-alone mode in this case.
When using, for example, remote JTAG, the packet rate is significantly higher, and the IPMC does not seem to have any problems processing it.
I though you said previously that you are not using the Xilinx virtual cable server on the IPMC?
I believe in order to make progress on this, we would need the exact copy of the files use to build the IPMC firmware that you are testing. We can then build the same firmware enabling some debug output in the networking functions.
cheers,
Stefan
Hi Petr,
Good to know that you got the notification 
I had the same question as Stefan: when I searched for the MAC address you sent, I could not find it in LanDB. Have you registered the IPMC? If not, how would you expect to get a reply to the DHCP DISCOVER?
Cheers,
Ralf
Hi Stefan,
I’m replying by email so that I can include Ulf and Constantinos. They can provide you with the files used to build the IPMC firmware.
I also put my answers inline below:
The IPMC on the EP-ESE development board is connected directly to the GPN network in B186, socket 0717/01.
I could not find any device with the MAC address 80:D3:36:00:42:74 from the log below in LanDB:
https://gitlab.cern.ch/hardware/phase2/ipmc_tests/-/blob/master/logs/IPMC_barebone.txt#L62
Did you not register the device on the network?
No, we do not register IPMC MAC addresses in LanDB when using GPN. I will put more information in the reply to Ralf.
I don’t know which TCP/IP stack the IPMC is using. Maybe lwIP? If so, would it be possible to find out which release/version is being used?
No external TCP/IP stack is used, the light-weight networking layer implemented on the IPMC was developed by PigeonPoint.
Ok, thanks.
These are artefacts caused by the fact that the IPMC does not have a connection to the Shelf Manager when it is running on the development board. One has to manually set the address via the management interface, e.g. with
SETHA 0x8C. After this command, the IPMC continues operating.
The development board is from EP-ESE, so I believe that you must have encountered this already. This does not happen when the IPMC is on the DTH in the ATCA crate and has a connection to the Shelf Manager. Therefore, this can be ignored.Operating the IPMC in normal mode without a connection to a shelf manager may cause issues. You should really reboot the IPMC in stand-alone mode in this case.
The same hard exception also appears when the IPMC is running in the ATCA crate and has a connection to the Shelf Manager. Therefore, I do not believe that running without the Shelf Manager is the cause of the issue in this case. That being said, we can also test the IPMC in stand-alone mode next time.
When using, for example, remote JTAG, the packet rate is significantly higher, and the IPMC does not seem to have any problems processing it.
I though you said previously that you are not using the Xilinx virtual cable server on the IPMC?
No, I don’t believe I said that. Using XVC over the IPMC to connect remotely to our embedded system is one of the functionalities that we tested, and it was working from Windows.
I believe in order to make progress on this, we would need the exact copy of the files use to build the IPMC firmware that you are testing. We can then build the same firmware enabling some debug output in the networking functions.
I will let Ulf/Constantinos send you the files.
It would be great if some additional debugging could be enabled. Ideally, in addition to the existing debug output, it would be very useful if the IPMC could print information about the packets it is receiving and sending. This could move us further and help to understand what exactly triggers the exception.
Thanks.
Best regards,
Petr
Hi Ralf,
I’m happy to receive the notifications now, indeed.
Regarding your question: No, we do not register IPMC MAC addresses in LanDB when using GPN.
If there is no DHCP reply, the IPMC simply keeps sending DHCP Discover packets. This is what happens on a private network. On GPN, the IT DHCP server always replies, at least with a temporary IP address from a dedicated subnet, allowing the device to get registered in LanDB.
In our lab setup in B40, we have a custom DHCP server that provides GPN IP addresses based on the Client ID. This is something that the IT DHCP servers do not support, so we cannot use the IT DHCP server for this setup.
What is strange is that in B186, when the IPMC is connected to GPN, the exception appears. Without custom DHCP servers.
So far, the problem is only reproducible in B186. However, as more people are slowly starting to use our boards with IPMCs, it is not excluded that we may find the same problem in other locations over time.
We need to find the actual cause, ideally by enabling some debug messages or dumping the packets. This would allow us to either fix the problem or find a way to avoid it.
In particular, this is important for the experiments. If the problem is triggered by a single packet, it has the potential to stop the whole experiment from operating.
Best regards,
Petr
Hi Petr,
Thanks for your reply.
Are you sure that IT DHCP servers reply to DHCP requests from nodes which are not registered? In any case, your logs do not show the IPMC receiving any DHCP offer and changing the IP address to something different than 0.0.0.0.
I think the best way to progress is to give Stefan access to your code, so that he can check if he can instrument it with DEBUG messages in order to see what is going on.
Cheers,
Ralf
Hi all,
Unluckily I’m on vacation and reading my emails unregularly. I remember that the firmware we are running on this IPMC is not reading out any sensor (with our standard sensor readout we have seen the same behavior).
I’m also traveling w/o laptop and have to find a way to provide you with our xml config file if you want to add debug requests. We are using the standard web interface to compile the firmware.
Greetings from the Côte d’Azur
Ulf
Hi Ralf,
Yes, an unknown MAC address will get a reply from the IT DHCP server with access to a restricted network, from which one can perform self-registration in CERN LanDB. This does not cause any problems for the IPMCs in B40.
To see exactly which packets the IPMC receives in B186, and whether one of them is causing the crash, we need to have a debug firmware, indeed. Currently, when the IPMC is connected to the network in B186, it hangs almost immediately without printing any messages. We have also verified this with a plain IPMC firmware, as described in the previous emails.
I believe Stefan does not need to wait for us (Ulf is on holiday) to provide the code. He can generate any generic IPMC firmware with the necessary debug messages enabled. We can then upload it to an IPMC and test it in B186.
Best regards,
Petr