Fred Klassen
2014-05-20 17:32:21 UTC
Hi,
I'm trying to understand why I cannot receive classic STP or RSTP packets. When sending the STP_only.pcap file with tcpreplay we see "rx_errors" with ethtool, and notice that the 82599 RLEC (Receive Length Error Count) increments. If I send the pattern to a 82574L (e1000e) the packets arrive.
Also, we note that MSTP (802.1s) does appear to work on 82599.
I am using the 3.18.7 ixgbe driver from Sourceforge.
Thanks, Fred.
From: Eli Urshanski <***@silicom.co.il<mailto:***@silicom.co.il>>
Date: Monday, May 19, 2014 at 3:57 AM
To: Fred Klassen <***@appneta.com<mailto:***@appneta.com>>
Cc: David Castiel <***@silicom.co.il<mailto:***@silicom.co.il>>, David Hendel <***@silicom.co.il<mailto:***@silicom.co.il>>, Matt Stevens <***@appneta.com<mailto:***@appneta.com>>, Michael Hustler <***@appneta.com<mailto:***@appneta.com>>, Daniel Schnabel <***@appneta.com<mailto:***@appneta.com>>, Maksim Mihailovich <***@silicom.co.il<mailto:***@silicom.co.il>>, Pavel Komarov <***@silicom.co.il<mailto:***@silicom.co.il>>
Subject: RE: Spanning tree packets missing or have invalid timestamps
Fred,
I tried to analyze the issue with the STP by generating / receiving different pcaps, that included Classic, RSTP and MSTP (802.1s) protocols.
The traffic was injected to the TS adapter with cross-connected 10 Gbe ports in TS disabled and TS enabled mode.
ixgbe driver 3.18.7; FPGA version 0x22-0x1-0x2 ; the commands used:
---------------------------------------------------------------------------
# tsctl_util start
# tsctl_util all set_ts off
# tsctl_util all get_ts ----->TS disabled
04:00.0 off
04:00.1 slave
# tcpreplay --mbps=0 --intf1=eth88 STP_only.pcap (Tx side).
sending out eth88
processing file: STP_only.pcap
Actual: 13 packets (780 bytes) sent in 0.03 seconds.
Statistics for network device: eth88
Attempted packets: 13
Successful packets: 13
Failed packets: 0
Retried packets (ENOBUFS): 0
Retried packets (EAGAIN): 0
# tcpdump -i eth89 -w eli.pcap (Rx side).
# ethtool ethtool -S eth89 | grep rx_er
rx_errors: 13
--------------------------------------------------------------------------------------------------
Investigation results: It seems, that the STP issue is related to the ixgbe driver, not to the TS driver.
The rx_error in ixgbe includes the count reported by the RLEC register, which is abbreviation for
"Receive Length Error count". The description of this register you could see from the 82599 data sheet.
"Number of packets with receive length errors. A length error occurs if an incoming
packet length field in the MAC header doesn't match the packet length"
I have seen this to be the case with Classic and RSTP protocols and it did not happen with RSTP protocol.
As you reported the issue do not occur with 1 Gbe (igb driver) as well.
So, the current work-around could be to change the Spanning Tree from Classic or RSTP to MSTP in case 10 Gbe switch support all the ptocols.
I continue to investigate what more could be done in this case.
Regarding your two additional questions:
1. If you need to upgrade the FPGA and complete the flashing, indeed it will be necessary to power off / on the server.
This was noted at the TS_FPGA Update User Guide page 7, clause 1.5
2. According to our records (see below) we shipped to Appneta 42 adapters Revision 3.7 (FPGA version x21-0x1-0x14).
So, please double check the FPGA version of the existing in-field adapters.
You could upgrade them without recall / RMA.
[cid:***@01CF736A.50DD1EF0]
3. The FPGA release 0x22-0x1-0x2 did not address the previously reported by you the recording the actual time stamp instead
of latch issue.
Regards, Eli.
From: Fred Klassen [mailto:***@appneta.com]
Sent: Tuesday, May 13, 2014 9:42 PM
To: Eli Urshanski
Cc: David Castiel; David Hendel; Matt Stevens; Michael Hustler; Daniel Schnabel; Maksim Mihailovich; Pavel Komarov
Subject: Re: Spanning tree packets missing or have invalid timestamps
Eli,
I upgraded our one and only x21-0x1-0x14 adapter to 0x22-0x1-0x2. Unfortunately with this release I don't see any STP packets, although if I send the trace to one of our GigE adapters they are properly captured. Do you have any idea why the TS adapter is blocking STP? I feel that they should be passing through so that bridging protocols will work properly.
A few more questions?
1. I found that I had to walk up to the machine and power cycle it at the completion of flashing. Is there any way to have the upgrade process complete without a power cycle? Or more specifically, is there a way that the upgrade can be done remotely?
2. Is there any way we can upgrade our existing machines without a recall and RMA?
3. Has the perviously reported issue been fixed with this new release? Specifically, if the timestamp is latched, RX packets receive the latched timestamp instead of recording the actual timestamp.
Fred.
From: Eli Urshanski <***@silicom.co.il<mailto:***@silicom.co.il>>
Date: Monday, May 12, 2014 at 11:04 PM
To: Fred Klassen <***@appneta.com<mailto:***@appneta.com>>
Cc: David Castiel <***@silicom.co.il<mailto:***@silicom.co.il>>, David Hendel <***@silicom.co.il<mailto:***@silicom.co.il>>, Matt Stevens <***@appneta.com<mailto:***@appneta.com>>, Michael Hustler <***@appneta.com<mailto:***@appneta.com>>, Daniel Schnabel <***@appneta.com<mailto:***@appneta.com>>, Maksim Mihailovich <***@silicom.co.il<mailto:***@silicom.co.il>>, Pavel Komarov <***@silicom.co.il<mailto:***@silicom.co.il>>
Subject: RE: Spanning tree packets missing or have invalid timestamps
Fred,
I just paid attention, that SW FPGA update is supported beginning from the version 0x21.0x01.0x11
(please refer to the FPGA Update User Guide page 5).
As you mentioned, you have the TS cards with FPGA versions 0x20-0x1-0x6 and 0x21-0x1-0x14.
I could advise the following procedure:
1. Upgrade only one TS adapter from FPGA version 0x21-0x1-0x14 to the release version 0x22-0x1-0x2 and check that STP packets
have valid time stamps and all other tests have been pass successfully.
2. Upgrade all the rest TS adapters with FPGA version 0x21-0x1-0x14.
3. Please do not upgrade the TS adapters with FPGA version 0x21-0x1-0x6 (the ones you have in the field), because they will fail FPGA update process.
If you come to decision, that you need to upgrade these adapters as well, we need to coordinate the RMA procedure.
Regards, Eli.
From: Eli Urshanski
Sent: Monday, May 12, 2014 6:04 PM
To: 'Fred Klassen'
Cc: David Castiel; David Hendel; Matt Stevens; Michael Hustler; Daniel Schnabel; Maksim Mihailovich; Pavel Komarov
Subject: RE: Spanning tree packets missing or have invalid timestamps
Fred,
Please find attached FPGA version (xamac xx.xx.xx bit file) ; TS FPGA Update User Guide and ts_update utility.
-----------------------------------------------------------------------------------------------------------------------------------------------------------
BTW we reported about "802.3 packets issue" and David provided the explanation (see the paste & cut from the mail ~ 4-5 month ago)
2. As the second issue, 802.3 packets less than 64 byte,:
On The Receive side, the FPGA RX MAC identifies it is a 802.3 short packet and "cuts" the trailer ( per 10G standard) and stamp the packet after the data. Than in the TX MAC append the trailer that causes the following issues"
1. Time stamp not at the end of the packet.
2. Since this packet is shorter than the minimum may cause to the time stamp not update its value of the next packet.
We have a preliminary debug FGPA version that fixes the second issue. We not likely make any change due to the first issue. If you have comments, please let us know.
This is a head up. Not related only to 0x20 we always had it. This is an L2 switching old protocol, so, this is all the information we have.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Regards, Eli.
From: Fred Klassen [mailto:***@appneta.com]
Sent: Monday, May 12, 2014 5:49 PM
To: Eli Urshanski
Cc: David Castiel; David Hendel; Matt Stevens; Michael Hustler; Daniel Schnabel; Maksim Mihailovich; Pavel Komarov
Subject: Re: Spanning tree packets missing or have invalid timestamps
Yes pls send FPGA code and I will upgrade from x21
Sent from my iPhone
On May 12, 2014, at 6:35 AM, "Eli Urshanski" <***@silicom.co.il<mailto:***@silicom.co.il>> wrote:
Hi Fred,
We analyzed the "received.pcap" file and it turned out, that with the STP packets the Time Stamp with Control Byte C3 (9 bytes) was inserted at
the beginning of the trailer. See below the print screen:
This happens with FPGA version 0x20-0x1-0x6.
<image002.jpg>
- Regarding your question what can be done about this..
We have the FPGA release version 0x22-0x1-0x2, that fixed these issue by placing the Timestamp the end of the packet.
See below the print screen, that was captured with TS card FPGA version 0x22-0x1-0x2.
I could send you the FPGA version 0x22-0x1-0x2 to run the tests in your lab and provide the FPGA upgrade instructions.
<image004.jpg>
Regards, Eli.
From: Fred Klassen [mailto:***@appneta.com]
Sent: Friday, May 09, 2014 1:51 AM
To: Eli Urshanski
Cc: David Castiel; Matt Stevens; Michael Hustler; Daniel Schnabel
Subject: Spanning tree packets missing or have invalid timestamps
Eli,
I am attempting to send the "sent.pcap" file to our TS adapters and having strange results.
On my version 0x21-0x1-0x14 adapter we are missing all spanning tree (STP) packets.
On our version common 0x20 0x1 0x6 adapters (the ones we have in the field) we are missing some STP packets, but some of the ones we receive have completely corrupted timestamps. This is illustrated in the "received.pcap" file. Notice that STP packets 73, 193 and 239 have valid timestamps (2014-05-08) but packets 318 and 340 are 2055-01-25. Also, packet 389 has a timestamp of 2038-12-07.
We see different values on different TS adapters, and sometimes we even get negative timestamps.
So, my questions are:
1. Why are the adapters we have in the field corrupting timestamps?
2. Why are some or all the STP packets being filtered out?
3. What can be done about this?
Thanks, Fred.
I'm trying to understand why I cannot receive classic STP or RSTP packets. When sending the STP_only.pcap file with tcpreplay we see "rx_errors" with ethtool, and notice that the 82599 RLEC (Receive Length Error Count) increments. If I send the pattern to a 82574L (e1000e) the packets arrive.
Also, we note that MSTP (802.1s) does appear to work on 82599.
I am using the 3.18.7 ixgbe driver from Sourceforge.
Thanks, Fred.
From: Eli Urshanski <***@silicom.co.il<mailto:***@silicom.co.il>>
Date: Monday, May 19, 2014 at 3:57 AM
To: Fred Klassen <***@appneta.com<mailto:***@appneta.com>>
Cc: David Castiel <***@silicom.co.il<mailto:***@silicom.co.il>>, David Hendel <***@silicom.co.il<mailto:***@silicom.co.il>>, Matt Stevens <***@appneta.com<mailto:***@appneta.com>>, Michael Hustler <***@appneta.com<mailto:***@appneta.com>>, Daniel Schnabel <***@appneta.com<mailto:***@appneta.com>>, Maksim Mihailovich <***@silicom.co.il<mailto:***@silicom.co.il>>, Pavel Komarov <***@silicom.co.il<mailto:***@silicom.co.il>>
Subject: RE: Spanning tree packets missing or have invalid timestamps
Fred,
I tried to analyze the issue with the STP by generating / receiving different pcaps, that included Classic, RSTP and MSTP (802.1s) protocols.
The traffic was injected to the TS adapter with cross-connected 10 Gbe ports in TS disabled and TS enabled mode.
ixgbe driver 3.18.7; FPGA version 0x22-0x1-0x2 ; the commands used:
---------------------------------------------------------------------------
# tsctl_util start
# tsctl_util all set_ts off
# tsctl_util all get_ts ----->TS disabled
04:00.0 off
04:00.1 slave
# tcpreplay --mbps=0 --intf1=eth88 STP_only.pcap (Tx side).
sending out eth88
processing file: STP_only.pcap
Actual: 13 packets (780 bytes) sent in 0.03 seconds.
Statistics for network device: eth88
Attempted packets: 13
Successful packets: 13
Failed packets: 0
Retried packets (ENOBUFS): 0
Retried packets (EAGAIN): 0
# tcpdump -i eth89 -w eli.pcap (Rx side).
# ethtool ethtool -S eth89 | grep rx_er
rx_errors: 13
--------------------------------------------------------------------------------------------------
Investigation results: It seems, that the STP issue is related to the ixgbe driver, not to the TS driver.
The rx_error in ixgbe includes the count reported by the RLEC register, which is abbreviation for
"Receive Length Error count". The description of this register you could see from the 82599 data sheet.
"Number of packets with receive length errors. A length error occurs if an incoming
packet length field in the MAC header doesn't match the packet length"
I have seen this to be the case with Classic and RSTP protocols and it did not happen with RSTP protocol.
As you reported the issue do not occur with 1 Gbe (igb driver) as well.
So, the current work-around could be to change the Spanning Tree from Classic or RSTP to MSTP in case 10 Gbe switch support all the ptocols.
I continue to investigate what more could be done in this case.
Regarding your two additional questions:
1. If you need to upgrade the FPGA and complete the flashing, indeed it will be necessary to power off / on the server.
This was noted at the TS_FPGA Update User Guide page 7, clause 1.5
2. According to our records (see below) we shipped to Appneta 42 adapters Revision 3.7 (FPGA version x21-0x1-0x14).
So, please double check the FPGA version of the existing in-field adapters.
You could upgrade them without recall / RMA.
[cid:***@01CF736A.50DD1EF0]
3. The FPGA release 0x22-0x1-0x2 did not address the previously reported by you the recording the actual time stamp instead
of latch issue.
Regards, Eli.
From: Fred Klassen [mailto:***@appneta.com]
Sent: Tuesday, May 13, 2014 9:42 PM
To: Eli Urshanski
Cc: David Castiel; David Hendel; Matt Stevens; Michael Hustler; Daniel Schnabel; Maksim Mihailovich; Pavel Komarov
Subject: Re: Spanning tree packets missing or have invalid timestamps
Eli,
I upgraded our one and only x21-0x1-0x14 adapter to 0x22-0x1-0x2. Unfortunately with this release I don't see any STP packets, although if I send the trace to one of our GigE adapters they are properly captured. Do you have any idea why the TS adapter is blocking STP? I feel that they should be passing through so that bridging protocols will work properly.
A few more questions?
1. I found that I had to walk up to the machine and power cycle it at the completion of flashing. Is there any way to have the upgrade process complete without a power cycle? Or more specifically, is there a way that the upgrade can be done remotely?
2. Is there any way we can upgrade our existing machines without a recall and RMA?
3. Has the perviously reported issue been fixed with this new release? Specifically, if the timestamp is latched, RX packets receive the latched timestamp instead of recording the actual timestamp.
Fred.
From: Eli Urshanski <***@silicom.co.il<mailto:***@silicom.co.il>>
Date: Monday, May 12, 2014 at 11:04 PM
To: Fred Klassen <***@appneta.com<mailto:***@appneta.com>>
Cc: David Castiel <***@silicom.co.il<mailto:***@silicom.co.il>>, David Hendel <***@silicom.co.il<mailto:***@silicom.co.il>>, Matt Stevens <***@appneta.com<mailto:***@appneta.com>>, Michael Hustler <***@appneta.com<mailto:***@appneta.com>>, Daniel Schnabel <***@appneta.com<mailto:***@appneta.com>>, Maksim Mihailovich <***@silicom.co.il<mailto:***@silicom.co.il>>, Pavel Komarov <***@silicom.co.il<mailto:***@silicom.co.il>>
Subject: RE: Spanning tree packets missing or have invalid timestamps
Fred,
I just paid attention, that SW FPGA update is supported beginning from the version 0x21.0x01.0x11
(please refer to the FPGA Update User Guide page 5).
As you mentioned, you have the TS cards with FPGA versions 0x20-0x1-0x6 and 0x21-0x1-0x14.
I could advise the following procedure:
1. Upgrade only one TS adapter from FPGA version 0x21-0x1-0x14 to the release version 0x22-0x1-0x2 and check that STP packets
have valid time stamps and all other tests have been pass successfully.
2. Upgrade all the rest TS adapters with FPGA version 0x21-0x1-0x14.
3. Please do not upgrade the TS adapters with FPGA version 0x21-0x1-0x6 (the ones you have in the field), because they will fail FPGA update process.
If you come to decision, that you need to upgrade these adapters as well, we need to coordinate the RMA procedure.
Regards, Eli.
From: Eli Urshanski
Sent: Monday, May 12, 2014 6:04 PM
To: 'Fred Klassen'
Cc: David Castiel; David Hendel; Matt Stevens; Michael Hustler; Daniel Schnabel; Maksim Mihailovich; Pavel Komarov
Subject: RE: Spanning tree packets missing or have invalid timestamps
Fred,
Please find attached FPGA version (xamac xx.xx.xx bit file) ; TS FPGA Update User Guide and ts_update utility.
-----------------------------------------------------------------------------------------------------------------------------------------------------------
BTW we reported about "802.3 packets issue" and David provided the explanation (see the paste & cut from the mail ~ 4-5 month ago)
2. As the second issue, 802.3 packets less than 64 byte,:
On The Receive side, the FPGA RX MAC identifies it is a 802.3 short packet and "cuts" the trailer ( per 10G standard) and stamp the packet after the data. Than in the TX MAC append the trailer that causes the following issues"
1. Time stamp not at the end of the packet.
2. Since this packet is shorter than the minimum may cause to the time stamp not update its value of the next packet.
We have a preliminary debug FGPA version that fixes the second issue. We not likely make any change due to the first issue. If you have comments, please let us know.
This is a head up. Not related only to 0x20 we always had it. This is an L2 switching old protocol, so, this is all the information we have.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Regards, Eli.
From: Fred Klassen [mailto:***@appneta.com]
Sent: Monday, May 12, 2014 5:49 PM
To: Eli Urshanski
Cc: David Castiel; David Hendel; Matt Stevens; Michael Hustler; Daniel Schnabel; Maksim Mihailovich; Pavel Komarov
Subject: Re: Spanning tree packets missing or have invalid timestamps
Yes pls send FPGA code and I will upgrade from x21
Sent from my iPhone
On May 12, 2014, at 6:35 AM, "Eli Urshanski" <***@silicom.co.il<mailto:***@silicom.co.il>> wrote:
Hi Fred,
We analyzed the "received.pcap" file and it turned out, that with the STP packets the Time Stamp with Control Byte C3 (9 bytes) was inserted at
the beginning of the trailer. See below the print screen:
This happens with FPGA version 0x20-0x1-0x6.
<image002.jpg>
- Regarding your question what can be done about this..
We have the FPGA release version 0x22-0x1-0x2, that fixed these issue by placing the Timestamp the end of the packet.
See below the print screen, that was captured with TS card FPGA version 0x22-0x1-0x2.
I could send you the FPGA version 0x22-0x1-0x2 to run the tests in your lab and provide the FPGA upgrade instructions.
<image004.jpg>
Regards, Eli.
From: Fred Klassen [mailto:***@appneta.com]
Sent: Friday, May 09, 2014 1:51 AM
To: Eli Urshanski
Cc: David Castiel; Matt Stevens; Michael Hustler; Daniel Schnabel
Subject: Spanning tree packets missing or have invalid timestamps
Eli,
I am attempting to send the "sent.pcap" file to our TS adapters and having strange results.
On my version 0x21-0x1-0x14 adapter we are missing all spanning tree (STP) packets.
On our version common 0x20 0x1 0x6 adapters (the ones we have in the field) we are missing some STP packets, but some of the ones we receive have completely corrupted timestamps. This is illustrated in the "received.pcap" file. Notice that STP packets 73, 193 and 239 have valid timestamps (2014-05-08) but packets 318 and 340 are 2055-01-25. Also, packet 389 has a timestamp of 2038-12-07.
We see different values on different TS adapters, and sometimes we even get negative timestamps.
So, my questions are:
1. Why are the adapters we have in the field corrupting timestamps?
2. Why are some or all the STP packets being filtered out?
3. What can be done about this?
Thanks, Fred.