Discussion:
Adding a ETH-type filter for CFM
(too old to reply)
Mattias Barthel
2015-10-05 12:19:13 UTC
Permalink
Greetings,

Im getting quite bad latency when using the igb driver on i350 on linux
regarding
ETH CFM packets. This compared to TWAMP (IPv4 UDP packets).
The environment is KVM with the i350 devices in PCI-passthrough.

So i figured I add an own rx-queue (filter) for those types of ethernet
protocol packets.

This is what i added:


wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable filter */
(1 << 29) | /* enable immediate interrupt */
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902 CFM eth protocol type */


ETH_P_8021A is 0x8902



Is this a correct/good approach?

Thanks in advance!
--
Mattias Barthel
Mattias Barthel
2015-10-08 05:21:31 UTC
Permalink
Hi again!

I realise my message might not have been so clear.

I will try to clarify. I am doing PM with software timestamping.

igb versions:
version: 3.0.6-k2
firmware-version: 0.147-0

I do timestamping (NTP format) of the packets in the same place for all
protocols. (in igb_clean_rx_irq_adv()).

For these non ip packets like IEEE 801.2ag CFM ETH-DM (ethertype 0x8902) I
get quite a lot worse accuracy in the timestamping.

For UDP/IP TWAMP I get for 10k pps a max delay of around 140us back to back.
For CFM ETH DM I get for the same packet rate and packet size a max delay
of around 900us.

So my hypothesis was:
Could FW be putting these L2 packets in another queue with different
characteristics?

I tried to add a ET-type filter as written in my previous mail but it
showed no difference.


Thanks for any help given!
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when using the igb driver on i350 on linux
regarding
ETH CFM packets. This compared to TWAMP (IPv4 UDP packets).
The environment is KVM with the i350 devices in PCI-passthrough.
So i figured I add an own rx-queue (filter) for those types of ethernet
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable filter */
(1 << 29) | /* enable immediate interrupt */
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902 CFM eth protocol type */
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
--
Mattias Barthel
Alexander Duyck
2015-10-08 07:02:43 UTC
Permalink
The difference could be Receive Side Scaling (RSS). The UDP/IP packets
can be load balanced across all queues via RSS based on a hash of the
source and destination addresses. So if you have multiple sources
generating these packets, or they are being received on multiple IP
addresses then it is possible the load is being spread out over multiple
CPUs. Your CFM packets are on an Ethertype other than IPv4 or IPv6 so
all packets will end up on queue 0 if I am not mistaken.

- Alex
Post by Mattias Barthel
Hi again!
I realise my message might not have been so clear.
I will try to clarify. I am doing PM with software timestamping.
version: 3.0.6-k2
firmware-version: 0.147-0
I do timestamping (NTP format) of the packets in the same place for all
protocols. (in igb_clean_rx_irq_adv()).
For these non ip packets like IEEE 801.2ag CFM ETH-DM (ethertype 0x8902) I
get quite a lot worse accuracy in the timestamping.
For UDP/IP TWAMP I get for 10k pps a max delay of around 140us back to back.
For CFM ETH DM I get for the same packet rate and packet size a max delay
of around 900us.
Could FW be putting these L2 packets in another queue with different
characteristics?
I tried to add a ET-type filter as written in my previous mail but it
showed no difference.
Thanks for any help given!
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when using the igb driver on i350 on linux
regarding
ETH CFM packets. This compared to TWAMP (IPv4 UDP packets).
The environment is KVM with the i350 devices in PCI-passthrough.
So i figured I add an own rx-queue (filter) for those types of ethernet
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable filter */
(1 << 29) | /* enable immediate interrupt */
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902 CFM eth protocol type */
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
------------------------------------------------------------------------------
Mattias Barthel
2015-10-08 07:33:50 UTC
Permalink
Hi
thanks for answer.

Unfortunately, the udp/ip stream just as the cfm stream is against only one
peer.
Furthemore on the guest side I have only one vcpu tied to two physical
siblings to minimize L2 cache misses on ISRs due to cache coherence.

I had an idea this morning, maybe I can make use of the ptp hw timestaming
for this? This should give better accuracy.
Is this possible or does it only work for ethertype ptp?
Post by Alexander Duyck
The difference could be Receive Side Scaling (RSS). The UDP/IP packets
can be load balanced across all queues via RSS based on a hash of the
source and destination addresses. So if you have multiple sources
generating these packets, or they are being received on multiple IP
addresses then it is possible the load is being spread out over multiple
CPUs. Your CFM packets are on an Ethertype other than IPv4 or IPv6 so
all packets will end up on queue 0 if I am not mistaken.
- Alex
Post by Mattias Barthel
Hi again!
I realise my message might not have been so clear.
I will try to clarify. I am doing PM with software timestamping.
version: 3.0.6-k2
firmware-version: 0.147-0
I do timestamping (NTP format) of the packets in the same place for all
protocols. (in igb_clean_rx_irq_adv()).
For these non ip packets like IEEE 801.2ag CFM ETH-DM (ethertype
0x8902) I
Post by Mattias Barthel
get quite a lot worse accuracy in the timestamping.
For UDP/IP TWAMP I get for 10k pps a max delay of around 140us back to
back.
Post by Mattias Barthel
For CFM ETH DM I get for the same packet rate and packet size a max delay
of around 900us.
Could FW be putting these L2 packets in another queue with different
characteristics?
I tried to add a ET-type filter as written in my previous mail but it
showed no difference.
Thanks for any help given!
On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel <
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when using the igb driver on i350 on linux
regarding
ETH CFM packets. This compared to TWAMP (IPv4 UDP packets).
The environment is KVM with the i350 devices in PCI-passthrough.
So i figured I add an own rx-queue (filter) for those types of ethernet
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable filter */
(1 << 29) | /* enable immediate interrupt */
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902 CFM eth protocol type */
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit
http://communities.intel.com/community/wired
--
Avis de confidentialité

Les informations contenues dans le présent message et dans toute piÚce qui
lui est jointe sont confidentielles et peuvent être protégées par le secret
professionnel. Ces informations sont à l’usage exclusif de son ou de ses
destinataires. Si vous recevez ce message par erreur, veuillez s’il vous
plait communiquer immédiatement avec l’expéditeur et en détruire tout
exemplaire. De plus, il vous est strictement interdit de le divulguer, de
le distribuer ou de le reproduire sans l’autorisation de l’expéditeur.
Merci.

Confidentiality notice

This e-mail message and any attachment hereto contain confidential
information which may be privileged and which is intended for the exclusive
use of its addressee(s). If you receive this message in error, please
inform sender immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or any attachment
hereto without the consent of the sender is strictly prohibited. Thank you.
Alexander Duyck
2015-10-08 16:01:38 UTC
Permalink
The i350 should support timestamping all packets as I recall. You
should probably look into getting the linuxptp package and seeing if you
can make use of the hwstamp_ctl command to enable timestamping for all
incoming packets on the adapter.

- Alex
Post by Mattias Barthel
Hi
thanks for answer.
Unfortunately, the udp/ip stream just as the cfm stream is against
only one peer.
Furthemore on the guest side I have only one vcpu tied to two physical
siblings to minimize L2 cache misses on ISRs due to cache coherence.
I had an idea this morning, maybe I can make use of the ptp hw
timestaming for this? This should give better accuracy.
Is this possible or does it only work for ethertype ptp?
On Thursday, October 8, 2015, Alexander Duyck
The difference could be Receive Side Scaling (RSS). The UDP/IP packets
can be load balanced across all queues via RSS based on a hash of the
source and destination addresses. So if you have multiple sources
generating these packets, or they are being received on multiple IP
addresses then it is possible the load is being spread out over multiple
CPUs. Your CFM packets are on an Ethertype other than IPv4 or IPv6 so
all packets will end up on queue 0 if I am not mistaken.
- Alex
Post by Mattias Barthel
Hi again!
I realise my message might not have been so clear.
I will try to clarify. I am doing PM with software timestamping.
version: 3.0.6-k2
firmware-version: 0.147-0
I do timestamping (NTP format) of the packets in the same place
for all
Post by Mattias Barthel
protocols. (in igb_clean_rx_irq_adv()).
For these non ip packets like IEEE 801.2ag CFM ETH-DM
(ethertype 0x8902) I
Post by Mattias Barthel
get quite a lot worse accuracy in the timestamping.
For UDP/IP TWAMP I get for 10k pps a max delay of around 140us
back to back.
Post by Mattias Barthel
For CFM ETH DM I get for the same packet rate and packet size a
max delay
Post by Mattias Barthel
of around 900us.
Could FW be putting these L2 packets in another queue with different
characteristics?
I tried to add a ET-type filter as written in my previous mail
but it
Post by Mattias Barthel
showed no difference.
Thanks for any help given!
On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when using the igb driver on i350
on linux
Post by Mattias Barthel
Post by Mattias Barthel
regarding
ETH CFM packets. This compared to TWAMP (IPv4 UDP packets).
The environment is KVM with the i350 devices in PCI-passthrough.
So i figured I add an own rx-queue (filter) for those types of
ethernet
Post by Mattias Barthel
Post by Mattias Barthel
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable filter */
(1 << 29) | /* enable immediate interrupt */
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902 CFM eth protocol type */
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit
http://communities.intel.com/community/wired
Avis de confidentialité
Les informations contenues dans le présent message et dans toute pièce
qui lui est jointe sont confidentielles et peuvent être protégées par
le secret professionnel. Ces informations sont à l’usage exclusif de
son ou de ses destinataires. Si vous recevez ce message par erreur,
veuillez s’il vous plait communiquer immédiatement avec l’expéditeur
et en détruire tout exemplaire. De plus, il vous est strictement
interdit de le divulguer, de le distribuer ou de le reproduire sans
l’autorisation de l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain confidential
information which may be privileged and which is intended for the
exclusive use of its addressee(s). If you receive this message in
error, please inform sender immediately and destroy any copy thereof.
Furthermore, any disclosure, distribution or copying of this message
and/or any attachment hereto without the consent of the sender is
strictly prohibited. Thank you.
------------------------------------------------------------------------------
Mattias Barthel
2015-10-12 15:04:54 UTC
Permalink
Hi again,

Started to try and timestamp all packets in the I350.

Driver version:

driver: igb
version: 3.0.6-k2
firmware-version: 0.147-0

Doing an ioctl that turns on timestamping for all packets.

hwconfig.tx_type = HWTSTAMP_TX_OFF;
hwconfig.rx_filter = HWTSTAMP_FILTER_ALL;
hwconfig_requested = hwconfig;
ioctl(sock, SIOCSHWTSTAMP, &hwtstamp)

The strangest thing is that although it works to read the timestamps the
clock seems to advance very slowly and/or incorrectly.

I just printout the HW timestamps alongside the systemclock and the hw
stamping is advancing the nanoseconds when the systemclock is advancing us
and seconds.

Is there a know bug in this version of the driver that maybe initializes
the clock incorrectly or maybe reads incorrectly from the clock?


[ 63.004116] igb_rx_hwtstamp
[ 63.004684] hwtstamp: 1444656871372825302, syststamp: 1444656871372845807
[ 63.005721] system timestamp: tv_sec: 1444656793, tv_usec: 683825
[ 63.006631] sys hw timestamp: tstamp: 1444656871372845807, tv_sec:
1444656871, tv_usec: 372845
[ 63.008044] hw timestamp: tstamp: 1444656871372825302, tv_sec:
1444656871, tv_usec: 372825
[ 63.009395]
[ 63.104111] igb_rx_hwtstamp
[ 63.104704] hwtstamp: 1444656871372825308, syststamp: 1444656871372845813
[ 63.105747] system timestamp: tv_sec: 1444656793, tv_usec: 783851
[ 63.106669] sys hw timestamp: tstamp: 1444656871372845813, tv_sec:
1444656871, tv_usec: 372845
[ 63.108075] hw timestamp: tstamp: 1444656871372825308, tv_sec:
1444656871, tv_usec: 372825
[ 63.109428]
[ 63.204109] igb_rx_hwtstamp
[ 63.204683] hwtstamp: 1444656871372825314, syststamp: 1444656871372845819
[ 63.206018] system timestamp: tv_sec: 1444656793, tv_usec: 884122
[ 63.207242] sys hw timestamp: tstamp: 1444656871372845819, tv_sec:
1444656871, tv_usec: 372845
[ 63.209096] hw timestamp: tstamp: 1444656871372825314, tv_sec:
1444656871, tv_usec: 372825
[ 63.210897]
[ 63.304068] igb_rx_hwtstamp
[ 63.304849] hwtstamp: 1444656871372825320, syststamp: 1444656871372845825
[ 63.306176] system timestamp: tv_sec: 1444656793, tv_usec: 984280
[ 63.307391] sys hw timestamp: tstamp: 1444656871372845825, tv_sec:
1444656871, tv_usec: 372845
[ 63.309252] hw timestamp: tstamp: 1444656871372825320, tv_sec:
1444656871, tv_usec: 372825
[ 63.311032]
[ 63.404107] igb_rx_hwtstamp
[ 63.404659] hwtstamp: 1444656871372825326, syststamp: 1444656871372845831
[ 63.405695] system timestamp: tv_sec: 1444656794, tv_usec: 83798
[ 63.406635] sys hw timestamp: tstamp: 1444656871372845831, tv_sec:
1444656871, tv_usec: 372845
[ 63.408044] hw timestamp: tstamp: 1444656871372825326, tv_sec:
1444656871, tv_usec: 372825
[ 63.409405]
[ 63.504084] igb_rx_hwtstamp
[ 63.504670] hwtstamp: 1444656871372825332, syststamp: 1444656871372845837
[ 63.505715] system timestamp: tv_sec: 1444656794, tv_usec: 183819
[ 63.506657] sys hw timestamp: tstamp: 1444656871372845837, tv_sec:
1444656871, tv_usec: 372845
[ 63.508090] hw timestamp: tstamp: 1444656871372825332, tv_sec:
1444656871, tv_usec: 372825


Any help will be very much appreciated.

Best regards,

Mattias
The i350 should support timestamping all packets as I recall. You should
probably look into getting the linuxptp package and seeing if you can make
use of the hwstamp_ctl command to enable timestamping for all incoming
packets on the adapter.
- Alex
Post by Mattias Barthel
Hi
thanks for answer.
Unfortunately, the udp/ip stream just as the cfm stream is against only
one peer.
Furthemore on the guest side I have only one vcpu tied to two physical
siblings to minimize L2 cache misses on ISRs due to cache coherence.
I had an idea this morning, maybe I can make use of the ptp hw
timestaming for this? This should give better accuracy.
Is this possible or does it only work for ethertype ptp?
The difference could be Receive Side Scaling (RSS). The UDP/IP packets
can be load balanced across all queues via RSS based on a hash of the
source and destination addresses. So if you have multiple sources
generating these packets, or they are being received on multiple IP
addresses then it is possible the load is being spread out over multiple
CPUs. Your CFM packets are on an Ethertype other than IPv4 or IPv6 so
all packets will end up on queue 0 if I am not mistaken.
- Alex
Post by Mattias Barthel
Hi again!
I realise my message might not have been so clear.
I will try to clarify. I am doing PM with software timestamping.
version: 3.0.6-k2
firmware-version: 0.147-0
I do timestamping (NTP format) of the packets in the same place
for all
Post by Mattias Barthel
protocols. (in igb_clean_rx_irq_adv()).
For these non ip packets like IEEE 801.2ag CFM ETH-DM
(ethertype 0x8902) I
Post by Mattias Barthel
get quite a lot worse accuracy in the timestamping.
For UDP/IP TWAMP I get for 10k pps a max delay of around 140us
back to back.
Post by Mattias Barthel
For CFM ETH DM I get for the same packet rate and packet size a
max delay
Post by Mattias Barthel
of around 900us.
Could FW be putting these L2 packets in another queue with different
characteristics?
I tried to add a ET-type filter as written in my previous mail
but it
Post by Mattias Barthel
showed no difference.
Thanks for any help given!
On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when using the igb driver on i350
on linux
Post by Mattias Barthel
Post by Mattias Barthel
regarding
ETH CFM packets. This compared to TWAMP (IPv4 UDP packets).
The environment is KVM with the i350 devices in PCI-passthrough.
So i figured I add an own rx-queue (filter) for those types of
ethernet
Post by Mattias Barthel
Post by Mattias Barthel
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable filter */
(1 << 29) | /* enable immediate interrupt */
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902 CFM eth protocol type */
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit
http://communities.intel.com/community/wired
Avis de confidentialité
Les informations contenues dans le présent message et dans toute piÚce
qui lui est jointe sont confidentielles et peuvent être protégées par le
secret professionnel. Ces informations sont à l’usage exclusif de son ou de
ses destinataires. Si vous recevez ce message par erreur, veuillez s’il
vous plait communiquer immédiatement avec l’expéditeur et en détruire tout
exemplaire. De plus, il vous est strictement interdit de le divulguer, de
le distribuer ou de le reproduire sans l’autorisation de l’expéditeur.
Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain confidential
information which may be privileged and which is intended for the exclusive
use of its addressee(s). If you receive this message in error, please
inform sender immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or any attachment
hereto without the consent of the sender is strictly prohibited. Thank you.
--
Mattias Barthel
Mattias Barthel
2015-10-13 07:53:20 UTC
Permalink
Hi, I think I managed to get around this problem.


in function init_hw_timer():
----
case e1000_i350:
case e1000_82580:

/*
* The 82580 timesync updates the system timer every 8ns by 8ns
* and the value cannot be shifted. Instead we need to shift
* the registers to generate a 64bit timer value. As a result
* SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by
* 24 in order to generate a larger value for synchronization.
*/
adapter->cycles.shift = IGB_82580_TSYNC_SHIFT;

----

I have the I350 and I dont think this cycles shift was meant for my adapter.
IGB_82580_TSYNC_SHIFT is 24.

Also when doing igb_read_clock,
this shifting is applied only for the 82580 and I guess it should not have
been set in the first place for the I350:
--------
/*
* The timestamp latches on lowest register read. For the 82580
* the lowest register is SYSTIMR instead of SYSTIML. However we never
* adjusted TIMINCA so SYSTIMR will just read as all 0s so ignore it.
*/
if (hw->mac.type == e1000_82580) {
stamp = rd32(E1000_SYSTIMR) >> 8;
shift = IGB_82580_TSYNC_SHIFT;
}

stamp |= (u64)rd32(E1000_SYSTIML) << shift;
stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32);
-------------

Setting adapter->cycles.shift to 0 makes the clock run at about the same
speed as the system clock.

Next problem I have is that the syststamp is not syncing correctly against
the system clock.

Any help is much appreciated.

Regards,

Mattias
Post by Mattias Barthel
Hi again,
Started to try and timestamp all packets in the I350.
driver: igb
version: 3.0.6-k2
firmware-version: 0.147-0
Doing an ioctl that turns on timestamping for all packets.
hwconfig.tx_type = HWTSTAMP_TX_OFF;
hwconfig.rx_filter = HWTSTAMP_FILTER_ALL;
hwconfig_requested = hwconfig;
ioctl(sock, SIOCSHWTSTAMP, &hwtstamp)
The strangest thing is that although it works to read the timestamps the
clock seems to advance very slowly and/or incorrectly.
I just printout the HW timestamps alongside the systemclock and the hw
stamping is advancing the nanoseconds when the systemclock is advancing us
and seconds.
Is there a know bug in this version of the driver that maybe initializes
the clock incorrectly or maybe reads incorrectly from the clock?
[ 63.004116] igb_rx_hwtstamp
1444656871372845807
[ 63.005721] system timestamp: tv_sec: 1444656793, tv_usec: 683825
1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.009395]
[ 63.104111] igb_rx_hwtstamp
1444656871372845813
[ 63.105747] system timestamp: tv_sec: 1444656793, tv_usec: 783851
1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.109428]
[ 63.204109] igb_rx_hwtstamp
1444656871372845819
[ 63.206018] system timestamp: tv_sec: 1444656793, tv_usec: 884122
1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.210897]
[ 63.304068] igb_rx_hwtstamp
1444656871372845825
[ 63.306176] system timestamp: tv_sec: 1444656793, tv_usec: 984280
1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.311032]
[ 63.404107] igb_rx_hwtstamp
1444656871372845831
[ 63.405695] system timestamp: tv_sec: 1444656794, tv_usec: 83798
1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.409405]
[ 63.504084] igb_rx_hwtstamp
1444656871372845837
[ 63.505715] system timestamp: tv_sec: 1444656794, tv_usec: 183819
1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
Any help will be very much appreciated.
Best regards,
Mattias
The i350 should support timestamping all packets as I recall. You should
probably look into getting the linuxptp package and seeing if you can make
use of the hwstamp_ctl command to enable timestamping for all incoming
packets on the adapter.
- Alex
Post by Mattias Barthel
Hi
thanks for answer.
Unfortunately, the udp/ip stream just as the cfm stream is against only
one peer.
Furthemore on the guest side I have only one vcpu tied to two physical
siblings to minimize L2 cache misses on ISRs due to cache coherence.
I had an idea this morning, maybe I can make use of the ptp hw
timestaming for this? This should give better accuracy.
Is this possible or does it only work for ethertype ptp?
The difference could be Receive Side Scaling (RSS). The UDP/IP packets
can be load balanced across all queues via RSS based on a hash of the
source and destination addresses. So if you have multiple sources
generating these packets, or they are being received on multiple IP
addresses then it is possible the load is being spread out over multiple
CPUs. Your CFM packets are on an Ethertype other than IPv4 or IPv6 so
all packets will end up on queue 0 if I am not mistaken.
- Alex
Post by Mattias Barthel
Hi again!
I realise my message might not have been so clear.
I will try to clarify. I am doing PM with software timestamping.
version: 3.0.6-k2
firmware-version: 0.147-0
I do timestamping (NTP format) of the packets in the same place
for all
Post by Mattias Barthel
protocols. (in igb_clean_rx_irq_adv()).
For these non ip packets like IEEE 801.2ag CFM ETH-DM
(ethertype 0x8902) I
Post by Mattias Barthel
get quite a lot worse accuracy in the timestamping.
For UDP/IP TWAMP I get for 10k pps a max delay of around 140us
back to back.
Post by Mattias Barthel
For CFM ETH DM I get for the same packet rate and packet size a
max delay
Post by Mattias Barthel
of around 900us.
Could FW be putting these L2 packets in another queue with
different
Post by Mattias Barthel
characteristics?
I tried to add a ET-type filter as written in my previous mail
but it
Post by Mattias Barthel
showed no difference.
Thanks for any help given!
On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when using the igb driver on i350
on linux
Post by Mattias Barthel
Post by Mattias Barthel
regarding
ETH CFM packets. This compared to TWAMP (IPv4 UDP packets).
The environment is KVM with the i350 devices in PCI-passthrough.
So i figured I add an own rx-queue (filter) for those types of
ethernet
Post by Mattias Barthel
Post by Mattias Barthel
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable filter */
(1 << 29) | /* enable immediate interrupt */
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902 CFM eth protocol type */
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit
http://communities.intel.com/community/wired
Avis de confidentialité
Les informations contenues dans le présent message et dans toute piÚce
qui lui est jointe sont confidentielles et peuvent être protégées par le
secret professionnel. Ces informations sont à l’usage exclusif de son ou de
ses destinataires. Si vous recevez ce message par erreur, veuillez s’il
vous plait communiquer immédiatement avec l’expéditeur et en détruire tout
exemplaire. De plus, il vous est strictement interdit de le divulguer, de
le distribuer ou de le reproduire sans l’autorisation de l’expéditeur.
Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain confidential
information which may be privileged and which is intended for the exclusive
use of its addressee(s). If you receive this message in error, please
inform sender immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or any attachment
hereto without the consent of the sender is strictly prohibited. Thank you.
--
Mattias Barthel
--
Mattias Barthel
Alexander Duyck
2015-10-13 15:51:43 UTC
Permalink
I'm assuming you are using either an older kernel or an out-of-tree
driver do I have that right? I ask because this code doesn't resemble
what is is currently in the latest Linux kernel.

Comments on what you have found are inline below.

- Alex
Post by Mattias Barthel
Hi, I think I managed to get around this problem.
----
/*
* The 82580 timesync updates the system timer every 8ns by 8ns
* and the value cannot be shifted. Instead we need to shift
* the registers to generate a 64bit timer value. As a result
* SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by
* 24 in order to generate a larger value for synchronization.
*/
adapter->cycles.shift = IGB_82580_TSYNC_SHIFT;
----
I have the I350 and I dont think this cycles shift was meant for my adapter.
IGB_82580_TSYNC_SHIFT is 24.
Also when doing igb_read_clock,
this shifting is applied only for the 82580 and I guess it should not
--------
/*
* The timestamp latches on lowest register read. For the 82580
* the lowest register is SYSTIMR instead of SYSTIML. However we never
* adjusted TIMINCA so SYSTIMR will just read as all 0s so ignore it.
*/
if (hw->mac.type == e1000_82580) {
stamp = rd32(E1000_SYSTIMR) >> 8;
shift = IGB_82580_TSYNC_SHIFT;
}
stamp |= (u64)rd32(E1000_SYSTIML) << shift;
stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32);
-------------
This is a bug in the code. The i350 should be shifted by the same value
as the 82580. Then the result would be the same as if you had set the
shift to 0. As I recall the reasoning behind the change was to make use
of as many bits of the time as possible, and on 82580 and newer the
SYSTIMH portion of the clock only contained 8 bits of actual clock data.

Without that change I believe there is a risk of the counter being
considered to have cycled too fast and causing errors as the cycle
counter will roll over at 40 bits instead of 64. The alternative would
be to update the cycle counter you are using so that it is aware that it
is limited to 40 bits and then use an shift of 0.
Post by Mattias Barthel
Setting adapter->cycles.shift to 0 makes the clock run at about the
same speed as the system clock.
Next problem I have is that the syststamp is not syncing correctly
against the system clock.
Any help is much appreciated.
Regards,
Mattias
On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel
Hi again,
Started to try and timestamp all packets in the I350.
driver: igb
version: 3.0.6-k2
firmware-version: 0.147-0
Doing an ioctl that turns on timestamping for all packets.
hwconfig.tx_type = HWTSTAMP_TX_OFF;
hwconfig.rx_filter = HWTSTAMP_FILTER_ALL;
hwconfig_requested = hwconfig;
ioctl(sock, SIOCSHWTSTAMP, &hwtstamp)
The strangest thing is that although it works to read the
timestamps the clock seems to advance very slowly and/or incorrectly.
I just printout the HW timestamps alongside the systemclock and
the hw stamping is advancing the nanoseconds when the systemclock
is advancing us and seconds.
Is there a know bug in this version of the driver that maybe
initializes the clock incorrectly or maybe reads incorrectly from
the clock?
[ 63.004116] igb_rx_hwtstamp
1444656871372845807
[ 63.005721] system timestamp: tv_sec: 1444656793, tv_usec: 683825
[ 63.006631] sys hw timestamp: tstamp: 1444656871372845807,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.009395]
[ 63.104111] igb_rx_hwtstamp
1444656871372845813
[ 63.105747] system timestamp: tv_sec: 1444656793, tv_usec: 783851
[ 63.106669] sys hw timestamp: tstamp: 1444656871372845813,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.109428]
[ 63.204109] igb_rx_hwtstamp
1444656871372845819
[ 63.206018] system timestamp: tv_sec: 1444656793, tv_usec: 884122
[ 63.207242] sys hw timestamp: tstamp: 1444656871372845819,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.210897]
[ 63.304068] igb_rx_hwtstamp
1444656871372845825
[ 63.306176] system timestamp: tv_sec: 1444656793, tv_usec: 984280
[ 63.307391] sys hw timestamp: tstamp: 1444656871372845825,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.311032]
[ 63.404107] igb_rx_hwtstamp
1444656871372845831
[ 63.405695] system timestamp: tv_sec: 1444656794, tv_usec: 83798
[ 63.406635] sys hw timestamp: tstamp: 1444656871372845831,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.409405]
[ 63.504084] igb_rx_hwtstamp
1444656871372845837
[ 63.505715] system timestamp: tv_sec: 1444656794, tv_usec: 183819
[ 63.506657] sys hw timestamp: tstamp: 1444656871372845837,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
Any help will be very much appreciated.
Best regards,
Mattias
On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck
The i350 should support timestamping all packets as I recall.
You should probably look into getting the linuxptp package and
seeing if you can make use of the hwstamp_ctl command to
enable timestamping for all incoming packets on the adapter.
- Alex
Hi
thanks for answer.
Unfortunately, the udp/ip stream just as the cfm stream is
against only one peer.
Furthemore on the guest side I have only one vcpu tied to
two physical siblings to minimize L2 cache misses on ISRs
due to cache coherence.
I had an idea this morning, maybe I can make use of the
ptp hw timestaming for this? This should give better accuracy.
Is this possible or does it only work for ethertype ptp?
On Thursday, October 8, 2015, Alexander Duyck
The difference could be Receive Side Scaling (RSS).
The UDP/IP
packets
can be load balanced across all queues via RSS based
on a hash of the
source and destination addresses. So if you have
multiple sources
generating these packets, or they are being received
on multiple IP
addresses then it is possible the load is being spread
out over
multiple
CPUs. Your CFM packets are on an Ethertype other than
IPv4 or IPv6 so
all packets will end up on queue 0 if I am not mistaken.
- Alex
Post by Mattias Barthel
Hi again!
I realise my message might not have been so clear.
I will try to clarify. I am doing PM with software
timestamping.
Post by Mattias Barthel
version: 3.0.6-k2
firmware-version: 0.147-0
I do timestamping (NTP format) of the packets in the
same place
for all
Post by Mattias Barthel
protocols. (in igb_clean_rx_irq_adv()).
For these non ip packets like IEEE 801.2ag CFM ETH-DM
(ethertype 0x8902) I
Post by Mattias Barthel
get quite a lot worse accuracy in the timestamping.
For UDP/IP TWAMP I get for 10k pps a max delay of
around 140us
back to back.
Post by Mattias Barthel
For CFM ETH DM I get for the same packet rate and
packet size a
max delay
Post by Mattias Barthel
of around 900us.
Could FW be putting these L2 packets in another
queue with different
Post by Mattias Barthel
characteristics?
I tried to add a ET-type filter as written in my
previous mail
but it
Post by Mattias Barthel
showed no difference.
Thanks for any help given!
On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when using the igb
driver on i350
on linux
Post by Mattias Barthel
Post by Mattias Barthel
regarding
ETH CFM packets. This compared to TWAMP (IPv4 UDP
packets).
Post by Mattias Barthel
Post by Mattias Barthel
The environment is KVM with the i350 devices in
PCI-passthrough.
Post by Mattias Barthel
Post by Mattias Barthel
So i figured I add an own rx-queue (filter) for
those types of
ethernet
Post by Mattias Barthel
Post by Mattias Barthel
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable filter */
(1 << 29) | /* enable immediate interrupt */
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902 CFM eth
protocol type */
Post by Mattias Barthel
Post by Mattias Barthel
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit
http://communities.intel.com/community/wired
Avis de confidentialité
Les informations contenues dans le présent message et dans
toute pièce qui lui est jointe sont confidentielles et
peuvent être protégées par le secret professionnel. Ces
informations sont à l’usage exclusif de son ou de ses
destinataires. Si vous recevez ce message par erreur,
veuillez s’il vous plait communiquer immédiatement avec
l’expéditeur et en détruire tout exemplaire. De plus, il
vous est strictement interdit de le divulguer, de le
distribuer ou de le reproduire sans l’autorisation de
l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain
confidential information which may be privileged and which
is intended for the exclusive use of its addressee(s). If
you receive this message in error, please inform sender
immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or
any attachment hereto without the consent of the sender is
strictly prohibited. Thank you.
--
Mattias Barthel
--
Mattias Barthel
------------------------------------------------------------------------------
Mattias Barthel
2015-10-13 17:01:10 UTC
Permalink
Yes, setting the cycles.shift to the same as on the 82580 and also shifting
the reads accordingly everything works. The sync of shhwtstamps->syststamp
too.

My driver version is 3.0.6-k2 and linux is 3.0.8.

Although this might not be very suited to my needs.
The incoming buffers wont be timestamped if you cant keep up with reading
the timestamping registers as I understand it.
This might be perfectly fine in the case of PTP where the packet rate is
fairly low. In my case Im looking at some 80kpps that all would need to be
timestamped.
There is a comment about this in the driver:

*/* If this bit is set, then the RX registers contain the time stamp.
No** other packet will be time stamped until we read these registers,
so**read the registers to make them available again. Because only
one** packet can be time stamped at a time, we know that the
register**values must belong to this one here and therefore we don't
need to** compare any of the additional attributes stored for it.**If
nothing went wrong, then it should have a shared tx_flags that we**
can turn into a skb_shared_hwtstamps. */*
I'm assuming you are using either an older kernel or an out-of-tree driver
do I have that right? I ask because this code doesn't resemble what is is
currently in the latest Linux kernel.
Comments on what you have found are inline below.
- Alex
Post by Mattias Barthel
Hi, I think I managed to get around this problem.
----
/*
* The 82580 timesync updates the system timer every 8ns by 8ns
* and the value cannot be shifted. Instead we need to shift
* the registers to generate a 64bit timer value. As a result
* SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by
* 24 in order to generate a larger value for synchronization.
*/
adapter->cycles.shift = IGB_82580_TSYNC_SHIFT;
----
I have the I350 and I dont think this cycles shift was meant for my adapter.
IGB_82580_TSYNC_SHIFT is 24.
Also when doing igb_read_clock,
this shifting is applied only for the 82580 and I guess it should not
--------
/*
* The timestamp latches on lowest register read. For the 82580
* the lowest register is SYSTIMR instead of SYSTIML. However we never
* adjusted TIMINCA so SYSTIMR will just read as all 0s so ignore it.
*/
if (hw->mac.type == e1000_82580) {
stamp = rd32(E1000_SYSTIMR) >> 8;
shift = IGB_82580_TSYNC_SHIFT;
}
stamp |= (u64)rd32(E1000_SYSTIML) << shift;
stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32);
-------------
This is a bug in the code. The i350 should be shifted by the same value
as the 82580. Then the result would be the same as if you had set the
shift to 0. As I recall the reasoning behind the change was to make use of
as many bits of the time as possible, and on 82580 and newer the SYSTIMH
portion of the clock only contained 8 bits of actual clock data.
Without that change I believe there is a risk of the counter being
considered to have cycled too fast and causing errors as the cycle counter
will roll over at 40 bits instead of 64. The alternative would be to
update the cycle counter you are using so that it is aware that it is
limited to 40 bits and then use an shift of 0.
Setting adapter->cycles.shift to 0 makes the clock run at about the same
Post by Mattias Barthel
speed as the system clock.
Next problem I have is that the syststamp is not syncing correctly
against the system clock.
Any help is much appreciated.
Regards,
Mattias
On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel <
Hi again,
Started to try and timestamp all packets in the I350.
driver: igb
version: 3.0.6-k2
firmware-version: 0.147-0
Doing an ioctl that turns on timestamping for all packets.
hwconfig.tx_type = HWTSTAMP_TX_OFF;
hwconfig.rx_filter = HWTSTAMP_FILTER_ALL;
hwconfig_requested = hwconfig;
ioctl(sock, SIOCSHWTSTAMP, &hwtstamp)
The strangest thing is that although it works to read the
timestamps the clock seems to advance very slowly and/or incorrectly.
I just printout the HW timestamps alongside the systemclock and
the hw stamping is advancing the nanoseconds when the systemclock
is advancing us and seconds.
Is there a know bug in this version of the driver that maybe
initializes the clock incorrectly or maybe reads incorrectly from
the clock?
[ 63.004116] igb_rx_hwtstamp
1444656871372845807
[ 63.005721] system timestamp: tv_sec: 1444656793, tv_usec: 683825
[ 63.006631] sys hw timestamp: tstamp: 1444656871372845807,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.009395]
[ 63.104111] igb_rx_hwtstamp
1444656871372845813
[ 63.105747] system timestamp: tv_sec: 1444656793, tv_usec: 783851
[ 63.106669] sys hw timestamp: tstamp: 1444656871372845813,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.109428]
[ 63.204109] igb_rx_hwtstamp
1444656871372845819
[ 63.206018] system timestamp: tv_sec: 1444656793, tv_usec: 884122
[ 63.207242] sys hw timestamp: tstamp: 1444656871372845819,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.210897]
[ 63.304068] igb_rx_hwtstamp
1444656871372845825
[ 63.306176] system timestamp: tv_sec: 1444656793, tv_usec: 984280
[ 63.307391] sys hw timestamp: tstamp: 1444656871372845825,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.311032]
[ 63.404107] igb_rx_hwtstamp
1444656871372845831
[ 63.405695] system timestamp: tv_sec: 1444656794, tv_usec: 83798
[ 63.406635] sys hw timestamp: tstamp: 1444656871372845831,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.409405]
[ 63.504084] igb_rx_hwtstamp
1444656871372845837
[ 63.505715] system timestamp: tv_sec: 1444656794, tv_usec: 183819
[ 63.506657] sys hw timestamp: tstamp: 1444656871372845837,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
Any help will be very much appreciated.
Best regards,
Mattias
On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck
The i350 should support timestamping all packets as I recall.
You should probably look into getting the linuxptp package and
seeing if you can make use of the hwstamp_ctl command to
enable timestamping for all incoming packets on the adapter.
- Alex
Hi
thanks for answer.
Unfortunately, the udp/ip stream just as the cfm stream is
against only one peer.
Furthemore on the guest side I have only one vcpu tied to
two physical siblings to minimize L2 cache misses on ISRs
due to cache coherence.
I had an idea this morning, maybe I can make use of the
ptp hw timestaming for this? This should give better accuracy.
Is this possible or does it only work for ethertype ptp?
On Thursday, October 8, 2015, Alexander Duyck
The difference could be Receive Side Scaling (RSS).
The UDP/IP
packets
can be load balanced across all queues via RSS based
on a hash of the
source and destination addresses. So if you have
multiple sources
generating these packets, or they are being received
on multiple IP
addresses then it is possible the load is being spread
out over
multiple
CPUs. Your CFM packets are on an Ethertype other than
IPv4 or IPv6 so
all packets will end up on queue 0 if I am not mistaken.
- Alex
Post by Mattias Barthel
Hi again!
I realise my message might not have been so clear.
I will try to clarify. I am doing PM with software
timestamping.
Post by Mattias Barthel
version: 3.0.6-k2
firmware-version: 0.147-0
I do timestamping (NTP format) of the packets in the
same place
for all
Post by Mattias Barthel
protocols. (in igb_clean_rx_irq_adv()).
For these non ip packets like IEEE 801.2ag CFM ETH-DM
(ethertype 0x8902) I
Post by Mattias Barthel
get quite a lot worse accuracy in the timestamping.
For UDP/IP TWAMP I get for 10k pps a max delay of
around 140us
back to back.
Post by Mattias Barthel
For CFM ETH DM I get for the same packet rate and
packet size a
max delay
Post by Mattias Barthel
of around 900us.
Could FW be putting these L2 packets in another
queue with different
Post by Mattias Barthel
characteristics?
I tried to add a ET-type filter as written in my
previous mail
but it
Post by Mattias Barthel
showed no difference.
Thanks for any help given!
On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when using the igb
driver on i350
on linux
Post by Mattias Barthel
Post by Mattias Barthel
regarding
ETH CFM packets. This compared to TWAMP (IPv4 UDP
packets).
Post by Mattias Barthel
Post by Mattias Barthel
The environment is KVM with the i350 devices in
PCI-passthrough.
Post by Mattias Barthel
Post by Mattias Barthel
So i figured I add an own rx-queue (filter) for
those types of
ethernet
Post by Mattias Barthel
Post by Mattias Barthel
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable filter */
(1 << 29) | /* enable immediate interrupt */
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902 CFM eth
protocol type */
Post by Mattias Barthel
Post by Mattias Barthel
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit
http://communities.intel.com/community/wired
Avis de confidentialité
Les informations contenues dans le présent message et dans
toute piÚce qui lui est jointe sont confidentielles et
peuvent être protégées par le secret professionnel. Ces
informations sont à l’usage exclusif de son ou de ses
destinataires. Si vous recevez ce message par erreur,
veuillez s’il vous plait communiquer immédiatement avec
l’expéditeur et en détruire tout exemplaire. De plus, il
vous est strictement interdit de le divulguer, de le
distribuer ou de le reproduire sans l’autorisation de
l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain
confidential information which may be privileged and which
is intended for the exclusive use of its addressee(s). If
you receive this message in error, please inform sender
immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or
any attachment hereto without the consent of the sender is
strictly prohibited. Thank you.
-- Mattias Barthel
--
Mattias Barthel
--
Avis de confidentialité

Les informations contenues dans le présent message et dans toute piÚce qui
lui est jointe sont confidentielles et peuvent être protégées par le secret
professionnel. Ces informations sont à l’usage exclusif de son ou de ses
destinataires. Si vous recevez ce message par erreur, veuillez s’il vous
plait communiquer immédiatement avec l’expéditeur et en détruire tout
exemplaire. De plus, il vous est strictement interdit de le divulguer, de
le distribuer ou de le reproduire sans l’autorisation de l’expéditeur.
Merci.

Confidentiality notice

This e-mail message and any attachment hereto contain confidential
information which may be privileged and which is intended for the exclusive
use of its addressee(s). If you receive this message in error, please
inform sender immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or any attachment
hereto without the consent of the sender is strictly prohibited. Thank you.
Alexander Duyck
2015-10-13 17:17:37 UTC
Permalink
I don't know if it is available in that driver or not, but there is an
option in later igb drivers to support a time stamp being written with
the packet data. Based on the fact that you are already seeing issues,
you might want to try getting the out-of-tree driver from e1000.sf.net
and seeing if you can use that with the time stamp in packet option that
is enabled when you select to timestamp all incoming Rx packets.

- Alex
Post by Mattias Barthel
Yes, setting the cycles.shift to the same as on the 82580 and also
shifting the reads accordingly everything works. The sync of
shhwtstamps->syststamp too.
My driver version is 3.0.6-k2 and linux is 3.0.8.
Although this might not be very suited to my needs.
The incoming buffers wont be timestamped if you cant keep up with
reading the timestamping registers as I understand it.
This might be perfectly fine in the case of PTP where the packet rate
is fairly low. In my case Im looking at some 80kpps that all would
need to be timestamped.
*//* If this bit is set, then the RX registers contain the time stamp.
No/**/ other packet will be time stamped until we read these
registers, so/* */read the registers to make them available again.
Because only one/**/ packet can be time stamped at a time, we know
that the register/* */values must belong to this one here and
therefore we don't need to/**/ compare any of the additional
attributes stored for it./* */If nothing went wrong, then it should
have a shared tx_flags that we/**/ can turn into a
skb_shared_hwtstamps. *//*
*//*
*//*
On Tuesday, October 13, 2015, Alexander Duyck
I'm assuming you are using either an older kernel or an
out-of-tree driver do I have that right? I ask because this code
doesn't resemble what is is currently in the latest Linux kernel.
Comments on what you have found are inline below.
- Alex
Hi, I think I managed to get around this problem.
----
/*
* The 82580 timesync updates the system timer every 8ns by 8ns
* and the value cannot be shifted. Instead we need to shift
* the registers to generate a 64bit timer value. As a result
* SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by
* 24 in order to generate a larger value for synchronization.
*/
adapter->cycles.shift = IGB_82580_TSYNC_SHIFT;
----
I have the I350 and I dont think this cycles shift was meant
for my adapter.
IGB_82580_TSYNC_SHIFT is 24.
Also when doing igb_read_clock,
this shifting is applied only for the 82580 and I guess it
--------
/*
* The timestamp latches on lowest register read. For the 82580
* the lowest register is SYSTIMR instead of SYSTIML. However we never
* adjusted TIMINCA so SYSTIMR will just read as all 0s so ignore it.
*/
if (hw->mac.type == e1000_82580) {
stamp = rd32(E1000_SYSTIMR) >> 8;
shift = IGB_82580_TSYNC_SHIFT;
}
stamp |= (u64)rd32(E1000_SYSTIML) << shift;
stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32);
-------------
This is a bug in the code. The i350 should be shifted by the same
value as the 82580. Then the result would be the same as if you
had set the shift to 0. As I recall the reasoning behind the
change was to make use of as many bits of the time as possible,
and on 82580 and newer the SYSTIMH portion of the clock only
contained 8 bits of actual clock data.
Without that change I believe there is a risk of the counter being
considered to have cycled too fast and causing errors as the cycle
counter will roll over at 40 bits instead of 64. The alternative
would be to update the cycle counter you are using so that it is
aware that it is limited to 40 bits and then use an shift of 0.
Setting adapter->cycles.shift to 0 makes the clock run at
about the same speed as the system clock.
Next problem I have is that the syststamp is not syncing
correctly against the system clock.
Any help is much appreciated.
Regards,
Mattias
On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel
Hi again,
Started to try and timestamp all packets in the I350.
driver: igb
version: 3.0.6-k2
firmware-version: 0.147-0
Doing an ioctl that turns on timestamping for all packets.
hwconfig.tx_type = HWTSTAMP_TX_OFF;
hwconfig.rx_filter = HWTSTAMP_FILTER_ALL;
hwconfig_requested = hwconfig;
ioctl(sock, SIOCSHWTSTAMP, &hwtstamp)
The strangest thing is that although it works to read the
timestamps the clock seems to advance very slowly and/or incorrectly.
I just printout the HW timestamps alongside the
systemclock and
the hw stamping is advancing the nanoseconds when the systemclock
is advancing us and seconds.
Is there a know bug in this version of the driver that maybe
initializes the clock incorrectly or maybe reads
incorrectly from
the clock?
[ 63.004116] igb_rx_hwtstamp
1444656871372845807
[ 63.005721] system timestamp: tv_sec: 1444656793,
tv_usec: 683825
[ 63.006631] sys hw timestamp: tstamp: 1444656871372845807,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.009395]
[ 63.104111] igb_rx_hwtstamp
1444656871372845813
[ 63.105747] system timestamp: tv_sec: 1444656793,
tv_usec: 783851
[ 63.106669] sys hw timestamp: tstamp: 1444656871372845813,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.109428]
[ 63.204109] igb_rx_hwtstamp
1444656871372845819
[ 63.206018] system timestamp: tv_sec: 1444656793,
tv_usec: 884122
[ 63.207242] sys hw timestamp: tstamp: 1444656871372845819,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.210897]
[ 63.304068] igb_rx_hwtstamp
1444656871372845825
[ 63.306176] system timestamp: tv_sec: 1444656793,
tv_usec: 984280
[ 63.307391] sys hw timestamp: tstamp: 1444656871372845825,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.311032]
[ 63.404107] igb_rx_hwtstamp
1444656871372845831
[ 63.405695] system timestamp: tv_sec: 1444656794,
tv_usec: 83798
[ 63.406635] sys hw timestamp: tstamp: 1444656871372845831,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.409405]
[ 63.504084] igb_rx_hwtstamp
1444656871372845837
[ 63.505715] system timestamp: tv_sec: 1444656794,
tv_usec: 183819
[ 63.506657] sys hw timestamp: tstamp: 1444656871372845837,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
Any help will be very much appreciated.
Best regards,
Mattias
On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck
The i350 should support timestamping all packets as I
recall. You should probably look into getting the
linuxptp package and
seeing if you can make use of the hwstamp_ctl command to
enable timestamping for all incoming packets on the adapter.
- Alex
Hi
thanks for answer.
Unfortunately, the udp/ip stream just as the cfm stream is
against only one peer.
Furthemore on the guest side I have only one vcpu tied to
two physical siblings to minimize L2 cache misses on ISRs
due to cache coherence.
I had an idea this morning, maybe I can make use of the
ptp hw timestaming for this? This should give better accuracy.
Is this possible or does it only work for
ethertype ptp?
On Thursday, October 8, 2015, Alexander Duyck
The difference could be Receive Side Scaling
(RSS). The UDP/IP
packets
can be load balanced across all queues via RSS based
on a hash of the
source and destination addresses. So if you have
multiple sources
generating these packets, or they are being received
on multiple IP
addresses then it is possible the load is being spread
out over
multiple
CPUs. Your CFM packets are on an Ethertype other than
IPv4 or IPv6 so
all packets will end up on queue 0 if I am not mistaken.
- Alex
Post by Mattias Barthel
Hi again!
I realise my message might not have been so
clear.
Post by Mattias Barthel
I will try to clarify. I am doing PM with
software
timestamping.
Post by Mattias Barthel
version: 3.0.6-k2
firmware-version: 0.147-0
I do timestamping (NTP format) of the
packets in the
same place
for all
Post by Mattias Barthel
protocols. (in igb_clean_rx_irq_adv()).
For these non ip packets like IEEE 801.2ag
CFM ETH-DM
(ethertype 0x8902) I
Post by Mattias Barthel
get quite a lot worse accuracy in the
timestamping.
Post by Mattias Barthel
For UDP/IP TWAMP I get for 10k pps a max
delay of
around 140us
back to back.
Post by Mattias Barthel
For CFM ETH DM I get for the same packet
rate and
packet size a
max delay
Post by Mattias Barthel
of around 900us.
Could FW be putting these L2 packets in another
queue with different
Post by Mattias Barthel
characteristics?
I tried to add a ET-type filter as written in my
previous mail
but it
Post by Mattias Barthel
showed no difference.
Thanks for any help given!
On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when using the igb
driver on i350
on linux
Post by Mattias Barthel
Post by Mattias Barthel
regarding
ETH CFM packets. This compared to TWAMP
(IPv4 UDP
packets).
Post by Mattias Barthel
Post by Mattias Barthel
The environment is KVM with the i350 devices in
PCI-passthrough.
Post by Mattias Barthel
Post by Mattias Barthel
So i figured I add an own rx-queue (filter) for
those types of
ethernet
Post by Mattias Barthel
Post by Mattias Barthel
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable
filter */
Post by Mattias Barthel
Post by Mattias Barthel
(1 << 29) | /* enable immediate
interrupt */
Post by Mattias Barthel
Post by Mattias Barthel
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902 CFM eth
protocol type */
Post by Mattias Barthel
Post by Mattias Barthel
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
<javascript:;>
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit
http://communities.intel.com/community/wired
Avis de confidentialité
Les informations contenues dans le présent message et dans
toute pièce qui lui est jointe sont confidentielles et
peuvent être protégées par le secret
professionnel. Ces
informations sont à l’usage exclusif de son ou de ses
destinataires. Si vous recevez ce message par erreur,
veuillez s’il vous plait communiquer immédiatement avec
l’expéditeur et en détruire tout exemplaire. De plus, il
vous est strictement interdit de le divulguer, de le
distribuer ou de le reproduire sans l’autorisation de
l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain
confidential information which may be privileged and which
is intended for the exclusive use of its
addressee(s). If
you receive this message in error, please inform sender
immediately and destroy any copy thereof.
Furthermore, any
disclosure, distribution or copying of this message and/or
any attachment hereto without the consent of the sender is
strictly prohibited. Thank you.
-- Mattias Barthel
--
Mattias Barthel
Avis de confidentialité
Les informations contenues dans le présent message et dans toute pièce
qui lui est jointe sont confidentielles et peuvent être protégées par
le secret professionnel. Ces informations sont à l’usage exclusif de
son ou de ses destinataires. Si vous recevez ce message par erreur,
veuillez s’il vous plait communiquer immédiatement avec l’expéditeur
et en détruire tout exemplaire. De plus, il vous est strictement
interdit de le divulguer, de le distribuer ou de le reproduire sans
l’autorisation de l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain confidential
information which may be privileged and which is intended for the
exclusive use of its addressee(s). If you receive this message in
error, please inform sender immediately and destroy any copy thereof.
Furthermore, any disclosure, distribution or copying of this message
and/or any attachment hereto without the consent of the sender is
strictly prohibited. Thank you.
------------------------------------------------------------------------------
Mattias Barthel
2015-10-13 17:26:53 UTC
Permalink
Thanks a lot Alex, there might still be some hope then.
Unfortunately Im bound to using the linux kernel 3.0.8. What out-of-tree
driver can I use with that, please?
Post by Alexander Duyck
I don't know if it is available in that driver or not, but there is an
option in later igb drivers to support a time stamp being written with the
packet data. Based on the fact that you are already seeing issues, you
might want to try getting the out-of-tree driver from e1000.sf.net and
seeing if you can use that with the time stamp in packet option that is
enabled when you select to timestamp all incoming Rx packets.
- Alex
Post by Mattias Barthel
Yes, setting the cycles.shift to the same as on the 82580 and also
shifting the reads accordingly everything works. The sync of
shhwtstamps->syststamp too.
My driver version is 3.0.6-k2 and linux is 3.0.8.
Although this might not be very suited to my needs.
The incoming buffers wont be timestamped if you cant keep up with reading
the timestamping registers as I understand it.
This might be perfectly fine in the case of PTP where the packet rate is
fairly low. In my case Im looking at some 80kpps that all would need to be
timestamped.
*//* If this bit is set, then the RX registers contain the time stamp.
No/**/ other packet will be time stamped until we read these registers,
so/* */read the registers to make them available again. Because only
one/**/ packet can be time stamped at a time, we know that the register/*
*/values must belong to this one here and therefore we don't need to/**/
compare any of the additional attributes stored for it./* */If nothing went
wrong, then it should have a shared tx_flags that we/**/ can turn into a
skb_shared_hwtstamps. *//*
*//*
*//*
I'm assuming you are using either an older kernel or an
out-of-tree driver do I have that right? I ask because this code
doesn't resemble what is is currently in the latest Linux kernel.
Comments on what you have found are inline below.
- Alex
Hi, I think I managed to get around this problem.
----
/*
* The 82580 timesync updates the system timer every 8ns by 8ns
* and the value cannot be shifted. Instead we need to shift
* the registers to generate a 64bit timer value. As a result
* SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by
* 24 in order to generate a larger value for synchronization.
*/
adapter->cycles.shift = IGB_82580_TSYNC_SHIFT;
----
I have the I350 and I dont think this cycles shift was meant
for my adapter.
IGB_82580_TSYNC_SHIFT is 24.
Also when doing igb_read_clock,
this shifting is applied only for the 82580 and I guess it
--------
/*
* The timestamp latches on lowest register read. For the 82580
* the lowest register is SYSTIMR instead of SYSTIML. However we never
* adjusted TIMINCA so SYSTIMR will just read as all 0s so ignore it.
*/
if (hw->mac.type == e1000_82580) {
stamp = rd32(E1000_SYSTIMR) >> 8;
shift = IGB_82580_TSYNC_SHIFT;
}
stamp |= (u64)rd32(E1000_SYSTIML) << shift;
stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32);
-------------
This is a bug in the code. The i350 should be shifted by the same
value as the 82580. Then the result would be the same as if you
had set the shift to 0. As I recall the reasoning behind the
change was to make use of as many bits of the time as possible,
and on 82580 and newer the SYSTIMH portion of the clock only
contained 8 bits of actual clock data.
Without that change I believe there is a risk of the counter being
considered to have cycled too fast and causing errors as the cycle
counter will roll over at 40 bits instead of 64. The alternative
would be to update the cycle counter you are using so that it is
aware that it is limited to 40 bits and then use an shift of 0.
Setting adapter->cycles.shift to 0 makes the clock run at
about the same speed as the system clock.
Next problem I have is that the syststamp is not syncing
correctly against the system clock.
Any help is much appreciated.
Regards,
Mattias
On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel
Hi again,
Started to try and timestamp all packets in the I350.
driver: igb
version: 3.0.6-k2
firmware-version: 0.147-0
Doing an ioctl that turns on timestamping for all packets.
hwconfig.tx_type = HWTSTAMP_TX_OFF;
hwconfig.rx_filter = HWTSTAMP_FILTER_ALL;
hwconfig_requested = hwconfig;
ioctl(sock, SIOCSHWTSTAMP, &hwtstamp)
The strangest thing is that although it works to read the
timestamps the clock seems to advance very slowly and/or incorrectly.
I just printout the HW timestamps alongside the
systemclock and
the hw stamping is advancing the nanoseconds when the systemclock
is advancing us and seconds.
Is there a know bug in this version of the driver that maybe
initializes the clock incorrectly or maybe reads
incorrectly from
the clock?
[ 63.004116] igb_rx_hwtstamp
1444656871372845807
[ 63.005721] system timestamp: tv_sec: 1444656793, tv_usec: 683825
[ 63.006631] sys hw timestamp: tstamp: 1444656871372845807,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.009395]
[ 63.104111] igb_rx_hwtstamp
1444656871372845813
[ 63.105747] system timestamp: tv_sec: 1444656793, tv_usec: 783851
[ 63.106669] sys hw timestamp: tstamp: 1444656871372845813,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.109428]
[ 63.204109] igb_rx_hwtstamp
1444656871372845819
[ 63.206018] system timestamp: tv_sec: 1444656793, tv_usec: 884122
[ 63.207242] sys hw timestamp: tstamp: 1444656871372845819,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.210897]
[ 63.304068] igb_rx_hwtstamp
1444656871372845825
[ 63.306176] system timestamp: tv_sec: 1444656793, tv_usec: 984280
[ 63.307391] sys hw timestamp: tstamp: 1444656871372845825,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.311032]
[ 63.404107] igb_rx_hwtstamp
1444656871372845831
[ 63.405695] system timestamp: tv_sec: 1444656794, tv_usec: 83798
[ 63.406635] sys hw timestamp: tstamp: 1444656871372845831,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
[ 63.409405]
[ 63.504084] igb_rx_hwtstamp
1444656871372845837
[ 63.505715] system timestamp: tv_sec: 1444656794, tv_usec: 183819
[ 63.506657] sys hw timestamp: tstamp: 1444656871372845837,
tv_sec: 1444656871, tv_usec: 372845
1444656871, tv_usec: 372825
Any help will be very much appreciated.
Best regards,
Mattias
On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck
The i350 should support timestamping all packets as I
recall. You should probably look into getting the
linuxptp package and
seeing if you can make use of the hwstamp_ctl command to
enable timestamping for all incoming packets on the adapter.
- Alex
Hi
thanks for answer.
Unfortunately, the udp/ip stream just as the cfm stream is
against only one peer.
Furthemore on the guest side I have only one vcpu tied to
two physical siblings to minimize L2 cache misses on ISRs
due to cache coherence.
I had an idea this morning, maybe I can make use of the
ptp hw timestaming for this? This should give
better accuracy.
Is this possible or does it only work for
ethertype ptp?
On Thursday, October 8, 2015, Alexander Duyck
The difference could be Receive Side Scaling
(RSS). The UDP/IP
packets
can be load balanced across all queues via RSS based
on a hash of the
source and destination addresses. So if you have
multiple sources
generating these packets, or they are being received
on multiple IP
addresses then it is possible the load is being spread
out over
multiple
CPUs. Your CFM packets are on an Ethertype other than
IPv4 or IPv6 so
all packets will end up on queue 0 if I am not mistaken.
- Alex
Post by Mattias Barthel
Hi again!
I realise my message might not have been so
clear.
Post by Mattias Barthel
I will try to clarify. I am doing PM with
software
timestamping.
Post by Mattias Barthel
version: 3.0.6-k2
firmware-version: 0.147-0
I do timestamping (NTP format) of the
packets in the
same place
for all
Post by Mattias Barthel
protocols. (in igb_clean_rx_irq_adv()).
For these non ip packets like IEEE 801.2ag
CFM ETH-DM
(ethertype 0x8902) I
Post by Mattias Barthel
get quite a lot worse accuracy in the
timestamping.
Post by Mattias Barthel
For UDP/IP TWAMP I get for 10k pps a max
delay of
around 140us
back to back.
Post by Mattias Barthel
For CFM ETH DM I get for the same packet
rate and
packet size a
max delay
Post by Mattias Barthel
of around 900us.
Could FW be putting these L2 packets in another
queue with different
Post by Mattias Barthel
characteristics?
I tried to add a ET-type filter as written in my
previous mail
but it
Post by Mattias Barthel
showed no difference.
Thanks for any help given!
On Mon, Oct 5, 2015 at 2:19 PM, Mattias Barthel
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when using the igb
driver on i350
on linux
Post by Mattias Barthel
Post by Mattias Barthel
regarding
ETH CFM packets. This compared to TWAMP
(IPv4 UDP
packets).
Post by Mattias Barthel
Post by Mattias Barthel
The environment is KVM with the i350 devices in
PCI-passthrough.
Post by Mattias Barthel
Post by Mattias Barthel
So i figured I add an own rx-queue (filter) for
those types of
ethernet
Post by Mattias Barthel
Post by Mattias Barthel
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable
filter */
Post by Mattias Barthel
Post by Mattias Barthel
(1 << 29) | /* enable immediate
interrupt */
Post by Mattias Barthel
Post by Mattias Barthel
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902 CFM eth
protocol type */
Post by Mattias Barthel
Post by Mattias Barthel
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit
http://communities.intel.com/community/wired
Avis de confidentialité
Les informations contenues dans le présent message et dans
toute piÚce qui lui est jointe sont confidentielles et
peuvent être protégées par le secret
professionnel. Ces
informations sont à l’usage exclusif de son ou de ses
destinataires. Si vous recevez ce message par erreur,
veuillez s’il vous plait communiquer immédiatement
avec
l’expéditeur et en détruire tout exemplaire. De
plus, il
vous est strictement interdit de le divulguer, de le
distribuer ou de le reproduire sans l’autorisation de
l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain
confidential information which may be privileged and which
is intended for the exclusive use of its
addressee(s). If
you receive this message in error, please inform sender
immediately and destroy any copy thereof.
Furthermore, any
disclosure, distribution or copying of this message and/or
any attachment hereto without the consent of the sender is
strictly prohibited. Thank you.
-- Mattias Barthel
-- Mattias Barthel
Avis de confidentialité
Les informations contenues dans le présent message et dans toute piÚce
qui lui est jointe sont confidentielles et peuvent être protégées par le
secret professionnel. Ces informations sont à l’usage exclusif de son ou de
ses destinataires. Si vous recevez ce message par erreur, veuillez s’il
vous plait communiquer immédiatement avec l’expéditeur et en détruire tout
exemplaire. De plus, il vous est strictement interdit de le divulguer, de
le distribuer ou de le reproduire sans l’autorisation de l’expéditeur.
Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain confidential
information which may be privileged and which is intended for the exclusive
use of its addressee(s). If you receive this message in error, please
inform sender immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or any attachment
hereto without the consent of the sender is strictly prohibited. Thank you.
--
Avis de confidentialité

Les informations contenues dans le présent message et dans toute piÚce qui
lui est jointe sont confidentielles et peuvent être protégées par le secret
professionnel. Ces informations sont à l’usage exclusif de son ou de ses
destinataires. Si vous recevez ce message par erreur, veuillez s’il vous
plait communiquer immédiatement avec l’expéditeur et en détruire tout
exemplaire. De plus, il vous est strictement interdit de le divulguer, de
le distribuer ou de le reproduire sans l’autorisation de l’expéditeur.
Merci.

Confidentiality notice

This e-mail message and any attachment hereto contain confidential
information which may be privileged and which is intended for the exclusive
use of its addressee(s). If you receive this message in error, please
inform sender immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or any attachment
hereto without the consent of the sender is strictly prohibited. Thank you.
Alexander Duyck
2015-10-13 18:45:06 UTC
Permalink
You can probably just download and use the latest driver from:
http://sourceforge.net/projects/e1000/files/igb%20stable/

It should be able to build on a 3.0.8 kernel provided nothing has been
back-ported that would change the APIs.

- Alex
Post by Mattias Barthel
Thanks a lot Alex, there might still be some hope then.
Unfortunately Im bound to using the linux kernel 3.0.8. What
out-of-tree driver can I use with that, please?
On Tuesday, October 13, 2015, Alexander Duyck
I don't know if it is available in that driver or not, but there
is an option in later igb drivers to support a time stamp being
written with the packet data. Based on the fact that you are
already seeing issues, you might want to try getting the
out-of-tree driver from e1000.sf.net <http://e1000.sf.net> and
seeing if you can use that with the time stamp in packet option
that is enabled when you select to timestamp all incoming Rx packets.
- Alex
Yes, setting the cycles.shift to the same as on the 82580 and
also shifting the reads accordingly everything works. The sync
of shhwtstamps->syststamp too.
My driver version is 3.0.6-k2 and linux is 3.0.8.
Although this might not be very suited to my needs.
The incoming buffers wont be timestamped if you cant keep up
with reading the timestamping registers as I understand it.
This might be perfectly fine in the case of PTP where the
packet rate is fairly low. In my case Im looking at some
80kpps that all would need to be timestamped.
*//* If this bit is set, then the RX registers contain the
time stamp. No/**/ other packet will be time stamped until we
read these registers, so/* */read the registers to make them
available again. Because only one/**/ packet can be time
stamped at a time, we know that the register/* */values must
belong to this one here and therefore we don't need to/**/
compare any of the additional attributes stored for it./* */If
nothing went wrong, then it should have a shared tx_flags that
we/**/ can turn into a skb_shared_hwtstamps. *//*
*//*
*//*
On Tuesday, October 13, 2015, Alexander Duyck
I'm assuming you are using either an older kernel or an
out-of-tree driver do I have that right? I ask because this code
doesn't resemble what is is currently in the latest Linux kernel.
Comments on what you have found are inline below.
- Alex
Hi, I think I managed to get around this problem.
----
/*
* The 82580 timesync updates the system timer every 8ns by 8ns
* and the value cannot be shifted. Instead we need to shift
* the registers to generate a 64bit timer value. As a result
* SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by
* 24 in order to generate a larger value for
synchronization.
*/
adapter->cycles.shift = IGB_82580_TSYNC_SHIFT;
----
I have the I350 and I dont think this cycles shift was meant
for my adapter.
IGB_82580_TSYNC_SHIFT is 24.
Also when doing igb_read_clock,
this shifting is applied only for the 82580 and I guess it
--------
/*
* The timestamp latches on lowest register read. For the 82580
* the lowest register is SYSTIMR instead of SYSTIML.
However
we never
* adjusted TIMINCA so SYSTIMR will just read as all 0s so
ignore it.
*/
if (hw->mac.type == e1000_82580) {
stamp = rd32(E1000_SYSTIMR) >> 8;
shift = IGB_82580_TSYNC_SHIFT;
}
stamp |= (u64)rd32(E1000_SYSTIML) << shift;
stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32);
-------------
This is a bug in the code. The i350 should be shifted by the same
value as the 82580. Then the result would be the same as if you
had set the shift to 0. As I recall the reasoning behind the
change was to make use of as many bits of the time as possible,
and on 82580 and newer the SYSTIMH portion of the clock only
contained 8 bits of actual clock data.
Without that change I believe there is a risk of the counter being
considered to have cycled too fast and causing errors as the cycle
counter will roll over at 40 bits instead of 64. The alternative
would be to update the cycle counter you are using so that it is
aware that it is limited to 40 bits and then use an shift of 0.
Setting adapter->cycles.shift to 0 makes the clock run at
about the same speed as the system clock.
Next problem I have is that the syststamp is not syncing
correctly against the system clock.
Any help is much appreciated.
Regards,
Mattias
On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel
Hi again,
Started to try and timestamp all packets in the I350.
driver: igb
version: 3.0.6-k2
firmware-version: 0.147-0
Doing an ioctl that turns on timestamping for all packets.
hwconfig.tx_type = HWTSTAMP_TX_OFF;
hwconfig.rx_filter = HWTSTAMP_FILTER_ALL;
hwconfig_requested = hwconfig;
ioctl(sock, SIOCSHWTSTAMP, &hwtstamp)
The strangest thing is that although it works to read the
timestamps the clock seems to advance very slowly
and/or
incorrectly.
I just printout the HW timestamps alongside the
systemclock and
the hw stamping is advancing the nanoseconds when the
systemclock
is advancing us and seconds.
Is there a know bug in this version of the driver that maybe
initializes the clock incorrectly or maybe reads
incorrectly from
the clock?
[ 63.004116] igb_rx_hwtstamp
1444656871372845807
[ 63.005721] system timestamp: tv_sec: 1444656793,
tv_usec: 683825
1444656871372845807,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825302,
1444656871, tv_usec: 372825
[ 63.009395]
[ 63.104111] igb_rx_hwtstamp
1444656871372845813
[ 63.105747] system timestamp: tv_sec: 1444656793,
tv_usec: 783851
1444656871372845813,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825308,
1444656871, tv_usec: 372825
[ 63.109428]
[ 63.204109] igb_rx_hwtstamp
1444656871372845819
[ 63.206018] system timestamp: tv_sec: 1444656793,
tv_usec: 884122
1444656871372845819,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825314,
1444656871, tv_usec: 372825
[ 63.210897]
[ 63.304068] igb_rx_hwtstamp
1444656871372845825
[ 63.306176] system timestamp: tv_sec: 1444656793,
tv_usec: 984280
1444656871372845825,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825320,
1444656871, tv_usec: 372825
[ 63.311032]
[ 63.404107] igb_rx_hwtstamp
1444656871372845831
[ 63.405695] system timestamp: tv_sec: 1444656794,
tv_usec: 83798
1444656871372845831,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825326,
1444656871, tv_usec: 372825
[ 63.409405]
[ 63.504084] igb_rx_hwtstamp
1444656871372845837
[ 63.505715] system timestamp: tv_sec: 1444656794,
tv_usec: 183819
1444656871372845837,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825332,
1444656871, tv_usec: 372825
Any help will be very much appreciated.
Best regards,
Mattias
On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck
The i350 should support timestamping all packets as I
recall. You should probably look into getting the
linuxptp package and
seeing if you can make use of the hwstamp_ctl command to
enable timestamping for all incoming packets
on the
adapter.
- Alex
Hi
thanks for answer.
Unfortunately, the udp/ip stream just as
the cfm
stream is
against only one peer.
Furthemore on the guest side I have only
one vcpu
tied to
two physical siblings to minimize L2 cache
misses
on ISRs
due to cache coherence.
I had an idea this morning, maybe I can
make use
of the
ptp hw timestaming for this? This should give
better accuracy.
Is this possible or does it only work for
ethertype ptp?
On Thursday, October 8, 2015, Alexander Duyck
The difference could be Receive Side Scaling
(RSS). The UDP/IP
packets
can be load balanced across all queues
via RSS
based
on a hash of the
source and destination addresses. So
if you have
multiple sources
generating these packets, or they are
being
received
on multiple IP
addresses then it is possible the load is
being spread
out over
multiple
CPUs. Your CFM packets are on an Ethertype
other than
IPv4 or IPv6 so
all packets will end up on queue 0 if
I am not
mistaken.
- Alex
Post by Mattias Barthel
Hi again!
I realise my message might not have
been so
clear.
Post by Mattias Barthel
I will try to clarify. I am doing PM
with
software
timestamping.
Post by Mattias Barthel
version: 3.0.6-k2
firmware-version: 0.147-0
I do timestamping (NTP format) of the
packets in the
same place
for all
Post by Mattias Barthel
protocols. (in igb_clean_rx_irq_adv()).
For these non ip packets like IEEE
801.2ag CFM ETH-DM
(ethertype 0x8902) I
Post by Mattias Barthel
get quite a lot worse accuracy in the
timestamping.
Post by Mattias Barthel
For UDP/IP TWAMP I get for 10k pps a max
delay of
around 140us
back to back.
Post by Mattias Barthel
For CFM ETH DM I get for the same packet
rate and
packet size a
max delay
Post by Mattias Barthel
of around 900us.
Could FW be putting these L2 packets
in another
queue with different
Post by Mattias Barthel
characteristics?
I tried to add a ET-type filter as
written in my
previous mail
but it
Post by Mattias Barthel
showed no difference.
Thanks for any help given!
On Mon, Oct 5, 2015 at 2:19 PM,
Mattias Barthel
<javascript:;>>
Post by Mattias Barthel
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when
using the igb
driver on i350
on linux
Post by Mattias Barthel
Post by Mattias Barthel
regarding
ETH CFM packets. This compared to TWAMP
(IPv4 UDP
packets).
Post by Mattias Barthel
Post by Mattias Barthel
The environment is KVM with the
i350 devices in
PCI-passthrough.
Post by Mattias Barthel
Post by Mattias Barthel
So i figured I add an own rx-queue
(filter) for
those types of
ethernet
Post by Mattias Barthel
Post by Mattias Barthel
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable
filter */
Post by Mattias Barthel
Post by Mattias Barthel
(1 << 29) | /* enable immediate
interrupt */
Post by Mattias Barthel
Post by Mattias Barthel
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902
CFM eth
protocol type */
Post by Mattias Barthel
Post by Mattias Barthel
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
<javascript:;>
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit
http://communities.intel.com/community/wired
Avis de confidentialité
Les informations contenues dans le présent
message
et dans
toute pièce qui lui est jointe sont confidentielles et
peuvent être protégées par le secret
professionnel. Ces
informations sont à l’usage exclusif de
son ou de ses
destinataires. Si vous recevez ce message par erreur,
veuillez s’il vous plait communiquer
immédiatement
avec
l’expéditeur et en détruire tout
exemplaire. De
plus, il
vous est strictement interdit de le divulguer, de le
distribuer ou de le reproduire sans l’autorisation de
l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain
confidential information which may be
privileged
and which
is intended for the exclusive use of its
addressee(s). If
you receive this message in error, please
inform
sender
immediately and destroy any copy thereof.
Furthermore, any
disclosure, distribution or copying of this
message and/or
any attachment hereto without the consent
of the
sender is
strictly prohibited. Thank you.
-- Mattias Barthel
-- Mattias Barthel
Avis de confidentialité
Les informations contenues dans le présent message et dans
toute pièce qui lui est jointe sont confidentielles et peuvent
être protégées par le secret professionnel. Ces informations
sont à l’usage exclusif de son ou de ses destinataires. Si
vous recevez ce message par erreur, veuillez s’il vous plait
communiquer immédiatement avec l’expéditeur et en détruire
tout exemplaire. De plus, il vous est strictement interdit de
le divulguer, de le distribuer ou de le reproduire sans
l’autorisation de l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain
confidential information which may be privileged and which is
intended for the exclusive use of its addressee(s). If you
receive this message in error, please inform sender
immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or any
attachment hereto without the consent of the sender is
strictly prohibited. Thank you.
Avis de confidentialité
Les informations contenues dans le présent message et dans toute pièce
qui lui est jointe sont confidentielles et peuvent être protégées par
le secret professionnel. Ces informations sont à l’usage exclusif de
son ou de ses destinataires. Si vous recevez ce message par erreur,
veuillez s’il vous plait communiquer immédiatement avec l’expéditeur
et en détruire tout exemplaire. De plus, il vous est strictement
interdit de le divulguer, de le distribuer ou de le reproduire sans
l’autorisation de l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain confidential
information which may be privileged and which is intended for the
exclusive use of its addressee(s). If you receive this message in
error, please inform sender immediately and destroy any copy thereof.
Furthermore, any disclosure, distribution or copying of this message
and/or any attachment hereto without the consent of the sender is
strictly prohibited. Thank you.
------------------------------------------------------------------------------
Mattias Barthel
2015-10-13 19:47:06 UTC
Permalink
Thanks again Alex,
It might be a good idea to upgrade the driver, who knows what more there is
in store for me using this one.

Anyways, regarding the timestamping in each rx buffer I think I found it.
-----
When the TSAUXC.Disable systime bit is cleared and the SRRCTL[n].Timestamp
bit is set to 1, packets received to the queue will be time stamped if they
meet one of the following conditions:
— Meet the criteria defined in the TSYNCRXCTL.Type field (See Section
8.16.1 and Section 8.16.26).
— Match the value defined in one of the ETQF registers with the 1588 time
stamp bit set (See Section 7.1.2.4) if TSYNCRXCTL.Type field defines time
stamping of L2 packets.

— Match a 2-tuple filter with the TTQF.1588 time stamp set (See Section
7.1.2.5) if TSYNCRXCTL.Type field defines time stamping of L4 packets.

When detecting a receive packet that should be time stamped, the I350 will:
• Place a 64 bit timestamp, indicating the time a packet was received by
the MAC, at the beginning of the receive buffer before the received packet.
• Set the TSIP bit in the RDESC.STATUS field of the last receive
descriptor.
• Update the RDESC.Packet Type field in the last receive descriptor. Value
in this field enables identifying that this is a PTP (Precision Time
Protocol) packet (this indication is only relevant for L2 packets).
• Update the RDESC.HDR_LEN and RDESC.PKT_LEN values to include size of
timestamp.
-----
In my driver I found this:

if (hw->mac.type == e1000_82580)
srrctl |= E1000_SRRCTL_TIMESTAMP;

So again, its only being done if its 82580. I will try to set this bit
tomorrow for the I350 before trying any driver upgrades which can be quite
painful.

Thank again!
Post by Alexander Duyck
http://sourceforge.net/projects/e1000/files/igb%20stable/
It should be able to build on a 3.0.8 kernel provided nothing has been
back-ported that would change the APIs.
- Alex
Post by Mattias Barthel
Thanks a lot Alex, there might still be some hope then.
Unfortunately Im bound to using the linux kernel 3.0.8. What out-of-tree
driver can I use with that, please?
I don't know if it is available in that driver or not, but there
is an option in later igb drivers to support a time stamp being
written with the packet data. Based on the fact that you are
already seeing issues, you might want to try getting the
out-of-tree driver from e1000.sf.net <http://e1000.sf.net> and
seeing if you can use that with the time stamp in packet option
that is enabled when you select to timestamp all incoming Rx packets.
- Alex
Yes, setting the cycles.shift to the same as on the 82580 and
also shifting the reads accordingly everything works. The sync
of shhwtstamps->syststamp too.
My driver version is 3.0.6-k2 and linux is 3.0.8.
Although this might not be very suited to my needs.
The incoming buffers wont be timestamped if you cant keep up
with reading the timestamping registers as I understand it.
This might be perfectly fine in the case of PTP where the
packet rate is fairly low. In my case Im looking at some
80kpps that all would need to be timestamped.
*//* If this bit is set, then the RX registers contain the
time stamp. No/**/ other packet will be time stamped until we
read these registers, so/* */read the registers to make them
available again. Because only one/**/ packet can be time
stamped at a time, we know that the register/* */values must
belong to this one here and therefore we don't need to/**/
compare any of the additional attributes stored for it./* */If
nothing went wrong, then it should have a shared tx_flags that
we/**/ can turn into a skb_shared_hwtstamps. *//*
*//*
*//*
On Tuesday, October 13, 2015, Alexander Duyck
I'm assuming you are using either an older kernel or an
out-of-tree driver do I have that right? I ask because this code
doesn't resemble what is is currently in the latest Linux kernel.
Comments on what you have found are inline below.
- Alex
Hi, I think I managed to get around this problem.
----
/*
* The 82580 timesync updates the system timer every 8ns by 8ns
* and the value cannot be shifted. Instead we need to shift
* the registers to generate a 64bit timer value. As a result
* SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by
* 24 in order to generate a larger value for
synchronization.
*/
adapter->cycles.shift = IGB_82580_TSYNC_SHIFT;
----
I have the I350 and I dont think this cycles shift was meant
for my adapter.
IGB_82580_TSYNC_SHIFT is 24.
Also when doing igb_read_clock,
this shifting is applied only for the 82580 and I guess it
--------
/*
* The timestamp latches on lowest register read. For the 82580
* the lowest register is SYSTIMR instead of SYSTIML.
However
we never
* adjusted TIMINCA so SYSTIMR will just read as all 0s so
ignore it.
*/
if (hw->mac.type == e1000_82580) {
stamp = rd32(E1000_SYSTIMR) >> 8;
shift = IGB_82580_TSYNC_SHIFT;
}
stamp |= (u64)rd32(E1000_SYSTIML) << shift;
stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32);
-------------
This is a bug in the code. The i350 should be shifted by the same
value as the 82580. Then the result would be the same as if you
had set the shift to 0. As I recall the reasoning behind the
change was to make use of as many bits of the time as possible,
and on 82580 and newer the SYSTIMH portion of the clock only
contained 8 bits of actual clock data.
Without that change I believe there is a risk of the counter being
considered to have cycled too fast and causing errors as the cycle
counter will roll over at 40 bits instead of 64. The alternative
would be to update the cycle counter you are using so that it is
aware that it is limited to 40 bits and then use an shift of 0.
Setting adapter->cycles.shift to 0 makes the clock run at
about the same speed as the system clock.
Next problem I have is that the syststamp is not syncing
correctly against the system clock.
Any help is much appreciated.
Regards,
Mattias
On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel
Hi again,
Started to try and timestamp all packets in the I350.
driver: igb
version: 3.0.6-k2
firmware-version: 0.147-0
Doing an ioctl that turns on timestamping for all packets.
hwconfig.tx_type = HWTSTAMP_TX_OFF;
hwconfig.rx_filter = HWTSTAMP_FILTER_ALL;
hwconfig_requested = hwconfig;
ioctl(sock, SIOCSHWTSTAMP, &hwtstamp)
The strangest thing is that although it works to read the
timestamps the clock seems to advance very slowly
and/or
incorrectly.
I just printout the HW timestamps alongside the
systemclock and
the hw stamping is advancing the nanoseconds when the
systemclock
is advancing us and seconds.
Is there a know bug in this version of the driver that maybe
initializes the clock incorrectly or maybe reads
incorrectly from
the clock?
[ 63.004116] igb_rx_hwtstamp
1444656871372845807
[ 63.005721] system timestamp: tv_sec: 1444656793,
tv_usec: 683825
1444656871372845807,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825302,
1444656871, tv_usec: 372825
[ 63.009395]
[ 63.104111] igb_rx_hwtstamp
1444656871372845813
[ 63.105747] system timestamp: tv_sec: 1444656793,
tv_usec: 783851
1444656871372845813,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825308,
1444656871, tv_usec: 372825
[ 63.109428]
[ 63.204109] igb_rx_hwtstamp
1444656871372845819
[ 63.206018] system timestamp: tv_sec: 1444656793,
tv_usec: 884122
1444656871372845819,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825314,
1444656871, tv_usec: 372825
[ 63.210897]
[ 63.304068] igb_rx_hwtstamp
1444656871372845825
[ 63.306176] system timestamp: tv_sec: 1444656793,
tv_usec: 984280
1444656871372845825,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825320,
1444656871, tv_usec: 372825
[ 63.311032]
[ 63.404107] igb_rx_hwtstamp
1444656871372845831
[ 63.405695] system timestamp: tv_sec: 1444656794,
tv_usec: 83798
1444656871372845831,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825326,
1444656871, tv_usec: 372825
[ 63.409405]
[ 63.504084] igb_rx_hwtstamp
1444656871372845837
[ 63.505715] system timestamp: tv_sec: 1444656794,
tv_usec: 183819
1444656871372845837,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825332,
1444656871, tv_usec: 372825
Any help will be very much appreciated.
Best regards,
Mattias
On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck
The i350 should support timestamping all packets as I
recall. You should probably look into getting the
linuxptp package and
seeing if you can make use of the hwstamp_ctl command to
enable timestamping for all incoming packets
on the
adapter.
- Alex
Hi
thanks for answer.
Unfortunately, the udp/ip stream just as
the cfm
stream is
against only one peer.
Furthemore on the guest side I have only
one vcpu
tied to
two physical siblings to minimize L2 cache
misses
on ISRs
due to cache coherence.
I had an idea this morning, maybe I can
make use
of the
ptp hw timestaming for this? This should give
better accuracy.
Is this possible or does it only work for
ethertype ptp?
On Thursday, October 8, 2015, Alexander Duyck
The difference could be Receive Side Scaling
(RSS). The UDP/IP
packets
can be load balanced across all queues
via RSS
based
on a hash of the
source and destination addresses. So
if you have
multiple sources
generating these packets, or they are
being
received
on multiple IP
addresses then it is possible the load is
being spread
out over
multiple
CPUs. Your CFM packets are on an Ethertype
other than
IPv4 or IPv6 so
all packets will end up on queue 0 if
I am not
mistaken.
- Alex
Post by Mattias Barthel
Hi again!
I realise my message might not have
been so
clear.
Post by Mattias Barthel
I will try to clarify. I am doing PM
with
software
timestamping.
Post by Mattias Barthel
version: 3.0.6-k2
firmware-version: 0.147-0
I do timestamping (NTP format) of the
packets in the
same place
for all
Post by Mattias Barthel
protocols. (in igb_clean_rx_irq_adv()).
For these non ip packets like IEEE
801.2ag CFM ETH-DM
(ethertype 0x8902) I
Post by Mattias Barthel
get quite a lot worse accuracy in the
timestamping.
Post by Mattias Barthel
For UDP/IP TWAMP I get for 10k pps a max
delay of
around 140us
back to back.
Post by Mattias Barthel
For CFM ETH DM I get for the same packet
rate and
packet size a
max delay
Post by Mattias Barthel
of around 900us.
Could FW be putting these L2 packets
in another
queue with different
Post by Mattias Barthel
characteristics?
I tried to add a ET-type filter as
written in my
previous mail
but it
Post by Mattias Barthel
showed no difference.
Thanks for any help given!
On Mon, Oct 5, 2015 at 2:19 PM,
Mattias Barthel
<javascript:;>>
Post by Mattias Barthel
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when
using the igb
driver on i350
on linux
Post by Mattias Barthel
Post by Mattias Barthel
regarding
ETH CFM packets. This compared to TWAMP
(IPv4 UDP
packets).
Post by Mattias Barthel
Post by Mattias Barthel
The environment is KVM with the
i350 devices in
PCI-passthrough.
Post by Mattias Barthel
Post by Mattias Barthel
So i figured I add an own rx-queue
(filter) for
those types of
ethernet
Post by Mattias Barthel
Post by Mattias Barthel
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable
filter */
Post by Mattias Barthel
Post by Mattias Barthel
(1 << 29) | /* enable immediate
interrupt */
Post by Mattias Barthel
Post by Mattias Barthel
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902
CFM eth
protocol type */
Post by Mattias Barthel
Post by Mattias Barthel
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
<javascript:;>
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit
http://communities.intel.com/community/wired
Avis de confidentialité
Les informations contenues dans le présent
message
et dans
toute piÚce qui lui est jointe sont
confidentielles et
peuvent être protégées par le secret
professionnel. Ces
informations sont à l’usage exclusif de
son ou de ses
destinataires. Si vous recevez ce message par erreur,
veuillez s’il vous plait communiquer
immédiatement
avec
l’expéditeur et en détruire tout
exemplaire. De
plus, il
vous est strictement interdit de le divulguer, de le
distribuer ou de le reproduire sans
l’autorisation de
l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain
confidential information which may be
privileged
and which
is intended for the exclusive use of its
addressee(s). If
you receive this message in error, please
inform
sender
immediately and destroy any copy thereof.
Furthermore, any
disclosure, distribution or copying of this
message and/or
any attachment hereto without the consent
of the
sender is
strictly prohibited. Thank you.
-- Mattias Barthel
-- Mattias Barthel
Avis de confidentialité
Les informations contenues dans le présent message et dans
toute piÚce qui lui est jointe sont confidentielles et peuvent
être protégées par le secret professionnel. Ces informations
sont à l’usage exclusif de son ou de ses destinataires. Si
vous recevez ce message par erreur, veuillez s’il vous plait
communiquer immédiatement avec l’expéditeur et en détruire
tout exemplaire. De plus, il vous est strictement interdit de
le divulguer, de le distribuer ou de le reproduire sans
l’autorisation de l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain
confidential information which may be privileged and which is
intended for the exclusive use of its addressee(s). If you
receive this message in error, please inform sender
immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or any
attachment hereto without the consent of the sender is
strictly prohibited. Thank you.
Avis de confidentialité
Les informations contenues dans le présent message et dans toute piÚce
qui lui est jointe sont confidentielles et peuvent être protégées par le
secret professionnel. Ces informations sont à l’usage exclusif de son ou de
ses destinataires. Si vous recevez ce message par erreur, veuillez s’il
vous plait communiquer immédiatement avec l’expéditeur et en détruire tout
exemplaire. De plus, il vous est strictement interdit de le divulguer, de
le distribuer ou de le reproduire sans l’autorisation de l’expéditeur.
Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain confidential
information which may be privileged and which is intended for the exclusive
use of its addressee(s). If you receive this message in error, please
inform sender immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or any attachment
hereto without the consent of the sender is strictly prohibited. Thank you.
--
Mattias Barthel
Mattias Barthel
2015-10-16 19:25:32 UTC
Permalink
It works now!

Thanks Alexander for your help! I really appreciate it and for me it has
been very great value.
Post by Mattias Barthel
Thanks again Alex,
It might be a good idea to upgrade the driver, who knows what more there
is in store for me using this one.
Anyways, regarding the timestamping in each rx buffer I think I found it.
-----
When the TSAUXC.Disable systime bit is cleared and the SRRCTL[n].Timestamp
bit is set to 1, packets received to the queue will be time stamped if
— Meet the criteria defined in the TSYNCRXCTL.Type field (See Section
8.16.1 and Section 8.16.26).
— Match the value defined in one of the ETQF registers with the 1588 time
stamp bit set (See Section 7.1.2.4) if TSYNCRXCTL.Type field defines time
stamping of L2 packets.
— Match a 2-tuple filter with the TTQF.1588 time stamp set (See Section
7.1.2.5) if TSYNCRXCTL.Type field defines time stamping of L4 packets.
• Place a 64 bit timestamp, indicating the time a packet was received by
the MAC, at the beginning of the receive buffer before the received packet.
• Set the TSIP bit in the RDESC.STATUS field of the last receive
descriptor.
• Update the RDESC.Packet Type field in the last receive descriptor. Value
in this field enables identifying that this is a PTP (Precision Time
Protocol) packet (this indication is only relevant for L2 packets).
• Update the RDESC.HDR_LEN and RDESC.PKT_LEN values to include size of
timestamp.
-----
if (hw->mac.type == e1000_82580)
srrctl |= E1000_SRRCTL_TIMESTAMP;
So again, its only being done if its 82580. I will try to set this bit
tomorrow for the I350 before trying any driver upgrades which can be quite
painful.
Thank again!
On Tue, Oct 13, 2015 at 8:45 PM, Alexander Duyck <
Post by Alexander Duyck
http://sourceforge.net/projects/e1000/files/igb%20stable/
It should be able to build on a 3.0.8 kernel provided nothing has been
back-ported that would change the APIs.
- Alex
Post by Mattias Barthel
Thanks a lot Alex, there might still be some hope then.
Unfortunately Im bound to using the linux kernel 3.0.8. What out-of-tree
driver can I use with that, please?
I don't know if it is available in that driver or not, but there
is an option in later igb drivers to support a time stamp being
written with the packet data. Based on the fact that you are
already seeing issues, you might want to try getting the
out-of-tree driver from e1000.sf.net <http://e1000.sf.net> and
seeing if you can use that with the time stamp in packet option
that is enabled when you select to timestamp all incoming Rx packets.
- Alex
Yes, setting the cycles.shift to the same as on the 82580 and
also shifting the reads accordingly everything works. The sync
of shhwtstamps->syststamp too.
My driver version is 3.0.6-k2 and linux is 3.0.8.
Although this might not be very suited to my needs.
The incoming buffers wont be timestamped if you cant keep up
with reading the timestamping registers as I understand it.
This might be perfectly fine in the case of PTP where the
packet rate is fairly low. In my case Im looking at some
80kpps that all would need to be timestamped.
*//* If this bit is set, then the RX registers contain the
time stamp. No/**/ other packet will be time stamped until we
read these registers, so/* */read the registers to make them
available again. Because only one/**/ packet can be time
stamped at a time, we know that the register/* */values must
belong to this one here and therefore we don't need to/**/
compare any of the additional attributes stored for it./* */If
nothing went wrong, then it should have a shared tx_flags that
we/**/ can turn into a skb_shared_hwtstamps. *//*
*//*
*//*
On Tuesday, October 13, 2015, Alexander Duyck
I'm assuming you are using either an older kernel or an
out-of-tree driver do I have that right? I ask because this code
doesn't resemble what is is currently in the latest Linux kernel.
Comments on what you have found are inline below.
- Alex
Hi, I think I managed to get around this problem.
----
/*
* The 82580 timesync updates the system timer every 8ns by 8ns
* and the value cannot be shifted. Instead we need to shift
* the registers to generate a 64bit timer value. As a result
* SYSTIMR/L/H, TXSTMPL/H, RXSTMPL/H all have to be shifted by
* 24 in order to generate a larger value for
synchronization.
*/
adapter->cycles.shift = IGB_82580_TSYNC_SHIFT;
----
I have the I350 and I dont think this cycles shift was meant
for my adapter.
IGB_82580_TSYNC_SHIFT is 24.
Also when doing igb_read_clock,
this shifting is applied only for the 82580 and I guess it
--------
/*
* The timestamp latches on lowest register read. For the 82580
* the lowest register is SYSTIMR instead of SYSTIML.
However
we never
* adjusted TIMINCA so SYSTIMR will just read as all 0s so
ignore it.
*/
if (hw->mac.type == e1000_82580) {
stamp = rd32(E1000_SYSTIMR) >> 8;
shift = IGB_82580_TSYNC_SHIFT;
}
stamp |= (u64)rd32(E1000_SYSTIML) << shift;
stamp |= (u64)rd32(E1000_SYSTIMH) << (shift + 32);
-------------
This is a bug in the code. The i350 should be shifted by the same
value as the 82580. Then the result would be the same as if you
had set the shift to 0. As I recall the reasoning behind the
change was to make use of as many bits of the time as possible,
and on 82580 and newer the SYSTIMH portion of the clock only
contained 8 bits of actual clock data.
Without that change I believe there is a risk of the counter being
considered to have cycled too fast and causing errors as the cycle
counter will roll over at 40 bits instead of 64. The alternative
would be to update the cycle counter you are using so that it is
aware that it is limited to 40 bits and then use an shift of 0.
Setting adapter->cycles.shift to 0 makes the clock run at
about the same speed as the system clock.
Next problem I have is that the syststamp is not syncing
correctly against the system clock.
Any help is much appreciated.
Regards,
Mattias
On Mon, Oct 12, 2015 at 5:04 PM, Mattias Barthel
Hi again,
Started to try and timestamp all packets in the I350.
driver: igb
version: 3.0.6-k2
firmware-version: 0.147-0
Doing an ioctl that turns on timestamping for all packets.
hwconfig.tx_type = HWTSTAMP_TX_OFF;
hwconfig.rx_filter = HWTSTAMP_FILTER_ALL;
hwconfig_requested = hwconfig;
ioctl(sock, SIOCSHWTSTAMP, &hwtstamp)
The strangest thing is that although it works to read the
timestamps the clock seems to advance very slowly
and/or
incorrectly.
I just printout the HW timestamps alongside the
systemclock and
the hw stamping is advancing the nanoseconds when the
systemclock
is advancing us and seconds.
Is there a know bug in this version of the driver
that maybe
initializes the clock incorrectly or maybe reads
incorrectly from
the clock?
[ 63.004116] igb_rx_hwtstamp
1444656871372845807
[ 63.005721] system timestamp: tv_sec: 1444656793,
tv_usec: 683825
1444656871372845807,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825302,
1444656871, tv_usec: 372825
[ 63.009395]
[ 63.104111] igb_rx_hwtstamp
1444656871372845813
[ 63.105747] system timestamp: tv_sec: 1444656793,
tv_usec: 783851
1444656871372845813,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825308,
1444656871, tv_usec: 372825
[ 63.109428]
[ 63.204109] igb_rx_hwtstamp
1444656871372845819
[ 63.206018] system timestamp: tv_sec: 1444656793,
tv_usec: 884122
1444656871372845819,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825314,
1444656871, tv_usec: 372825
[ 63.210897]
[ 63.304068] igb_rx_hwtstamp
1444656871372845825
[ 63.306176] system timestamp: tv_sec: 1444656793,
tv_usec: 984280
1444656871372845825,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825320,
1444656871, tv_usec: 372825
[ 63.311032]
[ 63.404107] igb_rx_hwtstamp
1444656871372845831
[ 63.405695] system timestamp: tv_sec: 1444656794,
tv_usec: 83798
1444656871372845831,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825326,
1444656871, tv_usec: 372825
[ 63.409405]
[ 63.504084] igb_rx_hwtstamp
1444656871372845837
[ 63.505715] system timestamp: tv_sec: 1444656794,
tv_usec: 183819
1444656871372845837,
tv_sec: 1444656871, tv_usec: 372845
1444656871372825332,
1444656871, tv_usec: 372825
Any help will be very much appreciated.
Best regards,
Mattias
On Thu, Oct 8, 2015 at 6:01 PM, Alexander Duyck
The i350 should support timestamping all packets as I
recall. You should probably look into getting the
linuxptp package and
seeing if you can make use of the hwstamp_ctl
command to
enable timestamping for all incoming packets
on the
adapter.
- Alex
Hi
thanks for answer.
Unfortunately, the udp/ip stream just as
the cfm
stream is
against only one peer.
Furthemore on the guest side I have only
one vcpu
tied to
two physical siblings to minimize L2 cache
misses
on ISRs
due to cache coherence.
I had an idea this morning, maybe I can
make use
of the
ptp hw timestaming for this? This should give
better accuracy.
Is this possible or does it only work for
ethertype ptp?
On Thursday, October 8, 2015, Alexander Duyck
The difference could be Receive Side Scaling
(RSS). The UDP/IP
packets
can be load balanced across all queues
via RSS
based
on a hash of the
source and destination addresses. So
if you have
multiple sources
generating these packets, or they are
being
received
on multiple IP
addresses then it is possible the load is
being spread
out over
multiple
CPUs. Your CFM packets are on an Ethertype
other than
IPv4 or IPv6 so
all packets will end up on queue 0 if
I am not
mistaken.
- Alex
Post by Mattias Barthel
Hi again!
I realise my message might not have
been so
clear.
Post by Mattias Barthel
I will try to clarify. I am doing PM
with
software
timestamping.
Post by Mattias Barthel
version: 3.0.6-k2
firmware-version: 0.147-0
I do timestamping (NTP format) of the
packets in the
same place
for all
Post by Mattias Barthel
protocols. (in igb_clean_rx_irq_adv()).
For these non ip packets like IEEE
801.2ag CFM ETH-DM
(ethertype 0x8902) I
Post by Mattias Barthel
get quite a lot worse accuracy in the
timestamping.
Post by Mattias Barthel
For UDP/IP TWAMP I get for 10k pps a
max
delay of
around 140us
back to back.
Post by Mattias Barthel
For CFM ETH DM I get for the same
packet
rate and
packet size a
max delay
Post by Mattias Barthel
of around 900us.
Could FW be putting these L2 packets
in another
queue with different
Post by Mattias Barthel
characteristics?
I tried to add a ET-type filter as
written in my
previous mail
but it
Post by Mattias Barthel
showed no difference.
Thanks for any help given!
On Mon, Oct 5, 2015 at 2:19 PM,
Mattias Barthel
Post by Mattias Barthel
Post by Mattias Barthel
Greetings,
Im getting quite bad latency when
using the igb
driver on i350
on linux
Post by Mattias Barthel
Post by Mattias Barthel
regarding
ETH CFM packets. This compared to
TWAMP
(IPv4 UDP
packets).
Post by Mattias Barthel
Post by Mattias Barthel
The environment is KVM with the
i350 devices in
PCI-passthrough.
Post by Mattias Barthel
Post by Mattias Barthel
So i figured I add an own rx-queue
(filter) for
those types of
ethernet
Post by Mattias Barthel
Post by Mattias Barthel
protocol packets.
wr32(E1000_ETQF(4),
(1 << 31) | /* queue enable */
(E1000_ETQF_FILTER_ENABLE) | /* enable
filter */
Post by Mattias Barthel
Post by Mattias Barthel
(1 << 29) | /* enable immediate
interrupt */
Post by Mattias Barthel
Post by Mattias Barthel
(0x4 << 16) | /* queue no. 4 */
(ETH_P_8021AG)); /* 0x8902
CFM eth
protocol type */
Post by Mattias Barthel
Post by Mattias Barthel
ETH_P_8021A is 0x8902
Is this a correct/good approach?
Thanks in advance!
--
Mattias Barthel
------------------------------------------------------------------------------
_______________________________________________
E1000-devel mailing list
<javascript:;>
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit
http://communities.intel.com/community/wired
Avis de confidentialité
Les informations contenues dans le présent
message
et dans
toute piÚce qui lui est jointe sont
confidentielles et
peuvent être protégées par le secret
professionnel. Ces
informations sont à l’usage exclusif de
son ou de ses
destinataires. Si vous recevez ce message
par erreur,
veuillez s’il vous plait communiquer
immédiatement
avec
l’expéditeur et en détruire tout
exemplaire. De
plus, il
vous est strictement interdit de le
divulguer, de le
distribuer ou de le reproduire sans
l’autorisation de
l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment
hereto contain
confidential information which may be
privileged
and which
is intended for the exclusive use of its
addressee(s). If
you receive this message in error, please
inform
sender
immediately and destroy any copy thereof.
Furthermore, any
disclosure, distribution or copying of this
message and/or
any attachment hereto without the consent
of the
sender is
strictly prohibited. Thank you.
-- Mattias Barthel
-- Mattias Barthel
Avis de confidentialité
Les informations contenues dans le présent message et dans
toute piÚce qui lui est jointe sont confidentielles et peuvent
être protégées par le secret professionnel. Ces informations
sont à l’usage exclusif de son ou de ses destinataires. Si
vous recevez ce message par erreur, veuillez s’il vous plait
communiquer immédiatement avec l’expéditeur et en détruire
tout exemplaire. De plus, il vous est strictement interdit de
le divulguer, de le distribuer ou de le reproduire sans
l’autorisation de l’expéditeur. Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain
confidential information which may be privileged and which is
intended for the exclusive use of its addressee(s). If you
receive this message in error, please inform sender
immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or any
attachment hereto without the consent of the sender is
strictly prohibited. Thank you.
Avis de confidentialité
Les informations contenues dans le présent message et dans toute piÚce
qui lui est jointe sont confidentielles et peuvent être protégées par le
secret professionnel. Ces informations sont à l’usage exclusif de son ou de
ses destinataires. Si vous recevez ce message par erreur, veuillez s’il
vous plait communiquer immédiatement avec l’expéditeur et en détruire tout
exemplaire. De plus, il vous est strictement interdit de le divulguer, de
le distribuer ou de le reproduire sans l’autorisation de l’expéditeur.
Merci.
Confidentiality notice
This e-mail message and any attachment hereto contain confidential
information which may be privileged and which is intended for the exclusive
use of its addressee(s). If you receive this message in error, please
inform sender immediately and destroy any copy thereof. Furthermore, any
disclosure, distribution or copying of this message and/or any attachment
hereto without the consent of the sender is strictly prohibited. Thank you.
--
Mattias Barthel
--
Mattias Barthel
Loading...