[E1000-devel] Intel 82574L rx_short_length_errors

[Aleksey Chudov]

Hello,



I have few identical low end servers with the following integrated NICs:



02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection

03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection



On all servers is constantly increasing counters rx_errors, rx_length_errors
and rx_short_length_errors.


I tried the following settings 82574L:

* Increase Rx and Tx buffers from 256 to 4096 bytes

* Manual distribution of interrupts among the processors (IRQ Affinity)

* Disable Active State Power Management (ASPM) in the BIOS and Kernel

* Update kernel from 2.6.34 to 2.6.39

* Different e1000e drivers 1.0.2-k2, 1.3.10-k2, 1.6.3-NAPI, 1.9.5-NAPI

* Driver settings InterruptThrottleRate = 1,1

* Legacy Interrupt Mode IntMode = 1,1

* Disable LRO, GRO, GSO, Autonegotiate with ethtool

* Increase memory limits of the Linux networking stack


It seems that the number of errors does not depend on configuration or
driver version or the amount of traffic.



Then I tied to insert in one server additional NIC 82576 connected to the
same switch through the same patch cable and errors completely disappeared.





Does anyone have any idea why there are errors with 82574L NIC ?









# ethtool -S eth0

NIC statistics:

rx_packets: 441142143

tx_packets: 640189607

rx_bytes: 51863921636

tx_bytes: 754569587969

rx_broadcast: 158

tx_broadcast: 1

rx_multicast: 0

tx_multicast: 0

rx_errors: 13

tx_errors: 0

tx_dropped: 0

multicast: 0

collisions: 0

rx_length_errors: 13

rx_over_errors: 0

rx_crc_errors: 0

rx_frame_errors: 0

rx_no_buffer_count: 0

rx_missed_errors: 0

tx_aborted_errors: 0

tx_carrier_errors: 0

tx_fifo_errors: 0

tx_heartbeat_errors: 0

tx_window_errors: 0

tx_abort_late_coll: 0

tx_deferred_ok: 0

tx_single_coll_ok: 0

tx_multi_coll_ok: 0

tx_timeout_count: 0

tx_restart_queue: 0

rx_long_length_errors: 0

rx_short_length_errors: 13

rx_align_errors: 0

tx_tcp_seg_good: 106335289

tx_tcp_seg_failed: 0

rx_flow_control_xon: 0

rx_flow_control_xoff: 0

tx_flow_control_xon: 0

tx_flow_control_xoff: 0

rx_long_byte_count: 51863921636

rx_csum_offload_good: 441123474

rx_csum_offload_errors: 4958

rx_header_split: 0

alloc_rx_buff_failed: 0

tx_smbus: 0

rx_smbus: 0

dropped_smbus: 0

rx_dma_failed: 0

tx_dma_failed: 0





02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection

Subsystem: ASUSTeK Computer Inc. Unknown device 8369

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-

Latency: 0, Cache Line Size: 64 bytes

Interrupt: pin A routed to IRQ 17

Region 0: Memory at fbde0000 (32-bit, non-prefetchable) [size=128K]

Region 2: I/O ports at bc00 [size=32]

Region 3: Memory at fbddc000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [c8] Power Management version 2

Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)

Status: D0 PME-Enable- DSel=0 DScale=1 PME-

Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-

Address: 0000000000000000 Data: 0000

Capabilities: [e0] Express Endpoint IRQ 0

Device: Supported: MaxPayload 256 bytes, PhantFunc 0,
ExtTag-

Device: Latency L0s <512ns, L1 <64us

Device: AtnBtn- AtnInd- PwrInd-

Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-

Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+

Device: MaxPayload 128 bytes, MaxReadReq 512 bytes

Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s L1, Port 0

Link: Latency L0s <128ns, L1 <64us

Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-

Link: Speed 2.5Gb/s, Width x1

Capabilities: [a0] MSI-X: Enable+ Mask- TabSize=5

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00002000



03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection

Subsystem: ASUSTeK Computer Inc. Unknown device 8369

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-

Latency: 0, Cache Line Size: 64 bytes

Interrupt: pin A routed to IRQ 16

Region 0: Memory at fbee0000 (32-bit, non-prefetchable) [size=128K]

Region 2: I/O ports at cc00 [size=32]

Region 3: Memory at fbedc000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [c8] Power Management version 2

Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)

Status: D0 PME-Enable- DSel=0 DScale=1 PME-

Capabilities: [d0] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-

Address: 0000000000000000 Data: 0000

Capabilities: [e0] Express Endpoint IRQ 0

Device: Supported: MaxPayload 256 bytes, PhantFunc 0,
ExtTag-

Device: Latency L0s <512ns, L1 <64us

Device: AtnBtn- AtnInd- PwrInd-

Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-

Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+

Device: MaxPayload 128 bytes, MaxReadReq 512 bytes

Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s L1, Port 0

Link: Latency L0s <128ns, L1 <64us

Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-

Link: Speed 2.5Gb/s, Width x1

Capabilities: [a0] MSI-X: Enable+ Mask- TabSize=5

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00002000



08:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network
Connection (rev 01)

Subsystem: Intel Corporation Gigabit ET2 Quad Port Server Adapter

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-

Latency: 0, Cache Line Size: 256 bytes

Interrupt: pin A routed to IRQ 32

Region 0: Memory at f9fe0000 (32-bit, non-prefetchable) [size=128K]

Region 1: Memory at fa000000 (32-bit, non-prefetchable) [size=4M]

Region 2: I/O ports at d880 [size=32]

Region 3: Memory at f9fdc000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [40] Power Management version 3

Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold-)

Status: D0 PME-Enable- DSel=0 DScale=1 PME-

Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
Queue=0/0 Enable-

Address: 0000000000000000 Data: 0000

Masking: 00000000 Pending: 00000000

Capabilities: [70] MSI-X: Enable+ Mask- TabSize=10

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00002000

Capabilities: [a0] Express Endpoint IRQ 0

Device: Supported: MaxPayload 512 bytes, PhantFunc 0,
ExtTag-

Device: Latency L0s <512ns, L1 <64us

Device: AtnBtn- AtnInd- PwrInd-

Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-

Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+

Device: MaxPayload 256 bytes, MaxReadReq 512 bytes

Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s L1, Port 4

Link: Latency L0s <4us, L1 <64us

Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-

Link: Speed 2.5Gb/s, Width x4



08:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network
Connection (rev 01)

Subsystem: Intel Corporation Gigabit ET2 Quad Port Server Adapter

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-

Latency: 0, Cache Line Size: 256 bytes

Interrupt: pin B routed to IRQ 42

Region 0: Memory at fa7e0000 (32-bit, non-prefetchable) [size=128K]

Region 1: Memory at fa800000 (32-bit, non-prefetchable) [size=4M]

Region 2: I/O ports at dc00 [size=32]

Region 3: Memory at fa7dc000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [40] Power Management version 3

Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold-)

Status: D0 PME-Enable- DSel=0 DScale=1 PME-

Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
Queue=0/0 Enable-

Address: 0000000000000000 Data: 0000

Masking: 00000000 Pending: 00000000

Capabilities: [70] MSI-X: Enable+ Mask- TabSize=10

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00002000

Capabilities: [a0] Express Endpoint IRQ 0

Device: Supported: MaxPayload 512 bytes, PhantFunc 0,
ExtTag-

Device: Latency L0s <512ns, L1 <64us

Device: AtnBtn- AtnInd- PwrInd-

Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-

Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+

Device: MaxPayload 256 bytes, MaxReadReq 512 bytes

Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s L1, Port 4

Link: Latency L0s <4us, L1 <64us

Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-

Link: Speed 2.5Gb/s, Width x4



09:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network
Connection (rev 01)

Subsystem: Intel Corporation Gigabit ET2 Quad Port Server Adapter

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-

Latency: 0, Cache Line Size: 256 bytes

Interrupt: pin A routed to IRQ 47

Region 0: Memory at fafe0000 (32-bit, non-prefetchable) [size=128K]

Region 1: Memory at fb000000 (32-bit, non-prefetchable) [size=4M]

Region 2: I/O ports at e880 [size=32]

Region 3: Memory at fafdc000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [40] Power Management version 3

Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)

Status: D0 PME-Enable- DSel=0 DScale=1 PME-

Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
Queue=0/0 Enable-

Address: 0000000000000000 Data: 0000

Masking: 00000000 Pending: 00000000

Capabilities: [70] MSI-X: Enable+ Mask- TabSize=10

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00002000

Capabilities: [a0] Express Endpoint IRQ 0

Device: Supported: MaxPayload 512 bytes, PhantFunc 0,
ExtTag-

Device: Latency L0s <512ns, L1 <64us

Device: AtnBtn- AtnInd- PwrInd-

Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-

Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+

Device: MaxPayload 256 bytes, MaxReadReq 512 bytes

Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s L1, Port 2

Link: Latency L0s <4us, L1 <64us

Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-

Link: Speed 2.5Gb/s, Width x4



09:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network
Connection (rev 01)

Subsystem: Intel Corporation Gigabit ET2 Quad Port Server Adapter

Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-

Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-

Latency: 0, Cache Line Size: 256 bytes

Interrupt: pin B routed to IRQ 41

Region 0: Memory at fb7e0000 (32-bit, non-prefetchable) [size=128K]

Region 1: Memory at fb800000 (32-bit, non-prefetchable) [size=4M]

Region 2: I/O ports at ec00 [size=32]

Region 3: Memory at fb7dc000 (32-bit, non-prefetchable) [size=16K]

Capabilities: [40] Power Management version 3

Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)

Status: D0 PME-Enable- DSel=0 DScale=1 PME-

Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
Queue=0/0 Enable-

Address: 0000000000000000 Data: 0000

Masking: 00000000 Pending: 00000000

Capabilities: [70] MSI-X: Enable+ Mask- TabSize=10

Vector table: BAR=3 offset=00000000

PBA: BAR=3 offset=00002000

Capabilities: [a0] Express Endpoint IRQ 0

Device: Supported: MaxPayload 512 bytes, PhantFunc 0,
ExtTag-

Device: Latency L0s <512ns, L1 <64us

Device: AtnBtn- AtnInd- PwrInd-

Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-

Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+

Device: MaxPayload 256 bytes, MaxReadReq 512 bytes

Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s L1, Port 2

Link: Latency L0s <4us, L1 <64us

Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-

Link: Speed 2.5Gb/s, Width x4





Best regards,

Aleksey

[Ben Greear]

Post by Aleksey Chudov
Hello,
02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
On all servers is constantly increasing counters rx_errors, rx_length_errors
and rx_short_length_errors.
* Increase Rx and Tx buffers from 256 to 4096 bytes
* Manual distribution of interrupts among the processors (IRQ Affinity)
* Disable Active State Power Management (ASPM) in the BIOS and Kernel
* Update kernel from 2.6.34 to 2.6.39
* Different e1000e drivers 1.0.2-k2, 1.3.10-k2, 1.6.3-NAPI, 1.9.5-NAPI
* Driver settings InterruptThrottleRate = 1,1
* Legacy Interrupt Mode IntMode = 1,1
* Disable LRO, GRO, GSO, Autonegotiate with ethtool
* Increase memory limits of the Linux networking stack
It seems that the number of errors does not depend on configuration or
driver version or the amount of traffic.
Then I tied to insert in one server additional NIC 82576 connected to the
same switch through the same patch cable and errors completely disappeared.

If you still have this system available for testing, please try
completely powering down the system for a minute or two
(a reboot is not sufficient) and see if the problems at least
temporarily go away after powering back up.

Also, try 'ethtool -t' on the interface.

We saw some strange errors that were 'fixed' by these
actions..but we never found the root cause and mostly
we just stopped using those NICs...

Thanks,
Ben

[Aleksey Chudov]

Hello,

Post by Ben Greear
If you still have this system available for testing, please try
completely powering down the system for a minute or two
(a reboot is not sufficient) and see if the problems at least
temporarily go away after powering back up.

Tried to shutdown one server for 5 minutes with no effect. Again at
least 1 error every 10 - 15 min.
As errors appears on all servers with 82574L NIC is seems like the
problem is with HW or driver.

Post by Ben Greear
Also, try 'ethtool -t' on the interface.

# ethtool -t eth1
The test result is PASS
The test extra info:
Register test (offline) 0
Eeprom test (offline) 0
Interrupt test (offline) 0
Loopback test (offline) 0
Link test (on/offline) 0


Any ideas?

[Wyborny, Carolyn]

-----Original Message-----
Sent: Tuesday, April 03, 2012 7:49 AM
Subject: [E1000-devel] Intel 82574L rx_short_length_errors
Then I tied to insert in one server additional NIC 82576 connected to the
same switch through the same patch cable and errors completely
disappeared.

Do the errors on the 82574 disappear when the 82576 is installed or is it that the errors don't appear on the 82576?

Thanks,

Carolyn

Carolyn Wyborny
Linux Development
LAN Access Division
Intel Corporation

[Aleksey Chudov]

Post by Wyborny, Carolyn

Post by Aleksey Chudov
Then I tied to insert in one server additional NIC 82576 connected to
the same switch through the same patch cable and errors completely
disappeared.

Do the errors on the 82574 disappear when the 82576 is installed or

is it that the errors don't appear on the 82576?

Errors don't appear on the 82576 but at the same server errors on the
82574L errors continue counting.


Regards,
Aleksey

[Wyborny, Carolyn]

-----Original Message-----
Sent: Tuesday, April 03, 2012 11:53 AM
To: Wyborny, Carolyn
Subject: Re: [E1000-devel] Intel 82574L rx_short_length_errors

Post by Wyborny, Carolyn

Post by Aleksey Chudov
Then I tied to insert in one server additional NIC 82576 connected

to

Post by Wyborny, Carolyn

Post by Aleksey Chudov
the same switch through the same patch cable and errors completely
disappeared.

Do the errors on the 82574 disappear when the 82576 is installed or

is it that the errors don't appear on the 82576?
Errors don't appear on the 82576 but at the same server errors on the
82574L errors continue counting.
Regards,
Aleksey

I appreciate the clarification. I think Jesse has a lot more info for you on this.

Thanks,

Carolyn

Carolyn Wyborny
Linux Development
LAN Access Division
Intel Corporation

[Jesse Brandeburg]

On Tue, 3 Apr 2012 17:49:25 +0300

Post by Aleksey Chudov
02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network
Connection
On all servers is constantly increasing counters rx_errors, rx_length_errors
and rx_short_length_errors.
# ethtool -S eth0
rx_packets: 441142143
tx_packets: 640189607
rx_bytes: 51863921636
tx_bytes: 754569587969
rx_broadcast: 158
tx_broadcast: 1
rx_errors: 13
rx_length_errors: 13
rx_short_length_errors: 13
tx_tcp_seg_good: 106335289
rx_long_byte_count: 51863921636
rx_csum_offload_good: 441123474
rx_csum_offload_errors: 4958

...

Post by Aleksey Chudov
It seems that the number of errors does not depend on configuration or
driver version or the amount of traffic.

All the troubleshooting items listed above are typically for
different issues than you are having.

Post by Aleksey Chudov
Then I tied to insert in one server additional NIC 82576 connected to the
same switch through the same patch cable and errors completely disappeared.

That indicates you might be having some issue at the physical
layer (the PHY) on the part. What kind of switch, and what does
'ethtool eth0' report for the negotiated settings?

Post by Aleksey Chudov
Does anyone have any idea why there are errors with 82574L NIC ?

your error rate is extremely small, and in my experience most
internet connected machines have some amount of bogus input.
That said you didn't have issues with the 82576, which has a
completely diffferent PHY.

the rx_short_length errors are extremely unusual and are another
strong indication of a physical layer problem (bad cable,
something else gone wrong with LAN controller PHY, possibly
negotiation of tx or rx power having issues)

is the cable run very long?

<snip>

please give us the output of ethtool -e from one of your 82574L having
issues.

[Aleksey Chudov]

Hello,

Post by Jesse Brandeburg

Post by Aleksey Chudov
Then I tied to insert in one server additional NIC 82576 connected to the
same switch through the same patch cable and errors completely disappeared.

That indicates you might be having some issue at the physical
layer (the PHY) on the part. What kind of switch, and what does
'ethtool eth0' report for the negotiated settings?
the rx_short_length errors are extremely unusual and are another
strong indication of a physical layer problem (bad cable,
something else gone wrong with LAN controller PHY, possibly
negotiation of tx or rx power having issues)
is the cable run very long?

In fact, I have more than 20 servers connected to different switches
(Cisco 2960 and Cisco 3750) and different length of patch cables (3m - 10m).
There is small amount rx_errors is on all 82574L and no errors on 82576
at the same servers.


# ethtool eth1
Settings for eth1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
Auto-negotiation: on
MDI-X: on
Supports Wake-on: pumbag
Wake-on: g
Current message level: 0x00000001 (1)
drv
Link detected: yes

# ethtool -a eth1
Pause parameters for eth1:
Autonegotiate: on
RX: off
TX: off

Post by Jesse Brandeburg

Post by Aleksey Chudov
Does anyone have any idea why there are errors with 82574L NIC ?

your error rate is extremely small, and in my experience most
internet connected machines have some amount of bogus input.
That said you didn't have issues with the 82576, which has a
completely diffferent PHY.

This is the first batch of servers that have 82574L. Earlier we used
only the 82576 and the error always means that there is a problem.

Post by Jesse Brandeburg
please give us the output of ethtool -e from one of your 82574L having
issues.

# ethtool -e eth0
Offset Values
------ ------
0x0000 f4 6d 04 2b b0 af 20 04 46 f7 80 10 ff ff ff ff
0x0010 ff ff ff ff 6b 02 69 83 43 10 d3 10 ff ff 58 a5
0x0020 00 00 01 20 74 7e ff ff 00 00 c8 00 00 00 04 27
0x0030 c9 6c 50 31 3e 07 0b 46 84 2d 40 01 00 f0 06 07
0x0040 00 60 80 00 04 0f ff 7f 01 4d ec 92 5c fc 83 f0
0x0050 28 00 33 02 50 00 1f 7d 61 19 53 04 a0 10 ff ff
0x0060 00 01 00 40 35 13 13 40 ff ff ff ff ff ff ff ff
0x0070 ff ff ff ff ff ff ff ff ff ff 30 01 ff ff a9 42
0x0080 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0090 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x00a0 00 00 1f 38 0a 00 04 00 80 02 00 20 ff ff ff 7f
0x00b0 00 00 1f 38 08 00 04 00 00 00 1f 38 09 00 04 00
0x00c0 00 00 3f 0a 14 00 20 00 01 7e 5f 10 3f 00 20 00
0x00d0 00 7c 9f 61 00 08 00 00 00 7c 5f 62 00 00 30 00
0x00e0 01 04 e6 08 ca fa e3 7f 02 07 00 20 02 04 06 09
0x00f0 03 04 26 09 01 7a e4 7f 81 03 00 20 0e 7a e4 7f
0x0100 02 00 1f 38 c4 04 04 00 0e 08 26 08 04 02 89 10
0x0110 10 04 26 08 06 02 49 10 01 12 49 10 00 00 1f 38
0x0120 30 03 04 00 00 00 1f 38 07 00 04 00 ff ff ff ff
0x0130 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0140 00 00 1f 38 0a 00 04 00 00 0a 00 20 ef ef ef ef
0x0150 00 00 1f 38 0b 00 04 00 00 00 1f 38 03 00 04 00
0x0160 00 00 1f 38 05 00 04 00 00 00 1f 38 02 00 04 00
0x0170 00 07 00 20 ef ef ef ef 00 04 00 00 04 04 00 00
0x0180 08 08 00 08 04 00 08 00 04 00 04 00 ff 00 00 00
0x0190 ff 00 00 04 00 00 00 00 00 00 00 00 00 00 df 09
0x01a0 98 11 00 00 00 00 1f 38 01 00 04 00 00 fd e4 67
0x01b0 00 01 00 00 82 12 00 20 00 fd e5 7f 81 50 00 00
0x01c0 82 0a 00 20 06 7a e2 7f 82 09 00 20 08 fe 04 40
0x01d0 00 fe 3f 60 02 02 80 11 02 02 80 11 02 02 80 11
0x01e0 02 7c 80 11 57 01 00 00 02 7c 40 11 06 00 00 42
0x01f0 78 fe 6a 50 04 80 21 08 02 fe 5f 60 40 02 1f 38
0x0200 e0 04 04 00 00 80 21 08 04 fe 5f 60 40 02 1f 38
0x0210 e0 04 04 00 0f fe ff 61 17 7a e6 7f 02 02 00 20
0x0220 00 17 e5 47 02 01 00 20 24 fc 84 10 00 00 00 00
0x0230 00 00 df 09 98 11 00 00 00 17 e5 47 01 fc c6 79
0x0240 00 00 00 00 00 1c 1f 10 98 11 00 00 00 00 1f 38
0x0250 0c 00 04 00 ff ff ff ff ff ff ff ff ff ff ff ff
0x0260 69 53 84 03 01 00 00 00 00 00 00 00 00 00 00 00
0x0270 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0290 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x02f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0300 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0310 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0320 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0330 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0340 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0350 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0360 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0370 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0380 00 00 00 00 00 00 00 00 03 3d 00 00 00 00 00 00
0x0390 00 00 00 00 00 00 00 00 00 00 00 00 bc 0c 00 00
0x03a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x03b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x03c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x03d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x03e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x03f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0400 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0420 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0430 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0440 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0450 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0460 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0470 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0480 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0490 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0500 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0510 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0520 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0530 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0540 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0550 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0560 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0570 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0580 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0590 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05e0 00 00 00 00 ff ff ff ff ff ff ff ff ff ff ff ff
0x05f0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0600 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0610 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0620 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0630 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0640 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0650 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0660 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0670 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0680 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0690 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x06a0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x06b0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x06c0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x06d0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x06e0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x06f0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0700 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0710 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0720 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0730 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0740 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0750 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0760 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0770 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0780 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0790 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07a0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07b0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07c0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07d0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07e0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07f0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0800 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0810 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0820 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0830 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0840 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0850 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0860 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0870 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0880 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0890 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08a0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08b0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08c0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08d0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08e0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08f0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0900 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0910 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0920 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0930 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0940 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0950 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0960 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0970 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0980 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0990 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09a0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09b0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09c0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09d0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09e0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09f0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a10 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a20 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a30 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a40 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a50 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a60 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a70 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a80 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a90 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0aa0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ab0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ac0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ad0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ae0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0af0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b10 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b20 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b30 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b40 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b50 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b60 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b70 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b80 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b90 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ba0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0bb0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0bc0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0bd0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0be0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0bf0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c10 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c20 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c30 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c40 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c50 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c60 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c70 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c80 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c90 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ca0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0cb0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0cc0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0cd0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ce0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0cf0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d10 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d20 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d30 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d40 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d50 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d60 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d70 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d80 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d90 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0da0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0db0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0dc0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0dd0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0de0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0df0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e10 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e20 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e30 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e40 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e50 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e60 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e70 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e80 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e90 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ea0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0eb0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ec0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ed0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ee0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ef0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f00 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f10 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f20 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f30 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f40 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f50 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f60 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f70 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f80 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f90 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fa0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fb0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fc0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fd0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fe0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ff0 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff