안녕하세요 쿤드입니다. 🍀

netstat --statistics 명령부터 시작해보겠습니다. (nstat 명령이 훨씬 간결하긴하지만..)

처음에는 각각의 인자들이 무엇을 의미하는지 확인하려고 했으나
이 페이지를 작성하다보니 하나씩 찾다보니 의미만 아는것은 별 필요가 없는것 같았습니다.

 

장애 당시를 기준으로 확인하는것이 좋을것 같아서 그때 확인한 factor들에 대해서

아래에 따로 정리해두었습니다.

작년에 제가 다니던 회사에서 중간에서 gateway역할을 해주는 application의 오류로 장애가 3시간정도 발생한적이 있었는데

그 당시 tcp 통계 변화를 바탕으로

내부적인 application 장애가 발생해서 외부에서 유입되는 traffic 처리를 못한 경우 어떤 차이가 발생하는지를 확인 해보도록 하겠습니다.

 

아래에 있는 내용들은 netstat으로 확인한 통계 값인데 

무엇을 의미하는건지 모르면 여기에 각각 설명이 잘 나와있습니다.

혹은 Linux Document

 

Ip:
    15829871449 total packets received (InReceives)
    0 forwarded (ForwDatagrams)
    0 incoming packets discarded (InDiscards)
    15829871405 incoming packets delivered (InDelivers)
    14182698760 requests sent out (OutRequests)
    31 outgoing packets dropped
    16 dropped because of missing route (OutNoRoutes)
    143 fragments failed
Icmp:
    12710040 ICMP messages received (InMsgs)
    167 input ICMP message failed. (InErrors)
    ICMP input histogram: 
        destination unreachable: 12263649 (InDestUnreachs)
        timeout in transit: 420248 (InTimeExcds)
        source quenches: 6 (InSrcQuenchs)
        redirects: 92 (InRedirects)
        echo requests: 26045 (InEchos)
    28937 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 2892 (OutDestUnreachs)
        echo replies: 26045 (OutEchoReps)
IcmpMsg:
        InType3: 12263649
        InType4: 6
        InType5: 92
        InType8: 26045
        InType11: 420248
        OutType0: 26045
        OutType3: 2892
Tcp:
    25768376 active connections openings (TcpActiveOpens)
    647707490 passive connection openings (TcpPassiveOpens)   
    4647341 failed connection attempts (AttemptFails)
    31117340 connection resets received (EstabResets)
    3537 connections established (CurrEstab) https://tools.ietf.org/html/rfc793   
    15817063195 segments received (InSegs)
    16491280655 segments send out (OutSegs)
    986318177 segments retransmited (RetransSegs)
    7440 bad segments received. (InErrs)
    91126171 resets sent (OutRsts)
Udp:
    95601 packets received (InDatagrams)
    2569 packets to unknown port received. (NoPorts)
    0 packet receive errors (InErrors)
    111544 packets sent (OutDatagrams)
    0 receive buffer errors (RcvbufErrors)
    0 send buffer errors (SndbufErrors)
UdpLite:
TcpExt:
    42045 SYN cookies sent (SyncookiesSent)
    12143 SYN cookies received (SyncookiesRecv)
    60209039 invalid SYN cookies received (SyncookiesFailed)
    4647134 resets received for embryonic SYN_RECV sockets (EmbryonicRsts)
    3821 packets pruned from receive queue because of socket buffer overrun (PruneCalled)
    63491 ICMP packets dropped because they were out-of-window (OutOfWindowIcmps)
    22286 ICMP packets dropped because socket was locked (LockDroppedIcmps)
    107834482 TCP sockets finished time wait in fast timer (TW)
    46352300 packets rejects in established connections because of timestamp (PAWSEstab)
    607439027 delayed acks sent (DelayedACKs)
    632566 delayed acks further delayed because of locked socket (DelayedACKLocked)
    Quick ack mode was activated 199302465 times (DelayedACKLost)
    208055 SYNs to LISTEN sockets dropped (ListenDrops)
    418093 packets directly queued to recvmsg prequeue. (TCPPrequeued)
    122606 bytes directly in process context from backlog (TCPDirectCopyFromBacklog)
    79146389 bytes directly received in process context from prequeue (TCPDirectCopyFromPrequeue)
    1574327100 packet headers predicted (TCPHPHits)
    402495 packets header predicted and directly queued to user (TCPHPHitsToUser)
    7766206385 acknowledgments not containing data payload received (TCPPureAcks)
    559462251 predicted acknowledgments
    24017 times recovered from packet loss due to fast retransmit
    72915270 times recovered from packet loss by selective acknowledgements
    586 bad SACK blocks received
    Detected reordering 889310 times using FACK
    Detected reordering 465211 times using SACK
    Detected reordering 697 times using reno fast retransmit
    Detected reordering 13789060 times using time stamp
    6344097 congestion windows fully recovered without slow start
    10638781 congestion windows partially recovered using Hoe heuristic
    1716234 congestion windows recovered without slow start by DSACK
    43272810 congestion windows recovered without slow start after partial ack
    TCPLostRetransmit: 1784385
    13185 timeouts after reno fast retransmit
    15322792 timeouts after SACK recovery
    26945066 timeouts in loss state
    104358514 fast retransmits
    30761536 forward retransmits
    72817949 retransmits in slow start
    189678472 other TCP timeouts
    TCPLossProbes: 324536377
    TCPLossProbeRecovery: 76481412
    12218 classic Reno fast retransmits failed
    27270545 SACK retransmits failed
    200116946 DSACKs sent for old packets (TCPDSACKOldSent)
    3581427 DSACKs sent for out of order packets 
    188637563 DSACKs received
    1897085 DSACKs for out of order packets received
    17269734 connections reset due to unexpected data (TCPAbortOnData)
    994645 connections reset due to early user close (TCPAbortOnClose)
    71206457 connections aborted due to timeout (TCPAbortOnTimeout)
    20 times unable to send RST due to no memory
    TCPSACKDiscard: 2614 (Number of SACK blocks are invalid)
    TCPDSACKIgnoredOld: 457719 (Number of DSACK block is invalid, undo_marker not set)
    TCPDSACKIgnoredNoUndo: 94644730 (Number of DSACK block is invalid, undo_marker set)
    TCPSpuriousRTOs: 3443956 (Number of The spurious retransmission timeout detected by the `F-RTO`_ algorithm)
    * skb: The linux networking stack stores data in sk_buff struct
    TCPSackShifted: 17209501 (Number of skb is shifted)
    TCPSackMerged: 57304616 (Number of skb is merged)
    TCPSackShiftFallback: 234570612 (Number of TCP stack doesn't merge it for some reasons.)
    TCPBacklogDrop: 43
    IPReversePathFilter: 3
    TCPReqQFullDoCookies: 42045
    TCPRetransFail: 271
    TCPRcvCoalesce: 606120335
    * OFO: Out Of Order
    TCPOFOQueue: 154893248 (Number of packets queued in OFO queue)
    TCPOFOMerge: 3581444 (Number of packets in OFO that were merged with other packets)
    TCPChallengeACK: 2105995
    TCPSYNChallenge: 64987
    TCPFastOpenCookieReqd: 3824
    TCPSpuriousRtxHostQueues: 332
    TCPAutoCorking: 19132
    TCPWantZeroWindowAdv: 6989
    TCPSynRetrans: 106752693 (Number of SYN and SYN/ACK retransmits to break down retransmissions into SYN,
                                                  fast-retransmits, timeout retransmits, etc.  참고링크)

    TCPOrigDataSent: 11168295644 (Number of outgoing packets with original data, 참고링크)
    TCPHystartTrainDetect: 97852 (Number of  the ACK train length threshold is detected)
    * CWND: Congestion Window
    TCPHystartTrainCwnd: 2647949 (The sum of CWND detected by ACK train length)
    TCPHystartDelayDetect: 21988563 (Number of the packet delay threshold is detected)
    TCPHystartDelayCwnd: 781061970 (The sum of CWND detected by packet delay)
    TCPACKSkippedSynRecv: 1717067 (If the ACk sending frequency is higher than tcp_invalid_ratelimit allows, then +1)
    * PAWS: Protect Against Wrapped Sequence numbers
    TCPACKSkippedPAWS: 1738488
    TCPACKSkippedSeq: 38312
    TCPACKSkippedFinWait2: 2294
    TCPACKSkippedTimeWait: 230801
    TCPACKSkippedChallenge: 8756
IpExt:
    InNoRoutes: 41
    InOctets: 8387403650946
    OutOctets: 12141527324794
    InNoECTPkts: 14298811581
    InECT1Pkts: 279253858
    InECT0Pkts: 271928848
    InCEPkts: 981945413

 

 

그런데 그 설명을 번역만 하는것으로는 도무지 뜬구름 잡는 얘기 같아서 하나도 이해가 가지 않습니다.

일단은 이 수치가 무엇을 의미하는지 먼저 알아보고

이 중에서 필요한것만 따로 모아서 다시 보아야 겠습니다.

 

TCP 통계는 TCP state diagram과 같이 봅니다.

 

TCP 통계를 카테고리화해서 묶어서 보면 한눈에 들어와서 tcp 통계 항목들이 좀 보입니다.
https://perthcharles.github.io/2015/11/10/wiki-netstat-proc/

그마나 찾긴했으나 제가 사용하는 OS와 다른지 통계 factor에 차이가 있습니다.

적당히 느낌만 가지고 넘어갑니다.

category Involve Counters
Realtime RtoAlgorithm、RtoMin、RtoMax、MaxConn
Connection ActiveOpens、PassiveOpens、AttemptFails、CurrEstab、EstabResets
Packet InSegs、OutSegs、RetransSegs、InErrs、OutRsts、InCsumErrors、EmbryonicRsts
syncookies SyncookiesSent、SyncookiesRecv、SyncookiesFailed
TIME_WAIT recycle TW、TWRecycled、TWKilled、TCPTimeWaitOverflow
RTO frequency TCPTimeouts、TCPSpuriousRTOs、TCPLossProbes、TCPLossProbeRecovery、
TCPRenoRecoveryFail、TCPSackRecoveryFail、
TCPRenoFailures、TCPSackFailures、
TCPLossFailures
Retrans Quantity TCPFastRetrans、TCPForwardRetrans、
TCPSlowStartRetrans、TCPLostRetransmit、
TCPRetransFail
FastOpen TCPFastOpenActive、TCPFastOpenPassive、
TCPFastOpenPassiveFail、TCPFastOpenListenOverflow、
TCPFastOpenCookieReqd
MD5 TCPMD5NotFound、TCPMD5Unexpected
DelayedACK DelayedACKs、DelayedACKLocked、DelayedACKLost、
TCPSchedulerFailed
DSACK TCPDSACKOldSent、TCPDSACKOfoSent、
TCPDSACKRecv、TCPDSACKOfoRecv、
TCPDSACKIgnoredOld、TCPDSACKIgnoredNoUndo
Reorder TCPFACKReorder、TCPSACKReorder、
TCPRenoReorder、TCPTSReorder
Recovery TCPRenoRecovery、TCPSackRecovery、
TCPRenoRecoveryFail、TCPSackRecoveryFail
Abort TCPAbortOnData、TCPAbortOnClose、
TCPAbortOnMemory、TCPAbortOnTimeout、
TCPAbortOnLingerTCPAbortFailed

 

 

 

반응형

'기타' 카테고리의 다른 글

TCP Status 상태에 대해서  (433) 2020.07.15
TCP 통계수치 변화 (장애 상황)  (1029) 2020.07.15
Kerberos Setup (KR)  (964) 2019.08.22
500 Error on Confluence Startup (KR)  (333) 2019.07.29
squid proxy  (4) 2018.12.19

+ Recent posts