wafl_cp_slovol_warning_1 with big latency spikes

classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

wafl_cp_slovol_warning_1 with big latency spikes

Fletcher Cocquyt
Yesterday morning one of the heads on our 3270 experienced large NFS latency spikes causing our VMware hosts and their VMs to log storage timeouts.
This latency does not correlate to any external metrics like CPU, network, OPS etc.

But in the logs do show CP events on the aggregate hosting the VMs:

Jan 14 05:27:56 [n04:wafl.cp.slovol:warning]: aggregate aggr2 is holding up the CP.

And the EMS log has CP events logged for the duration of the episode - what can we do to prevent these issues?

<wafl_cp_toolong_warning_1
        total_ms="117825"
        total_dbufs="32276"
        clean="4312"
        v_ino="3"
        v_bm="29"
        a_ino="0"
        a_bm="3428"
        flush="1209"/>
</LR>
<LR d="14Jan2013 05:19:38" n="irt-na04" t="1358169578" id="1335304168/148007" p="4" s="Ok" o="wafl_CP_proc" vf="" type="0" seq="633232" >
<wafl_cp_slovol_warning_1
        voltype="aggregate"
        volowner=""
        volname="aggr2"
        volident=""
        nt="35"
        nb="22045"
        clean="1346852"
        v_ino="0"
        v_bm="113"
        a_ino="0"
        a_bm="4"
        flush="0"
        rgid="2"/>

Netapp support wants me to run perfstats, but the issue is not ongoing - things are idle

thanks




Fletcher Cocquyt
Principal Engineer
Information Resources and Technology (IRT)
Stanford University School of Medicine



Email: fcocquyt at stanford.edu
Phone: (650) 724-7485








-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.teaparty.net/pipermail/toasters/attachments/20130115/24779eec/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-4.png
Type: image/png
Size: 72276 bytes
Desc: not available
URL: <http://www.teaparty.net/pipermail/toasters/attachments/20130115/24779eec/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-3.png
Type: image/png
Size: 10547 bytes
Desc: not available
URL: <http://www.teaparty.net/pipermail/toasters/attachments/20130115/24779eec/attachment-0003.png>

Reply | Threaded
Open this post in threaded view
|

wafl_cp_slovol_warning_1 with big latency spikes

Brian Beaulieu
What about other IO on the aggregate?
Any other volume activity (snaps, resize)

I feel your pain.  I've had one single VM cause IO spikes and thus latency.
 I had this happen when I moved the VM from an NFS datastore to a VMFS
datastore to get more IO potential. Well, that extra IO was too much of a
piece of the pie.

The other night I had all NFS/CIFS IO completely stop for a couple of
minutes during a volume resize.

I love NFS+VMWare+NetApp but sometimes I hate it as well.

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.teaparty.net/pipermail/toasters/attachments/20130117/7431e230/attachment.html>

Reply | Threaded
Open this post in threaded view
|

Re: wafl_cp_slovol_warning_1 with big latency spikes

Fletcher Cocquyt
Running a perfstat now for netapp support and monitoring the disk busy %
We seem to have one disk (3a.01) noticeably busier than the rest of the 95 disks in the aggregate
Is this a potential hotspot?
aggregate re-allocate indicated here?

thanks






On Jan 17, 2013, at 8:24 AM, Brian Beaulieu <[hidden email]> wrote:

What about other IO on the aggregate?
Any other volume activity (snaps, resize)

I feel your pain.  I've had one single VM cause IO spikes and thus latency.  I had this happen when I moved the VM from an NFS datastore to a VMFS datastore to get more IO potential. Well, that extra IO was too much of a piece of the pie.

The other night I had all NFS/CIFS IO completely stop for a couple of minutes during a volume resize.

I love NFS+VMWare+NetApp but sometimes I hate it as well.

Brian
_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|

Re: wafl_cp_slovol_warning_1 with big latency spikes

Fletcher Cocquyt
Resending with smaller (lower quality jpeg) to fit in this list's 100K limit



On Jan 17, 2013, at 9:18 AM, Fletcher Cocquyt <[hidden email]> wrote:

Running a perfstat now for netapp support and monitoring the disk busy %
We seem to have one disk (3a.01) noticeably busier than the rest of the 95 disks in the aggregate
Is this a potential hotspot?
aggregate re-allocate indicated here?

thanks

<PastedGraphic-14.png>





On Jan 17, 2013, at 8:24 AM, Brian Beaulieu <[hidden email]> wrote:

What about other IO on the aggregate?
Any other volume activity (snaps, resize)

I feel your pain.  I've had one single VM cause IO spikes and thus latency.  I had this happen when I moved the VM from an NFS datastore to a VMFS datastore to get more IO potential. Well, that extra IO was too much of a piece of the pie.

The other night I had all NFS/CIFS IO completely stop for a couple of minutes during a volume resize.

I love NFS+VMWare+NetApp but sometimes I hate it as well.

Brian
_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters



_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|

wafl_cp_slovol_warning_1 with big latency spikes

Fletcher Cocquyt
attempt #3



Running a perfstat now for netapp support and monitoring the disk busy %
We seem to have one disk (3a.01) noticeably busier than the rest of the 95 disks in the aggregate
Is this a potential hotspot?
aggregate re-allocate indicated here?

thanks

<PastedGraphic-14.png>

>>
>>
>>
>>
>>
>> On Jan 17, 2013, at 8:24 AM, Brian Beaulieu <brian.beaulieu at gmail.com> wrote:
>>
>>> What about other IO on the aggregate?
>>> Any other volume activity (snaps, resize)
>>>
>>> I feel your pain.  I've had one single VM cause IO spikes and thus latency.  I had this happen when I moved the VM from an NFS datastore to a VMFS datastore to get more IO potential. Well, that extra IO was too much of a piece of the pie.
>>>
>>> The other night I had all NFS/CIFS IO completely stop for a couple of minutes during a volume resize.
>>>
>>> I love NFS+VMWare+NetApp but sometimes I hate it as well.
>>>
>>> Brian
>>> _______________________________________________
>>> Toasters mailing list
>>> Toasters at teaparty.net
>>> http://www.teaparty.net/mailman/listinfo/toasters
>>
>



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.teaparty.net/pipermail/toasters/attachments/20130117/3d4eac8e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: diskbusy3.jpg
Type: image/jpg
Size: 40297 bytes
Desc: not available
URL: <http://www.teaparty.net/pipermail/toasters/attachments/20130117/3d4eac8e/attachment-0001.jpg>

Reply | Threaded
Open this post in threaded view
|

wafl_cp_slovol_warning_1 with big latency spikes

Christopher S Eno
In reply to this post by Brian Beaulieu

I have been fighting these issues since last September.  For our environment it has boiled down to drinking the kool-aid re: SATA/PAM cost savings.  The SATA disks just can't crank out the IOPS that the vmware environment demands, and the PAMII (256G) isn't helping (official response from NetApp engineers).  Right now I have 18TB of SATA disk space devoted to 2TB of vmware, just to get the spindles needed, but still seeing the "stuck" CP which ends up making the FAS3140 stop responding to NFS/CIFS/iSCSI out all interfaces.

We're preparing to move vmware to four DS2246 shelves with 10k 600GB disks, but now seeing that folks are having these issues with 15k disks is very concerning.

We're VMware + NFS too.

On Jan 17, 2013, at 11:24 AM, Brian Beaulieu <brian.beaulieu at gmail.com> wrote:

> What about other IO on the aggregate?
> Any other volume activity (snaps, resize)
>
> I feel your pain.  I've had one single VM cause IO spikes and thus latency.  I had this happen when I moved the VM from an NFS datastore to a VMFS datastore to get more IO potential. Well, that extra IO was too much of a piece of the pie.
>
> The other night I had all NFS/CIFS IO completely stop for a couple of minutes during a volume resize.
>
> I love NFS+VMWare+NetApp but sometimes I hate it as well.
>
> Brian
> _______________________________________________
> Toasters mailing list
> Toasters at teaparty.net
> http://www.teaparty.net/mailman/listinfo/toasters


Reply | Threaded
Open this post in threaded view
|

wafl_cp_slovol_warning_1 with big latency spikes

Brian Beaulieu
3rd time is the charm.

I've attached my sysstat from the other night when NFS/CIFS hung up... is
this what you've seen as well?

During that issue, FCP was also slow.. had some MPIO failovers happening on
our AIX LPARs.  But, AIX handles that just fine and at least has an
alternate path through the other filer.  NFS isn't so lucky.

I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB PAM
would be sufficient for what we do.
While I do have SATA in use for VMWare, it's not heavy hitting VMs.. it's
the dormant stuff, mostly.
I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP.

I'm drinking the PAM kool-aid too but do have some measurable results
primarily on our PeopleSoft DB2 databases.
I definitely wouldn't bet on SATA+PAM == FC/SAS performance.

Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.teaparty.net/pipermail/toasters/attachments/20130117/4df7abbf/attachment-0001.html>
-------------- next part --------------
netapp01b> sysstat -x 1
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
 15%    521      6      0     567    1484  10918    3912     24       0      0    21s    89%  100%  :s   12%       0     40      0      30    287       0      0
  9%    783      2      0     858    1682  16037    5972      0       0      0    21s    91%  100%  :s   13%       0     73      0       6    569       0      0
  7%    215      0      0     303     834   9426    5936      8       0      0    21s    85%  100%  :s   12%       0     88      0      43    619       0      0
  4%    143     12      0     240     133   2872    2608     24       0      0    21s    68%  100%  :s   38%       0     79      0      13    420       0      0
  4%      0     11      0      59      77      4     988      0       0      0    21s    10%  100%  :s   18%       0     48      0      30    365       0      0
  4%      0      2      0      47      38      2     632      0       0      0    21s     0%  100%  :s   38%       0     39      0       4    198       0      0
  2%      0      3      0      23      34      1     804     24       0      0    21s     0%  100%  :s   10%       0     20      0      25    131       0      0
  6%      0      3      0      51      45     11     836     16       0      0    21s     0%  100%  :s   12%       0     42      0       9    191       0      0
  3%      0     80      0     110     202     16     788      0       0      0    21s     0%  100%  :s   10%       0     30      0      50    112       0      0
  4%      0      8      0      99      17      2    1208     24       0      0    21s     0%  100%  :s   13%       0     87      0       4    423       0      0
  5%      0      8      0      61      11      3     960      0       0      0    21s     0%  100%  :s   11%       0     53      0      74    284       0      0
  4%      0      0      0      43       4      1     692      0       0      0    21s     0%  100%  :s   38%       0     41      0       5    223       0      0
  4%      0      8      0      94       8     80     816     32       0      0    21s     0%  100%  :s   12%       0     86      0      27    141       0      0
  3%      0      7      0      40      24      3    1032      0       0      0    21s     0%  100%  :s   12%       0     33      0       9    214       0      0
  3%      0      1      0      13      26      2     724      0       0      0    21s     0%  100%  :s   10%       0     12      0      25     62       0      0
  4%      0     18      0      32      17     10     760     24       0      0    21s     0%  100%  :s   10%       0     14      0       0     99       0      0
  3%      0      0      0      34      20      2     868      0       0      0    21s     0%  100%  :s   13%       0     34      0      29    198       0      0
  3%      0      1      0      96      46      2     856      8       0      0    21s     0%  100%  :s   13%       0     95      0      15    206       0      0
  3%      0      1      0      49      20      3    1088     24       0      0    21s     0%  100%  :s   12%       0     47      0      34    360       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
  4%      0      1      0      61      33      2    1144      0       0      0    21s     0%  100%  :s   11%       0     60      0       1    467       0      0
  7%      0      4      0      66       6      2     960      0       0      0    21s     0%  100%  :s   21%       0     58      0      29    379       0      0
  3%      0      4      0      76       6      2    1112     24       0      0    21s     0%  100%  :s   13%       0     70      0       2    436       0      0
  6%      0      0      0      34       8     17     792      8       0      0    21s     0%  100%  :s   10%       0     30      0      31    153       0      0
  4%      0      4      0      27      82      3     696      0       0      0    21s     0%  100%  :s    9%      15      8      0       1     42       0      0
  4%      0     14      0      71      25      4    1052     24       0      0    21s     0%  100%  :s   14%       0     54      0      38    385       0      0
  5%      0     16      0      58      14      4    1000      0       0      0    21s     0%  100%  :s   12%       0     41      0      75    295       0      0
  3%      0      2      0      33       6      8     852      0       0      0    21s     0%  100%  :s   13%       0     31      0      38    205       0      0
  3%      0     15      0      61      22     11    1012     40       0      0    21s     0%  100%  :s   13%       0     46      0       1    345       0      0
  4%      0      3      0      62      19      2     704      0       0      0    21s     0%  100%  :s   39%       0     59      0      26    274       0      0
  5%      0      2      0      52      15     10    1024      0       0      0    21s     0%  100%  :s   14%       0     50      0       1    385       0      0
  2%      0      1      0      20       2      1     772     24       0      0    21s     0%  100%  :s   12%       0     18      0      25     82       0      0
  3%      0      1      0      12       3      3     740      0       0      0    21s     0%  100%  :s   12%       0     11      0       8     66       0      0
  3%      0      4      0     113       4      3     860      8       0      0    21s     0%  100%  :s   12%       2    104      0      39    231       0      0
  3%      0     34      0      88      21      6     872     24       0      0    21s     0%  100%  :s   10%       0     50      0      12    208       0      0
  3%      0      0      0      35      12      1     940      0       0      0    21s     0%  100%  :s   14%       0     33      0      26    226       0      0
  2%      0      0      0      40      10      2     876      0       0      0    21s     0%  100%  :s    9%       0     34      0       6    170       0      0
  5%      0      7      0     665       7      3    1072     24       0      0    21s     1%  100%  :s   10%     543    115      0      29    371       0      0
  4%      0      0      0       9      13      2     692      8       0      0    21s     0%  100%  :s   13%       0      7      0       1      1       0      0
  2%      0      0      0      31      16      1     816      0       0      0    21s     0%  100%  :s   10%       0     31      0      25    149       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
  3%      0      1      0      50      11      2     859     24       0      0    21s     0%  100%  :s   12%       0     49      0       5    159       0      0
  6%      0      1      0      38       3      1     828      0       0      0    21s     0%  100%  :s   11%       0     37      0      25    214       0      0
  3%      0      0      0     111       3      2     912      0       0      0    21s     0%  100%  :s   12%       0    111      0      15    501       0      0
  4%      0      0      0      35       4      1     980     32       0      0    21s     0%  100%  :s   11%       0     35      0      13    234       0      0
  4%      0      1      0      19      12      3     764      0       0      0    21s     0%  100%  :s   11%       0     18      0      13     83       0      0
  3%      0      1      0      28      13      1     840      0       0      0    21s     0%  100%  :s   12%       1     26      0      13    193       0      0
  2%      0      4      0      25      12      3     780     24       0      0    21s     0%  100%  :s   19%       0     21      0      13    164       0      0
  3%      0      3      0      34       6      1     464      0       0      0    21s     0%  100%  :s   43%       0     27      0      27    104       0      0
  2%      0      7      0      36       6      3     776      8       0      0    21s     0%  100%  :s   19%       0     27      0      18    104       0      0
 47%   5036    207      0    6514    8379   8526   45355 121422       0      0    21s    95%  100%  Hn   47%     411    830      0    5909   2952       0      0
Tue Jan 15 17:18:08 CST [netapp01b:snapmirror.src.connDropped:error]: Error reading/writing to network, connection dropped.  
 89%   7981    424      0   11003  108298  26650   55239 121526       0      0    21s    94%  100%  :v   64%     152   2389      0   29619   4695       0      0
 84%   5701    220      0    7111   66538  11942   48168 212460       0      0     5     95%  100%  Bf   70%       5   1154      0    7126    872       0      0
 66%   7250    232      0    8410   71577  17018   36696  92984       0      0     5     91%  100%  Hs   46%       0    907      0    4519   1507       0      0
 42%   6463     32      0    7134   33268  10816    9376     16       0      0     5     92%  100%  :s   24%      78    542      2    2373    973       0      0
 60%   8348      5      0    8479    3627  12038    6312      0       0      0     5     80%  100%  :s   25%       0    126      0     425    698       0      0
 25%   3920      6      0    4022    1430  13030    8908     24       0      0     5     83%  100%  :s   37%       0     96      0     151    516       0      0
 13%    876      5      0     956     846  15896    6876      0       0      0     5     87%  100%  :s   26%       0     75      0      18    561       0      0
  4%     49      5      0     152     541    700    2092      0       0      0     5     35%  100%  :s   14%       0     98      0      21    746       0      0
  6%    116      5      0     247     342   2101    3468     32       0      0     5     63%  100%  :s   16%       0    126      0      48    939       0      0
  4%     14      9      0     111     266    122    1064      0       0      0     5      6%  100%  :s   13%       0     84      0      36    348       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
  5%      2     12      0     108     132     43    1416      0       0      0     5     24%  100%  :s   13%       0     94      0      18    713       0      0
 12%      3     24      0      99     109    357    1232     24       0      0     5     29%  100%  :s   12%       0     72      0      22    566       0      0
  4%      1      4      0      92      68      2    1336      0       0      0     5     22%  100%  :s   14%       0     87      0      23    676       0      0
  4%      0      7      0     115      21      3    1328      8       0      0     5      0%  100%  :s   12%       0    102      0      71    648       0      0
  4%      3     17      0     122     138     19    1376     24       0      0     5     26%  100%  :s   11%       0     98      0      15    621       0      0
  4%      1      2      0      84      50     20    1416      0       0      0     5     23%  100%  :s   12%       0     81      0      27    631       0      0
  4%      0      1      0      92      29      2    1620      0       0      0     5     30%  100%  :s   14%       0     91      0      15    845       0      0
  3%      0      2      0      58      63      3     980     24       0      0     5      0%  100%  :s   12%       0     50      0      73    215       0      0
  4%      0      2      0      82      30      2    1112      8       0      0     5      5%  100%  :s   13%       0     74      0      16    397       0      0
  2%      0      1      0      48      77      3    1040      0       0      0     5      0%  100%  :s   12%       0     47      0      18    369       0      0
  7%      0      6      0     123      17      2    1312     24       0      0     4     24%  100%  :s   12%      12    105      0      15    627       0      0
  3%      0      0      0      64      29      2    1164      0       0      0     4     14%  100%  :s   12%       0     64      0      14    460       0      0
  5%      0      3      0     172      21      1    1392      0       0      0     4     38%  100%  :s   13%       0    169      0      16    781       0      0
  5%      0      0      0      78      16     80    1256     32       0      0     4     24%  100%  :s   15%       0     78      0      28    582       0      0
  4%      0      0      0      67      16      1    1196      0       0      0     4     18%  100%  :s   12%       0     67      0      14    484       0      0
  4%      0      1      0      68      11      2    1064      0       0      0     4      0%  100%  :s   11%       8     59      0      14    467       0      0
  3%      0      1      0      68       8      1    1196     24       0      0     4     19%  100%  :s   11%       0     67      0      14    528       0      0
  3%      0      1      0     127       3      2    1116      0       0      0     4     29%  100%  :s   13%       0    126      0      15    477       0      0
  5%      0      2      0     139      21     10    1151      8       0      0     4     34%  100%  :s   13%       0    137      0      17    558       0      0
  7%      0      0      0      79      25     26    1212     24       0      0     4     32%  100%  :s   13%       0     79      0      21    619       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
 29%   2171     29      0    2342    3633   2098   38374  94278       0      0     4     93%  100%  Hn   29%       6    131      0     725    519       0      0
 85%   7382    107      0   10926   20776  17282   67964 133053       0      0     4     93%  100%  Hn   77%       2   3256      0   20214   2349       0      0
 47%   8386    354      0   10473    6857  24048   20660  10712       0      0     4     90%  100%  :n   28%       0   1710      0    9345   3575       0      0
 42%   4927    439      0    6708    8330  10233   13532  34528       0      0     4     92%  100%  :s   38%       0   1335      0    8840   1392       0      0
 59%   4540    433      0    6411    8667  15692    9776      0       0      0     4     93%  100%  :s   24%       0   1434      0    6496   8209       0      0
 67%   2143    424      0    2727   21097  15372   13672     24       0      0     4     91%  100%  :s   22%      11    142      0     642    836       0      0
 68%    267    399      0     762    1785   4405    6296      0       0      0     4     77%  100%  :s   13%       0     96      0     433    480       0      0
 67%    255    265      0     601     606   4158    7356      0       0      0     4     77%  100%  :s   16%       0     81      0      29    590       0      0
 67%    175      2      0     272     347   2510    6316     32       0      0     4     66%  100%  :s   18%       0     95      0      44    705       0      0
 69%     58      0      0     161      95    334    1852      0       0      0     4     29%  100%  :s   12%       0    103      0      23    609       0      0
 41%     26      0      0     142     516    634    1740      0       0      0     4     36%  100%  :s   14%       5    111      0      97    812       0      0
  7%    234      0      0     316    1225   3273    5240     24       0      0     4     67%  100%  :s   15%       0     81      0      56    557       0      0
  3%     11      3      0     103     119    105    1412      0       0      0     4     20%  100%  :s   16%       0     89      0      27    705       0      0
  5%    161      4      0     237     256   5317   10136      8       0      0     4     76%  100%  :s   14%       0     72      0      27    508       0      0
 10%    303      2      0     397     303  14478   18292     24       0      0     4     88%  100%  :s   14%       0     86      0      18    502       0      0
  7%    216      1      0     286     223  12667   15980      0       0      0     4     90%  100%  :s   12%       7     62      0      27    443       0      0
  4%      0      4      0      80      75      4    1144      0       0      0     4      0%  100%  :s   14%       0     70      0      20    423       0      0
  4%      0      0      0      38     104      3     916     24       0      0     4      0%  100%  :s   14%       0     38      0      18    271       0      0
  4%      0      1      0      63     175      4     992      8       0      0     4      0%  100%  :s   11%       0     58      0      19    281       0      0
  4%      0      0      0      66      81      3    1052      0       0      0     4      0%  100%  :s   12%       0     62      0      29    420       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
  3%      0      0      0     110      26      2    1240     24       0      0     4      0%  100%  :s   12%       5     98      0      17    481       0      0
  5%      0      0      0      33       4      2     896      0       0      0     4      0%  100%  :s   10%       0     33      0      13    198       0      0
  5%      0      5      0      59      10      2    1124      0       0      0     3      0%  100%  :s   12%       0     54      0      18    345       0      0
  6%      0      1      0      22      34      9     708     32       0      0     3      0%  100%  :s   13%       6     15      0      13     74       0      0
  3%      0      1      0     167      55      3     812      0       0      0     3     17%  100%  :s   12%       0    166      0      16    265       0      0
  3%      0      4      0      42      24      2     844      0       0      0     3      0%  100%  :s    9%       7     31      0      83    213       0      0
  4%      0      0      0      64       4      2    1236     24       0      0     3      0%  100%  :s   12%       0     64      0      14    487       0      0
  3%      0      1      0     113       4      1     956      0       0      0     3      0%  100%  :s   12%       0    112      0      15    358       0      0
  2%      0      1      0      45      18      2     936      8       0      0     3      0%  100%  :s   12%       0     44      0      14    296       0      0
  4%      0      0      0     359      22      1     908     24       0      0     3      0%  100%  :s   13%     322     37      0      17    271       0      0
  2%      0      2      0      32      27      2     864      0       0      0     3      0%  100%  :s   20%       0     30      0      13    201       0      0
  6%      0      0      0      32      36      2    1080    120       0      0     3      0%  100%  :s   11%       0     30      0      14    205       0      0
  2%      0      5      0      65      38      4     948     24       0      0     3      0%  100%  :s   12%       0     56      0      20    257       0      0
  3%      0    336      0     402      91     55     960      8       0      0     3      0%  100%  :s   12%      12     48      0      36    272       0      0
  3%      0      0      0      42      24      2     960      0       0      0     3      0%  100%  :s   12%       0     42      0      14    304       0      0
  3%      0      0      0      60      32      2    1092     24       0      0     3      0%  100%  :s   11%       7     53      0      18    418       0      0
  3%      0      1      0      87      18      2    1072      0       0      0     3      0%  100%  :s   12%       0     80      2      35    472       0      0
  5%      0      8      0      60       7     11    1036      0       0      0     3      0%  100%  :s   13%       1     51      0      14    393       0      0
  3%      0      2      0      50       6      2     952     32       0      0     3      0%  100%  :s   12%       0     48      0      13    353       0      0
  4%      0      2      0      83       7     10    1132      0       0      0     3      0%  100%  :s   13%       0     81      0      18    454       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
  3%      0      0      0      44       5      1     992      0       0      0     3      0%  100%  :s   11%       0     44      0      13    328       0      0
 12%      0      0      0      27      22    292     808     24       0      0     3      0%  100%  :s   11%       0     27      0      29    160       0      0
  3%      0      6      0      45       5      2     944      0       0      0     3      0%  100%  :s   12%       0     39      0      13    311       0      0
  3%      0      0      0      98      14      3     880      8       0      0     3      0%  100%  :s   11%       0     98      0      19    239       0      0
  3%      0      0      0     101      10      1     539     24       0      0     3      0%  100%  :s   38%       0     99      0      77    131       0      0
  2%      0      0      0      31      15      2     792      0       0      0     3      0%  100%  :s   11%       0     27      0      23    140       0      0
  6%      0      7      0      55      15      9     848      0       0      0     3      0%  100%  :s   10%       0     44      0      21    199       0      0
  2%      0      0      0     128       4      4    1176     24       0      0     3     10%  100%  :s   14%       0    128      0      15    517       0      0
  3%      0     12      0      68       8      4     964      8       0      0     3      0%  100%  :s   11%       0     54      0      18    281       0      0
  3%      0      1      0     107       6      1    1568      0       0      0     3     14%  100%  :s   14%       0    106      0      16    836       0      0
  3%      0      0      0      95       4      2    1392     24       0      0     3      0%  100%  :s   12%       0     94      0      15    553       0      0
  3%      0      0      0     101       4      1    1540      0       0      0     3      3%  100%  :s   17%       0    101      0      19    747       0      0
  3%      0      1      0      69       8      2    1012      0       0      0     3      0%  100%  :s   17%       0     64      0      18    415       0      0
  5%      0      1      0      62      18     80    1044     32       0      0     3      0%  100%  :s   16%       0     60      0      18    419       0      0
  3%      0     11      0      77      10      5    1156      0       0      0     3      0%  100%  :s   21%       0     66      0      14    476       0      0
  3%      0      0      0      60      17      4    1144      0       0      0     3      0%  100%  :s   12%       0     60      0      14    467       0      0
  3%      0      0      0      59       3      2    1156     24       0      0     3      0%  100%  :s   10%       0     59      0      14    471       0      0
  3%      0      1      0      72       5      1    1212      0       0      0     3      0%  100%  :s   13%       0     69      0      19    476       0      0
  3%      1      4      0     122       7      2    1526      8       0      0     3     11%  100%  :s   14%       0    115      0      16    856       0      0
  3%      0      0      0     124       6      1    1460     24       0      0     3     22%  100%  :s   11%       0    118      0      30    838       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
  3%      0      0      0      34       5      2     968      0       0      0     3      0%  100%  :s   10%       0     34      0      17    230       0      0
  8%      0      1      0      87       5      2    1244      0       0      0     3      0%  100%  :s   13%       0     85      0      59    590       0      0
  3%      0      1      0     132      13      2    1060     24       0      0     3      0%  100%  :s   10%       0    131      0      15    437       0      0
  4%      0      1      0     150      10      9    1100      8       0      0     3     27%  100%  :s   33%       0    147      0      98    599       0      0
  3%      0      1      0      75       4      2    1208      0       0      0     3      0%  100%  :s   13%       0     74      0      24    562       0      0
  4%      0      4      0      47      18      4     904     24       0      0     3      0%  100%  :s   37%       0     43      0      13    352       0      0
  5%      0      3      0     128      19      3    1168      0       0      0     3      0%  100%  :s   12%       0    125      0      15    485       0      0
  3%      0      8      0      88       7      8    1256      0       0      0     3      0%  100%  :s   12%       0     80      0      35    590       0      0
  2%      0      3      0      41       4      2     936     32       0      0     3      0%  100%  :s   11%       0     38      0      13    271       0      0
  3%      0      0      0      80       6      1    1120      0       0      0     3      0%  100%  :s   13%       0     77      0      14    421       0      0
  4%      0      6      0      55       6      2     988      0       0      0     3      0%  100%  :s   12%       0     47      0      14    321       0      0
  3%      0      1      0      47       4      5     980     24       0      0     3      0%  100%  :s   11%       0     43      0      27    239       0      0
  4%      0     16      0      79      22     15    1096      0       0      0     3      0%  100%  :s   12%       0     57      0      28    346       0      0
  3%      0     10      0      66       7      4    1064      8       0      0     3      0%  100%  :s   13%       0     56      0      15    403       0      0
  3%      0      0      0      51       4      2     928     24       0      0     3      0%  100%  :s   12%       0     50      0      13    289       0      0
  4%      0      1      0      30      12      1     900      0       0      0     4      0%  100%  :s   10%       0     29      0      13    213       0      0
  5%      0      2      0      33       7      9     832      0       0      0     4      0%  100%  :s   11%       0     31      0      13    193       0      0
  3%      0      1      0      33       2      2     848     24       0      0     4      0%  100%  :s   14%       0     32      0      17    221       0      0
  5%      0      0      0      47       6      5     876      8       0      0     4      0%  100%  :s   12%       0     45      0      13    240       0      0
  2%      0      5      0      40       7      7     932      0       0      0     4      0%  100%  :s   11%       0     34      0      18    238       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
  7%      0      0      0     309       4      1     788     24       0      0     4      0%  100%  :s   10%     265     44      0      13    135       0      0
  7%      0      0      0     123       3      2    1024      0       0      0     4      0%  100%  :s   11%       0    123      0      28    412       0      0
  3%      0     20      0     145      20      5    1008      0       0      0     4      0%  100%  :s   12%       0    123      0      81    379       0      0
  4%      0      1      0      53       7      2     964     32       0      0     4      0%  100%  :s   15%       0     50      0      36    296       0      0
  2%      0     27      0      81      12      6     920      0       0      0     4      0%  100%  :s   15%       0     52      0      51    272       0      0
  3%      0      0      0     121       9      2     864      0       0      0     4      3%  100%  :s   26%       0    117      0      17    355       0      0
 29%   4068     76      0    4402    3034   2133   29328 117028       0      0     4     96%  100%  :v   28%      16    239      0    1122   2713       0      0
Tue Jan 15 17:20:04 CST [netapp01b:snapmirror.src.connDropped:error]: Error reading/writing to network, connection dropped.  
 92%   6177    372      0    9293   56264  15706   57908 102804       0      0     4     95%  100%  Zf   78%     141   2471      0   16242   4159       0      0
 42%   7901    369      0    8779   19443   6051   13300  25320       0      0     2     83%  100%  Hn   22%       3    438      0    4965    814       0      0
 39%   3464     13      0    4396   38211   6079    9380  70448       0      0     2     92%  100%  :s   21%       1    899      0   33602    818       0      0
  5%     60     18      0     157     573    200    1104      0       0      0     2     21%  100%  :s   11%       0     79      0     371    369       0      0
  7%      2      4      0      32     488     43     792      0       0      0     1      0%  100%  :s   11%       0     26      0      76    156       0      0
 19%   4569      8      0    4632    1321   1734    1220     24       0      0     1     44%  100%  :s   15%       0     55      0      26    289       0      0
 18%   4779      5      0    4802    1384   1675     836      8       0      0     1     34%  100%  :s   10%       0     18      0      25    131       0      0
 11%   2621      1      0    2757     684   1227    1652      0       0      0     1     76%  100%  :s   12%       0    135      0      40   2686       0      0
  4%     37      0      0      84      86    164    1265     24       0      0     1      0%  100%  :s   13%       0     47      0      30    360       0      0
  7%    215      4      0     272    1348   2012    2664      0       0      0     1     51%  100%  :s   10%       0     53      0      42    365       0      0
  4%     20      9      0      52     116    176     924      0       0      0     1      0%  100%  :s   12%       0     23      0      67    123       0      0
  9%      6      6      0      60      74     74     916     32       0      0     1      0%  100%  :s   11%       0     44      0       2    288       0      0
  5%      2     10      0      76      51     24     948      0       0      0     1      0%  100%  :s   12%       0     64      0      31    283       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
  4%      4      3      0      53      53     85    1048      0       0      0     1      0%  100%  :s   12%       0     42      0       5    271       0      0
 10%      0      1      0      45      68    293     904     24       0      0     1      0%  100%  :s   10%       0     44      0      30    320       0      0
  5%      2      8      0      57      40      4     900      0       0      0     1      0%  100%  :s   10%       0     43      0      20    263       0      0
  3%      1      0      0      22      62      3     712      8       0      0     1      0%  100%  :s   11%       0     21      0      29     99       0      0
  4%      4      1      0      81      47      3    1064     24       0      0     1      0%  100%  :s   11%       0     72      0      55    380       0      0
  5%      1      5      0      54      47      3     940      0       0      0     1      0%  100%  :s   12%       0     46      0      40    271       0      0
  3%      3      4      0      19      37      5     692      0       0      0     1      0%  100%  :s    9%       0     12      0       0     50       0      0
  3%      0      1      0      64       6      2    1120     24       0      0     1      0%  100%  :s   12%       0     61      0      27    443       0      0
 10%      0      0      0      94      39      3     524      8       0      0     1     15%  100%  :s   41%       0     94      0       2    134       0      0
 14%      0      0      0      92      41      3     764      0       0      0     1      0%  100%  :s   11%       0     92      0      88    178       0      0
  6%      0    338      0     393      92     56     756     24       0      0     1      0%  100%  :s   10%       0     55      0      15    171       0      0
  3%      0      0      0      36      51      2     824      0       0      0     1      0%  100%  :s   12%       0     35      0      25    206       0      0
  5%      0      0      0      88       7      2     728      0       0      0     1      0%  100%  :s   10%       0     88      0       2    134       0      0
  4%      0      0      0      20      30     80     756     32       0      0     1      0%  100%  :s   10%       0     20      0      26     99       0      0
  3%      0      0      0      38      35      3     908      0       0      0     1      0%  100%  :s   12%       0     38      0       1    255       0      0
  3%      0      0      0      72      26      2    1136      0       0      0     1      0%  100%  :s   12%       0     68      0      28    469       0      0
  3%      0      2      0      63      13      2    1136     24       0      0     1      0%  100%  :s   13%       0     61      0       1    496       0      0
 12%      0      0      0      58       7      3    1008      0       0      0     1      0%  100%  :s   12%       0     54      0      31    337       0      0
  3%      0      3      0     167      21      3    1268      8       0      0     1     24%  100%  :s   22%       0    162      0       6    724       0      0
  3%      0      0      0      72      39      2    1003     24       0      0     1      0%  100%  :s   12%       3     65      0      91    329       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
  3%      0      1      0      61      30      2    1104      0       0      0     1      0%  100%  :s   13%       0     58      0      15    410       0      0
 10%      0      1      0      36       6      2     856      0       0      0     1      0%  100%  :s   12%       0     35      0      30    254       0      0
  3%      0      0      0      23       6      2     668     24       0      0     1      0%  100%  :s    9%       0     23      0       5     43       0      0
  3%      0      0      0       8      29      2     708      8       0      0     1      0%  100%  :s   10%       0      8      0      25     57       0      0
  2%      0      1      0      36      14      2     908      0       0      0     1      0%  100%  :s   13%       0     35      0      13    266       0      0
  3%      0      5      0      57      22      3    1060     24       0      0     1      0%  100%  :s   12%       0     52      0      26    410       0      0
  3%      0      2      0       7      14      8     656      0       0      0     1      0%  100%  :s   10%       0      5      0       4      9       0      0
  5%      0      0      0      90       8      9     764      0       0      0     1      0%  100%  :s   11%       0     90      0      35    173       0      0
  3%      0      0      0     109      18      2     868     32       0      0     1      0%  100%  :s   10%       0    105      0      70    224       0      0
  3%      0      1      0      29      34      3     640      0       0      0     1      0%  100%  :s    9%       0     28      0      39      4       0      0
  4%      0      2      0      25      31      2     580      0       0      0     1      0%  100%  :s   38%       0     21      0       2    120       0      0
  3%      0      1      0      29      10      2     652     24       0      0     1      0%  100%  :s   20%       0     25      0      26     91       0      0
  2%      0      2      0      94       9      2     704      0       0      0     1      4%  100%  :s   24%       0     90      0       2    206       0      0
  3%      0      0      0      48      31      2    1008      8       0      0     1      0%  100%  :s   11%       0     48      0      29    296       0      0
  3%      0      0      0      45      17      2     700     24       0      0     1      0%  100%  :s   34%       0     43      0       1    239       0      0
  3%      0      0      0      60      18      2    1060      0       0      0     1      0%  100%  :s   13%       0     60      0      26    459       0      0
  3%      0      0      0      40      17      2     864      0       0      0     1      0%  100%  :s   11%       5     35      0       1    234       0      0
  4%      0      0      0     115       8      2     996     24       0      0     1     10%  100%  :s   12%       0    115      0      27    395       0      0
  4%      0      0      0      46      16      2     868      8       0      0     1      0%  100%  :s   14%       0     46      0       4    248       0      0
  4%      0      1      0      57      37      3    1060      0       0      0     1      0%  100%  :s   11%       0     55      0      30    394       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
  3%      0      1      0      57      34      2    1080     24       0      0     1      0%  100%  :s   13%       0     55      0       1    394       0      0
  7%      0      2      0      81      13      2     972      0       0      0     1      0%  100%  :s   10%       0     75      0      28    345       0      0
  2%      0      4      0      41      11      2     924      0       0      0     1      0%  100%  :s   10%       1     36      0       1    214       0      0
  6%      0      1      0      43      20      2     848     32       0      0     2      0%  100%  :s   11%       0     40      0      27    235       0      0
  4%      0      1      0      34      19      3     824      0       0      0     2      0%  100%  :s   10%       0     31      0       2    169       0      0
  6%      0      5      0      32      22      3     782      0       0      0     2      0%  100%  :s   10%       0     27      0      25    180       0      0
 91%   8167    239      0   11591   52708  16201   86459 228663       0      0     2     96%  100%  Hf   61%     369   2789      0   16449  20657       0      0
 88%   9889    517      0   12541   71712  19875   61400 102624       0      0    58s    91%  100%  Bs   73%      86   1987      0    9250   5321       0      0
 63%  13385    223      0   17785   43419  25767   23972   8200       0      0     0s    91%  100%  :s   41%       0   4042      0    3604  25829       0      0
 48%   9547      1      0   13950    2738  23309   21812     24       0      0     0s    90%  100%  :s   40%       0   4402      0     635  28681       0      0
 45%    921      8      0    4600    1072  18716   18612      0       0      0     0s    88%  100%  :s   30%       0   3671      0     221  16761       0      0
 44%    177      4      0    4688     863   1079    2452      0       0      0     0s    87%  100%  :s   28%       4   4503      0     197  20869       0      0
 43%    400     15      0    4996     693   5676    7064     24       0      0     8     88%  100%  :s   18%       0   4581      0     143  21346       0      0
 43%   2396      0      0    6848     925  69841   77656      8       0      0     8     90%  100%  :s   46%       0   4452      0     123  21713       0      0
 56%   1694      0      0    6110     739  64401   67936      0       0      0     8     94%  100%  :s   51%       0   4416      0     119  20538       0      0
 64%   2612      1      0    6787    1150  88008  107108     24       0      0     7     90%  100%  :s   70%       0   4174      0     112  19640       0      0
 64%   3896     12      0    7886    1586 132631  139292      0       0      0     7     92%  100%  :s   84%       0   3977      0     125  18785       0      0
 63%   1842      0      0    5819     790  59792   62888      0       0      0     5     95%  100%  :s   32%       0   3971      0     113  18125       0      0
 67%     39      1      0    4215      67   1235    3956     32       0      0     5     96%  100%  :s   12%       0   4169      0     113  19056       0      0
 67%     31      2      0    4231      68    800    2352      0       0      0     5     96%  100%  :s   13%       0   4198      0     117  19144       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
 64%      0      4      0    4053      59    296     592      0       0      0     5     97%  100%  :s   44%       0   4049      0     110  18575       0      0
 67%      0     16      0    3735      64      8     976     24       0      0     5     95%  100%  :s   16%       9   3704      0     263  16973       0      0
 67%      0      0      0    4164      26      5     848      0       0      0     5     96%  100%  :s   13%       1   4155      0     114  19048       0      0
 67%      1      0      0    4252      12     39     928      8       0      0     5     96%  100%  :s   10%       0   4251      0     114  19481       0      0
 67%      1      0      0    3210      14      5     868     24       0      0     5     95%  100%  :s   12%       0   3209      0      89  14746       0      0
 67%      1     24      0      59      63     19     808      0       0      0     5      0%  100%  :s   11%       0     34      0      13    190       0      0
 58%   1821    166      0    3625    6902   3382   33636  91752       0      0     5     93%  100%  :v   36%       8   1625      0    4295   6126       0      0
 92%   4502    512      0    8300   72565  11554   78507 230229       0      0     5     94%   96%  Hf   58%       0   3153      0   13008   8201       0      0
 75%   3245    495      0    7642   49810  13132   54072  65277       0      0     5     96%  100%  Hs   60%       0   3766      0   16406  12865       0      0
 67%   2368   1067      0    7589   16473  36541   51456      0       0      0     0s    90%  100%  :s   60%       0   4138      0    2954  18392       0      0
 65%   3127   1069      0    8168   18139  36356   32652     24       0      0     0s    93%  100%  :s   38%       0   3959      0    1475  18493       0      0
 62%   1490    399      0    6115    7740  15443   16180      0       0      0     5     89%  100%  :s   30%       0   4204      0    1863  18211       0      0
 66%   1530    547      0    6388    7405  22797   20744      0       0      0     5     91%  100%  :s   27%       7   4296      0    1664  29103       0      0
 63%   1335     28      0    2178    1425  63376   18248     32       0      0     4     90%  100%  :s   21%       0    815      0     352   3671       0      0
 68%    476     15      0     544     947  15685   11148      0       0      0     4     83%  100%  :s   24%       0     53      0     232    251       0      0
 67%    161      8      0     225    1033    838    1888      0       0      0     4      9%  100%  :s   10%       0     56      0      46    394       0      0
 68%    282     20      0     379     938   2561    3184     24       0      0     4     56%  100%  :s   16%       0     76      0     186    430       0      0
 68%    405     68      0     573    1098   4154    4672      0       0      0     4     65%  100%  :s   12%       8     92      0      80    558       0      0
 68%     77      3      0     122     242    513    1404      8       0      0     4      0%  100%  :s   13%       0     42      0      11    295       0      0
 69%    123      1      0     175     238   3080    1032     24       0      0     4     80%  100%  :s   11%       0     51      0      31    377       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
 68%     17      4      0      46      83    100     928      0       0      0     4      0%  100%  :s   10%       2     23      0       9    131       0      0
 67%     20      5      0      62      69    107     904      0       0      0     4      0%  100%  :s   10%       0     31      0      35    156       0      0
 45%      5      0      0      50     100     42     788     24       0      0     4      0%  100%  :s   12%       5     34      0       5     93       0      0
  3%      0      0      0       5      95      3     688      8       0      0     4      0%  100%  :s    9%       0      5      0      25     25       0      0
  2%      0      4      0      24      92      4     796      0       0      0     4      0%  100%  :s   10%       0     20      0       0    152       0      0
  5%      0      2      0      88      47     10    1004     24       0      0     4      0%  100%  :s   12%       0     82      0     170    345       0      0
  4%      0      1      0      61      18      2     944      0       0      0     4      0%  100%  :s   10%       0     52      0       5    281       0      0
  5%      0      0      0      62      25      8    1052      0       0      0     4      0%  100%  :s   11%       5     57      0      26    426       0      0
 35%   2063    102      0    2852    8573   5683   56608  87658       0      0     4     92%  100%  Hn   34%       3    674      0    4391   2354       0      0
 96%   3703    528      0    5659   34305  15522   76686 204829       0      0     4     93%  100%  Hn   73%       0   1422      0    7088   1366       0      0
 30%   2273    620      0    3316   13698  16157   21692   1134       0      0     4     85%  100%  :s   28%       0    414      0    4724    980       0      0
 39%   2608    641      0    3439   10451  28779   26022  25385       0      0     4     91%  100%  :s   32%       0    175      0    1321    342       0      0
 91%   2413    234      0    2915    6493  37020   59619 134254       0      0     4     94%   97%  Zf   42%       9    249      0    2456    367       0      0
 92%   2296    381      0    2917   14615  16839   62294  73099       0      0     4     92%   96%  Zf   46%       0    233      0    1506    422       0      0
 96%   2085    586      0    2873   10800  11578   32996  39817       0      0     1s    87%   52%  Z    46%       1    199      0    2123    245       0      0
 99%   1966    808      0    2984   22409  32727   12008      0       0      0     1s    94%    0%  -    25%       2    201      0    1324    243       0      0
 81%   2876    574      0    3701   13631  79261   78963  55363       0      0     3     92%  100%  Hf   51%       0    244      0    1965    277       0      0
 99%   4226    473      0    4903   18303 123429   84811  21284       0      0     3     92%   55%  :    48%       8    188      0    2049    399       0      0
 90%   4841    437      0    5490   28280 117829   86171  59500       0      0     3     94%   87%  Hf   61%       0    204      0    1965    201       0      0
 97%   4266    515      0    5379   29829 104717  105214  87123       0      0    50s    93%   89%  Hf   64%       0    597      0    5019    432       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
 98%   3997    453      0    4978   61201  75588   72701 121084       0      0     0s    95%   74%  H    63%       0    521      0    4565   2392       0      0
 97%   3310    356      0    3851   53038  29665   54697  76238       0      0    47s    96%   37%  Hn   47%       0    178      0     730    281       0      0
 97%   3876    541      0    4610   12338 125200   59244  72148       0      0     1s    96%   34%  :    38%       5    180      0    1144    521       0      0
 99%   4167    504      0    5263   24660 139104   53083     32       0      0    47s    97%    0%  -    42%       2    581      0    4451    345       0      0
 97%   4021    471      0    4828   18158 138614   93042  64407       0      0     0s    97%   93%  Hf   43%       0    335      0    2965    347       0      0
 99%   3429    534      0    4377   18881 130407   77948  23932       0      0    48s    96%   58%  :    41%       9    398      0    2561    267       0      0
 94%   3071    387      0    3810   15491 125730  117926  47892       0      0    49s    96%   34%  Hf   64%       0    343      0    2335   3156       0      0
 99%   3761    420      0    5027   30305 114609   82248  38985       0      0    36s    96%   91%  Hn   80%      86    746      0    4470   4011       0      0
 92%   4163    482      0    4847   34214 117670   78359  84331       0      0     0s    95%   84%  :    54%       0    194      0    1427    973       0      0
 96%   4056    369      0    4580   63899  93558   76219  75451       0      0    18s    96%   75%  Hn   53%       0    153      0    1524    479       0      0
 95%   3792    499      0    4441   70282  82808   73917 156202       0      0    16s    95%   68%  Hf   50%       0    143      0    1618    206       0      0
 95%   4655    330      0    5164   81440 114214  123333  99273       0      0    14s    93%   66%  Hf   49%       0    172      0    1975    127       0      0
 92%   3652    247      0    4067   65820  95263   89928  86785       0      0    13s    94%   80%  Hf   49%       5    154      0     494    311       0      0
 97%   4248    250      0    4643   75626  93762   71401 132886       0      0    11s    96%  100%  Hf   62%       0    135      0    1229    398       0      0
Tue Jan 15 17:21:03 CST [netapp01b:snapmirror.src.connDropped:error]: Error reading/writing to network, connection dropped.  
 97%   4408    102      0    4667   96011  83539   69962 118777       0      0     9s    96%   88%  Hv   63%       0    151      0     845    391       0      0
 93%   3864    421      0    4446   63744  56728   72645 163331       0      0     9s    95%   79%  Hf   59%       4    146      0    1329    492       0      0
 97%   3613    122      0    3995   34255  98851   50330  25722       0      0     9s    95%   22%  Hs   58%       0    252      0    2087    404       0      0
 96%   4040    495      0    4686   73156  75929   59361 106496       0      0     0s    95%  100%  Hf   55%       5    141      0    2162    486       0      0
 97%   3392    205      0    3855   31478 126124   86275  92554       0      0     9s    94%   94%  Hf   82%       0    254      0    2384    346       0      0
 97%   3838    500      0    5343   53288 118991   99234  87892       0      0     8s    95%   80%  Hv   91%       0    997      0    6765    329       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
 99%   3958    623      0    4980   15986 136552   77387    142       0      0     8s    93%    9%  :    83%       0    392      0    2055    260       0      0
 91%   2705    413      0    3303   18202 113161   90664  55873       0      0     8s    94%   53%  Hf   75%       0    178      0    1089   1376       0      0
 99%   3066    441      0    3636   27950 127594   63473  32671       0      0     7s    95%   76%  :    80%       9    117      0    1803    382       0      0
 96%   3533    362      0    4187   33510 121438   99532  66564       0      0     5s    94%   56%  Hf   84%       0    277      0    2489    387       0      0
 99%   3666    471      0    4291   31519 141720   85535  21992       0      0     6s    95%   65%  Hn   78%       0    147      0    1555    223       0      0
 96%   3482    496      0    4157   33411 141626  103590  84558       0      0     7s    96%   69%  :    91%       1    169      0    1837    160       0      0
 95%   3788    559      0    4606   47558 130035   91980  76056       0      0     7s    96%   69%  Hf   84%       0    252      0    2212    394       0      0
 96%   3936    498      0    4537   36805 140628   94761  70038       0      0     8s    95%   66%  Hf   84%       6     95      0     424    378       0      0
 97%   4491    368      0    5382   43567 133593   82980  36653       0      0     9s    95%   45%  Hs   79%       0    358      0    1888    706       0      0
 95%   4053    388      0    5002   87889 111963   90906 142719       0      0     9s    96%   99%  Hf   76%       0    553      0    2614   1657       0      0
 98%   3719    420      0    4584   84077 114784   76319 122818       0      0    10s    96%   76%  Hf   75%       0    437      0    1609   1431       0      0
 98%   3616    462      0    4275   82091  93089   59445 114186       0      0    10s    95%   85%  Hv   68%       0    191      0     545    512       0      0
 96%   3328     97      0    3705   44457 121261   86366  81970       0      0    11s    95%   59%  Hf   75%       5    273      0    4757    443       0      0
 92%   3320    395      0    4178   31968 119444   88555  88496       0      0    11s    97%   86%  H    72%       0    455      0    1722    970       0      0
 95%   3189    507      0    4040   34786 113773   86077  44690       0      0    11s    97%   25%  Hf   75%       1    336      0    1647   1142       0      0
 98%   3559    445      0    4179   39509 120741   65519  64666       0      0     0s    96%   55%  Hs   69%       0    166      0     587    937       0      0
 98%   4109    372      0    4610   49512 126011   64523  42290       0      0    12s    96%   32%  :    72%       0    116      0    1080    203       0      0
 96%   3728    580      0    4466   40963 122230   73202  83417       0      0    12s    96%   89%  H    74%       9    147      0     580    179       0      0
 98%   3841    447      0    4507   56641 116725   74766  87593       0      0    12s    96%   61%  H    70%       0    214      0     777    315       0      0
 98%   3811    153      0    4295   48429 113835   82737  72863       0      0    13s    96%   70%  H    82%       0    329      0    2206    216       0      0
 CPU    NFS   CIFS   HTTP   Total     Net   kB/s    Disk   kB/s    Tape   kB/s  Cache  Cache    CP  CP  Disk   OTHER    FCP  iSCSI     FCP   kB/s   iSCSI   kB/s
                                       in    out    read  write    read  write    age    hit  time  ty  util                            in    out      in    out
 85%   3254    840      0    4350   72456  77604   84108 127393       0      0    13s    97%   94%  Hs   75%       0    256      0    1367    304       0      0
 89%   2965    303      0    3441   30377 115125   75361  54861       0      0    13s    95%   81%  Hs   77%       0    172      0     339    363       0      0
 91%   3347    645      0    4127   71092  63905   62150 101946       0      0    13s    96%   86%  Hf   57%       5    129      0     334    523       0      0
 96%   1858    792      0    2776   10768  60804   40851  21582       0      0    15s    95%   36%  :    63%       0    126      0     456    505       0      0
 95%   1714    595      0    2410    9844  59120   39732      0       0      0    16s    94%    0%  -    57%       0    100      0     375    280       0      0
 89%   1572      0      0    1667    8372  61292   41628     24       0      0    16s    94%    0%  -    62%       0     89      0     327    263       0      0
 89%   2270      1      0    2394    9742  65307   56988  26564       0      0    16s    93%   37%  Hf   55%       0    121      0     299    542       0      0
 94%   1787      4      0    1988    8976  60561   64635  50030       0      0    16s    94%  100%  :f   59%       5    191      0     324   1483       0      0
 96%   2336    256      0    2762   19168  58693   42536  14048       0      0    18s    94%   51%  :    48%       0    170      0     530    409       0      0
 71%   2872      5      0    3429   14587  60153   39280      0       0      0    18s    96%    0%  -    40%       0    552      0    2599    486       0      0
 56%   2090      0      0    2257   12452  62088   44564  54876       0      0    18s    94%   77%  Hs   41%       0    165      0     982    518       0      0
 61%   1691      5      0    1807   19164  53872   33008     24       0      0    21s    94%  100%  :s   44%       0    109      0     221    563       0      0
 62%   2746     53      0    2963   13796  84925   61465      0       0      0     0s    95%  100%  :s   50%       5    159      0    1019    638       0      0
 62%   1805    344      0    2323   13541  59160   36763      8       0      0    21s    94%  100%  :s   47%       0    173      0     692    541       0      0
 59%   2187      8      0    2359   17808  60449   28407     24       0      0    21s    95%  100%  :s   34%       0    162      0    1593    196       0      0
 63%   1471     14      0    1892    3988  60929   34812      0       0      0    21s    94%  100%  :s   40%       0    402      0    2574    337       0      0
 62%   1290      7      0    1771    5648  54676   35904      0       0      0    21s    95%  100%  :s   41%       0    471      0    1715   2820       0      0
 58%   1413      8      0    1799    7340  59447   35365     24       0      0    21s    96%  100%  :s   40%       7    370      0     695   3288       0      0
 34%   2099    245      0    2545   19171  62731   34708      8       0      0    21s    96%  100%  :s   32%       0    200      0     984    869       0      0
 30%   1827     22      0    2078    4973  58644   34864      0       0      0    22s    94%  100%  :s   39%       0    213      0    2261    320       0      0
 

Reply | Threaded
Open this post in threaded view
|

wafl_cp_slovol_warning_1 with big latency spikes

Christopher S Eno

Yes, this is what I see.  ":s" and all the other protocols go to "0".

There's been some correlation, when this happens, to cleanup of VMware snapshots (not NetApp snaps on the volumes, but VMware snapshots of vm's via vcenter).  But it happens other times too.



On Jan 17, 2013, at 1:47 PM, Brian Beaulieu <brian.beaulieu at gmail.com> wrote:

> 3rd time is the charm.
>
> I've attached my sysstat from the other night when NFS/CIFS hung up... is this what you've seen as well?
>
> During that issue, FCP was also slow.. had some MPIO failovers happening on our AIX LPARs.  But, AIX handles that just fine and at least has an alternate path through the other filer.  NFS isn't so lucky.
>
> I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB PAM would be sufficient for what we do.
> While I do have SATA in use for VMWare, it's not heavy hitting VMs.. it's the dormant stuff, mostly.
> I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP.
>
> I'm drinking the PAM kool-aid too but do have some measurable results primarily on our PeopleSoft DB2 databases.
> I definitely wouldn't bet on SATA+PAM == FC/SAS performance.
>
> Brian
> <sysstat - Copy.txt>_______________________________________________
> Toasters mailing list
> Toasters at teaparty.net
> http://www.teaparty.net/mailman/listinfo/toasters

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.teaparty.net/pipermail/toasters/attachments/20130117/dc2e8c9b/attachment.html>

Reply | Threaded
Open this post in threaded view
|

wafl_cp_slovol_warning_1 with big latency spikes

Jeff Mohler-2
:s is wafl updating special files in the CP process.


Going on _that_ long....???  A few seconds of special file updates in a CP
sure, but that much?

I'd be pretty pushy on getting an answer, id put that in the "its a big
bug" bucket.   That's not normal IO activity in a healthy system.

On Thu, Jan 17, 2013 at 8:07 PM, Scott Eno <s.eno at me.com> wrote:

>
> Yes, this is what I see.  ":s" and all the other protocols go to "0".
>
> There's been some correlation, when this happens, to cleanup of VMware
> snapshots (not NetApp snaps on the volumes, but VMware snapshots of vm's
> via vcenter).  But it happens other times too.
>
>
>
> On Jan 17, 2013, at 1:47 PM, Brian Beaulieu <brian.beaulieu at gmail.com>
> wrote:
>
> 3rd time is the charm.
>
> I've attached my sysstat from the other night when NFS/CIFS hung up... is
> this what you've seen as well?
>
> During that issue, FCP was also slow.. had some MPIO failovers happening
> on our AIX LPARs.  But, AIX handles that just fine and at least has an
> alternate path through the other filer.  NFS isn't so lucky.
>
> I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB
> PAM would be sufficient for what we do.
> While I do have SATA in use for VMWare, it's not heavy hitting VMs.. it's
> the dormant stuff, mostly.
> I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP.
>
> I'm drinking the PAM kool-aid too but do have some measurable results
> primarily on our PeopleSoft DB2 databases.
> I definitely wouldn't bet on SATA+PAM == FC/SAS performance.
>
> Brian
> <sysstat - Copy.txt>_______________________________________________
>
> Toasters mailing list
> Toasters at teaparty.net
> http://www.teaparty.net/mailman/listinfo/toasters
>
>
>
> _______________________________________________
> Toasters mailing list
> Toasters at teaparty.net
> http://www.teaparty.net/mailman/listinfo/toasters
>
>


--
---
Gustatus Similis Pullus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.teaparty.net/pipermail/toasters/attachments/20130117/dca5692f/attachment.html>

Reply | Threaded
Open this post in threaded view
|

wafl_cp_slovol_warning_1 with big latency spikes

Brian Beaulieu
Scott, what version of OnTAP are you on?
We're on 8.1.1P1

I saw a BURT that sounded related to this but it was apparently fixed by
8.1.1P1.

Brian


On Thu, Jan 17, 2013 at 2:21 PM, Jeff Mohler <speedtoys.racing at gmail.com>wrote:

> :s is wafl updating special files in the CP process.
>
>
> Going on _that_ long....???  A few seconds of special file updates in a CP
> sure, but that much?
>
> I'd be pretty pushy on getting an answer, id put that in the "its a big
> bug" bucket.   That's not normal IO activity in a healthy system.
>
>
> On Thu, Jan 17, 2013 at 8:07 PM, Scott Eno <s.eno at me.com> wrote:
>
>>
>> Yes, this is what I see.  ":s" and all the other protocols go to "0".
>>
>> There's been some correlation, when this happens, to cleanup of VMware
>> snapshots (not NetApp snaps on the volumes, but VMware snapshots of vm's
>> via vcenter).  But it happens other times too.
>>
>>
>>
>> On Jan 17, 2013, at 1:47 PM, Brian Beaulieu <brian.beaulieu at gmail.com>
>> wrote:
>>
>> 3rd time is the charm.
>>
>> I've attached my sysstat from the other night when NFS/CIFS hung up... is
>> this what you've seen as well?
>>
>> During that issue, FCP was also slow.. had some MPIO failovers happening
>> on our AIX LPARs.  But, AIX handles that just fine and at least has an
>> alternate path through the other filer.  NFS isn't so lucky.
>>
>> I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB
>> PAM would be sufficient for what we do.
>> While I do have SATA in use for VMWare, it's not heavy hitting VMs.. it's
>> the dormant stuff, mostly.
>> I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP.
>>
>> I'm drinking the PAM kool-aid too but do have some measurable results
>> primarily on our PeopleSoft DB2 databases.
>> I definitely wouldn't bet on SATA+PAM == FC/SAS performance.
>>
>> Brian
>> <sysstat - Copy.txt>_______________________________________________
>>
>> Toasters mailing list
>> Toasters at teaparty.net
>> http://www.teaparty.net/mailman/listinfo/toasters
>>
>>
>>
>> _______________________________________________
>> Toasters mailing list
>> Toasters at teaparty.net
>> http://www.teaparty.net/mailman/listinfo/toasters
>>
>>
>
>
> --
> ---
> Gustatus Similis Pullus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.teaparty.net/pipermail/toasters/attachments/20130117/de99e044/attachment-0001.html>

Reply | Threaded
Open this post in threaded view
|

wafl_cp_slovol_warning_1 with big latency spikes

Christopher S Eno

We're on 8.1.1P1 across the environment.

7-mode, of course.



On Jan 17, 2013, at 3:35 PM, Brian Beaulieu <brian.beaulieu at gmail.com> wrote:

> Scott, what version of OnTAP are you on?
> We're on 8.1.1P1
>
> I saw a BURT that sounded related to this but it was apparently fixed by 8.1.1P1.
>
> Brian
>
>
> On Thu, Jan 17, 2013 at 2:21 PM, Jeff Mohler <speedtoys.racing at gmail.com> wrote:
> :s is wafl updating special files in the CP process.
>
>
> Going on _that_ long....???  A few seconds of special file updates in a CP sure, but that much?
>
> I'd be pretty pushy on getting an answer, id put that in the "its a big bug" bucket.   That's not normal IO activity in a healthy system.
>
>
> On Thu, Jan 17, 2013 at 8:07 PM, Scott Eno <s.eno at me.com> wrote:
>
> Yes, this is what I see.  ":s" and all the other protocols go to "0".
>
> There's been some correlation, when this happens, to cleanup of VMware snapshots (not NetApp snaps on the volumes, but VMware snapshots of vm's via vcenter).  But it happens other times too.
>
>
>
> On Jan 17, 2013, at 1:47 PM, Brian Beaulieu <brian.beaulieu at gmail.com> wrote:
>
>> 3rd time is the charm.
>>
>> I've attached my sysstat from the other night when NFS/CIFS hung up... is this what you've seen as well?
>>
>> During that issue, FCP was also slow.. had some MPIO failovers happening on our AIX LPARs.  But, AIX handles that just fine and at least has an alternate path through the other filer.  NFS isn't so lucky.
>>
>> I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB PAM would be sufficient for what we do.
>> While I do have SATA in use for VMWare, it's not heavy hitting VMs.. it's the dormant stuff, mostly.
>> I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP.
>>
>> I'm drinking the PAM kool-aid too but do have some measurable results primarily on our PeopleSoft DB2 databases.
>> I definitely wouldn't bet on SATA+PAM == FC/SAS performance.
>>
>> Brian
>> <sysstat - Copy.txt>_______________________________________________
>>
>> Toasters mailing list
>> Toasters at teaparty.net
>> http://www.teaparty.net/mailman/listinfo/toasters
>
>
> _______________________________________________
> Toasters mailing list
> Toasters at teaparty.net
> http://www.teaparty.net/mailman/listinfo/toasters
>
>
>
>
> --
> ---
> Gustatus Similis Pullus
>
> _______________________________________________
> Toasters mailing list
> Toasters at teaparty.net
> http://www.teaparty.net/mailman/listinfo/toasters

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.teaparty.net/pipermail/toasters/attachments/20130117/721df74f/attachment.html>

Reply | Threaded
Open this post in threaded view
|

wafl_cp_slovol_warning_1 with big latency spikes

Brian Beaulieu
In reply to this post by Brian Beaulieu
I had opened a case when this happened and let the TSE know that I'm not
the only one having this issue.. asked for it to be escalated and treated
as a possible bug.
We'll see where it goes.

Brian

On Thu, Jan 17, 2013 at 4:16 PM, Scott Eno <s.eno at icloud.com> wrote:

> We're on 8.1.1P1 across the environment.
>
> 7-mode, of course.
>
>
> On Jan 17, 2013, at 3:35 PM, Brian Beaulieu <brian.beaulieu at gmail.com>
> wrote:
>
> Scott, what version of OnTAP are you on?
> We're on 8.1.1P1
>
> I saw a BURT that sounded related to this but it was apparently fixed by
> 8.1.1P1.
>
> Brian
>
>
> On Thu, Jan 17, 2013 at 2:21 PM, Jeff Mohler <speedtoys.racing at gmail.com>wrote:
>
>> :s is wafl updating special files in the CP process.
>>
>>
>> Going on _that_ long....???  A few seconds of special file updates in a
>> CP sure, but that much?
>>
>> I'd be pretty pushy on getting an answer, id put that in the "its a big
>> bug" bucket.   That's not normal IO activity in a healthy system.
>>
>>
>> On Thu, Jan 17, 2013 at 8:07 PM, Scott Eno <s.eno at me.com> wrote:
>>
>>>
>>> Yes, this is what I see.  ":s" and all the other protocols go to "0".
>>>
>>> There's been some correlation, when this happens, to cleanup of VMware
>>> snapshots (not NetApp snaps on the volumes, but VMware snapshots of vm's
>>> via vcenter).  But it happens other times too.
>>>
>>>
>>>
>>> On Jan 17, 2013, at 1:47 PM, Brian Beaulieu <brian.beaulieu at gmail.com>
>>> wrote:
>>>
>>> 3rd time is the charm.
>>>
>>> I've attached my sysstat from the other night when NFS/CIFS hung up...
>>> is this what you've seen as well?
>>>
>>> During that issue, FCP was also slow.. had some MPIO failovers happening
>>> on our AIX LPARs.  But, AIX handles that just fine and at least has an
>>> alternate path through the other filer.  NFS isn't so lucky.
>>>
>>> I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB
>>> PAM would be sufficient for what we do.
>>> While I do have SATA in use for VMWare, it's not heavy hitting VMs..
>>> it's the dormant stuff, mostly.
>>> I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP.
>>>
>>> I'm drinking the PAM kool-aid too but do have some measurable results
>>> primarily on our PeopleSoft DB2 databases.
>>> I definitely wouldn't bet on SATA+PAM == FC/SAS performance.
>>>
>>> Brian
>>> <sysstat - Copy.txt>_______________________________________________
>>>
>>> Toasters mailing list
>>> Toasters at teaparty.net
>>> http://www.teaparty.net/mailman/listinfo/toasters
>>>
>>>
>>>
>>> _______________________________________________
>>> Toasters mailing list
>>> Toasters at teaparty.net
>>> http://www.teaparty.net/mailman/listinfo/toasters
>>>
>>>
>>
>>
>> --
>> ---
>> Gustatus Similis Pullus
>
>
> _______________________________________________
> Toasters mailing list
> Toasters at teaparty.net
> http://www.teaparty.net/mailman/listinfo/toasters
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.teaparty.net/pipermail/toasters/attachments/20130117/f33f0802/attachment-0001.html>

Reply | Threaded
Open this post in threaded view
|

Re: wafl_cp_slovol_warning_1 with big latency spikes

Jeff Mohler-2
In reply to this post by Fletcher Cocquyt
Perhaps.  Need iop count between that bust dlsk and a representative non busy one in the same raid group.  

Or, a disk statistics snippet from a statit would be better.  Yes.  Better.  

Sent from my iPhone

On Jan 17, 2013, at 5:22 PM, Fletcher Cocquyt <[hidden email]> wrote:

<diskbusy2.jpg>
Resending with smaller (lower quality jpeg) to fit in this list's 100K limit



On Jan 17, 2013, at 9:18 AM, Fletcher Cocquyt <[hidden email]> wrote:

Running a perfstat now for netapp support and monitoring the disk busy %
We seem to have one disk (3a.01) noticeably busier than the rest of the 95 disks in the aggregate
Is this a potential hotspot?
aggregate re-allocate indicated here?

thanks

<PastedGraphic-14.png>





On Jan 17, 2013, at 8:24 AM, Brian Beaulieu <[hidden email]> wrote:

What about other IO on the aggregate?
Any other volume activity (snaps, resize)

I feel your pain.  I've had one single VM cause IO spikes and thus latency.  I had this happen when I moved the VM from an NFS datastore to a VMFS datastore to get more IO potential. Well, that extra IO was too much of a piece of the pie.

The other night I had all NFS/CIFS IO completely stop for a couple of minutes during a volume resize.

I love NFS+VMWare+NetApp but sometimes I hate it as well.

Brian
_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|

RE: wafl_cp_slovol_warning_1 with big latency spikes

Christopher S Eno
In reply to this post by Brian Beaulieu

Hi Brian, Toasters,

 

A week ago we were hit with the “ :s “ CP type again.  One vm, on a rather idle aggregate of 15 SATA disks, was storage vmotioned from one vol on the aggregate to another vol on the same aggr.  This took about 25 minutes and put the aggr disks to 100%.  Near the completion of the VM move, the nasty “ :s “ CP types began and the controller started refusing all protocols while this CP type ran for up to 30 seconds.

 

Looking at a graph, one sees a “dead cat bounce” as all traffic and activity on the filer goes to zero, then starts again, then goes to zero, etc., three times total, as the controller attempts to catch up with queued protocol requests, but also trying to handle the long running “ :s “.  Attempting to attach screen shot of block/s throughput on the aggr.

 

Brian, I have netapp’s attention on this right now, so if you still have a case open, shoot me the case number and I can let them know someone else has seen the same thing.

 

a5421a7013fbb1ea772a9a98f901b934.png

 

403c6ca88576065c6a644acdbd12269c.png

 

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Brian Beaulieu
Sent: Thursday, January 17, 2013 8:21 PM
To: Scott Eno
Cc: [hidden email] Lists
Subject: Re: wafl_cp_slovol_warning_1 with big latency spikes

 

I had opened a case when this happened and let the TSE know that I'm not the only one having this issue.. asked for it to be escalated and treated as a possible bug.

We'll see where it goes.

 

Brian

 

On Thu, Jan 17, 2013 at 4:16 PM, Scott Eno <[hidden email]> wrote:

We're on 8.1.1P1 across the environment.

 

7-mode, of course.

 

 

On Jan 17, 2013, at 3:35 PM, Brian Beaulieu <[hidden email]> wrote:



Scott, what version of OnTAP are you on?

We're on 8.1.1P1

 

I saw a BURT that sounded related to this but it was apparently fixed by 8.1.1P1.

 

Brian

 

On Thu, Jan 17, 2013 at 2:21 PM, Jeff Mohler <[hidden email]> wrote:

:s is wafl updating special files in the CP process.


Going on _that_ long....???  A few seconds of special file updates in a CP sure, but that much?

I'd be pretty pushy on getting an answer, id put that in the "its a big bug" bucket.   That's not normal IO activity in a healthy system.

 

On Thu, Jan 17, 2013 at 8:07 PM, Scott Eno <[hidden email]> wrote:

 

Yes, this is what I see.  ":s" and all the other protocols go to "0".

 

There's been some correlation, when this happens, to cleanup of VMware snapshots (not NetApp snaps on the volumes, but VMware snapshots of vm's via vcenter).  But it happens other times too.

 

 

 

On Jan 17, 2013, at 1:47 PM, Brian Beaulieu <[hidden email]> wrote:

 

3rd time is the charm.

 

I've attached my sysstat from the other night when NFS/CIFS hung up... is this what you've seen as well? 

 

During that issue, FCP was also slow.. had some MPIO failovers happening on our AIX LPARs.  But, AIX handles that just fine and at least has an alternate path through the other filer.  NFS isn't so lucky.

 

I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB PAM would be sufficient for what we do.

While I do have SATA in use for VMWare, it's not heavy hitting VMs.. it's the dormant stuff, mostly.

I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP.

 

I'm drinking the PAM kool-aid too but do have some measurable results primarily on our PeopleSoft DB2 databases.

I definitely wouldn't bet on SATA+PAM == FC/SAS performance.

 

Brian

<sysstat - Copy.txt>_______________________________________________

 


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters



--
---
Gustatus Similis Pullus

 

_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters

 

 


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|

RE: wafl_cp_slovol_warning_1 with big latency spikes (UNCLASSIFIED)

Blackburn, James M (Jim) Jr. CIV USARMY PEO EIS (US)
Classification: UNCLASSIFIED
Caveats: NONE

Is this really so unusual with an aggregate such as the one described?  Another contributor would be if the aggregate is attached to a filer with either a relatively-small amount of NVRAM/NVMEM, or if there's a lot of competing write traffic for any amount of NVRAM/NVMEM.  Also, streaming writes have never been a strength of NetApp FAS units.
--
Jim Blackburn
US Army
[hidden email]
Note: I am not authorized to obligate, commit, or execute government funds.  No part of this message should be construed as a request or directive to obligate, commit, or execute government funds.


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Christopher S Eno
Sent: Friday, March 01, 2013 9:35 AM
To: 'Brian Beaulieu'
Cc: [hidden email]
Subject: RE: wafl_cp_slovol_warning_1 with big latency spikes

Hi Brian, Toasters,



A week ago we were hit with the " :s " CP type again.  One vm, on a rather idle aggregate of 15 SATA disks, was storage vmotioned from one vol on the aggregate to another vol on the same aggr.  This took about 25 minutes and put the aggr disks to 100%.  Near the completion of the VM move, the nasty " :s " CP types began and the controller started refusing all protocols while this CP type ran for up to 30 seconds.



Looking at a graph, one sees a "dead cat bounce" as all traffic and activity on the filer goes to zero, then starts again, then goes to zero, etc., three times total, as the controller attempts to catch up with queued protocol requests, but also trying to handle the long running " :s ".  Attempting to attach screen shot of block/s throughput on the aggr.



Brian, I have netapp's attention on this right now, so if you still have a case open, shoot me the case number and I can let them know someone else has seen the same thing.



a5421a7013fbb1ea772a9a98f901b934.png



403c6ca88576065c6a644acdbd12269c.png







From: [hidden email] [mailto:[hidden email]] On Behalf Of Brian Beaulieu
Sent: Thursday, January 17, 2013 8:21 PM
To: Scott Eno
Cc: [hidden email] Lists
Subject: Re: wafl_cp_slovol_warning_1 with big latency spikes



I had opened a case when this happened and let the TSE know that I'm not the only one having this issue.. asked for it to be escalated and treated as a possible bug.

We'll see where it goes.



Brian



On Thu, Jan 17, 2013 at 4:16 PM, Scott Eno <[hidden email]> wrote:

        We're on 8.1.1P1 across the environment.



        7-mode, of course.





        On Jan 17, 2013, at 3:35 PM, Brian Beaulieu <[hidden email]> wrote:





                Scott, what version of OnTAP are you on?

                We're on 8.1.1P1



                I saw a BURT that sounded related to this but it was apparently fixed by 8.1.1P1.



                Brian



                On Thu, Jan 17, 2013 at 2:21 PM, Jeff Mohler <[hidden email]> wrote:

                        :s is wafl updating special files in the CP process.


                        Going on _that_ long....???  A few seconds of special file updates in a CP sure, but that much?

                        I'd be pretty pushy on getting an answer, id put that in the "its a big bug" bucket.   That's not normal IO activity in a healthy system.



                        On Thu, Jan 17, 2013 at 8:07 PM, Scott Eno <[hidden email]> wrote:



                                Yes, this is what I see.  ":s" and all the other protocols go to "0".



                                There's been some correlation, when this happens, to cleanup of VMware snapshots (not NetApp snaps on the volumes, but VMware snapshots of vm's via vcenter).  But it happens other times too.







                                On Jan 17, 2013, at 1:47 PM, Brian Beaulieu <[hidden email]> wrote:



                                        3rd time is the charm.



                                        I've attached my sysstat from the other night when NFS/CIFS hung up... is this what you've seen as well?



                                        During that issue, FCP was also slow.. had some MPIO failovers happening on our AIX LPARs.  But, AIX handles that just fine and at least has an alternate path through the other filer.  NFS isn't so lucky.



                                        I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB PAM would be sufficient for what we do.

                                        While I do have SATA in use for VMWare, it's not heavy hitting VMs.. it's the dormant stuff, mostly.

                                        I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP.



                                        I'm drinking the PAM kool-aid too but do have some measurable results primarily on our PeopleSoft DB2 databases.

                                        I definitely wouldn't bet on SATA+PAM == FC/SAS performance.



                                        Brian

                                        <sysstat - Copy.txt>_______________________________________________


                                        Toasters mailing list
                                        [hidden email]
                                        http://www.teaparty.net/mailman/listinfo/toasters




                                _______________________________________________
                                Toasters mailing list
                                [hidden email]
                                http://www.teaparty.net/mailman/listinfo/toasters





                        --
                        ---
                        Gustatus Similis Pullus



                _______________________________________________
                Toasters mailing list
                [hidden email]
                http://www.teaparty.net/mailman/listinfo/toasters






Classification: UNCLASSIFIED
Caveats: NONE



_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|

Re: wafl_cp_slovol_warning_1 with big latency spikes

Jeff Mohler-2
In reply to this post by Christopher S Eno
Again, this is a deeper bug.  :s CPs are not anything unusual, just updating special files.

VERY normal.

NOT normal for something -else- to hang up that process.


Just saying, dont fear the :s, its a victim of this bug as well..not a cause.



On Fri, Mar 1, 2013 at 6:34 AM, Christopher S Eno <[hidden email]> wrote:

Hi Brian, Toasters,

 

A week ago we were hit with the “ :s “ CP type again.  One vm, on a rather idle aggregate of 15 SATA disks, was storage vmotioned from one vol on the aggregate to another vol on the same aggr.  This took about 25 minutes and put the aggr disks to 100%.  Near the completion of the VM move, the nasty “ :s “ CP types began and the controller started refusing all protocols while this CP type ran for up to 30 seconds.

 

Looking at a graph, one sees a “dead cat bounce” as all traffic and activity on the filer goes to zero, then starts again, then goes to zero, etc., three times total, as the controller attempts to catch up with queued protocol requests, but also trying to handle the long running “ :s “.  Attempting to attach screen shot of block/s throughput on the aggr.

 

Brian, I have netapp’s attention on this right now, so if you still have a case open, shoot me the case number and I can let them know someone else has seen the same thing.

 

a5421a7013fbb1ea772a9a98f901b934.png

 

403c6ca88576065c6a644acdbd12269c.png

 

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Brian Beaulieu
Sent: Thursday, January 17, 2013 8:21 PM
To: Scott Eno


Cc: [hidden email] Lists
Subject: Re: wafl_cp_slovol_warning_1 with big latency spikes

 

I had opened a case when this happened and let the TSE know that I'm not the only one having this issue.. asked for it to be escalated and treated as a possible bug.

We'll see where it goes.

 

Brian

 

On Thu, Jan 17, 2013 at 4:16 PM, Scott Eno <[hidden email]> wrote:

We're on 8.1.1P1 across the environment.

 

7-mode, of course.

 

 

On Jan 17, 2013, at 3:35 PM, Brian Beaulieu <[hidden email]> wrote:



Scott, what version of OnTAP are you on?

We're on 8.1.1P1

 

I saw a BURT that sounded related to this but it was apparently fixed by 8.1.1P1.

 

Brian

 

On Thu, Jan 17, 2013 at 2:21 PM, Jeff Mohler <[hidden email]> wrote:

:s is wafl updating special files in the CP process.


Going on _that_ long....???  A few seconds of special file updates in a CP sure, but that much?

I'd be pretty pushy on getting an answer, id put that in the "its a big bug" bucket.   That's not normal IO activity in a healthy system.

 

On Thu, Jan 17, 2013 at 8:07 PM, Scott Eno <[hidden email]> wrote:

 

Yes, this is what I see.  ":s" and all the other protocols go to "0".

 

There's been some correlation, when this happens, to cleanup of VMware snapshots (not NetApp snaps on the volumes, but VMware snapshots of vm's via vcenter).  But it happens other times too.

 

 

 

On Jan 17, 2013, at 1:47 PM, Brian Beaulieu <[hidden email]> wrote:

 

3rd time is the charm.

 

I've attached my sysstat from the other night when NFS/CIFS hung up... is this what you've seen as well? 

 

During that issue, FCP was also slow.. had some MPIO failovers happening on our AIX LPARs.  But, AIX handles that just fine and at least has an alternate path through the other filer.  NFS isn't so lucky.

 

I have a 3250+1TB PAM sitting on deck.. you'd think that the 3240+512GB PAM would be sufficient for what we do.

While I do have SATA in use for VMWare, it's not heavy hitting VMs.. it's the dormant stuff, mostly.

I'm moving a lot of it, though, to 6xDS4243x600GB-15k shelves ASAP.

 

I'm drinking the PAM kool-aid too but do have some measurable results primarily on our PeopleSoft DB2 databases.

I definitely wouldn't bet on SATA+PAM == FC/SAS performance.

 

Brian

<sysstat - Copy.txt>_______________________________________________

 


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters



--
---
Gustatus Similis Pullus

 

_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters

 

 


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters




--
---
Gustatus Similis Pullus
_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|

Re: wafl_cp_slovol_warning_1 with big latency spikes

dave.withers
In reply to this post by Fletcher Cocquyt
Subscribed.

We run 3240's in HA 7-mode on 8.1.2 and we have been battling issues similar to the OP and going back and forth with netapp for the last 3 months on resolution.  We have moved hotspots that netapp identified from 24/7 perfstat logs to SAS off SATA and have basically removed virtually all write heavy IO applications from the SATA aggregate but we will still see the :s cp type and all of the sudden experience latency spikes across all protocols.  I think we have gone through 3 upgrades based on 'bugs' netapp claimed to have found/fixed.  We are definitely in a better place, but the latency issue is still too common to feel comfortable about.   Definitely would liek to be added to an escalation and would be happy to provide logs/stats/etc that may help get this issue noticed by netapp.
Reply | Threaded
Open this post in threaded view
|

Re: wafl_cp_slovol_warning_1 with big latency spikes

Christopher S Eno
Hi Dave,

We just went through an escalation to the higher floors of NetApp.  Professional Services came on-site and gathered data.  We ran a storage vmotion to re-create the :s and perfstat-ed the whole event.  After further analysis, the PSE claims to have found a "bug" related to the :s and the model of controller, FAS3160.  We are awaiting their report.  The "snowquester" here in DC has delayed that report.

If you have a case open, have your case owner check into case 2003994303.


On Mar 6, 2013, at 10:49 AM, dave.withers <[hidden email]> wrote:

> Subscribed.
>
> We run 3240's in HA 7-mode on 8.1.2 and we have been battling issues similar
> to the OP and going back and forth with netapp for the last 3 months on
> resolution.  We have moved hotspots that netapp identified from 24/7
> perfstat logs to SAS off SATA and have basically removed virtually all write
> heavy IO applications from the SATA aggregate but we will still see the :s
> cp type and all of the sudden experience latency spikes across all
> protocols.  I think we have gone through 3 upgrades based on 'bugs' netapp
> claimed to have found/fixed.  We are definitely in a better place, but the
> latency issue is still too common to feel comfortable about.   Definitely
> would liek to be added to an escalation and would be happy to provide
> logs/stats/etc that may help get this issue noticed by netapp.
>
>
>
>
> --
> View this message in context: http://network-appliance-toasters.10978.n7.nabble.com/wafl-cp-slovol-warning-1-with-big-latency-spikes-tp24495p24680.html
> Sent from the Network Appliance - Toasters mailing list archive at Nabble.com.
> _______________________________________________
> Toasters mailing list
> [hidden email]
> http://www.teaparty.net/mailman/listinfo/toasters


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|

Re: wafl_cp_slovol_warning_1 with big latency spikes

Jeff Mohler-2
Snowquester, funny.   ;)

:s is updating 'special files'..nothing fancy, wafl underlying maps/etc.

Something has the system with indigestion IN that process, not because of it.  I used to see this as well when in the service of Netapp as the field perf guy, but for my life, I do not recall what the root is/was at THAT time.

I'll be glad to see you guys get your response.   :)



On Wed, Mar 6, 2013 at 8:15 AM, Scott Eno <[hidden email]> wrote:
Hi Dave,

We just went through an escalation to the higher floors of NetApp.  Professional Services came on-site and gathered data.  We ran a storage vmotion to re-create the :s and perfstat-ed the whole event.  After further analysis, the PSE claims to have found a "bug" related to the :s and the model of controller, FAS3160.  We are awaiting their report.  The "snowquester" here in DC has delayed that report.

If you have a case open, have your case owner check into case 2003994303.


On Mar 6, 2013, at 10:49 AM, dave.withers <[hidden email]> wrote:

> Subscribed.
>
> We run 3240's in HA 7-mode on 8.1.2 and we have been battling issues similar
> to the OP and going back and forth with netapp for the last 3 months on
> resolution.  We have moved hotspots that netapp identified from 24/7
> perfstat logs to SAS off SATA and have basically removed virtually all write
> heavy IO applications from the SATA aggregate but we will still see the :s
> cp type and all of the sudden experience latency spikes across all
> protocols.  I think we have gone through 3 upgrades based on 'bugs' netapp
> claimed to have found/fixed.  We are definitely in a better place, but the
> latency issue is still too common to feel comfortable about.   Definitely
> would liek to be added to an escalation and would be happy to provide
> logs/stats/etc that may help get this issue noticed by netapp.
>
>
>
>
> --
> View this message in context: http://network-appliance-toasters.10978.n7.nabble.com/wafl-cp-slovol-warning-1-with-big-latency-spikes-tp24495p24680.html
> Sent from the Network Appliance - Toasters mailing list archive at Nabble.com.
> _______________________________________________
> Toasters mailing list
> [hidden email]
> http://www.teaparty.net/mailman/listinfo/toasters


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters



--
---
Gustatus Similis Pullus
_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|

RE: wafl_cp_slovol_warning_1 with big latency spikes

Christopher S Eno

We had our meeting with NetApp yesterday and went over the Professional Services findings.  Some things they listed are tasks we’ve been addressing since the slovol issues began, aligning mis-aligned VMs, adding disks to aggregates (or, in our case, moving VM’s to larger aggr w/ faster disks).  But one thing they confirmed, which was brought to my attention via an off-toasters email discussion hours before (I give that individual much thanks!!), was BURT 393877, “inefficient pre-fetching of metadata blocks delays WAFL Consistency Point.”

 

Data ONTAP's WAFL filesystem periodically commits user-modified data to the

back-end storage media (disk or otherwise) to achieve a Consistency Point (CP).

Although a Consistency Point typically takes only a few seconds, a constraint

has been designed into the software that all operations needed for a single

Consistency Point must be completed within 10 minutes.  If a CP has not been

completed before a 600-second timer expires, a "WAFL hung" panic is declared,

and a core dump is produced to permit diagnosis of the excessive CP delay.

 During the processing for a CP, some disk blocks are newly brought into use,

as fresh data is stored in the active filesystem, whereas some blocks may be

released from use. (Although a block which is no longer needed in the active

filesystem may remain in use in one or more snapshots, until all the snapshots

which use it are deleted.) But any changes in block usage must be reflected in

the accounting information kept in the volume metadata.  To make changes in

the block accounting, Data ONTAP must read metadata blocks from disk, bringing

them into the storage controller's physical memory.  Because the freeing of

blocks often occurs in a random ordering, the workload of updating the metadata

for block frees can be much higher than for updating the metadata to reflect

blocks just being brought into use.

 For greatest processing efficiency, Data ONTAP makes an effort to pre-fetch

blocks of metadata which are likely to be needed for a given Consistency Point.

However, in some releases of Data ONTAP, the pre-fetching of metadata is done

in an inefficient way, and therefore the processing for the Consistency Point

may run slower than it should.  This effect can be most pronounced for certain

workloads (especially overwrite workloads) in which many blocks may be freed

in unpredictable sequences.  And the problem may be compounded if other tasks

being performed by Data ONTAP attempt intensive use of the storage controller's

memory.  The competition for memory may cause metadata blocks to be evicted

before the Consistency Point is finished with them, leading to buffer thrashing

and a heavy disk-read load.

 In aggravated cases, the Consistency Point may be slowed so much that it cannot

be completed in 10 minutes, thus triggering a "WAFL hung" event.

 

The BURT doesn’t list any specific workarounds, as, apparently, there’s many depending on your environment and what’s causing it.  For us, they wanted to take each FAS3160 controller down to the boot prompt and make an environment change.  They didn’t say what this change was because it would have to be undone once a version of OnTap is released that fixes the issue.

 

On that topic, there was an almost guarantee that 8.1.2P3 will probably have the fix, but 8.1.3 will definitely have the fix.  I only get an OnTap upgrade window twice a year (April & October) so I hope 8.1.2P3 has the fix.  They were unsure as to the release date of 8.1.3.

 

Some, or most, of you may be aware of this already, but I wanted to follow up with our results in case someone else starts seeing this issue.  At least you’ll have a place to start with NetApp support.

 

Again, thanks to everyone that shared ideas on this topic!  This mailing list is an invaluable resource!

 

 

From: Jeff Mohler [mailto:[hidden email]]
Sent: Wednesday, March 06, 2013 11:26 AM
To: Scott Eno
Cc: dave.withers; [hidden email]
Subject: Re: wafl_cp_slovol_warning_1 with big latency spikes

 

Snowquester, funny.   ;)

:s is updating 'special files'..nothing fancy, wafl underlying maps/etc.

Something has the system with indigestion IN that process, not because of it.  I used to see this as well when in the service of Netapp as the field perf guy, but for my life, I do not recall what the root is/was at THAT time.

I'll be glad to see you guys get your response.   :)


On Wed, Mar 6, 2013 at 8:15 AM, Scott Eno <[hidden email]> wrote:

Hi Dave,

We just went through an escalation to the higher floors of NetApp.  Professional Services came on-site and gathered data.  We ran a storage vmotion to re-create the :s and perfstat-ed the whole event.  After further analysis, the PSE claims to have found a "bug" related to the :s and the model of controller, FAS3160.  We are awaiting their report.  The "snowquester" here in DC has delayed that report.

If you have a case open, have your case owner check into case 2003994303.



On Mar 6, 2013, at 10:49 AM, dave.withers <[hidden email]> wrote:


> Subscribed.
>
> We run 3240's in HA 7-mode on 8.1.2 and we have been battling issues similar
> to the OP and going back and forth with netapp for the last 3 months on
> resolution.  We have moved hotspots that netapp identified from 24/7
> perfstat logs to SAS off SATA and have basically removed virtually all write
> heavy IO applications from the SATA aggregate but we will still see the :s
> cp type and all of the sudden experience latency spikes across all
> protocols.  I think we have gone through 3 upgrades based on 'bugs' netapp
> claimed to have found/fixed.  We are definitely in a better place, but the
> latency issue is still too common to feel comfortable about.   Definitely
> would liek to be added to an escalation and would be happy to provide
> logs/stats/etc that may help get this issue noticed by netapp.
>
>
>
>
> --
> View this message in context: http://network-appliance-toasters.10978.n7.nabble.com/wafl-cp-slovol-warning-1-with-big-latency-spikes-tp24495p24680.html
> Sent from the Network Appliance - Toasters mailing list archive at Nabble.com.
> _______________________________________________
> Toasters mailing list
> [hidden email]
> http://www.teaparty.net/mailman/listinfo/toasters


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters




--
---
Gustatus Similis Pullus


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
12