qtree snapmirror inexplicably slow once a week

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

qtree snapmirror inexplicably slow once a week

BERNTSEN Basil

Hi folks,

 

I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site that's slated to be decommed, but for now, I need to keep the lights on. There is a qtree snapmirror that, aside from Mondays, runs in about 3-4 hours. Mondays, it takes 2-4 days. The change rate on the source doesn't appear to be any different. Also, this was working happily for years and only started acting up lately. There are only 5 disks on the system, and while they're at 100% now, they're also at 100% during a sample taken while this context wasn't updating, and while it was updating in a way that finished in 4 hours.

 

So far, I've checked statit during the event and while idle, and I couldn't see anything that stuck out as a massive difference, except maybe name cache hits? I have 68% name cache hits during this transfer, and 100% while idle.

 

I don't see anything suspicious in the wafl scan status, and reallocate status -v shows:

 

Reallocation scans are on

No reallocation status.

 

CPU is idle, network has no errors. Does anyone have any suggestions? What kinds of things should I be looking at?

 

Thanks,

 

Basil

 

************************************************************************* 
This message and any attachments (the "message") are confidential, intended solely for the addressee(s), and may contain legally privileged information.  Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. Please visit http://sgasdisclosure.com for important information regarding SG Americas Securities, LLC (“SGAS”). Please visit http://swapdisclosure.sgcib.com for important information regarding swap transactions with SOCIETE GENERALE.  
*************************************************************************


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: qtree snapmirror inexplicably slow once a week

Joel Krajden
Are there any other snapmirror qtrees happening on the same volume on
Mondays. If yes the locks on snapshots might be the problem.

J


On 12/13/2016 11:04 AM, BERNTSEN Basil wrote:

> Hi folks,
>
>
>
> I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site
> that's slated to be decommed, but for now, I need to keep the lights on.
> There is a qtree snapmirror that, aside from Mondays, runs in about 3-4
> hours. Mondays, it takes 2-4 days. The change rate on the source doesn't
> appear to be any different. Also, this was working happily for years and
> only started acting up lately. There are only 5 disks on the system, and
> while they're at 100% now, they're also at 100% during a sample taken
> while this context wasn't updating, and while it was updating in a way
> that finished in 4 hours.
>
>
>
> So far, I've checked statit during the event and while idle, and I
> couldn't see anything that stuck out as a massive difference, except
> maybe name cache hits? I have 68% name cache hits during this transfer,
> and 100% while idle.
>
>
>
> I don't see anything suspicious in the wafl scan status, and reallocate
> status -v shows:
>
>
>
> Reallocation scans are on
>
> No reallocation status.
>
>
>
> CPU is idle, network has no errors. Does anyone have any suggestions?
> What kinds of things should I be looking at?
>
>
>
> Thanks,
>
>
>
> Basil
>
>
>
> *************************************************************************
> This message and any attachments (the "message") are confidential,
> intended solely for the addressee(s), and may contain legally privileged
> information.  Any unauthorized use or dissemination is prohibited.
> E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any
> of its subsidiaries or affiliates shall be liable for the message if
> altered, changed or falsified. Please visit http://sgasdisclosure.com
> for important information regarding SG Americas Securities, LLC
> (“SGAS”). Please visit http://swapdisclosure.sgcib.com for important
> information regarding swap transactions with SOCIETE GENERALE.
> *************************************************************************
>
>
>
> _______________________________________________
> Toasters mailing list
> [hidden email]
> http://www.teaparty.net/mailman/listinfo/toasters
>


--
  Joel Krajden: Core Technologies Manager
  Engineering & Computer Science Concordia University
  EV-8.142  Tel: 514 848-2424 3052
  [hidden email]


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: qtree snapmirror inexplicably slow once a week

Joel Krajden

Check the etc/log/snapmirror file on the destination filer for clues.

J

On 12/14/2016 03:28 PM, Joel Krajden wrote:

> Are there any other snapmirror qtrees happening on the same volume on
> Mondays. If yes the locks on snapshots might be the problem.
>
> J
>
>
> On 12/13/2016 11:04 AM, BERNTSEN Basil wrote:
>> Hi folks,
>>
>>
>>
>> I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site
>> that's slated to be decommed, but for now, I need to keep the lights on.
>> There is a qtree snapmirror that, aside from Mondays, runs in about 3-4
>> hours. Mondays, it takes 2-4 days. The change rate on the source doesn't
>> appear to be any different. Also, this was working happily for years and
>> only started acting up lately. There are only 5 disks on the system, and
>> while they're at 100% now, they're also at 100% during a sample taken
>> while this context wasn't updating, and while it was updating in a way
>> that finished in 4 hours.
>>
>>
>>
>> So far, I've checked statit during the event and while idle, and I
>> couldn't see anything that stuck out as a massive difference, except
>> maybe name cache hits? I have 68% name cache hits during this transfer,
>> and 100% while idle.
>>
>>
>>
>> I don't see anything suspicious in the wafl scan status, and reallocate
>> status -v shows:
>>
>>
>>
>> Reallocation scans are on
>>
>> No reallocation status.
>>
>>
>>
>> CPU is idle, network has no errors. Does anyone have any suggestions?
>> What kinds of things should I be looking at?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Basil
>>
>>
>>
>> *************************************************************************
>> This message and any attachments (the "message") are confidential,
>> intended solely for the addressee(s), and may contain legally privileged
>> information.  Any unauthorized use or dissemination is prohibited.
>> E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any
>> of its subsidiaries or affiliates shall be liable for the message if
>> altered, changed or falsified. Please visit http://sgasdisclosure.com
>> for important information regarding SG Americas Securities, LLC
>> (“SGAS”). Please visit http://swapdisclosure.sgcib.com for important
>> information regarding swap transactions with SOCIETE GENERALE.
>> *************************************************************************
>>
>>
>>
>> _______________________________________________
>> Toasters mailing list
>> [hidden email]
>> http://www.teaparty.net/mailman/listinfo/toasters
>>
>
>


--
  Joel Krajden: Core Technologies Manager
  Engineering & Computer Science Concordia University
  EV-8.142  Tel: 514 848-2424 3052
  [hidden email]


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: qtree snapmirror inexplicably slow once a week

jordan slingerland-2
In reply to this post by BERNTSEN Basil
because the disks were at 100% while the snapmirror is slow and also 100% while the snapmirror finished in an acceptable time frame, does not mean you have a similar workload.  there is nowhere to go from there.
 You cant get many iops out of 5 disks, my first though would be to look at the clients, see which one is the hog and find out what it is doing. I suppose the first thing I would do is see what protocol is doing the i/o and go from there. 



sysstat -x 1 

might be useful for starters. 

--JMS

On Tue, Dec 13, 2016 at 11:04 AM, BERNTSEN Basil <[hidden email]> wrote:

Hi folks,

 

I'm at a loss. I have an unmaintained (fas2020) 7-mode HA pair in a site that's slated to be decommed, but for now, I need to keep the lights on. There is a qtree snapmirror that, aside from Mondays, runs in about 3-4 hours. Mondays, it takes 2-4 days. The change rate on the source doesn't appear to be any different. Also, this was working happily for years and only started acting up lately. There are only 5 disks on the system, and while they're at 100% now, they're also at 100% during a sample taken while this context wasn't updating, and while it was updating in a way that finished in 4 hours.

 

So far, I've checked statit during the event and while idle, and I couldn't see anything that stuck out as a massive difference, except maybe name cache hits? I have 68% name cache hits during this transfer, and 100% while idle.

 

I don't see anything suspicious in the wafl scan status, and reallocate status -v shows:

 

Reallocation scans are on

No reallocation status.

 

CPU is idle, network has no errors. Does anyone have any suggestions? What kinds of things should I be looking at?

 

Thanks,

 

Basil

 

************************************************************************* 
This message and any attachments (the "message") are confidential, intended solely for the addressee(s), and may contain legally privileged information.  Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. Please visit http://sgasdisclosure.com for important information regarding SG Americas Securities, LLC (“SGAS”). Please visit http://swapdisclosure.sgcib.com for important information regarding swap transactions with SOCIETE GENERALE.  
*************************************************************************


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters



_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Loading...