ONTAP9 NFSv4 + CentOS 6.8 does not work

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

ONTAP9 NFSv4 + CentOS 6.8 does not work

Momonth
Hi All,

We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I
discovered some freshly installed CentOS 6.8 hosts not being able to
mount file shares via NFS v4 protocol. The mount command just kind of
hangs. I verified it's not a connectivity issue, ie connections to the
filer's port 2049 are possible.

Also NFS v3 works just fine.

Did anyone experience anything like that?

Cheers,
Vladimir
_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ONTAP9 NFSv4 + CentOS 6.8 does not work

Tim McCarthy
Should check the IMT...
and that you should be on GA 6.8 and
Supported Kernel version:- kernel-2.6.32-642.el6.x86_64 Supported NFS-utils version:- nfs-utils-1.2.3-70.el6.x86_64 Supported libtirpc version:- libtirpc-0.2.1-11.el6.x86_64

This is for NFS4.1...which you may or may not be using....worth checking into.


--tmac

Tim McCarthy, Principal Consultant

Proud Member of the #NetAppATeam

I Blog at TMACsRack


On Wed, Nov 2, 2016 at 6:06 AM, Momonth <[hidden email]> wrote:
Hi All,

We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I
discovered some freshly installed CentOS 6.8 hosts not being able to
mount file shares via NFS v4 protocol. The mount command just kind of
hangs. I verified it's not a connectivity issue, ie connections to the
filer's port 2049 are possible.

Also NFS v3 works just fine.

Did anyone experience anything like that?

Cheers,
Vladimir
_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: ONTAP9 NFSv4 + CentOS 6.8 does not work

Parisi, Justin

That wouldn’t cause hangs though.

 

Official NetApp NFS support statement is that we support any client that follows the RFC specification. No custom kernels. RHEL/CentOS 6.8 should be fine.

 

I’d look into a getting a packet capture of the hang to see where they are occurring in the process.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of tmac
Sent: Wednesday, November 02, 2016 7:08 AM
To: Vladimir Zhigulin
Cc: [hidden email]
Subject: Re: ONTAP9 NFSv4 + CentOS 6.8 does not work

 

Should check the IMT...

and that you should be on GA 6.8 and

Supported Kernel version:- kernel-2.6.32-642.el6.x86_64 Supported NFS-utils version:- nfs-utils-1.2.3-70.el6.x86_64 Supported libtirpc version:- libtirpc-0.2.1-11.el6.x86_64

 

This is for NFS4.1...which you may or may not be using....worth checking into.

 


--tmac

 

Tim McCarthy, Principal Consultant

Proud Member of the #NetAppATeam

I Blog at TMACsRack

 

On Wed, Nov 2, 2016 at 6:06 AM, Momonth <[hidden email]> wrote:

Hi All,

We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I
discovered some freshly installed CentOS 6.8 hosts not being able to
mount file shares via NFS v4 protocol. The mount command just kind of
hangs. I verified it's not a connectivity issue, ie connections to the
filer's port 2049 are possible.

Also NFS v3 works just fine.

Did anyone experience anything like that?

Cheers,
Vladimir
_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters

 


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: ONTAP9 NFSv4 + CentOS 6.8 does not work

Fred Grieco
What version did you upgrade from? 


On Wed, Nov 2, 2016 at 11:02 AM, Parisi, Justin

That wouldn’t cause hangs though.

 

Official NetApp NFS support statement is that we support any client that follows the RFC specification. No custom kernels. RHEL/CentOS 6.8 should be fine.

 

I’d look into a getting a packet capture of the hang to see where they are occurring in the process.

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of tmac
Sent: Wednesday, November 02, 2016 7:08 AM
To: Vladimir Zhigulin
Cc: [hidden email]
Subject: Re: ONTAP9 NFSv4 + CentOS 6.8 does not work

 

Should check the IMT...

and that you should be on GA 6.8 and

Supported Kernel version:- kernel-2.6.32-642.el6.x86_64 Supported NFS-utils version:- nfs-utils-1.2.3-70.el6.x86_64 Supported libtirpc version:- libtirpc-0.2.1-11.el6.x86_64

 

This is for NFS4.1...which you may or may not be using....worth checking into.

 


--tmac

 

Tim McCarthy, Principal Consultant

Proud Member of the #NetAppATeam

I Blog at TMACsRack

 

On Wed, Nov 2, 2016 at 6:06 AM, Momonth <<a rel="nofollow" shape="rect" ymailto="mailto:momonth@gmail.com" target="_blank" href="javascript:return">momonth@...> wrote:

Hi All,

We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I
discovered some freshly installed CentOS 6.8 hosts not being able to
mount file shares via NFS v4 protocol. The mount command just kind of
hangs. I verified it's not a connectivity issue, ie connections to the
filer's port 2049 are possible.

Also NFS v3 works just fine.

Did anyone experience anything like that?

Cheers,
Vladimir
_______________________________________________
Toasters mailing list
<a rel="nofollow" shape="rect" ymailto="mailto:Toasters@teaparty.net" target="_blank" href="javascript:return">Toasters@...
http://www.teaparty.net/mailman/listinfo/toasters

 


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ONTAP9 NFSv4 + CentOS 6.8 does not work

Momonth
We upgraded from cDTO 8.3.2

Something to add: existing v4 clients (ie those that mounted the share
before ONTAP 9 upgrade) are just fine after the upgrade, only brand
new NFS clients are having issues.

On Wed, Nov 2, 2016 at 6:25 PM, Fred Grieco <[hidden email]> wrote:

> What version did you upgrade from?
>
> Sent from Yahoo Mail on Android
>
> On Wed, Nov 2, 2016 at 11:02 AM, Parisi, Justin
> <[hidden email]> wrote:
>
> That wouldn’t cause hangs though.
>
>
>
> Official NetApp NFS support statement is that we support any client that
> follows the RFC specification. No custom kernels. RHEL/CentOS 6.8 should be
> fine.
>
>
>
> I’d look into a getting a packet capture of the hang to see where they are
> occurring in the process.
>
>
>
> From: [hidden email] [mailto:[hidden email]]
> On Behalf Of tmac
> Sent: Wednesday, November 02, 2016 7:08 AM
> To: Vladimir Zhigulin
> Cc: [hidden email]
> Subject: Re: ONTAP9 NFSv4 + CentOS 6.8 does not work
>
>
>
> Should check the IMT...
>
> refers to
> http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1035160
>
> and that you should be on GA 6.8 and
>
> Supported Kernel version:- kernel-2.6.32-642.el6.x86_64 Supported NFS-utils
> version:- nfs-utils-1.2.3-70.el6.x86_64 Supported libtirpc version:-
> libtirpc-0.2.1-11.el6.x86_64
>
>
>
> This is for NFS4.1...which you may or may not be using....worth checking
> into.
>
>
>
>
> --tmac
>
>
>
> Tim McCarthy, Principal Consultant
>
> Proud Member of the #NetAppATeam
>
> I Blog at TMACsRack
>
>
>
> On Wed, Nov 2, 2016 at 6:06 AM, Momonth <[hidden email]> wrote:
>
> Hi All,
>
> We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I
> discovered some freshly installed CentOS 6.8 hosts not being able to
> mount file shares via NFS v4 protocol. The mount command just kind of
> hangs. I verified it's not a connectivity issue, ie connections to the
> filer's port 2049 are possible.
>
> Also NFS v3 works just fine.
>
> Did anyone experience anything like that?
>
> Cheers,
> Vladimir
> _______________________________________________
> Toasters mailing list
> [hidden email]
> http://www.teaparty.net/mailman/listinfo/toasters
>
>

_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ONTAP9 NFSv4 + CentOS 6.8 does not work

John Stoffel-5

Do new installs of older RHEL 6.x work with NFSv4 on the Ontap9 setup?
Just curious... and what does an 'strace' of the NFS process show?  Or
a tcpdump trace?  Can you share the options and full command line
you're using for the mounts?  Is your Authentication working properly
on both sides?

And have you disabled selinux as well?  That's always bit me in the
ass at times.

John



Momonth> We upgraded from cDTO 8.3.2
Momonth> Something to add: existing v4 clients (ie those that mounted the share
Momonth> before ONTAP 9 upgrade) are just fine after the upgrade, only brand
Momonth> new NFS clients are having issues.

Momonth> On Wed, Nov 2, 2016 at 6:25 PM, Fred Grieco <[hidden email]> wrote:

>> What version did you upgrade from?
>>
>> Sent from Yahoo Mail on Android
>>
>> On Wed, Nov 2, 2016 at 11:02 AM, Parisi, Justin
>> <[hidden email]> wrote:
>>
>> That wouldn’t cause hangs though.
>>
>>
>>
>> Official NetApp NFS support statement is that we support any client that
>> follows the RFC specification. No custom kernels. RHEL/CentOS 6.8 should be
>> fine.
>>
>>
>>
>> I’d look into a getting a packet capture of the hang to see where they are
>> occurring in the process.
>>
>>
>>
>> From: [hidden email] [mailto:[hidden email]]
>> On Behalf Of tmac
>> Sent: Wednesday, November 02, 2016 7:08 AM
>> To: Vladimir Zhigulin
>> Cc: [hidden email]
>> Subject: Re: ONTAP9 NFSv4 + CentOS 6.8 does not work
>>
>>
>>
>> Should check the IMT...
>>
>> refers to
>> http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1035160
>>
>> and that you should be on GA 6.8 and
>>
>> Supported Kernel version:- kernel-2.6.32-642.el6.x86_64 Supported NFS-utils
>> version:- nfs-utils-1.2.3-70.el6.x86_64 Supported libtirpc version:-
>> libtirpc-0.2.1-11.el6.x86_64
>>
>>
>>
>> This is for NFS4.1...which you may or may not be using....worth checking
>> into.
>>
>>
>>
>>
>> --tmac
>>
>>
>>
>> Tim McCarthy, Principal Consultant
>>
>> Proud Member of the #NetAppATeam
>>
>> I Blog at TMACsRack
>>
>>
>>
>> On Wed, Nov 2, 2016 at 6:06 AM, Momonth <[hidden email]> wrote:
>>
>> Hi All,
>>
>> We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I
>> discovered some freshly installed CentOS 6.8 hosts not being able to
>> mount file shares via NFS v4 protocol. The mount command just kind of
>> hangs. I verified it's not a connectivity issue, ie connections to the
>> filer's port 2049 are possible.
>>
>> Also NFS v3 works just fine.
>>
>> Did anyone experience anything like that?
>>
>> Cheers,
>> Vladimir
>> _______________________________________________
>> Toasters mailing list
>> [hidden email]
>> http://www.teaparty.net/mailman/listinfo/toasters
>>
>>

Momonth> _______________________________________________
Momonth> Toasters mailing list
Momonth> [hidden email]
Momonth> http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ONTAP9 NFSv4 + CentOS 6.8 does not work

Tim McCarthy
In reply to this post by Momonth
Are those clients set up the same...i.e the nfs domain is identical? Network/netmask is correct?


Per others, a packet trace from the client and the netapp may prove useful.


--tmac

Tim McCarthy, Principal Consultant

Proud Member of the #NetAppATeam

I Blog at TMACsRack




           RHCE6 110-107-141 

On Wed, Nov 2, 2016 at 3:37 PM, Momonth <[hidden email]> wrote:
We upgraded from cDTO 8.3.2

Something to add: existing v4 clients (ie those that mounted the share
before ONTAP 9 upgrade) are just fine after the upgrade, only brand
new NFS clients are having issues.

On Wed, Nov 2, 2016 at 6:25 PM, Fred Grieco <[hidden email]> wrote:
> What version did you upgrade from?
>
> Sent from Yahoo Mail on Android
>
> On Wed, Nov 2, 2016 at 11:02 AM, Parisi, Justin
> <[hidden email]> wrote:
>
> That wouldn’t cause hangs though.
>
>
>
> Official NetApp NFS support statement is that we support any client that
> follows the RFC specification. No custom kernels. RHEL/CentOS 6.8 should be
> fine.
>
>
>
> I’d look into a getting a packet capture of the hang to see where they are
> occurring in the process.
>
>
>
> From: [hidden email] [mailto:[hidden email]]
> On Behalf Of tmac
> Sent: Wednesday, November 02, 2016 7:08 AM
> To: Vladimir Zhigulin
> Cc: [hidden email]
> Subject: Re: ONTAP9 NFSv4 + CentOS 6.8 does not work
>
>
>
> Should check the IMT...
>
> refers to
> http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1035160
>
> and that you should be on GA 6.8 and
>
> Supported Kernel version:- kernel-2.6.32-642.el6.x86_64 Supported NFS-utils
> version:- nfs-utils-1.2.3-70.el6.x86_64 Supported libtirpc version:-
> libtirpc-0.2.1-11.el6.x86_64
>
>
>
> This is for NFS4.1...which you may or may not be using....worth checking
> into.
>
>
>
>
> --tmac
>
>
>
> Tim McCarthy, Principal Consultant
>
> Proud Member of the #NetAppATeam
>
> I Blog at TMACsRack
>
>
>
> On Wed, Nov 2, 2016 at 6:06 AM, Momonth <[hidden email]> wrote:
>
> Hi All,
>
> We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I
> discovered some freshly installed CentOS 6.8 hosts not being able to
> mount file shares via NFS v4 protocol. The mount command just kind of
> hangs. I verified it's not a connectivity issue, ie connections to the
> filer's port 2049 are possible.
>
> Also NFS v3 works just fine.
>
> Did anyone experience anything like that?
>
> Cheers,
> Vladimir
> _______________________________________________
> Toasters mailing list
> [hidden email]
> http://www.teaparty.net/mailman/listinfo/toasters
>
>


_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ONTAP9 NFSv4 + CentOS 6.8 does not work

Momonth
In reply to this post by John Stoffel-5
On Wed, Nov 2, 2016 at 10:49 PM, John Stoffel <[hidden email]> wrote:
>
> Do new installs of older RHEL 6.x work with NFSv4 on the Ontap9 setup?

I haven't tried anything else yet.

> Just curious... and what does an 'strace' of the NFS process show?  Or
> a tcpdump trace?  Can you share the options and full command line
> you're using for the mounts?  Is your Authentication working properly
> on both sides?

I'm going to collect tcpdump and strace.

We have provisioning system and puppetized configs, so all servers are
supposed to have the same settings etc.
However servers can land in different VLANs.

>
> And have you disabled selinux as well?  That's always bit me in the
> ass at times.
>

SELinux is disabled.
_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: ONTAP9 NFSv4 + CentOS 6.8 does not work

Steiner, Jeffrey
Just a thought - I ran into a similar problem once where what happened was I upgraded and ended up with a system enabled for 4.1 when previously only 4.0 was enabled. A few clients glitched because they detected the change and tried to connect with NFSv4.1. I had to explicitly force the mount to v4.0 only.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Momonth
Sent: Thursday, November 03, 2016 12:51 PM
To: John Stoffel <[hidden email]>
Cc: [hidden email]
Subject: Re: ONTAP9 NFSv4 + CentOS 6.8 does not work

On Wed, Nov 2, 2016 at 10:49 PM, John Stoffel <[hidden email]> wrote:
>
> Do new installs of older RHEL 6.x work with NFSv4 on the Ontap9 setup?

I haven't tried anything else yet.

> Just curious... and what does an 'strace' of the NFS process show?  Or
> a tcpdump trace?  Can you share the options and full command line
> you're using for the mounts?  Is your Authentication working properly
> on both sides?

I'm going to collect tcpdump and strace.

We have provisioning system and puppetized configs, so all servers are supposed to have the same settings etc.
However servers can land in different VLANs.

>
> And have you disabled selinux as well?  That's always bit me in the
> ass at times.
>

SELinux is disabled.
_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ONTAP9 NFSv4 + CentOS 6.8 does not work

Michael Bergman
I would like to just suggest... I'm aware that some ppl migh want to start a
long debate about this (which I will then just have to dodge)

If you really really want to deploy v4.x in an ubiquitous manner, scale it
up/out a lot so to speak for some good reason you have thought trough really
well.

Do it with NFS clients that can talk 4.1 (i.e. minorversion=1.)  NFS clients
which do not understand v4.1 but only 4.0, deny them any v4 access, this can
be done by simply disabling v4.0 in the vservers configs, but keep 4.1
enabled. Just make those older NFS clients fall back to v3 and all will
work, the users won't notice any difference and there's no real loss of
anything that you will miss

(AUTH_GSS, aka RPCSEC_GSS, with sec=krb5{,i,p} is supported and works just
fine with NFSv3 in ONTAP and in almost all if not all, wide-spread Linux
distros as well as Oracle Solaris. Don't know about *BSD).

/M

On 2016-11-03 19:41, Steiner, Jeffrey wrote:
> Just a thought - I ran into a similar problem once where what happened
> was I upgraded and ended up with a system enabled for 4.1 when previously
> only 4.0 was enabled. A few clients glitched because they detected the
> change and tried to connect with NFSv4.1. I had to explicitly force the
> mount to v4.0 only.

_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ONTAP9 NFSv4 + CentOS 6.8 does not work

Momonth
In reply to this post by Momonth
I'm not 100% sure what has happened, most likely in-house "server
provisioning tool" failed in such a way that some NFS related packages
were not installed. As after re-installing the server in question from
scratch everything runs fine. Unfortunately I was not able to collect
enough debug info to have better conclusions.

On Wed, Nov 2, 2016 at 11:06 AM, Momonth <[hidden email]> wrote:

> Hi All,
>
> We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I
> discovered some freshly installed CentOS 6.8 hosts not being able to
> mount file shares via NFS v4 protocol. The mount command just kind of
> hangs. I verified it's not a connectivity issue, ie connections to the
> filer's port 2049 are possible.
>
> Also NFS v3 works just fine.
>
> Did anyone experience anything like that?
>
> Cheers,
> Vladimir
_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Fwd: ONTAP9 NFSv4 + CentOS 6.8 does not work

Momonth
This issues re-occurred on already running (ie it's not freshly
provisioned servers) CentOS 6.6, and also on CentOS 7.2.1511. What I
observed is that NFS clients could not mount the share on ONTAP 9 SVM
after "lif migrate .." command.

This time I was able to collect tcpdump while running mount command.
It was as follows:

1. Client sends (decoded by wireshark):

Remote Procedure Call, Type:Call XID:0x9fc37eb7
Network File System, Ops(3): SETCLIENTID_CONFIRM, PUTROOTFH, GETATTR
    [Program Version: 4]
    [V4 Procedure: COMPOUND (1)]
    Tag: <EMPTY>
    minorversion: 0
    Operations (count: 3): SETCLIENTID_CONFIRM, PUTROOTFH, GETATTR
        Opcode: SETCLIENTID_CONFIRM (36)
            clientid: 0x36c1330000000076
            verifier: 0x0000000000000000
        Opcode: PUTROOTFH (24)
        Opcode: GETATTR (9)
            Attr mask: 0x00000400 (Lease_Time)
                reqd_attr: Lease_Time (10)
    [Main Opcode: SETCLIENTID_CONFIRM (36)]

2.  SVM responds with:

Network File System, Ops(2): SETCLIENTID_CONFIRM PUTROOTFH(NFS4ERR_DELAY)
    [Program Version: 4]
    [V4 Procedure: COMPOUND (1)]
    Status: NFS4ERR_DELAY (10008)
    Tag: <EMPTY>
        length: 0
        contents: <EMPTY>
    Operations (count: 2)
        Opcode: SETCLIENTID_CONFIRM (36)
            Status: NFS4_OK (0)
        Opcode: PUTROOTFH (24)
            Status: NFS4ERR_DELAY (10008)
    [Main Opcode: SETCLIENTID_CONFIRM (36)]

IMO, it was in an endless loop until I CTRL+C'ed the mount command.

On Tue, Nov 8, 2016 at 5:23 PM, Momonth <[hidden email]> wrote:

> I'm not 100% sure what has happened, most likely in-house "server
> provisioning tool" failed in such a way that some NFS related packages
> were not installed. As after re-installing the server in question from
> scratch everything runs fine. Unfortunately I was not able to collect
> enough debug info to have better conclusions.
>
> On Wed, Nov 2, 2016 at 11:06 AM, Momonth <[hidden email]> wrote:
>> Hi All,
>>
>> We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I
>> discovered some freshly installed CentOS 6.8 hosts not being able to
>> mount file shares via NFS v4 protocol. The mount command just kind of
>> hangs. I verified it's not a connectivity issue, ie connections to the
>> filer's port 2049 are possible.
>>
>> Also NFS v3 works just fine.
>>
>> Did anyone experience anything like that?
>>
>> Cheers,
>> Vladimir
_______________________________________________
Toasters mailing list
[hidden email]
http://www.teaparty.net/mailman/listinfo/toasters
Loading...