Write operations using NFS v2?

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Write operations using NFS v2?

Ragnar Lonn
Hello,

Thought I'd try out this list in the hope of sharing some info
with others as opposed to bugging NetApp support only for my own
sake ;)     (but I'll CC them at least)

I'm setting up an F540 as a newsspool NFS server. That is, I will
store Usenet news articles on it. A handful of machines will
mount the filesystem read-only and these machines will accept
user newsreader connections. Then I will have one single
machine that mount the filesystem read/write - it will be the
machine that stores incoming articles and removes old ones
from the filer.

Now, for news I've got the advice to use NFS v2 because clients
using v3 issue the readdir+ operation unique to v3 and this
operation does some extra stuff (as opposed to v2's readdir)
that most applications take advantage of but news software
(INN nnrpd) has no use of the extra data returned by readdir+
and this makes it a performance loss to use v3 in this
particular case.

But... I've read that NFS writes are much faster in NFS v3
than v2. This, of course, leads me to my question (ta-daa) :

Actually, I'll make it two questions. If the anser to the
first one is no, the second one is not so interesting anymore ;)

1. Are write operations slower on a NetApp server if you use
   NFS v2 compared to when you're using NFS v3?
   (and if so, about how much does it differ?)

2. (if the answer to the above question was 'yes')
   Could I leave the option "nfs.v3.enable" ON, on the NetApp
   and mount the filesystem from the reader clients using e.g
   "mount -o vers=2,proto=udp" (can't remember the exact syntax)
   while at the same time, from the client that is supposed to
   be doing a lot of *writing* to the NetApp, mount using
   version 3 to speed up the writes?  This machine won't be running
   nnrpd so it won't be doing any readdir+ in vain.

Regards,

  /Ragnar, Algonet




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Write operations using NFS v2?

Karl Swartz
> Now, for news I've got the advice to use NFS v2 because clients
> using v3 issue the readdir+ operation unique to v3 and this
> operation does some extra stuff (as opposed to v2's readdir)
> that most applications take advantage of but news software
> (INN nnrpd) has no use of the extra data returned by readdir+
> and this makes it a performance loss to use v3 in this
> particular case.

Right.  READDIR+ effectively does a stat() on each file that it
returns, and that data is not used by the netnews software.

> But... I've read that NFS writes are much faster in NFS v3
> than v2. This, of course, leads me to my question (ta-daa) :

That's news to me, unless you're referring to the ability ot NFSv3
to use larger block sizes.  For netnews, that's not likely to be of
much benefit since most articles are very small.

> 1. Are write operations slower on a NetApp server if you use
>    NFS v2 compared to when you're using NFS v3?
>    (and if so, about how much does it differ?)

Except for any block size effects, the performance should be nearly
identical.

> 2. (if the answer to the above question was 'yes')
>    Could I leave the option "nfs.v3.enable" ON, on the NetApp
>    and mount the filesystem from the reader clients using e.g
>    "mount -o vers=2,proto=udp" (can't remember the exact syntax)
>    while at the same time, from the client that is supposed to
>    be doing a lot of *writing* to the NetApp, mount using
>    version 3 to speed up the writes?  This machine won't be running
>    nnrpd so it won't be doing any readdir+ in vain.

Unfortunately, it's the "feeder" machine (the one which is writing to
the filer) that gets hit the hardest by using NFSv3.  One operation
that is hit particularly hard is renumbering the active file, though
there are others that behave similarly.



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Write operations using NFS v2?

Guy Harris
> > But... I've read that NFS writes are much faster in NFS v3
> > than v2. This, of course, leads me to my question (ta-daa) :
>
> That's news to me, unless you're referring to the ability ot NFSv3
> to use larger block sizes.

He could also be referring to asynchronous writes.  We *currently* don't
do asynchronous writes; if a client asks us to do an async write, we do
it synchronously - i.e., we log it to NVRAM - and return an indication
in the reply that it was done synchronously.  (We may do them
asynchronously at some point.)


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Write operations using NFS v2?

Ragnar Lonn
In reply to this post by Karl Swartz
On Thu, 13 Mar 1997, Karl Swartz wrote:

> > But... I've read that NFS writes are much faster in NFS v3
> > than v2. This, of course, leads me to my question (ta-daa) :
>
> That's news to me, unless you're referring to the ability ot NFSv3
> to use larger block sizes.  For netnews, that's not likely to be of
> much benefit since most articles are very small.
>

Maybe I should add that I read this in a book about performance
tuning for Solaris systems ;)  But it did look as if this write
performance improvement was connected to the version of the
NFS protocol. Quote from 'Configuration and capacity planning for
Solaris servers' by Brian Wong :

> In response to a variety of technical requests from customers,
> the NFS protocol was revised in 1993, with the resulting protocol
> being designated Version 3. The new protocol is similar to its
> predecessor, but it makes improvements in several key areas.
> In particular, the new protcol implements the following changes:
>
> * It permits write operations to be performed much more quickly
>   through the use of a two-phase commit protocol, while still
>   maintaining the server's stateless view of the protocol.
>
 [ some more features cut out ]
>
> Naturally, both the client and server systems must implement these
> features; since many hosts do not, some of the new features are
> optional and are negotiated between client and server when a file
> system is mounted by the client"

Is this not applicable to NetApp servers?

> Unfortunately, it's the "feeder" machine (the one which is writing to
> the filer) that gets hit the hardest by using NFSv3.  One operation
> that is hit particularly hard is renumbering the active file, though
> there are others that behave similarly.
>

But INND doesn't scan directories very much does it?  I can imagine
that the renumber operation would suffer but it doesn't do renumber
all that frequently. Isn't it nnrpd that accounts for most of the
readdir+ performance waste when using NFS v3?

Regards,

  /Ragnar, Algonet



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Write operations using NFS v2?

Brian Pawlowski
> > In response to a variety of technical requests from customers,

Uh-huh. :-)

> > the NFS protocol was revised in 1993, with the resulting protocol
> > being designated Version 3. The new protocol is similar to its
> > predecessor, but it makes improvements in several key areas.
> > In particular, the new protcol implements the following changes:
> >
> > * It permits write operations to be performed much more quickly
> >   through the use of a two-phase commit protocol, while still
> >   maintaining the server's stateless view of the protocol.
> >
>  [ some more features cut out ]
> >
> > Naturally, both the client and server systems must implement these
> > features; since many hosts do not, some of the new features are
> > optional and are negotiated between client and server when a file
> > system is mounted by the client"
>
> Is this not applicable to NetApp servers?

Well...

The async write of V3 had the most impact on non-HW accelerated NFS servers.
The NetApp box uses NVRAM to support a logging scheme that make writes
on NFS V2 and V3 faster.

Also, the NVRAM accelerates other modifying operations (create and remove, etc.)
that are not addressed in V3.


beepy


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Write operations using NFS v2?

Karl Swartz
In reply to this post by Ragnar Lonn
> > > But... I've read that NFS writes are much faster in NFS v3
> > > than v2. This, of course, leads me to my question (ta-daa) :
> >
> > That's news to me, unless you're referring to the ability ot NFSv3
> > to use larger block sizes.  For netnews, that's not likely to be of
> > much benefit since most articles are very small.
...

> Quote from 'Configuration and capacity planning for
> Solaris servers' by Brian Wong :
...
> > In particular, the new protcol implements the following changes:
> >
> > * It permits write operations to be performed much more quickly
> >   through the use of a two-phase commit protocol, while still
> >   maintaining the server's stateless view of the protocol.

Asynchronous writes, as Guy guessed.  He and Beepy noted that NetApp
does not currently implement async writes (we accept them and note
in the response that they were done synchronously) and that, with our
NVRAM, they wouldn't provide a substantial benefit as they can with
general purpose servers having no special hardware support.

Beyond that, "NFS Version 3 Design and Implementation" (from Summer
'94 Usenix) notes that async writes "are most effective for large
files."  Netnews mostly generates lots of little files, so even if
async writes had much value on a filer, they wouldn't have much
effect for netnews.

> Is this not applicable to NetApp servers?

Bottom line: no.

> > Unfortunately, it's the "feeder" machine (the one which is writing to
> > the filer) that gets hit the hardest by using NFSv3.  One operation
> > that is hit particularly hard is renumbering the active file, though
> > there are others that behave similarly.
>
> But INND doesn't scan directories very much does it?

No, it doesn't, but other things that run on the same machine do.

> I can imagine that the renumber operation would suffer but it
> doesn't do renumber all that frequently.

In the case of renumbering the active file, we saw one site drop from
12 hours to a few tens of minutes.  Still not too horrible if you only
run it once per night, except that it chews up a *lot* of resources on
the filer and can have a severe impact on useful work.

> Isn't it nnrpd that accounts for most of the readdir+ performance
> waste when using NFS v3?

*Any* operation than triggers READDIR+ may severely impact performance.
It only happens from nnrpd when you change or list a group other than
the one you've most recently looked at (or it's been a while).  Since
that's not something users do that frequently, "most" of the waste
depends on how many users you have.  Active file renumbering hammers
at it continuously until it's done, so unless you have a pretty large
number of users, I doubt nnrpd would account for "most" of the hit.

 -- Karl


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Write operations using NFS v2?

Ragnar Lonn
On Fri, 14 Mar 1997, Karl Swartz wrote:

> > > Unfortunately, it's the "feeder" machine (the one which is writing to
> > > the filer) that gets hit the hardest by using NFSv3.  One operation
> > > that is hit particularly hard is renumbering the active file, though
> > > there are others that behave similarly.
> >
> > But INND doesn't scan directories very much does it?
>
> No, it doesn't, but other things that run on the same machine do.
>

Hmm, but I still haven't a clue about *what* programs generate
the readdir+ requests?  If not nnrpd and not innd, then what's
left?  On a newsfeed machine I run only those programs plus
innfeed or possibly innxmit and I doubt that those scan
directories a lot (or do they?).

> depends on how many users you have.  Active file renumbering hammers
> at it continuously until it's done, so unless you have a pretty large
> number of users, I doubt nnrpd would account for "most" of the hit.
>

Hmm.... would it be possible to use two mount points for the same
filesystem where one is mounted using NFS v2 and one using NFS v3
and then let ones own software choose whether to do an operation
using v2 or v3 depending on what path it tries to open?

Maybe this doesn't matter much if there's no performance advantage
in general to using v3 instead of v2 of course... Then there's
no reason for me to bother about v3 at all as I'm only going to
use the NetApp as a newsspool host and nothing else.

Regards,

  /Ragnar, Algonet



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Write operations using NFS v2?

sthaug@nethelp.no
> Maybe this doesn't matter much if there's no performance advantage
> in general to using v3 instead of v2 of course... Then there's
> no reason for me to bother about v3 at all as I'm only going to
> use the NetApp as a newsspool host and nothing else.

I'm going to turn off v3 on our NetApp box tonight, just to see how
much of a difference it makes.

Another thing I've been wondering about is options.minra. I find the
documentation a bit thin, shall we say. If the 'aggressive read ahead'
mentioned in the documentation is *per file*, it's probably a good
thing for a NetApp used as a News spool - since you normally retrieve
a whole article at a time. If the read ahead is based on something
other than file, I'm not so sure.

Anybody played with minra on a NetApp news spool?

Steinar Haug, Nethelp consulting, sthaug at nethelp.no


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Write operations using NFS v2?

Karl Swartz
In reply to this post by Ragnar Lonn
> > > > Unfortunately, it's the "feeder" machine (the one which is writing to
> > > > the filer) that gets hit the hardest by using NFSv3.  One operation
> > > > that is hit particularly hard is renumbering the active file, though
> > > > there are others that behave similarly.
> > >
> > > But INND doesn't scan directories very much does it?
> >
> > No, it doesn't, but other things that run on the same machine do.
>
> Hmm, but I still haven't a clue about *what* programs generate
> the readdir+ requests?  If not nnrpd and not innd, then what's
> left?  On a newsfeed machine I run only those programs plus
> innfeed or possibly innxmit and I doubt that those scan
> directories a lot (or do they?).

Most people run expire and fastrm on their newsfeed machines.  If you
choose not to, we'll be happy to sell you more disk!  :-)  There's also
expireover, and whatever other grot gets run from news.daily.

I was mistaken about the active file renumbering, though, which is part
of innd and not a separate executable.

> > depends on how many users you have.  Active file renumbering hammers
> > at it continuously until it's done, so unless you have a pretty large
> > number of users, I doubt nnrpd would account for "most" of the hit.
>
> Hmm.... would it be possible to use two mount points for the same
> filesystem where one is mounted using NFS v2 and one using NFS v3
> and then let ones own software choose whether to do an operation
> using v2 or v3 depending on what path it tries to open?

With active file renumbering being part of innd, no, not without some
hacking on the code.  Even for the separate executables, keeping the
two straight might prove difficult.  Why, though?  For news, at least,
NFS v3 has little, if any, benefit, and can badly hurt.

 -- Karl


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Write operations using NFS v2?

Karl Swartz
In reply to this post by sthaug@nethelp.no
> Another thing I've been wondering about is options.minra. I find the
> documentation a bit thin, shall we say. If the 'aggressive read ahead'
> mentioned in the documentation is *per file*, it's probably a good
> thing for a NetApp used as a News spool - since you normally retrieve
> a whole article at a time.

It depends.  We're working on more complete documentation on this and
other issues for netnews, but the quick answer is as follows:

    Look at the "cache age" field in the output of sysstat on the
    filer during a time of heavy load.  If it's not close to zero,
    fiddling with minra probably won't help and may hurt.

    If the cache age *is* close to zero, do you have lots of users
    accessing news via slow connections (e.g., modems) reading lots
    of large articles (e.g., alt.binaries.*)?  If yes, and you can't
    add any more memory, there's a good chance you'll benefit from

        options minra on

    Otherwise, it probably won't help, but you may want to give it a
    try anyway.

Aggressive read ahead is per file.  The problem is that very slow
users (the connections, not the humans, though some of them may be
slow too ;-)) will trigger the read ahead during sequential reads,
but if there's enough memory contention, all that work will be for
naught because the data won't actually be requested until after it
has been tossed out of the cache in favor of more recent requests.
Pushed hard enough, the CPU will max out as it busily thrashes
memory, doing little useful work in the process.

 -- Karl


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Write operations using NFS v2?

Ragnar Lonn
In reply to this post by Karl Swartz
On Mon, 17 Mar 1997, Karl Swartz wrote:

> Most people run expire and fastrm on their newsfeed machines.  If you
> choose not to, we'll be happy to sell you more disk!  :-)  There's also
> expireover, and whatever other grot gets run from news.daily.
>

I bet you would. Alas, we're not Dejanews ;)

> > Hmm.... would it be possible to use two mount points for the same
> > filesystem where one is mounted using NFS v2 and one using NFS v3
> > and then let ones own software choose whether to do an operation
> > using v2 or v3 depending on what path it tries to open?
>
> With active file renumbering being part of innd, no, not without some
> hacking on the code.  Even for the separate executables, keeping the
> two straight might prove difficult.  Why, though?  For news, at least,
> NFS v3 has little, if any, benefit, and can badly hurt.
>

Yeah, the presumption was that writes were quicker in v3 but as
they aren't, there is no need to use v3 at all. I'll just give it
a miss I think!

Thanks for straightening things out.

Regards,

  /Ragnar, Algonet



Loading...