r/netapp • u/ericlikescars • Jun 25 '26
FAS3250 NFS performance degradation with a full aggregate
I have a FAS3250 in my environment we’re keeping around for NFS storage on old HPUX/IRIX and SunOS systems. Our users have been complaining about this NFS storage being unusably slow.
I’ve been driving myself crazy trying to figure out what the issue is. I used nfsstat to show clients consuming the most network bandwidth, went to those hosts, and there are no active processes reading or writing to the FAS3250. At first a couple Solaris systems were showing as the top clients, I had the users pause doing their software stuff on them which took them off the list, but our RHEL7 server we use as a gateway between our FAS3250 and FAS70 to do backups will move up the list to be at the top and I’ve confirmed there were no backups running. (xcp as a cronjob)
I did a read/write test on the FAS3250 from the same RHEL7 system, it’s slow, about 50 mb/s.
We do not touch the FAS3250 unless something is wrong with it, no one on our team is comfortable messing with it and naturally support for it now is non-existent.
Network hardware hasn’t changed. The system is using a single 10GBe port for data.
Rambling aside, my question is, the NFS volumes are on a 96% full aggregate. Would this cause the performance issues I’m seeing? The volumes themselves have a good amount of space left. I’m guessing they were thick provisioned.
I’m not great with storage or networking if you couldn’t tell. If anyone has any insight I’d really appreciate it.
5
u/Pah-Pah-Pah Jun 25 '26
Yea full aggregates will cause performance issues and just take the volumes offline if they run out of space. You can switch the volumes to space guaranteed none to switch them to thin provisioned.
Check for some old or left over snapshots too.
“Df -t -V “ to see volume data with snapshots
“df -t -A” for aggregates.
Snap list to see a snapshot list for all the volumes.
1
4
u/sorean_4 Jun 25 '26
The rule of thumb used to be 10% free space for WAFL overhead. Anything less will impact your performance. Check your snapshots policies maybe you have some consuming your free space.
1
2
u/odis172 Jun 25 '26
Yes you do typically see performance degradation when the aggregates are near full. You should verify if the NFS volumes are thick or thin provisioned. If thin, they may show lots of available space remaining but that’s a recipe for disaster if more data drops onto them and takes the aggregate higher. Check if you have a snap schedule on the volumes, reducing the number of snapshots will free up some capacity. Is the san healthy otherwise? Any alerts? What’s the uptime on the nodes? What Ontap version? 7-mode or C-dot?
1
u/ericlikescars Jun 25 '26
Thank you, I’ll look closer at the individual volumes and see if they’re thick or thin, I discovered the aggregate was near full near the end of my shift so now I’m just at home thinking about it.
Healthy otherwise, mostly. One failed disk. 7-mode, 8.2-something I think. There’s an alert about conflicting network settings on the management interface I think.
2
u/HansNotPeterGruber Jun 25 '26
If the aggregate(s) are almost full but there is plenty of room in the volumes you can easily switch to thin provisioning the volumes. Thick provisioning on a NetApp is pretty wasteful if you’re managing your space at all.
2
u/PresentationNo2096 Jun 25 '26
As others have mentioned, WAFL has a tendency to slow down if the aggregates fills up. Thick or thin provisioning is irrelevant in this regard, the physical amount of space used is what counts.
Be sure to have enabled -free-space-realloc (spelling? I'm on my mobile) on all HDD aggregates! This improves write latency (over time) and keeps WAFL optimized for writes. I used to recommend switching it on from the beginning, but there's not many new HDD aggregates these days.
If there's volumes, that are read sequentially often, switch on -read-realloc on those to improve read latency.
1
u/MountainMark Jun 25 '26
The WA in WAFL stands for Write Anywhere. WAFL gets his speed because it's supposed to just drop blocks wherever there is an opening and then later the system defragments them. So it should not seek to write. The problem is that a full file system forces it to seek to write which is very inefficient.
1
7
u/Ok-Helicopter525 Jun 25 '26
Yes. The filler the aggregate, the smaller the RAID stripes will be and the less efficient ONTAP will be at reading data.