FAS Research Computing - h-nfs15-p inaccessible – Incident details

All systems operational

Status page for the Harvard FAS Research Computing cluster and other resources.

Cluster Utilization (VPN and FASRC login required): Cannon | FASSE


Please scroll down to see details on any Incidents or maintenance notices.
Monthly maintenance occurs on the first Monday of the month (except holidays).

GETTING HELP
Documentation: https://docs.rc.fas.harvard.edu | Account Portal https://portal.rc.fas.harvard.edu
Email: rchelp@rc.fas.harvard.edu | Support Hours

The colors shown in the bars below were chosen to increase visibility for color-blind visitors.
For higher contrast, switch to light mode at the bottom of this page if the background is dark and colors are muted.

h-nfs15-p inaccessible

Resolved
Partial outage
Started 6 days agoLasted about 20 hours

Affected

Storage

Partial outage from 7:00 PM to 3:05 PM

Boston Tier 2 NFS (new)

Partial outage from 7:00 PM to 3:05 PM

Updates
  • Resolved
    UTC
    Resolved

    XFS journal recovery has completed, and kou_lab is remounted. The lab should verify the filesystem and any data that may have been written after 11:00 AM on 8/11/26 (Tuesday).

    In simple terms, too many open files caused the server to become unresponsive, which triggered an automatic reset. After the reboot, XFS required an extended recovery period before the filesystem could be mounted again.

    This does not mean the lab caused the issue. The file descriptor limit is a system-wide kernel resource shared by all filesystems and workloads on the server, so any share or process running a large job could have contributed. kou_lab was essentially collateral damage from machine-wide resource exhaustion, a “noisy neighbor” effect on a shared system.

  • Identified
    UTC
    Identified

    Most shares on h-nfs15-p are back up and accessible at this time.

    kou_lab storage is still down.

    We are continuing to work on a fix for this incident. Updates to come.

  • Investigating
    UTC
    Investigating

    Storage shares on h-nfs15-p may be inaccessible at this time. Affected labs include:

    • bellono_lab

    • debivort_lab

    • shakhnovich_lab

    • doyle_lab

    • holbrook_lab

    • kou_lab

    • koutrakis_lab

    • brennan_lab

    We are currently investigating this incident. We will continue to update as we know more

For issues not shown here, please contact FASRC via
https://portal.rc.fas.harvard.edu or email rchelp@rc.fas.harvard.edu