FAS Research Computing - Notice history

Globus Data Transfer experiencing major outage

Status page for the Harvard FAS Research Computing cluster and other resources.

Cluster Utilization (VPN and FASRC login required): Cannon | FASSE


Please scroll down to see details on any Incidents or maintenance notices.
Monthly maintenance occurs on the first Monday of the month (except holidays).

GETTING HELP
Documentation: https://docs.rc.fas.harvard.edu | Account Portal https://portal.rc.fas.harvard.edu
Email: rchelp@rc.fas.harvard.edu | Support Hours

The colors shown in the bars below were chosen to increase visibility for color-blind visitors.
For higher contrast, switch to light mode at the bottom of this page if the background is dark and colors are muted.
Operational

SLURM Scheduler - Cannon - Operational

Cannon Compute Cluster (Holyoke) - Operational

Boston Compute Nodes - Operational

GPU nodes (Holyoke) - Operational

seas_compute - Operational

Operational

SLURM Scheduler - FASSE - Operational

FASSE Compute Cluster (Holyoke) - Operational

Operational

Kempner Cluster CPU - Operational

Kempner Cluster GPU - Operational

Operational

FASSE login nodes - Operational

Operational

Cannon Open OnDemand - Operational

FASSE Open OnDemand - Operational

Major outage

Netscratch (Global Scratch) - Operational

Home Directory Storage - Boston - Operational

Tape - (Tier 3) - Operational

Holylabs - Operational

Isilon Storage Holyoke (Tier 1) - Operational

Holystore01 (Tier 0) - Operational

HolyLFS04 (Tier 0) - Operational

HolyLFS05 (Tier 0) - Operational

HolyLFS06 (Tier 0) - Operational

Holyoke Tier 2 NFS - Operational

Holyoke Specialty Storage - Operational

holECS - Operational

Isilon Storage Boston (Tier 1) - Operational

BosLFS02 (Tier 0) - Operational

Boston Tier 2 NFS - Operational

CEPH Storage Boston (Tier 2) - Operational

Boston Specialty Storage - Operational

bosECS - Operational

Samba Cluster - Operational

Globus Data Transfer - Major outage

Notice history

View current status

Nov 2026

CEPH shares down
ResolvedMajor outage2 hours 6 minutes
  • ResolvedUTC

    The networking change has been addressed, and CEPH shares are available from the cluster.

    This incident has been resolved.

  • InvestigatingUTC

    CEPH shares are currently unavailable due to networking change.

    We are currently investigating this incident.

Globus authentication/log-in
IdentifiedMajor outage

  • We are renewing a certificate to allow authentication to FASRC collections (Harvard FAS RC Boston/Harvard FAS RC Holyoke) on Globus

    UTC
OOD apps down
ResolvedPartial outage23 hours 18 minutes
  • ResolvedUTC

    The underlying issue has been fixed, and OOD apps should launch normally now.

    This incident has been resolved.

  • IdentifiedUTC

    The bug causing this has been identified, and we are continuing to work on a fix for this incident.

    Although the browser NoVNC connection may not connect, you can use the session credentials in the VNC Desktop Client tab for the session to connect to the running session from a VNC client like TurboVNC.

  • InvestigatingUTC

    Several OOD apps on both Cannon and FASSE may display a "Failed to connect to server" error after the job has started. They include:

    • Remote Desktop

    • Matlab

    • Stata

    • SAS

    • Knime

    We are currently investigating this incident.

NOV2
Monthly maintenance November 2nd, 2026 8am-6pm.
CompletedMaintenance10 hours
  • CompletedNovember 02, 2026 at 11:00 PMUTC
    Maintenance has completed successfully
  • In progressNovember 02, 2026 at 1:00 PMUTC
    Maintenance is now in progress
  • PlannedOctober 21, 2026 at 3:21 PMUTC

    Our maintenance tasks should be completed between 8am-6pm.

    NOTICES:

    • New: Netscratch retention will be reduced to 60 days starting with the December maintenance. More information will be announced soon.

    • Home directory permissions have been reset to allow only owner access per FASRC and Harvard policy. Data in home directories cannot be shared among users. Shared data should be stored in lab storage. Please reach out to rchelp@rc.fas.harvard.edu with questions or concerns.

    • Python/conda/mamba/venv security issue: A security problem was found with some of the certificates that software uses to confirm it is talking to the real website. Browsers have already been fixed, but Python tools like conda and mamba keep their own copy of these certificates in two packages, ca-certificates and certifi. Please update both to version 2026.10.19 or newer. We know of no attacks against these tools, but updating is a simple precaution. For more details, see the eMudhra Security Update.

    • Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at https://www.rc.fas.harvard.edu/upcoming-training/

    • Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at https://status.rc.fas.harvard.edu/ (click Get Updates for options).

    MAINTENANCE TASKS

    Cannon cluster will be paused during this maintenance?: YES
    FASSE cluster will be paused during this maintenance?: YES

    • Holyoke/MGHPCC rack 8a26 power work - 8am-6pm

      • Audience: All Cannon cluster users

      • Impact: The power distribution unit in the holy8a26 rack needs to be replaced, which necessitates us powering down the nodes in the rack. This will take place between 8am and 6pm on November 2nd. This is a partial outage for most partitions, jobs will adjust around the down nodes. For partitions suffering a full outage, please use other resources for the duration.

      • Public partitions:

        • test
          sapphire
          bigmem

      • Group or private partitions:

        • hsph
          itc_cluster

        • eddy

          janson_sapphire

          yao

          yao_alphatns

          yao_gpu

    • Slurm 26.05.5 Upgrade

      • Audience: All cluster users (Cannon and FASSE)

      • Impact: Jobs will be paused and resumed after maintenance

    • Login node reboots

      • Audience: All login nodes

      • Impact: Login nodes will be unavailable until after maintenance

    • OOD/Open OnDemand reboots

      • Audience: All OOD users

      • Impact: OOD will be unavailable until after maintenance

    • Netscratch 90-day retention cleanup

      • Audience: All netscratch users

      • Update: Due to low space on netscratch, please note that retention cleanup will be run more frequently. In addition, the retention period will be reduced to 60 days next month.

      • Impact: Files older than 90 days will be removed per our scratch policy. Please note that this cleanup can happen at any time, not just during maintenance.

Oct 2026

boslfs02 cutover
ResolvedMajor outage46 hours 48 minutes
  • ResolvedUTC

    boslfs02 cutover has been completed.

    This incident has been resolved.

  • InvestigatingUTC

    As per email communications with affected groups, the cutover from boslfs02 to /n/lab_storage is ongoing from October 27-October 29. All storage folders are set to read-only and all running jobs have been stopped.

    Once the cutover process is complete, affected users will receive an email confirming their new storage is ready for use.

boslogin07 slowness / high load
Resolved17 minutes
  • ResolvedUTC

    boslogin07 has been rebooted and is responsive. This incident has been resolved.

  • InvestigatingUTC

    boslogin07 is experiencing long delays on login we will reboot the system at 10:30

holystore01 issues
ResolvedMajor outage30 hours 4 minutes
  • ResolvedUTC
    This incident has been resolved.
  • MonitoringUTC

    holystore01 is back up.

    Draining nodes are being addressed and will be put back into service at the earliest.

  • InvestigatingUTC

    holystore01 is We are currently experiencing performance issues that have impacted some compute nodes, causing them to drain, investigating this incident.

OCT24
Globus scheduled maintenance
CompletedMaintenance1 hour
  • CompletedOctober 24, 2026 at 3:30 PMUTC
    Maintenance has completed successfully
  • In progressOctober 24, 2026 at 2:30 PMUTC
    Maintenance is now in progress
  • PlannedOctober 19, 2026 at 3:30 PMUTC

    Globus services will be unavailable for a period between 9:30am and 10:30am U.S. Central time on Saturday, October 24, 2026 due to planned maintenance (https://www.globus.org/blog/2026-10-24-maintenance) that includes database upgrades.

    Please refer to the posted link for additional information on how services and in-progress work will be impacted during this window.

holylfs06 degraded
ResolvedDegraded performance5 hours 15 minutes
  • ResolvedUTC

    This incident has been resolved.

  • InvestigatingUTC

    holylfs06 is in a very degraded or stuck state.

    Expect degraded to no access until the system is back up.

Sep 2026

h-nfs11-p down
ResolvedPartial outage144 hours 33 minutes
  • ResolvedUTC

    The vendor has replaced the necessary hardware components, and FASRC engineers have installed updates and confirmed working status. No data was affected.

    All h-nfs11-p shares should be accessible again once more.

    If you notice any irregularities, please reach out at rchelp@rc.fas.harvard.edu

  • UpdateUTC

    The vendor has dispatched parts. The vendor has indicated that this may not get fixed until Monday, but the storage team is pressing them to accelerate that timeline. We will update here when we know more.

    Thanks for your understanding.

  • UpdateUTC

    After lengthy troubleshooting, h-nfs11-p is still down. The vendor is scheduling a datacenter visit for on-site repairs. We will update as we know more.

  • IdentifiedUTC

    Our engineers are diagnosing the root cause of the failure and are working with the vendor to troubleshoot further

    We are continuing to work on a fix for this incident.

  • InvestigatingUTC

    h-nfs11-p is currently down. The following shares are inaccessible on FASSE

    • ncf_cnl01

    • ncf_cnl02

    • ncf_cnl03

    • ncf_cnl04

    • ncf_gspdata

    • ncf_mri_l3

    • cnl05

    • cnl06

    • eldaief

    • ncf_apps

    • jcamprodon_lab_l3

    We are investigating this incident.

Authentication outage
ResolvedMajor outage30 minutes
  • ResolvedUTC

    Openauth/radius is now operational. This update was created by an automated monitoring service.

  • InvestigatingUTC

    Authentication issues with openauth/radius. This incident was created by an automated monitoring service.

FASRC VPN (Boston) is back up
ResolvedMajor outage19 minutes
  • ResolvedUTC
    FASRC VPN (Boston) is back up. This incident was automatically resolved by Instatus monitoring.
  • InvestigatingUTC
    FASRC VPN (Boston) is down at the moment. This incident was automatically created by Instatus monitoring.
FASRC VPN (Cambridge) is back up
ResolvedMajor outage9 minutes
  • ResolvedUTC
    FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring.
  • InvestigatingUTC
    FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring.
FASRC VPN (Cambridge) is back up
ResolvedMajor outage9 minutes
  • ResolvedUTC
    FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring.
  • InvestigatingUTC
    FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring.

Previous

Sep 2026 to Nov 2026

Next

For issues not shown here, please contact FASRC via
https://portal.rc.fas.harvard.edu or email rchelp@rc.fas.harvard.edu