FAS Research Computing Status - Incident history

login.rc.fas.harvard.edu is responding normally

Sun, 19 Apr 2026 11:55:47 +0000

Type: Incident

Affected Components: , , ,

Apr 19, 11:55:47 GMT+0
Investigating - login.rc.fas.harvard.edu is not responding normally. This incident was automatically created..

Apr 19, 12:55:25 GMT+0
Resolved - \\\[login.rc.fas.harvard.edu\\\](http://login.rc.fas.harvard.edu) is responding normally. This incident was automatically resolved..

Authentication outage

Sun, 19 Apr 2026 06:44:52 +0000

Type: Incident

Duration: 7 hours and 30 minutes

Affected Components:

Apr 19, 06:44:52 GMT+0
Investigating - Authentication issues with openauth/radius. This incident was created by an automated monitoring service..

Apr 19, 14:14:53 GMT+0
Resolved - Openauth/radius is now operational. This update was created by an automated monitoring service..

MGHPCC Power Loss

Sun, 19 Apr 2026 06:18:00 +0000

Type: Incident Duration: 8 hours and 54 minutes Affected Components: FASSE Compute Cluster (Holyoke), FASRC Two-Factor (OpenAuth), HolyLFS04 (Tier 0), GPU nodes (Holyoke), Infiniband - Holyoke/MGHPCC, SLURM Scheduler - FASSE, Holyoke Tier 2 NFS (new), Holylabs, Holyoke Firewall, Network - Holyoke/MGHPCC, HolyLFS06 (Tier 0), Holyoke/MGHPCC Data Center, Cannon Compute Cluster (Holyoke), seas_compute, Cannon Open OnDemand/VDI, FASSE login nodes, Kempner Cluster GPU, Kempner Cluster CPU, FASSE Open OnDemand/VDI, Virtual Infrastructure - Holyoke, Holystore01 (Tier 0), Login Nodes - Holyoke, NESE (NorthEast Storage Exchange), SLURM Scheduler - Cannon, Isilon Storage Holyoke (Tier 1), holECS, Holyoke Specialty Storage, Login Nodes - Boston, HolyLFS05 (Tier 0), Netscratch (Global Scratch), Tape - (Tier 3), Boston Compute Nodes Apr 19, 06:18:00 GMT+0 - Investigating - At 2:18am on April 19th MGHPCC (our Holyoke datacenter) lost cooling which caused the entire facility to shutdown. This caused the loss of all jobs that were running. Storage and data on that storage should be safe. The facility is working on restoring cooling and power. Unfortunately we do not have an ETA. Apr 19, 13:06:16 GMT+0 - Identified - Power was fully restored to MGHPCC at 7:39am on April 19th. FASRC staff has restored functionality to most systems except for FASSE Open OnDemand. All other services are up and operating normally. If you continue to see issues with any system that is marked operational please let us know. We will deal with any non urgent requests in normal working hours. Apr 19, 15:12:24 GMT+0 - Resolved - All services, including FASSE OOD, should be functional at this time. If you continue to see issues with any system that is marked operational please let us know by sending an email to [rchelp@rc.fas.harvard.edu](mailto:rchelp@rc.fas.harvard.edu) We will deal with any non urgent requests in normal working hours. This incident has been resolved.

Starfish down

Mon, 13 Apr 2026 16:30:00 +0000

Type: Incident

Duration: 1 day and 48 minutes

Affected Components:

Apr 13, 16:30:00 GMT+0
Investigating - Starfish is currently unavailable, due to a network card issue. Updates to come. We are currently investigating this incident..

Apr 14, 14:17:21 GMT+0
Identified - Staff will be at the datacenter today to check on the physical status of the server. Updates to come. We are continuing to work on a fix for this incident..

Apr 14, 17:17:30 GMT+0
Resolved - The network card has been replaced, and Starfish is back up. This incident has been resolved..

Coldfront is down.

Wed, 8 Apr 2026 13:31:31 +0000

Type: Incident

Duration: 43 minutes

Affected Components:

Apr 8, 13:31:31 GMT+0
Investigating - Coldfront logins are producing an error message. We are currently investigating this incident..

Apr 8, 14:14:35 GMT+0
Resolved - Coldfront is back up and accepting logins..

Starfish dashboard inaccessible

Mon, 6 Apr 2026 14:27:54 +0000

Type: Incident

Duration: 1 hour and 3 minutes

Affected Components:

Apr 6, 14:27:54 GMT+0
Investigating - The Starfish dashboard is inaccessible. We are looking into the issue..

Apr 6, 15:31:02 GMT+0
Resolved - Starfish has resolved the issue and the dashboard is once again available..

FASRC monthly maintenance April 6th 2026 9am-1pm

Mon, 6 Apr 2026 13:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components: , , , , , , , , ,

Apr 6, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 6, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 6, 13:00:00 GMT+0
Identified - FASRC monthly maintenance will take place on April 6th 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **NO** FASSE cluster will be paused during this maintenance?: **NO** * [two-factor.rc.fas.harvard.edu](http://two-factor.rc.fas.harvard.edu) [OpenAuth](https://docs.rc.fas.harvard.edu/kb/openauth/) cut-over to new server * Audience: New accounts or anyone requesting an OpenAuth token * Impact: two-factor will be unavailable while moving to a new server * RStudio Server (Open OnDemand) * Audience: RStudio Server users on Cannon and FASSE * Impact: We will be decommissioning some versions of RStudio Server so we can properly maintain all production versions. Versions to be decommissioned: * R 4.1.3 (Bioconductor 3.14, RStudio 2022.02.0) * R 4.1.0 (Bioconductor 3.13, RStudio 1.4.1717) * R 4.0.3 (Bioconductor 3.12, Rstudio 1.3.1093) * R 4.0.0 (Bioconductor 3.11, Rstudio 1.3.1093) * If you use one of these versions, we recommend replacing it with the most recent version, R 4.4.2 (Bioconductor 3.20, RStudio 2024.12.0). You must reinstall previously installed libraries. * Domain controller replacement * Audience: Internal * Impact: End users should not see any impact * OOD/Open OnDemand reboots * Audience: All OOD users, reboot of the head nodes * Impact: Running sessions will _not_ be affected * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window * Netscratch 90-day retention cleanup * Audience; All netscratch users * Impact: Files older than 90 days will be removed per our [scratch policy](https://docs.rc.fas.harvard.edu/kb/policy-scratch/). Please note that this cleanup can happen at any time, not just during maintenance. Thank you, FAS Research Computing .

Scheduler is degraded

Tue, 31 Mar 2026 21:15:24 +0000

Type: Incident

Duration: 14 hours and 56 minutes

Affected Components: , , , , , , , ,

Apr 1, 12:11:15 GMT+0
Resolved - This incident has been resolved. The scheduler is running normally..

Mar 31, 21:15:24 GMT+0
Investigating - The scheduler is in a degraded state due to [thrashing](https://en.wikipedia.org/wiki/Thrashing%5F%28computer%5Fscience%29) We are actively working to resolve this problem..

two-factor.rc.fas.harvard.edu (openauth) error

Tue, 31 Mar 2026 15:32:20 +0000

Type: Incident

Duration: 1 hour and 12 minutes

Affected Components:

Mar 31, 15:32:20 GMT+0
Investigating - We are currently investigating this incident. Requesting a new token or re-requesting your token from two-factor is not currently working. .

Mar 31, 16:44:08 GMT+0
Resolved - This incident has been resolved. two-factor.rc.fas.harvard.edu is working normally again..

The web front end to two-factor.rc.fas.harvard.edu is currently not allowing logins, generating new tokens is currently unavailable

Wed, 25 Mar 2026 14:30:00 +0000

Type: Incident

Duration: 6 days and 30 minutes

Affected Components:

Mar 25, 14:30:00 GMT+0
Investigating - We are currently investigating this incident..

Mar 31, 15:00:25 GMT+0
Resolved - This incident has been resolved..

Network issues - Cluster degraded

Wed, 25 Mar 2026 13:34:01 +0000

Type: Incident

Duration: 5 days, 7 hours and 7 minutes

Affected Components: , , , , , , ,

Mar 25, 14:10:34 GMT+0
Identified - Mounts to Holyoke Isilon (specifically /n/sw) are broken on numerous nodes across the cluster. We have a check rolling out to find these nodes so we can remediate them individually. Until remediated the cluster will be in a degraded state. Running jobs may randomly die or fail as they hit nodes that have stale mounts. It will be risky to run jobs for the next hour and then, after that point, the cluster will have a large number of nodes closed waiting for them to drain so we can reboot them and fix the mounts..

Mar 25, 13:34:01 GMT+0
Investigating - A network issue affecting storage critical to the cluster is It's causing instability. The cluster is currently in a degraded state as a result. We are looking into the problem. Updates to follow...

Mar 25, 14:31:18 GMT+0
Monitoring - Mounts to Holyoke Isilon (specifically /n/sw) are broken on numerous nodes across the cluster. We have a check rolling out to find these nodes so we can remediate them individually. Until remediated the cluster will be in a degraded state. Running jobs may randomly die or fail as they hit nodes that have stale mounts. It will be risky to run jobs for the next hour and then, after that point, the cluster will have a large number of nodes closed waiting for them to drain so we can reboot them and fix the mounts. At this time we are unaware of any holy-isilon problems other than the effect this had on cluster nodes/running jobs. We will update should we identify any data storage concerns..

Mar 30, 20:41:25 GMT+0
Resolved - This incident has been resolved by draining and rebooting any nodes with stuck mounts..

ColdFront is down.

Thu, 19 Mar 2026 12:58:36 +0000

Type: Incident

Duration: 1 hour and 54 minutes

Affected Components:

Mar 19, 12:58:36 GMT+0
Identified - ColdFront is down. We are working to bring it back up. The instance got replaced last night, but it had trouble configuring itself on the way up again..

Mar 19, 14:52:54 GMT+0
Resolved - Cold front is back up. Thank you for your patience..

Key access issue to CSBN, HERS, FIINE, Portal Approve (p3approve)

Fri, 13 Mar 2026 20:35:07 +0000

Type: Incident

Duration: 5 days, 21 hours and 59 minutes

Affected Components:

Mar 13, 20:35:07 GMT+0
Investigating - We are currently investigating this incident. This only affects specific services. Users of CSBN, HERS, FIINE, Portal Approve (p3approve) may be affected. Email coming from these systems may also be delayed. No ETA.

Mar 19, 18:33:57 GMT+0
Resolved - This incident has been resolved..

Starfish dashboard unavailable

Mon, 2 Mar 2026 16:22:42 +0000

Type: Incident

Duration: 1 hour and 16 minutes

Affected Components:

Mar 2, 16:22:42 GMT+0
Investigating - The Starfish dashboard is not responding. We are currently investigating this issue with the vendor..

Mar 2, 17:38:26 GMT+0
Resolved - This incident has been resolved..

FASRC monthly maintenance Monday March 2nd, 2026 9am-1pm

Mon, 2 Mar 2026 14:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components: , , , , , , , , , , , , , , , , , , , ,

Mar 2, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 2, 14:00:00 GMT+0
Identified - Monthly maintenance will take place on Monday March 2nd, 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * Slurm scheduler update * Audience: All cluster users * Impact: Jobs will be paused during maintenance * OOD node reboots * Audience; All Open OnDemand users * Impact: OOD nodes will reboot during the maintenance window * Login node reboots * Audience: All login node users * Impact: Login nodes will reboot during the maintenance window * Netscratch retention purge * Audience: All users of Netscratch * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing [https://www.rc.fas.harvard.edu/](https://www.rc.fas.harvard.edu/upcoming-training/).

Mar 2, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

Tape outage

Fri, 27 Feb 2026 21:27:09 +0000

Type: Incident

Duration: 4 days, 17 hours and 42 minutes

Affected Components: ,

Mar 4, 15:09:37 GMT+0
Resolved - This incident has been resolved. Normal tape operations are restored..

Feb 27, 21:27:09 GMT+0
Investigating - NESE Tape service will be down or operating with degraded service (no store and recall) Friday from 12 Noon EST until as late as Monday, 2 March at 9 AM. SUMMARY OF ISSUE: NESE Tape service is currently not able to store or recall files to and from tape due to vendor firmware issues in the IBM TS4500 tape library. The issue is related to the library robotics and cartridge database and we do NOT expect any data loss from this issue. The issue is apparently due to an issue with the inventory database related to a recent firmware update. This database can be scrubbed and reconstructed by the library, which will scan the bar code labels on all the cartridges to rebuild the inventory. Association of files in Globus to tapes is handled separately from the tape library and is not affected by the firmware update..

Mar 2, 14:03:01 GMT+0
Identified - NESE Tape Service is still working with IBM technical support at restoring the inventory. The expected downtime is extended until Tuesday March 3rd, 9am. Apologies for the inconvenvenience..

Mar 3, 14:04:46 GMT+0
Monitoring - The tape library outage is further extended to Wednesday March 4th at 9am awaiting a hardware replacement part due today. Data can still be uploaded to lab collections via Globus, but be mindful of the 10 TB buffer file limit. The outage affects storage and recall from tape..

Starfish dashboard is unavailable

Thu, 26 Feb 2026 14:13:35 +0000

Type: Incident

Duration: 1 day, 7 hours and 50 minutes

Affected Components:

Feb 27, 22:04:05 GMT+0
Resolved - This incident has been resolved. The Starfish dashboard is available..

Feb 26, 14:13:35 GMT+0
Investigating - The starfish dashboard is unavailable. We are currently investigating this issue with Starfish...

Starfish maintenance Feb 25, 2026 all day

Wed, 25 Feb 2026 14:00:00 +0000

Type: Maintenance

Duration: 1 day

Affected Components:

Feb 25, 14:00:00 GMT+0
Identified - Starfish will be unavailable starting Wednesday, February 25th at 9AM until Thursday, February 26th at 9AM, for routine maintenance. The online dashboard will be inaccessible during this time..

Feb 26, 14:00:00 GMT+0
Completed - Maintenance has completed successfully.

Feb 25, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Authentication outage

Tue, 24 Feb 2026 15:39:56 +0000

Type: Incident

Duration: 5 minutes

Affected Components:

Feb 24, 15:39:56 GMT+0
Investigating - Authentication issues with openauth/radius. This incident was created by an automated monitoring service..

Feb 24, 15:44:57 GMT+0
Resolved - Openauth/radius is now operational. This update was created by an automated monitoring service..

NESE tape maintenance Feb 19th 2026

Thu, 19 Feb 2026 13:00:00 +0000

Type: Maintenance

Duration: 9 hours

Affected Components:

Feb 19, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 19, 13:00:00 GMT+0
Identified - From our partners at NESE. Details follow: We are installing four new tape frames, which will bring the tape system raw storage capacity to 253 petabytes. **Service Affected:** NESE Tape Service **Maintenance Window:** 8:00 AM - 5:00 PM (EST) * The tape service will be unavailable. * All upgrade activities are expected to be completed on the same day. NOTES: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor for real-time updates on progress Subscribe to for updates and announcements.

Feb 19, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

OOD inaccessible

Wed, 11 Feb 2026 16:15:00 +0000

Type: Incident

Duration: 3 hours and 30 minutes

Affected Components: , ,

Feb 11, 19:45:06 GMT+0
Resolved - This incident has been resolved and OOD is working normally..

Feb 11, 16:15:00 GMT+0
Investigating - OpenOnDemand for both Cannon and FASSE may be inaccessible for some users. Errors may include: "Error -- can't find user for " "502 proxy errors" For users that are able to access OOD, performance may be degraded or sessions may get stuck. We are currently investigating the root causes of this incident. Updates to follow .

Security updates needed for www.rc.fas.harvard.edu and docs.rc.fas.harvard.edu

Mon, 9 Feb 2026 21:10:00 +0000

Type: Maintenance

Duration: 8 minutes

Affected Components: ,

Feb 9, 21:10:00 GMT+0
Identified - Security updates will require a brief interruption for our primary websites [www.rc.fas.harvard.edu](http://www.rc.fas.harvard.edu) and [docs.rc.fas.harvard.edu](http://docs.rc.fas.harvard.edu) We will endeavour to keep this update as short as possible. Each site may be unavailable for a few minutes..

Feb 9, 21:10:01 GMT+0
Identified - Maintenance is now in progress.

Feb 9, 21:17:51 GMT+0
Completed - Maintenance has completed successfully..

License server issue

Mon, 9 Feb 2026 18:54:59 +0000

Type: Incident

Duration: 18 minutes

Affected Components: , , , , , , , , , , , , , , ,

Feb 9, 18:54:59 GMT+0
Investigating - New sessions of Matlab are hanging. We are currently investigating this incident..

Feb 9, 19:04:52 GMT+0
Identified - The affected softwares include: Matlab Mathematica Gurobi We are continuing to work on a fix for this incident..

Feb 9, 19:13:20 GMT+0
Resolved - The license server is back up, and all software should be performing as expected. This incident has been resolved..

NESE tape maintenance Feb 9th 2026

Mon, 9 Feb 2026 13:00:00 +0000

Type: Maintenance

Duration: 9 hours

Affected Components:

Feb 9, 13:00:00 GMT+0
Identified - From our partners at NESE. Details follow: In the process of the tape front-end file caching system upgrade, we will be installing a new IBM Storage Scale System 6000\. We will provide an additional update for when the software integration and data transfer from the current IBM Elastic Storage System 5000 will be performed. **Service Affected:** NESE Tape Service **Maintenance Window: No Downtime expected** NOTES: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor for real-time updates on progress * Subscribe to for updates and announcements.

Feb 9, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 9, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Grafana Cloud (FASRC) is down

Fri, 6 Feb 2026 20:44:10 +0000

Type: Incident

Affected Components:

Feb 6, 20:44:11 GMT+0
Investigating - Grafana Cloud (FASRC) is down at the moment. This incident was automatically created by Instatus monitoring..

Feb 6, 20:48:52 GMT+0
Resolved - Grafana Cloud (FASRC) is back up. This incident was automatically resolved by Instatus monitoring..

FASRC monthly maintenance Monday February 2nd, 2026 9am-1pm

Mon, 2 Feb 2026 14:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components: , , , , , , , , , , , , , , , , , , , ,

Feb 2, 14:00:00 GMT+0
Identified - Monthly maintenance will take place on Monday February 2nd, 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * MaxTime change * Audience: Cluster users * Impact: In order to improve scheduling efficiency and stability, we will be setting a maximum run time on all partitions that have MaxTime set to UNLIMITED to a MaxTime of 3 days. The unrestricted partition will be set to 365 days. Partitions that already have MaxTime set will retain their current setting. Partition owners wishing to set a different MaxTime for their partition should contact FASRC. Note that we do no guarantee uptime and so users should utilize checkpointing to save state in case of node failure. * Slurm upgrade to 25.11.2 * Audience: All cluster users * Impact: Jobs will be paused during maintenance * OOD node reboots * Audience; All Open OnDemand users * Impact: OOD nodes will reboot during the maintenance window * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window Thank you, FAS Research Computing [https://www.rc.fas.harvard.edu/](https://www.rc.fas.harvard.edu/upcoming-training/).

Feb 2, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 2, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

Coldfront down

Thu, 22 Jan 2026 15:06:02 +0000

Type: Incident

Duration: 59 minutes

Affected Components:

Jan 22, 16:04:46 GMT+0
Resolved - Coldfront is operational. Thank you for your patience..

Jan 22, 15:06:02 GMT+0
Investigating - We are currently investigating an issues with Coldfront. No ETA..

holystore01 down

Wed, 21 Jan 2026 16:28:00 +0000

Type: Incident

Duration: 1 hour and 1 minute

Affected Components:

Jan 21, 16:28:00 GMT+0
Investigating - The filesystem holystore01 is experiencing a network failure and in a bad state. Some files on holystore01 may not be accessible while this is ongoing. We are working to restore access, and apologize for the inconvenience. .

Jan 21, 17:29:01 GMT+0
Resolved - holystore01 is back up and usable. This incident has been resolved..

Grafana Cloud is down

Thu, 15 Jan 2026 15:34:57 +0000

Type: Incident

Jan 15, 15:34:57 GMT+0
Investigating - is down at the moment. This incident was automatically created by Instatus monitoring..

Jan 15, 15:46:56 GMT+0
Resolved - is back up. This incident was automatically resolved by Instatus monitoring..

holystore01 is wedging. We are rebooting.

Tue, 13 Jan 2026 14:57:56 +0000

Type: Incident

Duration: 14 minutes

Affected Components:

Jan 13, 14:57:56 GMT+0
Identified - holystore01 is wedging. We are rebooting..

Jan 13, 15:12:13 GMT+0
Resolved - This incident has been resolved..

FASRC monthly maintenance Monday January 12th, 2026 9am-1pm

Mon, 12 Jan 2026 14:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components: , , , , , , , , , , , , , , , , , , ,

Jan 12, 14:00:00 GMT+0
Identified - Monthly maintenance will take place on January 12th, 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Changes to SEAS partitions, please see tasks below. * Changes to job age priority weighting, please see tasks below. * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?:**YES** * Slurm upgrade to 25.11.1 * Audience: All cluster users (Cannon and FASSE) * Impact: Jobs will be paused during maintenance * In conjunction with SEAS we will modify seas\_gpu and seas\_compute time limits * Audience: SEAS users * Impact: seas\_gpu: will be set to 2 days maximum seas\_compute: will be set to 3 days maximum Existing pending jobs longer than these limits will be set to 2 day and 3 day run times depending on partition. * Job Age Priority Weight Change * Audience: Cluster users * Impact: We will be adjusting the weight applied to the priority earned by jobs by virtue of their age. Currently job priority is made up of two factors, Fairshare and Job Age. The Job Age factor is currently set such that jobs gain priority over 3 days with a maximum priority equivalent to jobs with Fairshare of 0.5\. This keeps low fairshare jobs from languishing at the bottom of the queue. With the current settings though, users with low fairshare can gain significant advantage over users with higher relative fairshare. To remedy this we will be adjusting the Job Age weight to cap out at an equivalent Fairshare of 0.1\. This will still allow jobs with 0 fairshare to gain priority and thus not languish while letting fairshare govern a wider range of higher priority jobs. * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window * Open OnDemand (OOD) node reboots * Audienc:; All OOD users * Impact: OOD nodes will reboot during the maintenance window * Netscratch retention will run * Audience: All cluster netscratch users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing [https://www.rc.fas.harvard.edu/](https://www.rc.fas.harvard.edu/upcoming-training/).

Jan 12, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 12, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

fasselogin01 reboot

Fri, 9 Jan 2026 16:32:56 +0000

Type: Incident

Duration: 26 minutes

Affected Components:

Jan 9, 16:32:56 GMT+0
Identified - fasselogin01 will be rebooted at 11:45 to fix some mounts.

Jan 9, 16:58:42 GMT+0
Resolved - fasselogin01 is back with the correct mounts..

Slurm down

Fri, 9 Jan 2026 00:11:25 +0000

Type: Incident Duration: 2 hours and 7 minutes Affected Components: NESE (NorthEast Storage Exchange), Holyoke Tier 2 NFS (new), Holyoke-Boston fiber link (long path), Globus Data Transfer, GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), Isilon Storage Holyoke (Tier 1), Starfish, Boston Tier 2 NFS (new), Infiniband - Holyoke/MGHPCC, HolyLFS04 (Tier 0), SLURM Scheduler - FASSE, Web Proxies, Network - Holyoke/MGHPCC, Virtual Infrastructure - Holyoke, Holyoke-Boston fiber link (short path), Infiniband - Boston, HolyLFS06 (Tier 0), bosECS, Boston Specialty Storage, Authentication, Boston Data Center, Samba Cluster, Software & Modules, Holyoke/MGHPCC Data Center, Cannon Compute Cluster (Holyoke), Network - Boston, CEPH Storage Boston (Tier 2), Cannon Open OnDemand/VDI, Cambridge firewall and other redundancy, SLURM Scheduler - Cannon, Holylabs, FASSE Open OnDemand/VDI, FIINE billing portal, License Servers, seas_compute, Login Nodes - Holyoke, FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, holECS, Holyoke Specialty Storage, FASRC Two-Factor (OpenAuth), FASRC Downloads Site, Citrix, Login Nodes - Boston, HolyLFS05 (Tier 0), Virtual Infrastructure - Boston, Network - Cambridge, FASRC VPN (Cambridge) , FASRC VPN (Boston), Holystore01 (Tier 0), Grafana Cloud (FASRC), BosLFS02 (Tier 0), Isilon Storage Boston (Tier 1), Home Directory Storage - Boston, docs.rc.fas.harvard.edu, portal.rc.fas.harvard.edu, Spinal, Bauer MiniLIMS, www.rc.fas.harvard.edu, Netscratch (Global Scratch), Tape - (Tier 3), FASRC Offsite Hosting, FASRC Ticket System (ServiceNow), Boston Compute Nodes, Harvard DNS System, Coldfront, Holyoke Firewall Jan 9, 00:11:25 GMT+0 - Investigating - The Slurm scheduler is currently down and no new jobs are able to be scheduled. We are currently investigating this incident and will provide updates. Jan 9, 02:18:39 GMT+0 - Resolved - The rogue job has been found and removed. The scheduler is running normally again and all partitions are open. Jan 9, 02:20:18 GMT+0 - Resolved - This incident has been resolved.

Monthly Maintenance and MGHPCC Power Work - Dec. 8, 2025 6am-6pm

Mon, 8 Dec 2025 11:00:00 +0000

Type: Maintenance

Duration: 12 hours

Affected Components: , , , , , , , , , , , , , , , , , , , , , , , ,

Dec 8, 11:00:00 GMT+0
Identified - Monthly maintenance will take place on December 8th. Our maintenance tasks should be completed between 9am-1pm. However: _Additionally_, MGHPCC will be performing power upgrades on the odd side of Row 8A where much of our computer resides. This is the final upgrade for this row. Current estimate for this work is a 12 hour window 6am-6pm. A list of the affected partitions is provided at the bottom of this notice. The nodes in those partitions will be drained prior to the work and will be powered down. Once the work is completed, those nodes will be returned to service. **Notices:** * New FASSE partition `fasse_gpu_h200`. This partitions has 2 H200 nodes and a 3day limit. It is available now. * 11/26 - 11/28 are university holidays (Thanksgiving). No on-site support, FASRC staff will return on 12/1. * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** FASSE cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** * Power work on Row 8A odd * Audience: Users of the partitions listed below * Impact: These nodes and partitions will be fully or partially down all day * OneFS (Isilon) upgrade * Audience: All Isilon (Tier 1) shares * Impact: Some VMs will be impacted including Cannon OOD, CBScentral, MCZapps/MCZbase, Portal, and Rclic1 (license server) * Slurm upgrade to 25.05.5 * Audience: All cluster users * Impact: Jobs will be paused during maintenance * Login node reboots * Audience: All login node users * Impact: Login nodes will reboot during the maintenance window **Impacted Cannon Partitions (Full or Partial Outage):** * arguelles\_delgado\_gpu\_a100 * arguelles\_delgado\_gpu\_mixed * bigmem\_intermediate * blackhole\_gpu * eddy * gershman * gpu\_requeue * hejazi * hernquist\_ice * hoekstra * huce\_ice * iaifi\_gpu * iaifi\_gpu\_priority * iaifi\_gpu\_requeue * itc\_gpu * jshapiro * kempner * kempner\_dev * kempner\_priority * kempner\_h100 * kempner\_h100\_priority * kempner\_h100\_priority2 * kempner\_h100\_priority3 * kempner\_interactive * kempner\_requeue * kovac * kozinsky * kozinsky\_gpu * kozinsky\_priority * kozinsky\_requeue * murphy\_ice * ortegahernandez\_ice * rivas * seas\_compute * seas\_gpu * serial\_requeue * siag\_combo * siag\_gpu * sur * zhuang.

Dec 8, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 8, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

holylfs04 migrations

Fri, 5 Dec 2025 14:00:00 +0000

Type: Maintenance

Duration: 4 days, 1 hour and 10 minutes

Affected Components:

Dec 5, 14:00:00 GMT+0
Identified - The holylfs04 migration to holylfs06 has begun. All holylfs04 folders will be **read-only** for the duration of the migration, from **Friday, December 5th at 9AM until end of day on Monday, December 8th.** All labs with holylfs04 have been informed via email; please email [rdm@rc.fas.harvard.edu](mailto:rdm@rc.fas.harvard.edu) if you have any questions..

Dec 9, 15:09:30 GMT+0
Completed - Maintenance has completed successfully..

NESE tape system maintenance 12/1/25-12/5/25

Mon, 1 Dec 2025 11:00:00 +0000

Type: Maintenance

Duration: 5 days

Affected Components: ,

Dec 1, 11:00:00 GMT+0
Identified - NESE, the Northeast Storage Exchange at MGHPCC which supplies the Tier3 tape service used by FASRC, will be offline for maintenance on the system Dec 1st - 5th. There will be ongoing performance-affecting maintenance until Dec 12th. Please see below for details. WHO: Any lab who has or is moving data to tape. IMPACT: No access 12/1/25 - 12/5/25\. Reduced performance 12/5/25 - 12/12/25. > NESE tape system maintenance and major software upgrade is scheduled to begin on December 1, 2025\. As a result, the NESE Tape service will be offline from December 1 to December 5. > > Starting December 8 through December 12, the service will be back online with reduced performance. All maintenance activities are planned to conclude on December 12, 2025. > > * Monitor: for real-time updates on progress > * Subscribe to for updates and announcements.

Dec 1, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 6, 11:00:00 GMT+0
Completed - Maintenance has completed successfully.

Starfish dashboard maintenance Nov. 14th 5-6PM

Fri, 14 Nov 2025 22:00:00 +0000

Type: Maintenance

Duration: 1 hour

Affected Components:

Nov 14, 22:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 14, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

Nov 14, 22:00:00 GMT+0
Identified - There is a planned upgrade of the Starfish dashboard scheduled for Friday November 14th starting at 5PM. The dashboard will be down for an hour while the upgrade is performed..

Monthly Maintenance and MGHPCC Power Work - Nov. 3, 2025 6am-6pm

Mon, 3 Nov 2025 11:00:00 +0000

Type: Maintenance

Duration: 9 hours and 4 minutes

Affected Components: , , , , , , , , , , , , , , , , , , ,

Nov 3, 11:00:00 GMT+0
Identified - Monthly maintenance will take place on November 3rd. Additionally, MGHPCC will be performing power upgrades on the even side of Row 8A where much of our computer resides. A further upgrade will take place Dec. 8th on the odd side. A list of the affected partitions is provided at the bottom of this notice. The nodes in those partitions will be drained prior to the work and will be powered down. Once the work is completed, those nodes will be returned to service. Current estimate is a 12 hour window. We will adjust as we know more. **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** FASSE cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** * Power work on Row 8A Even * Audience: Users of the partitions listed below * Impact: These nodes and partitions will be fully or partially down all day * Slurm upgrade to 25.05.4 * Audience: All cluster users * Impact: Jobs will be paused during maintenance * Block repo.anaconda.com cluster wide * Audience: Anyone attempting to use repo.anaconda.com * Impact: This change should not impact your Python workflow on the cluster. But if it does, consider using the open-source channel, `conda-forge`, through Miniforge distribution to install Python packages. This can be done by following our instructions on * Change Slurm User to Local User * Audience: All cluster users * Impact: Behind the scenes. No impact to users * Login node reboots (morning) * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will rebooted during this maintenance window * Netscratch cleanup ( ) * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. **AFFECTED PARTITIONS** Nov. 3, 2025 - All Day Power Work Partial or Full Outage Apples to: arguelles\_delgado\_h100 bigmem dvorkin eddy enos gpu gpu\_h200 gpu\_requeue hsph hsph\_gpu intermediate itc\_cluster joonholee jshapiro kempner\_dev kemkpner\_eng kempner\_requeue mweber\_compute mweber\_gpu olveczky\_sapphire sapphire seas\_compute seas\_gpu serial\_requeue yao yao\_gpu yao\_priority test.

Nov 3, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 3, 20:04:13 GMT+0
Completed - Maintenance has completed successfully including power work at MGHPCC. A reminder that additional all-day power work will take place on Dec 8th, along with our maintenance from 9am-1pm.

FASRC monthly maintenance Monday October 6th, 2025 9am-1pm

Mon, 6 Oct 2025 13:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components: , , , , , , ,

Oct 6, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 6, 13:00:00 GMT+0
Identified - FASRC monthly maintenance will take place Monday October 6th, 2025 from 9am-1pm **NOTICES** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at (click Get Updates for options). * Upcoming holidays: Columbus / Indigenous Peoples’ Day - October 13 **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **NO** FASSE cluster will be paused during this maintenance?: **NO** * DNS server reboots * Audience: All FASRC services * Impact: Rolling reboot should have no impact * Login node reboots * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will rebooted during this maintenance window * Netscratch cleanup ( ) * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing .

Oct 6, 17:00:00 GMT+0
Completed - Maintenance is now in progress.

VPN concentrator rolling updates overnight

Tue, 30 Sep 2025 06:00:00 +0000

Type: Maintenance

Sep 30, 07:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 30, 06:00:00 GMT+0
Completed - Networking will be patching the VPN concentrators overnight. This will be done in a rolling order so that one ore more are always online. This may cause active VPN connections to drop, but they can be re-connected shortly after. ETA is one hour total..

Sep 30, 06:00:00 GMT+0
Identified - Networking will be patching the VPN concentrators overnight. This will be done in a rolling order so that one ore more are always online. This may cause active VPN connections to drop, but they can be re-connected shortly after. ETA is one hour total..

FASRC monthly maintenance Monday September 8th, 2025 9am-1pm

Mon, 8 Sep 2025 13:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components: , , , , , , , , , , , , , , , , , , ,

Sep 8, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 8, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 8, 13:00:00 GMT+0
Identified - FASRC monthly maintenance will take place Monday September 8th, 2025 from 9am-1pm **NOTICES** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at (click Get Updates for options). * Upcoming holidays: Labor Day, Monday September 1st **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * Slurm Upgrade to 25.05.2 * Audience: All cluster users * Impact: Jobs and the scheduler will be paused during this upgrade * Domain controller work * Audience: Internal network * Impact: No impact expected * Login node reboots * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will rebooted during this maintenance window * Netscratch cleanup ( ) * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing .

Starfish upgrade Wednesday, August 20th 5PM-7PM

Wed, 20 Aug 2025 21:00:00 +0000

Type: Maintenance

Duration: 2 hours

Affected Components:

Aug 20, 21:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 20, 21:00:00 GMT+0
Identified - Starfish will be performing an upgrade on Wednesday, August 20th from 5PM-7PM. The web interface will be unavailable during that timeframe..

Aug 20, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

SEAS: seas_gpu partition GPU upgrades 8/11 - 8/14

Mon, 11 Aug 2025 10:00:00 +0000

Type: Maintenance

Duration: 4 days, 9 hours and 39 minutes

Affected Components:

Aug 11, 10:00:00 GMT+0
Identified - Between 8/11/25 6AM - 8/14/25 5PM FASRC will be upgrading 14 of the H100 GPU nodes in the `seas_gpu `partition to H200 GPUs. This sill also affect `mweber_gpu` A reservation has been set which will drain the nodes of jobs prior to the maintenance. The SEAS GPU partition will be running at 75% capacity during these updates. FASRC has hundreds of GPU, so users should feel free to utilize `gpu_requeue` if needed for their jobs. Affected nodes: `mweber_gpu` nodes (13): ``` holygpu8a[18204,18301-18304,18401-18404,18501-18502,18601-18602] ``` seas\_gpu nodes (14): ``` holygpu8a[16101-16104,16201-16204,16301-16304,16401-16402] ``` Please reach out to [rchelp@rc.fas.harvard.edu](mailto:rchelp@rc.fas.harvard.edu) if you have any questions or concerns..

Aug 15, 14:13:24 GMT+0
Identified - Maintenance is still in progress - imaging of the new H200 nodes is ongoing. Current ETA is end of day Friday. For further questions, please contact rchelp@rc.fas.harvard.edu.

Aug 15, 19:39:27 GMT+0
Completed - Maintenance has completed successfully. SEAS H200 nodes have been imaged and are back in service. .

Aug 11, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Monthly maintenance August 4, 2025 9am-1pm

Mon, 4 Aug 2025 13:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components: , , , ,

Aug 4, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 4, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Aug 4, 13:00:00 GMT+0
Identified - > _Important note about changes to the gpu\_test, test, and remoteviz partitions_ > > On Monday July 28th we will make the following changes. These changes are necessary in order to reduce congestion on these partitions. > > gpu\_test will have a 2 job limit per user - Reminder, these partitions are for testing and debugging, not for production work. > > gpu\_test, test, and remoteviz partitions will no longer be available for multi-partition submission FASRC monthly maintenance will take place Monday August 4th, 2025 from 9am-1pm **MONTHLY NOTICES** * Do you have a success story about your use of the FASRC clusters or services? We'd love to hear it and post it on our [new User Stories page.](https://www.rc.fas.harvard.edu/user-stories/) * New Quota tool available. Type quota -h to see the full instructions for usage or visit the usage doc. * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar at **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **NO** FASSE cluster will be paused during this maintenance?: **NO** * Please see note above about partition changes * Those changes will happen on July 28th * FASRC websites (www.rc and docs.rc) will be updated * Audience: Anyone browsing our websites * Impact: The sites will be periodically unavailable during reboots * Login node reboots * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will reboot during this maintenance window * Netscratch cleanup () * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing .

FASRC away for all-hands meeting 7/17

Thu, 17 Jul 2025 13:00:00 +0000

Type: Maintenance

Duration: 8 hours

Affected Components:

Jul 17, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 17, 13:00:00 GMT+0
Identified - FASRC staff will be attending an all-hands meeting all day Thursday 7/17/25. Ticket response will be delayed..

Jul 17, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

FASRC Monthly maintenance July 7, 2025 9AM-1PM

Mon, 7 Jul 2025 13:00:00 +0000

Type: Maintenance

Duration: 4 hours

Affected Components: , , , , , , , , , , , , , , , , , , ,

Jul 7, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Jul 7, 13:00:00 GMT+0
Identified - FASRC monthly maintenance will take place Monday July 7th, 2025 from 9am-1pm **NOTICES** * New Quota tool available (/usr/local/sbin/quota) - Works on _all_ filesystem types (home directory, lustre, isilon, netscratch, etc.) Type `quota -h` to see the full instructions for usage o visit [the usage doc](https://docs.rc.fas.harvard.edu/kb/checking-quota-and-usage/). * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at (click Get Updates for options). * Upcoming holidays: Juneteenth - Thur. June 19 / Independence Day - Fri. July 4 **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * Slurm Upgrade to 24.11.5 * Audience: All cluster users * Impact: Jobs and the scheduler will be paused during this upgrade * Login node OS upgrades * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will upgraded and unavailable during this maintenance window * Start of cluster OS upgrades - July 7 -10 * Audience: All cluster users * Impact: **Over 4 days,** July 7 through 10, we will upgrade the OS on 25% of the cluster each day. During that time, total capacity will be reduced across the cluster by 1/4 each day. This will require draining each sub-set of nodes ahead of time. * Netscratch cleanup ( ) * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing .

Jul 7, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Rolling cluster OS upgrades July 7 - 10

Mon, 7 Jul 2025 13:00:00 +0000

Type: Maintenance

Duration: 4 days, 3 hours and 3 minutes

Affected Components: , , , , , , , , , , ,

Jul 7, 13:00:01 GMT+0
Identified - Cannon rolling upgrades are in progress. Not all nodes are available. .

Jul 7, 13:00:01 GMT+0
Identified - **UPDATE:** 7/7/25 6M FASSE is operational. ~~Please be aware that FASSE jobs cannot be launched at this time due to the upgrades.~~ ~~We will return all FASSE nodes to normal services as soon as possible.~~ .

Jul 7, 13:00:00 GMT+0
Identified - Cluster OS upgrades - July 7 -10 * Audience: All cluster users * Impact: **Over 4 days**, July 7 through 10, we will upgrade the OS on **25%** of the cluster each day. During that time, total capacity will be reduced across the cluster by 1/4 each day. This will require draining each sub-set of nodes ahead of time. Work begins during the July 7th maintenance (login nogdes will be upgraded during the 7/7 maintenance window) and will continue through July 10th. Additional details and a breakdown of each phase: [2025 Compute OS Upgrade](https://www.rc.fas.harvard.edu/blog/2025-compute-os-upgrade/).

Jul 11, 16:02:45 GMT+0
Completed - All upgrades are complete. A small number of nodes need clean-up, but the cluster is back to normal operation with all nodes running Rocky 8.10\. Thanks for your patience..

June 5-6 MGHPCC pod 7c cooling updates - See partition list below

Thu, 5 Jun 2025 11:00:00 +0000

Type: Maintenance

Duration: 8 hours and 3 minutes

Affected Components: ,

Jun 5, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 5, 19:03:03 GMT+0
Completed - The work on row 7c is complete. Returning idled nodes to normal service..

Jun 5, 11:00:00 GMT+0
Identified - There will be additional scheduled maintenance at MGHPCC between June 5th and 6th. As part of the work during the MGHPCC Outage, one of the Cooling Distribution Unit (CDU) in Pod 7c will be replaced. This will allow for future expansion into this space. This work will run from Thursday Jun 5th until the evening of Friday June 6th. This means nodes whose names begin with holy7c02, 04, 06, 08, 10, 12 _will not_ come back online after the outage and will remain down until this CDU update is complete. This impacts the following partitions. If you are using one of those partitions please use the public sapphire partition while your equipment is being serviced. These nodes will be returned to service once the CDU work is complete: * blackhole * blackhole\_priority * davies * desai * eddy * huce\_cascade * huce\_cascade\_priority * huttenhower * jacobsen2 * janson * janson\_cascade * ke * lukin * nguyen * seas\_compute * shared * tambe * vishwanath * whipple * xlin.

2025 MGHPCC power downtime June 2-4, 2025

Mon, 2 Jun 2025 13:00:00 +0000

Type: Maintenance Duration: 3 days Affected Components: SLURM Scheduler - Cannon, Holyoke/MGHPCC Data Center, Software & Modules, NESE (NorthEast Storage Exchange), Holystore01 (Tier 0), FASSE login nodes, FASSE Open OnDemand/VDI, seas_compute, Kempner Cluster CPU, Isilon Storage Holyoke (Tier 1), Holyoke Firewall, Holyoke Tier 2 NFS (new), HolyLFS05 (Tier 0), Globus Data Transfer, Netscratch (Global Scratch), GPU nodes (Holyoke), Infiniband - Holyoke/MGHPCC, HolyLFS04 (Tier 0), SLURM Scheduler - FASSE, Boston Compute Nodes, Samba Cluster, Virtual Infrastructure - Holyoke, Network - Holyoke/MGHPCC, License Servers, Cannon Compute Cluster (Holyoke), FASSE Compute Cluster (Holyoke), Holylabs, Cannon Open OnDemand/VDI, Login Nodes - Holyoke, Kempner Cluster GPU, holECS, Holyoke Specialty Storage, Login Nodes - Boston, HolyLFS06 (Tier 0), Tape - (Tier 3) Jun 2, 13:00:00 GMT+0 - Identified - The yearly power downtime at our Holyoke data center, MGHPCC, has been scheduled. This year's power downtime will take place on Tuesday June 3, 2025\. This will require FASRC to begin shutdown of our systems beginning at _9AM on Monday, June 2nd_. We have worked to reduce the total outage time this year. We will begin power-up on Wednesday June 4th with an expected return to full service by _9AM Thursday June 5th_. * **Monday June 2nd** \- Power-down begins at 9AM * **Tuesday June 3rd** \- Power out at MGHPCC * **Wednesday June 4th** \- Maintenance tasks and then power-up begins * **Thursday June 5th** \- Expected return to full service by 9AM **Maintenance:** During this downtime, Holylabs (/n/holylabs) will move to new hardware. Starfish, Coldfront, and the Portal will be unavailable during the downtime. For more details including a graphical timeline, please see: **Updates will be posted here on our status page:** Note that you can subscribe to receive updates as they happen. On the status page, click Get Updates. Notices and reminders will also be sent to all users via our mailing lists. Jun 5, 13:00:00 GMT+0 - Completed - Maintenance has completed successfully Jun 2, 13:00:01 GMT+0 - Identified - Maintenance is now in progress

Starfish upgrade Thursday, May 29th from 5PM-6PM

Thu, 29 May 2025 21:00:00 +0000

Type: Maintenance

Duration: 17 hours and 27 minutes

Affected Components:

May 29, 21:00:00 GMT+0
Identified - Starfish upgrade Thursday, May 29th from 5PM-6PM. Starfish will be unavailable during that time.

May 30, 14:27:29 GMT+0
Completed - Maintenance has completed successfully..