tag:status.rc.fas.harvard.edu,2005:/history FAS Research Computing Status - Incident history 2026-06-15T13:00:00.000+00:00 FAS Research Computing tag:status.rc.fas.harvard.edu,2005:Maintenance/cmouer4mn03r4amtc5shv8576 2026-06-15T13:00:00.000+00:00 2026-06-15T13:00:00.000+00:00 2026 MGHPCC power downtime June 15-18, 2026 Type: Maintenance

Affected Components: FASRC Two-Factor (OpenAuth), HolyLFS04 (Tier 0), GPU nodes (Holyoke), Coldfront, Infiniband - Holyoke/MGHPCC, FASSE Compute Cluster (Holyoke), SLURM Scheduler - FASSE, Boston Tier 2 NFS (new), Holyoke Tier 2 NFS (new), Holylabs, Holyoke Firewall, Network - Holyoke/MGHPCC, Holyoke-Boston fiber link (short path), Infiniband - Boston, HolyLFS06 (Tier 0), bosECS, Boston Specialty Storage, Samba Cluster, Software & Modules, Holyoke/MGHPCC Data Center, Cannon Compute Cluster (Holyoke), Network - Boston, Cambridge firewall and other redundancy, FIINE billing portal, License Servers, seas_compute, Cannon Open OnDemand/VDI, FASSE login nodes, Kempner Cluster GPU, Kempner Cluster CPU, FASSE Open OnDemand/VDI, Starfish, Web Proxies, CEPH Storage Boston (Tier 2), Authentication, Globus Data Transfer, Holyoke-Boston fiber link (long path), Virtual Infrastructure - Holyoke, Holystore01 (Tier 0), Login Nodes - Holyoke, Boston Data Center, NESE (NorthEast Storage Exchange), SLURM Scheduler - Cannon, Isilon Storage Holyoke (Tier 1), holECS, Holyoke Specialty Storage, FASRC Downloads Site, Citrix, Login Nodes - Boston, HolyLFS05 (Tier 0), Virtual Infrastructure - Boston, Network - Cambridge, FASRC VPN (Cambridge) , FASRC VPN (Boston), Grafana Cloud (FASRC), BosLFS02 (Tier 0), Isilon Storage Boston (Tier 1), Home Directory Storage - Boston, portal.rc.fas.harvard.edu, Netscratch (Global Scratch), Tape - (Tier 3), Boston Compute Nodes

Jun 15, 13:00:00 GMT+0
Identified - The yearly power downtime at our Holyoke data center, MGHPCC, has been scheduled by the facility. This year's power downtime will take place on Tuesday June 15th - 18th, 2025\. There will be no June monthly maintenance as a result. Since the facility will be powered down for two days this year, we will not be performing the usual maintenance tasks. That said, networking and other key infrastructure will be doing maintenance. **IMPORTANT NOTE**: FASRC storage at both Holyoke and Boston **will be** affected and should not be expected to be available throughout the downtime. Please plan ahead accordingly. * **Monday June 15th** \- Power-down begins at 9AM * **Tuesday June 16th** \- Power out at MGHPCC * **Wednesday June 17th** \- Power out at MGHPCC * **Thursday June 18th** \- Expected return to full service by 5PM * **Friday June 19th** \- Please note that June 19th is a university holiday ![Monday June 15th -  Power-down begins at 9AM Tuesday June 16th - Power out at MGHPCC Wednesday June 17th - Power out at MGHPCC Thursday June 18th - Expected return to full service by 5PM](https://www.rc.fas.harvard.edu/wp-content/uploads/2026/05/mghpcc_powerdown_2026-1.jpg) **For more detailed information and follow-up, please see:** <https://www.rc.fas.harvard.edu/mghpcc-yearly-shutdown> **or this** [**Status Page**](https://status.rc.fas.harvard.edu/).

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmoagu0310052elrw0kbo3tbu 2026-05-18T11:00:00.000+00:00 2026-05-18T11:00:00.000+00:00 MGHPCC power work - Part 2 May 18 Type: Maintenance

Affected Components: FASSE Compute Cluster (Holyoke), GPU nodes (Holyoke), SLURM Scheduler - FASSE, Cannon Compute Cluster (Holyoke), seas_compute, Kempner Cluster GPU, Kempner Cluster CPU, SLURM Scheduler - Cannon, Boston Compute Nodes

May 18, 11:00:00 GMT+0
Identified - Our Holyoke data center, MGHPCC, will be doing power work on Row 8A. This work, which is being completed over the course of 2 weeks, will bring online another power feed which will increase power capacity. In order to do this work, it will require us to idle half the nodes in 8a for the duration of the week. This means all partitions in this row will be at half capacity. Existing jobs should drain naturally and no job should need to be canceled. The impacted partitions are: ``` arguelles_delgado_h100 bigmem bigmem_intermediate blackhole_gpu dvorkin eddy enos gershman gpu gpu_h200 gpu_requeue hejazi hernquist_ice hoekstra hsph hsph_gpu huce_ice iaifi_gpu_requeue intermediate itc_cluster itc_gpu janson_sapphire joonholee jshapiro kempner kempner_priority kempner_dev kempner_eng kempner_h200_priority kempner_h100 kempner_h100_priority kempner_h100_priority2 kempner_h100_priority3 kempner_h100_priority4 kempner_interactive kovac kozinsky kozinsky_gpu kozinsky_requeue murphy_ice mweber_compute mweber_gpu olveczky_sapphire ortegahernandez_ice rivas sapphire seas_compute siag siag_combo test yao yao_priority zhuang ```.

May 18, 11:00:00 GMT+0
Identified - Rescheduled to May 18.

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmoagr8fc0009m6hqfi4wd20x 2026-05-11T11:00:00.000+00:00 2026-05-11T11:00:00.000+00:00 MGHPCC power work - Part 1 May 11 Type: Maintenance

Duration: 5 days and 12 hours

Affected Components: GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), SLURM Scheduler - FASSE, Cannon Compute Cluster (Holyoke), seas_compute, Kempner Cluster GPU, Kempner Cluster CPU, SLURM Scheduler - Cannon, Boston Compute Nodes

May 11, 11:00:00 GMT+0
Identified - Rescheduled to May 11.

May 11, 11:00:00 GMT+0
Identified - Our Holyoke data center, MGHPCC, will be doing power work on Row 8A. This work, which will occur this week and next week, will bring online another power feed which will increase power capacity. In order to do this work, it will require us to idle half the nodes in 8a for the duration of the week. This means all partitions in this row will be at half capacity. Existing jobs should drain naturally and no job should need to be canceled. The impacted partitions are: ``` arguelles_delgado_h100 bigmem bigmem_intermediate blackhole_gpu dvorkin eddy enos gershman gpu gpu_h200 gpu_requeue hejazi hernquist_ice hoekstra hsph hsph_gpu huce_ice iaifi_gpu_requeue intermediate itc_cluster itc_gpu janson_sapphire joonholee jshapiro kempner kempner_priority kempner_dev kempner_eng kempner_h200_priority kempner_h100 kempner_h100_priority kempner_h100_priority2 kempner_h100_priority3 kempner_h100_priority4 kempner_interactive kovac kozinsky kozinsky_gpu kozinsky_requeue murphy_ice mweber_compute mweber_gpu olveczky_sapphire ortegahernandez_ice rivas sapphire seas_compute siag siag_combo test yao yao_priority zhuang ```.

May 16, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 11, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmou9kqfs000jano84c7xx9bc 2026-05-06T16:20:55.382+00:00 2026-05-06T16:20:55.581+00:00 www.rc.fas.harvard.edu is back up Type: Incident

Affected Components: www.rc.fas.harvard.edu

May 6, 16:20:55 GMT+0
Investigating - www.rc.fas.harvard.edu is down at the moment. This incident was created automatically..

May 6, 16:30:56 GMT+0
Resolved - www.rc.fas.harvard.edu is back up. This incident was resolved automatically..

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmoa68qts000ceyvghagi44uw 2026-05-04T13:00:00.000+00:00 2026-05-04T13:00:00.000+00:00 Monthly maintenance May 4th 2026 9am-1pm Type: Maintenance

Duration: 4 hours

Affected Components: FASRC Two-Factor (OpenAuth), GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), SLURM Scheduler - FASSE, Cannon Compute Cluster (Holyoke), seas_compute, Cannon Open OnDemand/VDI, FASSE login nodes, Kempner Cluster GPU, Kempner Cluster CPU, FASSE Open OnDemand/VDI, Login Nodes - Holyoke, SLURM Scheduler - Cannon, Login Nodes - Boston, Netscratch (Global Scratch), Boston Compute Nodes

May 4, 13:00:00 GMT+0
Identified - FASRC monthly maintenance will take place on May 4th 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Annual data center power downtime: The annual downtime at MGHPCC will take place June 15 - June 18\. This year's downtime will be one day longer. More details will be sent to all users next month. * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * Slurm 25.11.5 Upgrade * Audience: All cluster users * Impact: Jobs will be paused during the upgrade * Reboot remaining stuck nodes from power outage * Audience: N/A * Impact: No visible impact to user * Two-Factor/OpenAuth ([two-factor.rc.fas.harvard.edu](http://two-factor.rc.fas.harvard.edu)) replacement * Audience: All account holders * Impact: The server will be unavailable during maintenance. You will be unable to obtain a new or replacement OpenAuth token during this period. * Domain controller replacement * Audience: Internal * Impact: End users should not see any impact * OOD/Open OnDemand reboots * Audience: All OOD users, reboot of the head nodes * Impact: Running sessions will _not_ be affected * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window * Netscratch 90-day retention cleanup * Audience; All netscratch users * Impact: Files older than 90 days will be removed per our [scratch policy](https://docs.rc.fas.harvard.edu/kb/policy-scratch/). Please note that this cleanup can happen at any time, not just during maintenance. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

May 4, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

May 4, 14:56:02 GMT+0
Identified - The scheduler is re-opened and jobs un-paused. Other, non-impacting, work continues..

May 4, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmo8wbelf004s7x6qrzk183z3 2026-05-01T20:00:00.000+00:00 2026-05-01T20:00:00.000+00:00 Starfish maintenance Type: Maintenance

Duration: 1 hour

Affected Components: Starfish

May 1, 20:00:00 GMT+0
Identified - Starfish will be upgraded to the latest version on Friday, May 1st from 4pm-5pm. The service and dashboard will be down during this time. .

May 1, 20:00:01 GMT+0
Identified - Maintenance is now in progress.

May 1, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmohlvitv03efhvfzdaadj1lz 2026-04-30T12:00:00.000+00:00 2026-04-30T12:00:00.000+00:00 OpenOnDemand maintenance Type: Maintenance

Duration: 2 hours

Affected Components: Cannon Open OnDemand/VDI, FASSE Open OnDemand/VDI

Apr 30, 12:00:00 GMT+0
Identified - At 8am on Thursday April 30th we will be upgrading from Open OnDemand version 4.0.7 to 4.1.4 on both the Cannon and FASSE clusters. This is not expected to impact running jobs. This upgrade adds the Jobs->Project Manager menu item and fixes an issue that affected access to the Clusters->Shell Access menu item when using Firefox..

Apr 30, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 30, 14:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmokvaw2500ewyhgkqgzfuort 2026-04-30T02:31:25.900+00:00 2026-04-30T02:31:26.123+00:00 login.rc.fas.harvard.edu is responding normally Type: Incident

Affected Components: , Login Nodes - Holyoke, Login Nodes - Boston, , Login Nodes → login.rc.fas.harvard.edu →

Apr 30, 02:31:26 GMT+0
Investigating - login.rc.fas.harvard.edu is not responding normally. This incident was automatically created..

Apr 30, 15:43:53 GMT+0
Resolved - \\\[login.rc.fas.harvard.edu\\\](http://login.rc.fas.harvard.edu) is responding normally. This incident was automatically resolved..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmokukngw04fg48dopehgznvm 2026-04-30T02:11:01.535+00:00 2026-04-30T02:11:01.535+00:00 Login and OOD node access restricted due to serious security issue - No ETA Type: Incident

Duration: 16 hours and 2 minutes

Affected Components: Cannon Open OnDemand/VDI, FASSE login nodes, FASSE Open OnDemand/VDI, Login Nodes - Holyoke, Login Nodes - Boston

Apr 30, 02:11:01 GMT+0
Identified - **Due to a serious in-the-wild exploit which can compromise Fedora-based Linux distributions including Rocky, which is used on the cluster, we need to restrict access. All login and OOD nodes are shut down until a fix can be put in place. Jobs running on the cluster will continue running.** **No ETA, There is not fix at this time. We will update our status page in the morning once we have more information or a fix to roll out.** **This is a serious exploit and we do not take this measure lightly. Please follow this status page for updates and eventual resolution.**.

Apr 30, 14:44:58 GMT+0
Identified - We are developing a plan of attack to mitigate this exploit. Please know that this is a very serious issue and so we are treating it as such. Thank you for your understanding. We are currently awaiting further information from the Redhat/Fedora/Rocky community but building a plan in the meantime with the information we have. More details to follow as we can share them. If you need to access storage (except scratch and home directories), [Globus ](https://docs.rc.fas.harvard.edu/kb/globus-file-transfer/)is still online and available. But again, login nodes and OOD are not available..

Apr 30, 15:40:41 GMT+0
Identified - To mitigate this exploit we will need to restart -all nodes- on the cluster. This **will begin at 1PM** and run until all nodes have restarted (no ETA). This **will mean** any un-finished jobs will be **terminated**. There is no way to avoid this. We will then be validating the fix before re-opening the login. OOD nodes, and scheduler. Next steps and updates will be posted here..

Apr 30, 18:12:41 GMT+0
Resolved - The cluster has been rebooted and all nodes, including login and OOD, have been patched. The scheduler is re-opened and jobs which were preempted/requeued have priority for re-scheduling. Some non-standard, lab-owned nodes may still require patching. The owners of these machines may be contacted about this. Thank you for your patience. This is a global issue and is being addressed at centers everywhere..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmok8r4au00pamcnk0tz7d2ve 2026-04-29T16:00:11.873+00:00 2026-04-29T16:00:11.873+00:00 holylfs06 down Type: Incident

Duration: 33 minutes

Affected Components: HolyLFS06 (Tier 0)

Apr 29, 16:00:11 GMT+0
Identified - Holylfs06 storage is down. We are investigating. More details as they are known..

Apr 29, 16:32:45 GMT+0
Resolved - Holylfs06 is accessible again. This incident has been resolved..

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmoiqv16102yytc5y4eabtkxp 2026-04-28T17:00:00.000+00:00 2026-04-28T17:00:00.000+00:00 Website security maintenance (www.rc and docs.rc) 4-28-26 1pm Type: Maintenance

Duration: 17 minutes

Affected Components: docs.rc.fas.harvard.edu, www.rc.fas.harvard.edu

Apr 28, 17:00:00 GMT+0
Identified - Security updates are required for [www.rc.fas.harvard.edu](http://www.rc.fas.harvard.edu) and [docs.rc.fas.harvard.edu](http://docs.rc.fas.harvard.edu) This work will take place today between 1pm and 2pm Both sites will be down for very short periods during the updates..

Apr 28, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 28, 17:16:58 GMT+0
Completed - Website maintenance has completed successfully..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmo7omopz0041p3pjvxgqi747 2026-04-20T21:03:38.448+00:00 2026-04-20T21:03:38.448+00:00 Starfish - Out of date scans Type: Incident

Duration: 7 days and 18 hours

Affected Components: Starfish

Apr 20, 21:03:38 GMT+0
Identified - Some starfish scans restarted following earlier issue and may be up to a week out of date due to the delayed scans Two scanning agents are still down. So please bear this in mind when viewing Starfish data..

Apr 21, 13:48:40 GMT+0
Monitoring - The down agents are back in service. Scans are on-going but data will still be out of date until they catch up..

Apr 28, 15:03:28 GMT+0
Resolved - Scans are running normally. Any out of date stats will catch up shortly..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmo5pma7u0n7mjomagp7omaed 2026-04-19T11:55:47.130+00:00 2026-04-19T11:55:47.329+00:00 login.rc.fas.harvard.edu is responding normally Type: Incident

Affected Components: , Login Nodes - Holyoke, Login Nodes - Boston, , Login Nodes → login.rc.fas.harvard.edu →

Apr 19, 11:55:47 GMT+0
Investigating - login.rc.fas.harvard.edu is not responding normally. This incident was automatically created..

Apr 19, 12:55:25 GMT+0
Resolved - \\\[login.rc.fas.harvard.edu\\\](http://login.rc.fas.harvard.edu) is responding normally. This incident was automatically resolved..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmo5eigd3025we79u8go90tos 2026-04-19T06:44:52.695+00:00 2026-04-19T06:44:52.709+00:00 Authentication outage Type: Incident

Duration: 7 hours and 30 minutes

Affected Components: Authentication

Apr 19, 06:44:52 GMT+0
Investigating - Authentication issues with openauth/radius. This incident was created by an automated monitoring service..

Apr 19, 14:14:53 GMT+0
Resolved - Openauth/radius is now operational. This update was created by an automated monitoring service..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmo5mx8uq0wy7113vg0ey0q9t 2026-04-19T06:18:00.000+00:00 2026-04-19T06:18:00.000+00:00 MGHPCC Power Loss Type: Incident

Duration: 8 hours and 54 minutes

Affected Components: FASSE Compute Cluster (Holyoke), FASRC Two-Factor (OpenAuth), HolyLFS04 (Tier 0), GPU nodes (Holyoke), Infiniband - Holyoke/MGHPCC, SLURM Scheduler - FASSE, Holyoke Tier 2 NFS (new), Holylabs, Holyoke Firewall, Network - Holyoke/MGHPCC, HolyLFS06 (Tier 0), Holyoke/MGHPCC Data Center, Cannon Compute Cluster (Holyoke), seas_compute, Cannon Open OnDemand/VDI, FASSE login nodes, Kempner Cluster GPU, Kempner Cluster CPU, FASSE Open OnDemand/VDI, Virtual Infrastructure - Holyoke, Holystore01 (Tier 0), Login Nodes - Holyoke, NESE (NorthEast Storage Exchange), SLURM Scheduler - Cannon, Isilon Storage Holyoke (Tier 1), holECS, Holyoke Specialty Storage, Login Nodes - Boston, HolyLFS05 (Tier 0), Netscratch (Global Scratch), Tape - (Tier 3), Boston Compute Nodes

Apr 19, 06:18:00 GMT+0
Investigating - At 2:18am on April 19th MGHPCC (our Holyoke datacenter) lost cooling which caused the entire facility to shutdown. This caused the loss of all jobs that were running. Storage and data on that storage should be safe. The facility is working on restoring cooling and power. Unfortunately we do not have an ETA..

Apr 19, 13:06:16 GMT+0
Identified - Power was fully restored to MGHPCC at 7:39am on April 19th. FASRC staff has restored functionality to most systems except for FASSE Open OnDemand. All other services are up and operating normally. If you continue to see issues with any system that is marked operational please let us know. We will deal with any non urgent requests in normal working hours..

Apr 19, 15:12:24 GMT+0
Resolved - All services, including FASSE OOD, should be functional at this time. If you continue to see issues with any system that is marked operational please let us know by sending an email to [rchelp@rc.fas.harvard.edu](mailto:rchelp@rc.fas.harvard.edu) We will deal with any non urgent requests in normal working hours. This incident has been resolved..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmnyotr4d003l6i6q561of3ku 2026-04-13T16:30:00.000+00:00 2026-04-13T16:30:00.000+00:00 Starfish down Type: Incident

Duration: 1 day and 48 minutes

Affected Components: Starfish

Apr 13, 16:30:00 GMT+0
Investigating - Starfish is currently unavailable, due to a network card issue. Updates to come. We are currently investigating this incident..

Apr 14, 14:17:21 GMT+0
Identified - Staff will be at the datacenter today to check on the physical status of the server. Updates to come. We are continuing to work on a fix for this incident..

Apr 14, 17:17:30 GMT+0
Resolved - The network card has been replaced, and Starfish is back up. This incident has been resolved..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmnq3714o000vs7adgfjwe485 2026-04-08T13:31:31.202+00:00 2026-04-08T13:31:31.202+00:00 Coldfront is down. Type: Incident

Duration: 43 minutes

Affected Components: Coldfront

Apr 8, 13:31:31 GMT+0
Investigating - Coldfront logins are producing an error message. We are currently investigating this incident..

Apr 8, 14:14:35 GMT+0
Resolved - Coldfront is back up and accepting logins..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmnnabux30qmbq6fm9rbvrvx8 2026-04-06T14:27:54.839+00:00 2026-04-06T14:27:54.839+00:00 Starfish dashboard inaccessible Type: Incident

Duration: 1 hour and 3 minutes

Affected Components: Starfish

Apr 6, 14:27:54 GMT+0
Investigating - The Starfish dashboard is inaccessible. We are looking into the issue..

Apr 6, 15:31:02 GMT+0
Resolved - Starfish has resolved the issue and the dashboard is once again available..

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmn6ac2960e2v140x1wt0rf80 2026-04-06T13:00:00.000+00:00 2026-04-06T13:00:01.000+00:00 FASRC monthly maintenance April 6th 2026 9am-1pm Type: Maintenance

Duration: 4 hours

Affected Components: FASRC Two-Factor (OpenAuth), , , Cannon Open OnDemand/VDI, FASSE login nodes, FASSE Open OnDemand/VDI, Login Nodes - Holyoke, Login Nodes - Boston, Netscratch (Global Scratch), , Login Nodes → VDI/OpenOnDemand → login.rc.fas.harvard.edu →

Apr 6, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 6, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 6, 13:00:00 GMT+0
Identified - FASRC monthly maintenance will take place on April 6th 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **NO** FASSE cluster will be paused during this maintenance?: **NO** * [two-factor.rc.fas.harvard.edu](http://two-factor.rc.fas.harvard.edu) [OpenAuth](https://docs.rc.fas.harvard.edu/kb/openauth/) cut-over to new server * Audience: New accounts or anyone requesting an OpenAuth token * Impact: two-factor will be unavailable while moving to a new server * RStudio Server (Open OnDemand) * Audience: RStudio Server users on Cannon and FASSE * Impact: We will be decommissioning some versions of RStudio Server so we can properly maintain all production versions. Versions to be decommissioned: * R 4.1.3 (Bioconductor 3.14, RStudio 2022.02.0) * R 4.1.0 (Bioconductor 3.13, RStudio 1.4.1717) * R 4.0.3 (Bioconductor 3.12, Rstudio 1.3.1093) * R 4.0.0 (Bioconductor 3.11, Rstudio 1.3.1093) * If you use one of these versions, we recommend replacing it with the most recent version, R 4.4.2 (Bioconductor 3.20, RStudio 2024.12.0). You must reinstall previously installed libraries. * Domain controller replacement * Audience: Internal * Impact: End users should not see any impact * OOD/Open OnDemand reboots * Audience: All OOD users, reboot of the head nodes * Impact: Running sessions will _not_ be affected * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window * Netscratch 90-day retention cleanup * Audience; All netscratch users * Impact: Files older than 90 days will be removed per our [scratch policy](https://docs.rc.fas.harvard.edu/kb/policy-scratch/). Please note that this cleanup can happen at any time, not just during maintenance. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmnf48s6u02wctz2d7zg0wjmn 2026-03-31T21:15:24.557+00:00 2026-04-01T12:11:15.205+00:00 Scheduler is degraded Type: Incident

Duration: 14 hours and 56 minutes

Affected Components: , GPU nodes (Holyoke), , Cannon Compute Cluster (Holyoke), seas_compute, Kempner Cluster GPU, Kempner Cluster CPU, SLURM Scheduler - Cannon, Boston Compute Nodes, Cannon Cluster → Kempner Cluster →

Apr 1, 12:11:15 GMT+0
Resolved - This incident has been resolved. The scheduler is running normally..

Mar 31, 21:15:24 GMT+0
Investigating - The scheduler is in a degraded state due to [thrashing](https://en.wikipedia.org/wiki/Thrashing%5F%28computer%5Fscience%29) We are actively working to resolve this problem..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmnerzlwh0g8hr8lzi8oq5mr2 2026-03-31T15:32:20.956+00:00 2026-03-31T15:32:20.956+00:00 two-factor.rc.fas.harvard.edu (openauth) error Type: Incident

Duration: 1 hour and 12 minutes

Affected Components: FASRC Two-Factor (OpenAuth)

Mar 31, 15:32:20 GMT+0
Investigating - We are currently investigating this incident. Requesting a new token or re-requesting your token from two-factor is not currently working. .

Mar 31, 16:44:08 GMT+0
Resolved - This incident has been resolved. two-factor.rc.fas.harvard.edu is working normally again..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmnes00kk0cz8e9ndj1i7o3xk 2026-03-25T14:30:00.000+00:00 2026-03-25T14:30:00.000+00:00 The web front end to two-factor.rc.fas.harvard.edu is currently not allowing logins, generating new tokens is currently unavailable Type: Incident

Duration: 6 days and 30 minutes

Affected Components: FASRC Two-Factor (OpenAuth)

Mar 25, 14:30:00 GMT+0
Investigating - We are currently investigating this incident..

Mar 31, 15:00:25 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmn634bsn0co6fzrz7gjvlmtv 2026-03-25T13:34:01.221+00:00 2026-03-25T14:10:34.144+00:00 Network issues - Cluster degraded Type: Incident

Duration: 5 days, 7 hours and 7 minutes

Affected Components: , GPU nodes (Holyoke), Network - Holyoke/MGHPCC, Cannon Compute Cluster (Holyoke), seas_compute, SLURM Scheduler - Cannon, Isilon Storage Holyoke (Tier 1), Boston Compute Nodes, Cannon Cluster →

Mar 25, 14:10:34 GMT+0
Identified - Mounts to Holyoke Isilon (specifically /n/sw) are broken on numerous nodes across the cluster. We have a check rolling out to find these nodes so we can remediate them individually. Until remediated the cluster will be in a degraded state. Running jobs may randomly die or fail as they hit nodes that have stale mounts. It will be risky to run jobs for the next hour and then, after that point, the cluster will have a large number of nodes closed waiting for them to drain so we can reboot them and fix the mounts..

Mar 25, 13:34:01 GMT+0
Investigating - A network issue affecting storage critical to the cluster is It's causing instability. The cluster is currently in a degraded state as a result. We are looking into the problem. Updates to follow...

Mar 25, 14:31:18 GMT+0
Monitoring - Mounts to Holyoke Isilon (specifically /n/sw) are broken on numerous nodes across the cluster. We have a check rolling out to find these nodes so we can remediate them individually. Until remediated the cluster will be in a degraded state. Running jobs may randomly die or fail as they hit nodes that have stale mounts. It will be risky to run jobs for the next hour and then, after that point, the cluster will have a large number of nodes closed waiting for them to drain so we can reboot them and fix the mounts. At this time we are unaware of any holy-isilon problems other than the effect this had on cluster nodes/running jobs. We will update should we identify any data storage concerns..

Mar 30, 20:41:25 GMT+0
Resolved - This incident has been resolved by draining and rebooting any nodes with stuck mounts..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmmxh7njd00qve87a87odxjyz 2026-03-19T12:58:36.175+00:00 2026-03-19T12:58:36.175+00:00 ColdFront is down. Type: Incident

Duration: 1 hour and 54 minutes

Affected Components: Coldfront

Mar 19, 12:58:36 GMT+0
Identified - ColdFront is down. We are working to bring it back up. The instance got replaced last night, but it had trouble configuring itself on the way up again..

Mar 19, 14:52:54 GMT+0
Resolved - Cold front is back up. Thank you for your patience..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmmpcvot800j93wkll4ba6gwu 2026-03-13T20:35:07.736+00:00 2026-03-13T20:35:07.736+00:00 Key access issue to CSBN, HERS, FIINE, Portal Approve (p3approve) Type: Incident

Duration: 5 days, 21 hours and 59 minutes

Affected Components: portal.rc.fas.harvard.edu

Mar 13, 20:35:07 GMT+0
Investigating - We are currently investigating this incident. This only affects specific services. Users of CSBN, HERS, FIINE, Portal Approve (p3approve) may be affected. Email coming from these systems may also be delayed. No ETA.

Mar 19, 18:33:57 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmm9e0oz109onj5z9yxz1dcjr 2026-03-02T16:22:42.655+00:00 2026-03-02T16:22:42.655+00:00 Starfish dashboard unavailable Type: Incident

Duration: 1 hour and 16 minutes

Affected Components: Starfish

Mar 2, 16:22:42 GMT+0
Investigating - The Starfish dashboard is not responding. We are currently investigating this issue with the vendor..

Mar 2, 17:38:26 GMT+0
Resolved - This incident has been resolved..

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmls7rae90625fg9xy9b3jc5m 2026-03-02T14:00:00.000+00:00 2026-03-02T14:00:01.000+00:00 FASRC monthly maintenance Monday March 2nd, 2026 9am-1pm Type: Maintenance

Duration: 4 hours

Affected Components: Login Nodes - Holyoke, FASSE Compute Cluster (Holyoke), SLURM Scheduler - Cannon, , , GPU nodes (Holyoke), , SLURM Scheduler - FASSE, , Cannon Compute Cluster (Holyoke), , seas_compute, Cannon Open OnDemand/VDI, FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, FASSE Open OnDemand/VDI, Login Nodes - Boston, Netscratch (Global Scratch), , Boston Compute Nodes, Login Nodes → Cannon Cluster → VDI/OpenOnDemand → Kempner Cluster → FASSE Cluster → login.rc.fas.harvard.edu →

Mar 2, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 2, 14:00:00 GMT+0
Identified - Monthly maintenance will take place on Monday March 2nd, 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * Slurm scheduler update * Audience: All cluster users * Impact: Jobs will be paused during maintenance * OOD node reboots * Audience; All Open OnDemand users * Impact: OOD nodes will reboot during the maintenance window * Login node reboots * Audience: All login node users * Impact: Login nodes will reboot during the maintenance window * Netscratch retention purge * Audience: All users of Netscratch * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> [https://www.rc.fas.harvard.edu/](https://www.rc.fas.harvard.edu/upcoming-training/).

Mar 2, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmm5ekmnr0020fzjvxkjv1iox 2026-02-27T21:27:09.251+00:00 2026-03-04T15:09:37.681+00:00 Tape outage Type: Incident

Duration: 4 days, 17 hours and 42 minutes

Affected Components: NESE (NorthEast Storage Exchange), Tape - (Tier 3)

Mar 4, 15:09:37 GMT+0
Resolved - This incident has been resolved. Normal tape operations are restored..

Feb 27, 21:27:09 GMT+0
Investigating - NESE Tape service will be down or operating with degraded service (no store and recall) Friday from 12 Noon EST until as late as Monday, 2 March at 9 AM. SUMMARY OF ISSUE: NESE Tape service is currently not able to store or recall files to and from tape due to vendor firmware issues in the IBM TS4500 tape library. The issue is related to the library robotics and cartridge database and we do NOT expect any data loss from this issue. The issue is apparently due to an issue with the inventory database related to a recent firmware update. This database can be scrubbed and reconstructed by the library, which will scan the bar code labels on all the cartridges to rebuild the inventory. Association of files in Globus to tapes is handled separately from the tape library and is not affected by the firmware update..

Mar 2, 14:03:01 GMT+0
Identified - NESE Tape Service is still working with IBM technical support at restoring the inventory. The expected downtime is extended until Tuesday March 3rd, 9am. Apologies for the inconvenvenience..

Mar 3, 14:04:46 GMT+0
Monitoring - The tape library outage is further extended to Wednesday March 4th at 9am awaiting a hardware replacement part due today. Data can still be uploaded to lab collections via Globus, but be mindful of the 10 TB buffer file limit. The outage affects storage and recall from tape..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmm3jn9o500pvn3vlsrovaqif 2026-02-26T14:13:35.695+00:00 2026-02-27T22:04:05.371+00:00 Starfish dashboard is unavailable Type: Incident

Duration: 1 day, 7 hours and 50 minutes

Affected Components: Starfish

Feb 27, 22:04:05 GMT+0
Resolved - This incident has been resolved. The Starfish dashboard is available..

Feb 26, 14:13:35 GMT+0
Investigating - The starfish dashboard is unavailable. We are currently investigating this issue with Starfish...

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmlvay24p0nlme0oeqgve2zzk 2026-02-25T14:00:00.000+00:00 2026-02-25T14:00:00.000+00:00 Starfish maintenance Feb 25, 2026 all day Type: Maintenance

Duration: 1 day

Affected Components: Starfish

Feb 25, 14:00:00 GMT+0
Identified - Starfish will be unavailable starting Wednesday, February 25th at 9AM until Thursday, February 26th at 9AM, for routine maintenance. The online dashboard will be inaccessible during this time..

Feb 26, 14:00:00 GMT+0
Completed - Maintenance has completed successfully.

Feb 25, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmm0ruk3v0117ca6jfq7m0o6l 2026-02-24T15:39:56.874+00:00 2026-02-24T15:39:56.889+00:00 Authentication outage Type: Incident

Duration: 5 minutes

Affected Components: Authentication

Feb 24, 15:39:56 GMT+0
Investigating - Authentication issues with openauth/radius. This incident was created by an automated monitoring service..

Feb 24, 15:44:57 GMT+0
Resolved - Openauth/radius is now operational. This update was created by an automated monitoring service..

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmkx2dbd201zt5svdsbm0pm92 2026-02-19T13:00:00.000+00:00 2026-02-19T13:00:01.000+00:00 NESE tape maintenance Feb 19th 2026 Type: Maintenance

Duration: 9 hours

Affected Components: NESE (NorthEast Storage Exchange)

Feb 19, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 19, 13:00:00 GMT+0
Identified - From our partners at NESE. Details follow: We are installing four new tape frames, which will bring the tape system raw storage capacity to 253 petabytes. **Service Affected:** NESE Tape Service **Maintenance Window:** 8:00 AM - 5:00 PM (EST) * The tape service will be unavailable. * All upgrade activities are expected to be completed on the same day. NOTES: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor <https://nese.instatus.com/> for real-time updates on progress Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements.

Feb 19, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmlidbff700e2v2z4sn8zkflz 2026-02-11T16:15:00.000+00:00 2026-02-11T19:45:06.985+00:00 OOD inaccessible Type: Incident

Duration: 3 hours and 30 minutes

Affected Components: , Cannon Open OnDemand/VDI, FASSE Open OnDemand/VDI, VDI/OpenOnDemand →

Feb 11, 19:45:06 GMT+0
Resolved - This incident has been resolved and OOD is working normally..

Feb 11, 16:15:00 GMT+0
Investigating - OpenOnDemand for both Cannon and FASSE may be inaccessible for some users. Errors may include: "Error -- can't find user for <username>" "502 proxy errors" For users that are able to access OOD, performance may be degraded or sessions may get stuck. We are currently investigating the root causes of this incident. Updates to follow .

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmlfnw5qt0xla10jy3plyxdx0 2026-02-09T21:10:00.000+00:00 2026-02-09T21:10:00.000+00:00 Security updates needed for www.rc.fas.harvard.edu and docs.rc.fas.harvard.edu Type: Maintenance

Duration: 8 minutes

Affected Components: docs.rc.fas.harvard.edu, www.rc.fas.harvard.edu

Feb 9, 21:10:00 GMT+0
Identified - Security updates will require a brief interruption for our primary websites [www.rc.fas.harvard.edu](http://www.rc.fas.harvard.edu) and [docs.rc.fas.harvard.edu](http://docs.rc.fas.harvard.edu) We will endeavour to keep this update as short as possible. Each site may be unavailable for a few minutes..

Feb 9, 21:10:01 GMT+0
Identified - Maintenance is now in progress.

Feb 9, 21:17:51 GMT+0
Completed - Maintenance has completed successfully..

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmlfj7meu0zfk5p7vmpixtiwd 2026-02-09T18:54:59.922+00:00 2026-02-09T18:54:59.922+00:00 License server issue Type: Incident

Duration: 18 minutes

Affected Components: Software & Modules, , FIINE billing portal, License Servers, Starfish, FASRC Downloads Site, Citrix, Grafana Cloud (FASRC), docs.rc.fas.harvard.edu, portal.rc.fas.harvard.edu, Spinal, Bauer MiniLIMS, www.rc.fas.harvard.edu, FASRC Offsite Hosting, FASRC Ticket System (ServiceNow), Coldfront, Websites & Tools →

Feb 9, 18:54:59 GMT+0
Investigating - New sessions of Matlab are hanging. We are currently investigating this incident..

Feb 9, 19:04:52 GMT+0
Identified - The affected softwares include: Matlab Mathematica Gurobi We are continuing to work on a fix for this incident..

Feb 9, 19:13:20 GMT+0
Resolved - The license server is back up, and all software should be performing as expected. This incident has been resolved..

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmkx2d5pn01lsytjor57r552r 2026-02-09T13:00:00.000+00:00 2026-02-09T13:00:00.000+00:00 NESE tape maintenance Feb 9th 2026 Type: Maintenance

Duration: 9 hours

Affected Components: NESE (NorthEast Storage Exchange)

Feb 9, 13:00:00 GMT+0
Identified - From our partners at NESE. Details follow: In the process of the tape front-end file caching system upgrade, we will be installing a new IBM Storage Scale System 6000\. We will provide an additional update for when the software integration and data transfer from the current IBM Elastic Storage System 5000 will be performed. **Service Affected:** NESE Tape Service **Maintenance Window: No Downtime expected** NOTES: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor <https://nese.instatus.com/> for real-time updates on progress * Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements.

Feb 9, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 9, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmlbcsgmd000u3t5jsy7daltl 2026-02-06T20:44:10.404+00:00 2026-02-06T20:44:11.631+00:00 Grafana Cloud (FASRC) is down Type: Incident

Affected Components: Grafana Cloud (FASRC)

Feb 6, 20:44:11 GMT+0
Investigating - Grafana Cloud (FASRC) is down at the moment. This incident was automatically created by Instatus monitoring..

Feb 6, 20:48:52 GMT+0
Resolved - Grafana Cloud (FASRC) is back up. This incident was automatically resolved by Instatus monitoring..

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmkvgvpg4095lzxnkkb2spait 2026-02-02T14:00:00.000+00:00 2026-02-02T14:00:00.000+00:00 FASRC monthly maintenance Monday February 2nd, 2026 9am-1pm Type: Maintenance

Duration: 4 hours

Affected Components: Cannon Open OnDemand/VDI, Login Nodes - Holyoke, , , FASSE Compute Cluster (Holyoke), GPU nodes (Holyoke), , SLURM Scheduler - FASSE, , SLURM Scheduler - Cannon, Cannon Compute Cluster (Holyoke), , FASSE Open OnDemand/VDI, seas_compute, FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, Login Nodes - Boston, Netscratch (Global Scratch), , Boston Compute Nodes, Login Nodes → Cannon Cluster → VDI/OpenOnDemand → Kempner Cluster → FASSE Cluster → login.rc.fas.harvard.edu →

Feb 2, 14:00:00 GMT+0
Identified - Monthly maintenance will take place on Monday February 2nd, 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * MaxTime change * Audience: Cluster users * Impact: In order to improve scheduling efficiency and stability, we will be setting a maximum run time on all partitions that have MaxTime set to UNLIMITED to a MaxTime of 3 days. The unrestricted partition will be set to 365 days. Partitions that already have MaxTime set will retain their current setting. Partition owners wishing to set a different MaxTime for their partition should contact FASRC. Note that we do no guarantee uptime and so users should utilize checkpointing to save state in case of node failure. * Slurm upgrade to 25.11.2 * Audience: All cluster users * Impact: Jobs will be paused during maintenance * OOD node reboots * Audience; All Open OnDemand users * Impact: OOD nodes will reboot during the maintenance window * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> [https://www.rc.fas.harvard.edu/](https://www.rc.fas.harvard.edu/upcoming-training/).

Feb 2, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 2, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.rc.fas.harvard.edu,2005:Incident/cmkpl3ux008buvdviozj30q33 2026-01-22T15:06:02.990+00:00 2026-01-22T16:04:46.351+00:00 Coldfront down Type: Incident

Duration: 59 minutes

Affected Components: Coldfront

Jan 22, 16:04:46 GMT+0
Resolved - Coldfront is operational. Thank you for your patience..

Jan 22, 15:06:02 GMT+0
Investigating - We are currently investigating an issues with Coldfront. No ETA..

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmk1ivije002e49jtjj5n83yl 2026-01-12T14:00:00.000+00:00 2026-01-12T14:00:00.000+00:00 FASRC monthly maintenance Monday January 12th, 2026 9am-1pm Type: Maintenance

Duration: 4 hours

Affected Components: , , FASSE Compute Cluster (Holyoke), SLURM Scheduler - Cannon, GPU nodes (Holyoke), , SLURM Scheduler - FASSE, , Cannon Compute Cluster (Holyoke), , Cannon Open OnDemand/VDI, FASSE Open OnDemand/VDI, seas_compute, Login Nodes - Holyoke, FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, Login Nodes - Boston, , Boston Compute Nodes, Login Nodes → Cannon Cluster → VDI/OpenOnDemand → Kempner Cluster → FASSE Cluster → login.rc.fas.harvard.edu →

Jan 12, 14:00:00 GMT+0
Identified - Monthly maintenance will take place on January 12th, 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Changes to SEAS partitions, please see tasks below. * Changes to job age priority weighting, please see tasks below. * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?:**YES** * Slurm upgrade to 25.11.1 * Audience: All cluster users (Cannon and FASSE) * Impact: Jobs will be paused during maintenance * In conjunction with SEAS we will modify seas\_gpu and seas\_compute time limits * Audience: SEAS users * Impact: seas\_gpu: will be set to 2 days maximum seas\_compute: will be set to 3 days maximum Existing pending jobs longer than these limits will be set to 2 day and 3 day run times depending on partition. * Job Age Priority Weight Change * Audience: Cluster users * Impact: We will be adjusting the weight applied to the priority earned by jobs by virtue of their age. Currently job priority is made up of two factors, Fairshare and Job Age. The Job Age factor is currently set such that jobs gain priority over 3 days with a maximum priority equivalent to jobs with Fairshare of 0.5\. This keeps low fairshare jobs from languishing at the bottom of the queue. With the current settings though, users with low fairshare can gain significant advantage over users with higher relative fairshare. To remedy this we will be adjusting the Job Age weight to cap out at an equivalent Fairshare of 0.1\. This will still allow jobs with 0 fairshare to gain priority and thus not languish while letting fairshare govern a wider range of higher priority jobs. * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window * Open OnDemand (OOD) node reboots * Audienc:; All OOD users * Impact: OOD nodes will reboot during the maintenance window * Netscratch retention will run * Audience: All cluster netscratch users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> [https://www.rc.fas.harvard.edu/](https://www.rc.fas.harvard.edu/upcoming-training/).

Jan 12, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 12, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmi4yi1m400u3o9chvqic8p37 2025-12-08T11:00:00.000+00:00 2025-12-08T11:00:00.000+00:00 Monthly Maintenance and MGHPCC Power Work - Dec. 8, 2025 6am-6pm Type: Maintenance

Duration: 12 hours

Affected Components: Login Nodes - Holyoke, , GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), Isilon Storage Holyoke (Tier 1), , SLURM Scheduler - FASSE, Virtual Infrastructure - Holyoke, , Cannon Compute Cluster (Holyoke), , Cannon Open OnDemand/VDI, SLURM Scheduler - Cannon, FASSE Open OnDemand/VDI, License Servers, seas_compute, , FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, Login Nodes - Boston, Virtual Infrastructure - Boston, Isilon Storage Boston (Tier 1), , Boston Compute Nodes, Login Nodes → VDI/OpenOnDemand → Kempner Cluster → FASSE Cluster → Cannon Cluster → login.rc.fas.harvard.edu →

Dec 8, 11:00:00 GMT+0
Identified - Monthly maintenance will take place on December 8th. Our maintenance tasks should be completed between 9am-1pm. However: _Additionally_, MGHPCC will be performing power upgrades on the odd side of Row 8A where much of our computer resides. This is the final upgrade for this row. Current estimate for this work is a 12 hour window 6am-6pm. A list of the affected partitions is provided at the bottom of this notice. The nodes in those partitions will be drained prior to the work and will be powered down. Once the work is completed, those nodes will be returned to service. **Notices:** * New FASSE partition `fasse_gpu_h200`. This partitions has 2 H200 nodes and a 3day limit. It is available now. * 11/26 - 11/28 are university holidays (Thanksgiving). No on-site support, FASRC staff will return on 12/1. * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** FASSE cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** * Power work on Row 8A odd * Audience: Users of the partitions listed below * Impact: These nodes and partitions will be fully or partially down all day * OneFS (Isilon) upgrade * Audience: All Isilon (Tier 1) shares * Impact: Some VMs will be impacted including Cannon OOD, CBScentral, MCZapps/MCZbase, Portal, and Rclic1 (license server) * Slurm upgrade to 25.05.5 * Audience: All cluster users * Impact: Jobs will be paused during maintenance * Login node reboots * Audience: All login node users * Impact: Login nodes will reboot during the maintenance window **Impacted Cannon Partitions (Full or Partial Outage):** * arguelles\_delgado\_gpu\_a100 * arguelles\_delgado\_gpu\_mixed * bigmem\_intermediate * blackhole\_gpu * eddy * gershman * gpu\_requeue * hejazi * hernquist\_ice * hoekstra * huce\_ice * iaifi\_gpu * iaifi\_gpu\_priority * iaifi\_gpu\_requeue * itc\_gpu * jshapiro * kempner * kempner\_dev * kempner\_priority * kempner\_h100 * kempner\_h100\_priority * kempner\_h100\_priority2 * kempner\_h100\_priority3 * kempner\_interactive * kempner\_requeue * kovac * kozinsky * kozinsky\_gpu * kozinsky\_priority * kozinsky\_requeue * murphy\_ice * ortegahernandez\_ice * rivas * seas\_compute * seas\_gpu * serial\_requeue * siag\_combo * siag\_gpu * sur * zhuang.

Dec 8, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 8, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmisykt3v0ayak7rmnsl4btnt 2025-12-05T14:00:00.000+00:00 2025-12-05T14:00:00.000+00:00 holylfs04 migrations Type: Maintenance

Duration: 4 days, 1 hour and 10 minutes

Affected Components: HolyLFS04 (Tier 0)

Dec 5, 14:00:00 GMT+0
Identified - The holylfs04 migration to holylfs06 has begun. All holylfs04 folders will be **read-only** for the duration of the migration, from **Friday, December 5th at 9AM until end of day on Monday, December 8th.** All labs with holylfs04 have been informed via email; please email [rdm@rc.fas.harvard.edu](mailto:rdm@rc.fas.harvard.edu) if you have any questions..

Dec 9, 15:09:30 GMT+0
Completed - Maintenance has completed successfully..

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmhjab8ow00xbfmtmh0g1qmos 2025-12-01T11:00:00.000+00:00 2025-12-01T11:00:00.000+00:00 NESE tape system maintenance 12/1/25-12/5/25 Type: Maintenance

Duration: 5 days

Affected Components: NESE (NorthEast Storage Exchange), Tape - (Tier 3)

Dec 1, 11:00:00 GMT+0
Identified - NESE, the Northeast Storage Exchange at MGHPCC which supplies the Tier3 tape service used by FASRC, will be offline for maintenance on the system Dec 1st - 5th. There will be ongoing performance-affecting maintenance until Dec 12th. Please see below for details. WHO: Any lab who has or is moving data to tape. IMPACT: No access 12/1/25 - 12/5/25\. Reduced performance 12/5/25 - 12/12/25. > NESE tape system maintenance and major software upgrade is scheduled to begin on December 1, 2025\. As a result, the NESE Tape service will be offline from December 1 to December 5. > > Starting December 8 through December 12, the service will be back online with reduced performance. All maintenance activities are planned to conclude on December 12, 2025. > > * Monitor: <https://nese.instatus.com/> for real-time updates on progress > * Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements.

Dec 1, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 6, 11:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmhnyg8g701g4yv5cvuwz2hjz 2025-11-14T22:00:00.000+00:00 2025-11-14T22:00:01.000+00:00 Starfish dashboard maintenance Nov. 14th 5-6PM Type: Maintenance

Duration: 1 hour

Affected Components: Starfish

Nov 14, 22:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 14, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

Nov 14, 22:00:00 GMT+0
Identified - There is a planned upgrade of the Starfish dashboard scheduled for Friday November 14th starting at 5PM. The dashboard will be down for an hour while the upgrade is performed..

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmggy7q9801vdtke67ycm4dxq 2025-11-03T11:00:00.000+00:00 2025-11-03T11:00:00.000+00:00 Monthly Maintenance and MGHPCC Power Work - Nov. 3, 2025 6am-6pm Type: Maintenance

Duration: 9 hours and 4 minutes

Affected Components: , , SLURM Scheduler - FASSE, , Kempner Cluster CPU, Cannon Compute Cluster (Holyoke), , Cannon Open OnDemand/VDI, SLURM Scheduler - Cannon, FASSE Open OnDemand/VDI, seas_compute, Kempner Cluster GPU, Login Nodes - Holyoke, , GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), FASSE login nodes, Login Nodes - Boston, Netscratch (Global Scratch), Boston Compute Nodes, Login Nodes → VDI/OpenOnDemand → Kempner Cluster → FASSE Cluster → Cannon Cluster →

Nov 3, 11:00:00 GMT+0
Identified - Monthly maintenance will take place on November 3rd. Additionally, MGHPCC will be performing power upgrades on the even side of Row 8A where much of our computer resides. A further upgrade will take place Dec. 8th on the odd side. A list of the affected partitions is provided at the bottom of this notice. The nodes in those partitions will be drained prior to the work and will be powered down. Once the work is completed, those nodes will be returned to service. Current estimate is a 12 hour window. We will adjust as we know more. **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** FASSE cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** * Power work on Row 8A Even * Audience: Users of the partitions listed below * Impact: These nodes and partitions will be fully or partially down all day * Slurm upgrade to 25.05.4 * Audience: All cluster users * Impact: Jobs will be paused during maintenance * Block repo.anaconda.com cluster wide * Audience: Anyone attempting to use repo.anaconda.com * Impact: This change should not impact your Python workflow on the cluster. But if it does, consider using the open-source channel, `conda-forge`, through Miniforge distribution to install Python packages. This can be done by following our instructions on <https://docs.rc.fas.harvard.edu/kb/python-package-installation/> * Change Slurm User to Local User * Audience: All cluster users * Impact: Behind the scenes. No impact to users * Login node reboots (morning) * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will rebooted during this maintenance window * Netscratch cleanup ( <https://docs.rc.fas.harvard.edu/kb/policy-scratch/> ) * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. **AFFECTED PARTITIONS** Nov. 3, 2025 - All Day Power Work Partial or Full Outage Apples to: arguelles\_delgado\_h100 bigmem dvorkin eddy enos gpu gpu\_h200 gpu\_requeue hsph hsph\_gpu intermediate itc\_cluster joonholee jshapiro kempner\_dev kemkpner\_eng kempner\_requeue mweber\_compute mweber\_gpu olveczky\_sapphire sapphire seas\_compute seas\_gpu serial\_requeue yao yao\_gpu yao\_priority test.

Nov 3, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 3, 20:04:13 GMT+0
Completed - Maintenance has completed successfully including power work at MGHPCC. A reminder that additional all-day power work will take place on Dec 8th, along with our maintenance from 9am-1pm.

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmg1cda8s098z4gs4ylropic7 2025-10-06T13:00:00.000+00:00 2025-10-06T13:00:01.000+00:00 FASRC monthly maintenance Monday October 6th, 2025 9am-1pm Type: Maintenance

Duration: 4 hours

Affected Components: , Network - Cambridge, Network - Boston, Network - Holyoke/MGHPCC, Login Nodes - Boston, Netscratch (Global Scratch), Login Nodes - Holyoke, FASSE login nodes, Login Nodes →

Oct 6, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 6, 13:00:00 GMT+0
Identified - FASRC monthly maintenance will take place Monday October 6th, 2025 from 9am-1pm **NOTICES** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * Upcoming holidays: Columbus / Indigenous Peoples’ Day - October 13 **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **NO** FASSE cluster will be paused during this maintenance?: **NO** * DNS server reboots * Audience: All FASRC services * Impact: Rolling reboot should have no impact * Login node reboots * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will rebooted during this maintenance window * Netscratch cleanup ( <https://docs.rc.fas.harvard.edu/kb/policy-scratch/> ) * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

Oct 6, 17:00:00 GMT+0
Completed - Maintenance is now in progress.

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmg5awjiw04n710wcsl3zi0gv 2025-09-30T06:00:00.000+00:00 2025-09-30T07:00:00.000+00:00 VPN concentrator rolling updates overnight Type: Maintenance

Sep 30, 07:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 30, 06:00:00 GMT+0
Completed - Networking will be patching the VPN concentrators overnight. This will be done in a rolling order so that one ore more are always online. This may cause active VPN connections to drop, but they can be re-connected shortly after. ETA is one hour total..

Sep 30, 06:00:00 GMT+0
Identified - Networking will be patching the VPN concentrators overnight. This will be done in a rolling order so that one ore more are always online. This may cause active VPN connections to drop, but they can be re-connected shortly after. ETA is one hour total..

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmevo0dad0012rwy6qv5mv34j 2025-09-08T13:00:00.000+00:00 2025-09-08T13:00:01.000+00:00 FASRC monthly maintenance Monday September 8th, 2025 9am-1pm Type: Maintenance

Duration: 4 hours

Affected Components: Kempner Cluster CPU, Kempner Cluster GPU, , Cannon Compute Cluster (Holyoke), Boston Compute Nodes, Netscratch (Global Scratch), , , SLURM Scheduler - FASSE, , GPU nodes (Holyoke), Login Nodes - Boston, FASSE login nodes, Login Nodes - Holyoke, , FASSE Open OnDemand/VDI, Cannon Open OnDemand/VDI, SLURM Scheduler - Cannon, FASSE Compute Cluster (Holyoke), seas_compute, Login Nodes → Cannon Cluster → FASSE Cluster → VDI/OpenOnDemand → Kempner Cluster →

Sep 8, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Sep 8, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Sep 8, 13:00:00 GMT+0
Identified - FASRC monthly maintenance will take place Monday September 8th, 2025 from 9am-1pm **NOTICES** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * Upcoming holidays: Labor Day, Monday September 1st **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * Slurm Upgrade to 25.05.2 * Audience: All cluster users * Impact: Jobs and the scheduler will be paused during this upgrade * Domain controller work * Audience: Internal network * Impact: No impact expected * Login node reboots * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will rebooted during this maintenance window * Netscratch cleanup ( <https://docs.rc.fas.harvard.edu/kb/policy-scratch/> ) * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cmehjmf9q000dam879fafq3vq 2025-08-20T21:00:00.000+00:00 2025-08-20T21:00:01.000+00:00 Starfish upgrade Wednesday, August 20th 5PM-7PM Type: Maintenance

Duration: 2 hours

Affected Components: Starfish

Aug 20, 21:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 20, 21:00:00 GMT+0
Identified - Starfish will be performing an upgrade on Wednesday, August 20th from 5PM-7PM. The web interface will be unavailable during that timeframe..

Aug 20, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

]]> tag:status.rc.fas.harvard.edu,2005:Maintenance/cme1un2le0626okcc24iokqze 2025-08-11T10:00:00.000+00:00 2025-08-11T10:00:00.000+00:00 SEAS: seas_gpu partition GPU upgrades 8/11 - 8/14 Type: Maintenance

Duration: 4 days, 9 hours and 39 minutes

Affected Components: seas_compute

Aug 11, 10:00:00 GMT+0
Identified - Between 8/11/25 6AM - 8/14/25 5PM FASRC will be upgrading 14 of the H100 GPU nodes in the `seas_gpu `partition to H200 GPUs. This sill also affect `mweber_gpu` A reservation has been set which will drain the nodes of jobs prior to the maintenance. The SEAS GPU partition will be running at 75% capacity during these updates. FASRC has hundreds of GPU, so users should feel free to utilize `gpu_requeue` if needed for their jobs. Affected nodes: `mweber_gpu` nodes (13): ``` holygpu8a[18204,18301-18304,18401-18404,18501-18502,18601-18602] ``` seas\_gpu nodes (14): ``` holygpu8a[16101-16104,16201-16204,16301-16304,16401-16402] ``` Please reach out to [rchelp@rc.fas.harvard.edu](mailto:rchelp@rc.fas.harvard.edu) if you have any questions or concerns..

Aug 15, 14:13:24 GMT+0
Identified - Maintenance is still in progress - imaging of the new H200 nodes is ongoing. Current ETA is end of day Friday. For further questions, please contact rchelp@rc.fas.harvard.edu.

Aug 15, 19:39:27 GMT+0
Completed - Maintenance has completed successfully. SEAS H200 nodes have been imaged and are back in service. .

Aug 11, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

]]>