FAS Research Computing - Notice history

System Under Maintenance

Status page for the Harvard FAS Research Computing cluster and other resources.

Cluster Utilization (VPN and FASRC login required): Cannon | FASSE


Please scroll down to see details on any Incidents or maintenance notices.
Monthly maintenance occurs on the first Monday of the month (except holidays).

GETTING HELP
Documentation: https://docs.rc.fas.harvard.edu | Account Portal https://portal.rc.fas.harvard.edu
Email: rchelp@rc.fas.harvard.edu | Support Hours

The colors shown in the bars below were chosen to increase visibility for color-blind visitors.
For higher contrast, switch to light mode at the bottom of this page if the background is dark and colors are muted.
Under maintenance

SLURM Scheduler - Cannon - Under maintenance

Cannon Compute Cluster (Holyoke) - Under maintenance

Boston Compute Nodes - Under maintenance

GPU nodes (Holyoke) - Under maintenance

seas_compute - Under maintenance

Under maintenance

SLURM Scheduler - FASSE - Under maintenance

FASSE Compute Cluster (Holyoke) - Under maintenance

Under maintenance

Kempner Cluster CPU - Under maintenance

Kempner Cluster GPU - Under maintenance

Under maintenance

FASSE login nodes - Under maintenance

Under maintenance

Cannon Open OnDemand/VDI - Under maintenance

FASSE Open OnDemand/VDI - Under maintenance

Under maintenance

Netscratch (Global Scratch) - Under maintenance

Home Directory Storage - Boston - Under maintenance

Tape - (Tier 3) - Under maintenance

Holylabs - Under maintenance

Isilon Storage Holyoke (Tier 1) - Under maintenance

Holystore01 (Tier 0) - Under maintenance

HolyLFS04 (Tier 0) - Under maintenance

HolyLFS05 (Tier 0) - Under maintenance

HolyLFS06 (Tier 0) - Under maintenance

Holyoke Tier 2 NFS (new) - Under maintenance

Holyoke Specialty Storage - Under maintenance

holECS - Under maintenance

Isilon Storage Boston (Tier 1) - Under maintenance

BosLFS02 (Tier 0) - Under maintenance

Boston Tier 2 NFS (new) - Under maintenance

CEPH Storage Boston (Tier 2) - Under maintenance

Boston Specialty Storage - Under maintenance

bosECS - Under maintenance

Samba Cluster - Under maintenance

Globus Data Transfer - Under maintenance

Notice history

Mar 2022

Websites & Tools - MiniLIMs (FAS Informatics) isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Websites & Tools - MiniLIMs (FAS Informatics) is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Websites & Tools - MiniLIMs (FAS Informatics) cannot be accessed at the moment. This incident was created by an automated monitoring service.

Long path fibre to MGHPCC Holyoke is also down
  • Resolved
    Resolved

    The long path has returned to operation and initial checks indicate no substantial job or cluster issues as a result.

    The short path remains down, so latency between Boston and Holyoke will still be higher than normal. See short path incident for updates as that progresses (no ETA at this time)

  • Identified
    Identified

    The short path fibre connection to Holyoke/MGHPCC suffered a failure late yesterday (3/15/22). The short path is still down as of this morning.

    However, we have just seen the long path also drop connection and are awaiting more information on this incident. It is our hope this is a minor disconnect related to their repair work.

The short path fibre to MGHPCC Holyoke is down - long path remains up
  • Resolved
    Resolved

    The short path repair is complete. Normal network operation and speed/latency between Boston and Holyoke have been restored.

  • Identified
    Identified

    Additional note: The long path was briefly down, but was quickly restored. No knock-on effects observed and long path is fully operational.

    No ETA on short path repair.

  • Investigating
    Investigating

    We've been informed that the 'short path' fibre connection to Holyoke is down. No ETA.

    The 'long path' remains in operation, but this means slightly more latency when performing operations between Boston and Holyoke.

    We will relay updates when we receive them.

Websites & Tools - MiniLIMs (FAS Informatics) isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Websites & Tools - MiniLIMs (FAS Informatics) is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Websites & Tools - MiniLIMs (FAS Informatics) cannot be accessed at the moment. This incident was created by an automated monitoring service.

FASRC ticket system - tickets and replies delayed
  • Resolved
    Resolved

    A fix was implemented earlier and after monitoring this is deemed resolved.

  • Investigating
    Investigating

    We are currently working on this issue and expect the ticket system back in operation shortly.

    New tickets and replies should be queued and will get processed as soon as the system is back to normal.

Feb 2022

Websites & Tools - MiniLIMs (FAS Informatics) isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Websites & Tools - MiniLIMs (FAS Informatics) is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Websites & Tools - MiniLIMs (FAS Informatics) cannot be accessed at the moment. This incident was created by an automated monitoring service.

Websites & Tools - MiniLIMs (FAS Informatics) isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Websites & Tools - MiniLIMs (FAS Informatics) is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Websites & Tools - MiniLIMs (FAS Informatics) cannot be accessed at the moment. This incident was created by an automated monitoring service.

Academic Cluster home directory storage full
  • Resolved
    Resolved

    This issue has been resolved.

  • Identified
    Identified

    The home directory storage for the academic cluster has quickly filled to capacity. We are working to mitigate this issue.

Websites & Tools - Portal (portal.rc) isn't accessible at the moment (automated)
  • Resolved
    Resolved

    Websites & Tools - Portal (portal.rc) is now operational! This update was created by an automated monitoring service.

  • Investigating
    Investigating

    Websites & Tools - Portal (portal.rc) cannot be accessed at the moment. This incident was created by an automated monitoring service.

holystore01 mounts on boslogin nodes
  • Resolved
    Resolved

    This issue has been resolved and holystore01 is back to normal.

  • Monitoring
    Monitoring

    boslogin nodes are up but are unable to mount holystore01 at this time. Replacement parts for holystore01 will be installed at the earliest possible time to rectify this issue.

  • Identified
    Identified

    holystore01 may be inaccessible on some cluster nodes.

  • Investigating
    Investigating

    holystore01 is not currently accessible from boslogin nodes. These nodes will be rebooted again at 2:30pm to fix.

    Please login to holylogin nodes if you need to access holystore01.

Jan 2022

Tier3 Tape system maintenance/updates
  • Completed
    February 01, 2022 at 2:49 PM
    Completed
    February 01, 2022 at 2:49 PM

    Maintenance has completed successfully.

  • Planned
    January 31, 2022 at 3:00 PM
    Planned
    January 31, 2022 at 3:00 PM

    The NESE tape system, which is used for our Tier3 tape storage, will be under maintenance Monday January 31st from 10am - 12pm

    The Tier3 tape system will be unavailable during that period. Please plan accordingly.

Globus holyoke endpoint is not responding
  • Resolved
    Resolved

    Hung mount has been fixed and the Holyoke endpoint is accessible again.

  • Investigating
    Investigating

    The Harvard FAS RC Holyoke globus endpoint is not responding.

    We are investigating

Holyoke VM issues - affects FASSE login and VDI, and Academic cluster
  • Resolved
    Resolved

    Holyoke VM issues have been resolved.

  • Identified
    Identified

    The previous issue with the Holyoke VM infrastructure has returned. We are working to resolve as quickly as possible.

VM issue affecting FASSE VDI and Tier2 storage
  • Resolved
    Resolved

    We have found the root issue and restored functionality. We will continue to monitor, but services are restored.

    Email rchelp@rc.fas.harvard.edu if you still have issues with FASSE VDI/login or Tier2 storage.

  • Identified
    Identified

    We are still working on the underlying issue.

    A correction: This does not affect Academic VDI or vdi.rc. It primarily affects FASSE VDI and some Tier2 storage.

  • Investigating
    Investigating

    There is an issue with several VMs (virtual machines) including the VDI (Open OnDemand) nodes.

    We are investigating and will update here as this issue progresses.

    Please not that this affects the Academic cluster as well as FASSE VDI, NCF VDI, and VDI.rc

Winter maintenance - January 5th 8am-5pm
  • Completed
    January 05, 2022 at 7:56 PM
    Completed
    January 05, 2022 at 7:56 PM

    Maintenance has completed successfully.

  • Update
    January 05, 2022 at 7:52 PM
    Update
    January 05, 2022 at 7:52 PM

    All data center work is complete.

    Remaining tasks are:

    • rebooting login and VDI nodes
    • running scratch 90-day retention
  • In progress
    January 05, 2022 at 5:51 PM
    In progress
    January 05, 2022 at 5:51 PM

    Maintenance on-track.

    Several smaller tasks are complete, but water cooling maintenance is the larger task today and still on-going.

  • Planned
    January 05, 2022 at 1:00 PM
    Planned
    January 05, 2022 at 1:00 PM

    Our Winter maintenance will occur on January 5th, from 8am - 5pm. This will be an all-day event and will require that jobs will be suspended for the duration and scratch (holyscratch01) will be offline for updates.

    Storage firmware will also be updated that day, potentially in both data centers, so short storage interruptions may occur.

    This is one of the two times each year we can plan maintenance around our Boston data center and also make changes that involve maintaining, modifying, or adding to the water-cooling system in MGHPCC.

    Tasks will include:

    • Water cooling maintenance in MGHPCC row 7-C
    • Slurm 21.08.5 Upgrade (jobs paused during maintenance)
    • Firmware upgrades
    • Scratch (holyscratch01) maintenance (offline during maintenance)
    • Login and VDI node reboots
    • Scratch 90-day retention cleanup

Jan 2022 to Mar 2022

Next

For issues not shown here, please contact FASRC via
https://portal.rc.fas.harvard.edu or email rchelp@rc.fas.harvard.edu