FAS Research Computing - Tape outage – Incident details

All systems operational

Status page for the Harvard FAS Research Computing cluster and other resources.

Cluster Utilization (VPN and FASRC login required): Cannon | FASSE


Please scroll down to see details on any Incidents or maintenance notices.
Monthly maintenance occurs on the first Monday of the month (except holidays).

GETTING HELP
Documentation: https://docs.rc.fas.harvard.edu | Account Portal https://portal.rc.fas.harvard.edu
Email: rchelp@rc.fas.harvard.edu | Support Hours

The colors shown in the bars below were chosen to increase visibility for color-blind visitors.
For higher contrast, switch to light mode at the bottom of this page if the background is dark and colors are muted.

Tape outage

Resolved
Degraded performance
Started 10 days agoLasted 5 days

Affected

Storage

Major outage from 9:27 PM to 2:04 PM, Degraded performance from 2:04 PM to 3:09 PM

Tape - (Tier 3)

Major outage from 9:27 PM to 2:04 PM, Degraded performance from 2:04 PM to 3:09 PM

External Resources - Service Providers

Major outage from 9:27 PM to 2:04 PM, Degraded performance from 2:04 PM to 3:09 PM

NESE (NorthEast Storage Exchange)

Major outage from 9:27 PM to 2:04 PM, Degraded performance from 2:04 PM to 3:09 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved. Normal tape operations are restored.

  • Monitoring
    Monitoring

    The tape library outage is further extended to Wednesday March 4th at 9am awaiting a hardware replacement part due today. Data can still be uploaded to lab collections via Globus, but be mindful of the 10 TB buffer file limit. The outage affects storage and recall from tape.

  • Identified
    Identified

    NESE Tape Service is still working with IBM technical support at restoring the inventory. The expected downtime is extended until Tuesday March 3rd, 9am.
    Apologies for the inconvenvenience.

  • Investigating
    Investigating

    NESE Tape service will be down or operating with degraded service (no store and recall) Friday from 12 Noon EST until as late as Monday, 2 March at 9 AM.

    SUMMARY OF ISSUE:

    NESE Tape service is currently not able to store or recall files to and from tape due to vendor firmware issues in the IBM TS4500 tape library. The issue is related to the library robotics and cartridge database and we do NOT expect any data loss from this issue.

    The issue is apparently due to an issue with the inventory database related to a recent firmware update. This database can be scrubbed and reconstructed by the library, which will scan the bar code labels on all the cartridges to rebuild the inventory. Association of files in Globus to tapes is handled separately from the tape library and is not affected by the firmware update.

For issues not shown here, please contact FASRC via
https://portal.rc.fas.harvard.edu or email rchelp@rc.fas.harvard.edu