OAuth Authorization
Public and partner API endpoints, authentication, quotas, and developer tooling.
Maintenance notice
A small number of nonessential services may have limited performance while routine background work completes.
Starts: Oct 14, 2026, 1:15 PM UTC
Ends: Oct 14, 2026, 9:15 PM UTC
Current system health
Live health information for distribution, metadata, media, APIs, partner services, and internal infrastructure.
Live summary
2 services with limited performance
Last refreshed moments ago
Component directory
Manual component overrides, maintenance notices, and incident records are administered through the internal operations dashboard. The base operational state is Online.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Public and partner API endpoints, authentication, quotas, and developer tooling.
Routine background maintenance until 9:15 PM
Spotify endpoint monitoring for ddex ern release delivery.
Spotify endpoint monitoring for resource and asset delivery.
Spotify endpoint monitoring for metadata update and takedown.
Spotify endpoint monitoring for acknowledgement and receipt.
Apple Music endpoint monitoring for ddex ern release delivery.
Apple Music endpoint monitoring for resource and asset delivery.
Apple Music endpoint monitoring for metadata update and takedown.
Apple Music endpoint monitoring for acknowledgement and receipt.
iTunes Store endpoint monitoring for ddex ern release delivery.
iTunes Store endpoint monitoring for resource and asset delivery.
iTunes Store endpoint monitoring for metadata update and takedown.
iTunes Store endpoint monitoring for acknowledgement and receipt.
Amazon Music endpoint monitoring for ddex ern release delivery.
Amazon Music endpoint monitoring for resource and asset delivery.
Amazon Music endpoint monitoring for metadata update and takedown.
Amazon Music endpoint monitoring for acknowledgement and receipt.
YouTube Music endpoint monitoring for ddex ern release delivery.
YouTube Music endpoint monitoring for resource and asset delivery.
YouTube Music endpoint monitoring for metadata update and takedown.
YouTube Music endpoint monitoring for acknowledgement and receipt.
YouTube Content ID endpoint monitoring for ddex ern release delivery.
YouTube Content ID endpoint monitoring for resource and asset delivery.
YouTube Content ID endpoint monitoring for metadata update and takedown.
YouTube Content ID endpoint monitoring for acknowledgement and receipt.
TikTok endpoint monitoring for ddex ern release delivery.
TikTok endpoint monitoring for resource and asset delivery.
TikTok endpoint monitoring for metadata update and takedown.
TikTok endpoint monitoring for acknowledgement and receipt.
TikTok Commercial Music Library endpoint monitoring for ddex ern release delivery.
TikTok Commercial Music Library endpoint monitoring for resource and asset delivery.
TikTok Commercial Music Library endpoint monitoring for metadata update and takedown.
TikTok Commercial Music Library endpoint monitoring for acknowledgement and receipt.
Deezer endpoint monitoring for ddex ern release delivery.
Deezer endpoint monitoring for resource and asset delivery.
Deezer endpoint monitoring for metadata update and takedown.
Deezer endpoint monitoring for acknowledgement and receipt.
TIDAL endpoint monitoring for ddex ern release delivery.
TIDAL endpoint monitoring for resource and asset delivery.
TIDAL endpoint monitoring for metadata update and takedown.
TIDAL endpoint monitoring for acknowledgement and receipt.
Pandora endpoint monitoring for ddex ern release delivery.
Pandora endpoint monitoring for resource and asset delivery.
Pandora endpoint monitoring for metadata update and takedown.
Pandora endpoint monitoring for acknowledgement and receipt.
SoundCloud endpoint monitoring for ddex ern release delivery.
SoundCloud endpoint monitoring for resource and asset delivery.
SoundCloud endpoint monitoring for metadata update and takedown.
SoundCloud endpoint monitoring for acknowledgement and receipt.
Audiomack endpoint monitoring for ddex ern release delivery.
Audiomack endpoint monitoring for resource and asset delivery.
Audiomack endpoint monitoring for metadata update and takedown.
Audiomack endpoint monitoring for acknowledgement and receipt.
Anghami endpoint monitoring for ddex ern release delivery.
Anghami endpoint monitoring for resource and asset delivery.
Anghami endpoint monitoring for metadata update and takedown.
Anghami endpoint monitoring for acknowledgement and receipt.
Boomplay endpoint monitoring for ddex ern release delivery.
Boomplay endpoint monitoring for resource and asset delivery.
Boomplay endpoint monitoring for metadata update and takedown.
Boomplay endpoint monitoring for acknowledgement and receipt.
JioSaavn endpoint monitoring for ddex ern release delivery.
JioSaavn endpoint monitoring for resource and asset delivery.
JioSaavn endpoint monitoring for metadata update and takedown.
JioSaavn endpoint monitoring for acknowledgement and receipt.
Gaana endpoint monitoring for ddex ern release delivery.
Gaana endpoint monitoring for resource and asset delivery.
Gaana endpoint monitoring for metadata update and takedown.
Gaana endpoint monitoring for acknowledgement and receipt.
Hungama Music endpoint monitoring for ddex ern release delivery.
Hungama Music endpoint monitoring for resource and asset delivery.
Hungama Music endpoint monitoring for metadata update and takedown.
Hungama Music endpoint monitoring for acknowledgement and receipt.
KKBOX endpoint monitoring for ddex ern release delivery.
KKBOX endpoint monitoring for resource and asset delivery.
KKBOX endpoint monitoring for metadata update and takedown.
KKBOX endpoint monitoring for acknowledgement and receipt.
LINE MUSIC endpoint monitoring for ddex ern release delivery.
LINE MUSIC endpoint monitoring for resource and asset delivery.
LINE MUSIC endpoint monitoring for metadata update and takedown.
LINE MUSIC endpoint monitoring for acknowledgement and receipt.
AWA endpoint monitoring for ddex ern release delivery.
AWA endpoint monitoring for resource and asset delivery.
AWA endpoint monitoring for metadata update and takedown.
AWA endpoint monitoring for acknowledgement and receipt.
Rakuten Music endpoint monitoring for ddex ern release delivery.
Rakuten Music endpoint monitoring for resource and asset delivery.
Rakuten Music endpoint monitoring for metadata update and takedown.
Rakuten Music endpoint monitoring for acknowledgement and receipt.
RecoChoku endpoint monitoring for ddex ern release delivery.
RecoChoku endpoint monitoring for resource and asset delivery.
RecoChoku endpoint monitoring for metadata update and takedown.
RecoChoku endpoint monitoring for acknowledgement and receipt.
Tencent Music endpoint monitoring for ddex ern release delivery.
Tencent Music endpoint monitoring for resource and asset delivery.
Tencent Music endpoint monitoring for metadata update and takedown.
Routine background maintenance until 9:15 PM
Tencent Music endpoint monitoring for acknowledgement and receipt.
QQ Music endpoint monitoring for ddex ern release delivery.
QQ Music endpoint monitoring for resource and asset delivery.
QQ Music endpoint monitoring for metadata update and takedown.
QQ Music endpoint monitoring for acknowledgement and receipt.
Kugou Music endpoint monitoring for ddex ern release delivery.
Kugou Music endpoint monitoring for resource and asset delivery.
Kugou Music endpoint monitoring for metadata update and takedown.
Kugou Music endpoint monitoring for acknowledgement and receipt.
Kuwo Music endpoint monitoring for ddex ern release delivery.
Kuwo Music endpoint monitoring for resource and asset delivery.
Kuwo Music endpoint monitoring for metadata update and takedown.
Kuwo Music endpoint monitoring for acknowledgement and receipt.
NetEase Cloud Music endpoint monitoring for ddex ern release delivery.
NetEase Cloud Music endpoint monitoring for resource and asset delivery.
NetEase Cloud Music endpoint monitoring for metadata update and takedown.
NetEase Cloud Music endpoint monitoring for acknowledgement and receipt.
Melon endpoint monitoring for ddex ern release delivery.
Melon endpoint monitoring for resource and asset delivery.
Melon endpoint monitoring for metadata update and takedown.
Melon endpoint monitoring for acknowledgement and receipt.
Genie Music endpoint monitoring for ddex ern release delivery.
Genie Music endpoint monitoring for resource and asset delivery.
Genie Music endpoint monitoring for metadata update and takedown.
Genie Music endpoint monitoring for acknowledgement and receipt.
FLO endpoint monitoring for ddex ern release delivery.
FLO endpoint monitoring for resource and asset delivery.
FLO endpoint monitoring for metadata update and takedown.
FLO endpoint monitoring for acknowledgement and receipt.
VIBE endpoint monitoring for ddex ern release delivery.
VIBE endpoint monitoring for resource and asset delivery.
VIBE endpoint monitoring for metadata update and takedown.
VIBE endpoint monitoring for acknowledgement and receipt.
Bugs! endpoint monitoring for ddex ern release delivery.
Bugs! endpoint monitoring for resource and asset delivery.
Bugs! endpoint monitoring for metadata update and takedown.
Bugs! endpoint monitoring for acknowledgement and receipt.
Yandex Music endpoint monitoring for ddex ern release delivery.
Yandex Music endpoint monitoring for resource and asset delivery.
Yandex Music endpoint monitoring for metadata update and takedown.
Yandex Music endpoint monitoring for acknowledgement and receipt.
VK Music endpoint monitoring for ddex ern release delivery.
VK Music endpoint monitoring for resource and asset delivery.
VK Music endpoint monitoring for metadata update and takedown.
VK Music endpoint monitoring for acknowledgement and receipt.
Zvuk endpoint monitoring for ddex ern release delivery.
Zvuk endpoint monitoring for resource and asset delivery.
Zvuk endpoint monitoring for metadata update and takedown.
Zvuk endpoint monitoring for acknowledgement and receipt.
Qobuz endpoint monitoring for ddex ern release delivery.
Qobuz endpoint monitoring for resource and asset delivery.
Qobuz endpoint monitoring for metadata update and takedown.
Qobuz endpoint monitoring for acknowledgement and receipt.
Napster endpoint monitoring for ddex ern release delivery.
Napster endpoint monitoring for resource and asset delivery.
Napster endpoint monitoring for metadata update and takedown.
Napster endpoint monitoring for acknowledgement and receipt.
iHeartRadio endpoint monitoring for ddex ern release delivery.
iHeartRadio endpoint monitoring for resource and asset delivery.
iHeartRadio endpoint monitoring for metadata update and takedown.
iHeartRadio endpoint monitoring for acknowledgement and receipt.
Claro Música endpoint monitoring for ddex ern release delivery.
Claro Música endpoint monitoring for resource and asset delivery.
Claro Música endpoint monitoring for metadata update and takedown.
Claro Música endpoint monitoring for acknowledgement and receipt.
JOOX endpoint monitoring for ddex ern release delivery.
JOOX endpoint monitoring for resource and asset delivery.
JOOX endpoint monitoring for metadata update and takedown.
JOOX endpoint monitoring for acknowledgement and receipt.
TREBEL endpoint monitoring for ddex ern release delivery.
TREBEL endpoint monitoring for resource and asset delivery.
TREBEL endpoint monitoring for metadata update and takedown.
TREBEL endpoint monitoring for acknowledgement and receipt.
Beatport endpoint monitoring for ddex ern release delivery.
Beatport endpoint monitoring for resource and asset delivery.
Beatport endpoint monitoring for metadata update and takedown.
Beatport endpoint monitoring for acknowledgement and receipt.
Beatsource endpoint monitoring for ddex ern release delivery.
Beatsource endpoint monitoring for resource and asset delivery.
Beatsource endpoint monitoring for metadata update and takedown.
Beatsource endpoint monitoring for acknowledgement and receipt.
Traxsource endpoint monitoring for ddex ern release delivery.
Traxsource endpoint monitoring for resource and asset delivery.
Traxsource endpoint monitoring for metadata update and takedown.
Traxsource endpoint monitoring for acknowledgement and receipt.
7digital endpoint monitoring for ddex ern release delivery.
7digital endpoint monitoring for resource and asset delivery.
7digital endpoint monitoring for metadata update and takedown.
7digital endpoint monitoring for acknowledgement and receipt.
Bandcamp endpoint monitoring for ddex ern release delivery.
Bandcamp endpoint monitoring for resource and asset delivery.
Bandcamp endpoint monitoring for metadata update and takedown.
Bandcamp endpoint monitoring for acknowledgement and receipt.
Soundtrack Your Brand endpoint monitoring for ddex ern release delivery.
Soundtrack Your Brand endpoint monitoring for resource and asset delivery.
Soundtrack Your Brand endpoint monitoring for metadata update and takedown.
Soundtrack Your Brand endpoint monitoring for acknowledgement and receipt.
TouchTunes endpoint monitoring for ddex ern release delivery.
TouchTunes endpoint monitoring for resource and asset delivery.
TouchTunes endpoint monitoring for metadata update and takedown.
TouchTunes endpoint monitoring for acknowledgement and receipt.
SiriusXM endpoint monitoring for ddex ern release delivery.
SiriusXM endpoint monitoring for resource and asset delivery.
SiriusXM endpoint monitoring for metadata update and takedown.
SiriusXM endpoint monitoring for acknowledgement and receipt.
Peloton endpoint monitoring for ddex ern release delivery.
Peloton endpoint monitoring for resource and asset delivery.
Peloton endpoint monitoring for metadata update and takedown.
Peloton endpoint monitoring for acknowledgement and receipt.
Meta Music endpoint monitoring for ddex ern release delivery.
Meta Music endpoint monitoring for resource and asset delivery.
Meta Music endpoint monitoring for metadata update and takedown.
Meta Music endpoint monitoring for acknowledgement and receipt.
Facebook endpoint monitoring for ddex ern release delivery.
Facebook endpoint monitoring for resource and asset delivery.
Facebook endpoint monitoring for metadata update and takedown.
Facebook endpoint monitoring for acknowledgement and receipt.
Instagram endpoint monitoring for ddex ern release delivery.
Instagram endpoint monitoring for resource and asset delivery.
Instagram endpoint monitoring for metadata update and takedown.
Instagram endpoint monitoring for acknowledgement and receipt.
Snapchat endpoint monitoring for ddex ern release delivery.
Snapchat endpoint monitoring for resource and asset delivery.
Snapchat endpoint monitoring for metadata update and takedown.
Snapchat endpoint monitoring for acknowledgement and receipt.
CapCut endpoint monitoring for ddex ern release delivery.
CapCut endpoint monitoring for resource and asset delivery.
CapCut endpoint monitoring for metadata update and takedown.
CapCut endpoint monitoring for acknowledgement and receipt.
Shazam endpoint monitoring for ddex ern release delivery.
Shazam endpoint monitoring for resource and asset delivery.
Shazam endpoint monitoring for metadata update and takedown.
Shazam endpoint monitoring for acknowledgement and receipt.
Gracenote endpoint monitoring for ddex ern release delivery.
Gracenote endpoint monitoring for resource and asset delivery.
Gracenote endpoint monitoring for metadata update and takedown.
Gracenote endpoint monitoring for acknowledgement and receipt.
Musixmatch endpoint monitoring for ddex ern release delivery.
Musixmatch endpoint monitoring for resource and asset delivery.
Musixmatch endpoint monitoring for metadata update and takedown.
Musixmatch endpoint monitoring for acknowledgement and receipt.
Mixcloud endpoint monitoring for ddex ern release delivery.
Mixcloud endpoint monitoring for resource and asset delivery.
Mixcloud endpoint monitoring for metadata update and takedown.
Mixcloud endpoint monitoring for acknowledgement and receipt.
Twitch endpoint monitoring for ddex ern release delivery.
Twitch endpoint monitoring for resource and asset delivery.
Twitch endpoint monitoring for metadata update and takedown.
Twitch endpoint monitoring for acknowledgement and receipt.
Vevo endpoint monitoring for ddex ern release delivery.
Vevo endpoint monitoring for resource and asset delivery.
Vevo endpoint monitoring for metadata update and takedown.
Vevo endpoint monitoring for acknowledgement and receipt.
Incident history
42 structured reports spanning 2022–2026, with impact summaries, root-cause findings, corrective actions, affected components, and response timelines.
Showing 42 of 42 matching reports
Analytics, Reporting & Search
Recently updated releases could take longer than normal to appear in nonessential catalog search results. Delivery and release processing continued normally.
Resolution time
Jul 18, 2026, 3:19 PM UTC
1h 14m
Impact
Recently updated releases could take longer than normal to appear in nonessential catalog search results. Delivery and release processing continued normally.
Root cause
A scheduled search-index compaction job exceeded its expected memory budget and paused incremental refresh workers.
Resolution
The compaction job was restarted with a reduced batch size, queued index updates were replayed, and search freshness returned to its normal target.
Corrective action
Added batch-size guardrails, memory-pressure alerts, and an automatic fallback to incremental-only indexing during compaction.
Affected components
Response timeline
Investigating · Jul 18, 2026, 2:05 PM UTC
Operations began reviewing analytics, reporting & search telemetry and affected processing queues.
Identified · Jul 18, 2026, 2:20 PM UTC
A scheduled search-index compaction job exceeded its expected memory budget and paused incremental refresh workers.
Monitoring · Jul 18, 2026, 2:57 PM UTC
The compaction job was restarted with a reduced batch size, queued index updates were replayed, and search freshness returned to its normal target. Teams monitored recovery indicators before closing the report.
Resolved · Jul 18, 2026, 3:19 PM UTC
Normal performance was confirmed. Follow-up action: Added batch-size guardrails, memory-pressure alerts, and an automatic fallback to incremental-only indexing during compaction.
DDEX Delivery Endpoints
Delivery acknowledgements for a subset of partner routes were delayed, while outbound DDEX packages continued to queue safely without data loss.
Resolution time
Jun 29, 2026, 5:53 AM UTC
3h 33m
Impact
Delivery acknowledgements for a subset of partner routes were delayed, while outbound DDEX packages continued to queue safely without data loss.
Root cause
A receipt-normalization worker entered a retry loop after receiving an unexpected acknowledgement envelope variant.
Resolution
The envelope was isolated, the parser compatibility rule was expanded, and the receipt backlog was drained in chronological order.
Corrective action
Introduced schema-variant fixtures, retry-loop circuit breakers, and queue-age dashboards for partner acknowledgements.
Affected components
Response timeline
Investigating · Jun 29, 2026, 2:20 AM UTC
Operations began reviewing ddex delivery endpoints telemetry and affected processing queues.
Identified · Jun 29, 2026, 3:03 AM UTC
A receipt-normalization worker entered a retry loop after receiving an unexpected acknowledgement envelope variant.
Monitoring · Jun 29, 2026, 4:49 AM UTC
The envelope was isolated, the parser compatibility rule was expanded, and the receipt backlog was drained in chronological order. Teams monitored recovery indicators before closing the report.
Resolved · Jun 29, 2026, 5:53 AM UTC
Normal performance was confirmed. Follow-up action: Introduced schema-variant fixtures, retry-loop circuit breakers, and queue-age dashboards for partner acknowledgements.
Audio & Media Processing
Artwork validation completed more slowly for newly uploaded releases. Audio intake and metadata delivery were unaffected.
Resolution time
May 11, 2026, 7:33 PM UTC
51 minutes
Impact
Artwork validation completed more slowly for newly uploaded releases. Audio intake and metadata delivery were unaffected.
Root cause
A malformed embedded ICC profile triggered repeated fallback conversions in the image validation worker.
Resolution
The problematic profile path was bypassed, affected files were reprocessed, and validation latency normalized.
Corrective action
Added malformed-profile detection, per-file conversion limits, and dedicated test fixtures for uncommon color profiles.
Affected components
Response timeline
Investigating · May 11, 2026, 6:42 PM UTC
Operations began reviewing audio & media processing telemetry and affected processing queues.
Identified · May 11, 2026, 6:52 PM UTC
A malformed embedded ICC profile triggered repeated fallback conversions in the image validation worker.
Monitoring · May 11, 2026, 7:18 PM UTC
The problematic profile path was bypassed, affected files were reprocessed, and validation latency normalized. Teams monitored recovery indicators before closing the report.
Resolved · May 11, 2026, 7:33 PM UTC
Normal performance was confirmed. Follow-up action: Added malformed-profile detection, per-file conversion limits, and dedicated test fixtures for uncommon color profiles.
Rights, Royalties & Finance
Some nonurgent statement exports remained queued longer than expected. Earnings calculations and stored balances were not changed.
Resolution time
Apr 23, 2026, 10:53 AM UTC
1h 38m
Impact
Some nonurgent statement exports remained queued longer than expected. Earnings calculations and stored balances were not changed.
Root cause
A large batch of multi-territory exports saturated the document-rendering worker pool.
Resolution
Worker concurrency was temporarily increased and the queued exports were completed in priority order.
Corrective action
Added export-size estimation, fair-queue scheduling, and reserved capacity for interactive export requests.
Affected components
Response timeline
Investigating · Apr 23, 2026, 9:15 AM UTC
Operations began reviewing rights, royalties & finance telemetry and affected processing queues.
Identified · Apr 23, 2026, 9:35 AM UTC
A large batch of multi-territory exports saturated the document-rendering worker pool.
Monitoring · Apr 23, 2026, 10:24 AM UTC
Worker concurrency was temporarily increased and the queued exports were completed in priority order. Teams monitored recovery indicators before closing the report.
Resolved · Apr 23, 2026, 10:53 AM UTC
Normal performance was confirmed. Follow-up action: Added export-size estimation, fair-queue scheduling, and reserved capacity for interactive export requests.
Support & Communications
Status updates were visible on the public page, but a subset of email notifications arrived later than normal.
Resolution time
Mar 7, 2026, 9:44 PM UTC
36 minutes
Impact
Status updates were visible on the public page, but a subset of email notifications arrived later than normal.
Root cause
A notification batch exceeded the provider connection pool and waited for available sending slots.
Resolution
The batch was split into smaller segments and remaining notifications were delivered.
Corrective action
Implemented adaptive batch sizing, provider pool telemetry, and an in-app fallback notice when email delivery is delayed.
Affected components
Response timeline
Investigating · Mar 7, 2026, 9:08 PM UTC
Operations began reviewing support & communications telemetry and affected processing queues.
Identified · Mar 7, 2026, 9:16 PM UTC
A notification batch exceeded the provider connection pool and waited for available sending slots.
Monitoring · Mar 7, 2026, 9:33 PM UTC
The batch was split into smaller segments and remaining notifications were delivered. Teams monitored recovery indicators before closing the report.
Resolved · Mar 7, 2026, 9:44 PM UTC
Normal performance was confirmed. Follow-up action: Implemented adaptive batch sizing, provider pool telemetry, and an in-app fallback notice when email delivery is delayed.
Core Platform
New release records remained safely stored but synchronized to downstream gateway records more slowly than normal.
Resolution time
Feb 16, 2026, 10:16 AM UTC
4h 46m
Impact
New release records remained safely stored but synchronized to downstream gateway records more slowly than normal.
Root cause
A malformed optional territory value caused repeated per-record synchronization retries without sufficient backoff.
Resolution
The malformed value was normalized, retry queues were reset, and all pending UPC-indexed records were synchronized.
Corrective action
Added payload prevalidation, exponential backoff, dead-letter isolation, and per-record retry ceilings.
Affected components
Response timeline
Investigating · Feb 16, 2026, 5:30 AM UTC
Operations began reviewing core platform telemetry and affected processing queues.
Identified · Feb 16, 2026, 6:27 AM UTC
A malformed optional territory value caused repeated per-record synchronization retries without sufficient backoff.
Monitoring · Feb 16, 2026, 8:50 AM UTC
The malformed value was normalized, retry queues were reset, and all pending UPC-indexed records were synchronized. Teams monitored recovery indicators before closing the report.
Resolved · Feb 16, 2026, 10:16 AM UTC
Normal performance was confirmed. Follow-up action: Added payload prevalidation, exponential backoff, dead-letter isolation, and per-record retry ceilings.
Metadata & Identifier Services
Some recording relationship lookups took longer to complete, while direct ISRC validation remained available.
Resolution time
Jan 28, 2026, 2:19 PM UTC
1h 7m
Impact
Some recording relationship lookups took longer to complete, while direct ISRC validation remained available.
Root cause
A metadata cache shard was rebuilding after routine maintenance and served a higher-than-normal percentage of uncached queries.
Resolution
Cache warming was prioritized for high-frequency recording identifiers and lookup latency returned to normal.
Corrective action
Added prewarming manifests, cache-rebuild progress indicators, and request coalescing for duplicate lookups.
Affected components
Response timeline
Investigating · Jan 28, 2026, 1:12 PM UTC
Operations began reviewing metadata & identifier services telemetry and affected processing queues.
Identified · Jan 28, 2026, 1:25 PM UTC
A metadata cache shard was rebuilding after routine maintenance and served a higher-than-normal percentage of uncached queries.
Monitoring · Jan 28, 2026, 1:59 PM UTC
Cache warming was prioritized for high-frequency recording identifiers and lookup latency returned to normal. Teams monitored recovery indicators before closing the report.
Resolved · Jan 28, 2026, 2:19 PM UTC
Normal performance was confirmed. Follow-up action: Added prewarming manifests, cache-rebuild progress indicators, and request coalescing for duplicate lookups.
DSP Delivery Network
Content ID asset receipts were delayed for a limited batch. Assets remained queued and no duplicate submissions were generated.
Resolution time
Jan 14, 2026, 3:02 AM UTC
2h 22m
Impact
Content ID asset receipts were delayed for a limited batch. Assets remained queued and no duplicate submissions were generated.
Root cause
A partner receipt payload included an unrecognized nested status code that was routed to manual review.
Resolution
The status code was mapped, held receipts were replayed, and asset delivery states were reconciled.
Corrective action
Expanded receipt code coverage and added automated safe handling for unknown nonterminal status values.
Affected components
Response timeline
Investigating · Jan 14, 2026, 12:40 AM UTC
Operations began reviewing dsp delivery network telemetry and affected processing queues.
Identified · Jan 14, 2026, 1:08 AM UTC
A partner receipt payload included an unrecognized nested status code that was routed to manual review.
Monitoring · Jan 14, 2026, 2:19 AM UTC
The status code was mapped, held receipts were replayed, and asset delivery states were reconciled. Teams monitored recovery indicators before closing the report.
Resolved · Jan 14, 2026, 3:02 AM UTC
Normal performance was confirmed. Follow-up action: Expanded receipt code coverage and added automated safe handling for unknown nonterminal status values.
API & Developer Services
Sandbox authorization-code exchanges completed slowly for some developer sessions. Production API authentication was unaffected.
Resolution time
Jan 3, 2026, 5:38 PM UTC
43 minutes
Impact
Sandbox authorization-code exchanges completed slowly for some developer sessions. Production API authentication was unaffected.
Root cause
A sandbox-only signing-key cache was cold after a routine deployment.
Resolution
Signing keys were preloaded across sandbox workers and pending token exchanges completed successfully.
Corrective action
Added deployment-time key prewarming and separate latency alerts for sandbox authentication paths.
Affected components
Response timeline
Investigating · Jan 3, 2026, 4:55 PM UTC
Operations began reviewing api & developer services telemetry and affected processing queues.
Identified · Jan 3, 2026, 5:04 PM UTC
A sandbox-only signing-key cache was cold after a routine deployment.
Monitoring · Jan 3, 2026, 5:25 PM UTC
Signing keys were preloaded across sandbox workers and pending token exchanges completed successfully. Teams monitored recovery indicators before closing the report.
Resolved · Jan 3, 2026, 5:38 PM UTC
Normal performance was confirmed. Follow-up action: Added deployment-time key prewarming and separate latency alerts for sandbox authentication paths.
DDEX Delivery Endpoints
Metadata delivery receipts for a subset of updates appeared later than expected. Outbound delivery remained available.
Resolution time
Dec 9, 2025, 12:53 PM UTC
1h 29m
Impact
Metadata delivery receipts for a subset of updates appeared later than expected. Outbound delivery remained available.
Root cause
Receipt polling was temporarily constrained by a conservative partner-route rate limit.
Resolution
Polling intervals were redistributed and delayed receipts were reconciled without resubmitting releases.
Corrective action
Added route-specific rate-limit forecasting and staggered receipt polling for large update batches.
Affected components
Response timeline
Investigating · Dec 9, 2025, 11:24 AM UTC
Operations began reviewing ddex delivery endpoints telemetry and affected processing queues.
Identified · Dec 9, 2025, 11:42 AM UTC
Receipt polling was temporarily constrained by a conservative partner-route rate limit.
Monitoring · Dec 9, 2025, 12:26 PM UTC
Polling intervals were redistributed and delayed receipts were reconciled without resubmitting releases. Teams monitored recovery indicators before closing the report.
Resolved · Dec 9, 2025, 12:53 PM UTC
Normal performance was confirmed. Follow-up action: Added route-specific rate-limit forecasting and staggered receipt polling for large update batches.
Catalog & Release Operations
Large catalog CSV uploads remained in processing longer than normal, but files stayed intact and were processed in upload order.
Resolution time
Oct 27, 2025, 9:14 PM UTC
2h 8m
Impact
Large catalog CSV uploads remained in processing longer than normal, but files stayed intact and were processed in upload order.
Root cause
Several unusually large catalog files arrived simultaneously and exhausted the standard parsing worker pool.
Resolution
Additional parsing workers were allocated and the backlog was drained with deduplication checks preserved.
Corrective action
Added file-size-aware queue partitioning, burst capacity, and estimated completion times for large imports.
Affected components
Response timeline
Investigating · Oct 27, 2025, 7:06 PM UTC
Operations began reviewing catalog & release operations telemetry and affected processing queues.
Identified · Oct 27, 2025, 7:32 PM UTC
Several unusually large catalog files arrived simultaneously and exhausted the standard parsing worker pool.
Monitoring · Oct 27, 2025, 8:36 PM UTC
Additional parsing workers were allocated and the backlog was drained with deduplication checks preserved. Teams monitored recovery indicators before closing the report.
Resolved · Oct 27, 2025, 9:14 PM UTC
Normal performance was confirmed. Follow-up action: Added file-size-aware queue partitioning, burst capacity, and estimated completion times for large imports.
Audio & Media Processing
Validated timed lyrics took longer than normal to move from intake to publishing queues. Audio deliveries were unaffected.
Resolution time
Sep 13, 2025, 8:42 AM UTC
57 minutes
Impact
Validated timed lyrics took longer than normal to move from intake to publishing queues. Audio deliveries were unaffected.
Root cause
A character-normalization edge case caused repeated validation retries on a small group of TTML documents.
Resolution
The normalization rule was corrected and queued lyric documents were revalidated.
Corrective action
Added Unicode edge-case fixtures and a retry cap that routes malformed documents directly to review.
Affected components
Response timeline
Investigating · Sep 13, 2025, 7:45 AM UTC
Operations began reviewing audio & media processing telemetry and affected processing queues.
Identified · Sep 13, 2025, 7:56 AM UTC
A character-normalization edge case caused repeated validation retries on a small group of TTML documents.
Monitoring · Sep 13, 2025, 8:25 AM UTC
The normalization rule was corrected and queued lyric documents were revalidated. Teams monitored recovery indicators before closing the report.
Resolved · Sep 13, 2025, 8:42 AM UTC
Normal performance was confirmed. Follow-up action: Added Unicode edge-case fixtures and a retry cap that routes malformed documents directly to review.
DSP Delivery Network
Preview asset processing was delayed for selected TikTok-bound releases while metadata delivery continued.
Resolution time
Aug 2, 2025, 6:14 AM UTC
2h 56m
Impact
Preview asset processing was delayed for selected TikTok-bound releases while metadata delivery continued.
Root cause
A media transcode worker group was reduced during maintenance and did not scale back up automatically.
Resolution
Worker capacity was restored and queued preview assets were processed by original priority.
Corrective action
Added post-maintenance capacity verification and automatic minimum-worker enforcement.
Affected components
Response timeline
Investigating · Aug 2, 2025, 3:18 AM UTC
Operations began reviewing dsp delivery network telemetry and affected processing queues.
Identified · Aug 2, 2025, 3:53 AM UTC
A media transcode worker group was reduced during maintenance and did not scale back up automatically.
Monitoring · Aug 2, 2025, 5:21 AM UTC
Worker capacity was restored and queued preview assets were processed by original priority. Teams monitored recovery indicators before closing the report.
Resolved · Aug 2, 2025, 6:14 AM UTC
Normal performance was confirmed. Follow-up action: Added post-maintenance capacity verification and automatic minimum-worker enforcement.
Metadata & Identifier Services
A small number of repeated UPC checks returned inconsistent cached messages, though final checksum decisions remained correct.
Resolution time
Jun 21, 2025, 4:35 PM UTC
45 minutes
Impact
A small number of repeated UPC checks returned inconsistent cached messages, though final checksum decisions remained correct.
Root cause
Two cache versions used different explanatory message formats after a rolling release.
Resolution
The older cache namespace was invalidated and all workers were aligned to the current validation response format.
Corrective action
Versioned validation caches and added cross-worker contract tests for response consistency.
Affected components
Response timeline
Investigating · Jun 21, 2025, 3:50 PM UTC
Operations began reviewing metadata & identifier services telemetry and affected processing queues.
Identified · Jun 21, 2025, 3:59 PM UTC
Two cache versions used different explanatory message formats after a rolling release.
Monitoring · Jun 21, 2025, 4:21 PM UTC
The older cache namespace was invalidated and all workers were aligned to the current validation response format. Teams monitored recovery indicators before closing the report.
Resolved · Jun 21, 2025, 4:35 PM UTC
Normal performance was confirmed. Follow-up action: Versioned validation caches and added cross-worker contract tests for response consistency.
Audio & Media Processing
Atmos package verification took longer than normal for new submissions. Stereo processing and existing deliveries were unaffected.
Resolution time
May 9, 2025, 2:22 AM UTC
3h 51m
Impact
Atmos package verification took longer than normal for new submissions. Stereo processing and existing deliveries were unaffected.
Root cause
An unusually complex set of channel-layout validations increased average verification time and saturated the specialized worker queue.
Resolution
Complex packages were moved to a dedicated queue and standard validations resumed at normal speed.
Corrective action
Added complexity scoring, dedicated high-complexity capacity, and queue-depth alerts for immersive-audio validation.
Affected components
Response timeline
Investigating · May 8, 2025, 10:31 PM UTC
Operations began reviewing audio & media processing telemetry and affected processing queues.
Identified · May 8, 2025, 11:17 PM UTC
An unusually complex set of channel-layout validations increased average verification time and saturated the specialized worker queue.
Monitoring · May 9, 2025, 1:13 AM UTC
Complex packages were moved to a dedicated queue and standard validations resumed at normal speed. Teams monitored recovery indicators before closing the report.
Resolved · May 9, 2025, 2:22 AM UTC
Normal performance was confirmed. Follow-up action: Added complexity scoring, dedicated high-complexity capacity, and queue-depth alerts for immersive-audio validation.
Analytics, Reporting & Search
Recently changed nonessential endpoint labels took longer to appear in public search results. Live component health values remained current.
Resolution time
Mar 19, 2025, 11:16 AM UTC
1h 2m
Impact
Recently changed nonessential endpoint labels took longer to appear in public search results. Live component health values remained current.
Root cause
A static directory cache did not invalidate after a content-only metadata update.
Resolution
The directory cache was purged and rebuilt from the current component registry.
Corrective action
Linked content updates to cache invalidation and added freshness timestamps to directory responses.
Affected components
Response timeline
Investigating · Mar 19, 2025, 10:14 AM UTC
Operations began reviewing analytics, reporting & search telemetry and affected processing queues.
Identified · Mar 19, 2025, 10:26 AM UTC
A static directory cache did not invalidate after a content-only metadata update.
Monitoring · Mar 19, 2025, 10:57 AM UTC
The directory cache was purged and rebuilt from the current component registry. Teams monitored recovery indicators before closing the report.
Resolved · Mar 19, 2025, 11:16 AM UTC
Normal performance was confirmed. Follow-up action: Linked content updates to cache invalidation and added freshness timestamps to directory responses.
Rights, Royalties & Finance
Some territory-level report views loaded slowly while aggregate earnings totals remained available.
Resolution time
Feb 4, 2025, 5:49 AM UTC
1h 44m
Impact
Some territory-level report views loaded slowly while aggregate earnings totals remained available.
Root cause
A currency reference table refresh held a read lock longer than expected during report aggregation.
Resolution
The refresh was moved to a nonblocking swap operation and pending reports completed.
Corrective action
Implemented atomic reference-table replacement and query-time fallbacks to the last validated currency snapshot.
Affected components
Response timeline
Investigating · Feb 4, 2025, 4:05 AM UTC
Operations began reviewing rights, royalties & finance telemetry and affected processing queues.
Identified · Feb 4, 2025, 4:26 AM UTC
A currency reference table refresh held a read lock longer than expected during report aggregation.
Monitoring · Feb 4, 2025, 5:18 AM UTC
The refresh was moved to a nonblocking swap operation and pending reports completed. Teams monitored recovery indicators before closing the report.
Resolved · Feb 4, 2025, 5:49 AM UTC
Normal performance was confirmed. Follow-up action: Implemented atomic reference-table replacement and query-time fallbacks to the last validated currency snapshot.
Identity, Security & Storage
Organization-domain profile edits were saved but appeared with a short delay in account views. Authentication and permissions were unaffected.
Resolution time
Jan 12, 2025, 6:17 PM UTC
39 minutes
Impact
Organization-domain profile edits were saved but appeared with a short delay in account views. Authentication and permissions were unaffected.
Root cause
A profile cache invalidation event was not emitted for one update path.
Resolution
The missing invalidation event was restored and affected profiles were refreshed.
Corrective action
Added update-path coverage tests and cache-consistency checks after profile writes.
Affected components
Response timeline
Investigating · Jan 12, 2025, 5:38 PM UTC
Operations began reviewing identity, security & storage telemetry and affected processing queues.
Identified · Jan 12, 2025, 5:46 PM UTC
A profile cache invalidation event was not emitted for one update path.
Monitoring · Jan 12, 2025, 6:05 PM UTC
The missing invalidation event was restored and affected profiles were refreshed. Teams monitored recovery indicators before closing the report.
Resolved · Jan 12, 2025, 6:17 PM UTC
Normal performance was confirmed. Follow-up action: Added update-path coverage tests and cache-consistency checks after profile writes.
Catalog & Release Operations
A subset of ERN messages using an uncommon namespace pattern required manual review before release records could be created.
Resolution time
Nov 18, 2024, 11:28 AM UTC
3h 16m
Impact
A subset of ERN messages using an uncommon namespace pattern required manual review before release records could be created.
Root cause
The namespace resolver expected a canonical prefix even though the incoming XML used a valid alternate prefix.
Resolution
Namespace handling was updated to resolve by URI, and held messages were reprocessed successfully.
Corrective action
Added alternate-prefix fixtures and namespace-agnostic extraction tests for supported DDEX message families.
Affected components
Response timeline
Investigating · Nov 18, 2024, 8:12 AM UTC
Operations began reviewing catalog & release operations telemetry and affected processing queues.
Identified · Nov 18, 2024, 8:51 AM UTC
The namespace resolver expected a canonical prefix even though the incoming XML used a valid alternate prefix.
Monitoring · Nov 18, 2024, 10:29 AM UTC
Namespace handling was updated to resolve by URI, and held messages were reprocessed successfully. Teams monitored recovery indicators before closing the report.
Resolved · Nov 18, 2024, 11:28 AM UTC
Normal performance was confirmed. Follow-up action: Added alternate-prefix fixtures and namespace-agnostic extraction tests for supported DDEX message families.
DSP Delivery Network
Territory update receipts were delayed for a limited batch. Existing availability remained unchanged until confirmation.
Resolution time
Oct 3, 2024, 9:51 PM UTC
1h 11m
Impact
Territory update receipts were delayed for a limited batch. Existing availability remained unchanged until confirmation.
Root cause
A route-specific acknowledgement poller restarted during routine infrastructure maintenance.
Resolution
The poller resumed from its stored cursor and all pending receipts were reconciled.
Corrective action
Added cursor checkpoint validation and automatic restart verification for route pollers.
Affected components
Response timeline
Investigating · Oct 3, 2024, 8:40 PM UTC
Operations began reviewing dsp delivery network telemetry and affected processing queues.
Identified · Oct 3, 2024, 8:54 PM UTC
A route-specific acknowledgement poller restarted during routine infrastructure maintenance.
Monitoring · Oct 3, 2024, 9:30 PM UTC
The poller resumed from its stored cursor and all pending receipts were reconciled. Teams monitored recovery indicators before closing the report.
Resolved · Oct 3, 2024, 9:51 PM UTC
Normal performance was confirmed. Follow-up action: Added cursor checkpoint validation and automatic restart verification for route pollers.
Audio & Media Processing
Large high-resolution masters remained in validation longer than usual. Files were retained securely and no audio data was modified.
Resolution time
Aug 26, 2024, 3:08 PM UTC
2h 13m
Impact
Large high-resolution masters remained in validation longer than usual. Files were retained securely and no audio data was modified.
Root cause
Integrity scans were running in a low-throughput compatibility mode after a scanner update.
Resolution
The scanner configuration was corrected and queued masters were rescanned at normal throughput.
Corrective action
Added throughput benchmarks to scanner deployments and an automatic rollback threshold for validation speed.
Affected components
Response timeline
Investigating · Aug 26, 2024, 12:55 PM UTC
Operations began reviewing audio & media processing telemetry and affected processing queues.
Identified · Aug 26, 2024, 1:22 PM UTC
Integrity scans were running in a low-throughput compatibility mode after a scanner update.
Monitoring · Aug 26, 2024, 2:28 PM UTC
The scanner configuration was corrected and queued masters were rescanned at normal throughput. Teams monitored recovery indicators before closing the report.
Resolved · Aug 26, 2024, 3:08 PM UTC
Normal performance was confirmed. Follow-up action: Added throughput benchmarks to scanner deployments and an automatic rollback threshold for validation speed.
Third Party and Partners
Broad metadata searches were temporarily slowed by upstream request limits. Exact identifier lookups remained available.
Resolution time
Jul 9, 2024, 2:05 AM UTC
48 minutes
Impact
Broad metadata searches were temporarily slowed by upstream request limits. Exact identifier lookups remained available.
Root cause
A scheduled catalog enrichment task produced more broad searches than forecast.
Resolution
The enrichment task was throttled and interactive searches were assigned a separate request budget.
Corrective action
Separated background and interactive quotas and added predictive request-volume controls.
Affected components
Response timeline
Investigating · Jul 9, 2024, 1:17 AM UTC
Operations began reviewing third party and partners telemetry and affected processing queues.
Identified · Jul 9, 2024, 1:27 AM UTC
A scheduled catalog enrichment task produced more broad searches than forecast.
Monitoring · Jul 9, 2024, 1:51 AM UTC
The enrichment task was throttled and interactive searches were assigned a separate request budget. Teams monitored recovery indicators before closing the report.
Resolved · Jul 9, 2024, 2:05 AM UTC
Normal performance was confirmed. Follow-up action: Separated background and interactive quotas and added predictive request-volume controls.
Analytics, Reporting & Search
Historical delivery activity reports were behind current processing by approximately one reporting cycle. Live status indicators remained current.
Resolution time
May 30, 2024, 5:37 PM UTC
1h 34m
Impact
Historical delivery activity reports were behind current processing by approximately one reporting cycle. Live status indicators remained current.
Root cause
An aggregation job skipped a partition after detecting a transient checksum mismatch.
Resolution
The partition was verified, replayed, and reporting freshness returned to normal.
Corrective action
Added automatic partition retry after checksum verification and clearer separation of live versus historical freshness metrics.
Affected components
Response timeline
Investigating · May 30, 2024, 4:03 PM UTC
Operations began reviewing analytics, reporting & search telemetry and affected processing queues.
Identified · May 30, 2024, 4:22 PM UTC
An aggregation job skipped a partition after detecting a transient checksum mismatch.
Monitoring · May 30, 2024, 5:09 PM UTC
The partition was verified, replayed, and reporting freshness returned to normal. Teams monitored recovery indicators before closing the report.
Resolved · May 30, 2024, 5:37 PM UTC
Normal performance was confirmed. Follow-up action: Added automatic partition retry after checksum verification and clearer separation of live versus historical freshness metrics.
DDEX Delivery Endpoints
A small batch of high-resolution assets was held before partner delivery while format declarations were revalidated.
Resolution time
Apr 14, 2024, 9:09 AM UTC
2h 47m
Impact
A small batch of high-resolution assets was held before partner delivery while format declarations were revalidated.
Root cause
Container metadata and declared bit depth disagreed on several files, triggering a safety hold.
Resolution
The files were reanalyzed, valid assets were released, and mismatched assets were returned with precise correction guidance.
Corrective action
Added preflight comparison of container metadata, declared specifications, and actual decoded stream properties.
Affected components
Response timeline
Investigating · Apr 14, 2024, 6:22 AM UTC
Operations began reviewing ddex delivery endpoints telemetry and affected processing queues.
Identified · Apr 14, 2024, 6:55 AM UTC
Container metadata and declared bit depth disagreed on several files, triggering a safety hold.
Monitoring · Apr 14, 2024, 8:19 AM UTC
The files were reanalyzed, valid assets were released, and mismatched assets were returned with precise correction guidance. Teams monitored recovery indicators before closing the report.
Resolved · Apr 14, 2024, 9:09 AM UTC
Normal performance was confirmed. Follow-up action: Added preflight comparison of container metadata, declared specifications, and actual decoded stream properties.
Support & Communications
New documentation sections were available by direct link but did not immediately appear in documentation search.
Resolution time
Feb 23, 2024, 12:09 AM UTC
59 minutes
Impact
New documentation sections were available by direct link but did not immediately appear in documentation search.
Root cause
The documentation crawler deferred a reindex after detecting a temporary sitemap fetch error.
Resolution
The sitemap was refetched and the documentation index was rebuilt.
Corrective action
Added sitemap retry with cached fallback and visible documentation-index freshness reporting.
Affected components
Response timeline
Investigating · Feb 22, 2024, 11:10 PM UTC
Operations began reviewing support & communications telemetry and affected processing queues.
Identified · Feb 22, 2024, 11:22 PM UTC
The documentation crawler deferred a reindex after detecting a temporary sitemap fetch error.
Monitoring · Feb 22, 2024, 11:51 PM UTC
The sitemap was refetched and the documentation index was rebuilt. Teams monitored recovery indicators before closing the report.
Resolved · Feb 23, 2024, 12:09 AM UTC
Normal performance was confirmed. Follow-up action: Added sitemap retry with cached fallback and visible documentation-index freshness reporting.
Catalog & Release Operations
A small number of detail pages displayed a stale status label after a successful delivery update. Underlying records were correct.
Resolution time
Jan 6, 2024, 2:16 PM UTC
32 minutes
Impact
A small number of detail pages displayed a stale status label after a successful delivery update. Underlying records were correct.
Root cause
A detail-view cache key omitted the release update timestamp.
Resolution
The cache key was corrected and affected detail views were invalidated.
Corrective action
Added status-consistency checks between list and detail views and included record versioning in cache keys.
Affected components
Response timeline
Investigating · Jan 6, 2024, 1:44 PM UTC
Operations began reviewing catalog & release operations telemetry and affected processing queues.
Identified · Jan 6, 2024, 1:52 PM UTC
A detail-view cache key omitted the release update timestamp.
Monitoring · Jan 6, 2024, 2:06 PM UTC
The cache key was corrected and affected detail views were invalidated. Teams monitored recovery indicators before closing the report.
Resolved · Jan 6, 2024, 2:16 PM UTC
Normal performance was confirmed. Follow-up action: Added status-consistency checks between list and detail views and included record versioning in cache keys.
DSP Delivery Network
Takedown requests were submitted, but confirmation receipts were delayed for selected routes. Requests were not duplicated.
Resolution time
Dec 12, 2023, 8:39 AM UTC
4h 4m
Impact
Takedown requests were submitted, but confirmation receipts were delayed for selected routes. Requests were not duplicated.
Root cause
The acknowledgement reconciler paused after detecting an out-of-order receipt sequence.
Resolution
Receipt ordering rules were corrected and pending confirmations were reconciled against original request IDs.
Corrective action
Added out-of-order receipt support, idempotency checks, and route-specific reconciliation alerts.
Affected components
Response timeline
Investigating · Dec 12, 2023, 4:35 AM UTC
Operations began reviewing dsp delivery network telemetry and affected processing queues.
Identified · Dec 12, 2023, 5:24 AM UTC
The acknowledgement reconciler paused after detecting an out-of-order receipt sequence.
Monitoring · Dec 12, 2023, 7:26 AM UTC
Receipt ordering rules were corrected and pending confirmations were reconciled against original request IDs. Teams monitored recovery indicators before closing the report.
Resolved · Dec 12, 2023, 8:39 AM UTC
Normal performance was confirmed. Follow-up action: Added out-of-order receipt support, idempotency checks, and route-specific reconciliation alerts.
Metadata & Identifier Services
Complex artist relationship queries took longer than normal. Direct artist and release lookups remained available.
Resolution time
Oct 29, 2023, 7:33 PM UTC
1h 13m
Impact
Complex artist relationship queries took longer than normal. Direct artist and release lookups remained available.
Root cause
A highly connected entity generated a large uncached relationship graph.
Resolution
The query was paginated internally and frequently requested relationship paths were cached.
Corrective action
Added graph-size limits, progressive relationship loading, and high-cardinality query telemetry.
Affected components
Response timeline
Investigating · Oct 29, 2023, 6:20 PM UTC
Operations began reviewing metadata & identifier services telemetry and affected processing queues.
Identified · Oct 29, 2023, 6:35 PM UTC
A highly connected entity generated a large uncached relationship graph.
Monitoring · Oct 29, 2023, 7:11 PM UTC
The query was paginated internally and frequently requested relationship paths were cached. Teams monitored recovery indicators before closing the report.
Resolved · Oct 29, 2023, 7:33 PM UTC
Normal performance was confirmed. Follow-up action: Added graph-size limits, progressive relationship loading, and high-cardinality query telemetry.
Analytics, Reporting & Search
Recent audit events briefly appeared out of sequence in one interface. Event records and timestamps remained intact.
Resolution time
Sep 5, 2023, 9:49 AM UTC
41 minutes
Impact
Recent audit events briefly appeared out of sequence in one interface. Event records and timestamps remained intact.
Root cause
The compact feed sorted equal-millisecond timestamps by arrival order instead of event identifier.
Resolution
A deterministic secondary sort by event identifier was added and the feed refreshed.
Corrective action
Standardized audit sorting across all views and added equal-timestamp test coverage.
Affected components
Response timeline
Investigating · Sep 5, 2023, 9:08 AM UTC
Operations began reviewing analytics, reporting & search telemetry and affected processing queues.
Identified · Sep 5, 2023, 9:16 AM UTC
The compact feed sorted equal-millisecond timestamps by arrival order instead of event identifier.
Monitoring · Sep 5, 2023, 9:37 AM UTC
A deterministic secondary sort by event identifier was added and the feed refreshed. Teams monitored recovery indicators before closing the report.
Resolved · Sep 5, 2023, 9:49 AM UTC
Normal performance was confirmed. Follow-up action: Standardized audit sorting across all views and added equal-timestamp test coverage.
Catalog & Release Operations
New territory settings were stored correctly but some cached availability views reflected the previous territory set.
Resolution time
Jul 17, 2023, 4:50 AM UTC
1h 58m
Impact
New territory settings were stored correctly but some cached availability views reflected the previous territory set.
Root cause
A territory-only update did not trigger the same cache invalidation path as a full metadata update.
Resolution
Territory caches were invalidated and availability views were rebuilt from current release records.
Corrective action
Unified cache invalidation for all delivery-affecting metadata changes and added territory-specific regression tests.
Affected components
Response timeline
Investigating · Jul 17, 2023, 2:52 AM UTC
Operations began reviewing catalog & release operations telemetry and affected processing queues.
Identified · Jul 17, 2023, 3:16 AM UTC
A territory-only update did not trigger the same cache invalidation path as a full metadata update.
Monitoring · Jul 17, 2023, 4:15 AM UTC
Territory caches were invalidated and availability views were rebuilt from current release records. Teams monitored recovery indicators before closing the report.
Resolved · Jul 17, 2023, 4:50 AM UTC
Normal performance was confirmed. Follow-up action: Unified cache invalidation for all delivery-affecting metadata changes and added territory-specific regression tests.
API & Developer Services
Some signed upload URLs took longer to generate, while existing upload URLs remained valid.
Resolution time
May 26, 2023, 3:23 PM UTC
1h 4m
Impact
Some signed upload URLs took longer to generate, while existing upload URLs remained valid.
Root cause
A storage permission policy refresh temporarily increased authorization checks.
Resolution
The refreshed policy was cached across workers and pending URL requests completed.
Corrective action
Added policy preloading and separate latency monitoring for signed URL generation.
Affected components
Response timeline
Investigating · May 26, 2023, 2:19 PM UTC
Operations began reviewing api & developer services telemetry and affected processing queues.
Identified · May 26, 2023, 2:32 PM UTC
A storage permission policy refresh temporarily increased authorization checks.
Monitoring · May 26, 2023, 3:04 PM UTC
The refreshed policy was cached across workers and pending URL requests completed. Teams monitored recovery indicators before closing the report.
Resolved · May 26, 2023, 3:23 PM UTC
Normal performance was confirmed. Follow-up action: Added policy preloading and separate latency monitoring for signed URL generation.
DSP Delivery Network
A subset of channel mapping searches timed out and required retry. Existing channel mappings were unaffected.
Resolution time
Apr 2, 2023, 11:23 PM UTC
37 minutes
Impact
A subset of channel mapping searches timed out and required retry. Existing channel mappings were unaffected.
Root cause
A background preference-refresh task competed with interactive mapping searches for the same connection pool.
Resolution
Interactive requests were moved to a reserved pool and timed-out lookups succeeded on retry.
Corrective action
Separated background refresh traffic and added pool-saturation protection for interactive lookups.
Affected components
Response timeline
Investigating · Apr 2, 2023, 10:46 PM UTC
Operations began reviewing dsp delivery network telemetry and affected processing queues.
Identified · Apr 2, 2023, 10:54 PM UTC
A background preference-refresh task competed with interactive mapping searches for the same connection pool.
Monitoring · Apr 2, 2023, 11:12 PM UTC
Interactive requests were moved to a reserved pool and timed-out lookups succeeded on retry. Teams monitored recovery indicators before closing the report.
Resolved · Apr 2, 2023, 11:23 PM UTC
Normal performance was confirmed. Follow-up action: Separated background refresh traffic and added pool-saturation protection for interactive lookups.
Catalog & Release Operations
A subset of imported delivery timestamps displayed with an incorrect offset, while delivery status values remained accurate.
Resolution time
Feb 15, 2023, 1:42 PM UTC
2h 31m
Impact
A subset of imported delivery timestamps displayed with an incorrect offset, while delivery status values remained accurate.
Root cause
One accepted timestamp format was interpreted in local time instead of UTC during normalization.
Resolution
Affected timestamps were recalculated from source records and the parser was corrected.
Corrective action
Added explicit timezone parsing, source-format fixtures, and timestamp round-trip validation.
Affected components
Response timeline
Investigating · Feb 15, 2023, 11:11 AM UTC
Operations began reviewing catalog & release operations telemetry and affected processing queues.
Identified · Feb 15, 2023, 11:41 AM UTC
One accepted timestamp format was interpreted in local time instead of UTC during normalization.
Monitoring · Feb 15, 2023, 12:57 PM UTC
Affected timestamps were recalculated from source records and the parser was corrected. Teams monitored recovery indicators before closing the report.
Resolved · Feb 15, 2023, 1:42 PM UTC
Normal performance was confirmed. Follow-up action: Added explicit timezone parsing, source-format fixtures, and timestamp round-trip validation.
Identity, Security & Storage
A small number of background partner jobs waited for token refresh before continuing. No credentials were exposed.
Resolution time
Jan 9, 2023, 4:17 AM UTC
52 minutes
Impact
A small number of background partner jobs waited for token refresh before continuing. No credentials were exposed.
Root cause
Refresh requests converged at the same expiration boundary and temporarily serialized on a signing lock.
Resolution
Refresh requests were jittered and queued jobs resumed after tokens were renewed.
Corrective action
Added expiration jitter, proactive refresh windows, and lock-contention alerts.
Affected components
Response timeline
Investigating · Jan 9, 2023, 3:25 AM UTC
Operations began reviewing identity, security & storage telemetry and affected processing queues.
Identified · Jan 9, 2023, 3:35 AM UTC
Refresh requests converged at the same expiration boundary and temporarily serialized on a signing lock.
Monitoring · Jan 9, 2023, 4:01 AM UTC
Refresh requests were jittered and queued jobs resumed after tokens were renewed. Teams monitored recovery indicators before closing the report.
Resolved · Jan 9, 2023, 4:17 AM UTC
Normal performance was confirmed. Follow-up action: Added expiration jitter, proactive refresh windows, and lock-contention alerts.
Identity, Security & Storage
A planned nonessential storage migration ran longer than expected. Existing public assets remained available while new background copies queued safely.
Resolution time
Dec 3, 2022, 12:42 PM UTC
5h 12m
Impact
A planned nonessential storage migration ran longer than expected. Existing public assets remained available while new background copies queued safely.
Root cause
Checksum verification took longer on several large media objects than the migration estimate allowed.
Resolution
The migration window was extended, checksums completed, and queued copies finalized without data loss.
Corrective action
Improved migration sizing with object-level throughput sampling and added automatic window-extension notices.
Affected components
Response timeline
Investigating · Dec 3, 2022, 7:30 AM UTC
Operations began reviewing identity, security & storage telemetry and affected processing queues.
Identified · Dec 3, 2022, 8:32 AM UTC
Checksum verification took longer on several large media objects than the migration estimate allowed.
Monitoring · Dec 3, 2022, 11:08 AM UTC
The migration window was extended, checksums completed, and queued copies finalized without data loss. Teams monitored recovery indicators before closing the report.
Resolved · Dec 3, 2022, 12:42 PM UTC
Normal performance was confirmed. Follow-up action: Improved migration sizing with object-level throughput sampling and added automatic window-extension notices.
Third Party and Partners
Bulk ISRC enrichment slowed temporarily while direct release processing continued.
Resolution time
Oct 18, 2022, 5:25 PM UTC
1h 23m
Impact
Bulk ISRC enrichment slowed temporarily while direct release processing continued.
Root cause
A batch enrichment task exceeded the conservative external lookup budget.
Resolution
The task was throttled and remaining identifiers were processed within the available request budget.
Corrective action
Added adaptive rate control, request deduplication, and cached reuse of recent identifier results.
Affected components
Response timeline
Investigating · Oct 18, 2022, 4:02 PM UTC
Operations began reviewing third party and partners telemetry and affected processing queues.
Identified · Oct 18, 2022, 4:19 PM UTC
A batch enrichment task exceeded the conservative external lookup budget.
Monitoring · Oct 18, 2022, 5:00 PM UTC
The task was throttled and remaining identifiers were processed within the available request budget. Teams monitored recovery indicators before closing the report.
Resolved · Oct 18, 2022, 5:25 PM UTC
Normal performance was confirmed. Follow-up action: Added adaptive rate control, request deduplication, and cached reuse of recent identifier results.
Rights, Royalties & Finance
Territory-level report exports completed later than normal. Source statements and totals remained unchanged.
Resolution time
Sep 1, 2022, 7:51 AM UTC
2h 7m
Impact
Territory-level report exports completed later than normal. Source statements and totals remained unchanged.
Root cause
A reporting query generated an inefficient plan after a statistics refresh.
Resolution
The query plan was stabilized and delayed reports were regenerated.
Corrective action
Added plan regression checks and pinned tested query strategies for large territory datasets.
Affected components
Response timeline
Investigating · Sep 1, 2022, 5:44 AM UTC
Operations began reviewing rights, royalties & finance telemetry and affected processing queues.
Identified · Sep 1, 2022, 6:09 AM UTC
A reporting query generated an inefficient plan after a statistics refresh.
Monitoring · Sep 1, 2022, 7:13 AM UTC
The query plan was stabilized and delayed reports were regenerated. Teams monitored recovery indicators before closing the report.
Resolved · Sep 1, 2022, 7:51 AM UTC
Normal performance was confirmed. Follow-up action: Added plan regression checks and pinned tested query strategies for large territory datasets.
Audio & Media Processing
New artwork was accepted but took longer to appear at a subset of edge locations.
Resolution time
Jul 24, 2022, 8:02 PM UTC
46 minutes
Impact
New artwork was accepted but took longer to appear at a subset of edge locations.
Root cause
One cache region delayed invalidation after an artwork replacement.
Resolution
The affected cache region was purged and the current artwork propagated globally.
Corrective action
Added regional propagation checks and automatic re-purge when edge versions disagree.
Affected components
Response timeline
Investigating · Jul 24, 2022, 7:16 PM UTC
Operations began reviewing audio & media processing telemetry and affected processing queues.
Identified · Jul 24, 2022, 7:25 PM UTC
One cache region delayed invalidation after an artwork replacement.
Monitoring · Jul 24, 2022, 7:48 PM UTC
The affected cache region was purged and the current artwork propagated globally. Teams monitored recovery indicators before closing the report.
Resolved · Jul 24, 2022, 8:02 PM UTC
Normal performance was confirmed. Follow-up action: Added regional propagation checks and automatic re-purge when edge versions disagree.
Catalog & Release Operations
A repeated upload remained queued for review longer than expected. No duplicate release record was published.
Resolution time
Jun 10, 2022, 12:09 PM UTC
1h 31m
Impact
A repeated upload remained queued for review longer than expected. No duplicate release record was published.
Root cause
The deduplication fingerprint was generated after the job entered the parsing queue instead of before enqueue.
Resolution
The job was safely cancelled and fingerprint generation was moved ahead of queue insertion.
Corrective action
Added prequeue duplicate detection and explicit duplicate-job status messages.
Affected components
Response timeline
Investigating · Jun 10, 2022, 10:38 AM UTC
Operations began reviewing catalog & release operations telemetry and affected processing queues.
Identified · Jun 10, 2022, 10:56 AM UTC
The deduplication fingerprint was generated after the job entered the parsing queue instead of before enqueue.
Monitoring · Jun 10, 2022, 11:42 AM UTC
The job was safely cancelled and fingerprint generation was moved ahead of queue insertion. Teams monitored recovery indicators before closing the report.
Resolved · Jun 10, 2022, 12:09 PM UTC
Normal performance was confirmed. Follow-up action: Added prequeue duplicate detection and explicit duplicate-job status messages.
Core Platform
Recently edited release details could require a manual refresh to display the latest values. Saved data remained current.
Resolution time
Apr 29, 2022, 12:03 AM UTC
58 minutes
Impact
Recently edited release details could require a manual refresh to display the latest values. Saved data remained current.
Root cause
A cache invalidation event was dropped during a brief client reconnection.
Resolution
The affected query keys were invalidated and detail pages refreshed from source records.
Corrective action
Added reconnect-time invalidation replay and stale-data indicators for release detail views.
Affected components
Response timeline
Investigating · Apr 28, 2022, 11:05 PM UTC
Operations began reviewing core platform telemetry and affected processing queues.
Identified · Apr 28, 2022, 11:17 PM UTC
A cache invalidation event was dropped during a brief client reconnection.
Monitoring · Apr 28, 2022, 11:46 PM UTC
The affected query keys were invalidated and detail pages refreshed from source records. Teams monitored recovery indicators before closing the report.
Resolved · Apr 29, 2022, 12:03 AM UTC
Normal performance was confirmed. Follow-up action: Added reconnect-time invalidation replay and stale-data indicators for release detail views.
Catalog & Release Operations
Some release summaries displayed Pending longer than expected after partner delivery receipts arrived.
Resolution time
Mar 12, 2022, 4:43 PM UTC
2h 53m
Impact
Some release summaries displayed Pending longer than expected after partner delivery receipts arrived.
Root cause
Status-derivation jobs were waiting behind a large catalog normalization batch.
Resolution
Status jobs were moved to a priority queue and affected release summaries were recalculated.
Corrective action
Separated delivery-state work from bulk catalog normalization and reserved capacity for receipt-driven updates.
Affected components
Response timeline
Investigating · Mar 12, 2022, 1:50 PM UTC
Operations began reviewing catalog & release operations telemetry and affected processing queues.
Identified · Mar 12, 2022, 2:25 PM UTC
Status-derivation jobs were waiting behind a large catalog normalization batch.
Monitoring · Mar 12, 2022, 3:51 PM UTC
Status jobs were moved to a priority queue and affected release summaries were recalculated. Teams monitored recovery indicators before closing the report.
Resolved · Mar 12, 2022, 4:43 PM UTC
Normal performance was confirmed. Follow-up action: Separated delivery-state work from bulk catalog normalization and reserved capacity for receipt-driven updates.
Support & Communications
A routine maintenance notice appeared later than its scheduled publication time. Service health data remained accessible.
Resolution time
Jan 20, 2022, 3:22 AM UTC
1h 8m
Impact
A routine maintenance notice appeared later than its scheduled publication time. Service health data remained accessible.
Root cause
The notice scheduler used an incorrect timezone offset for one publication rule.
Resolution
The schedule was corrected and the notice was published with accurate timing.
Corrective action
Standardized schedules on UTC storage with explicit display-zone conversion and prepublication checks.
Affected components
Response timeline
Investigating · Jan 20, 2022, 2:14 AM UTC
Operations began reviewing support & communications telemetry and affected processing queues.
Identified · Jan 20, 2022, 2:28 AM UTC
The notice scheduler used an incorrect timezone offset for one publication rule.
Monitoring · Jan 20, 2022, 3:02 AM UTC
The schedule was corrected and the notice was published with accurate timing. Teams monitored recovery indicators before closing the report.
Resolved · Jan 20, 2022, 3:22 AM UTC
Normal performance was confirmed. Follow-up action: Standardized schedules on UTC storage with explicit display-zone conversion and prepublication checks.
Incident history
4 maintenance windows that have concluded, retained here as part of the status history. Affected services have returned to their normal operating state.
Apple Podcasts Connect and Apple Podcasts Analytics are within an extended scheduled-maintenance period. Apple indicates normal operation is expected to be restored on August 8, 2026.
Apple timeline
Maintenance begins
Jul 30, 2026, 2:00 PM PDT · Jul 30, 2026, 5:00 PM EDT
Apple begins the scheduled maintenance window. Apple Podcasts Connect and Analytics access may become unavailable.
Access interruption expected
Jul 30, 2026, 2:05 PM PDT · Jul 30, 2026, 5:05 PM EDT
Podcast creators may be unable to sign in to Apple Podcasts Connect or view Analytics during the maintenance period.
Mid-window monitoring
Jul 30, 2026, 3:00 PM PDT · Jul 30, 2026, 6:00 PM EDT
SSO continues monitoring Apple service availability and partner delivery acknowledgements while maintenance is in progress.
Apple published maintenance window begins
Aug 8, 2026, 6:00 AM PDT · Aug 8, 2026, 9:00 AM EDT
Apple’s published August 8 maintenance window begins at 6:00 a.m. PDT. Connect and Analytics access may be unavailable for up to two hours.
Normal operation expected
Aug 8, 2026, 8:00 AM PDT · Aug 8, 2026, 11:00 AM EDT
Apple indicates normal Apple Podcasts Connect and Analytics operation is expected to be restored by August 8 at approximately 8:00 a.m. PDT.
The iTunes Connect publishing portal used for Apple Music and Apple Books is within an extended scheduled-maintenance period. Transporter deliveries are expected to remain available, although processing delays may occur. Apple indicates normal operation is expected to be restored on August 8, 2026.
Apple timeline
Publishing portal maintenance begins
Jul 30, 2026, 2:00 PM PDT · Jul 30, 2026, 5:00 PM EDT
Apple begins maintenance affecting iTunes Connect access for Apple Music and Apple Books publishing workflows.
Connect access unavailable
Jul 30, 2026, 2:05 PM PDT · Jul 30, 2026, 5:05 PM EDT
Users may be unable to access the iTunes Connect publishing portal or complete portal-based changes during the window.
Transporter remains available
Jul 30, 2026, 2:10 PM PDT · Jul 30, 2026, 5:10 PM EDT
New music and book deliveries sent through Transporter are expected to continue, but Apple may process them more slowly until maintenance completes.
Delivery processing monitored
Jul 30, 2026, 3:00 PM PDT · Jul 30, 2026, 6:00 PM EDT
SSO monitors Apple Music and Apple Books delivery receipts, processing acknowledgements, and any downstream delays.
Apple published maintenance window begins
Aug 8, 2026, 6:00 AM PDT · Aug 8, 2026, 9:00 AM EDT
Apple’s published August 8 maintenance window begins at 6:00 a.m. PDT for iTunes Connect publishing services used by Apple Music and Apple Books.
Normal operation expected
Aug 8, 2026, 8:00 AM PDT · Aug 8, 2026, 11:00 AM EDT
Apple indicates normal iTunes Connect access and publishing-portal processing are expected to be restored by August 8 at approximately 8:00 a.m. PDT.
SSO Gateway analytics services are operating in a limited state while SSO engineers and developers complete backend and server updates related to new integrations, server rehauls, and merges to new server grids. Related UI work has been completed; the limited performance is scoped to analytics endpoints and reporting surfaces.
Apple timeline
Limited-performance window begins
Aug 2, 2026, 5:30 PM PDT · Aug 2, 2026, 8:30 PM EDT
SSO Gateway analytics endpoints move into a limited-performance state as backend and server-grid work begins.
UI work complete
Aug 2, 2026, 6:00 PM PDT · Aug 2, 2026, 9:00 PM EDT
Gateway UI updates related to this maintenance have been completed; remaining impact is limited to analytics services.
Backend and server rehaul underway
Aug 6, 2026, 5:00 PM PDT · Aug 6, 2026, 8:00 PM EDT
Engineering continues backend updates, new integration wiring, server rehauls, and merges to the new server grids.
Final verification
Aug 13, 2026, 5:00 AM PDT · Aug 13, 2026, 8:00 AM EDT
SSO engineers verify analytics endpoints against the rebuilt server grids ahead of restoration.
Full restoration expected
Aug 13, 2026, 9:00 PM PDT · Aug 14, 2026, 12:00 AM EDT
SSO Gateway analytics services are expected to return to normal performance as the maintenance window closes.
SSO Gateway platform services outside of music delivery are in scheduled maintenance while SSO engineers and developers complete major updates. UI work has been completed; maintenance covers music videos, podcasts, and advanced DDEX services (DDEX file tools and DDEX file ingestion via PATH and APIs from SSO Digital Solutions subscriptions) as backend and server updates, new integrations, server rehauls, and merges to new server grids are performed.
Apple timeline
Maintenance begins
Aug 2, 2026, 5:30 PM PDT · Aug 2, 2026, 8:30 PM EDT
SSO Gateway platform maintenance begins for music videos, podcasts, and advanced DDEX tooling. Music delivery is excluded and continues operating.
UI updates completed
Aug 2, 2026, 6:00 PM PDT · Aug 2, 2026, 9:00 PM EDT
UI updates tied to this maintenance have been completed; remaining work is backend-only.
Backend and server-grid work continues
Aug 6, 2026, 5:00 PM PDT · Aug 6, 2026, 8:00 PM EDT
SSO engineers continue backend updates, new integration wiring, server rehauls, and merges to the new server grids.
Verification phase
Aug 13, 2026, 5:00 AM PDT · Aug 13, 2026, 8:00 AM EDT
Engineering verifies DDEX file tools, PATH/API ingestion, music video, and podcast workflows against the rebuilt infrastructure.
Restoration expected
Aug 13, 2026, 9:00 PM PDT · Aug 14, 2026, 12:00 AM EDT
SSO Gateway platform services are expected to return to normal operation as the maintenance window closes.
450+ Global Platforms
Representative streaming, social, retail, metadata, fitness, and commercial-music destinations monitored across the delivery network.
View complete endpoint directory