Skip to main content

Operations AI Conversations

Real interations with the operations assistant for monitoring processes, curves and data

Operations Assistant

Your workflow automation expert for managing data pipelines, monitoring operations, and troubleshooting issues.

Expertise Areas
  • Data loader configuration and monitoring
  • Workflow automation design
  • Operational alerts and exception handling
  • System health monitoring
  • Troubleshooting data quality issues
  • Scheduled task managements

Analyse a process execution​

The operations assistant take a deep-dive into why process executions and workflows fail

In this case, the process could not retrieve a file from the ftp server on the 31st December 2025, it correctly determined that the process was trying to retrieve the file based on the year for tomorrow, i.e. 2026

Can you check process execution b16fc5c0-d14a-4e7b-808d-a042bb8bf1e0 to see why the sftp is not working?
Answer from assistant: Operations

Investigation Results: SFTP Issue in EPEX_IDA2_SMARTLOADER

I found the root cause of the SFTP problem. Here's what happened:

The Problem:

The process executed successfully but encountered FTP service errors during the file retrieval phase. Specifically:

Error Code: [6502] - File not found

The SFTP server cannot find the requested files:

  • intraday_auction_spot_prices_SEM GB IDA2_2026.csv
  • intraday_auction_spot_volumes_SEM GB IDA2_2026.csv

Root Cause Analysis:

The process is looking for files with year 2026 in their filename:

sftp://opendsl.marketdata:@sftp.marketdata.epexspot.com/great-britain/Intraday%20Auction/GB%2030min%20IDM%20Auction%20D/Current/Prices_Volumes/intraday_auction_spot_prices_SEM%20GB%20IDA2_2026.csv

However, the process was executed on December 31, 2025 and the URL construction uses ${date:"tomorrow"}.year, which resolved to 2026.

The likely issue: The EPEX SFTP server may not have published the 2026 files yet, or they follow a different naming convention. The process expects yesterday's data but is asking for tomorrow's data by year.

Recommendations:

  1. Verify the SFTP directory structure - Connect to sftp.marketdata.epexspot.com and check what files are actually available in that directory

  2. Review the date logic - The expression ${date:"tomorrow"}.year might not be appropriate if you're processing historical data

  3. Check with EPEX - Contact the EPEX data provider to confirm:

    • When 2026 files will be available
    • The exact naming convention they use
    • If there's a file availability schedule

  4. Add error handling - Consider implementing retry logic or conditional file checks to handle missing files gracefully

  5. Update monitoring - Set up alerts specifically for FTP file not found errors (6502) to catch this issue earlier

Would you like me to check for any related alerts or examine other recent process executions to see if this is a pattern?