FOC Devnet scenarios run report (frontier)

[FilOzzy]

The frontier scenarios run failed ❌.
See the workflow run for details.

Scenarios Tests

Description	Data
Type	frontier
Date	26-June-2026 07:27:11 GMT +0
Status	PASS ✅:4, FAIL 🟥:1, Total:5
CI run	https://github.com/FilOzone/foc-devnet/actions/runs/28221902095

Versions info

foc-devnet version: not available

Tests summary

✅ test_containers - PASS (0s)

[ OK ] container foc-20260626T0709_KlutzKiki-lotus is running
[ OK ] container foc-20260626T0709_KlutzKiki-lotus-miner is running
[ OK ] container foc-20260626T0709_KlutzKiki-curio-1 is running
[ OK ] container foc-20260626T0709_KlutzKiki-curio-2 is running

✅ test_basic_balances - PASS (0s)

[ OK ] cast is installed
[ OK ] at least one user exists
[ OK ] USER_1 FIL balance > 0
[ OK ] USER_1 USDFC balance > 0
[ OK ] USER_2 FIL balance > 0
[ OK ] USER_2 USDFC balance > 0
[ OK ] USER_3 FIL balance > 0
[ OK ] USER_3 USDFC balance > 0

❌ test_storage_e2e - FAIL (67s)

[FAIL] NETWORK=devnet node utils/example-storage-e2e.js random_file (exit=1) Error: RPC Request failed.

URL: http://localhost:5701/rpc/v1
Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}
 
Raw Call Arguments:
  from:  0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
  to:    0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7
  data:  0x0a9cb4a7
 
Contract Call:
  address:   0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7
  function:  getProviderIds()

Docs: https://viem.sh/docs/contract/readContract
Details: refusing explicit call due to state fork at epoch
Version: viem@2.52.2
Caused by: RPC Request failed.

URL: http://localhost:5701/rpc/v1
Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}
 
Raw Call Arguments:
  from:  0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
  to:    0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7
  data:  0x0a9cb4a7

Details: refusing explicit call due to state fork at epoch
Version: viem@2.52.2
ContractFunctionExecutionError: RPC Request failed.

URL: http://localhost:5701/rpc/v1
Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}
 
Raw Call Arguments:
  from:  0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
  to:    0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7
  data:  0x0a9cb4a7
 
Contract Call:
  address:   0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7
  function:  getProviderIds()

Docs: https://viem.sh/docs/contract/readContract
Details: refusing explicit call due to state fork at epoch
Version: viem@2.52.2
    at getContractError (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/utils/errors/getContractError.js:34:12)
    at readContract (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/actions/public/readContract.js:58:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async getEndorsedProviderIds (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/packages/synapse-core/dist/src/endorsements/get-endorsed-provider-ids.js:4:18)
    ... 4 lines matching cause stack trace ...
    at async StorageManager.createContexts (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/packages/synapse-sdk/src/storage/manager.ts:921:22)
    at async StorageManager.prepare (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/packages/synapse-sdk/src/storage/manager.ts:667:11) {
  cause: CallExecutionError: RPC Request failed.
  
  URL: http://localhost:5701/rpc/v1
  Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}
   
  Raw Call Arguments:
    from:  0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
    to:    0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7
    data:  0x0a9cb4a7
  
  Details: refusing explicit call due to state fork at epoch
  Version: viem@2.52.2
      at getCallError (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/utils/errors/getCallError.js:11:12)
      at call (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/actions/public/call.js:175:15)
      at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
      at async readContract (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/actions/public/readContract.js:45:26)
      at async getEndorsedProviderIds (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/packages/synapse-core/dist/src/endorsements/get-endorsed-provider-ids.js:4:18)
      at async Promise.all (index 1)
      at async fetchProviderSelectionInput (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/packages/synapse-core/dist/src/warm-storage/fetch-provider-selection-input.js:5:51)
      at async StorageContext.smartSelect (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/packages/synapse-sdk/src/storage/context.ts:576:19)
      at async StorageContext.createContexts (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/packages/synapse-sdk/src/storage/context.ts:275:21)
      at async StorageManager.createContexts (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/packages/synapse-sdk/src/storage/manager.ts:921:22) {
    cause: RpcRequestError: RPC Request failed.
    
    URL: http://localhost:5701/rpc/v1
    Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}
    
    Details: refusing explicit call due to state fork at epoch
    Version: viem@2.52.2
        at request (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/clients/transports/http.js:72:27)
        at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
        at async withRetry.delay.count.count (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/utils/buildRequest.js:30:24)
        at async attemptRetry (file:///tmp/synapse-sdk-rqrqpaqz/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/utils/promise/withRetry.js:24:30) {
      details: 'refusing explicit call due to state fork at epoch',
      docsPath: undefined,
      metaMessages: [Array],
      shortMessage: 'RPC Request failed.',
      version: '2.52.2',
      code: 1,
      data: undefined,
      url: 'http://localhost:5701/rpc/v1',
      [cause]: [Object]
    },
    details: 'refusing explicit call due to state fork at epoch',
    docsPath: undefined,
    metaMessages: [
      'URL: http://localhost:5701/rpc/v1',
      'Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}',
      ' ',
      'Raw Call Arguments:',
      '  from:  0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c\n' +
        '  to:    0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7\n' +
        '  data:  0x0a9cb4a7'
    ],
    shortMessage: 'RPC Request failed.',
    version: '2.52.2'
  },
  details: 'refusing explicit call due to state fork at epoch',
  docsPath: '/docs/contract/readContract',
  metaMessages: [
    'URL: http://localhost:5701/rpc/v1',
    'Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}',
    ' ',
    'Raw Call Arguments:',
    '  from:  0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c\n' +
      '  to:    0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7\n' +
      '  data:  0x0a9cb4a7',
    ' ',
    'Contract Call:',
    '  address:   0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7\n' +
      '  function:  getProviderIds()'
  ],
  shortMessage: 'RPC Request failed.',
  version: '2.52.2',
  abi: [
    { type: 'constructor', inputs: [], stateMutability: 'nonpayable' },
    {
      type: 'function',
      inputs: [Array],
      name: 'addProviderId',
      outputs: [],
      stateMutability: 'nonpayable'
    },
    {
      type: 'function',
      inputs: [Array],
      name: 'containsProviderId',
      outputs: [Array],
      stateMutability: 'view'
    },
    {
      type: 'function',
      inputs: [],
      name: 'getProviderIds',
      outputs: [Array],
      stateMutability: 'view'
    },
    {
      type: 'function',
      inputs: [],
      name: 'owner',
      outputs: [Array],
      stateMutability: 'view'
    },
    {
      type: 'function',
      inputs: [Array],
      name: 'removeProviderId',
      outputs: [],
      stateMutability: 'nonpayable'
    },
    {
      type: 'function',
      inputs: [],
      name: 'renounceOwnership',
      outputs: [],
      stateMutability: 'nonpayable'
    },
    {
      type: 'function',
      inputs: [Array],
      name: 'transferOwnership',
      outputs: [],
      stateMutability: 'nonpayable'
    },
    {
      type: 'event',
      anonymous: false,
      inputs: [Array],
      name: 'OwnershipTransferred'
    },
    { type: 'error', inputs: [Array], name: 'OwnableInvalidOwner' },
    {
      type: 'error',
      inputs: [Array],
      name: 'OwnableUnauthorizedAccount'
    },
    { type: 'error', inputs: [Array], name: 'ProviderIdNotFound' },
    { type: 'error', inputs: [Array], name: 'ProviderIdTooLarge' }
  ],
  args: [],
  contractAddress: '0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7',
  formattedArgs: undefined,
  functionName: 'getProviderIds',
  sender: undefined
}
[ OK ] git is installed
[ OK ] node is installed
[ OK ] pnpm is installed
[ OK ] clone synapse-sdk
[ OK ] checkout master HEAD
[INFO] synapse-sdk commit: c18a0de03935002487ece0a99f697be490cd3eee
[ OK ] pnpm install
[ OK ] pnpm build
[INFO] Creating random file (20971520 bytes)
[ OK ] random_file created with exact size 20971520 bytes
[INFO] Running Synapse SDK storage e2e script against devnet

✅ test_multi_copy_upload - PASS (56s)

[ OK ] node is installed
[ OK ] npm is installed
[ OK ] npm init
[ OK ] pin filecoin-pin dependencies
[ OK ] npm install
[INFO] @filoz/synapse-core streaming upload already free of Content-Length header; no patch needed
[INFO] Creating random file (20971520 bytes)
[ OK ] random_file created with exact size 20971520 bytes
[INFO] Running filecoin-pin multi-copy upload script against devnet
[INFO] filecoin-pin add attempt 1 (default multi-copy)
[INFO] Verified retrieval URL 1: http://host.docker.internal:5721/ipfs/bafybeicscvnp3dhanfl3yrgdavs35em6i4ycrkon5gvoggfnn6gfqdfj4q
[INFO] Verified retrieval URL 2: http://host.docker.internal:5725/ipfs/bafybeicscvnp3dhanfl3yrgdavs35em6i4ycrkon5gvoggfnn6gfqdfj4q

✅ test_caching_subsystem - PASS (176s)

[ OK ] git is installed
[ OK ] node is installed
[ OK ] pnpm is installed
[INFO] Run index: 1, seeds: small=43, large=86
[INFO] cqlsh version: cqlsh 6.2.0
[INFO] Yugabyte cassandra port: localhost:5705
[INFO] Initial row count = 0
[ OK ] clone synapse-sdk
[ OK ] checkout master HEAD
[INFO] synapse-sdk commit: c18a0de03935002487ece0a99f697be490cd3eee
[ OK ] pnpm install
[ OK ] pnpm build
[INFO] Uploading 20MB piece (below 32MB threshold)
[INFO] Loading devnet info from: /home/runner/.foc-devnet/state/latest/devnet-info.json
Devnet run: 20260626T0709_KlutzKiki
Using user: USER_1 (0x47cc9101fd026fc112d7fadf6b3c9df5be7d4a8c)
SPs available: 2
=== Synapse SDK Storage E2E Example ===

Reading file...
  small_20mb (20 MB)

--- Initializing Synapse SDK ---
Network: FOC DevNet
RPC URL: http://localhost:5701/rpc/v1
Wallet address: 0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
Synapse instance created

--- Checking Balances ---
FIL balance: 999.9999699069999 FIL
USDFC balance: 999.999970 USDFC

--- Preparing Account ---
Estimated costs:
  Per epoch (30s): 0.000001 USDFC
  Per month: 0.048095 USDFC
  Deposit needed: 0.000000 USDFC
  Ready: true
Account already ready

--- Uploading ---
Uploading small_20mb (20 MB) via stream...

  Selected SP 1 (0x446339aE7245e3cd1FeD701b685C196C69af695e)
  Selected SP 2 (0xC60bc6fb60901c2d28ba6e4F8AD4915Bd2106497)
  Upload progress: 10 MB (50.0%)
  Upload progress: 20 MB (100.0%)
  Stored on SP 1: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Pulling to SP 2: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa (pending)
  Pulling to SP 2: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa (inProgress)
  Pulling to SP 2: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa (complete)
  Copied to SP 2: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Pieces committed on SP 1, tx: 0x9469088af68a164c14716cd80ee945d3ca929f878594348ed35c2ba38b020cb1
    bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Pieces committed on SP 2, tx: 0x8b540b7fb49f7c6c08185e661927980f81c29aa1a4d0cc4207716ffe6487cfbb
    bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Data set 4 confirmed on SP 1
    bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa -> pieceId 0
  Data set 3 confirmed on SP 2
    bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa -> pieceId 0

--- Upload Summary ---

File: small_20mb
  PieceCID: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Size: 20 MB
  [Primary]   Provider 1 - pieceId: 0, dataSetId: 4
  [Secondary] Provider 2 - pieceId: 0, dataSetId: 3

--- Downloading and Verifying ---

Downloading bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa...
  VERIFIED: 20 MB matches original

=== SUCCESS: All files uploaded, replicated, and verified ===
The service provider(s) will periodically prove they still have your data.
You are being charged based on the storage size and duration.
[ OK ] upload 20MB piece
[INFO] Waiting 10s for caching tasks
[INFO] row_count after 'upload 20MB piece' = 0
[ OK ] cache rows count should not increase
[INFO] Uploading 80MB piece (above 32MB threshold)
[INFO] Loading devnet info from: /home/runner/.foc-devnet/state/latest/devnet-info.json
Devnet run: 20260626T0709_KlutzKiki
Using user: USER_1 (0x47cc9101fd026fc112d7fadf6b3c9df5be7d4a8c)
SPs available: 2
=== Synapse SDK Storage E2E Example ===

Reading file...
  large_80mb (80 MB)

--- Initializing Synapse SDK ---
Network: FOC DevNet
RPC URL: http://localhost:5701/rpc/v1
Wallet address: 0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
Synapse instance created

--- Checking Balances ---
FIL balance: 999.9999699069999 FIL
USDFC balance: 999.999970 USDFC

--- Preparing Account ---
Estimated costs:
  Per epoch (30s): 0.000001 USDFC
  Per month: 0.048478 USDFC
  Deposit needed: 0.000000 USDFC
  Ready: true
Account already ready

--- Uploading ---
Uploading large_80mb (80 MB) via stream...

  Selected SP 1 (0x446339aE7245e3cd1FeD701b685C196C69af695e)
  Creating new data set: 4
  Selected SP 2 (0xC60bc6fb60901c2d28ba6e4F8AD4915Bd2106497)
  Creating new data set: 3
  Upload progress: 10 MB (12.5%)
  Upload progress: 20 MB (25.0%)
  Upload progress: 30 MB (37.5%)
  Upload progress: 40 MB (50.0%)
  Upload progress: 50 MB (62.5%)
  Upload progress: 60 MB (75.0%)
  Upload progress: 70 MB (87.5%)
  Upload progress: 80 MB (100.0%)
  Stored on SP 1: bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima
  Pulling to SP 2: bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima (pending)
  Pulling to SP 2: bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima (inProgress)
  Pulling to SP 2: bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima (complete)
  Copied to SP 2: bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima
  Pieces committed on SP 1, tx: 0x710b0c9d68879ade262c83c770e29eedd20e6feea2a8e7045ed19e444c8e9030
    bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima
  Pieces committed on SP 2, tx: 0x06db8eb4d42735d7dc7d764a3e04e879e9a82b677680eb6bd5047e94f60b923a
    bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima
  Data set 4 confirmed on SP 1
    bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima -> pieceId 1
  Data set 3 confirmed on SP 2
    bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima -> pieceId 1

--- Upload Summary ---

File: large_80mb
  PieceCID: bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima
  Size: 80 MB
  [Primary]   Provider 1 - pieceId: 1, dataSetId: 4
  [Secondary] Provider 2 - pieceId: 1, dataSetId: 3

--- Downloading and Verifying ---

Downloading bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima...
  VERIFIED: 80 MB matches original

=== SUCCESS: All files uploaded, replicated, and verified ===
The service provider(s) will periodically prove they still have your data.
You are being charged based on the storage size and duration.
[ OK ] upload 80MB piece
[INFO] Waiting 10s for caching tasks
[INFO] row_count after 'upload 80MB piece' = 32
[ OK ] cache rows count should increase

[Action Run link](https://github.com/FilOzone/foc-devnet/actions/runs/28221902095)

[FilOzzy]

The frontier scenarios run failed ❌.
See the workflow run for details.

Scenarios Tests

Description	Data
Type	frontier
Date	26-June-2026 08:00:32 GMT +0
Status	PASS ✅:4, FAIL 🟥:1, Total:5
CI run	https://github.com/FilOzone/foc-devnet/actions/runs/28221902095

Versions info

foc-devnet version: not available

Tests summary

✅ test_containers - PASS (0s)

[ OK ] container foc-20260626T0746_WonkyBug-lotus is running
[ OK ] container foc-20260626T0746_WonkyBug-lotus-miner is running
[ OK ] container foc-20260626T0746_WonkyBug-curio-1 is running
[ OK ] container foc-20260626T0746_WonkyBug-curio-2 is running

✅ test_basic_balances - PASS (0s)

[ OK ] cast is installed
[ OK ] at least one user exists
[ OK ] USER_1 FIL balance > 0
[ OK ] USER_1 USDFC balance > 0
[ OK ] USER_2 FIL balance > 0
[ OK ] USER_2 USDFC balance > 0
[ OK ] USER_3 FIL balance > 0
[ OK ] USER_3 USDFC balance > 0

✅ test_storage_e2e - PASS (89s)

[ OK ] git is installed
[ OK ] node is installed
[ OK ] pnpm is installed
[ OK ] clone synapse-sdk
[ OK ] checkout master HEAD
[INFO] synapse-sdk commit: c18a0de03935002487ece0a99f697be490cd3eee
[ OK ] pnpm install
[ OK ] pnpm build
[INFO] Creating random file (20971520 bytes)
[ OK ] random_file created with exact size 20971520 bytes
[INFO] Running Synapse SDK storage e2e script against devnet
[INFO] Loading devnet info from: /home/runner/.foc-devnet/state/latest/devnet-info.json
Devnet run: 20260626T0746_WonkyBug
Using user: USER_1 (0x47cc9101fd026fc112d7fadf6b3c9df5be7d4a8c)
SPs available: 2
=== Synapse SDK Storage E2E Example ===

Reading file...
  random_file (20 MB)

--- Initializing Synapse SDK ---
Network: FOC DevNet
RPC URL: http://localhost:5701/rpc/v1
Wallet address: 0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
Synapse instance created

--- Checking Balances ---
FIL balance: 999.9999698778 FIL
USDFC balance: 999.999970 USDFC

--- Preparing Account ---
Estimated costs:
  Per epoch (30s): 0.000001 USDFC
  Per month: 0.048095 USDFC
  Deposit needed: 0.000000 USDFC
  Ready: true
Account already ready

--- Uploading ---
Uploading random_file (20 MB) via stream...

  Selected SP 1 (0x446339aE7245e3cd1FeD701b685C196C69af695e)
  Selected SP 2 (0xC60bc6fb60901c2d28ba6e4F8AD4915Bd2106497)
  Upload progress: 10 MB (50.0%)
  Upload progress: 20 MB (100.0%)
  Stored on SP 1: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Pulling to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq (pending)
  Pulling to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq (inProgress)
  Pulling to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq (complete)
  Copied to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Pieces committed on SP 2, tx: 0xc65815a9786fc1678a94e8175d49c741db60ea6fd70762d16fbccfc90b307909
    bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Pieces committed on SP 1, tx: 0x194a823e6f653b9b94dbf9c147cec869d55b3fe54e263d943fcd551331092026
    bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Data set 2 confirmed on SP 1
    bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq -> pieceId 0
  Data set 1 confirmed on SP 2
    bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq -> pieceId 0

--- Upload Summary ---

File: random_file
  PieceCID: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Size: 20 MB
  [Primary]   Provider 1 - pieceId: 0, dataSetId: 2
  [Secondary] Provider 2 - pieceId: 0, dataSetId: 1

--- Downloading and Verifying ---

Downloading bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq...
  VERIFIED: 20 MB matches original

=== SUCCESS: All files uploaded, replicated, and verified ===
The service provider(s) will periodically prove they still have your data.
You are being charged based on the storage size and duration.
[ OK ] NETWORK=devnet node utils/example-storage-e2e.js random_file

✅ test_multi_copy_upload - PASS (49s)

[ OK ] node is installed
[ OK ] npm is installed
[ OK ] npm init
[ OK ] pin filecoin-pin dependencies
[ OK ] npm install
[INFO] @filoz/synapse-core streaming upload already free of Content-Length header; no patch needed
[INFO] Creating random file (20971520 bytes)
[ OK ] random_file created with exact size 20971520 bytes
[INFO] Running filecoin-pin multi-copy upload script against devnet
[INFO] filecoin-pin add attempt 1 (default multi-copy)
[INFO] Verified retrieval URL 1: http://host.docker.internal:5721/ipfs/bafybeicscvnp3dhanfl3yrgdavs35em6i4ycrkon5gvoggfnn6gfqdfj4q
[INFO] Verified retrieval URL 2: http://host.docker.internal:5725/ipfs/bafybeicscvnp3dhanfl3yrgdavs35em6i4ycrkon5gvoggfnn6gfqdfj4q

❌ test_caching_subsystem - FAIL (42s)

[FAIL] upload 20MB piece (exit=1) Error: RPC Request failed.

URL: http://localhost:5701/rpc/v1
Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}
 
Raw Call Arguments:
  from:  0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
  to:    0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7
  data:  0x0a9cb4a7
 
Contract Call:
  address:   0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7
  function:  getProviderIds()

Docs: https://viem.sh/docs/contract/readContract
Details: refusing explicit call due to state fork at epoch
Version: viem@2.52.2
Caused by: RPC Request failed.

URL: http://localhost:5701/rpc/v1
Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}
 
Raw Call Arguments:
  from:  0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
  to:    0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7
  data:  0x0a9cb4a7

Details: refusing explicit call due to state fork at epoch
Version: viem@2.52.2
ContractFunctionExecutionError: RPC Request failed.

URL: http://localhost:5701/rpc/v1
Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}
 
Raw Call Arguments:
  from:  0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
  to:    0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7
  data:  0x0a9cb4a7
 
Contract Call:
  address:   0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7
  function:  getProviderIds()

Docs: https://viem.sh/docs/contract/readContract
Details: refusing explicit call due to state fork at epoch
Version: viem@2.52.2
    at getContractError (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/utils/errors/getContractError.js:34:12)
    at readContract (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/actions/public/readContract.js:58:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async getEndorsedProviderIds (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/packages/synapse-core/dist/src/endorsements/get-endorsed-provider-ids.js:4:18)
    ... 4 lines matching cause stack trace ...
    at async StorageManager.createContexts (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/packages/synapse-sdk/src/storage/manager.ts:921:22)
    at async StorageManager.prepare (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/packages/synapse-sdk/src/storage/manager.ts:667:11) {
  cause: CallExecutionError: RPC Request failed.
  
  URL: http://localhost:5701/rpc/v1
  Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}
   
  Raw Call Arguments:
    from:  0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
    to:    0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7
    data:  0x0a9cb4a7
  
  Details: refusing explicit call due to state fork at epoch
  Version: viem@2.52.2
      at getCallError (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/utils/errors/getCallError.js:11:12)
      at call (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/actions/public/call.js:175:15)
      at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
      at async readContract (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/actions/public/readContract.js:45:26)
      at async getEndorsedProviderIds (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/packages/synapse-core/dist/src/endorsements/get-endorsed-provider-ids.js:4:18)
      at async Promise.all (index 1)
      at async fetchProviderSelectionInput (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/packages/synapse-core/dist/src/warm-storage/fetch-provider-selection-input.js:5:51)
      at async StorageContext.smartSelect (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/packages/synapse-sdk/src/storage/context.ts:576:19)
      at async StorageContext.createContexts (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/packages/synapse-sdk/src/storage/context.ts:275:21)
      at async StorageManager.createContexts (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/packages/synapse-sdk/src/storage/manager.ts:921:22) {
    cause: RpcRequestError: RPC Request failed.
    
    URL: http://localhost:5701/rpc/v1
    Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}
    
    Details: refusing explicit call due to state fork at epoch
    Version: viem@2.52.2
        at request (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/clients/transports/http.js:72:27)
        at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
        at async withRetry.delay.count.count (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/utils/buildRequest.js:30:24)
        at async attemptRetry (file:///tmp/synapse-sdk-cache-0r9ooh5v/synapse-sdk/node_modules/.pnpm/viem@2.52.2_typescript@6.0.3_zod@4.4.3/node_modules/viem/_esm/utils/promise/withRetry.js:24:30) {
      details: 'refusing explicit call due to state fork at epoch',
      docsPath: undefined,
      metaMessages: [Array],
      shortMessage: 'RPC Request failed.',
      version: '2.52.2',
      code: 1,
      data: undefined,
      url: 'http://localhost:5701/rpc/v1',
      [cause]: [Object]
    },
    details: 'refusing explicit call due to state fork at epoch',
    docsPath: undefined,
    metaMessages: [
      'URL: http://localhost:5701/rpc/v1',
      'Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}',
      ' ',
      'Raw Call Arguments:',
      '  from:  0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c\n' +
        '  to:    0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7\n' +
        '  data:  0x0a9cb4a7'
    ],
    shortMessage: 'RPC Request failed.',
    version: '2.52.2'
  },
  details: 'refusing explicit call due to state fork at epoch',
  docsPath: '/docs/contract/readContract',
  metaMessages: [
    'URL: http://localhost:5701/rpc/v1',
    'Request body: {"method":"eth_call","params":[{"data":"0x0a9cb4a7","from":"0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c","to":"0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7"},"latest"]}',
    ' ',
    'Raw Call Arguments:',
    '  from:  0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c\n' +
      '  to:    0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7\n' +
      '  data:  0x0a9cb4a7',
    ' ',
    'Contract Call:',
    '  address:   0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7\n' +
      '  function:  getProviderIds()'
  ],
  shortMessage: 'RPC Request failed.',
  version: '2.52.2',
  abi: [
    { type: 'constructor', inputs: [], stateMutability: 'nonpayable' },
    {
      type: 'function',
      inputs: [Array],
      name: 'addProviderId',
      outputs: [],
      stateMutability: 'nonpayable'
    },
    {
      type: 'function',
      inputs: [Array],
      name: 'containsProviderId',
      outputs: [Array],
      stateMutability: 'view'
    },
    {
      type: 'function',
      inputs: [],
      name: 'getProviderIds',
      outputs: [Array],
      stateMutability: 'view'
    },
    {
      type: 'function',
      inputs: [],
      name: 'owner',
      outputs: [Array],
      stateMutability: 'view'
    },
    {
      type: 'function',
      inputs: [Array],
      name: 'removeProviderId',
      outputs: [],
      stateMutability: 'nonpayable'
    },
    {
      type: 'function',
      inputs: [],
      name: 'renounceOwnership',
      outputs: [],
      stateMutability: 'nonpayable'
    },
    {
      type: 'function',
      inputs: [Array],
      name: 'transferOwnership',
      outputs: [],
      stateMutability: 'nonpayable'
    },
    {
      type: 'event',
      anonymous: false,
      inputs: [Array],
      name: 'OwnershipTransferred'
    },
    { type: 'error', inputs: [Array], name: 'OwnableInvalidOwner' },
    {
      type: 'error',
      inputs: [Array],
      name: 'OwnableUnauthorizedAccount'
    },
    { type: 'error', inputs: [Array], name: 'ProviderIdNotFound' },
    { type: 'error', inputs: [Array], name: 'ProviderIdTooLarge' }
  ],
  args: [],
  contractAddress: '0x4cfC6473C32aD43a88e6cE0c5a9618Ff5b6Ff8C7',
  formattedArgs: undefined,
  functionName: 'getProviderIds',
  sender: undefined
}
[ OK ] git is installed
[ OK ] node is installed
[ OK ] pnpm is installed
[INFO] Run index: 1, seeds: small=43, large=86
[INFO] cqlsh version: cqlsh 6.2.0
[INFO] Yugabyte cassandra port: localhost:5705
[INFO] Initial row count = 0
[ OK ] clone synapse-sdk
[ OK ] checkout master HEAD
[INFO] synapse-sdk commit: c18a0de03935002487ece0a99f697be490cd3eee
[ OK ] pnpm install
[ OK ] pnpm build
[INFO] Uploading 20MB piece (below 32MB threshold)

[Action Run link](https://github.com/FilOzone/foc-devnet/actions/runs/28221902095)

[rjan90]

I reran the failed frontier job to check whether this was a one-off. The rerun also failed: https://github.com/FilOzone/foc-devnet/actions/runs/28221902095/job/83612889693

Interesting bit: the failure moved between scenarios. In the original frontier run, test_storage_e2e failed. In the rerun, test_storage_e2e passed, but test_caching_subsystem failed during its first 20MB upload.

The underlying error is the same in both cases:

Details: refusing explicit call due to state fork at epoch

Both failures come from Synapse SDK/viem doing an eth_call to the Endorsements contract, specifically getProviderIds(). So this looks frontier-only and not tied to one specific scenario. It seems like a recurring FEVM/RPC read instability during Synapse provider selection.

Suggested next step: add a small diagnostic PR in this repo (foc-devnet) to print the chain head/epoch and eth block number around these SDK upload calls, and to capture Lotus logs after scenario failures. That should help confirm whether the refused eth_call lines up with a state-fork epoch before we decide whether to mitigate with targeted retry/backoff or chase an upstream frontier regression.

[rjan90]

Opened diagnostic draft PR: #137

I dispatched the nightly workflow against that branch with reporting disabled: https://github.com/FilOzone/foc-devnet/actions/runs/28227304981

Result from this diagnostic run:

• Frontier passed all 5 scenarios: https://github.com/FilOzone/foc-devnet/actions/runs/28227304981/job/83625936078
• Stability failed test_storage_e2e with the same underlying Endorsements.getProviderIds() eth_call error: https://github.com/FilOzone/foc-devnet/actions/runs/28227304981/job/83622269578

The new scenario diagnostics caught the direct probe failing before the SDK upload:

ChainHead height=201
eth_blockNumber=200 raw=0xc8
getProviderIds error: {"code": 1, "message": "refusing explicit call due to state fork at epoch"}

The immediate post-failure probe and the 5s retry both stayed at the same head/block and returned the same error. Later in the same stability run, test_caching_subsystem succeeded and the same direct getProviderIds() probe succeeded at higher heights.

So the latest diagnostic evidence suggests this may not be strictly frontier-only. It looks more like the eth_call latest refusal happens around a specific early chain height/state-fork boundary, and whichever Synapse scenario hits provider selection there fails.

[rjan90]

Follow-up from the diagnostic run and Lotus version check:

• The failing RPC is still the same concrete error: refusing explicit call due to state fork at epoch.
• The diagnostic run caught the failure at ChainHead height=201 / eth_blockNumber=200; the same direct Endorsements.getProviderIds() eth_call succeeded later once the chain advanced.
• The run is using Lotus v1.36.0, which shipped NV28. In that tag, the 2k params still have the named NV28/FireHorse upgrade at UpgradeFireHorseHeight = 200 and do not have a live LOTUS_XX_HEIGHT override. So our existing LOTUS_XX_HEIGHT=-1 env var is not affecting this release.
• That means this is probably not frontier-specific. It is timing-sensitive: whichever scenario hits Synapse provider selection while Lotus is sitting on the expensive state-fork boundary can fail.

I updated the draft fix PR here: #137

The PR now removes the temporary diagnostics and adds a narrow retry around Synapse uploads only when the command output contains the exact Lotus state-fork refusal. This should handle FireHorse in v1.36.0, the future XX skeleton, and future named upgrades without chasing release-specific env vars.