FOC Devnet scenarios run report (stability)

[FilOzzy]

The stability scenarios run failed ❌.
See the workflow run for details.

Scenarios Tests

Description	Data
Type	stability
Date	25-June-2026 07:04:17 GMT +0
Status	PASS ✅:4, FAIL 🟥:1, Total:5
CI run	https://github.com/FilOzone/foc-devnet/actions/runs/28151857223

Versions info

foc-devnet version: not available

Tests summary

✅ test_containers - PASS (0s)

[ OK ] container foc-20260625T0645_GoobsMimi-lotus is running
[ OK ] container foc-20260625T0645_GoobsMimi-lotus-miner is running
[ OK ] container foc-20260625T0645_GoobsMimi-curio-1 is running
[ OK ] container foc-20260625T0645_GoobsMimi-curio-2 is running

❌ test_basic_balances - FAIL (0s)

[FAIL] USER_1 USDFC balance > 0 (not an int: '')
[ OK ] cast is installed
[ OK ] at least one user exists
[ OK ] USER_1 FIL balance > 0

✅ test_storage_e2e - PASS (89s)

[ OK ] git is installed
[ OK ] node is installed
[ OK ] pnpm is installed
[ OK ] clone synapse-sdk
[ OK ] checkout master HEAD
[INFO] synapse-sdk commit: c18a0de03935002487ece0a99f697be490cd3eee
[ OK ] pnpm install
[ OK ] pnpm build
[INFO] Creating random file (20971520 bytes)
[ OK ] random_file created with exact size 20971520 bytes
[INFO] Running Synapse SDK storage e2e script against devnet
[INFO] Loading devnet info from: /home/runner/.foc-devnet/state/latest/devnet-info.json
Devnet run: 20260625T0645_GoobsMimi
Using user: USER_1 (0x47cc9101fd026fc112d7fadf6b3c9df5be7d4a8c)
SPs available: 2
=== Synapse SDK Storage E2E Example ===

Reading file...
  random_file (20 MB)

--- Initializing Synapse SDK ---
Network: FOC DevNet
RPC URL: http://localhost:5701/rpc/v1
Wallet address: 0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
Synapse instance created

--- Checking Balances ---
FIL balance: 999.9999697906001 FIL
USDFC balance: 999.999970 USDFC

--- Preparing Account ---
Estimated costs:
  Per epoch (30s): 0.000001 USDFC
  Per month: 0.048095 USDFC
  Deposit needed: 0.000000 USDFC
  Ready: true
Account already ready

--- Uploading ---
Uploading random_file (20 MB) via stream...

  Selected SP 1 (0x446339aE7245e3cd1FeD701b685C196C69af695e)
  Selected SP 2 (0xC60bc6fb60901c2d28ba6e4F8AD4915Bd2106497)
  Upload progress: 10 MB (50.0%)
  Upload progress: 20 MB (100.0%)
  Stored on SP 1: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Pulling to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq (pending)
  Pulling to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq (inProgress)
  Pulling to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq (complete)
  Copied to SP 2: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Pieces committed on SP 2, tx: 0x1edd8a804c1b7b0229f3effe97a8696254428634a7c1f68fb6082080715ba08b
    bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Pieces committed on SP 1, tx: 0x8b7ae7ff62c82b8dc9cc53966820c7ae1ac35c3e8389c59ff98e1c7928c0cf0e
    bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Data set 2 confirmed on SP 1
    bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq -> pieceId 0
  Data set 1 confirmed on SP 2
    bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq -> pieceId 0

--- Upload Summary ---

File: random_file
  PieceCID: bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq
  Size: 20 MB
  [Primary]   Provider 1 - pieceId: 0, dataSetId: 2
  [Secondary] Provider 2 - pieceId: 0, dataSetId: 1

--- Downloading and Verifying ---

Downloading bafkzcibfqcapabiuuwbw4j3fajxirwml7a4bkrhop6gipo3u47drjeusofmnyitg3yaq...
  VERIFIED: 20 MB matches original

=== SUCCESS: All files uploaded, replicated, and verified ===
The service provider(s) will periodically prove they still have your data.
You are being charged based on the storage size and duration.
[ OK ] NETWORK=devnet node utils/example-storage-e2e.js random_file

✅ test_multi_copy_upload - PASS (53s)

[ OK ] node is installed
[ OK ] npm is installed
[ OK ] npm init
[ OK ] pin filecoin-pin dependencies
[ OK ] npm install
[INFO] @filoz/synapse-core streaming upload already free of Content-Length header; no patch needed
[INFO] Creating random file (20971520 bytes)
[ OK ] random_file created with exact size 20971520 bytes
[INFO] Running filecoin-pin multi-copy upload script against devnet
[INFO] filecoin-pin add attempt 1 (default multi-copy)
[INFO] Verified retrieval URL 1: http://host.docker.internal:5721/ipfs/bafybeicscvnp3dhanfl3yrgdavs35em6i4ycrkon5gvoggfnn6gfqdfj4q
[INFO] Verified retrieval URL 2: http://host.docker.internal:5725/ipfs/bafybeicscvnp3dhanfl3yrgdavs35em6i4ycrkon5gvoggfnn6gfqdfj4q

✅ test_caching_subsystem - PASS (156s)

[ OK ] git is installed
[ OK ] node is installed
[ OK ] pnpm is installed
[INFO] Run index: 1, seeds: small=43, large=86
[INFO] cqlsh version: cqlsh 6.2.0
[INFO] Yugabyte cassandra port: localhost:5705
[INFO] Initial row count = 0
[ OK ] clone synapse-sdk
[ OK ] checkout master HEAD
[INFO] synapse-sdk commit: c18a0de03935002487ece0a99f697be490cd3eee
[ OK ] pnpm install
[ OK ] pnpm build
[INFO] Uploading 20MB piece (below 32MB threshold)
[INFO] Loading devnet info from: /home/runner/.foc-devnet/state/latest/devnet-info.json
Devnet run: 20260625T0645_GoobsMimi
Using user: USER_1 (0x47cc9101fd026fc112d7fadf6b3c9df5be7d4a8c)
SPs available: 2
=== Synapse SDK Storage E2E Example ===

Reading file...
  small_20mb (20 MB)

--- Initializing Synapse SDK ---
Network: FOC DevNet
RPC URL: http://localhost:5701/rpc/v1
Wallet address: 0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
Synapse instance created

--- Checking Balances ---
FIL balance: 999.9999697906001 FIL
USDFC balance: 999.999970 USDFC

--- Preparing Account ---
Estimated costs:
  Per epoch (30s): 0.000001 USDFC
  Per month: 0.048191 USDFC
  Deposit needed: 0.000000 USDFC
  Ready: true
Account already ready

--- Uploading ---
Uploading small_20mb (20 MB) via stream...

  Selected SP 1 (0x446339aE7245e3cd1FeD701b685C196C69af695e)
  Creating new data set: 2
  Selected SP 2 (0xC60bc6fb60901c2d28ba6e4F8AD4915Bd2106497)
  Creating new data set: 1
  Upload progress: 10 MB (50.0%)
  Upload progress: 20 MB (100.0%)
  Stored on SP 1: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Pulling to SP 2: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa (pending)
  Pulling to SP 2: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa (complete)
  Copied to SP 2: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Pieces committed on SP 2, tx: 0xf330a7bb2c9c99e46b89c906542fb136d1c2f8e155eea231f889d6ad0097e22f
    bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Pieces committed on SP 1, tx: 0x4906c8b343f942c1f2619a4fb9e2fc3224f8674658947bb2cbaf9ca98d46536a
    bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Data set 2 confirmed on SP 1
    bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa -> pieceId 1
  Data set 1 confirmed on SP 2
    bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa -> pieceId 1

--- Upload Summary ---

File: small_20mb
  PieceCID: bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa
  Size: 20 MB
  [Primary]   Provider 1 - pieceId: 1, dataSetId: 2
  [Secondary] Provider 2 - pieceId: 1, dataSetId: 1

--- Downloading and Verifying ---

Downloading bafkzcibfqcapabiuoe2wvpyktx3aunlchjyqizqov5lhagoi4ycwnq344ql6k4gctmwa...
  VERIFIED: 20 MB matches original

=== SUCCESS: All files uploaded, replicated, and verified ===
The service provider(s) will periodically prove they still have your data.
You are being charged based on the storage size and duration.
[ OK ] upload 20MB piece
[INFO] Waiting 10s for caching tasks
[INFO] row_count after 'upload 20MB piece' = 0
[ OK ] cache rows count should not increase
[INFO] Uploading 80MB piece (above 32MB threshold)
[INFO] Loading devnet info from: /home/runner/.foc-devnet/state/latest/devnet-info.json
Devnet run: 20260625T0645_GoobsMimi
Using user: USER_1 (0x47cc9101fd026fc112d7fadf6b3c9df5be7d4a8c)
SPs available: 2
=== Synapse SDK Storage E2E Example ===

Reading file...
  large_80mb (80 MB)

--- Initializing Synapse SDK ---
Network: FOC DevNet
RPC URL: http://localhost:5701/rpc/v1
Wallet address: 0x47CC9101fD026fC112D7FADf6B3c9DF5bE7D4A8c
Synapse instance created

--- Checking Balances ---
FIL balance: 999.9999697906001 FIL
USDFC balance: 999.999970 USDFC

--- Preparing Account ---
Estimated costs:
  Per epoch (30s): 0.000001 USDFC
  Per month: 0.048574 USDFC
  Deposit needed: 0.000000 USDFC
  Ready: true
Account already ready

--- Uploading ---
Uploading large_80mb (80 MB) via stream...

  Selected SP 1 (0x446339aE7245e3cd1FeD701b685C196C69af695e)
  Creating new data set: 2
  Selected SP 2 (0xC60bc6fb60901c2d28ba6e4F8AD4915Bd2106497)
  Creating new data set: 1
  Upload progress: 10 MB (12.5%)
  Upload progress: 20 MB (25.0%)
  Upload progress: 30 MB (37.5%)
  Upload progress: 40 MB (50.0%)
  Upload progress: 50 MB (62.5%)
  Upload progress: 60 MB (75.0%)
  Upload progress: 70 MB (87.5%)
  Upload progress: 80 MB (100.0%)
  Stored on SP 1: bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima
  Pulling to SP 2: bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima (pending)
  Pulling to SP 2: bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima (inProgress)
  Pulling to SP 2: bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima (complete)
  Copied to SP 2: bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima
  Pieces committed on SP 2, tx: 0xeeae8efa73ced3743384cf4964849b3b38dce2d69f3ed170bab522840bed7324
    bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima
  Pieces committed on SP 1, tx: 0x0681b6e79a4aaf27b790c7869c1a7d9719055dd3ea6f5b22e8d8a5c40f74a585
    bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima
  Data set 2 confirmed on SP 1
    bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima -> pieceId 2
  Data set 1 confirmed on SP 2
    bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima -> pieceId 2

--- Upload Summary ---

File: large_80mb
  PieceCID: bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima
  Size: 80 MB
  [Primary]   Provider 1 - pieceId: 2, dataSetId: 2
  [Secondary] Provider 2 - pieceId: 2, dataSetId: 1

--- Downloading and Verifying ---

Downloading bafkzcibfqcamafywumo5i24e5ryghcohzkeyuzp6hosbgkftd7inxohiveq2tm7dkima...
  VERIFIED: 80 MB matches original

=== SUCCESS: All files uploaded, replicated, and verified ===
The service provider(s) will periodically prove they still have your data.
You are being charged based on the storage size and duration.
[ OK ] upload 80MB piece
[INFO] Waiting 10s for caching tasks
[INFO] row_count after 'upload 80MB piece' = 32
[ OK ] cache rows count should increase

[Action Run link](https://github.com/FilOzone/foc-devnet/actions/runs/28151857223)

[rjan90]

Confirmed this looks like a scenario-test false failure rather than a devnet stability failure.

Evidence: the failing test_basic_balances path only saw empty stdout from its cast call, while the same run later confirmed USER_1 had USDFC via the SDK/viem path (USDFC balance: 999.999970 USDFC) and the storage scenarios passed. The Python helper also discarded stderr/nonzero exits, which is why the report collapsed the useful error into not an int: ''.

I opened a draft PR with the focused fix: #135

The PR makes the USDFC cast call run with --from <checked user> and makes the shell helper report command stderr on failures. I would keep this issue open until that lands and a stability rerun passes, then close it with the rerun linked.

[rjan90]

PR #135 has merged with the scenario fix and improved diagnostics, so closing this as addressed. The next stability run will verify the fix in the nightly path; if the balance probe fails again, it should now report the underlying stderr/RPC error instead of not an int: ''.