Environment
• Mergin DB Sync 2.3.0
• GeoDiff 2.3.0
• Windows 11
• Python 3.12
Description
While investigating another issue, I found that Mergin DB Sync crashes with a UnicodeDecodeError when GeoDiff returns an error message containing non-UTF-8 characters on Windows.
Instead of reporting the original GeoDiff error, DB Sync terminates while decoding the stderr output produced by GeoDiff.
The relevant code is:
geodiff_stderr = res.stderr.decode()
This assumes that the error output is UTF-8 encoded. On Windows, this assumption may not hold depending on how GeoDiff emits its error messages.
Actual Result
DB Sync terminates with:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97 in position ...
As a consequence, the original GeoDiff error is hidden, making it difficult to diagnose the underlying problem.
Investigation
To investigate the issue, I modified the run_geodiff() function to preserve the original stdout and stderr output.
Instead of:
geodiff_stderr = res.stderr.decode()
I temporarily changed the code to:
geodiff_stdout = res.stdout.decode(errors="replace")
geodiff_stderr = res.stderr.decode(errors="replace")
This allowed DB Sync to continue running and display the original GeoDiff error.
In my case, the hidden error was:
Error: Missing 'modified' file when opening sqlite driver:
C:/Users/.../camada_mergin_maps—_estacoes_amostragem_teste__estaes_de_amostragem.gpkg
Without this modification, the actual GeoDiff error could not be identified because DB Sync terminated first with the UnicodeDecodeError.
Expected Result
DB Sync should always display the original GeoDiff error, regardless of the encoding used by the underlying process.
It should not terminate while decoding the stderr output.
Possible Improvement
Instead of:
geodiff_stderr = res.stderr.decode()
consider one of the following approaches:
geodiff_stderr = res.stderr.decode(errors="replace")
or
import locale
geodiff_stderr = res.stderr.decode(
locale.getpreferredencoding(False),
errors="replace",
)
Either approach would prevent DB Sync from crashing while still preserving the original GeoDiff error message.
Related Issue
While investigating this behaviour, I identified the underlying GeoDiff issue:
MerginMaps/geodiff#258
DB Sync currently hides that error because of the UTF-8 decoding failure.
I'd be happy to test a fix on Windows if needed.
Environment
• Mergin DB Sync 2.3.0
• GeoDiff 2.3.0
• Windows 11
• Python 3.12
Description
While investigating another issue, I found that Mergin DB Sync crashes with a UnicodeDecodeError when GeoDiff returns an error message containing non-UTF-8 characters on Windows.
Instead of reporting the original GeoDiff error, DB Sync terminates while decoding the stderr output produced by GeoDiff.
The relevant code is:
geodiff_stderr = res.stderr.decode()
This assumes that the error output is UTF-8 encoded. On Windows, this assumption may not hold depending on how GeoDiff emits its error messages.
Actual Result
DB Sync terminates with:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x97 in position ...
As a consequence, the original GeoDiff error is hidden, making it difficult to diagnose the underlying problem.
Investigation
To investigate the issue, I modified the run_geodiff() function to preserve the original stdout and stderr output.
Instead of:
geodiff_stderr = res.stderr.decode()
I temporarily changed the code to:
geodiff_stdout = res.stdout.decode(errors="replace")
geodiff_stderr = res.stderr.decode(errors="replace")
This allowed DB Sync to continue running and display the original GeoDiff error.
In my case, the hidden error was:
Error: Missing 'modified' file when opening sqlite driver:
C:/Users/.../camada_mergin_maps—_estacoes_amostragem_teste__estaes_de_amostragem.gpkg
Without this modification, the actual GeoDiff error could not be identified because DB Sync terminated first with the UnicodeDecodeError.
Expected Result
DB Sync should always display the original GeoDiff error, regardless of the encoding used by the underlying process.
It should not terminate while decoding the stderr output.
Possible Improvement
Instead of:
geodiff_stderr = res.stderr.decode()
consider one of the following approaches:
geodiff_stderr = res.stderr.decode(errors="replace")
or
import locale
geodiff_stderr = res.stderr.decode(
locale.getpreferredencoding(False),
errors="replace",
)
Either approach would prevent DB Sync from crashing while still preserving the original GeoDiff error message.
Related Issue
While investigating this behaviour, I identified the underlying GeoDiff issue:
MerginMaps/geodiff#258
DB Sync currently hides that error because of the UTF-8 decoding failure.
I'd be happy to test a fix on Windows if needed.