refactor: remove ffmpeg decoding and use OS apis

[mdydek]

Closes #

⚠️ Breaking changes ⚠️

Introduced changes

• removed ffmpeg decoding several formats, it uses sytem apis instead for that
• one of the parts connected to api refactor to use ffmpeg only for essentials

Checklist

• Linked relevant issue
• Updated relevant documentation
• Added/Conducted relevant tests
• Performed self-review of the code
• Updated Web Audio API coverage
• Added support for web
• Updated old arch android spec file

[maciejmakowski2003]

General idea is great, although there are a lot of places that are not aligned with SOLID principles.

Let's start with my desired architecture explanation. I assume three layers:

• clients (AudioDecoding, AudioFileConcatenator, SeekDecoderDaemon)
• API interface (IncrementalDecoder, DecoderFactory)
• backend (IOSDecoder, AndroidDecoder, MiniAudioDecoder, FfmpegDecoder, RawPcmDecoder - all implementing IncrementalDecoder)

flowchart TB
    subgraph Clients
        AD["AudioDecoding.cpp\ndecodeAll, probeDuration, decodeWithFilePath..."]
        SDD["SeekDecoderDaemon.cpp\nstreaming loop, seek handling"]
        AFC["AudioFileConcatenator.cpp\nread chunks → WAV encoder / FFmpeg remux"]
    end

    subgraph API interface
        DS["DecoderSource"]
        DF["DecoderFactory::createDecoder()"]
        IAD["IncrementalAudioDecoder"]
    end

    subgraph Backends
        OS["OsCoreDecoder"]
        MA["MiniAudioDecoder"]
        FF["FfmpegDecoder"]
        PCM["RawPcmDecoder"]
    end

    AD --> DF
    SDD --> DF
    AFC --> DF

    DF --> IAD
    IAD --> OS
    IAD --> MA
    IAD --> FF
    IAD --> PCM

Loading

Suggested interfaces

// decoding/DecoderSource.h

namespace audioapi::decoding {

struct LocalFileSource {
  std::string path;
  int outputSampleRate = 0; // 0 = native rate
};

struct EncodedMemorySource {
  const void *data = nullptr;
  size_t size = 0;
  int outputSampleRate = 0;
};

struct RemoteUrlSource {
  std::string url;
  int outputSampleRate = 0;
  std::map<std::string, std::string> httpHeaders; // only for remote
};

struct RawPcmSource {
  const void *data = nullptr;
  size_t size = 0;
  float sampleRate = 0.0f;
  int channelCount = 0;
  bool interleaved = true;
  // optional PcmSampleFormat enum
};

struct RawPcmBase64Source {
  std::string base64;
  float sampleRate = 0.0f;
  int channelCount = 0;
  bool interleaved = true;
  // optional PcmSampleFormat enum
};

using DecoderSource = std::variant<
    LocalFileSource,
    EncodedMemorySource,
    RemoteUrlSource,
    RawPcmSource,
    RawPcmBase64Source>;
}

// decoding/IncrementalAudioDecoder.h
namespace audioapi::decoding {
  using DecoderError = std::string;
  using DecoderResult = Result<NoneType, DecoderError>;
  class IncrementalAudioDecoder {
   public:
    static constexpr size_t CHUNK_SIZE = 4096;
    IncrementalAudioDecoder() = default;
    virtual ~IncrementalAudioDecoder() = default;
    DELETE_COPY_AND_MOVE(IncrementalAudioDecoder);
    [[nodiscard]] virtual size_t readPcmFrames(float *outInterleaved, size_t frameCount) = 0;
    [[nodiscard]] virtual DecoderResult seekToTime(double seconds) = 0;
    virtual void close() = 0;
    [[nodiscard]] virtual bool isOpen() const = 0;
    [[nodiscard]] virtual int outputChannels() const = 0;
    [[nodiscard]] virtual int outputSampleRate() const = 0;
    [[nodiscard]] virtual float getDurationInSeconds() const = 0;
    [[nodiscard]] virtual float getCurrentPositionInSeconds() const = 0;
    [[nodiscard]] virtual bool isHlsStreaming() const { return false; }
    // NO open(const DecoderSource&)
  };
} // namespace audioapi::decoding

// decoding/backends/...

class IOSDecoder : public OsDecoderBase {
 public:
  [[nodiscard]] DecoderResult open(const LocalFileSource &source);
  [[nodiscard]] DecoderResult open(const EncodedMemorySource &source);
  ...
};

class RawPcmDecoder : public IncrementalAudioDecoder {
 public:
  [[nodiscard]] DecoderResult open(const RawPcmSource &source);
  [[nodiscard]] DecoderResult open(const RawPcmBase64Source &source);
  ....
};

// decoding/DecoderFactory.h

namespace audioapi::decoding {
  /// Returns the decoder implementation for @p source.
  /// It should pick the right decoder backend based on format and open it.
  [[nodiscard]] std::unique_ptr<IncrementalAudioDecoder> createDecoder(const DecoderSource &source);
} // namespace audioapi::decoding

// decoding/DecoderFactory.cpp
template <typename Decoder, typename Source>
Result<std::unique_ptr<IncrementalAudioDecoder>, DecoderError>
createAndOpen(const Source &source) {
  auto decoder = std::make_unique<Decoder>();
  if (auto opened = decoder->open(source); opened.is_err()) {
    decoder->close();
    return Err(opened.unwrap_err());
  }
  return Ok(std::unique_ptr<IncrementalAudioDecoder>(std::move(decoder)));
}

Result<std::unique_ptr<IncrementalAudioDecoder>, DecoderError>
createDecoder(const DecoderSource &source) {
  return std::visit(overloaded{
    [](const LocalFileSource &s) {
      if (isMiniAudioExtensionPath(s.path)) {
        return createAndOpen<MiniAudioExtensionDecoder>(s);
      }
#if RN_AUDIO_API_HAS_OS_DECODER
      return createAndOpen<os_decoder::Decoder>(s);
#else
      return createAndOpen<MiniAudioExtensionDecoder>(s);
#endif
    },
    [](const EncodedMemorySource &s) { ... },
    [](const RemoteUrlSource &s) {
      return createAndOpen<FfmpegExtensionDecoder>(s);
    },
    [](const RawPcmSource &s) {
      return createAndOpen<RawPcmDecoder>(s);
    },
    // ...
  }, source);
}

[maciejmakowski2003]

I do not think we need OSDecoderBase anymore

[mdydek]

i did not completely removed it, but onboard some of the code from osdecoderbase to decodingbackend and it reduced amount of lines, i think its good

[maciejmakowski2003]

I would say that it could be stronger guard e.g. HlsPlaylistSource

[mdydek]

missed that, ffmpeg should not operate on localfilesource at all

[maciejmakowski2003]

maybe lest try to extract it to shared utils

[maciejmakowski2003]

have you considered dropping mp3, wav and flac support?

[mdydek]

it dropped the .so library size on android by 0.6 MB, which is roughly 10%, when I turned off unnecessary options