Probabilistic models allow for uncertainty aware predictions. This is relevant for safety critical applications but also enables the use of active learning or Bayesian optimization. Within our research, we develop Bayesian optimization based methods for biological sequence design. This includes DNA, RNA but also protein sequences. The relation between sequence and functionality (i.e. expression levels) can be represented naturally by sequence-to-function neural networks, with ensemble neural networks being a natural method for generating probabilistic predictions. Large ensembles allow for straight forward implementation and well calibrated uncertainty representation but can lead to substantial computational overhead. To this end, we envisioned the creation of approximate ensembles by implementing the ensembling only at the later stages of the network. A special focus then applies to the quality of uncertainty calibration. The goal of this project is to implement and advance the development of these approximate ensembles with a special focus on sequence-to-function neural networks. The project is also applicable as a group project.