ONNX

ONNX (Open Neural Network Exchange) is an open file-format specification that defines a common set of operators and a graph structure for representing machine learning models. ONNX Runtime is a separate, cross-platform inference engine—developed by Microsoft—that loads and executes those .onnx model files with hardware acceleration on CPU, GPU, and edge devices. You can use the ONNX format without ever touching ONNX Runtime, but the two are frequently paired in production pipelines.

Yes, PyTorch ships with a built-in torch.onnx.export() function that traces or scripts your trained model into the ONNX graph format. Once exported, the resulting .onnx file can be served by ONNX Runtime, a custom C++ engine, or any ONNX-compatible runtime with zero dependency on PyTorch at inference time. This decoupling of training and deployment is one of the primary reasons ONNX was created.

PyTorch, TensorFlow, Keras, scikit-learn, and MATLAB all support exporting to ONNX through either native API support or community-maintained converters such as tf2onnx and sklearn-onnx. The ONNX GitHub ecosystem repository maintains a regularly updated list of converters and their coverage of operators. This breadth of exporter support is what makes ONNX a practical interchange standard across the major training frameworks.

ONNX defines dedicated quantized operators—such as QLinearConv, QLinearMatMul, and QGemm—that allow models to run with 8-bit integer arithmetic, cutting model size and improving latency on edge hardware. ONNX Runtime additionally ships dynamic and static quantization tools that can convert a standard float32 ONNX model into a quantized version. This makes ONNX a viable deployment format for mobile, embedded, and IoT inference scenarios.