Previously today, LinkedIn revealed it was open-sourcing AvroTensorDataset, which is a “TensorFlow dataset for reading, parsing, and processing Avro information.” Apache Avro is the main storage format that LinkedIn utilizes for its training information.
According to LinkedIn, it was experiencing traffic jams in its maker finding out work that were brought on by the requirement to check out several terabytes of input information. AvroTensorDataset can accelerate preprocessing of information by several orders of magnitude, according to the business.
The tool was constructed internally at LinkedIn, and it wished to open-source the job so that others might experience the big efficiency enhances to training work. It has actually remained in production for over a year currently at LinkedIn.
LinkedIn states that with this tool it has actually had the ability to enhance processing speed by 162x compared to existing options and has actually reduced total training time by 66%
” ATDSDataset is LinkedIn’s option to effectively check out Avro information into TensorFlow. Through several efficiency improvements, we had the ability to accelerate I/O throughput by orders of magnitude over existing Avro reader options. Our group at LinkedIn worked carefully with the TensorFlow I/O neighborhood to open-source this function, and we hope that by open-sourcing it, the TensorFlow neighborhood can likewise gain from these efficiency improvements,” Jonathan Hung, personnel software application engineer at LinkedIn, composed in a article
