Even Faster I/O (#1369)

* try multithreading for faster IO

* smaller batch size

* Account for pread returning less than size

* nit

---------

Co-authored-by: Angelos Katharopoulos <a_katharopoulos@apple.com>
This commit is contained in:
Awni Hannun
2024-08-28 11:49:07 -07:00
committed by GitHub
parent 4e22a1dffe
commit fcb65a3897
6 changed files with 157 additions and 27 deletions

View File

@@ -146,6 +146,11 @@ class PyFileReader : public io::Reader {
}
}
void read(char* data, size_t n, size_t offset) override {
seek(offset);
read(data, n);
}
std::string label() const override {
return "python file object";
}