5 Commits (839fee9ef4f6c806fd49b2a21cf741bf961f642c)

Author	SHA1	Message	Date
Joe Tsai	d603d18956	taildrop: fix TestResume (#9874) ... Previously, the test simply relied on: defer close() to cleanup file handles. This works fine on Unix-based systems, but not on Windows, which dislikes deleting files where an open file handle continues to exist. Fix the test by explicitly closing the file handle after we are done with the resource. Updates tailscale/corp#14772 Signed-off-by: Joe Tsai <joetsai@digital-static.net>	1 year ago
Joe Tsai	cb00eac850	taildrop: disable TestResume (#9873) ... This test is currently failing on Windows. Updates tailscale/corp#14772 Signed-off-by: Joe Tsai <joetsai@digital-static.net>	1 year ago
Joe Tsai	3f27087e9d	taildrop: switch hashing to be streaming based (#9861) ... While the previous logic was correct, it did not perform well. Resuming is a dance between the client and server, where 1. the client requests hashes for a partial file, 2. the server then computes those hashes, 3. the client computes hashes locally and compares them. 4. goto 1 while the partial file still has data While step 2 is running, the client is sitting idle. While step 3 is running, the server is sitting idle. By streaming over the block hash immediately after the server computes it, the client can start checking the hash, while the server works on the next hash (in a pipelined manner). This performs dramatically better and also uses less memory as we don't need to hold a list of hashes, but only need to handle one hash at a time. There are two detriments to this approach: * The HTTP API relies on a JSON stream, which is not a standard REST-like pattern. However, since we implement both client and server, this is fine. * While the stream is on-going, we hold an open file handle on the server side while the file is being hashed. On really slow streams, this could hold a file open forever. Updates tailscale/corp#14772 Signed-off-by: Joe Tsai <joetsai@digital-static.net> Co-authored-by: Rhea Ghosh <rhea@tailscale.com>	1 year ago
Joe Tsai	c2a551469c	taildrop: implement asynchronous file deletion (#9844) ... File resumption requires keeping partial files around for some time, but we must still eventually delete them if never resumed. Thus, we implement asynchronous file deletion, which could spawn a background goroutine to delete the files. We also use the same mechanism for deleting files on Windows, where a file can't be deleted if there is still an open file handle. We can enqueue those with the asynchronous file deleter as well. Updates tailscale/corp#14772 Signed-off-by: Joe Tsai <joetsai@digital-static.net>	1 year ago
Joe Tsai	b1867eb23f	taildrop: add logic for resuming partial files (#9785) ... We add the following API: * type FileChecksums * type Checksum * func Manager.PartialFiles * func Manager.HashPartialFile * func ResumeReader The Manager methods provide the ability to query for partial files and retrieve a list of checksums for a given partial file. The ResumeReader function is a helper that wraps an io.Reader to discard content that is identical locally and remotely. The FileChecksums type represents the checksums of a file and is safe to JSON marshal and send over the wire. Updates tailscale/corp#14772 Signed-off-by: Joe Tsai <joetsai@digital-static.net> Co-authored-by: Rhea Ghosh <rhea@tailscale.com>	1 year ago