Commit Graph

18 Commits (2aa9125ac438ffa902158b5bedf9791c93117b9b)

Author SHA1 Message Date
Joe Tsai 52ef27ab7c
taildrop: fix defer in loop (#13757)
However, this affects the scope of a defer.

Updates #11038

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
3 weeks ago
Brad Fitzpatrick c763b7a7db syncs: delete Map.Range, update callers to iterators
Updates #11038

Change-Id: I2819fed896cc4035aba5e4e141b52c12637373b1
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
3 weeks ago
Joe Tsai 2e404b769d
all: use new AppendEncode methods available in Go 1.22 (#11079)
Updates #cleanup

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
9 months ago
Rhea Ghosh 4ce33c9758
taildrop: remove breaking abstraction layers for apple (#10728)
Removes the avoidFinalRename logic and all associated code as it is no longer required by the Apple clients.
Enables resume logic to be usable for Apple clients.

Fixes tailscale/corp#14772

Signed-off-by: Rhea Ghosh <rhea@tailscale.com>
10 months ago
Rhea Ghosh dbe70962b1
taildrop: Allow category Z unicode characters (#10750)
This will expand the unicode character categories that we allow for valid filenames to go from "L, M, N, P, S, and the ASCII space character" to "L, M, N, P, S, Zs"

Fixes #10105

Signed-off-by: Rhea Ghosh <rhea@tailscale.com>
10 months ago
Joe Tsai 975c5f7684
taildrop: lazily perform full deletion scan after first taildrop use (#10137)
Simply reading the taildrop directory can pop up security dialogs
on platforms like macOS. Avoid this by only performing garbage collection
of partial and deleted files after the first received taildrop file,
which would have prompted the security dialog window.

Updates tailscale/corp#14772

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
12 months ago
Joe Tsai 6ada33db77
taildrop: fix theoretical race condition in fileDeleter.Init (#9876)
It is possible that upon a cold-start, we enqueue a partial file
for deletion that is resumed shortly after startup.

If the file transfer happens to last longer than deleteDelay,
we will delete the partial file, which is unfortunate.
The client spent a long time uploading a file,
only for it to be accidentally deleted.
It's a very rare race, but also a frustrating one
if it happens to manifest.

Fix the code to only delete partial files that
do not have an active puts against it.

We also fix a minor bug in ResumeReader
where we read b[:blockSize] instead of b[:cs.Size].
The former is the fixed size of 64KiB,
while the latter is usually 64KiB,
but may be less for the last block.

Updates tailscale/corp#14772

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
1 year ago
Joe Tsai d603d18956
taildrop: fix TestResume (#9874)
Previously, the test simply relied on:
	defer close()
to cleanup file handles.

This works fine on Unix-based systems,
but not on Windows, which dislikes deleting files
where an open file handle continues to exist.

Fix the test by explicitly closing the file handle
after we are done with the resource.

Updates tailscale/corp#14772

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
1 year ago
Joe Tsai cb00eac850
taildrop: disable TestResume (#9873)
This test is currently failing on Windows.

Updates tailscale/corp#14772

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
1 year ago
Joe Tsai a8fbe284b2
taildrop: fix theoretical race condition (#9866)
WaitGroup.Wait should not be concurrently called WaitGroup.Add.
In other words, we should not start new goroutines after shutodwn is called.
Thus, add a conditional to check that shutdown has not been called
before starting off a new waitAndDelete goroutine.

Updates tailscale/corp#14772

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
1 year ago
Joe Tsai 3f27087e9d
taildrop: switch hashing to be streaming based (#9861)
While the previous logic was correct, it did not perform well.
Resuming is a dance between the client and server, where
1. the client requests hashes for a partial file,
2. the server then computes those hashes,
3. the client computes hashes locally and compares them.
4. goto 1 while the partial file still has data

While step 2 is running, the client is sitting idle.
While step 3 is running, the server is sitting idle.

By streaming over the block hash immediately after the server
computes it, the client can start checking the hash,
while the server works on the next hash (in a pipelined manner).
This performs dramatically better and also uses less memory
as we don't need to hold a list of hashes, but only need to
handle one hash at a time.

There are two detriments to this approach:
* The HTTP API relies on a JSON stream,
  which is not a standard REST-like pattern.
  However, since we implement both client and server,
  this is fine.
* While the stream is on-going, we hold an open file handle
  on the server side while the file is being hashed.
  On really slow streams, this could hold a file open forever.

Updates tailscale/corp#14772

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Co-authored-by: Rhea Ghosh <rhea@tailscale.com>
1 year ago
Joe Tsai c2a551469c
taildrop: implement asynchronous file deletion (#9844)
File resumption requires keeping partial files around for some time,
but we must still eventually delete them if never resumed.
Thus, we implement asynchronous file deletion, which could
spawn a background goroutine to delete the files.

We also use the same mechanism for deleting files on Windows,
where a file can't be deleted if there is still an open file handle.
We can enqueue those with the asynchronous file deleter as well.

Updates tailscale/corp#14772

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
1 year ago
Joe Tsai 8f948638c5
taildrop: minor cleanups and fixes (#9786)
Perform the same m==nil check in Manager.{PartialFiles,HashPartialFile}
as we do in the other methods.

Fix HashPartialFile is properly handle a length of -1.

Updates tailscale/corp#14772

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
1 year ago
Joe Tsai b1867eb23f
taildrop: add logic for resuming partial files (#9785)
We add the following API:
* type FileChecksums
* type Checksum
* func Manager.PartialFiles
* func Manager.HashPartialFile
* func ResumeReader

The Manager methods provide the ability to query for partial files
and retrieve a list of checksums for a given partial file.
The ResumeReader function is a helper that wraps an io.Reader
to discard content that is identical locally and remotely.
The FileChecksums type represents the checksums of a file
and is safe to JSON marshal and send over the wire.

Updates tailscale/corp#14772

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Co-authored-by: Rhea Ghosh <rhea@tailscale.com>
1 year ago
Joe Tsai 37c646d9d3
taildrop: improve the functionality and reliability of put (#9762)
Changes made:
* Move all HTTP related functionality from taildrop to ipnlocal.
* Add two arguments to taildrop.Manager.PutFile to specify
  an opaque client ID and a resume offset (both unused for now).
* Cleanup the logic of taildrop.Manager.PutFile
  to be easier to follow.
* Implement file conflict handling where duplicate files are renamed
  (e.g., "IMG_1234.jpg" -> "IMG_1234 (2).jpg").
* Implement file de-duplication where "renaming" a partial file
  simply deletes it if it already exists with the same contents.
* Detect conflicting active puts where a second concurrent put
  results in an error.

Updates tailscale/corp#14772

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Co-authored-by: Rhea Ghosh <rhea@tailscale.com>
1 year ago
Joe Tsai e4cb83b18b
taildrop: document and cleanup the package (#9699)
Changes made:
* Unexport declarations specific to internal taildrop functionality.
* Document all exported functionality.
* Move TestRedactErr to the taildrop package.
* Rename and invert Handler.DirectFileDoFinalRename as AvoidFinalRename.

Updates tailscale/corp#14772

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
1 year ago
Rhea Ghosh 557ddced6c
{ipn/ipnlocal, taildrop}: move put logic to taildrop (#9680)
Cleaning up taildrop logic for sending files.

Updates tailscale/corp#14772

Signed-off-by: Rhea Ghosh <rhea@tailscale.com>
Co-authored-by: Joe Tsai <joetsai@digital-static.net>
1 year ago
Rhea Ghosh dc1c7cbe3e
taildrop: initial commit of taildrop functionality refactoring (#9676)
Over time all taildrop functionality will be contained in the
taildrop package. This will include end to end unit tests. This is
simply the first smallest piece to move over.

There is no functionality change in this commit.

Updates tailscale/corp#14772

Signed-off-by: Rhea Ghosh <rhea@tailscale.com>
Co-authored-by: Joseph Tsai <joetsai@tailscale.com>
1 year ago