Commit Graph
58 Commits
Author SHA1 Message Date
James Turner 537776e1f8 TerraSync: fix local-file copying
Avoid downloading data corresponding to files shipped in FGData.
2020-11-04 11:22:48 +00:00
James Turner a59c4e2c8b TerraSync: make tarball extraction asynchronous 2020-11-03 17:49:19 +00:00
James Turner 9305417207 Terrasync: tarball extraction, use larger buffer
Use a 1MB buffer, 2kByte is very 1979 :)
2020-11-01 11:11:31 +00:00
James Turner 11da8b33f9 TerraSync: switch to per-directory hash caching
Avoid a central hash cache becoming enormous, now we use the selective
download scheme for the tile dirs.

Hash name is changed to co-exist with older versions.
2020-11-01 11:11:27 +00:00
James Turner ec3829addb TerraSync: validate local dirs incrementally
Add a process() method to HTTPRepository, and use this to
incrementally validate subdirs after the .dirIndex is received. This 
avoids large pauses of the TerraSync thread, when all of Airports/
is validated at once.
2020-11-01 11:11:15 +00:00
James Turner 96bafef3f3 TerraSync: use an unordered_map for the hash cache
Linear-scan is a bit slow in debug builds, for the large Airports/ tree;
switch to an unordered_map.

Will back-port to the LTS once tested a bit more.
2020-11-01 11:11:12 +00:00
Automatic Release Builder bd9f04d980 TerraSync: retry after socket failures
Assume socket failures are intermittent, up to some maximum count
(currently configured as 16). Add a test case to cover this.
2020-10-13 11:38:35 +01:00
Automatic Release Builder 9c530d6978 TerraSync: Rate-limit hash-cache writes
This helps with IO-limited performance on Windows
2020-10-07 12:23:33 +01:00
Automatic Release Builder 41e43eeba0 TerraSync: fix Windows behaviour
SGDir on Windows returns dot-files by default, which includes our
hash-cache, causing it to be orphaned.
2020-10-06 23:51:06 +01:00
Automatic Release Builder 852058150b TerraSync: fast start Airports/ sync
If TerraSync/Airports is missing, download Airport_archive.tgz instead.
In all other cases update as normal. This has the effect, that fresh
installs make many fewer requests to TerraSync servers.
2020-10-05 11:58:03 +01:00
Automatic Release Builder 0ab81d36b9 TerraSync: tweak warnings around checksum failures
Trying to reduce log spam when there is a server-side failure.
2020-09-29 17:43:18 +01:00
Automatic Release Builder 99c159d46e Harden Repo::computeHashForPath
Check for some error cases in computeHashForPath, since this showed up
in some crash reports.
2020-09-23 22:08:55 +01:00
James Turner 36dca92c2b HTTPRepo: Fix ownership of HTTPDirectory 2020-08-26 10:37:54 +01:00
James Turner 2fe60c9635 Terraysnc: fix incorrect handling of ‘entirely ocean’ tiles
We had some confusion between two error codes, meaning we would log an
error (and treat as a failure) a pure ocean 1x1 area, even though this
was entirely correct and expected behaviour.

Terrasync.cxx expected REPO_ERROR_NOT_FOUND, we need to convert the
file-level error to that, and avoid logging a warning.
2020-08-25 21:10:54 +01:00
Stuart Buchanan 3fe85809b2 Add support for tarballs in Terrasync 2020-02-16 20:26:20 +00:00
James Turner 05da3db2ed Fix a debug message left in the terrasync code 2018-09-24 14:54:20 +01:00
James Turner c192071f03 TerraSync: improve copy-installed-files logic 2018-08-31 16:05:48 +01:00
James Turner f77724a646 Fix terrasync hash persistence 2018-07-19 07:40:23 +01:00
James Turner a0c4913f84 Fix .dirindex preservation on Windows
Ensure we don't treat the special files as regular children
when updating.
2018-07-18 16:30:44 +01:00
James Turner 3dfce43de2 Fixes to HTTP repo tests on Windows.
Large alloca fails, and SGFIle dtor doesn't close the handle,
which breaks _wunlink on Windows :(
2018-07-18 07:20:18 +01:00
Stuart Buchanan 1d7c3984ca Fix minor warning for no return value. 2018-06-05 21:55:23 +01:00
James Turner cf03307b70 More verbose error logging from Repository code 2018-03-30 17:24:58 +01:00
James Turner dd3cdf63e6 Terrasync: Fix exception deleting orphaned directories. 2017-09-29 12:47:42 +01:00
James Turner c170f576b6 Ensure <simgear_config.h> is always included.
This is going to become important soon - prep for enabling SIMGEAR_SHARED on Windows.
2017-03-28 09:36:53 +01:00
Florent Rougon 79f869a7f3 Move IOStreams-related files to simgear/io/iostreams; rename zfstream.[ch]xx to gzfstream.[ch]xx
- Rename zfstream.cxx (resp. zfstream.hxx) to gzfstream.cxx (resp.
  gzfstream.hxx)

  This is because these files only deal with the gzip format (RFC 1952),
  while zlib can actually read and write two slightly different formats:
  this one and the "ZLIB Compressed Data Format" (RFC 1950). Since I am
  going to add std::streambuf and std::istream subclasses able to deal
  with both formats (and supporting data sources that are general
  std::istream instances, not just files), this renaming will make
  things a bit clearer, I hope.

- Add new folder simgear/io/iostreams and move the following files to
  this folder:

    simgear/misc/gzcontainerfile.cxx
    simgear/misc/gzcontainerfile.hxx
    simgear/misc/gzfstream.cxx
    simgear/misc/gzfstream.hxx
    simgear/misc/sgstream.cxx
    simgear/misc/sgstream.hxx
    simgear/misc/sgstream_test.cxx

- Adapt other files accordingly (mainly #includes and CMakeLists.txt
  files).
2017-02-12 21:18:52 +01:00
Torsten Dreyer 1b8dfb2bef Accept time token for .dirindex files
print the timestamp on log-level "info"
2017-02-12 17:39:12 +01:00
James Turner a2b111bb09 Bugfix: reject dubious paths in HTTP repos.
This avoids a malicious repository writing to files outside the local
storage root.
2016-12-15 17:37:28 +00:00
Erik Hofman e768553a4a Replace auto_ptr with unique_ptr 2016-12-07 11:03:49 +01:00
James Turner 2aaad212e8 Revert partial-update mode for TerraSync repos.
Going to implement this a different, simpler way now,
2016-11-25 22:19:51 +00:00
James Turner ab70090a0a Finally fix bogus terrasync download numbers.
(Staying a hotel with sufficiently slow wi-fi to debug this!)
2016-11-13 22:18:32 +01:00
James Turner ab1d4e651e Bugfix: avoid bogus download size on start
Missed a default init of the HTTP repo download content size, so until
the request response header was received, this could report a very
large value.
2016-11-07 15:49:07 +01:00
James Turner 87590cafb2 Fix terrasync behaviour when net is down. 2016-09-07 22:49:03 +01:00
James Turner d7d59b08a2 Copy-from-install support in TerraSync
Allows repositories to be initialised based on data in the install.
This avoids duplicate downloading of the Model and Airport data,
and the starting scenery.

Requires a corresponding FlightGear change to be functional.
2016-08-18 16:21:31 +01:00
James Turner b862cf7e54 Fix repository test. 2016-07-01 09:02:52 +01:00
James Turner f21eac8473 Building with clean SGPath API 2016-06-30 16:17:52 +01:00
James Turner a6bed69d19 Fix user-after-free in HTTP repo code 2016-06-02 23:53:15 +01:00
James Turner 1b0289b11f HTTP: Always use absolute paths for hashes
Ensure we only compute / retrieve hashes for absolute paths. Fixes
duplicate paths in repository hash cache.
2016-05-26 23:36:03 +01:00
James Turner 00f4248137 Partial update support for HTTP repos.
Not used yet, we set full-update mode on repositories for the
moment. Pending Terrasync change will enable partial mode.
2016-05-26 23:35:11 +01:00
James Turner 7d59dd977f AbstractRepository interface removed. 2016-05-26 17:38:02 +01:00
James Turner 6e4da15a58 Use SGBinaryFile in a couple more places.
Most importantly, when computing a hash from the on-disk
state.
2016-05-07 10:03:48 +01:00
Torsten Dreyer 561e451192 Introduce SGBinaryFile
For Windows, default file mode is TEXT.
If binary files should be created _O_BINARY shall be or'ed to the open
flags. This is not necessary on *nixes.

Introduce a SGBinaryFile as extension to SGFile which adds the flag
upon construction on Windows.

This should keep existing behaviour for all other usages of SGFile.
2016-05-06 22:01:42 +02:00
Torsten Dreyer 74b4df9452 Don't use object returned from vector::end()
Replace all use of c->name by it->file() in
updateChildrenBasedOnHash()'s fsChildren loop
to avoid confusion.

Thanks to Geoff for spotting this!
2016-05-06 16:21:43 +02:00
Torsten Dreyer 5fb8891e86 Fix one more crash on Windows in HTTPRepository
inner scope it seems to overwrite out scope it on Win :-/

Anyway, the erase-remove-idiom is a better solution for that task
https://en.wikipedia.org/wiki/Erase%E2%80%93remove_idiom

Also remove some useless debug messages
2016-05-06 10:48:16 +02:00
Torsten Dreyer f3d68a170e Don't continue parsing after processing version line 2016-05-05 21:59:35 +02:00
Torsten Dreyer 639e16b722 use safer stl functions instead of pointer operations 2016-05-04 21:10:12 +02:00
Torsten Dreyer 9456ff838c Add a bunch of debug messages to help fixing the Windows crash
This patch should be reverted once the bug is fixed
2016-05-04 15:44:55 +02:00
Torsten Dreyer 87eaec3ce7 Fix some warnings reported from msvc 2016-05-02 20:54:28 +02:00
James Turner c17324b5ea Fixes for HTTP cancellation.
(Forgot to stage these from previous commit)
2016-04-14 09:42:04 +01:00
James Turner aa679ffb86 HTTP terra sync: fix hash cache handling
Avoids very long pauses blocking the terrasync thread, while existing
file trees are verified.

Also split the request queue so we don’t submit vary large numbers of
requests from a single repository, and hence block other repositories
from getting traffic.
2016-03-24 19:15:20 +00:00
James Turner c199c95218 Expose total bytes to download / remaining
For HTTP repositories, support some additional metrics about ongoing
transfers. Not currently exposed via properties / TerraSync API, but
will be shortly.
2016-03-22 21:14:17 +00:00