This project is mirrored from https://github.com/cockroachdb/cockroach. Pull mirroring updated .
  1. 14 Jul, 2020 12 commits
    • craig[bot]'s avatar
      Merge #51329 · bc2f84c7
      craig[bot] authored
      51329: roach{test,prod},acceptance: no longer rely on auto-init r=irfansharif a=irfansharif
      
      `cockroach start` without `--join` auto-initializes the cluster. This
      was deprecated in 19.2 and will be removed in a future commit. We update
      test code that relies on the previous behavior. To (mostly) retain
      existing roachprod behavior, we auto-initialize when `roachprod start`-ing 
      n1. We introduce a new `--skip-init` flag for when that is
      not desired (for example in roachtests that restart n1, but don't intend
      to re-initialize the cluster). Most of the roachtest changes are exactly
      to this effect, where tests than invoke `roachprod start` a second time
      on n1 are changed to now specify this new `--skip-init` flag.
      
      Release note: None
      Co-authored-by: default avatarirfan sharif <[email protected]>
      bc2f84c7
    • irfan sharif's avatar
      roachprod: explicitly bootstrap cluster instead of relying on auto-init · ad380a4b
      irfan sharif authored
      `cockroach start` without `--join` auto-initializes the cluster. This
      is deprecated behavior as of 19.2 and will be removed soon. `roachprod
      start` inherited similar semantics, auto-initializing a cluster on
      `roachprod start`. It did so by cutting around the join flags and using
      auto-init behaviour of crdb. We keep the same external API of
      `roachprod start`, but we now explicitly `cockroach init` the managed
      clusters (using a dedicated `cluster-bootstrapped` signal file to avoid
      double initialization).
      
      Release note: None
      ad380a4b
    • craig[bot]'s avatar
      Merge #51072 · ab32cf17
      craig[bot] authored
      51072: backupccl: fix backup protected timestamp test r=ajwerner,dt a=pbardea
      
      The protected timestamp test for backup only tested that the timestamp
      was released after the backup, not if the timestamp was actually
      protected in the first place. This commit adds an AOST select at what
      should otherwise be a timestamp that has been GC'd.
      
      The test will fail on this SELECT command if the timestamp is not
      protected.
      
      Fixes #51068.
      
      Release note: None
      Co-authored-by: default avatarPaul Bardea <[email protected]>
      ab32cf17
    • craig[bot]'s avatar
      Merge #45611 #51407 · 36e3d7f0
      craig[bot] authored
      45611: Makefile: speed up `buildshort` and introduce `cleanshort` target r=irfansharif a=irfansharif
      
      Makefile: speed up `buildshort` and introduce `cleanshort` target
      
      I took some time to investigate where all time was going every \`make
      buildshort\` invocation, and here are a few changes to tighten it up a
      bit. Here's what my workflow tends to look like:
      
      ```sh
      $ make clean       # Used gratuitously, like `clear`. Probably shouldn't.
      $ make buildshort  # To check if everything compiles.
      $ make buildshort  # As I'm iterating on things, to keep checking things.
      ```
      
      Almost all of my changes are exclusively go code, and as far as
      generated code goes, it's only ever protobuf generated code. Others may
      also use `make generate` to regenerate code (including execgen, optgen,
      etc.), but I assume the workflow is roughly the same for all. I also
      assume protobufs are changed far more often than `execgen` or `optgen`
      code.
      
      To speed things up for most folks, this PR aims to make the common
      sequences of "iteratively compile crdb" and "clean up some stuff and
      compile crdb" faster than it was before. It does so by doing the
      following:
      
      1. Not tagging `build.utcTime` on every `buildshort` invocation, which
         lets us make much better use of the go build cache.
      
      ```sh
      $ make clean; time make buildshort       # Fresh builds before
        156.40 real       553.84 user       380.26 sys
      $ make clean; time make buildshort       # Fresh builds after
        121.04 real       384.52 user       278.03 sys
      
      $ make buildshort; time make buildshort  # No-op builds before
        27.44 real        21.91 user        32.50 sys
      $ make buildshort; time make buildshort  # No-op builds after
        4.79 real         3.21 user        30.79 sys
      ```
      
      2. Not rebuilding the settings HTML page or the docgen page for
         `buildshort`. Given we're not building the UI, I think this is fine
         (and shaves a few seconds)
      
      3. Introducing a `cleanshort` target, which leaves the go build cache
      and C++ artifacts intact. It should suffice for most folks editing
      `.proto` files and regular go code. `{exec,opt}gen` and storage folks
      may still need to fall back to `make {clean,generate}`, but nobody else.
      
      ```sh
      $ make clean; time make buildshort        # Fresh build; nuking most things.
        121.04 real       384.52 user       278.03 sys
      
      $ make cleanshort; time make buildshort   # Fresh-ish build; nuking only go protos.
        31.10 real        29.04 user        36.76 sys
      ```
      
      ---
      
      We also add a loud warning if `ccache` is not present on the
      maintainer's system, as it really should be.
      
      ```sh
      $ make clean; time make build   # Without ccache
      409.45 real      3251.96 user       598.34 sys
      
      $ make clean; time make build   # With ccache
      156.40 real       553.84 user       380.26 sys
      ```
      
      Release note: None
      
      
      51407: deps: bump datadriven r=RaduBerinde a=knz
      
      This ensures that datadriven tests do not report all directives on
      their output unless `-v` is passed to `go test`, as was originally
      intended.
      
      Noticed/requested by @RaduBerinde.
      
      Release note: None
      Co-authored-by: default avatarirfan sharif <[email protected]>
      Co-authored-by: default avatarRaphael 'kena' Poss <[email protected]>
      36e3d7f0
    • craig[bot]'s avatar
      Merge #50211 · 2e0635be
      craig[bot] authored
      50211: kvserver: detail the range ID and replica details upon lease error r=irfansharif,tbg a=knz
      
      This logging if it had been present, would have simplified the investigation on https://github.com/cockroachlabs/support/issues/482
      
      
      Release note: None
      Co-authored-by: default avatarRaphael 'kena' Poss <[email protected]>
      2e0635be
    • Raphael 'kena' Poss's avatar
      1ba28219
    • craig[bot]'s avatar
      Merge #51408 · 10f0c57d
      craig[bot] authored
      51408: kvserver: skip TestGCQueueTransactionTable r=knz a=tbg
      
      It's quite flaky.
      
      https://github.com/cockroachdb/cockroach/issues/51402
      
      Release note: None
      Co-authored-by: default avatarTobias Schottdorf <[email protected]>
      10f0c57d
    • Tobias Schottdorf's avatar
      kvserver: skip TestGCQueueTransactionTable · f88e1332
      Tobias Schottdorf authored
      It's quite flaky.
      
      https://github.com/cockroachdb/cockroach/issues/51402
      
      Release note: None
      f88e1332
    • Raphael 'kena' Poss's avatar
      deps: bump datadriven · bdea159e
      Raphael 'kena' Poss authored
      This ensures that datadriven tests do not report all directives on
      their output unless `-v` is passed to `go test`, as was originally
      intended.
      
      Noticed/requested by @RaduBerinde.
      
      Release note: None
      bdea159e
    • craig[bot]'s avatar
      Merge #51185 · 161d68e8
      craig[bot] authored
      51185: cli: fix dump to gracefully handle temp tables, views and sequences r=adityamaru a=adityamaru
      
      pg_dump ignores temporary tables, views and sequences.  This change
      makes `cockroach dump` follow the same behavior by skipping descriptors
      with schema name having `pg_temp` as its prefix, when aggregating the
      metadata for the tables to be dumped.
      
      When no table names are specified in the `cockroach dump` command, we
      silently ignore all temp objects.  If the user explicitly asks for a
      temp object to be dumped, we error out with an informative message.
      
      Fixes:#50899
      Fixes:#51166
      
      Release note (bug fix): `cockroach dump` no longer errors out when
      dumping temporary tables, view or sequences.  It either ignores them or
      throws an informative error if the temp object is explicitly requested
      to be dumped via the CLI.
      Co-authored-by: default avatarAditya Maru <[email protected]>
      161d68e8
    • craig[bot]'s avatar
      Merge #50934 · 41e8c9f5
      craig[bot] authored
      50934: opt: calculate partial index scan statistics r=mgartner a=mgartner
      
      This commit updates the statistics builder to account for the
      selectivity of predicate expressions for scans over partial indexes.
      
      Fixes #50220
      Co-authored-by: default avatarMarcus Gartner <[email protected]>
      41e8c9f5
    • craig[bot]'s avatar
      Merge #50967 · 550d32ab
      craig[bot] authored
      50967: backupccl: Support BACKUP and RESTORE TENANT r=dt a=dt
      
      backupccl: tenant backup/restore support
      
      Tenants are made up of three things:
       1) a tenant ID 2) some metadata bytes and 3) the tenant's key-span.
      
      Backing up a tenant consists of exporting their keyspan, and restoring a
      tenant means creating a tenant record for that ID and importing that
      keyspan.
      
      In many ways this is much simpler than table, database or cluster backup
      and restore, since we don't need to worry about individual tables or
      databases or how they are related or not -- by simply importing and
      exporting the tenant's entire keyspan, we've captured all of their data,
      and things like jobs or schema changes or migrations will simply pick up
      from where they left off just as if they were resuming from a reboot.
      
      BACKUP and RESTORE TENANT are only available to the system tenant.
      
      Tenants can only be backed up and restored completely -- there is no way
      currently for the system tenant to peek into a tenant's key-space and
      find individual tables or databases to backup. Similarly backed up
      tenants are opaque spans of keys in the backup -- we do not know what or
      even how many tables are in that backed up keyspan.
      
      Additionally, tenants can only be restored to the same ID -- i.e. if you
      BACKUP TENANT 5, you can only restore them as TENANT 5 in the restoring
      cluster, which thus must not already contain an existing tenant 5. In
      theory we could re-key the tenant's data the same way RESTORE can rekey
      table data, however the fact that the tenant ID may appear in values
      as well complicates this. One example is jobs which checkpoint spans or
      high-watermark keys in their progress info -- these are encoded with the
      tenant ID, meaning they'd need to be found and rewritten to support
      RESTORE'ing to a new tenant ID.
      
      Since a tenant is just a target span like a table, other backup/restore
      features work the same on tenant backups as they do on table backups:
      Changes to a tenant's key=space since the last backup can be backed up
      incrementally, just like with table backups, and other features like
      partitioning, AS OF SYSTEM TIME, revision_history and encryption should
      work as well.
      
      Release note (enterprise change): Support BACKUP TENANT and RESTORE TENANT.
      Co-authored-by: default avatarDavid Taylor <[email protected]>
      550d32ab
  2. 13 Jul, 2020 28 commits
    • craig[bot]'s avatar
      Merge #51227 #51247 · 94ffb1e9
      craig[bot] authored
      51227: kvcoord: simplify rangeCache.clearOlderOverlapping r=andreimatei a=andreimatei
      
      This method had too much code - it was traversing the tree twice for no
      good reason.
      Also added some more tests for it.
      
      Release note: None
      
      51247: kv/concurrency: use errors.AssertionFailedf in lockTable r=nvanbenschoten a=nvanbenschoten
      
      Informs #50996.
      
      This avoids unwanted redaction in Sentry reports from these assertion failures.
      Co-authored-by: default avatarAndrei Matei <[email protected]>
      Co-authored-by: default avatarNathan VanBenschoten <[email protected]>
      94ffb1e9
    • Marcus Gartner's avatar
      opt: calculate partial index scan statistics · d750b919
      Marcus Gartner authored
      This commit updates the statistics builder to account for the
      selectivity of predicate expressions for scans over partial indexes.
      
      Release note: None
      d750b919
    • Aditya Maru's avatar
      cli: fix dump to gracefully handle temp tables, views and sequences · d8c41da6
      Aditya Maru authored
      pg_dump ignores temporary tables, views and sequences.  This change
      makes `cockroach dump` follow the same behavior by skipping descriptors
      with schema name having `pg_temp` as its prefix, when aggregating the
      metadata for the tables to be dumped.
      
      When no table names are specified in the `cockroach dump` command, we
      silently ignore all temp objects.  If the user explicitly asks for a
      temp object to be dumped, we error out with an informative message.
      
      Fixes:#50899
      
      Release note (bug fix): `cockroach dump` no longer errors out when
      dumping temporary tables, view or sequences.  It either ignores them or
      throws an informative error if the temp object is explicitly requested
      to be dumped via the CLI.
      d8c41da6
    • Aditya Maru's avatar
      cli: change statement diag to use UTC time instead of Local time · 82e33ce9
      Aditya Maru authored
      Previously, in the case of cli tests we overrode Local to be equal to
      UTC time so that the output is always the same. This assignment however
      resulted in data race conditions in several CLI unit tests.
      
      Egs:
      https://teamcity.cockroachdb.com/repository/download/Cockroach_UnitTests/
      2078378:id/testrace/logTestDumpTempTables_dump_db_only_temp068989627/
      clitest-stderr.4ab17fe2fbed.roach.2020-07-09T20_12_24Z.018111.log
      
      This change ensures all statement diag times are always show in UTC, thereby
      eliminating the need to override Local.
      
      Release note (cli change): statement-diag command will now show all
      times in UTC.
      82e33ce9
    • craig[bot]'s avatar
      Merge #51184 #51389 · 6e39e58c
      craig[bot] authored
      51184: storage: optimize MVCCGarbageCollect for large numbers of versions r=itsbilal a=ajwerner
      
      The first two commits rework the benchmarks a tad. The third commit is the meat.
      
      Touches and maybe fixes #50194.
      
      Release note (performance improvement): Improved the efficiency of garbage
      collection when there are a large number of versions of a single key,
      commonly found when utilizing sequences.
      
      
      
      51389: vendor: Bump pebble to 36c1b9c7a833, and set FlushSplitBytes r=itsbilal a=itsbilal
      
      Pulls in these changes:
       - 36c1b9c7a833bce8d1ab6f2013780d2d6f3a97f6 db: relax deletion-only compaction conditions
       - a383f2cde93937d8decb8a802f7377163d8bcb7c db: implement delete-only compactions
       - b6a03ca6a613247a86eb62b73ded803fe1c7f5fa internal/manifest: add LevelIterator struct
       - 9f08cf3d3dc69568de3b72b87abe3cee83df5299 *: Always pick base compactions with minCompactionDepth = 1
       - 3a9adc12d6c366ef0f876b773639ed4a98136fcd internal/metamorphic: generate deterministically from prng seed
       - c9a380a7f499f2844aca2990fd012a4068f4c775 tool: Only encode sublevel changes in lsm json object
       - 58d55986b0c37aaf06301ebba4d50bfa95a9cacf docs: fix mouse position calculation on zoom/pan
       - 6cdafa7b75584e1e886ebe19dcdae3e25b6ba9ee docs: use flexbox for layout of nightly benchmarks page
       - 884334184862059f43737420479a4caecdcdb33f docs: display annotation in chart title when hovered
       - 9c8d7390339278b65647bb849f36759747fad606 docs: allow hovering over the detail line
       - 709c5f7d9380b01b2f71b321cd540d2f7df6bb89 docs: add last updated text to nightly benchmarks
       - 0f71f3c8a72a1ce454576a825cad3d44c88c3424 db: clarify Batch.Reset docs
       - deb60d0bf8b53f975884090cc639780de7fefe80 internal/mkbench: walk through symlinks
       - 3b1143d859ffaee68d7f3060a1bf346c008d7de9 docs: indicate data for missing days
       - 999b9646a584ba354a87b9ba60b57b8018b99e93 internal/mkbench: be more permissive in data.js parsing
       - 3cd66bb123c09088a450092363765ad6e2bccace cmd: Address TODO on measuring scan read-amp
       - 8446ccc4c58c02515dbe66e4f9e4818645e0d5c9 internal/manifest: Have flush split bytes account for # of sublevels
       - f8d8b9654d3ed5a66e0d80c2cce65cbefe1926cf internal/mkbench: don't require bzip compression of raw data
       - f32ddecd06a3e06f53ace9d3451d9ef5d68c8440 internal/mkbench: utility for processing nightly benchmark data
       - 5a444e1b29fa1a96da37557315e5133d78bb4c86 docs: serve benchmark data out of S3
       - c1d4e3ce262e0be2849740a9f8b3924abf47858d compaction: Create pickedCompaction struct for use in compaction_picker
       - f02d55b076e0fa277482e43017dd030b7e764fe1 internal/manifest: Don't create new sublevel if overlap is on sentinel key
       - 7fce6da1eb8c0c34238becc1c0885f4acd600bf8 Add panning and zooming
       - 5bf72881988e9d33a48ad0ca522c3fbc9eb1eebc docs: add nightly benchmarks
       - a452ae2c916a74a035d264e42c9b761061a988b5 db: log events unredacted where possible
       - 9dcb6e30ee67f8b5e8ce200e755e6663c0ee81c1 db: truncate tombstones to file bounds when calculating table stats
      
      Based on discussion and benchmarks done as part of
      cockroachdb/pebble#781, it's
      more idiomatic to tie FlushSplitBytes to the L0 TargetFileSize.
      The 2x factor helps to reduce cases of excessive flush splitting.
      
      Release note: None.
      Co-authored-by: default avatarAndrew Werner <[email protected]>
      Co-authored-by: default avatarBilal Akhtar <[email protected]>
      6e39e58c
    • irfan sharif's avatar
      Makefile: speed up `buildshort` and introduce `cleanshort` target · 36918d5b
      irfan sharif authored
      I took some time to investigate where all time was going every `make
      buildshort` invocation, and here are a few changes to tighten it up a
      bit. Here's what my workflow tends to look like:
      
      ```sh
      $ make clean       # Used gratuitously, like `clear`. Probably shouldn't.
      $ make buildshort  # To check if everything compiles.
      $ make buildshort  # As I'm iterating on things, to keep checking things.
      ```
      
      Almost all of my changes are exclusively go code, and as far as
      generated code goes, it's only ever protobuf generated code. Others may
      also use `make generate` to regenerate code (including execgen, optgen,
      etc.), but I assume the workflow is roughly the same for all. I also
      assume protobufs are changed far more often than `execgen` or `optgen`
      code.
      
      To speed things up for most folks, this PR aims to make the common
      sequences of "iteratively compile crdb" and "clean up some stuff and
      compile crdb" faster than it was before. It does so by doing the
      following not tagging `build.utcTime` on every `buildshort` invocation,
      which lets us make much better use of the go build cache.
      
      ```sh
      $ make clean; time make buildshort       # Fresh builds before
        156.40 real       553.84 user       380.26 sys
      $ make clean; time make buildshort       # Fresh builds after
        121.04 real       384.52 user       278.03 sys
      
      $ make buildshort; time make buildshort  # No-op builds before
        27.44 real        21.91 user        32.50 sys
      $ make buildshort; time make buildshort  # No-op builds after
        4.79 real         3.21 user        30.79 sys
      ```
      
      We also introduce a `cleanshort` target, which leaves the go build cache
      and C++ artifacts intact. It should suffice for most folks editing
      `.proto` files and regular go code. `{exec,opt}gen` and storage folks
      may still need to fall back to `make {clean,generate}`, but nobody else.
      
      ```sh
      $ make clean; time make buildshort        # Fresh build; nuking most things.
        121.04 real       384.52 user       278.03 sys
      
      $ make cleanshort; time make buildshort   # Fresh-ish build; nuking only go protos.
        31.10 real        29.04 user        36.76 sys
      ```
      
      We stopped short of not rebuilding the settings HTML page or the docgen
      page for `buildshort` (but there are additional seconds to be shaved off
      here).
      
      ---
      
      We also add a loud warning if `ccache` is not present on the
      maintainer's system, as it really should be.
      
      ```sh
      $ make clean; time make build   # Without ccache
      409.45 real      3251.96 user       598.34 sys
      
      $ make clean; time make build   # With ccache
      156.40 real       553.84 user       380.26 sys
      ```
      
      Release note: None
      36918d5b
    • craig[bot]'s avatar
      Merge #51149 · dde3cfcb
      craig[bot] authored
      51149: sqlbase: add validation for type cross references in table descs r=rohany a=rohany
      
      This PR adds validation for any referenced types within a table
      descriptor.
      
      Release note: None
      Co-authored-by: default avatarRohan Yadav <[email protected]>
      dde3cfcb
    • craig[bot]'s avatar
      Merge #50953 · 1b70ae49
      craig[bot] authored
      50953: sql: add benchmark for counting how many roundtrips a ddl statement performs r=solongordon a=RichardJCai
      
      sql: add benchmark for counting how many roundtrips a ddl statement performs
      
      Release note: None
      
      Example output:
      ```
      goos: darwin
      goarch: amd64
      pkg: github.com/cockroachdb/cockroach/pkg/bench
      BenchmarkGrantTables/grant_on_1_table-8         	1000000000	         0.164 ns/op	        10.0 roundtrips
      BenchmarkGrantTables/grant_on_10_tables-8       	1000000000	         0.445 ns/op	       160 roundtrips
      --- BENCH: BenchmarkGrantTables
          test_log_scope.go:77: test logs captured to: /var/folders/s9/5m7lxn217451rj169w3sh8bh0000gp/T/logBenchmarkGrantTables101520749
          test_log_scope.go:58: use -show-logs to present logs inline
      BenchmarkDropTables/drop_1_table-8              	       1	1043545344 ns/op	         7.00 roundtrips
      BenchmarkDropTables/drop_2_tables-8             	       1	1032635967 ns/op	        16.0 roundtrips
      BenchmarkDropTables/drop_3_tables-8             	       1	1032938773 ns/op	        27.0 roundtrips
      --- BENCH: BenchmarkDropTables
          test_log_scope.go:77: test logs captured to: /var/folders/s9/5m7lxn217451rj169w3sh8bh0000gp/T/logBenchmarkDropTables622127676
          test_log_scope.go:58: use -show-logs to present logs inline
      PASS
      ```
      Co-authored-by: default avatarrichardjcai <[email protected]>
      1b70ae49
    • richardjcai's avatar
      sql: add benchmark for counting how many roundtrips · 0517ff2e
      richardjcai authored
      a ddl statement performs.
      
      Release note: None
      0517ff2e
    • richardjcai's avatar
      0ae1d652
    • Andrew Werner's avatar
      storage: optimize MVCCGarbageCollect · 8e5423b1
      Andrew Werner authored
      Prior to this change, MVCCGarbageCollect performed a linear scan of all
      versions of a key, not just the versions being garbage collected. Given
      the pagination of deleting versions above this call, the linear behavior
      can result in quadratic runtime of GC when the number of versions vastly
      exceeds the page size. The benchmark results demonstrate the change's
      effectiveness.
      
      It's worth noting that for a single key with a single version, the change
      has a negative performance impact. I suspect this is due to the allocation
      of a key in order to construct the iterator. In cases involving more keys,
      I theorize the positive change is due to the fact that now the iterator
      is never seeked backwards due to the sorting of the keys. It's worth
      noting that since 20.1, the GC queue has been sending keys in the GC
      request in reverse order. I anticipate that this sorting is likely a good
      thing in that case too.
      
      The stepping optimization seemed important in the microbenchmarks for cases
      where most of the data was garbage. Without it, the change had small negative
      impact on performance.
      
      ```
      name                                                                                                     old time/op  new time/op  delta
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1/numVersions=2/deleteVersions=1-24           3.39µs ± 1%  3.96µs ± 0%  +16.99%  (p=0.004 n=6+5)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1/numVersions=1024/deleteVersions=1-24         319µs ± 3%    10µs ±12%  -96.88%  (p=0.002 n=6+6)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1/numVersions=1024/deleteVersions=16-24        319µs ± 2%    16µs ±10%  -94.95%  (p=0.002 n=6+6)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1/numVersions=1024/deleteVersions=32-24        319µs ± 3%    21µs ± 5%  -93.52%  (p=0.002 n=6+6)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1/numVersions=1024/deleteVersions=512-24       337µs ± 1%   182µs ± 3%  -46.00%  (p=0.002 n=6+6)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1/numVersions=1024/deleteVersions=1015-24      361µs ± 0%   353µs ± 2%   -2.32%  (p=0.010 n=4+6)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1/numVersions=1024/deleteVersions=1023-24      361µs ± 3%   350µs ± 2%   -3.14%  (p=0.009 n=6+6)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1024/numVersions=2/deleteVersions=1-24        2.00ms ± 3%  2.25ms ± 2%  +12.53%  (p=0.004 n=6+5)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1024/numVersions=1024/deleteVersions=1-24      388ms ± 3%    16ms ± 5%  -95.76%  (p=0.002 n=6+6)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1024/numVersions=1024/deleteVersions=16-24     387ms ± 1%    27ms ± 3%  -93.14%  (p=0.002 n=6+6)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1024/numVersions=1024/deleteVersions=32-24     393ms ± 5%    35ms ± 4%  -91.09%  (p=0.002 n=6+6)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1024/numVersions=1024/deleteVersions=512-24    463ms ± 4%   276ms ± 3%  -40.43%  (p=0.004 n=5+6)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1024/numVersions=1024/deleteVersions=1015-24   539ms ± 5%   514ms ± 3%   -4.64%  (p=0.016 n=5+5)
      MVCCGarbageCollect/rocksdb/keySize=128/valSize=128/numKeys=1024/numVersions=1024/deleteVersions=1023-24   533ms ± 4%   514ms ± 1%     ~     (p=0.093 n=6+6)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1/numVersions=2/deleteVersions=1-24            1.97µs ± 3%  2.29µs ± 2%  +16.58%  (p=0.002 n=6+6)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1/numVersions=1024/deleteVersions=1-24          139µs ± 1%     5µs ± 6%  -96.40%  (p=0.004 n=5+6)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1/numVersions=1024/deleteVersions=16-24         140µs ± 1%     8µs ± 1%  -94.13%  (p=0.004 n=6+5)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1/numVersions=1024/deleteVersions=32-24         143µs ± 4%    11µs ± 2%  -92.03%  (p=0.002 n=6+6)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1/numVersions=1024/deleteVersions=512-24        178µs ± 9%   109µs ± 1%  -38.75%  (p=0.004 n=6+5)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1/numVersions=1024/deleteVersions=1015-24       201µs ± 1%   213µs ± 1%   +5.80%  (p=0.008 n=5+5)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1/numVersions=1024/deleteVersions=1023-24       205µs ±11%   215µs ± 6%     ~     (p=0.126 n=5+6)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1024/numVersions=2/deleteVersions=1-24         1.43ms ± 1%  1.34ms ± 1%   -5.82%  (p=0.004 n=6+5)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1024/numVersions=1024/deleteVersions=1-24       218ms ± 9%     9ms ± 2%  -96.00%  (p=0.002 n=6+6)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1024/numVersions=1024/deleteVersions=16-24      216ms ± 3%    15ms ± 2%  -93.19%  (p=0.004 n=5+6)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1024/numVersions=1024/deleteVersions=32-24      219ms ± 4%    20ms ± 5%  -90.77%  (p=0.004 n=5+6)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1024/numVersions=1024/deleteVersions=512-24     303ms ± 4%   199ms ± 4%  -34.47%  (p=0.004 n=5+6)
      MVCCGarbageCollect/pebble/keySize=128/valSize=128/numKeys=1024/numVersions=1024/deleteVersions=1015-24    382ms ±16%   363ms ± 8%     ~     (p=0.485 n=6+6)
      [email protected]:~/go/src/github.com/cockroachdb/cockroach$ %ns=1024/deleteVersions=1023-24    363ms ± 4%   354ms ± 4%     ~     (p=0.222 n=5+5)
      ```
      
      Release note (performance improvement): Improved the efficiency of garbage
      collection when there are a large number of versions of a single key,
      commonly found when utilizing sequences.
      8e5423b1
    • craig[bot]'s avatar
      Merge #51244 #51352 · 21536de0
      craig[bot] authored
      51244: opt: add fittin.gs external stats and example queries r=mjibson a=mjibson
      
      Release note: None
      
      51352: sql: implement datetime_precision column in information_schema.columns r=r6min a=r6min
      
      Before: The datetime_precision column was null for all types.
      
      Why: Useful for users to reverse engineer the data type of columns in
      a user table.
      
      Now: The datetime_precision column shows the precision for columns with
      type `TIME`/`TIMETZ`, `TIMESTAMP`/`TIMESTAMPTZ` and `INTERVAL`, and is null for
      any other types.
      
      Resolves #51286
      
      Release note (sql change): Implement the datetime_precision column in
      the information_schema.columns table for time-stored values.
      Co-authored-by: default avatarMatt Jibson <[email protected]>
      Co-authored-by: default avatarRyan Min <[email protected]>
      21536de0
    • Bilal Akhtar's avatar
      storage: Set FlushSplitBytes to 2*(L0 TargetFileSize) · b097461e
      Bilal Akhtar authored
      Based on discussion and benchmarks done as part of
      https://github.com/cockroachdb/pebble/pull/781, it seems
      more idiomatic to tie FlushSplitBytes to the L0 TargetFileSize.
      The 2x factor helps to reduce cases of excessive flush splitting.
      
      Release note: None
      b097461e
    • Bilal Akhtar's avatar
      vendor: Bump pebble to 36c1b9c7a833 · 42159725
      Bilal Akhtar authored
      Pulls in these changes:
       - 36c1b9c7a833bce8d1ab6f2013780d2d6f3a97f6 db: relax deletion-only compaction conditions
       - a383f2cde93937d8decb8a802f7377163d8bcb7c db: implement delete-only compactions
       - b6a03ca6a613247a86eb62b73ded803fe1c7f5fa internal/manifest: add LevelIterator struct
       - 9f08cf3d3dc69568de3b72b87abe3cee83df5299 *: Always pick base compactions with minCompactionDepth = 1
       - 3a9adc12d6c366ef0f876b773639ed4a98136fcd internal/metamorphic: generate deterministically from prng seed
       - c9a380a7f499f2844aca2990fd012a4068f4c775 tool: Only encode sublevel changes in lsm json object
       - 58d55986b0c37aaf06301ebba4d50bfa95a9cacf docs: fix mouse position calculation on zoom/pan
       - 6cdafa7b75584e1e886ebe19dcdae3e25b6ba9ee docs: use flexbox for layout of nightly benchmarks page
       - 884334184862059f43737420479a4caecdcdb33f docs: display annotation in chart title when hovered
       - 9c8d7390339278b65647bb849f36759747fad606 docs: allow hovering over the detail line
       - 709c5f7d9380b01b2f71b321cd540d2f7df6bb89 docs: add last updated text to nightly benchmarks
       - 0f71f3c8a72a1ce454576a825cad3d44c88c3424 db: clarify Batch.Reset docs
       - deb60d0bf8b53f975884090cc639780de7fefe80 internal/mkbench: walk through symlinks
       - 3b1143d859ffaee68d7f3060a1bf346c008d7de9 docs: indicate data for missing days
       - 999b9646a584ba354a87b9ba60b57b8018b99e93 internal/mkbench: be more permissive in data.js parsing
       - 3cd66bb123c09088a450092363765ad6e2bccace cmd: Address TODO on measuring scan read-amp
       - 8446ccc4c58c02515dbe66e4f9e4818645e0d5c9 internal/manifest: Have flush split bytes account for # of sublevels
       - f8d8b9654d3ed5a66e0d80c2cce65cbefe1926cf internal/mkbench: don't require bzip compression of raw data
       - f32ddecd06a3e06f53ace9d3451d9ef5d68c8440 internal/mkbench: utility for processing nightly benchmark data
       - 5a444e1b29fa1a96da37557315e5133d78bb4c86 docs: serve benchmark data out of S3
       - c1d4e3ce262e0be2849740a9f8b3924abf47858d compaction: Create pickedCompaction struct for use in compaction_picker
       - f02d55b076e0fa277482e43017dd030b7e764fe1 internal/manifest: Don't create new sublevel if overlap is on sentinel key
       - 7fce6da1eb8c0c34238becc1c0885f4acd600bf8 Add panning and zooming
       - 5bf72881988e9d33a48ad0ca522c3fbc9eb1eebc docs: add nightly benchmarks
       - a452ae2c916a74a035d264e42c9b761061a988b5 db: log events unredacted where possible
       - 9dcb6e30ee67f8b5e8ce200e755e6663c0ee81c1 db: truncate tombstones to file bounds when calculating table stats
      
      Release note: None.
      42159725
    • craig[bot]'s avatar
      Merge #51231 · 14ea7236
      craig[bot] authored
      51231: opt: add per-row costs to all filters r=mjibson a=mjibson
      
      Previously only the geo lookup join accounted for per-row filter costs,
      thus over costing it relative to other joins and select.
      Co-authored-by: default avatarMatt Jibson <[email protected]>
      14ea7236
    • Rohan Yadav's avatar
      sqlbase: add validation for type cross references in table descs · bffe18e3
      Rohan Yadav authored
      This PR adds validation for any referenced types within a table
      descriptor.
      
      Release note (enterprise change): Disables the use of `IMPORT` with user
      defined types. Users should use `IMPORT INTO` instead.
      bffe18e3
    • Andrew Werner's avatar
      storage: augment BenchmarkGarbageCollect to different deletion ratios · 035f8a8c
      Andrew Werner authored
      Before this change, BenchmarkGarbageCollect always benchmarked collecting
      all but the last version of a key. This change expands the test to benchmark
      different numbers of deleted versions to highlight that the current
      implementation is linear in the number of versions.
      
      Release note: None
      035f8a8c
    • Andrew Werner's avatar
      storage: unify BenchmarkGarbageCollect between engines · 422b7251
      Andrew Werner authored
      This commit unifies `BenchmarkGarbageCollect` in the style of
      `BenchmarkExportToSst` and moves both from engine-specific
      files to bench_test.go
      
      Release note: None
      422b7251
    • craig[bot]'s avatar
      Merge #51327 · 51398fb4
      craig[bot] authored
      51327: opt: make partial index predicate implication tests more robust r=RaduBerinde a=mgartner
      
      #### opt: make partial index predicate implication tests more robust
      
      This commit makes the partial index predicate implication more robust by
      ensuring that predicates are normalized in `Implicator` tests the same
      way that they are normalized during Select queries. A new function,
      `factory.NormalizePartialIndexPredicate`, performs predicate-specific
      normalization and is called both in `partialidx` tests and in
      `optbuilder/select.go`.
      
      This commit also changes the `partialidx` tests so that query filters
      are normalized in the context of a Select expression. This allows
      normalization rules specific to filters within Selects to be performed.
      The resulting filters better mimic filters in a real Select query.
      
      These two changes should help prevent cases where implication proofs in
      tests differ from proofs in real Select queries.
      
      Release note: None
      
      #### opt: fix partial index predicate implication for inlined const vars
      
      This commit fixes a false-negative in partial index predicate
      implication due to constant variables being inlined in the query filter
      but not in the predicate.
      
      Fixes #51177.
      
      Release note: None
      Co-authored-by: default avatarMarcus Gartner <[email protected]>
      51398fb4
    • Matt Jibson's avatar
      opt: add per-row costs to all filters · 89f0350f
      Matt Jibson authored
      Previously only the geo lookup join accounted for per-row filter costs,
      thus over costing it relative to other joins and select.
      
      Release note: None
      89f0350f
    • craig[bot]'s avatar
      Merge #51110 #51335 #51354 #51376 · 834dfe9a
      craig[bot] authored
      51110: sql: add a `TypeDescriptor.Validate` function r=rohany a=rohany
      
      This PR adds a validation function for `TypeDescriptor` and calls it
      before `TypeDescriptor`s are written to disk.
      
      Release note: None
      
      51335: sql: disallow row-level locking in READ-ONLY txns r=RaduBerinde a=hiepd
      
      Closes #50476
      
      Release note (sql change): Disallow row-level locking in READ-ONLY txns
                          to be consistent with Postgres behavior
      
      51354: sql: clean up explain flags r=RaduBerinde a=RaduBerinde
      
      This change simplifies explain flags and moves them to `exec/explain`.
      
      Release note: None
      
      51376: colexec: implement Close in Columnarizer r=yuzefovich a=asubiotto
      
      A Columnarizer needs to call ConsumerClosed on the wrapped input tree since
      those inputs may not be drained in some cases.
      
      Release note: None
      
      Fixes #51179 
      Co-authored-by: default avatarRohan Yadav <[email protected]>
      Co-authored-by: default avatarmanhhiep92 <[email protected]>
      Co-authored-by: default avatarRadu Berinde <[email protected]>
      Co-authored-by: default avatarAlfonso Subiotto Marques <[email protected]>
      834dfe9a
    • Matt Jibson's avatar
      opt: factor out setup and per-row cost in computeFiltersCost · c33fe858
      Matt Jibson authored
      This is in preparation to use computeFiltersCost consistently in all
      places.
      
      Release note: None
      c33fe858
    • Alfonso Subiotto Marques's avatar
      colexec: implement Close in Columnarizer · b264cd57
      Alfonso Subiotto Marques authored
      A Columnarizer needs to call ConsumerClosed on the wrapped input tree since
      those inputs may not be drained in some cases.
      
      Release note: None
      b264cd57
    • Andrei Matei's avatar
      kvcoord: simplify rangeCache.clearOlderOverlapping · dab84349
      Andrei Matei authored
      This method had too much code - it was traversing the tree twice for no
      good reason.
      Also added some more tests for it.
      
      Release note: None
      dab84349
    • Andrei Matei's avatar
      kvcoord: clarify locking requirement of clearOlderOverlapping · ee5dd35d
      Andrei Matei authored
      This method requires exclusive access to the cache, so renamed it to
      *Locked(). Also added a version that takes the lock, to be used by tests
      - which previously were technically missing the locking (the omission
        was benign since the tests are single-threaded).
      
        Release note: None
      ee5dd35d
    • Andrei Matei's avatar
      kvclient: improve a DistSender test · 5ab8d8e8
      Andrei Matei authored
      Remove log.Fatal()s from a goroutine.
      Also teach RangeIterator.Seek() to return a wrapped error.
      
      Release note: None
      5ab8d8e8
    • craig[bot]'s avatar
      Merge #51108 · 0848ed62
      craig[bot] authored
      51108: release: ship libgeos.* and libgeos_c.* files r=jlinder a=otan
      
      This commit adds libgeos.* and libgeos_c.* files to be uploaded to S3 at
      `s3://cockroach/cockroach/lib/` as well as adding the files into the
      .tar.gz and .zip archives.
      
      Release note (general change): The CockroachDB tarballs now include
      libgeos and libgeos_c static libraries, which are required to be copied
      to /usr/local/lib/cockroach for UNIX/MacOS systems to perform certain
      geospatial operations. The Windows location will be finalized at a later
      date. This location is configurable with the `--geo-libs` flag.
      Co-authored-by: default avatarOliver Tan <[email protected]>
      0848ed62
    • Matt Jibson's avatar
      opt: add fittin.gs external stats and example queries · 7f5434eb
      Matt Jibson authored
      Release note: None
      7f5434eb