RocksDB Administration and Data Access Tool

Ldb Tool

The ldb command-line tool provides various data access and database management commands. Below are some samples.

Data Access Example

$./ldb --db=/tmp/test_db --create_if_missing put a1 b1
    OK

    $ ./ldb --db=/tmp/test_db get a1
    b1

    $ ./ldb --db=/tmp/test_db get a2
    Failed: NotFound:

    $ ./ldb --db=/tmp/test_db scan
    a1 : b1

    $ ./ldb --db=/tmp/test_db scan --hex
    0x6131 : 0x6231

    $ ./ldb --db=/tmp/test_db put --key_hex 0x6132 b2
    OK

    $ ./ldb --db=/tmp/test_db scan
    a1 : b1
    a2 : b2

    $ ./ldb --db=/tmp/test_db get --value_hex a2
    0x6232

    $ ./ldb --db=/tmp/test_db get --hex 0x6131
    0x6231

    $ ./ldb --db=/tmp/test_db batchput a3 b3 a4 b4
    OK

    $ ./ldb --db=/tmp/test_db scan
    a1 : b1
    a2 : b2
    a3 : b3
    a4 : b4

    $ ./ldb --db=/tmp/test_db batchput "multiple words key" "multiple words value"
    OK

    $ ./ldb --db=/tmp/test_db scan
    Created bg thread 0x7f4a1dbff700
    a1 : b1
    a2 : b2
    a3 : b3
    a4 : b4
    multiple words key : multiple words value

Dump leveldb database data in HEX format

./ldb --db=/tmp/test_db dump --hex > /tmp/dbdump

Load data from HEX format dump into a new leveldb database

cat /tmp/dbdump | ./ldb --db=/tmp/test_db_new load --hex --compression_type=bzip2 --block_size=65536 --create_if_missing --disable_wal

Compact an existing leveldb database

./ldb --db=/tmp/test_db_new compact --compression_type=bzip2 --block_size=65536

The --column_family=<string> command-line argument can be used to specify the column family of the query operation, and the –try_load_options command-line argument can be used to load the DB’s option file to open the DB. It is recommended to always set this parameter when operating on the DB. If the default option is used when opening the DB instead of the information in the DB’s option file, it may mess up the LSM-tree and subsequent automatic recovery will not be possible.

SST dump tool

The sst_dump tool can dump data and analyze SST files. The sst_dump tool provides several methods of operation on SST files.

$ ./sst_dump
file or directory must be specified.

sst_dump --file=<data_dir_OR_sst_file> [--command=check|scan|raw]
    --file=<data_dir_OR_sst_file>
      Path to SST file or directory containing SST files

    --command=check|scan|raw|verify
        check: Iterate over entries in files but dont print anything except if an error is encounterd (default command)
        scan: Iterate over entries in files and print them to screen
        raw: Dump all the table contents to <file_name>_dump.txt
        verify: Iterate all the blocks in files verifying checksum to detect possible coruption but dont print anything except if a corruption is encountered
        recompress: reports the SST file size if recompressed with different
                    compression types

    --output_hex
      Can be combined with scan command to print the keys and values in Hex

    --from=<user_key>
      Key to start reading from when executing check|scan

    --to=<user_key>
      Key to stop reading at when executing check|scan

    --prefix=<user_key>
      Returns all keys with this prefix when executing check|scan
      Cannot be used in conjunction with --from

    --read_num=<num>
      Maximum number of entries to read when executing check|scan

    --verify_checksum
      Verify file checksum when executing check|scan

    --input_key_hex
      Can be combined with --from and --to to indicate that these values are encoded in Hex

    --show_properties
      Print table properties after iterating over the file when executing
      check|scan|raw

    --set_block_size=<block_size>
      Can be combined with --command=recompress to set the block size that will
      be used when trying different compression algorithms

    --compression_types=<comma-separated list of CompressionType members, e.g.,
      kSnappyCompression>
      Can be combined with --command=recompress to run recompression for this
      list of compression types

    --parse_internal_key=<0xKEY>
      Convenience option to parse an internal key on the command line. Dumps the
      internal key in hex format {'key' @ SN: type}

Dump SST file blocks

./sst_dump --file=/path/to/sst/000829.sst --command=raw

This command generates a txt file named /path/to/sst/000829.sst. The file contains all the Index blocks and data blocks in hex encoding, as well as table property-related information, footer details, and meta index details.

Printing entries in SST file

./sst_dump --file=/path/to/sst/000829.sst --command=scan --read_num=5

The following command will print the first 5 keys in the SST file

'Key1' @ 5: 1 => Value1
'Key2' @ 2: 1 => Value2
'Key3' @ 4: 1 => Value3
'Key4' @ 3: 1 => Value4
'Key5' @ 1: 1 => Value5

The input format can be abstracted as

'<key>' @ <sequence number>: <type> => <value>

If the key contains non-ASCII characters, they cannot be displayed on the screen, in which case it is best to use –output_hex

./sst_dump --file=/path/to/sst/000829.sst --command=scan --read_num=5 --output_hex

You can also set the starting and ending positions of the read operation

./sst_dump --file=/path/to/sst/000829.sst --command=scan --from="key2" --to="key4"

You can also pass from-to parameters in HEX

./sst_dump --file=/path/to/sst/000829.sst --command=scan --from="0x6B657932" --to="0x6B657934" --input_key_hex

Check SST file

./sst_dump --file=/path/to/sst/000829.sst --command=check --verify_checksum

This command will iterate through all the entries in the SST file, but will not print any information unless SST file problems are found. This command operation will also verify the checksum. Print SST file properties

./sst_dump --file=/path/to/sst/000829.sst --show_properties

This command reads the properties of the SST file and prints them, outputting as follows:

from [] to []
Process /path/to/sst/000829.sst
Sst file format: block-based
Table Properties:
------------------------------
  # data blocks: 26541
  # entries: 2283572
  raw key size: 264639191
  raw average key size: 115.888262
  raw value size: 26378342
  raw average value size: 11.551351
  data block size: 67110160
  index block size: 3620969
  filter block size: 0
  (estimated) table size: 70731129
  filter policy name: N/A
  # deleted keys: 571272

SST_dump can be used to check the file size under different compression algorithms

./sst_dump --file=/path/to/sst/000829.sst --show_compression_sizes

With the –show_compression_sizes parameter, sst_dump will rebuild the SST file in memory and can use different compression algorithms, then report the file size, as output below

from [] to []
Process /path/to/sst/000829.sst
Sst file format: block-based
Block Size: 16384
Compression: kNoCompression Size: 103974700
Compression: kSnappyCompression Size: 103906223
Compression: kZlibCompression Size: 80602892
Compression: kBZip2Compression Size: 76250777
Compression: kLZ4Compression Size: 103905572
Compression: kLZ4HCCompression Size: 97234828
Compression: kZSTDNotFinalCompression Size: 79821573

These files are generated in memory, and the block size in the file is 16KB, which can be modified by configuring –set_block_size.

Buy Me a Coffee

Back to Table of Contents

Disclaimer

License under CC BY-NC 4.0
Copyright issue feedback me#imzye.com, replace # with @
Not all the commands and scripts are tested in production environment, use at your own risk
No personal information is collected
Partial content rewritten by AI, verified by humans

Feedback