Inspect data in BoltDB

Vault clusters configured to use Integrated Storage (Raft), the cluster persists all encrypted data to a bbolt key/value store.

You can inspect data in clusters which use Integrated Storage by operating Vault in recovery mode and using the sys/raw API as described in Inspect Data in Integrated Storage. That process is useful for troubleshooting issues in operable Vault clusters and provides the greatest detail possible for issue resolution and troubleshooting.

You can also use information in this tutorial to access low level details from the database by directly examining the database files.

This is a tool of last resort, but it's helpful when you suspect database corruption, when a Vault cluster is inoperable, or for querying the free list.

Prerequisites

To follow the examples in this tutorial, you need the following.

• A Go development environment to install the tools

Learn monitoring repository

You can get some example Vault database files to use when following along in this tutorial by cloning the learn-vault-monitoring repository or downloading a zip file from GitHub.

1. Clone the repository:

   $ git clone https://github.com/hashicorp-education/learn-vault-monitoring.git

   Or, download the repository contents:

   

   Tip

   If you downloaded the zip archive, be sure to unzip the file before proceeding.

   This repository has supporting content for Vault learn tutorials. The content specific to this tutorial is in a subdirectory.

2. Change the working directory to learn-vault-monitoring/example-raft-database.

   $ cd learn-vault-monitoring/example-raft-database

3. Examine the directory structure of a data directory from a Vault server configured to use Integrated Storage.

   $ tree.├── raft│   ├── raft.db│   └── snapshots└── vault.db

   This is the most minimal example, with a raft directory containing the cluster coordinating data in raft.db, and a vault.db that holds the Vault operational data and secrets.

   Were this from a busier cluster, the snapshots directory would also contain some snapshot data.

With your example Vault data in place, you are now ready to inspect the files with the bbolt CLI and via a web browser with boltd.

Notes about storage architecture and terms

bbolt uses a database file represented by pages. The first 2 pages store important database metadata, while the third page holds the "free list". The free list keeps track of pages marked as deleted, which are free for writing to again.

All remaining pages contain the buckets. All keys and values get stored in buckets, represented by B-tree data structures.

Using the bbolt CLI

The bbolt CLI tool is part of the bbolt distribution.

1. Install bbolt into your $GOBIN path where GOBIN is the directory where go install and go get will place binaries after building main packages.

   $ go install go.etcd.io/bbolt/cmd/bbolt@latest

2. Check the bbolt installation to make sure it's on your $PATH.

   $ bbolt helpBbolt is a tool for inspecting bbolt databases. Usage:   bbolt command [arguments] The commands are:     bench       run synthetic benchmark against bbolt    buckets     print a list of buckets    check       verifies integrity of bbolt database    compact     copies a bbolt database, compacting it in the process    dump        print a hexadecimal dump of a single page    get         print the value of a key in a bucket    info        print basic info    keys        print a list of keys in a bucket    help        print this screen    page        print one or more pages in human readable format    pages       print list of pages with their types    page-item   print the key and value of a page item.    stats       iterate over all pages and generate usage stats Use "bbolt [command] -h" for more information about a command.

   Tip

   If you meet with an error instead, ensure that the go install succeeded and that bbolt is in your $PATH with which bbolt.

Check database file integrity

One useful feature of bbolt is the check command. It verifies that all pages are accessible or marked as freed, and also checks that no pages are double referenced.

If you suspect issues with the database file, the first thing you can do is check its integrity.

$ bbolt check vault.dbOK

The expected successful result is OK.

If there are issues with the database, the tool will emit errors.

Statistics

You can also gather essential statistics about the database.

$ bbolt stats vault.dbAggregate statistics for 2 buckets Page count statistics  Number of logical branch pages: 11  Number of physical branch overflow pages: 0  Number of logical leaf pages: 297  Number of physical leaf overflow pages: 135Tree statistics  Number of keys/value pairs: 1314  Number of levels in B+tree: 3Page size utilization  Bytes allocated for physical branch pages: 45056  Bytes actually used for branch data: 22775 (50%)  Bytes allocated for physical leaf pages: 1769472  Bytes actually used for leaf data: 1238220 (69%)Bucket statistics  Total number of buckets: 2  Total number on inlined buckets: 1 (50%)  Bytes used for inlined buckets: 317 (0%)

The stats command performs an extensive search of the database to track every page reference. It starts at the current meta page and recursively iterates through every accessible bucket. The command will emit errors if the database is corrupt.

From the example output, you learn that this database consists of 1314 key/value pairs and that there have been 1238220 bytes actually used for the leaf data.

More information is available from the help output with bbolt stats -h.

List buckets

The Vault database starts with 2 buckets: config and data.

You can list them like this:

$ bbolt buckets vault.dbconfigdata

List and get keys

The goal of this section is for you to learn how to list keys in more detail.

1. List all keys present in the config bucket.

   $ bbolt keys vault.db configlatest_configlatest_indexeslocal_node_config

   The keys in the config bucket contain information about the latest cluster configuration, including a list of cluster members, latest index state, and whether the local node is a voter.

2. Get count of the keys in the data bucket.

   $ bbolt keys vault.db data | wc -l    1311

   The data bucket holds the majority of all data in Vault, and should also represent the majority of the keys found in the database.

   Since there are several keys present in this example, you can explore a smaller subset of the keys contained in the data bucket.

3. List just the keys which make up the Vault core information.

   $ bbolt keys vault.db data | grep corecore/auditcore/authcore/autoloaded-licensecore/cluster/feature-flagscore/cluster/local/infocore/hsm/barrier-unseal-keyscore/index-header-hmac-keycore/keyringcore/leader/dceb963e-72a2-f1d4-ea8f-c903dbb57dd1core/local-auditcore/local-authcore/local-mountscore/lockcore/mastercore/mountscore/raft/tlscore/seal-configcore/shamir-kekcore/wrapping/jwtkey

   These keys represent the Vault core, and Vault uses them internally to contain critical configuration information.

4. A more practical example than demonstrating exploration is to find the count of valid tokens in the current Vault data. You can do so by counting the entries under the path sys/token/id/.

   $ bbolt keys vault.db data | grep 'sys/token/id/' | less | wc -l    228

5. Another practical example is confirming the existence of a particular policy by name. If for some reason you need to verify that the cluster has a particular named policy, you can list all names.

   $ bbolt keys vault.db data | grep 'sys/policy/' | cut -d '/' -f3adminscontrol-groupdefaultlab-userldap-devldap-userprometheusresponse-wrappingvaultron-example-namespace-nsvaultron-example-root-nsvaultron-example-token-adminvaultron-example-token-identityvaultron-sudo

   Note

   Along with listing keys, you can get key values for a limited subset of keys which allow unauthenticated access. Vault encrypts all user supplied inputs at rest, and you will not be able to decrypt them using the techniques and tools in this tutorial.

   One such exception is the list of cluster members found in the latest_config key in the config bucket.

6. Get the latest_config key values.

   $ bbolt get vault.db config latest_config}%vaultron-vault-010.10.42.200:8201%vaultron-vault-110.10.42.201:8201%vaultron-vault-210.10.42.202:8201%vaultron-vault-310.10.42.203:8201%vaultron-vault-410.10.42.204:8201

   These are the 5 servers which made up the Vault cluster these data are from.

If you have concerns about your terminal session, you can also try testing the output with file. Using the earlier examples from listing keys making up the Vault core, you can check if it might be safe to view the server's seal configuration in core/seal-config.

$ bbolt get vault.db data core/seal-config | file -/dev/stdin: JSON data

This is JSON data that provides some helpful information about the server's seal configuration.

$ bbolt get vault.db data core/seal-config{"type":"shamir","secret_shares":1,"secret_threshold":1,"pgp_keys":null,"nonce":"","backup":false,"stored_shares":1}

While generally not as useful as the vault.db, you can also inspect data in the raft.db. For example, to find the cluster member which was the last candidate voted a leader, you can check the LastVoteTerm key in the conf bucket.

$ bbolt get raft/raft.db conf LastVoteCand10.10.42.201:8201

Pages and the free list

This section is most useful for users of Vault 1.8.0 and higher who wish to learn more about the free list status.

1. To begin, list the first 2 pages.

   $ bbolt pages vault.db | head -n4ID       TYPE       ITEMS  OVRFLW======== ========== ====== ======0        meta       01        meta       0

   The list has 2 meta pages. You can use the most current one with the highest transaction ID value to find the freelist location.

2. Check the transaction ID for page 0.

   $ bbolt page vault.db 0 | grep -B2 'Txn ID'Freelist:   <pgid=306>HWM:        <pgid=477>Txn ID:     1712

3. Check the transaction ID for page 1.

   $ bbolt page vault.db 1 | grep -B2 'Txn ID'Freelist:   <pgid=174>HWM:        <pgid=477>Txn ID:     1711

   Page 0 has the highest Txn ID value, and is the meta page you should examine to find the root of the tree. Note that its Freelist value points to the correct freelist page ID.

4. List page 0 to find the root page ID.

   $ bbolt page vault.db 0 | grep RootRoot:       <pgid=235>

5. The root page ID is 235, and from here you can walk the entire tree of pages. Begin by listing page 235.

   $  bbolt page vault.db 235Page ID:    235Page Type:  leafTotal Size: 4096 bytesItem Count: 2 "config": <pgid=0,seq=0>"data": <pgid=212,seq=0>

   The output shows the pages representing the config and data buckets.

6. You can retrieve the data page at 212.

   $ bbolt page vault.db 212Page ID:    212Page Type:  branchTotal Size: 4096 bytesItem Count: 10 "audit/4487ee65-4449-6a32-d0af-518c380a0ede/salt": <pgid=139>"index-dr/pages/99": <pgid=58>"sys/expire/id/auth/userpass/login/lab-user-1/   h04a2ea39ac9f9b5fca932aab0f7ed25140db7e06e620926ffb43530a6911b61b": <pgid=162>"sys/expire/id/auth/userpass/login/lab-user-3/   h999915ecf7b1956d2287c913979ebe3ae6e2d35cca151f7da5e840ecf56b212d": <pgid=302>"sys/expire/id/auth/userpass/login/lab-user-5/   h74f9c9511f932e06e8f3e574947dd0e8f7279f2cc50cc3d3957320d539e2cfe7": <pgid=221>"sys/expire/id/auth/userpass/login/lab-user-7/   h44ba7953b65aceec11658b9371546743162fa86f999d7c86ecb3ce39abfa42e7": <pgid=217>"sys/expire/id/auth/userpass/login/lab-user-7/   hc6386db098e528405526f7b288703038a0358c4e90372e8f5b77e6f62e8b4222": <pgid=301>"sys/token/accessor/d1db3bdfd1cae11be408a41157996f32646f62af": <pgid=281>"sys/token/id/h60a215b77e0bc911cae8e988de36a8c1df736c57cf0e9023c0bb2430ea8538f3": <pgid=196>"sys/token/id/he32264f2dc56be09df07c4ed7b2b45987b440e459c9c622183527ddbf4ca41dc": <pgid=194>

7. Page 162 represents a username and password auth method lease id value. You can list this page ID to view its contents.

   $ bbolt page vault.db 162Page ID:    162Page Type:  branchTotal Size: 4096 bytesItem Count: 17 "sys/expire/id/auth/userpass/login/lab-user-1/h04a2ea39ac9f9b5fca932aab0f7ed25140db7e06e620926ffb43530a6911b61b": <pgid=64>"sys/expire/id/auth/userpass/login/lab-user-1/h4a13e2e920de1a69720984c2c643b172d7a451bcbf4d1bd90b1d5ba3bc56ed2d": <pgid=98>"sys/expire/id/auth/userpass/login/lab-user-1/h9f7b901d7bddc6407d82ef442c79bb6407804c5717c999ea74cc81a434f4efde": <pgid=41>"sys/expire/id/auth/userpass/login/lab-user-1/he93aa7f0493831dbdb5fada98f790f6b66eb1cbde985dbc49962629b7cd95fe0": <pgid=125>"sys/expire/id/auth/userpass/login/lab-user-3/h021e7aadf3008a2bb47ea917a6de8402439bd4637b78b11d56b372f5f1872627": <pgid=169>"sys/expire/id/auth/userpass/login/lab-user-3/h0cf7e2818d1f43250793cc3adf37d2cd65ea6421fe2ca1411b8ed59885f23b8b": <pgid=53>"sys/expire/id/auth/userpass/login/lab-user-3/h16a29ff005344742a5ca44a27d04bcc26467012229b9aba5977266e4a7e5888c": <pgid=118>"sys/expire/id/auth/userpass/login/lab-user-3/h20fef3012a68b1f3d9a0e3f42d08d5f22dfc9d5584582777538e8be1aeda9d78": <pgid=198>"sys/expire/id/auth/userpass/login/lab-user-3/h32be213efb25de70f7401a6955a3b716f0e4dcddc654629edfacaa5754742e41": <pgid=134>"sys/expire/id/auth/userpass/login/lab-user-3/h4109fc3b1d299624d81eabc9d7e5be4e0bf03d7bfe28d4b4a1de439b99f0f03e": <pgid=155>"sys/expire/id/auth/userpass/login/lab-user-3/h44e2ad5e382b80ceb2ab45d10710d9feb1468d0baf853c7f94df57e2b0db7aa8": <pgid=4>"sys/expire/id/auth/userpass/login/lab-user-3/h494323e1dcad8ae427e163d0fffcc6bbf317c8a6d9c3489d98d3cf22fe98cfdb": <pgid=181>"sys/expire/id/auth/userpass/login/lab-user-3/h5d7b4c21ae89c98952a45bb6bbae4698f2a1c0d7715022e2f9456092ab423b7f": <pgid=105>"sys/expire/id/auth/userpass/login/lab-user-3/h69290ffe0e01a35568c381ac6b4a6cea66c2947d30c4266d8410a45f65794216": <pgid=66>"sys/expire/id/auth/userpass/login/lab-user-3/h7396c835ae280c4115c01835e5f83b072eafef5480e758fe254bc7135fd610a2": <pgid=138>"sys/expire/id/auth/userpass/login/lab-user-3/h78b7a2a0688a3f6afe17e6d34cb9b96cbf926fc26a891300c220390fbc5adba8": <pgid=122>"sys/expire/id/auth/userpass/login/lab-user-3/h885ab1f14922f3310924bac61019c8a894dc3d55bbb1fa98f1cc71ddb1e2d318": <pgid=123>

8. Now you can view the leaf page for an individual lease id, for example the one at page ID 64.

   $ bbolt page vault.db 64Page ID:    64Page Type:  leafTotal Size: 4096 bytesItem Count: 3 "sys/expire/id/auth/userpass/login/lab-user-1/h04a2ea39ac9f9b5fca932aab0f7ed25140db7e06e620926ffb43530a6911b61b": 000000010232b3d254a9581ad479364b7ba235aa11a91bc225adc8c447e0ec267ba2c489492ff4d5df7891aa3fc91de61cd30bab462a91674f13117f613e9f6ae7bb1e903eaaed485db5bde4e98debd601a0f5303fcc5f0244df3d4795f1b5a5db7fb8ff9101e8dabd43b092902043af6dc59755acafe0730850410d37c8717f65cd09e015c15ddc665515b37ca8be8f51d67abc7ab093183a8d11a4c2edf56bce6a53691e0b066c7b2c37ff3d7062169850f79fa8d89602ad4b6bb5fd3c7d741939fa8bc3dd84373a1e86f79ebabcc1ff91c38728d57806483d2d4e03d6aef414ba0c8826ddc9b4d953220c7dda59cde87b71a4db778e365dd41f27e6d0f7089074aa82eb0ced12d5cf3cd9c241094e0ba034fdaab13f264adf4a73341a3944c08b16ba7581cc5c546cfea21909adde5792530aec8c45d6b3a491541f48ad194164a8ffd773908247c7f649f9d07e962a2addabc2742aaea6864417ae8e3e86a914652f2f72cc1b8abab2be7348208c46ca485887a7bb12579ea0e6e13f6d61ada5f339f1a5f2fe8999a46efb05ce9582d942094e2055828642c69e67648de867d9e1e14bdfc5f526e10c569de74e695b1c58e649ce50d02eab8268456010a3126fc079012af2dfede424913fcd756ea1f7498a7eb584f03bfc59b408a3205bbed61c931131cd2ae67a2dd985439a2924efb73f9b49ba7acdeea23e67226fbc4225f158e2975b7cd363661f7efb1ee3ae2ac7ff0930e2af8fe59478920f3b86fd840dd1280c901d2ad1ea174c850b893612d58fcb5d45a32e5e08a1167925a2a506d57aef0c0e0cf5d11205d346737904dfa3f5c89c624728241d7b98176a26dc8deb5e6e7e9ddbe6df7847922d7976920ad699f79a94c2465627f8b41b1680170ad2b5c07d872739b73e78f3b3b44a8b9a00438316c611d66d99ec6db0bc07522324d3b1c9be79f76cfd179e6b2e65a3afc771c3e9a84dac58471fc1e54390b00a3c4f0166e67fce6ff121527abe45b363e63404c6f0290687bdc7fa1ff29ae61f35f4bad69ccadc71da8e2af4632ac565e931cfa0ad4919fa802a2c1b5e557796b9e8ca4b93b25b2c991a87079c8d18fc03debb416d8be6ae3f4d10b49f65fef46415428c14111d4f20a767b1218da34f7f3343bde1268461e547fd74d8841cd0fc22acd1116dda4e4be9d8bbd100b125318e22ec04568ba2603c9aebc249b9fdc7837557d87bf77ff8a496c159a25bd531c3f5fcb5a11267f52c1fdfd51bc330fb3ed84903d7755850be54fa7cdf9726a9da77e330f1dd934d84c3e6b4afe13d4ba574fe95695ece561c8e9b936b56ff905fc045b07c9bc2494bf48ce60588bef4ec5b213384ad97af5e9a82dc3ffdd8721931a30aa707262d8ef796f300eed4eb3f89d2ae7b0586a4d4a889ccb55d29ce348a517fe40622b1addb8e2f24d92b0271ea37bd7674de741749af6ded672abcbc5d238aad4f39c8496f996db894cb9029ab8fbbe4b7c29924b67606daaef0f2f686cc5362544306f1fba53d380f1ed5777e8f38d47e9fbf011f45...snip...

   The result is a key sys/expire/id/auth/userpass/login/lab-user-1/h04a2ea39ac9f9b5fca932aab0f7ed25140db7e06e620926ffb43530a6911b61b with a long hexadecimal string representing the encrypted value.

   You can explore the entire tree this way.

9. With the freelist itself, recall that meta page 0 was the page ID containing the most recent Txn ID. Check the Freelist value for meta page 0.

   $ bbolt page vault.db 1 | grep -B2 'Txn ID'Freelist:   <pgid=174>

10. The freelist page ID is 174. Go ahead and list it next.

    $ bbolt page vault.db 174 | head -n 15Page ID:    174Page Type:  freelistTotal Size: 4096 bytesItem Count: 31Overflow: 0 5758151189194212235306313

    Note the presence of 31 pages in the freelist.

    The other way to verify a page is free, is to check for its page ID in the pages output, and ensure that free appears next to it.

11. Confirm that page 313 is free.

    $ bbolt pages vault.db | grep -w 313313      free

Using the boltd web UI

Another tool from the original BoltDB project, boltd exists to explore the database visually.

You can use boltd by attaching it to a Vault database and connecting a web browser to the web UI it exposes.

1. Install boltd.

   $ go get github.com/boltdb/boltd/...

2. Start boltd and attach it to vault.db.

   $ boltd vault.dbListening on http://localhost:9000

3. Access http://localhost:9000 in a browser.

   

   The user interface is minimal, and defaults to displaying a view of pages.

   You can use it to browse facets of your Vault data like the bbolt CLI tool, by navigating links or jumping directly to page numbers in the Go to page input.

   You can also explore a page usage histogram by selecting Show Page Usage.

   

Summary

You have learned about how to inspect data in the bbolt database file for a Vault server using the Integrated Storage backend. You learned about 2 tools which you can use for this purpose along with caveats around their usage.

You can learn more about inspecting Vault data in Inspect Data in Integrated Storage and Inspecting Data in Consul Storage.

Clean up

When you finish the hands on lab, you can clean up by changing to the parent directory and removing the learn-vault-monitoring directory.

$ cd ../..

rm -rf learn-vault-monitoring

Help and reference

• bbolt
• boltd