Interfacing with the Bitcoin blockchain services

Toptal academy Blockchain lectures #3

2018-02-28

ivan.voras@toptal.com

Today's agenda

• Wallets and passwords
• "Accounts" (do not exist)
• JSON-RPC
• Important Bitcoin APIs
• Use case: a web shop
• Data flow for detecting a transaction
• Ethereum's RPC

A note about wallets and passwords

A "wallet" is a small database file containing (usually) one or more private keys, and a cache of unspent transaction outputs available for spending by this wallet.

The private key is usually encrypted with a user-selectable password. I.e. it is "locked".

Some operations cannot be done on a locked wallet.

When the user enters the correct password, it "unlocks" the wallet: the software decrypts the private key into RAM, so that certain operation (such as spending coins) can be done.

Bitcoin "accounts"

In the early days of Bitcoin, there was an idea that a single wallet can contain multiple private keys grouped into named "accounts".

So when some software requests "account balance", it sums up all the unspent coins belonging to a single "account."

This is *only* a front-end concept, implemented by the original Bitcoin Core wallet / node software. There are no "accounts" in the blockchain, only addresses.

Nodes / Wallets / Daemons...

To bootstrap the concept, the original Bitcoin Core software did everything in a single executable:

• Recorded and validated the blockchain
• Mined new blocks
• Relayed blocks and transactions to the P2P
  (peer-to-peer) network
• Acted as "wallet software", i.e. maintained a wallet file with a private key and allowed transactions to be created
• Acted as a RPC server between 3rd party software and the blockchain & P2P network
• Acted as a RPC client

Nodes / Wallets / Daemons...

• Nodes are all instances of compatible software connected to the P2P network
• Full nodes are instances which also contain the entire blockchain (i.e. all the 145 GB)
• Wallets are software which deals with private keys and allows transactions to be signed and sent into the network
• Light wallets are wallets which do not store the entire blockchain (or block headers), just deal with transactions
• Payment gateways offer customers to pay (with BTC in this case), monitor addresses for received payments, and notify the web shop that the payment has arrived.
• "Daemon" an archaic word for software running as a background service on a server

Command lines and config files

The Bitcoin Core software can be configured using almost exactly the same configuration options either by command-line arguments or by adding them to a textual configuration file.

Starting

bitcoind -rpcuser=user -rpcpassword=password

Is equivalent to having a file with these lines:

rpcuser=user
rpcpassword=password

And then starting bitcoind with

bitcoind -conf=/path/to/config.file

A Bitcoin full note syncing

Accessing "RPC" commands

The same set of operations / commands can be done in 3 ways:

1. On the shell command-line, invoking "bitcoin-cli COMMAND". In this way, bitcoin-cli acts as a JSON-RPC client!
2. In the GUI, Help->Debug Window->Console
3. Issue real JSON-RPC calls (if the server is started and configured)

The same commands are available over all three interfaces.

The "Console"

Enter RPC commands here

JSON-RPC

Looks like this:

{"jsonrpc": "2.0", "method": "subtract", "params": [42, 23], "id": 1}

It's a light-weight text-only protocol for doing
remote procedure calls (request-response style).

There are libraries for all major development environments /
programming languages.

JSON-RPC and Bitcoin

Bitcoin implements 60-ish JSON-RPC calls
(depending on the version).

It enables third party software to do almost all available operations with transactions and blocks, without interfacing with the actual Bitcoin P2P network, which uses a binary protocol.

The list of all official RPC APIs is available at:

https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list

Enabling JSON-RPC in Bitcoin

Bitcoin software (either the command-line interface variant or the GUI variant) needs to be started with additional options:

-server

-rpcuser=username

-rpcpassword=password

(and possibly -rpcallowip, -rpcbind, etc.)

Note: username and password are plaintext over the network!
The usual way of doing this is over localhost only!

"RPC" commands in CLI and GUI

A RPC method can be documented as

method_name([args...])

for example:

getblockhash(block_no)

When used in the CLI or the GUI console, the arguments are passed separated by spaces, for example:

getblockhash 3412 (in GUI console)
bitcoin-cli getblockhash 3412 (in CLI)

Important API: getblockcount

Returns the number of block currently known to be in the blockchain by this node.

This number increases as new blocks are added or historical blocks synchronised.

Example response:

511326

Important API: getblockhash

Returns the hash of a block identified by its "height" or "index".

Canonically, blocks should always be referenced by their hash, but it's often convenient to reference them by their sequential place in the block (genesis block is at index 0).

Example:

getblockhash(123456)

Result:

0000000000002917ed80650c6174aac8dfc46f5fe36480aaef682ff6cd83c3ca

Important API: getblock

Returns the structure of a block identified by its hash. The returned value is a JSON object.

Example:

getblock("0000000000002917ed80650c6174aac8dfc46f5fe36480aaef682ff6cd83c3ca")

Result:

(next slide)

{  "hash": "0000000000002917ed80650c6174aac8dfc46f5fe36480aaef682ff6cd83c3ca",
  "confirmations": 387874,
  "strippedsize": 4179,
  "size": 4179,
  "weight": 16716,
  "height": 123456,
  "version": 1,
  "versionHex": "00000001",
  "merkleroot": "0e60651a9934e8f0decd1c5fde39309e48fca0cd1c84a21ddfde95033762d86c",
  "tx": [
    "5b75086dafeede555fc8f9a810d8b10df57c46f9f176ccc3dd8d2fa20edd685b",
...
    "864a102aeedf53dd9b2baab4eeb898c5083fde6141113e0606b664c41fe15e1f"
  ],
  "time": 1305200806,
  "mediantime": 1305197900,
  "nonce": 2436437219,
  "bits": "1a6a93b3", "difficulty": 157416.4018436489,
  "chainwork": "000000000000000000000000000000000000000000000000541788211ac227bc",
  "previousblockhash": "0000000000000b60bc96a44724fd72daf9b92cf8ad00510b5224c6253ac40095",
  "nextblockhash": "000000000000129f5f02be247070bf7334d3753e4ddee502780c2acaecec6d66" }

Important API: getrawtransaction

Returns the content of the transaction. Can only be used for arbitrary transactions if bitcoind was started with -txindex.

Example:

getrawtransaction 864a102aeedf53dd9b2baab4eeb898c5083fde6141113e0606b664c41fe15e1f true

This method accepts two arguments, the first is the tx hash, the second is whether to return a JSON object (true) or a hex dump of the transaction (false, the default).

Result: (next slides)

{
  "txid": "864a102aeedf53dd9b2baab4eeb898c5083fde6141113e0606b664c41fe15e1f",
  "hash": "864a102aeedf53dd9b2baab4eeb898c5083fde6141113e0606b664c41fe15e1f",
  "version": 1,
  "size": 257,
  "vsize": 257,
  "locktime": 0,

...

  "vin": [
    {
      "txid": "38be76d39f91dbcad95c6a9ebf79a80b6e60eb50e9b85362cefb61788381056a",
      "vout": 0,
      "scriptSig": {
        "asm": "304402206c7308a8fa8d45c082da032880270c10d436d2a4623ba6e13819d45eecf0f3b90220356f4e9855a101487b680d0b4e69b10c509015293236f2cfbaf9da6cb6791466[ALL] 04cfd5868f564bae61bf22479e8d22e030fbdbf9a01a9a0b61ddfb580a9552b47b15f8a28e7c40b8fe73d6b9c0e0cdd1527266602b327dcc272bd8c1a33b87e4de",
        "hex": "47304402206c7308a8fa8d45c082da032880270c10d436d2a4623ba6e13819d45eecf0f3b90220356f4e9855a101487b680d0b4e69b10c509015293236f2cfbaf9da6cb6791466014104cfd5868f564bae61bf22479e8d22e030fbdbf9a01a9a0b61ddfb580a9552b47b15f8a28e7c40b8fe73d6b9c0e0cdd1527266602b327dcc272bd8c1a33b87e4de"
      },
      "sequence": 4294967295
    }
  ],

...

  "vout": [
    {
      "value": 0.07889992,
      "n": 0,
      "scriptPubKey": {
        "asm": "OP_DUP OP_HASH160 ceb552bdf23d002aed04c317c92cf8987e550df9 OP_EQUALVERIFY OP_CHECKSIG",
        "hex": "76a914ceb552bdf23d002aed04c317c92cf8987e550df988ac",
        "reqSigs": 1,
        "type": "pubkeyhash",
        "addresses": [
          "1KqyR5L31Gkig5GJwHugbhmcyM2RHyv5xJ"
        ]
      }
    },

...

    {
      "value": 0.01000000,
      "n": 1,
      "scriptPubKey": {
        "asm": "OP_DUP OP_HASH160 586ce63c59b47a3ad08ccac6f132dc04ccbea0e2 OP_EQUALVERIFY OP_CHECKSIG",
        "hex": "76a914586ce63c59b47a3ad08ccac6f132dc04ccbea0e288ac",
        "reqSigs": 1,
        "type": "pubkeyhash",
        "addresses": [
          "194YvLncdC6L91xMF48TiPv6zB7cqe431K"
        ]
      }
    }
  ],

...

  "hex": "01000000016a0581837861fbce6253b8e950eb606e0ba879bf9e6a5cd9cadb919fd376be38000000008a47304402206c7308a8fa8d45c082da032880270c10d436d2a4623ba6e13819d45eecf0f3b90220356f4e9855a101487b680d0b4e69b10c509015293236f2cfbaf9da6cb6791466014104cfd5868f564bae61bf22479e8d22e030fbdbf9a01a9a0b61ddfb580a9552b47b15f8a28e7c40b8fe73d6b9c0e0cdd1527266602b327dcc272bd8c1a33b87e4deffffffff0248647800000000001976a914ceb552bdf23d002aed04c317c92cf8987e550df988ac40420f00000000001976a914586ce63c59b47a3ad08ccac6f132dc04ccbea0e288ac00000000",
  "blockhash": "0000000000002917ed80650c6174aac8dfc46f5fe36480aaef682ff6cd83c3ca",
  "confirmations": 387874,
  "time": 1305200806,
  "blocktime": 1305200806
}

Some of this data is in tx, some is synthetic (e.g. confirmations, blocktime).

Note: pretty much undocumented - "whatever works"

Important API: getnewaddress

Generates a new address for receiving coins, usually by generating a new keypair (unless HD wallets are used). 

Example:

getnewaddress

Response:

12xBPi37rXL2DD9tTY2UMKcTqLEb1dEHvu

Important API: sendtoaddress

Sends coins to an address.

Example:

sendtoaddress 12xBPi37rXL2DD9tTY2UMKcTqLEb1dEHvu 0.1

Response:

273826051ac6919893b5efd65e0c83d201b1d3f4283f90febcd270c361df168f (txid)

Important API: getrawmempool

Returns a list of all transaction IDs currently in the "memory pool" (i.e. waiting to be included in blocks, "unconfirmed").

Example:

getrawmempool

Result:

[   "ec07f582ea477c29f11ac39310adc828f0a54dc7cb4581c88373103fa7952b24",
  "2ab68a41b503a15c5cf2bb1da5290690a6023b0e13a2d4ec5748c9bd9021ddd5",
...
  "b360ae7f9f0794ae1f5a8266ba7681ae98bff7f33f2d632100ebd2d57b8601d5",
  "3b447ccdfbc92731adab5bf1231a38dc46f692b0210ef3dba35a4d9c5084fc17"
]

There can be tens
of thousands of
unconfirmed transactions
at a time.

Often used to "preview" transactions.

"RPC" commands in CLI or GUI

Homework

Start your Bitcoin node (you don't have to wait until it's synced) and using any type of interface (the GUI "console", the command line, or a JSON-RPC client), call these methods:

getblockcount()
getblockhash(0)
getblock("000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f")
getrawmempool()

Note that in the CLI or GUI console, they are inputed as:
"getblockhash 0", etc.

How to make a webshop accept Bitcoin

Supposing there is a web shop which wants to accept Bitcoin.
The general steps are:

1. The webshop generates a new address to receive the payment
2. The webshop (optionally) presents this address as QR code to the client
3. The client pays with his wallet
4. The webshop monitors the blockchain to detect that the payment transaction has been included
5. The webshop (optionally) waits until the relevant transaction has N confirmations

Generating a new address each time

It's a "best practice" when Bitcoin is concerned (not universally accepted) to generate a new address for each individual payment, especially by shops.

Address generation is trivial, fast, and costs nothing.

Benefits:

• Better pseudonimity: money is harder to trace if each payment is made to a different address
• It's easier to detect if an address receives the exact number of coins.

Note: (usually) NO REFUNDS, NO CHANGE!

A payment gateway to support a shop

The algorithm for processing payments

A = getnewaddress()
notify_user_to_pay_to(A, amount)
while not A in R = getrawmempool(): sleep()
if get_amount(getrawtransaction(get_txid(A, R))) == amount:
   notify_user_ok_unconfirmed()
else:
   notify_user_pebkac()
while not A in getblock(BC = getblockcount()): sleep()
notify_user_confirmed(1)
while getblockcount() - BC + 1 < N_CONFIRMATIONS: sleep()
notify_user_confirmed(N_CONFIRMATIONS)

Make an address monitoring script

Homework

Make a script which interfaces with Bitcoin's JSON-RPC services to perform the following operations:

• Whenever a new block is detected, print out its hash
• For every new block, if it contains a transaction where the address which receives a payment contain the capital letter "E", print out the address and the value of its payment
• For every new block, if a transaction in it pays more than 10 BTC to any address, print out "BINGO!" and the address.

Ethereum's RPC

Much better designed than Bitcoin's.
(actually has some useful documentation)

Relies on a JavaScript / JSON-ish view of the world, rather than the C/C++-ish world of Bitcoin.

The API is divided into "libraries". The node software (e.g. "geth") needs to have a certain library enabled / exposed before it can be used from the RPC.

Example:

geth --rpcapi eth,web3,personal,txpool --rpc

Ethereum's APIs equivalent to Bitcoin's

• eth_blockNumber() -- returns blockchain length
• eth_getBlockbyHash(hash) -- returns block data
• eth_getBlockbyNumber(n) -- returns block data
• eth_getTransactionByHash(hash) -- returns tx data
• eth_getTransactionByBlockHashAndIndex(hash, n) -- ditto
• eth_sendTransaction({"from":...,"to":...,"value":...})

Ethereum APIs are better defined and more complex,
especially when dealing with smart contracts.

https://github.com/ethereum/wiki/wiki/JSON-RPC

THE END

ivan.voras@toptal.com

Blockchain lecture #2: Interfacing with Bitcoin software

February 2018

Q&A?